Operating Method of a Processor Device

Abstract

To provide a method of operating a processor device.SOLUTION: A method of operating a processor device for displaying angiographic and intravascular representations of a blood vessel includes generating a set of optical coherence tomography image data in response to distance measurement of the blood vessel obtained during a pullback of a probe through the blood vessel using an optical coherence tomography system, where the set of optical coherence tomography image data comprising a plurality of cross-sectional images at a plurality of positions along the blood vessel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] In part, the present invention generally relates to the field of imaging and data collection of the vascular and peripheral vascular systems.

Background Art

[0002] Cardiologists performing interventions employ various diagnostic tools during catheterization procedures for therapy planning, guidance, and evaluation. Fluoroscopy is generally used to perform angiographic imaging of blood vessels. And such vascular imaging is used by physicians to diagnose, locate, and treat vascular diseases during interventions such as bypass surgery or stent placement. Intravascular imaging techniques such as optical coherence tomography (OCT), acoustic techniques such as intravascular ultrasound (IVUS), and other techniques are also valuable tools that can be used instead of or in combination with fluoroscopy to obtain high-resolution data regarding the condition of a given subject's blood vessels.

[0003] The fractional flow reserve (FFR) can also be used to evaluate blood vessels during imaging and angiography. Intravascular OCT, IVUS, and FFR are invasive catheter-based systems that collect optical data, ultrasonic data, and pressure data respectively from within blood vessels or regarding a target specimen. Angiography is a non-invasive X-ray imaging method that collects data from outside the body during the injection of an X-ray opaque contrast medium.

[0004] Intravascular optical coherence tomography is a catheter-based imaging modality that uses light to peer into the walls of coronary arteries and generate images for study. OCT can provide in-vivo tomography at video rate within diseased blood vessels at a resolution of the order of micrometers by utilizing coherent light, interferometric spectroscopy, and micro-optics. By using an optical fiber probe to view subsurface structures with high resolution, OCT becomes particularly useful for minimally invasive imaging of internal tissues and organs. Such a level of detail made possible by OCT enables clinicians not only to monitor but also to diagnose the progression of coronary artery disease.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Considering the complexity of the above-mentioned various technologies and the complexity of the datasets generated by each related technology, it takes a great deal of time to perform co-registration between technologies based on two images such as OCT and angiography. Therefore, problems remain regarding real-time co-registration of intravascular data and angiography image data. Some co-registration techniques rely heavily on interaction with the user. Unfortunately, due to the need to manually match corresponding points in the images, the long waiting period for the algorithm to return co-registration, and important user interactions during co-registration such as verification of the final result, such methods impose a heavy burden on the operator, making them difficult to implement in many clinical scenarios. Furthermore, other methods use data from asynchronous, i.e., third-party controlled sources, which makes timing irregular. Additionally, contrast agents such as dyes are used with some intravascular imaging modalities that interfere with other non-invasive imaging modalities, which may cause imaging artifacts and errors that interfere with co-registration between such modalities.

[0006] Therefore, there is a need to address one or more of the problems identified above related to intravascular imaging and angiography imaging. Embodiments of the present invention address these and other problems.

Means for Solving the Problems

[0007] One embodiment of the present invention relates to a method for alignment between two imaging modalities such as angiography and OTC. One embodiment of the present invention relates to one or more methods for performing co-registration between an angiography image and an OCT image.

[0008] One embodiment of the present invention relates to a method for detecting a marker band that is stationary on a frame without using a contrast agent such as a dye and using a contrast agent. Further, one embodiment of the present invention provides tracking of such a marker band as it moves through the lumen of a blood vessel such that the marker band is tracked in subsequent pullback frames, which includes tracking from frames without a contrast agent to frames with a contrast agent.

[0009] In one embodiment, the period for alignment between about 20 frames to 100 frames of angiography image frames and about 100 frames to about 1500 frames of OCT image frames ranges from about 2 seconds to about 30 seconds. In one embodiment, the alignment of angiography image data and OCT image data obtained during OCT pullback is co-registered in less than about 10 seconds. In one embodiment, the pullback of the data collection probe ranges from about 4 seconds to about 10 seconds. In one embodiment, the frames of the angiography method are obtained in real time using a frame grabber. The frames of the angiography data are captured in synchronization with the OCT image data frames obtained as a result of the pullback.

[0010] In one embodiment, the co-registration method co-registers the OCT frames of the image data obtained during pullback imaging with the frames of the angiography data obtained during such pullback within an alignment period of about 3 seconds to about 5 seconds.

[0011] In one embodiment, the present invention relates to an image data processing system including an OCT system, one or more computing devices, and a user interface configured to perform imaging during pullback of a data collection probe having a marker passing through a blood vessel and generate OCT image data with time stamps related to the blood vessel, wherein the frame grabber is configured to obtain time-stamped frames of angiography image data related to the blood vessel.

[0012] In one embodiment, the video capture of the angiography image data is generated by an OCT system. In one embodiment, the user manually specifies a marker band on the angiography image. In one embodiment, the specified marker band appears on the angiography image without using a contrast agent. In one embodiment, the user interface includes a longitudinal OCT image panel, a cross-sectional OCT image panel, one or more controls, and an angiography image panel. In one embodiment, the user interface includes a registration control, i.e., a button, that causes a computing device to execute one or more software modules configured to overlay and register OCT image data and angiography image data. In one embodiment, a timestamp is used to provide a primary alignment between an angiography frame and its corresponding OCT frame, thereby allowing the position of the nearest angiography frame to be determined for each OCT frame, and vice versa. Further, timestamped events such as pullback start and stop are also recorded to assist the coregistration process.

[0013] In one embodiment, a cursor or other identifier on the angiography image indicates the location of the OCT catheter reference marker that coincides with the selected OCT pullback frame. In one embodiment, the cursor or other identifier can also indicate the proximal reference frame and the distal reference frame selected by the user within which the MLA is calculated, and can also indicate the average diameter of the blood vessel. The scrolling of the overlaid and registered OCT and angiography images can be controlled via the OCT L mode, i.e., a cursor on the angiography frame as a remote controller or as part of the user interface.

[0014] In one embodiment, a filter kernel, such as a convolution matrix, is implemented as a matrix that includes elements configured to perform image processing for performing row and column, as well as intensification, sharpening, pattern recognition, detection, tracking, and other image processing tasks. The filter kernel can be used in various preprocessing and other processing stages to perform image processing on angiography image data or other image data.

[0015] In one embodiment, the present invention relates to a processor-based method for displaying angiographic and intravascular representations of blood vessels. The method includes: generating a set of OCT image data including a plurality of cross-sectional images at a plurality of positions along a blood vessel in response to distance measurements of the blood vessel using an optical coherence tomography system; generating a set of angiographic image data including a plurality of two-dimensional images at a plurality of positions along the blood vessel; and registering the angiographic image and the OCT image by overlaying based on one or more of a time stamp, a relationship between time stamps, a match between features in the OCT image and features in the angiographic image, and determination of a centerline of the blood vessel and overlay registration of the OCT image and the angiographic image using the centerline.

[0016] In one aspect, the present invention relates to a processor-based method for displaying angiographic and intravascular representations of blood vessels. The method includes: generating a set of optical coherence tomography image data in response to distance measurements of the blood vessel obtained during retraction of a probe through the blood vessel using an optical coherence tomography system, the set of optical coherence tomography image data including a plurality of cross-sectional images at a plurality of positions along the blood vessel; generating a set of angiographic image data using an angiographic system during the retraction of the probe through the blood vessel using the optical coherence tomography system, the set of angiographic image data including a plurality of two-dimensional images obtained at different points in time during the retraction; displaying a first panel including a first longitudinal view of the blood vessel generated using the OCT image data; and displaying a second panel including a frame of the angiographic image data identifying the blood vessel using one or more points within the frame and a blood vessel centerline passing through the one or more points. 2 including.

[0017] In one embodiment, the present invention further includes registering and overlaying the OCT image data and the angiography data using the vascular centerline to generate continuous alignment of the marker to be tracked, wherein the marker to be tracked is disposed on an OCT data collection probe. In one embodiment, the present invention further includes registering and overlaying the OCT image data and the angiography data such that the frame identifier in the first longitudinal view is changed by selecting points along the vascular centerline through a user interface. In one embodiment, the present invention further includes excluding candidates for the marker to be tracked based on the pull-back length and / or the pull-back speed by performing an iterative search using the pull-back speed or the pull-back length, based on positions that the marker can take.

[0018] In one embodiment, the present invention is such that the vascular centerline is generated using a shortest path technique and a plurality of processing steps from Dijkstra's algorithm. In one embodiment, the present invention further includes the step of removing a guide catheter image from one or more frames of the angiography data using an overlay of luminance profiles. In one embodiment, the present invention is such that the vascular centerline is generated using path information generated from one or more angiography frames in the substantial absence of a contrast agent solution. In one embodiment, the present invention further includes generating a confidence score for each of the detection of the angiography data and the optical coherence tomography data, and the registration of the overlay between the angiography data and the optical coherence tomography data.

[0019] In one aspect, the present invention is a method for detecting an intravascular probe marker, comprising: obtaining a first frame of angiography image data that substantially does not contain a contrast agent and contains the intravascular probe marker; obtaining a second frame of the angiography image data that contains contrast agent image data near the intravascular probe marker; and detecting the intravascular probe marker in the first frame and the second frame.

[0020] In one embodiment, the present invention includes the steps of applying an image processing transformation to the second frame to remove or modify features within the second frame; and increasing the luminance of a plurality of pixels, wherein the plurality of pixels includes a guide wire image within the second frame. In one embodiment, the present invention further includes the step of generating an average luminance value for a plurality of images and subtracting the average luminance from the first or second frame. In one embodiment, the present invention further includes applying a bottom hat operator to the second frame and applying a morphological closing operation.

[0021] In one embodiment, detecting the intravascular probe marker includes filtering candidate markers including pixels of the first frame and the second frame by applying a multi-scale LoG (Laplacian of Gaussian) operator to the first frame and the second frame; and performing non-maximum suppression processing to identify small blocks having a maximum value in the vicinity of the pixels.

[0022] In one embodiment, the present invention further includes applying a Euclidean distance transform to a binary image to generate a guide wire-based potential. This method includes applying an exponent to a negative fractional power of the distance transform to calculate a potential function. In one embodiment, this method further includes determining a plurality of geodesic distances based on the guide wire-based potential using a fast marching method.

[0023] In one embodiment, the present invention further includes removing shadows from the first frame and the second frame; increasing the contrast level of a guide wire of one of the first frame or the second frame; and performing morphological image reconstruction for each marker candidate. In one embodiment, the present invention further includes processing a plurality of pull-back frames using a Hessian-based vascularity filter; and tracking the intravascular probe marker from one of the first frame or the second frame through all of the pull-back frames through the plurality of pull-back frames using template matching. In one embodiment, the present invention further includes tracking the intravascular probe marker through a plurality of frames obtained during pull-back using the Viterbi dynamic programming method.

[0024] In one aspect, the present invention relates to a processor-based method of overlay registering angiographic image data with intravascular image data obtained during pull-back through a blood vessel. The method includes: storing a plurality of frames of optical coherence tomography data in memory; storing a plurality of frames of the angiographic image data in memory; processing the plurality of frames of the angiographic image data such that one or more shadows are substantially reduced; detecting a catheter in the plurality of frames of the angiographic image data; removing the detected catheter in the plurality of frames of the angiographic image data; generating a vessel centerline for the plurality of frames of the angiographic image data; detecting a probe marker in the plurality of frames of the angiographic image data; tracking the position of the probe marker along one or more of the vessel centerlines; and overlay registering the plurality of frames of the angiographic image data with the plurality of frames of the optical coherence tomography data using the tracked position.

[0025] In one embodiment, the present invention further includes a step of generating a score indicating a level of confidence in the overlay registration between a frame of the angiography image data and a frame of the optical coherence tomography data. In one embodiment, the step of removing the detected catheter in the present invention is performed using an overlay of luminance profiles generated based on sampling of the region of the detected catheter.

[0026] In one embodiment, the step of overlay registering the plurality of frames of the angiography image data and the plurality of frames of the optical coherence tomography data in the present invention includes generating a co-registration table using a computing device, and the co-registration table includes an angiography image frame, a plurality of OCT timestamps per frame, a plurality of angiography timestamps per frame, and an optical coherence tomography image frame. In one embodiment, the present invention further includes displaying a stent representation on the OCT image and the angiography image within a user interface using the co-registration table and a computing device.

[0027] In one embodiment, the present invention further includes identifying the side branch in one or more of the OCT images and the angiography images using the co-registration table and a user interface configured to display the side branch. In one embodiment, the present invention further includes adjusting a pull-back speed change by setting an interval between frames of OCT data based on the co-registration table, and displaying a longitudinal view on the user interface based on the interval.

[0028] The figures are not necessarily to scale and, overall, emphasize principles of illustration. The figures are to be regarded as illustrative in all respects and not intended to limit the invention, the scope of which is defined only by the claims.

Brief Description of the Drawings

[0029]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

Figure 6I

Figure 6J

Figure 6K

Figure 6L

Figure 6M

Figure 6N

Figure 7A

Figure 7F

Figure 8A

Figure 8C

Figure 9A

Figure 9E

Figure 10A

Figure 10B

Figure 11A

Figure 11B

Figure 12

Figure 13A

Figure 13B

Figure 14

Figure 15

Best Mode for Carrying Out the Invention

[0030] The following description refers to the accompanying drawings that illustrate specific embodiments of the invention. Other embodiments are contemplated and modifications may be made to the above embodiments without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to limit the invention, and the scope of the invention is defined by the claims.

[0031] As described above, there are problems related to vascular diagnostic systems and peripheral vascular diagnostic systems, such as issues related to implementing co-registration for multiple imaging techniques such as angiography, OCT, and IVUS. In part, the present invention relates to various systems, components of various systems, and methods for collecting data from a subject and using it in a catheter laboratory or other facility to help improve one or more of these limitations. The data collected typically relates to a patient's cardiovascular or peripheral vascular system and may include image data, pressure data, heart rate, and other types of data as described herein.

[0032] Furthermore, in one embodiment, the image data is collected using an optical coherence tomography probe and other related OCT components. In one embodiment, the image data is collected using an IVUS probe and other related IVUS components. Further, in one embodiment, the pressure data is collected using an FFR probe and other related FFR components. In yet another embodiment, EKG, heart rate, and other subject data are collected using electrodes and other related components.

[0033] Furthermore, some embodiments of the present invention are suitable for handling multiple imaging modalities. Thus, in part, the present invention relates to a multimodal diagnostic system and components of a multimodal diagnostic system configured to overlay and register one or more of OCT, IVUS, FFR, and angiography. OCT data and image processing results can be used to improve the processing of frames of angiography images by providing input to a software module dedicated to angiography.

[0034] Also, IVUS imaging features can be incorporated into a data collection probe that is used in conjunction with collecting angiography data in one embodiment. Additionally, FFR pressure measurements can also be performed using a suitable pressure transducer and probe. In one embodiment, the FFR data collection probe or transducer can include a wireless transmitter, and use a wireless receiver to receive FFR data and communicate the FFR data to a server. The comparison and coregistration of OCT images and / or IVUS images with angiography images is achieved by interfacing the system with an angiography device or a hospital data network where angiography data is stored.

[0035] In one embodiment, a user such as a clinician interacts with a workstation or server having a relevant user interface for displaying images of longitudinal cross-sections from the top to the bottom of a subject's blood vessel or cross-sections substantially parallel to the longitudinal axis of the blood vessel. The coregistration process may include various steps as well as image processing and feature detection software modules. In one embodiment, the user or the system activates intravascular imaging during the acquisition of angiography images. The blood vessel being imaged intravascularly and the imaging catheter can be displayed as part of a graphical user interface. The boundaries of the lumen of the blood vessel can be identified and associated in each intravascular image and angiography image, thereby maintaining the same blood vessel segment in different views.

[0036] Since the imaging catheter is introduced by a guide wire, the guide wire can be used as an anchor path and can also be used to provide directional information such as which endpoint is distal and which endpoint is proximal within the associated imaging segment. In one embodiment, the guide catheter slides along the guide wire to position the probe tip having one or more imaging devices within the blood vessel. In one embodiment, the angiography image data is processed to remove the guide catheter from the image after the identification of the guide catheter.

[0037] In one embodiment, one or more software modules are used to generate and track the centerline of a blood vessel for a given frame of angiography data. In one embodiment, the centerline of the blood vessel, also referred to herein as the centerline, is a model or simulation generated based on the iterative evaluation of each candidate subset of a frame of angiography data for a marker band associated with an optical sensor or an acoustic sensor or other imaging or data collection sensor introduced during the collection of angiography data. In one embodiment, a dynamic program software module such as a software module implementing one or more steps of the Viterbi algorithm can be used to track the marker band. In one embodiment, the Viterbi algorithm is used for radiopaque marker tracking. The creation and tracking of the centerline are typically processed by other algorithms or combinations of algorithms. In one embodiment, the centerline of the blood vessel is generated by a combination of algorithms or processes for finding the shortest path between two distant points, such as a fast marching algorithm on the Hessian image and a modified Dijkstra algorithm.

[0038] FIG. 1 shows a system 5 including various data collection subsystems suitable for collecting data, or detecting features of a subject 10, or detecting the state of the subject 10, or performing other diagnoses of the subject 10. In one embodiment, the subject is placed on a suitable support 12, such as a chair or other suitable support, from a table bed. The subject 10 is typically a human or another animal having a specific target site 25.

[0039] In part, embodiments of the present invention relate to the co - registration of intravascular images or data obtained by an imaging catheter that traverses a blood vessel and an external angiographic image of that blood vessel taken when the catheter traverses. A general schematic with the target site enlarged is shown in FIG. 2A.

[0040] In a typical OCT data acquisition procedure, the catheter is inserted through a guidewire to advance the probe to the distal end of the target blood vessel. The probe 30 can include one or more markers. In one embodiment, the marker disposed on the probe 30 is an X - ray opaque marker band. The torque wire 110 that partially surrounds the optical fiber 33 is also shown in FIG. 2A. The probe 30 is disposed within the lumen 50 of the blood vessel. The guidewire 115 is also shown within the lumen 50. The guidewire 115 is used to position the probe tip and the torque wire disposed within the catheter with respect to the lumen 50. The light λ from the probe tip is shown directed towards the wall of the blood vessel having the lumen 50.

[0041] Additional details related to a typical intravascular data collection probe are shown in FIG. 2B. As shown in FIG. 2B, an intravascular data collection probe 120, such as an OCT, IVUS, FFR, or other data collection probe, includes an optical fiber 33 configured to direct light as shown by the dotted line as part of the probe tip. A sheath, such as the polymer sheath 125, surrounds the probe tip that includes a beam - directing element such as a lens or a reflector. The light λ emitted from the beam - directing device along the dotted line is illustrated. The optical fiber 33 is disposed on the torque wire 110 that is also disposed within the sheath 120. The optical fiber 33 is coupled to the PIU 35 as shown.

[0042] As shown in FIG. 2B, a marker or marker band 130, such as a radiopaque marker, is part of the data collection probe 120. The marker is detectable by the angiography system and can be tracked according to the movement of the marker across the frames of the angiography data. As shown, the distance from the right end of the torque wire 127 to the beam directing element, such as a lens or reflector, is L1.

[0043] Furthermore, the distance from the right end of the torque wire 127 to the right end of the marker 130 is L2. The thickness of the marker 130 is L3. The distance from the distal edge of the marker 130 (shown as the left side of the marker) to the torque wire 127 is L3 + L2. In one embodiment, L1 ranges from about 0.3 mm to about 0.9 mm. In one embodiment, L2 is about 0.6 mm to about 1.4 mm. In one embodiment, L3 ranges from about 0.5 mm to about 1.5 mm.

[0044] In one embodiment, a data collection probe, such as an OCT probe, can include three radiopaque marker bands. The distal marker located at the distal end of the probe remains stationary throughout acquisition. The central marker is located in the imaging core that resides 27 mm from the distal marker before retraction. The proximal marker is located 50 mm from the imaging core, and this distance remains fixed during retraction.

[0045] During retraction, a processor-based system, such as the system 22 of FIG. 1, records a live algorithm and displays the blood vessel along with the contrast agent and the marker or probe. Typically, the marker is visible at most times. Optionally, some frames are recorded without using a contrast agent, as shown in FIG. 6C, whereby the guide wire and the marker are clearly visible. This provides a good indication of the retraction path through the blood vessel.

[0046] FIG. 3A shows a typical graphical user interface configured to display a plurality of panels. The graphical user interface can be implemented using a computing device such as server 50 or workstation 87 or another suitable computing device. The upper right panel shows frame angiography image data. As shown in the image, a section of the blood vessel disposed between the upper point, i.e., cursor 3, and the lower point, i.e., cursor 4, was imaged using intravascular imaging technology as part of the pullback. Specifically, the angiography data was obtained during the execution of the OCT pullback.

[0047] A typical cross-section of an artery is shown in the upper left panel. In the upper left OCT image, a side branch is shown on the right side of the cross-section of the data collection probe. The lower panel, which substantially spans the user interface, includes a longitudinal image of the blood vessel disposed between the points, i.e., the distal endpoint and the proximal endpoint, indicated by cursors 3 and 4 in the angiography image. The magnifying glass icon can be used to zoom in or out on either the OCT image or the angiography image. The pencil icon can be used to make measurements on either the OCT image or the angiography image. The angiography frame of the data can be played back as a video in the upper right panel by using the playback, review, or forward video user interface controls.

[0048] In the upper left OCT image, the diagonal axis indicates the cross-section used to display the longitudinal mode in the lower panel. The longitudinal mode is generated by combining a plurality of cross-sectional views as shown in the upper left quadrant interface. In the L mode, the triangle 4’ is configured to indicate the bookmarked location of the target frame. The longitudinal view, i.e., the L mode, can be advanced, reviewed, or displayed as a video using the review, playback, or forward L mode user interface. However, the vertical line shown in the L mode corresponds to the cross-sectional slice of the blood vessel shown in the above cross-sectional OCT image. By selecting the playback button and the review button in the L mode, the corresponding vertical line advances or retreats so that the vertical line moves in the L mode of the lower panel as different cross-sections are shown in the upper OCT image.

[0049] In one embodiment, the computing device used to display and execute the user interfaces of FIGS. 3A and 3B includes a memory area containing image data such as cross-sectional views of blood vessels. The computing device may include a machine-readable medium or other memory containing one or more software modules for displaying a graphical user interface such as interface 142. The interface may include a plurality of panels, menus, or other displayable areas. These panels or areas can be displayed on one or more monitors such as display 82. The computing device can exchange data such as image data with monitor 23 using a network that may include one or more wired, optical, wireless, or other data exchange connections.

[0050] The controller or input device 127 can perform wired communication, optical communication, or other communication with other devices or systems shown on the network 120. The controller can be used to send command signals to the computing system 100 that runs the interface 142. The interface 142 can display data from the system 5 of FIG. 1, the system 300 of FIG. 14, or the data sources, systems, or software modules described herein. The interface 142 may include one or more menus and other sections that change in response to control signals from the controller 127. The controller 127 may include a processor or a suitable programmable ASIC. The control signals can be transmitted through the network 120 or via another connection.

[0051] The computing device 100 may include a server computer, a client user computer, a personal computer (PC), a laptop computer, a tablet PC, a desktop computer, a control system, a microprocessor, or any computing device capable of executing a set (sequential or otherwise) of instructions that specify the actions to be taken by the computing device. Further, although a single computing device is shown, the term "computing device" shall also be construed to include any collection of computing devices that individually or jointly execute a set (or sets) of instructions to perform any one or more of the software features or methods, such as the interface 142.

[0052] Figure 3B shows the representation of the graphical user interface 142. The interface 142 includes a plurality of panels. As shown in the illustration, in one embodiment, there are four main panels 150, 155, 160, and 165. These include an auxiliary display panel 150 showing angiographic data in this embodiment, a cross-sectional view or B-mode display panel 155, a lumen profile panel 160, and an L-mode display panel 165. In one embodiment, the interface also includes a plurality of toolbars B1, B2, and B3. In panel 150, three markers are shown as crosses superimposed on the angiographic image. The upper marker corresponds to the proximal reference frame shown in panel 160. The central marker corresponds to the minimum lumen area frame shown in panel 160 or the active OCT frame shown in panel 155. The lower marker corresponds to the distal reference frame shown in panel 160. The angiographic frames and OCT frames that can be displayed using the interface in FIGS. 3A and 3B can be processed and registered in overlay as outlined herein. In one embodiment, the computing device accesses a co-registration table to display the frames that are registered in overlay.

[0053] Figure 3B shows a minimum lumen area plot as part of the lumen profile for a blood vessel imaged during retraction of the OCT probe in panel 160. The D arrow and P arrow indicate the proximal and distal directions along the imaged blood vessel. The cross-section shown as a line having sections L1 and L2 is shown in the cross-sectional view of panel 155 and is also shown by sections L1 and L2 in the L-mode panel 165. An information bar B1, a measurement bar B2, and a menu bar B3 are shown.

[0054] As shown, the distance of a blood vessel such as an artery can be measured with respect to two endpoints, as indicated by a typical measurement distance of 119.88 mm. Further, the average diameter can be shown at the ends of each of the selected reference frames for measuring the blood vessel, for example, average diameter values of 39.2 mm and 44.2 mm at the distal reference frame and the proximal reference frame, respectively. As shown, the MLA is about 22 mm 2 is. In the MLA frame, the average blood vessel diameter is about 2.11 mm, and the percent diameter stenosis is 25.4% compared to the average diameters of the proximal reference frame and the distal reference frame.

[0055] All three images shown in the user interfaces of FIGS. 3A and 3B are overlaid and registered such that movement along the line between the ends of the blood vessel in the angiographic image can be indicated by a moving point that is synchronized with the frame within the OCT image. Thus, as one image moves along the blood vessel segment, the movement along the centerline shown in the angiographic image is also indicated by the movement frame identifier of the cross-sectional OCT image or the L-mode OCT image or both.

[0056] First, since the proximal marker band may be resident near the coronary ostium of the coronary artery branch, it is occluded by the turbidity of the contrast agent during retraction. The catheter is retracted at a constant speed through the blood vessel. Since multiple blood vessel segments exhibit different constrictions along the retraction, the marker does not move at a constant speed in the angiographic image plane (2D). Further, due to the movement of the heart, the marker exhibits a unique "sawing" motion with respect to the anatomical structure of the blood vessel. At some of the positions of the angiographic frame, the marker band appears blurred / indistinct due to the high-speed retraction motion combined with the high-speed heart motion. The contrast of the local nearby marker may be low. Other features such as the narrowed bifurcation point, background structure, etc. may be mistaken for any of the marker bands.

[0057] The data collection system 5 includes a minimally invasive imaging system such as magnetic resonance imaging, X-ray, computed tomography, or other suitable minimally invasive imaging techniques. As a non-limiting example of such a minimally invasive imaging system, as shown in the figure, an angiography system 20 suitable for generating cine, etc. is shown. The angiography system 20 may include a fluoroscopy system. The angiography system 20 is configured to minimally invasively image the subject 10, whereby a frame of angiography data, typically in the form of a frame of image data, is generated while a retrieval procedure is being performed using the probe 30, thereby imaging the blood vessels of the site 25 of the subject 10 using angiography in one or more imaging techniques such as OCT or IVUS.

[0058] The angiography system 20 communicates with an angiography data storage and image management system 22, which can be implemented as a workstation or a server in one embodiment. In one embodiment, data processing related to the angiography signals collected is performed directly on the detector of the angiography system 20. Images from the system 20 are stored and managed by the angiography data storage and image management 22. In one embodiment, a system server 50 or a workstation 87 handles the functions of the system 22. In one embodiment, the entire system 20 generates electromagnetic radiation such as X-rays. The system 20 also receives such radiation after passing through the subject 10. And the data processing system 22 uses the signals from the angiography system 20 to image one or more sites of the subject 10 including the site 25.

[0059] As shown in this particular example, the target site 25 is a subset of the vasculature or peripheral vasculature such as a particular blood vessel. This can be imaged using OCT. The catheter-based data collection probe 30 is introduced into the subject 10 and positioned within the lumen of a particular blood vessel such as a coronary artery. The probe 30 can be various types of data collection probes such as, for example, an OCT probe, an FFR probe, an IVUS probe, a probe that combines two or more of the above features, and other probes suitable for imaging within a blood vessel. The probe 30 typically includes a probe tip, one or more radiopaque markers, an optical fiber, and a torque wire. Further, the probe tip includes one or more data collection subsystems such as an optical beam steering device, an acoustic beam steering device, a pressure detector sensor, other transducers or detectors, and combinations of the above.

[0060] In the case of a probe that includes an optical beam steering device, the optical fiber 33 is in optical communication with the probe with the beam steering device. The torque wire defines a bore through which the optical fiber is disposed. In FIG. 1, the optical fiber 33 is shown without the torque wire surrounding it. Further, the probe 30 also includes a sheath such as a polymeric sheath (not shown) that forms a portion of the catheter. In the case of an OCT system, the optical fiber 33, which is a part of the sample arm of the interferometer, is optically coupled to the patient interface unit (PIU) 35 as shown.

[0061] The patient interface unit 35 includes a probe connector adapted to receive the end of the probe 30 and optically couple to the end of the probe 30. Typically, the data collection probe 30 is disposable. The PIU 35 includes appropriate junctions and elements based on the type of data collection probe being used. For example, a combination of OCT and IVUS data collection probes requires OCT and IVUS PIUs. The PIU 35 typically also includes a motor suitable for retracting the torque wire, sheath, and optical fiber 33 disposed therein as part of a retraction procedure. In addition to being retracted, the probe tip is typically rotated by the PIU 35. In this way, the blood vessels of the subject 10 can be imaged longitudinally or through a cross-section. The probe 30 can also be used to measure certain parameters such as FFR or other pressure measurements.

[0062] And the PIU 35 is connected to one or more intravascular data collection systems 40. The intravascular data collection system 40 can be an OCT system, an IVUS system, another imaging system, and combinations thereof. For example, the system 40 when the probe 30 is an OCT probe can include a sample arm of an interferometer, a reference arm of the interferometer, a photodiode, a control system, and a patient interface unit. Similarly, as another example, in the case of an IVUS system, the intravascular data collection system 40 can include an ultrasonic signal generation and processing circuit, a noise filter, a rotatable junction, a motor, and an interface unit. In one embodiment, the data collection system 40 and the angiography system 20 have a shared clock signal or other timing signal configured to synchronize the angiography video frame time stamp and the OCT image frame time stamp.

[0063] In addition to the invasive and non-invasive image data collection systems and devices of FIG. 1, various other types of data can be collected regarding the subject's site 25 and other subject parameters of interest. For example, the data collection probe 30 may include one or more pressure sensors such as a pressure wire. The pressure wire can be used without an OCT component or an ultrasonic component. Pressure measurements are obtained along segments of blood vessels at the site 25 of the subject 10.

[0064] Such measurements can be relayed via either a wired or wireless connection. As shown in the fractional flow reserve data collection system 45, the wireless transceiver 47 is configured to receive pressure measurements from the probe 30 and transmit them to the system to generate FFR measurements, or to more locations along the blood vessel being measured. One or more displays 82 can also be used to show angiographic frames of data, OCT frames, user interfaces for OCT, and angiographic data, as well as other reference and feature of interest.

[0065] Intravascular image data, such as frames of intravascular data generated using the data collection probe 30, can be sent to the data collection processing system 40 coupled to the probe via the PIU 35. Low-invasive image data generated using the angiography system 22 can be transmitted to, stored in, and processed by one or more servers or workstations such as the co-registration server 50, the workstation 87, etc. A video frame grab device 55, such as a computer board configured to capture angiographic image data from the system 22, can be used in various embodiments.

[0066] In one embodiment, the server 50 includes one or more co-registration software modules 60 stored in the memory 70 and executed by the processor 80. The server 50 may include other typical components for a processor-based computing server. One or more databases, such as the database 90, may be configured to receive image data generated by one or more of the systems, devices, or components shown in FIG. 1, subject parameters generated by, received by, or transferred to the database 90 by one or more of the systems, devices, or components shown in FIG. 1, and other information. The database 90 is shown as being stored in the memory at the workstation 87 while being connected to the server 50, but this is merely one typical configuration. For example, the software module 60 may be executed on the processor of the workstation 87, and the database 90 may be located in the memory of the server 50. Devices or systems used to execute various software modules are provided as examples. In various embodiments, the hardware and software described herein can be used to obtain frames of image data, process such image data, and align such image data.

[0067] Alternatively, as referred to herein, the software module 60 may include software such as pre-processing software, deformations, matrices, and other software-based components that are used to process image data or respond to patient triggers to facilitate co-registration of various types of image data by other software-based components 60 or to perform such co-registration in other ways.

[0068] The database 90 can be configured to receive and store angiographic image data 92 such as image data generated by the angiography system 20 and obtained by the frame grabber 55 server 50. The database 90 can be configured to receive and store OCT image data 95 such as image data generated by the OCT system 40 and obtained by the frame grabber 55 server 50. The database 90 can be configured to receive and store an angiography table such as the table shown in FIG. 14 and a co-registration table such as the table shown in FIG. 15.

[0069] Furthermore, the subject 10 can be electrically coupled to one or more monitors, such as the monitor 49, via one or more electrodes. The monitor 49 may include, but is not limited to, an electrocardiogram monitor configured to generate data related to the function of the heart and indicative of various conditions of the subject, such as systole and diastole. Since the geometric shape of the heart, including the coronary arteries, is substantially the same at a particular heart stage even in different heart cycles, knowing the heart stage can be used to assist in the tracking of the vascular centerline.

[0070] Therefore, when the angiographic data spans several heart cycles, the primary matching of the vascular centerline at the same heart stage can assist in the tracking of the centerline throughout the pullback. Furthermore, since most of the movement of the heart occurs during systole, the movement of the blood vessels is expected to be higher near systole and attenuate towards diastole. Thereby, data is provided as an indication of the amount of movement expected between consecutive angiographic frames to one or more software modules. The knowledge of the expected movement can be used by one or more software modules to improve the quality of the tracking and the quality of the vascular centerline by enabling adaptive constraints based on the expected movement.

[0071] The use of arrowheads indicating directionality or lack thereof in a given figure is not intended to limit or require the direction in which information may flow. For example, in the case of certain connectors such as arrows and lines shown as connecting the elements shown in FIG. 1, information may flow in one or more directions, or only in one direction, as appropriate for a given embodiment. The connection may include various suitable data transmission connections such as optical connections, wired connections, power connections, wireless connections, or electrical connections.

[0072] Furthermore, although the FFR data collection system 45 is shown as having a wireless system 47 suitable for wirelessly transmitting and receiving information, other systems and components shown in FIG. 1 may also include a wireless system such as the system 47 and, in one embodiment, may be capable of wirelessly transmitting and receiving information.

[0073] One or more software modules can be used to process frames of angiography data received from an angiography system such as the system 22 shown in FIG. 1. The various software modules may include, without limitation, software, software components, or one or more steps of a software-based method or method executed by a processor and can be used in a given embodiment of the present invention.

[0074] Examples of such software modules include a video processing software module, a preprocessing software module, an image file size reduction software module, a catheter removal software module, a shadow removal software module, a blood vessel image quality improvement software module, a patch image quality improvement software module, a LoG filter or modified software module, a guide wire detection software module, an anatomical structure feature detection software module, a stationary marker detection software module, a background subtraction module, a Frangi vesselness software module, an image luminance sampling module, a moving marker software detection module, an iterative centerline testing software module, a background subtraction software module, a morphological close operation software module, a feature tracking software module, a catheter detection software module, a bottom hat filter software module, a path detection software module, a Dijkstra software module, a Viterbi software module, a software module based on a fast marching method, a blood vessel centerline generation software module, a blood vessel centerline tracking module software module, a Hessian software module, a luminance sampling software module, an image luminance superposition software module, and other suitable software modules described herein may be included without limitation. The software module 60 shown in FIG. 1 may include one or more of the above-described software modules and other software modules described herein.

[0075] Image data processing features and exemplary embodiments As shown in FIGS. 4A and 4B, various processing stages, processing steps, or processing software modules are generalized to provide a high-level summary of the process of registering and overlaying angiographic image data and image data obtained using intravascular imaging techniques such as OCT, IVUS, or others. In one embodiment, frames of angiographic data are captured at an OCT or IVUS server or workstation using a frame grabber or other data capture device. Capturing images from both imaging modalities in real time ensures accurate timestamping of the two sources relative to each other. It is essentially not possible to calibrate the DICOM angiographic data acquisition time to match the timing of the OCT data. For example, a video software module can be controlled via a user interface to obtain and store individual frames of timestamped angiographic data as well, and the angiographic video can be presented to a frame grabber that can store it. In one embodiment, the OCT data and the angiographic data are stamped with dates by two respective processes that run in parallel on the same computer and thus share the same time reference.

[0076] Once the angiographic data frames are cached or otherwise stored, each stored frame can be modified during a preprocessing stage. Various matrices such as convolution matrices, Hessians, etc. can be applied on a pixel-by-pixel basis to change the brightness, remove a given angiographic image frame, or perform other modifications. As described herein, the preprocessing stage effectively enhances, modifies, or removes features of the angiographic image to improve the accuracy, processing speed, success rate, and other characteristics of subsequent processing stages.

[0077] As shown in FIG. 4A, various software-based processing stages 140 are shown. First, one or more frames of the angiographic images are processed during a preprocessing stage 140a prior to the various detection and tracking stages to assist in the registration of superimposition with frames related to other image data obtained using other imaging techniques such as OCT, IVUS, etc. and combinations thereof. The next stage is a stage 140b for determining or calculating the vessel centerline. As shown in the user interface of FIG. 3, the vessel centerline is generated by one or more software modules and superimposed on or otherwise displayed on the angiographic image.

[0078] In one embodiment, the centerline represents the trajectory of a probe such as the data collection probe 30 of FIG. 1 passing through the vessel being imaged during retraction. In one embodiment, the centerline is also referred to as a trace. Another stage is the detection of marker bands in the angiographic frame 140c. In one embodiment, the last stage is the co-registration stage. These stages as well as other stages and methods described herein can be performed in different orders, interactively, in parallel, sequentially, or in combination. Additional steps and stages can also be added before, after, or between a given stage or step. Examples of additional typical stages and steps for listing additional details are shown in FIGS. 4B and 5A - 5C.

[0079] As shown in FIG. 4B, various software-based processing stages or processing steps 145 are shown that include additional details regarding the processing stages or processing steps shown in FIG. 4A. First, preprocessing of the angiographic frames is performed 150a. Detection of the guidewire on frames without contrast agent is performed as shown in FIG. 6D 150c. FIG. 6N is a typical angiographic image showing the result of guidewire detection. As shown in FIG. 6N, the distal portion of the guidewire is detected.

[0080] Next, in one embodiment, generation of a vessel centerline in one frame is performed at 150e. In one embodiment, user input such as selection of a guidewire endpoint within the lumen being imaged via the user interface is stored as the endpoint selected by the user, which is also referred to as a hint point. Such a hint point can be used to generate a vessel centerline on one frame such that a trace between the hint point and the peripheral site is generated for the relevant frame of the angiographic data. In one embodiment, such relevant frames are obtained without placing a contrast agent in the blood vessel.

[0081] Referring also to FIG. 4B here, tracking of the vessel centerline along the angiographic frame is performed at 150f. In one embodiment, such tracking of the vessel centerline is performed for all or substantially all of the angiographic frames obtained during retraction. Tracking and / or marker detection of radiopaque markers in the angiographic frames is performed at 150h. In one embodiment, the Viterbi algorithm is used to perform marker tracking. Overlay registration of the OCT image and the angiographic image is performed at 150j. Generation of a confidence score / performance index is performed at 150l.

[0082] The generation of the confidence score / figure of merit (FOM) is performed using one or more software modules 150l. In one embodiment, the confidence score or (FOM) is provided to the user by a graphic display on a computer monitor by, for example, providing a color code on an X-ray image or an OCT image indicating regions of OCT pullback having high or low confidence, such as where overlay registration is being performed. Regions of low confidence may be indicated, for example, by red stripes or bars on an X-ray image near a vascular segment where a low FOM was obtained. The FOM / score reflects the confidence criterion of the returned result. The score is in the range of [0,1], where 0 reflects the lowest confidence and 1 reflects the highest confidence. An FOM threshold can be selected to define the boundary between high confidence registration results and low confidence registration results. This threshold can be chosen to give the desired sensitivity and specificity for identifying high error locations by generating a receiver operating characteristic (ROC). If low FOM values are obtained for most of the frames in a given pullback and thus the overall quality of the registration is suspect, the registration results may not be displayed to the user.

[0083] The FOM determination is a scoring process based on one or more factors such as the quality of the detected blob (contrast or luminance of the detected blob compared to the contrast or luminance of the immediate neighborhood, shape, size, etc.), the distance of the detected blob from its nominally expected position (based on pullback speed, frame rate calculation), the number of blob candidates detected within the same neighborhood (the more candidates, the lower the FOM), and the luminance-based z-score, the overall score of the Viterbi algorithm (how well the overall set of detected blobs represents the pullback), as well as other factors and criteria. In one embodiment, a weighted average including one or more of the parameters listed herein can be used to generate the FOM or score.

[0084] As shown in FIGS. 4A and 4B and as described herein, the various steps and stages, such as those described herein, can be performed automatically, in whole or in part, in various embodiments. Some additional details related to certain specific examples, such as those related to FIGS. 5A-5C, among some of the steps and methods of FIGS. 4A and 4B, will be described herein. For example, FIG. 5A shows a flowchart related to some typical preprocessing steps or preprocessing stages.

[0085] Typical angiography image data preprocessing embodiments In part, as shown in FIGS. 4A and 4B and as described herein, the present invention includes, in part, one or more preprocessing steps, preprocessing software modules, and related methods with respect to the collected frames of angiography data. In one embodiment, the image preprocessing is performed on a frame-by-frame basis with respect to the frames of angiography data, such as the data generated by the system 20 of FIG. 1. The preprocessing steps may include, without limitation, methods, steps, and software components suitable for improving vascular image quality, removing catheters, removing shadows, removing cardiac shadows, improving the quality of small masses, etc., by applying, for example, multi-scale LoG (Laplacian of Gaussian), detecting anatomical structure features, generating skeletons, reducing the size of angiography images, background subtraction, bottom-hat filters, etc., and other components.

[0086] Various matrices, such as Hessian and other types of filters and masks, can be applied to improve the image quality of the frames of angiography data before the frames of angiography data are subjected to additional processing for tracking markers, generating centerlines, and registering with OCT, IVUS, or other images or data for overlay. One or more image processing steps can be used to preprocess the frames of angiography data received, for example, from an angiography system such as the system 22 or server 50 or workstation 87 shown in FIG. 1.

[0087] FIG. 5A shows a process flow 160 related to some additional specific typical preprocessing steps or preprocessing stages. As shown, angiographic images can be processed in multiple stages in parallel. In one embodiment, LoG filtering is performed at multiple scales 160a. Each scale corresponds to the size of the elements in the image acted upon by the filter. The LoG multi-scale based filter can be used, in one embodiment, to improve the image quality of patches corresponding to moving markers on the imaging probe. Different scales are used for markers of different sizes. In one embodiment, the LoG operator is calculated at several scales because it is sensitive to patches of different sizes and less sensitive to noise. An example of the LoG filter is shown in FIG. 6A. As an example of a patch (a set of pixels from the angiographic image) corresponding to a marker whose image quality is improved by applying the LoG of FIG. 6A as part of the imaging processing software image quality improvement, an example of a patch is shown in FIG. 6B. In one embodiment, background subtraction is performed to reduce the influence of static features based on the average of several frames of the angiographic image.

[0088] Furthermore, in one embodiment, a bottom-hat filter or bottom-hat transform 160c can be applied to the angiographic data to enhance the visibility of the guide wire within the image. In one embodiment, the bottom-hat filter is configured to eliminate features larger than the size of certain structural elements in a given angiogram, such as diaphragms, skeletal features, etc. An example of the bottom-hat filter or bottom-hat operator applied to an angiographic image is shown in FIG. 6E. In one embodiment, multiple image averaging is used for background subtraction. Furthermore, in one embodiment, Hessian filtering at a scale such as scale 1 is performed following the bottom-hat filter or bottom-hat transform 160e. The Hessian filter at scale 1 is performed to improve the image quality of the wire while smoothing the noisy image after the application of the bottom-hat operator. An example of the Hessian filter at scale 1 applied to the image is shown in FIG. 6F.

[0089] In one embodiment, a morphological closing operation is performed on the image data. The morphological closing operation is mainly used to fill possible gaps that may be obtained in the step of applying the bottom-hat transform. The bottom-hat transform is applied with a small filter kernel to improve the image quality of narrow features such as guidewires, for example.

[0090] Binary image map feature By applying a set of preprocessing steps to each angiography image, a binary map is created that is used to determine where the contrast agent is present. In one embodiment, the binary map refers to an image of the same size as the original angiography image, and the pixels are either black or white, i.e., pixels with dye are black and pixels without dye are white, or vice versa. The binary map may have regions of vascular pixels separated by the inherent imperfections of the binary map. Then a distance map can be calculated based on the binary map. An exemplary distance map calculated using the FMM algorithm is shown in FIG. 6H.

[0091] The distance map is an image of the same size, and the value of each pixel is determined according to the distance from the nearest "black" pixel in the binary map. Specifically, pixels determined to have dye present in the binary map (pixels with a distance of 0 from the dye region, i.e., "black" pixels) remain black, and pixels surrounding the immediate vicinity of the region of black pixels (with a distance of 1 from the dye region) have a luminance that is only "1" lower. The next layer of pixel luminance is lowered by only "2", and so on. As shown in FIG. 6H, the various luminance values are mapped to pixels arranged along the x-axis and y-axis with respect to the pixel location. A scale encoded by color or other markings can be used to map the luminance value for each pixel location. In one embodiment, the scale is a color scale. The various typical luminance values on the scale are shown in the figure. The central region has the lowest luminance value corresponding to B. The T luminance value increases relative to the B value. The Y luminance value increases relative to the T value, and the R value increases relative to the Y luminance value.

[0092] The resulting distance map is such that the regions of the dye / contrast agent in the original binary map appear as raised lines with slopes descending on either side. If two such raised lines are close enough (small distance in the binary map), they will appear as a connected raised line in the distance map. The dark central spot having the minimum value of the distance map belongs to the user hint point where the propagation starts at the front end. Due to the potential configuration, this propagates along the wire. The distal endpoint of the trace has the maximum value on the distance map. One application of the distance map is to determine separate segments of the dye / contrast agent that are close enough to be connectable. In one embodiment, the distance map is a tool used to determine the vascular skeleton from the binary map. The distance map can be used for various purposes.

[0093] Embodiments for detection of typical anatomical structure features / generation of prior data Furthermore, in one embodiment, anatomical structure feature detection is performed as part of the pre - processing of the angiography image. In one embodiment, this can be performed to generate specific prior information related to the path taken by the imaging probe through the blood vessels. For example, generation of line segments such as through a skeleton generation process can be used for feature detection. In one embodiment, the skeleton is a static object such as one or more line segments created to assist in tracing the blood vessels of the subject being imaged.

[0094] The use of a skeleton - or line - segment - based approach for generating candidate paths through blood vessels for a data - collection probe that can be used to notify center - line generation and marker tracking provides several advantages over refraining from using such an approach. For example, a skeleton - based approach can prevent or eliminate the generation of certain center - line traces that would otherwise pass through branches or the imaging - probe catheter. The generation of a skeleton provides a method for determining an initial candidate of the geometry of the imaged blood vessels as well as branches and other blood vessels as a map or framework to facilitate center - line generation. By generating a skeleton, it is possible to extract points of interest such as branch points and blood - vessel segments, stabilize marker tracking and center - line tracing, and verify tracking quality across the entire frame of angiographic - image data.

[0095] In one embodiment, the process of generating a skeleton to detect anatomical - structure features such as branches and the geometry of blood vessels is implemented during pre - processing 160d of the angiographic image. The skeleton can be used to detect anatomical - structure features such as major branches (170l) and outliers (170m). Further, the skeleton can be used to detect and generate (170f) a smooth blood - vessel center - line. For example, the skeleton can be used with a Dijkstra algorithm. The skeleton can be generated based on a pre - processed Hessian image. Points selected by a user on an angiographic image such as the image of FIG. 7A, which are related to the guide - wire position, can be used to reduce noise and facilitate skeleton generation.

[0096] In FIG. 7D, the endpoints selected by the user and the endpoints determined by the computer are indicated by X. The binary image generated from the Hessian image can be used to generate a skeleton in the angiographic image shown in FIG. 7B. After generating the skeleton, the skeleton can be eroded to eliminate small branches. For example, the small branches of the skeleton can be removed or subtracted from the image until only the main trunk section remains. Thresholds related to branch thickness and other parameters can be used to manage skeleton erosion. Removal of the small branches of the skeleton can be performed in pixel units in one embodiment until the final skeleton results as shown in FIG. 7C.

[0097] In one embodiment, the position of the junctions on the skeleton is located by detecting branches and other gaps indicated by the regions surrounded by the circles in FIG. 7D. These junctions are used to decompose the skeleton into branches as shown by branches 1-13 in FIG. 7E. And each branch of the tree that is too small to represent a blood vessel branch can be eroded and eliminated. In one embodiment, all branches are eroded equally (by the same number of pixels in length). As a result, relatively long branches survive and relatively short branches are eliminated. The remaining skeleton branches can then be converted into a connected graph as shown in FIG. 7F. The distances between graph nodes such as nodes 2 and 4 in FIG. 7F, i.e., between skeleton branches, are based on the angular change. When i = 2 and j = 4 for a node, the following distance relationship:

Number

Number

[0098] Figures 8A - 8C show the best paths resulting from the detection regarding the skeleton of FIG. 7E based on the application of the Dijkstra shortest path algorithm to the graph generated in FIG. 7F. FIG. 8A shows a path passing through nodes 2, 4, 8, 7, 6, 3, and 1. FIG. 8B shows a path passing through nodes 4, 6, 9, 3, and 1. FIG. 8C shows a path passing through nodes 2, 6, 8, 9, 7, 5, 3, and 1. The use of angles for distance measurement is useful when considering the three - dimensional nature of how the nodes and branches are arranged within the subject.

[0099] Typical catheter detection embodiments Furthermore, in one embodiment, as part of the pre - processing of angiography images, catheter detection is performed at 160f. The presence of a catheter within the field of view may interfere with various steps and processing stages of the co - registration method. The intersection of a catheter and a blood vessel may be interpreted as a false branch that can lead to unstable tracking. The tracking of markers and centerlines may be negatively affected by the presence of a catheter delivering an intravascular imaging device. Another problem associated with such catheters is that the shortest path between two points along a blood vessel may pass through the catheter instead of the blood vessel. As a result, the catheter may lead to errors and the generation of false centerlines.

[0100] Therefore, it is desirable to be able to remove the catheter from each frame of the angiography data, for example, to assist in centerline generation, before proceeding with subsequent processing and detection. For example, as shown in FIG. 9A, for a given input angiography image, vector fields as shown in FIGS. 9B - 9D can be superimposed on the image based on the detection of a section of the moving image and a section of the image showing a direction field as shown in FIG. 9B where the catheter extends into the central part of the figure and the blood vessel intersects the catheter at an angle approximately in the center of the figure. FIG. 9C shows a vector field map of the blood vessel region. While FIG. 9D shows vectors that are substantially straight and perpendicular within the catheter region.

[0101] The vectors of the vector fields shown in FIGS. 9C and 9D are eigenvectors corresponding to the eigenvalues of the Hessian matrix calculated by local quadratic analysis. In FIG. 9C, all scales from 1 to 5 were used with the Frangi filter. An example of such a filter is described in A.F. Frangi, W.J. Niessen, K.L. Vincken, M.A. Viergever, "Multiscale vessel enhancement filtering", Medical Image Computing and Computer-Assisted Intervention (MICCAI), 1998, pages 130 - 137, and thus the disturbing effects outside the blood vessels are explained. In FIG. 9D, only scale sigma - 4 was used, and thus the isolated orientations on the catheter were used, while in the outer regions, the eigenvectors have zero weight. Regarding the sigma parameter, this parameter represents the scale of the Gaussian used in the convolution calculation. Sigma = 4 reflects the typical width of the pixels for the catheter observed in the angiography dataset.

[0102] In one embodiment, catheter detection is based on the primary assumption of the orientation of the catheter and the fact that the catheter always intersects the lower boundary of the image as shown in FIG. 9D. Locally, although the catheter and the blood vessels are generally indistinguishable from each other considering their tubular structures. From the perspective of the shape of the catheter, the catheter can be globally distinguished from the blood vessels because it crosses substantially the entire image and has a substantially straight shape. In one embodiment, the vector orientation is used to distinguish the catheter and the blood vessels. Locally, the blood vessels may have small regions with orientations similar to the orientation of the catheter. The eigenvector orientation of the catheter is generally closed up to 90 degrees, while the eigenvector orientation of the blood vessels is not closed.

[0103] In one embodiment, a method of catheter detection is used that incorporates a frangible filter for vasculature in addition to shape features. In one embodiment, the method involves determining a direction image based on the eigenvectors of the vasculature reference image and the Hessian image at only one scale (sigma = 4, which reflects the typical width in pixels of the catheter as observed in the angiography dataset). Catheters in a given image frame of the angiography data can be isolated using various criteria. These criteria include direction (threshold of the direction image), and the length of the connected component containing the catheter (the length of the catheter profile must be at least half of the maximum image dimension in x (or y)).

[0104] As a constraint of the image processing software, when a catheter is detected such that it is displayed in a given image, it is typically the case that the catheter intersects substantially the entire image. In one embodiment, the system is programmed to assume that the catheter always cuts the lower boundary of the image. As a result, the lower boundary can be set based on the size of the detected object. Further, when the region of the angiography image associated with the catheter is detected, it can be useful to slightly expand the boundary around the centerline of the catheter or otherwise enlarge it to ensure that a sufficiently large feature has been detected. An example of a catheter detected based on the steps outlined above is shown in FIG. 9E.

[0105] Typical catheter removal embodiment As described above, the presence of a catheter within the field of view of a given angiographic image may interfere with the various steps and processing stages described herein. Thus, when a catheter is detected, for example, by a software-based method enumerated herein, it is desirable to remove the catheter. The bounded region shown in FIG. 9A shows a catheter that obliquely overlaps with a blood vessel. While striving to maintain the integrity of the image, various object elimination approaches for removing the catheter can be used. Based on a mask of the catheter that can be generated from the output of the catheter detection process being used, or as the output of the catheter detection process being used, a software module can be configured to remove the catheter mask by excluding the catheter.

[0106] One advantageous technique for removing a catheter uses the principle of superposition of an extinguishing function and a removing function when they are out of phase with respect to each other. In one embodiment, a software module based on superposition is used to perform catheter removal, for example, by estimating its luminance profile and reducing it from the image. The catheter luminance profile can be generated based on sampling points in the image identified as part of the catheter through a catheter detection software module.

[0107] As shown in FIGS. 10A and 10B, a typical cylinder 190 can be displayed with various longitudinal slices of thicknesses T0, T1, and T2 as shown. The cylinder 190 can be displayed as a model representation of a catheter. To the extent that the catheter and the cylinder 190 are filled with a contrast agent, the luminance change caused by the contrast agent is greater in the center along the thickness T0, then decreases as it moves away from the center T0 to the slice T1, and then further decreases when it reaches the slice T2. Thus, since there is less contrast agent at the thinner edge of the catheter compared to the center of the catheter, to remove the catheter from the image, a luminance profile for the catheter can be generated and added to the region of the image where the catheter was detected. A typical representation of the relevant catheter removal method is shown in FIG. 11A.

[0108] Considering that the catheter is detected as described herein, a mask associated with the pixels in the image constituting the catheter can be generated by using a mask region such as the region shown in FIG. 9E. In one embodiment, the image luminance is sampled in the catheter region, for example, on a line perpendicular to the catheter line. These vertical lines gradually decrease from one side of the catheter until they reach a low or relative extreme corresponding to the thickest central part of the catheter, and span the slope of the contrast agent luminance change that gradually increases again when the cross-section of the catheter becomes thinner at the edge of the catheter, as shown in FIGS. 10A and 10B. Each line sampled in the catheter region generates a luminance curve. The various luminance curves can be averaged into a single curve. This luminance curve is inverted and then superimposed on the vertical lines constituting the catheter region as shown in FIG. 11A, and the catheter can be effectively removed from that region.

[0109] Typical shadow removal embodiments A standard Hessian-based filter is part of the preprocessing and is based on the eigenvalues of the Hessian of the image. In one embodiment, the Hessian is calculated at several discrete scales, and then the maximum response among them is taken. In one embodiment of the shadow removal process, scales from 1 to 5 are used. Scale 5 can be chosen as the scale representing the typical maximum observed vessel width in the available data. Examples of the original image and the subsequent processing to remove shadows and other features are shown in FIGS. 6I - 6N.

[0110] The shadow removal preprocessing step is applied to convert the original image into a modified image having an improved contrast level. Further, the modified image is changed by a process of applying a Hessian so that it is not affected by the shadows of the heart and diaphragm that may induce some regions or planes of different contrast. Removing these shadows is desirable because such regions or planes may lead to incorrect vessel centerlines. In one embodiment, the shadow removal step includes applying a bottom hat operator with a filter kernel configured to have a distance parameter much larger than the width of a typical blood vessel. FIGS. 6L and 6J show the modified images improved by performing the shadow removal process.

[0111] Typical Vessel Centerline (Trace) Generation Embodiment Two anchor points, namely the distal point and the proximal point, mark the endpoints and start points of the vessel centerline. Another point is reflected on the vessel skeleton and the Dijkstra algorithm is applied to find the shortest path from the perspective of smoothness. The FMM is also applied to find the shortest path from the perspective of luminance (the FMM is executed on the Hessian image with improved image quality). The result of the FMM is combined with the Dijkstra result to yield the best vessel centerline (trace) between the two anchor points. The vessel centerlines in other angiographic frames are generated by applying an equiangular mapping combined with the FMM to the trace initially generated.

[0112] In one embodiment, a fast marching technique or method addresses the efficient calculation of geodesic distances based on a potential. In one embodiment, when a contrast agent is present, the potential may be an image with improved image quality. In one embodiment, when, for example, no contrast agent is present, or when only a guide wire is present (even if it is visibly distinct on an angiographic image), the potential is adjusted by constructing a function based on a distance transform. One method for calculating the potential function propagated by the leading front can be performed by the guide wire-based potential by applying the Euclidean distance transform to a binary image. Once the distance transform is generated, such a transform can be further modified to the potential function by applying an exponent to a negative fractional power of the distance transform. A typical guide wire potential is shown in FIG. 6G.

[0113] FIG. 5B shows a process flow 170 related to vascular centerline generation. In one embodiment, a Hessian having a scale of 1 is applied 170a to a frame of angiographic data. This application of the Hessian leads to an improvement in the image quality of thin ridges in the image, such as a guide wire. In one embodiment, automatic detection of the guide wire and selection of an anchor point on the guide wire are performed 170c. Once the guide wire is detected, in one embodiment, the point of the highest LoG response is identified as the anchor point. Next, tracking 170e of the distal guide wire anchor point to all the retracted angiographic frames is performed. The proximal anchor point is detected in a single frame. The distal anchor point is also detected in a single frame. In one embodiment, each anchor point is a feature that can be easily detected in other frames by tracking. Next, the anchor points are tracked to all the frames such that each angiographic frame has two endpoints for vascular centerline generation (tracing).

[0114] In one embodiment, a point selected by a user, such as a guidewire point on an angiographic image, is selected 170j. Similarly, a Hessian of (up to approximately 5) scale can be applied to the angiographic image to improve the image quality of blood vessels 170k. The image modified as a result of the Hessian application can then be used to perform the detection 170l of the nearest branch anchor point. This detection step can use, as input, the point selected by the user, i.e., the hint point. Next, the detection of the extrapolation of the anchor point is performed 170m. Identify which anchor points are detected. Next, tracking the anchor points to all the backward angiographic frames is performed 170n.

[0115] In one embodiment, the system then uses a graph search software module, such as Dijkstra's shortest path solution for a graph. Applying Dijkstra's algorithm or other shortest path algorithms combined with FMM to select the best initial vessel centerline can then be performed with respect to the angiographic backward frames 170f. Tracking the vessel centerline in the angiographic backward frames using FMM in a narrow band based on conformal mapping is then performed 170g. In this context, a narrow band means constructing a narrow band region around the trace of interest. This narrow band is aimed at increasing the efficiency of the FMM algorithm for calculating geodesic distances in a limited region of the image. These centerlines can be stored in one or more tables and displayed on applicable angiographic images.

[0116] Typical marker detection and co-registration embodiments Figure 5C shows a process flow 180 related to marker detection and co-registration. As used herein, the term trace is interchangeable with centerline. Initially, as input, a centerline (trace) from the pullback frame is provided as input for the sampling direction 180a. Further, a LoG is applied from the pullback frame to the image 180c. Sampling of the LoG image perpendicular to the trace is performed 180e. In one embodiment, dynamic programming or iteration is performed at different starting or ending points to find the marker position in all frames 180g. In one embodiment, the dynamic programming or iterative process can be implemented using the Viterbi algorithm. Next, selection of the most likely solution for the marker in frame units is performed 180h. In all frames, calculation of the marker position is performed along with the marker normalized arc length position along the vessel centerline 180l.

[0117] Next, all combinations of OCT frames and angiography frames can be registered by overlaying based on the marker positions calculated from the perspective of arc length. Since all vessel centerlines start and end at the same anatomical structure features of all angiography frames, each centerline matches the other centerlines of other frames. Therefore, the length or arc length of the centerline can be used as a basis for co-registration. The marker positions from the perspective of arc length are preserved (up to some error) in all frames.

[0118] One issue encountered when trying to resolve the opaque marker band of the sensor or data collection probe is the use of contrast fluid as part of the OCT pullback. In one embodiment, it is useful to process frames of angiography data prior to the introduction of contrast fluid so that an initial path through the vessel can be provided using a guidewire and an imaging catheter. This initial data set can be repeatedly improved using other information and parameters as described herein.

[0119] The Viterbi-based algorithm automatically detects the radiopaque markers in each image of the pullback. This algorithm can be used to obtain a global solution based on patch luminance and location prediction (constant speed along the trace). As a prerequisite for this algorithm, a process of detecting and tracking the vessel centerline (trace) is executed. The trace is used to create a continuous co-registration between OCT and angiography. These curves are calculated using the fast marching method. The fast marching method enables the efficient calculation of the path (trace) between the proximal point (which may be a point selected by the user, i.e., a hint point) and the distal stationary marker in each frame. The stationary markers are detected on the frame (with and / or without the use of contrast agent / dye). Template matching techniques are utilized to track the proximal and distal points over subsequent sequences.

[0120] The Viterbi algorithm is configured to maintain a balance between exogenous and endogenous factors. The exogenous factor (marker band representation) is derived from the marker band LoG map by resampling the map with discrete strips perpendicular to the trace for each angiography frame. The endogenous factor is the arc length progression over time. This endogenous factor models the advancement of the marker band along the arc length of the pullback. The basic idea is that while the average pace is determined by the pullback speed, there is a penalty for deviating from this pace. This factor takes into account the natural "tailback" profile by imposing different penalties for forward / backward movement.

[0121] FIG. 12 shows a data collection and coregistration system 300 that includes various software and hardware components suitable for processing intravascular image data and angiography data. In one embodiment, after one or more frames of OCT image data and angiography image data are overlay registered, the output is a registration table. In one embodiment, frames of OCT data can be monitored to check if there is a clear frame display state, and this clear frame display can be used to trigger cine so that frames of angiography data can be captured. In one embodiment, in the case of a predetermined retraction procedure in which the probe is retracted through the blood vessel while probe data and angiography data are being collected, frame timestamp stamping, registration table addition, and image processing features and other processes may be performed.

[0122] The user interface (UI) 308 communicates with the OCT adapter 320. The image processing module 330 communicates with the OCT adapter 320. In one embodiment, the image processing module 330 performs operations or transformations, or applies them to frames of angiography data, for example, for shadow removal, guide wire detection, catheter removal, and other image processing steps outlined herein. The optical coherence tomography system 310 communicates with the OCT adapter 320. The optical coherence tomography system 310 may include a frame grabber 302 or communicate with a frame grabber 320. Angiography frames are acquired using a frame grabber and fetched by a software module.

[0123] The OCT frame table 315 includes information and images of blood vessels obtained during retraction of the imaging probe through the blood vessel. The role of the OCT adapter 320 is to provide a software interface between the angiography system and the OCT system.

[0124] The software-based system such as a server or a workstation described in this specification, and the software module configured to automatically execute, capture, and tag each image according to the acquisition time thereof support the co-registration of intravascular data tagged with the acquisition time. The image processing module 330 that may include a co-registration software module automatically detects X-ray opacity markers on each angiographic image corresponding to intravascular acquisition. A single user input may be required to assist in the detection as shown in FIG. 5B. The co-registration software module calculates the path of the intravascular imaging catheter on all angiographic images corresponding to intravascular image acquisition during the retraction of the probe through the blood vessel during imaging. The co-registration software module generates a co-registration table of external images including the FOM associated with each co-registration result, which serves as a reference for the intravascular image of the acquisition, the location of the X-ray impermeable markers on each angiographic image, the position of each intravascular image / data point on each angiographic image, and the level of confidence in the accuracy of the result.

[0125] The user is presented with a graphical representation of the intravascular image and the angiographic image, as well as the correspondence between two such as a specific intravascular image on the angiographic image as part of the user interface when the co-registration is completed in one embodiment. During the co-registration procedure, if the FOM or confidence score is unacceptable, additional user input or other parameters from the OCT system may be requested or may be obtained automatically.

[0126] Typical Confidence Score / Performance Index Embodiment For each detection of the probe marker, a confidence score, also called FOM, is assigned to each detected probe marker. The score is based on one or more of the blob luminance, the number of dark blobs in the vicinity of the predicted region of the marker, the marker arc length along the trace, the blob movement, and the stability of the trace. The FOM / score reflects the confidence criterion of the returned result. In one embodiment, the score is in the range [0,1], where 0 reflects the lowest confidence and 1 reflects the highest confidence.

[0127] For example, software modules related to angiography, such as one or more modules described herein, typically evaluate images generated using an imaging device disposed outside the subject's body. In contrast, data collection probes such as OCT, IVUS, FFR, pressure, or other data collection modalities can be placed inside the patient's blood vessels. As a result, data obtained from the data collection probe during retraction, or parameters known to be associated with the data collection probe, can be used by the angiography software to improve the operation of the methods and steps described herein. An adapter software module or other software module can be used to provide OCT information to the angiography image frame processing software module, and vice versa.

[0128] For example, the following parameters related to data obtained regarding a blood vessel as part of intravascular data collection can be sent to an angiography software module or other software module for analysis, or to assist in the evaluation of a subject, or to associate with different data sets, retraction length in millimeters, start of retraction, end of retraction, display of branch points such as side branches from collected OCT data, data collected regarding a frame prior to introduction of a contrast agent or dye, OCT and angiography synchronous frame time tags, retraction speed, distance between the distal and proximal markers of a catheter, and other factors and parameters obtained regarding a given data collection modality such as longitudinal blood vessel image data, pressure data, EKG data, systolic state during retraction, diastolic state of the heart during retraction, and other states available regarding the subject.

[0129] Angiography table An angiography table, such as that shown in FIG. 14, includes not only each acquired angiography frame but also information explaining the angiography retraction. The angiography table is created by an angiography software module at the time of acquisition, with timestamp data partially added. This table is extracted and stored by the OCT module upon completion of acquisition. The table is then provided to the angiography software module during co - registration when a co - registration - dependent field is added.

[0130] Co - registration table The co - registration table, as shown in FIG. 15, includes the results of successful co - registration. The co - registration table includes all of the OCT / angiography cross - reference information necessary to drive the co - registration GUI toolset. This table includes an entry for each OCT frame, which is a list with an entry for each angiography frame including the acquisition timestamp of that frame and OCT marker position information. In one embodiment, the co - registration table associates the OCT frame index with the registered angiography frame index. Further, the table may include entries that associate the OCT frame and the angiography frame.

[0131] Additional multimodal co - registration features and embodiments In one embodiment, co-registration refers to synchronizing frames from two or more data collection modalities or combining information from two or more data collection modalities. For example, a bifurcation detected in an OCT image can be used as an anchor with respect to a bifurcation detected in an angiography image. The co-registration features enumerated here are not limited to OCT. Instead, the features described here related to the vasculature and individual blood vessels or other data collection modalities can be extended to other intravascular imaging modalities. In one embodiment of the present invention, the centerline of a blood vessel is determined from the path of a guidewire or catheter tracked by a tracking system such as Mediguide's Medical Position System during the advancement of the guidewire or catheter through the blood vessel.

[0132] In one embodiment, a side branch detected in an OCT frame of data using OCT image processing can be used as an input to improve co-registration with angiography data. For example, in one embodiment, each OCT frame includes a flag (yes / no) indicating whether a side branch is present. Further, when co-registration is obtained, stents, calcium deposits, lipid deposits, thrombi, asymptomatic ruptured plaques (TCFA or "vulnerable plaques"), vessel normalization, side branch detection, FFR values (which may be calculated as vascular resistance ratio (VRR) values based on OCT image data), lumen size values, stents, and various other data described herein can be overlaid on angiography images or OCT images taking into account co-registration between data sets.

[0133] Live stent implantation guidance embodiments and features In one embodiment, following OCT / angiography co - registration, the guide wire is retained after being retracted for stent particles through another catheter. The process of imaging the blood vessel that was the target of the retraction proceeds via continuous fluoroscopic imaging that is co - registered with the OCT. As it moves along the OCT frame or the angiography frame, collateral branches and other information can be viewed. In one embodiment, various processing steps are performed on OCT data such as pre - stent, 3 - D co - registered virtual tissue structure, lumen detection, guide wire detection, stent incomplete apposition, plaque detection, etc. Since the OCT frame and the angiography frame are registered, the information detected in the OCT frame can be overlaid on the angiography image that the operator uses to deploy the stent. If the collateral branches can be shown on the angiography map on the user interface, this may help avoid unwanted entrapment of the collateral branches during stent deployment.

[0134] Furthermore, various types of overlays related to stents previously deployed or candidates for deployment can be displayed on one or both of the OCT image and the angiography image following co - registration. For example, a bioabsorbable scaffold (BVS), i.e., a new type of stent that is radiopaque, can be detected in the OCT frame using OCT image processing. This OCT image data can be used to provide a particular type of stent overlay that is important in relation to the angiography data since such stents are not visible by X - ray. As another special case of data overlay, in one embodiment, information on stent incomplete apposition from a given OCT frame can be used to modify the stent image on the X - ray image by color - coding or generating other marks to indicate the regions of incomplete apposition.

[0135] Furthermore, considering that markers on the stent delivery probe can be tracked, the stimulated stent can be shown with respect to the marker in OCT longitudinal mode, i.e., L-mode. Angiography / OCT co-registration can show the cross-reference of tissue features, lumen features, and movement features such as the insertion of a balloon or stent, etc., together with an overlay and with the display of elements such as the stent cross-section in L-mode. When the scan of the stent is obtained as a wireframe model or selected from a drop-down menu prior to stent placement, the diameter and length can be used to display the stent in L-mode or angiography with higher accuracy.

[0136] In one embodiment, bands on OCT images and / or angiography images indicating sites for avoiding stent placement such as branches and target deployment sites based on stenosis / MLA calculation can be used. The angiography display and OCT display can be used to show a higher level of granularity with an overlay to help the user properly position the stent within the target area. Furthermore, considering the wireframe model of the stent and the lumen area calculated from OCT frames co-registered with the location of the stent in the angiography system, a visual guide for stent expansion targets can be provided and displayed. In one embodiment, this can be performed using an inflation balloon that is used to selectively expand the stimulated wireframe of the stent and one or both ends of the stent. These types of investigations using OCT and angiography can be used before stent insertion, after stent insertion, or as part of future follow-up.

[0137] In one embodiment, when retraction is performed, the OCT data and angiography data are stored. This stored data can be used to generate an image or model of the artery. Using such a model, live stent placement during subsequent retraction is enhanced. In this way, the prior existing OCT / angiography co-registration information can be used as a baseline.

[0138] Also, the angiography data can also be used to notify, improve, or correct OCT image display features or detection algorithms. One correction of OCT from angiography data is to re-space the OCT frames in L-mode to show the actual physical separation between frames measured by a co-registration tool. This compensates for the interval error that results from assuming a constant pull-back speed compared to the vessel lumen. In reality, the pull-back speed varies significantly due to heart motion, and our frames are not equally spaced. Once the co-registered OCT and angiography data sets are generated, a software module can be used to accurately measure the frame-to-frame interval. The frame-by-frame correction can be applied to re-space the L-mode view at a given user interface. This can also be applied to 3D OCT rendering and will provide a more accurate visual representation of the vessel.

[0139] Generally, various additional advantages can be considered by having a co-registered set of frames and two-way communication between an angiography system and an OCT system. The angiography information includes traces generated for different vessels. The junctions of these branches can be mapped to specific frames to notify OCT side branch detection. In one embodiment, by storing the angiography data and OCT obtained during angiography, a record can be constructed over time that can be used to co-register OCT images at different times with the angiography data that acts as a bridge or linker between two different OCT data sets.

[0140] In one embodiment, a pressure probe or other data collection modality can be used to collect data to improve the representation of a vessel using another imaging modality or parameter. In one embodiment, VRR can be used to calculate the percentage contribution of each stenosis to the overall FFR value and display the percentage relative to the angiography data. Further, the side branch location information from OCT images or angiography images can be used to improve VRR calculations by identifying additional junctions and flow points in the area near the imaged vessel.

[0141] In one embodiment, the system and method can be used to monitor a thrombectomy catheter in OCT L-mode. That is, this can be used in conjunction with the inductive stenting technique that uses the stents and alignment data stimulated as described herein. Generally, in part, the present invention relates to the tracking of any therapeutic device having radiopaque marker bands and displaying its position in OCT L-mode images and previously acquired, superimposed and registered X-ray images. The therapeutic device may be a stent or a thrombectomy catheter, or a balloon device such as an angiographic balloon or a drug-eluting balloon, or an ablation device such as a rotational atherectomy probe (rotablator).

[0142] Non-limiting software features and embodiments for implementing angiography and intravascular data collection methods and systems The following description is intended to provide an overview of the device hardware and other operating components suitable for performing the methods of the present invention described herein. This description is not intended to limit the applicable embodiments or scope of the present invention. Similarly, the hardware and other operating components may be suitable as parts of the devices described herein. The present invention can be practiced on a personal computer, a multiprocessor system, a microprocessor-based electronic device or a programmable electronic device, a network PC, a minicomputer, a mainframe computer, etc.

[0143] Some parts of the embodiments for carrying out the invention are presented from the perspective of algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations of algorithms can be used by those skilled in the art related to the fields of computers and software. In one embodiment, an algorithm is here, and generally, considered to be a consistent sequence of operations leading to a desired result. The methods, steps, or other operations described herein are operations that require physical operations on physical quantities. Although not necessarily, usually these quantities take the form of electrical or magnetic signals capable of storage, transfer, combination, conversion, comparison, and other operations.

[0144] Unless clearly stated otherwise as apparent from the following description, throughout the description, descriptions using terms such as "process" or "calculate" or "compute" or "compare" or "measure arc length" or "detect" or "trace" or "mask" or "sample" or "operate" or "generate" or "determine" or "display" refer to actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities within the registers and memories of the computer system and convert them into other data similarly represented as physical quantities within the computer system memory or registers or other such information storage devices, transmission devices, or display devices.

[0145] In some embodiments, the present invention also relates to a device for performing the operations herein. This device may be specifically constructed for the required purpose, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.

[0146] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. It may be convenient to use a variety of general purpose systems in conjunction with programs according to the teachings herein, or to construct more specialized apparatus to perform the required method steps. The required structure for various of these systems will appear from the following description.

[0147] Embodiments of the invention may be implemented in many different forms including, but not limited to, computer program logic for use with a processor (e.g., a microprocessor, a microcontroller, a digital signal processor, or a general purpose computer), programmable logic for use with a programmable logic circuit (e.g., a field programmable gate array (FPGA) or other PLD), discrete components, integrated circuit fabric (e.g., an application specific integrated circuit (ASIC)), or any other means including any combination thereof. In typical embodiments of the invention, some or all of the processing of data collected using an OCT probe, an FFR probe, an angiography system, and other imaging and subject monitoring devices, and processor-based systems, is converted into a computer-executable form, e.g., stored on a computer-readable medium and implemented as a set of computer program instructions executed by a microprocessor under the control of an operating system. Thus, for example, user interface instructions and triggers based on the completion of a pullback or co-registration request are converted into instructions understandable by a processor suitable for generating OCT data and performing image processing using the various features and other features and embodiments described above.

[0148] The computer program logic implementing all or part of the functionality described herein may be embodied in a variety of forms including, but in no way limited to, source code form, computer-executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator). The source code may include a series of computer program instructions implemented in any of a variety of programming languages (e.g., object code, assembly language, or high-level languages such as Fortran, C, C++, JAVA (registered trademark), or HTML) for use with a variety of operating systems or operating environments. The source code may define and use a variety of data structures and communication messages. The source code may be in computer-executable form (e.g., via an interpreter), or the source code may be converted to computer-executable form (e.g., via a translator, assembler, or compiler).

[0149] A computer program may be permanently or temporarily fixed in any form (e.g., source code form, computer-executable form, or intermediate form) on a tangible storage medium such as a semiconductor storage device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic storage device (e.g., a floppy disk or a fixed disk), an optical storage device (e.g., a CD-ROM), a PC card (e.g., a PCMCIA card), or other storage devices. The computer program may be fixed in any form in a signal that can be transmitted to a computer using any of various communication technologies including, but by no means limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth (registered trademark)), networking technology, and Internetworking technology. The computer program may be pre-loaded in a computer system (e.g., on a system ROM or a fixed disk), or distributed as a removable storage medium together with an attached printed document or electronic document (e.g., packaged software) distributed from a server or an electronic bulletin board on a communication system (e.g., the Internet or the World Wide Web).

[0150] The hardware logic that implements all or part of the functionality described above in this specification (including programmable logic for use with programmable logic circuits) may be designed using conventional manual methods, or electronically designed, incorporated, simulated, or documented using various tools such as computer-aided design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL).

[0151] Programmable logic may be permanently or temporarily fixed on a tangible storage medium such as, for example, a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a floppy disk or a fixed disk), an optical memory device (e.g., a CD-ROM), or other storage devices. Programmable logic may be fixed in a signal transmissible to a computer using any of a variety of communication technologies including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology. Programmable logic may be distributed in any form as a removable storage medium together with an attached printed or electronic document (e.g., packaged software) that is pre-loaded in a computer system (e.g., on a system ROM or a fixed disk) or distributed from a server or an electronic bulletin board on a communication system (e.g., the Internet or the World Wide Web).

[0152] Various examples of suitable processing modules are described in more detail below. As described herein, a module refers to software, hardware, or firmware suitable for performing a particular data processing task or data transmission task. In one embodiment, a module refers to an instruction or a software routine, program, or other memory resident application suitable for receiving, converting, sending, and processing various types of data such as, for example, angiography data, OCT data, FFR data, IVUS data, co-registration table data, centerlines, shadows, pixels, luminance patterns, and information for other purposes described herein.

[0153] The computers and computer systems described in this specification may include a computer-readable medium operably coupled, such as a memory, for storing software applications used when obtaining, processing, storing, and / or communicating data. It is understood that such a memory may be internal, external, remote, or locally present to the operably coupled computer or computer system.

[0154] Also, the memory may include any means for storing software instructions or other instructions, including, by way of example and without limitation, hard disks, optical disks, floppy (registered trademark) disks, DVDs (Digital Versatile Disks), CDs (Compact Disks), memory sticks, flash memories, ROMs (Read Only Memories), RAMs (Random Access Memories), DRAMs (Dynamic Random Access Memories), PROMs (Programmable ROMs), EEPROMs (Electrically Erasable PROMs), and / or other similar computer-readable media.

[0155] Generally, the computer-readable memory media applicable in connection with the embodiments of the invention described in this specification may include any memory media capable of storing instructions executable by a programmable device. If necessary, the method steps described in this specification may be embodied or executed as instructions stored in one or more computer-readable memory media. These instructions may be software embodied in various programming languages such as C++, C, Java (registered trademark), and / or various other types of software programming languages that may be applied to create instructions in accordance with the embodiments of the invention.

[0156] Aspects, embodiments, features, and examples of the invention should be considered illustrative in all respects and not intended to limit the invention, the scope of which is defined only by the claims. Other embodiments, modifications, and uses will be apparent to those skilled in the art without departing from the spirit and scope of the claimed invention.

[0157] The use of the headings and sections of this application is not intended to limit the invention, and each section is applicable to any aspect, embodiment or feature of the invention.

[0158] Throughout this application, when a composition is described as having, including, or comprising a particular component, or a process is described as having, including, or comprising a particular process step, it is intended that the composition of the present teachings also consists of or consists essentially of the recited components, and that the process of the present teachings consists of or consists essentially of the recited process steps.

[0159] In this application, when an element or component is said to be included in and / or selected from a list of recited elements or components, it should be understood that the element or component may be any one of the recited elements or components, and can be selected from a group consisting of two or more of the recited elements or components. Further, it should be understood that the elements and / or features of the compositions, devices or methods described herein can be combined in various ways without departing from the spirit and scope of the present teachings, whether explicit or implicit herein.

[0160] The use of the terms "include", "includes", "including", "have", "has", or "having" should generally be interpreted as open-ended and non-limiting, unless expressly stated otherwise.

[0161] The use of the singular form herein includes the plural (and vice versa) unless it is specifically stated otherwise. Further, the singular forms "a" and "the" include the plural unless the context clearly indicates otherwise. Further, when the term "about" is used in front of a quantitative value, the present teachings include the specific quantitative value itself unless specifically stated otherwise.

[0162] It should be understood that the order of steps or the order for performing a particular action is not important as long as the present teachings are operable. Further, two or more steps or actions may be performed simultaneously.

[0163] When a range or list of values is provided, each intermediate value between the upper and lower limits of the range or list of values is considered individually and is incorporated into the present invention as if each value were explicitly recited herein. Further, ranges between the upper and lower limits of a given range, as well as smaller ranges including the upper and lower limits of a given range, are considered and incorporated into the present invention. A typical listing or range of values is not a negation of other values or ranges between the upper and lower limits of a given range, as well as other values or ranges including the upper and lower limits of a given range.

[0164] It should be understood that the various aspects of the claimed invention are directed to subsets and substeps of the techniques disclosed herein. Further, the terms and expressions used herein are used as terms of description rather than limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Accordingly, the desired content guaranteed by the patent is the present invention as defined and differentiated in the following claims, which includes all equivalents.

[0165] [Appendix 1] A processor-based method for displaying angiographic and intravascular representations of blood vessels, comprising: generating a set of optical coherence tomography image data in response to distance measurements of the blood vessels obtained during retraction of a probe through the blood vessels using an optical coherence tomography system, the set of optical coherence tomography image data including a plurality of cross-sectional images at a plurality of positions along the blood vessels; During the retraction of the probe through the blood vessel using the optical coherence tomography system, generating a set of angiography image data using an angiography system, wherein the set of angiography image data includes a plurality of two-dimensional images obtained at different points in time during the retraction; displaying a first panel including a first longitudinal view of the blood vessel generated using the OCT image data; and displaying a first panel including a frame of the angiography image data identifying the blood vessel, using one or more points within the frame and a blood vessel centerline passing through the one or more points A method comprising: [Appendix 2] further comprising registering and superimposing the OCT image data and the angiography data using the blood vessel centerline to generate a continuous alignment of the markers to be tracked, wherein the marker to be tracked is disposed on an OCT data collection probe, the method according to Appendix 1. [Appendix 3] further comprising registering and superimposing the OCT image data and the angiography data such that a frame identifier in the first longitudinal view is changed by selecting a point along the blood vessel centerline through a user interface, the method according to Appendix 1. [Appendix 4] further comprising excluding candidates for the marker to be tracked based on the retraction length and / or the retraction speed, based on the positions that the marker can take based on the retraction length and / or the retraction speed, by performing an iterative search using the retraction speed or the retraction length, the method according to Appendix 2. [Appendix 5] wherein the blood vessel centerline is generated using a plurality of processing steps from shortest path techniques and Dijkstra's algorithm, the method according to Appendix 1. [Appendix 6] further comprising removing a guide catheter image from one or more frames of the angiography data using an overlay of luminance profiles, the method according to Appendix 1. [Supplementary Note 7] The method according to Supplementary Note 1, wherein the vascular center line is generated using path information generated from one or more angiography frames in the substantial absence of a contrast agent solution. [Supplementary Note 8] The method according to Supplementary Note 1, further comprising generating a confidence score for each of detection of the angiography data and the optical coherence tomography data, and registration of superimposition between the angiography data and the optical coherence tomography data. [Supplementary Note 9] A method for detecting an intravascular probe marker, comprising: obtaining a first frame of angiography image data that substantially does not contain a contrast agent and contains the intravascular probe marker; obtaining a second frame of the angiography image data that contains contrast agent image data near the intravascular probe marker; and detecting the intravascular probe marker in the first frame and the second frame The method comprising. [Supplementary Note 10] applying an image processing transformation to the second frame to remove or correct features within the second frame; and increasing the luminance of a plurality of pixels, the plurality of pixels including a guide wire image within the second frame, the step of increasing The method according to Supplementary Note 9, further comprising. [Supplementary Note 11] The method according to Supplementary Note 9, further comprising generating an average luminance value for a plurality of images and subtracting the average luminance from the first or second frame. [Supplementary Note 12] The method according to Supplementary Note 9, further comprising applying a bottom-hat operator to the second frame and applying a morphological closing operation. [Supplementary Note 13] Detecting the intravascular probe marker comprises: Filtering candidate markers including pixels in the first frame and the second frame by applying a multi-scale LoG (Laplacian of Gaussian) operator to the first frame and the second frame; and Performing non-maximum suppression processing to identify small blocks having a maximum value in the vicinity of pixels The method according to Appendix 9, comprising: [Appendix 14] The method according to Appendix 9, further comprising the step of generating a guide wire-based potential by applying Euclidean distance transformation to a binary image and applying an exponent to a negative fractional power of the distance transformation. [Appendix 15] The method according to Appendix 14, further comprising determining a plurality of geodesic distances based on the guide wire-based potential using a fast marching method. [Appendix 16] Removing shadows from the first frame and the second frame; Increasing the contrast level of a guide wire of one of the first frame or the second frame; and Performing morphological image reconstruction for each marker candidate The method according to Appendix 9, further comprising: [Appendix 17] Processing a plurality of backward frames using a Hessian-based vascularity filter; and Tracking the intravascular probe marker from one of the first frame or the second frame through all the backward frames to all the backward frames using template matching The method according to Appendix 9, further comprising: [Appendix 18] The method according to Appendix 9, further comprising tracking the intravascular probe marker through a plurality of frames obtained during backward using the Viterbi dynamic programming method. [Appendix 19] A processor-based method of registering by overlaying angiographic image data and intravascular image data obtained during withdrawal through a blood vessel, comprising: Storing a plurality of frames of optical coherence tomography data in a memory; Storing a plurality of frames of the angiographic image data in a memory; Processing the plurality of frames of the angiographic image data so that one or more shadows are substantially reduced; Detecting a catheter in the plurality of frames of the angiographic image data; Removing the detected catheter from within the plurality of frames of the angiographic image data; Generating a vessel centerline for the plurality of frames of the angiographic image data; Detecting a probe marker in the plurality of frames of the angiographic image data; Tracking the position of the probe marker along one or more of the vessel centerlines; and Registering the plurality of frames of the angiographic image data and the plurality of frames of the optical coherence tomography data by overlaying them using the tracked position A method comprising the above steps. [Appendix 20] The method according to Appendix 19, further comprising the step of generating a score indicating a level of confidence in the overlay registration between a certain frame of the angiographic image data and a certain frame of the optical coherence tomography data. [Appendix 21] The method according to Appendix 19, wherein the step of removing the detected catheter is performed using the superposition of luminance profiles generated based on sampling of the region of the detected catheter. [Appendix 22] The step of registering the plurality of frames of the angiographic image data and the plurality of frames of the optical coherence tomography data by overlaying them includes generating a co-registration table using a computing device, The registration table includes an angiographic image frame, a plurality of OCT time stamps for each frame, a plurality of angiographic time stamps for each frame, and an optical coherence tomography image frame, and is the method described in Appendix 19. [Appendix 23] The method according to Appendix 22, further comprising displaying a stent representation on an OCT image and an angiographic image in a user interface using the registration table and a computing device. [Appendix 24] The method according to Appendix 22, further comprising identifying the collateral using the registration table and a user interface configured to display the collateral in one or more of the OCT images and the angiographic images. [Appendix 25] The method according to Appendix 22, further comprising adjusting a pull-back speed change by setting an interval between frames of OCT data based on the registration table, and displaying a longitudinal view on a user interface based on the interval.

Claims

1. A method of operating a processor device for displaying angiographic and intravascular representations of blood vessels, comprising: generating intravascular image data in response to distance measurements of the blood vessels obtained during retraction of a probe through the blood vessels using an intravascular imaging system, said intravascular image data including a plurality of cross-sectional images at a plurality of positions along said blood vessels; generating angiographic data using an angiographic system during retraction of said probe through said blood vessels using said intravascular imaging system, said angiographic data including a plurality of two-dimensional images obtained at different times during said retraction; overlaying and registering said intravascular image data and said angiographic data; displaying a first panel including a two-dimensional representation of said blood vessels generated using said intravascular image data; displaying a second panel including a frame of said angiographic data; and wherein movement along the trajectory of said probe within said frame of said angiographic data within said second panel, as selected by a user, is also indicated by moving a frame identifier within said first panel. A method.

2. The method of claim 1, further comprising generating a score indicative of a level of confidence of said overlay registration between a frame of angiographic image data and a frame of intravascular image data.

3. The method of claim 2, further comprising identifying a portion of said blood vessels using one or more points within a frame of said angiographic image data.

4. The method of claim 3, further comprising displaying said one or more points in said second panel.

5. The method of claim 3, wherein a centerline of said blood vessels passes through said one or more points.

6. The method of claim 1, further comprising overlaying and registering said intravascular image data and said angiographic data using a blood vessel centerline to create a continuous registration of a tracked marker, said tracked marker being disposed on an intravascular data collection probe.

7. The method of claim 1, wherein said trajectory of said probe corresponds to a blood vessel centerline shown in said frame of said angiographic data within said second panel.

8. The step of registering by superimposing the intravascular image data and the angiography data further includes detecting candidates for the marker positions of the intravascular data collection probe, and using the withdrawal speed or the withdrawal length to exclude the detected candidates for the marker positions based on the withdrawal speed and / or the withdrawal length. The method according to claim 6.

9. The method according to claim 1, wherein the intravascular image data is optical coherence tomography image data or intravascular ultrasound image data.

10. The method according to claim 1, wherein the intravascular image system is an optical coherence tomography system or an intravascular ultrasound system.

11. An operating method of a processor device for displaying an angiographic representation of a blood vessel and an intravascular representation of the blood vessel, generating a set of intravascular image data in response to a distance measurement of the blood vessel obtained during withdrawal of a probe through the blood vessel using an intravascular image system, the set of intravascular image data including a plurality of cross-sectional images at a plurality of positions along the blood vessel; displaying a first panel including a first longitudinal cross-sectional view of the blood vessel generated using the set of intravascular image data; displaying a second panel including a frame of angiographic image data, the angiographic image data being generated using an angiography system during the withdrawal of the probe through the blood vessel using the intravascular image system, the angiographic image data including a plurality of two-dimensional images obtained at different times during the withdrawal; removing a guide catheter image from one or more frames of the angiographic image data; wherein movement along the trajectory of the probe within the frame of the angiographic image data within the second panel by a user selecting a point is also shown by moving a frame identifier within the first panel. Method.

12. The method according to claim 11, wherein the step of removing the guide catheter image further includes removing the guide catheter image from one or more frames of the angiographic image data using superposition of luminance profiles.

13. The method according to claim 11, wherein the set of intravascular image data is optical coherence tomography image data or intravascular ultrasound image data.

14. The method according to claim 11, wherein the intravascular imaging system is an optical coherence tomography system or an intravascular ultrasound system.

15. A method of operating a processor device for displaying an angiographic representation of a blood vessel and an intravascular representation of the blood vessel, generating a set of intravascular image data in response to distance measurements of the blood vessel obtained during retraction of a probe through the blood vessel using an intravascular imaging system, the set of intravascular image data including a plurality of cross-sectional images at a plurality of positions along the blood vessel; displaying a first panel including a two-dimensional representation of the blood vessel generated using the set of intravascular image data; displaying a second panel including a frame of angiographic image data identifying the blood vessel, the angiographic image data being generated using an angiographic system during retraction of the probe through the blood vessel using the intravascular imaging system, the set of angiographic image data including a plurality of two-dimensional images obtained at different times during the retraction; generating a vessel centerline using path information of the probe generated from one or more angiographic frames in a substantially contrast-agent-free state; and wherein movement along the vessel centerline within the frame of the angiographic image data within the second panel by a user selecting a point is also indicated by moving a frame identifier within the first panel. Method.

16. The method according to claim 15, further comprising identifying a portion of the blood vessel using one or more points of the frame of the angiographic image data.

17. The method according to claim 15, wherein the vessel centerline passes through one or more points of the frame of the angiographic image data.

18. The method according to claim 15, wherein the set of intravascular image data is optical coherence tomography image data or intravascular ultrasound image data.

19. The method according to claim 15, wherein the intravascular imaging system is an optical coherence tomography system or an intravascular ultrasound system.

20. A method of operating a processor device for registering and superimposing angiographic image data and intravascular image data obtained during retraction through a blood vessel, storing a plurality of frames of the intravascular image data in a memory; Storing a plurality of frames of angiographic image data in a memory; Processing the plurality of frames of the angiographic image data so that one or more shadows are substantially reduced; Detecting a catheter in the plurality of frames of the angiographic image data; Removing the detected catheter in the plurality of frames of the angiographic image data; Generating a vascular centerline for the plurality of frames of the angiographic image data; Detecting a probe marker in the plurality of frames of the angiographic image data; Tracking the position of the probe marker along one or more vascular centerlines; Using the tracked position to overlay and register the plurality of frames of the angiographic image data and the plurality of frames of the intravascular image data; Displaying a first panel including a two-dimensional representation of the blood vessel generated using the intravascular image data; Displaying a second panel including a frame of the angiographic image data; and Movement along the one or more vascular centerlines in the frame of the angiographic image data in the second panel by a user selecting a point is also shown by moving a frame identifier in the first panel. Method.

21. The method according to claim 20, wherein the intravascular image data is optical coherence tomography image data or intravascular ultrasound image data.

22. The method according to claim 20, further comprising generating a score indicating a level of confidence in the overlay registration between a frame of the angiographic image data and a frame of the intravascular image data.

23. The method according to claim 20, wherein the step of removing the detected catheter is performed using an overlay of luminance profiles generated based on sampling of the region of the detected catheter.

24. The method according to claim 20, wherein the step of overlaying and registering the plurality of frames of the angiographic image data and the plurality of frames of the intravascular image data includes generating a co-registration table using a computing device, the co-registration table including an angiographic image frame, a plurality of intravascular timestamps per frame, a plurality of angiographic timestamps per frame, and an intravascular image frame. **Claim 25** The method according to claim 24, further comprising the step of displaying, on a user interface, the representation of the stent in the intravascular image and the angiographic image using the registration table and the computing device. **Claim 26** The method according to claim 24, further comprising the step of identifying side branches in one or more intravascular images or angiographic images using the registration table and a user interface configured to display the side branches. **Claim 27** The method according to claim 24, further comprising the step of setting an interval between the plurality of frames of the intravascular image data based on the registration table to adjust a change in a pull-back speed, and the step of displaying a longitudinal sectional view on a user interface based on the interval.

Images