SYSTEM AND METHOD FOR LOCATING POTENTIAL FALSE POSITIVES AND BLIND SPOT POSITIVES IN CATHETER-BASED MULTIMODAL IMAGING - Patent application

Abstract

A system, method, and computer-readable medium configured to process image data to detect blind spot locations in multi-modality images acquired by an imaging catheter, the method including: acquiring a multi-modality image of a blood vessel, the multi-modality image including first and second image data that are simultaneously collected and co-registered by scanning an inner wall of the blood vessel with two or more wavelengths of light transmitted through the catheter; analyzing the first image data to identify characteristic properties of the blood vessel; detecting potential blind spot locations in the second image data of the multi-modality image based on the characteristic properties of the blood vessel; and displaying the multi-modality image on a display device together with markers that indicate the potential blind spot locations in the second image data with respect to the first image data.

Description

[Technical field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 062226, filed August 6, 2020. The disclosure of said provisional application is incorporated herein by reference in its entirety for all purposes. The benefit of priority is claimed under 35 U.S.C. § 119(e).

[0002] The present disclosure relates to image processing, and more particularly, to methods, systems and computer readable media configured to perform data processing of medical images acquired from an anatomical lumen by a multi-modality imaging catheter and algorithmic detection of blind spot regions within said images. [Background technology]

[0003] Imaging catheters configured to acquire image data from anatomical lumens are well known. In imaging systems such as photoacoustic intravascular ultrasound (IVUS) and optical coherence tomography (OCT), a motor coupled to a proximal end of the catheter rotates and translates an imaging probe located at a distal end of the catheter at a known rate. This allows the catheter to generate response signals by circumferentially and axially scanning the inner wall of a patient's blood vessel with light. The response signals are collected by a photodetector as a function of time. The rate at which the motor rotates and translates the imaging probe relative to the vessel wall is known a priori and / or is precisely tracked during the procedure. A processor receives the data collected by the detector, calculates the position at which the data was collected based on the rate at which the motor rotates and translates the imaging probe, and associates the position of the vessel wall with the data to generate an image of the patient's vasculature. The processor then displays an image showing the data as a function of the position and structural quality of the vessel wall. See, for example, U.S. Pat. No. 7,292,715, incorporated herein by reference for all purposes.

[0004] OCT is a high-resolution imaging modality that uses backscattered light to generate two-dimensional (2D) and three-dimensional (3D) images of tissue microstructure in situ and in real time. Intravascular OCT is widely used in percutaneous coronary intervention (PCI) for the investigation, diagnosis, and management of coronary artery disease. See, e.g., U.S. Pat. No. 10,109,058, incorporated herein by reference for all purposes. In addition, OCT has recently been proposed as an imaging modality for the evaluation of intracranial lesions and atherosclerotic occlusions, such as the diagnosis and evaluation of cerebral aneurysms. See, e.g., Gounis et al., “Intravascular OCT for Neurointervention,” published by the American Heart Association (AHA), Stroke, Volume 50, Issue 1, January 2019; pp. 218-223.

[0005] Although OCT provides an unprecedented level of morphological detail in the visualization of morphological features associated with vulnerable plaque, OCT still has limitations (OCT cannot identify the necrotic core or chemicals / molecules associated with plaque progression and rupture). For example, vascular tissue and the fluids contained therein (such as blood contained in blood vessels) are highly scattering, so OCT cannot identify important health-related parameters of the imaged tissue. To complement the capabilities of OCT, it has been proposed to add fluorescence as a secondary modality, to acquire and co-register images simultaneously with OCT. Fluorescence is proposed because biological tissues contain endogenous fluorophores that naturally fluoresce (i.e., absorb and re-emit light) (when illuminated with light of a certain wavelength). For example, fluorescence can detect molecules specific to the necrotic core and correlate near-infrared autofluorescence (NIRAF) intensity with plaque type. Thus, the simultaneous use of OCT imaging and fluorescence imaging modalities, such as near-infrared autofluorescence (NIRAF) and near-infrared fluorescence (NIRF), has shown the potential to improve medical imaging of tissue microstructure in situ and in real time and to improve diagnostic outcomes.See, for example, Wang et al., "Ex vivo catheter-based imaging of coronary atherosclerosis using multimodality OCT and NIRAF excited at 633 nm," Biomedical Optics Express 6(4), 1363-1375 (2015); Ughi et al., "Dual modality intravascular optical coherence tomography (OCT) and near-infrared fluorescence (NIRF) imaging: a fully automated algorithm for the distance-calibration of NIRF signal intensity for quantitative molecular imaging," Int. J. Cardiovascular. Imagine 31, 259-268 (2014); and patent publications (US2016 / 0228097, US2017 / 0209049, US2019 / 0099079, U.S. Patent No. 9,557,154, etc.).

[0006] However, recent studies have shown that image interpretation is generally user-dependent, and too much data can lead to user confusion and fatigue, leading to misinterpretation of imaging data. For example, excessive data can lead to “false positive” results that indicate the presence of a particular disease or condition when in fact such a condition is not present. To avoid misinterpretation of image data, techniques have been proposed to detect “false positive” and “false negative” results. See, for example, U.S. Pregrant Patent Application Publication Nos. 2005 / 0283058 and 2021 / 0110534, and “False positives and false negatives in functional near-infrared spectroscopy: issues, challenges, and the way forward” by Tachtsidis et al., Neurophotonics, 2016:3(3), Non-Patent Literature (NPL). These methods are generally automated and in some cases also use machine learning techniques.

[0007] Nevertheless, there is no way to notify the user of completely unclassified cases (i.e., undefined or blind spot cases) where a particular region may contain false positive or false negative cases. In such cases, the algorithm cannot adequately classify the image region as a true false positive or true false negative, and it may be advantageous to give the user an opportunity to make a practical judgment based on their own expertise. Furthermore, there is a debate among clinical supervising physicians and clinical teams about how much information should be displayed to the user for safe and accurate image interpretation. Thus, there is a need for a system and method that can notify the user of false positives, possible false negatives, and / or potential undefined ("blind spot") cases in a timely manner without presenting excessive information that may lead to user confusion, fatigue, and possible misinterpretation or misdiagnosis. Summary of the Invention

[0008] In accordance with at least one embodiment of the present disclosure, one or more systems, methods and / or storage media are provided for characterizing tissue in an image of an object or sample and determining potential blind spot locations. The images are acquired using a catheter with multiple imaging modalities (such as simultaneous OCT and fluorescence). Blind spot locations are image locations with undefined image data (which may include false positive and / or false negative image data). In this manner, the system, method and / or storage medium can notify a user of possible false positive, false negative and / or blind spot issues while simultaneously enabling the user to quickly make an actual (real-time) accurate determination of such false positive, false negative and / or blind spot cases.

[0009] According to one embodiment, the method includes: simultaneously acquiring OCT-NIRAF data of a sample using a multi-modality catheter; analyzing OCT data of the OCT-NIRAF data to identify characteristic features of the sample; detecting locations of potential false positives, false negatives and / or blind spots in the NIRAF data based on the analysis of the OCT data; and displaying the locations of the detected potential false positives, false negatives and / or blind spots together with the OCT-NIRAF data.

[0010] According to one or more embodiments, a system (100) and method (400) for detecting locations of false positives and / or blind spots in a multi-modality image includes: displaying (S402) images of each of a plurality of imaging modalities on a display device (300); analyzing the displayed image of at least one of the plurality of imaging modalities to detect one or more characteristic features of the displayed image (S408); detecting locations of potential false positives and / or blind spots for the one or more characteristic features (S410-S412); and updating (S414) the displayed image of each of the plurality of imaging modalities with markers indicating locations of potential false positives and / or blind spots for the one or more characteristic features. According to one embodiment, the plurality of imaging modalities includes an OCT modality and a fluorescence modality configured to simultaneously scan a lumen, and the characteristic features include calcified plaque in the lumen.

[0011] According to another embodiment, a method (400) for detecting locations of false positives and / or blind spots in images of a multi-modality probe includes the steps of: acquiring images of each of a plurality of imaging modalities included in the multi-modality probe (S401); analyzing a first image acquired by a first imaging modality and identifying one or more characteristic properties of the first image (S408); detecting locations of potential false positives and / or blind spots in a second image acquired by a second imaging modality based on the one or more characteristic properties of the first image (S410-S412); and displaying the first image and the second image together with markers (325) indicating locations of potential false positives and / or blind spots in the second image with respect to the one or more characteristic properties in the first image (S414).

[0012] These and other objects, features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments of the present disclosure, taken in conjunction with the accompanying drawings and the appended claims. [Brief description of the drawings]

[0013] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description when taken in conjunction with the accompanying drawings illustrating exemplary embodiments of the present disclosure.

[0014] [Figure 1] FIG. 1 illustrates an exemplary catheter-based multi-modality imaging system 100 that includes an interferometric OCT modality and a fluorescence modality. [Diagram 2] 2A and 2B show longitudinal and axial views, respectively, of the pullback operation of catheter-based multi-modality imaging system 100. [Diagram 3] FIG. 3 illustrates a display device 300 with an exemplary graphical user interface (GUI) 321 for displaying the locations of potential false positives, false negatives and / or blind spots in multi-modality data obtained by scanning a biological sample with a multi-modality imaging catheter. [Figure 4] FIG. 4 illustrates an example algorithm 400 for detecting potential false positives, false negatives, and / or blind spot locations according to one embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates a workflow process 500 for detecting potential false positives, false negatives, and / or blind spot locations according to another embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates a workflow process 600 for detecting potential false positives, false negatives, and / or blind spot locations according to yet another embodiment of the present disclosure. [Figure 7] FIG. 7 shows a comparison of histological (H&E, Masson's Trichrome), OCT-NIRAF and confocal NIRAF images to identify the location of potential false positives, false negatives and / or blind spots of fluorescence. [Figure 8] Figure 8A shows an algorithm for registering an OCT-NIRAF image with a confocal NIRAF image, and Figure 8B shows a registered image divided into multiple segments to analyze characteristic features and detect the location of false positives, false negatives, and / or blind spots in the fluorescence data. [Figure 9-1] 9A-9C illustrate an exemplary embodiment of a graphical user interface (GUI) configured to display the locations of potential false positives, false negatives, and / or blind spots in a multi-modality OCT-fluorescence image acquired by the catheter-based imaging system 100. [Figure 9-2] 9D-9F illustrate an exemplary embodiment of a graphical user interface (GUI) configured to display the locations of potential false positives, false negatives, and / or blind spots in a multi-modality OCT-fluorescence image acquired by the catheter-based imaging system 100. [Figure 10] 10A, 10B, 10C and 10D illustrate various examples of further embodiments of graphical user interfaces configured to display the locations of potential false positives, false negatives and / or blind spots of fluorescence in a multi-modality OCT-fluorescence image. [Figure 11] FIG. 11 illustrates a display device 300 with a graphical user interface (GUI) 321 configured to display the locations of potential false positives, false negatives and / or blind spots using a simplified binary panel. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure provides one or more systems, methods, and storage media configured to analyze image data to characterize tissue and determine the location of potential false positives, false negatives, and / or blind spots in an image of an object or sample. The images are acquired using multiple imaging modalities (such as simultaneous OCT and fluorescence) contained within a single fiber-based catheter. Several embodiments of the present disclosure (which may be implemented by one or more embodiments of the systems, methods, and / or computer-readable storage media of the present disclosure) are diagrammatically and visually described with reference to FIGS. 1-11.

[0016] Before various embodiments are described in further detail, it is to be understood that the present disclosure is not limited to particular embodiments, and it is also to be understood that the terminology used herein is for the purpose of describing example embodiments only and is not intended to be limiting.

[0017] Throughout the figures, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the present disclosure will be described in detail with reference to the enclosed figures, it is done so in connection with the illustrated embodiments. It is intended that changes and modifications can be made to the illustrated exemplary embodiments without departing from the true scope and spirit of the present disclosure as defined by the appended claims. While the drawings represent some possible configurations and approaches, the drawings are not necessarily to scale and certain features may be exaggerated, omitted or partially cut off in order to more clearly illustrate and explain certain aspects of the present disclosure. The description set forth herein is not intended to be exhaustive or otherwise limit or restrict the scope of the claims to the exact forms and configurations shown in the drawings and disclosed in the following detailed description.

[0018] Those skilled in the art will appreciate that the terms used herein, in general, and in the appended claims, particularly (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," "includes" should be interpreted as "including, but not limited to," etc.). Moreover, those skilled in the art will appreciate that, where specific numbers of introduced claim recitations are intended, such intent will be expressly set forth in the claims, and, in the absence of such recitation, no such intent exists. For example, as an aid to understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that the introduction of a claim statement with the indefinite article "a" or "an" means that a particular claim containing such an introduced claim statement is limited to claims containing only one such statement, even if the same claim contains both the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should typically be construed to mean "at least one" or "one or more"). The same is true for the use of definite articles used to introduce claim statements.

[0019] Furthermore, even if specific numbers in the claims are explicitly stated, it is understood by those skilled in the art that such a statement should typically be interpreted to mean at least the stated number (e.g., when only "two statements" is stated without other modifiers, it typically means at least two statements, or two or more statements). Furthermore, when a definition similar to "at least one of A, B, and C, etc." is used, such a syntax is generally intended in the sense that a person skilled in the art can understand the definition (e.g., "a system having at least one of A, B, and C" may include a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together). When a definition similar to "at least one of A, B, or C, etc." is used, such syntax is generally intended in the sense that one of ordinary skill in the art would understand the definition (e.g., "a system having at least one of A, B, or C" can include a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Furthermore, one of ordinary skill in the art will appreciate that typically disjunctions and / or phrases (whether in the specification, claims, or drawings) that express two or more alternative terms should be understood to contemplate the possibility of including one of the terms, either of the terms, or both of the terms, unless the context indicates otherwise. For example, the expression "A or B" is typically understood to include the possibilities of "A" or "B" or "A and B."

[0020] In this specification, when a feature or element is referred to as being "on" another feature or element, it may be directly on the other feature or element, or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It should also be understood that when a feature or element is referred to as being "connected," "attached," "coupled," or the like, to another feature or element, it may be directly connected, attached, or coupled to the other feature, or there may be intervening features or elements present. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or illustrated with respect to one embodiment, features and elements so described or illustrated in one embodiment may be applicable to other embodiments. Additionally, as will be understood by one of ordinary skill in the art, a reference to a structure or feature being located "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0021] In this specification, terms such as first, second, third, etc. may be used to describe various elements, components, regions, parts, and / or portions. It is to be understood that these elements, components, regions, parts, and / or portions are not limited by these designated terms. These designated terms are used only to distinguish one element, component, region, part, or portion from another region, part, or portion. Thus, a first element, component, region, part, or portion described below can be referred to as a second element, component, region, part, or portion merely for purposes of distinction, but without limitation, and without departing from the structural or functional meaning.

[0022] As used herein, the singular is intended to include the plural unless the context clearly dictates otherwise. Furthermore, it should be understood that the terms "comprise," "comprising," and "consisting," when used in this specification and claims, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof that are not expressly recited. Furthermore, in this disclosure, the transitional phrase "consisting of" excludes any element, step, or component not specified in the claim. It should also be noted that some claims or some features of claims may be drafted to exclude any element, and such claims may use exclusive language such as "solely," "only," or "negative" limitations in connection with the recitation of the claim elements.

[0023] The term "about" or "approximately" as used herein means, for example, within 10%, within 5% or less. In some embodiments, the term "about" may mean within measurement error. In this regard, when described or claimed, all numerical values ​​may be read as if they were preceded by the word "about" or "approximately", even if the term is not explicitly indicated. The phrase "about" or "approximately" may be used when describing a size and / or location to indicate that the stated value and / or location is within a reasonable expected range of value and / or location. For example, a numerical value may include values ​​that are ±0.1% of the stated value (or range of values), ±1% of the stated value (or range of values), ±2% of the stated value (or range of values), ±5% of the stated value (or range of values), ±10% of the stated value (or range of values), etc. As described herein, any numerical range is intended to include the end value and includes all subranges therein, unless specifically stated otherwise. As used herein, the term "substantially" means allowing for deviations from the descriptor that do not adversely affect the intended purpose. For example, deviations resulting from limitations in measurements, differences within manufacturing tolerances, or variations of less than 5% can be considered to be within substantially the same range. The specified descriptors can be absolute (e.g., substantially spherical, substantially vertical, substantially concentric, etc.) or relative terms (e.g., substantially similar, substantially the same, etc.).

[0024] Unless specifically stated otherwise, and as will be apparent from the disclosure that follows, discussions throughout this disclosure using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like are understood to refer to operations and processes of a computer system, or similar electronic computing device, or data processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of a computer system into other data that is also represented as physical quantities in the memory or registers of a computer system or other device for storing, transmitting, or displaying such information. Computer or electronic operations described herein or recited in the appended claims can generally be performed in any order, unless the context dictates otherwise. Also, while various operational flow diagrams are presented in a sequential order, it will be understood that various operations can be performed in orders other than those shown or claimed, or operations can be performed simultaneously. Examples of such alternative orderings include overlapping, interleaving, interrupting, reordering, incrementing, preparing, supplementing, simultaneous, reverse, or other variations of ordering, unless the context dictates otherwise. Moreover, terms such as "in response to," "in response to," "in connection with," "based on," or other similar past tense adjectives generally do not intend to exclude such variations unless the context indicates otherwise.

[0025] The present disclosure generally relates to medical devices and illustrates embodiments of a combination optical probe and a fluorescence imaging probe, such as an optical coherence tomography (OCT) catheter, i.e., a multi-modality OCT (MMOCT) catheter or probe. The optical probe and portions thereof are described with respect to their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates), the term "orientation" refers to the rotational configuration of an object or a portion of an object (three rotational degrees of freedom - e.g., roll, pitch, yaw), the term "pose" refers to the position of an object or a portion of an object in at least one translational degree of freedom and the orientation of an object or a portion of an object in at least one rotational degree of freedom (up to six degrees of freedom in total), and the term "shape" refers to a set of poses, positions and / or orientations measured along the elongated body of an object.

[0026] As is known in the medical instrument art, the terms "proximal" and "distal" are used in reference to the operating ends of an instrument that extend from a user to a surgical or diagnostic site. In this regard, the term "proximal" refers to the portion of the instrument closer to the user (e.g., the handle) and the term "distal" refers to the portion of the instrument that is further from the user and closer to the surgical or diagnostic site (the tip). It will further be appreciated that for convenience and brevity, spatial terms such as "vertical," "parallel," "upper," and "lower" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0027] As used herein, the term "catheter" generally refers to a flexible, thin, tubular device made of medical grade materials designed to be inserted through a narrow opening into a body cavity (e.g., a blood vessel) to perform a wide range of medical functions. The more specific term "optical catheter" refers to a medical device that includes an elongated bundle of one or more flexible, light-conducting fibers with optical imaging capabilities disposed within a protective sheath made of medical grade materials. A specific example of an optical catheter is a fiber optic catheter that includes a sheath, a coil, a protector, and an optical probe. In some applications, the catheter may include a "guide catheter" that functions similarly to the sheath.

[0028] In this disclosure, terms such as "optical fiber" or simply "fiber" refer to an elongated, flexible, light-conducting tube capable of conducting light from one end to the other by an effect known as total internal reflection. The terms "light-guiding component" or "waveguide" may also refer to or function as optical fibers. The term "fiber" may refer to one or more light-conducting fibers. Optical fibers generally have a transparent, homogenous core through which light is guided and the core is surrounded by a homogenous cladding. The refractive index of the core is greater than the refractive index of the cladding. Depending on design choices, some fibers may have multiple claddings surrounding the core.

[0029] As used herein, a "false positive" value is an error in binary classification where a test result erroneously indicates the presence of a condition when the condition is not present, and a "false negative" is the opposite error where a test result erroneously fails to indicate the presence of a condition when the condition is actually present. These are two types of errors in contrast to the two correct results in a binary test: true positive and true negative. The terms "potential" false positive, false negative, and / or blind spot refer to candidate locations identified by automated (algorithmic) analysis. According to the present disclosure, "potential" false positive, potential false negative, and / or potential blind spot results, occurrences, or events are determined by the system and accepted or rejected by the user.

[0030] <Multi-modality OCT (MMOCT) imaging system> According to one embodiment, a multi-modality OCT-NIRAF imaging system including a system console and a multi-modality catheter is configured to acquire OCT-NIRAF images from a luminal sample, such as a cardiac vessel. The OCT images can be acquired using a swept-wavelength laser with a central wavelength of 1310 nm and a bandwidth of 127 nm. The NIRAF images can be acquired by exciting the luminal sample at 633 nm and detecting the fluorescence emission in the wavelength range of 660 nm to 740 nm. The OCT and NIRAF illumination light was delivered to the vessel through a double-clad fiber in the catheter. The catheter was connected to the imaging system through a patient interface unit that provides mechanical helical scanning. The system acquires synchronized and coregiven OCT and NIRAF data at a rate of 200 frames per second with a pullback speed of 10 to 40 mm / s. Each OCT-NIRAF image contains 500 A-lines.

[0031] FIG. 1 illustrates an exemplary multi-modality OCT (MMOCT) imaging system 100 including an interferometric OCT modality and a fluorescent modality. The system 100 may be used for intravascular imaging or may be configured with a balloon catheter for esophageal imaging or imaging of other similar body cavities. As shown in FIG. 1, the OCT modality consists of an interferometer (e.g., a Michelson interferometer) having a sample arm 10 and a reference arm 20, an OCT light source 110, a detector 120, a data acquisition (DAQ) unit 130, and a computer 200. The computer 200 is connected to an external system such as a display device 300 or a picture archiving and communication system (PACS) 1400. The sample arm 10 includes a patient interface unit (PIU) 150 and a fiber-based catheter 160. The fluorescent modality consists of an excitation light source 180, a catheter 160, a photodetector 183, a data acquisition (DAQ) unit 130, and a computer 200. In the fluorescence modality, a light source 180 is connected to the PIU 150 via an optical fiber 181 .

[0032] The PIU 150 includes a beam combiner (not shown), a fiber optic rotary joint 152, and a pullback unit 151 (e.g., a precision linear stage). In one embodiment, the system 100 uses a swept-wavelength laser (1310 nm±50 nm) as the OCT light source 110 for the OCT modality and a helium-neon (He:Ne) laser with a central wavelength of about 633 nm as the excitation light source 180 for the fluorescence modality. The catheter 160 includes an imaging core consisting of a double-clad fiber (DCF) 167 and a distal optics assembly 168. The distal optics assembly may include a polished ball lens at the tip of the DCF 167 for side-view imaging. Alternatively, the distal optics assembly 168 may include a gradient index (GRIN) lens and a reflective element (diffraction grating) attached to the tip of the DCF 167. At the proximal end, the catheter 160 is connected to the PIU 150 via a catheter connector 161.

[0033] The imaging system 100 is configured to simultaneously acquire OCT and fluorescence images from a sample 170, which may include a biological lumen such as a blood vessel. To that end, light or other electromagnetic radiation (radiation at a first wavelength) from an OCT light source 110 is directed through a sample arm 10 to the sample 170, through a reference arm 20 to a reflector 140, and the light returns along its respective optical path, thereby generating OCT interference fringes. Specifically, the light from the light source 110 is split (e.g., 50 / 50) by a splitter 102 (fiber splitter or beam splitter) into a sample beam and a reference beam, which are sent to the sample arm 10 and the reference arm 20, respectively, via respective optical fibers. In the sample arm 10, the sample beam enters a circulator 105 and travels to a fiber coupler 108 via a single mode (SM) fiber 106, and the sample beam is sent to the PIU 150 via a double clad fiber 107. The catheter 160 is connected to the PIU 150, which is connected to the computer 200 (through electronic connections not shown). Under the control of the computer 200, the PIU 150 controls the rotation of the imaging core of the catheter 160 to irradiate the sample beam to the sample 170 in a scanning manner. The light of the sample beam reflected and / or scattered by the sample 170 is collected by a distal optics 168 (optical probe) located at the distal end of the catheter 160, and the collected light is sent back to the PIU 150 through a double clad fiber 167. The collected light (sample beam) travels from the PIU through a fiber 107 to a fiber coupler 108. The fiber coupler 108 sends a portion of the returned light to the circulator 105 via a SM fiber 106. The circulator 105 then directs a portion of the returned light to the combiner 104. In addition, the fiber coupler 108 couples another portion of the returned light through a multimode fiber 109 to a second detector 122 .

[0034] In the reference arm 20, the light of the reference beam enters the circulator 103 and is sent to the reflector 140 via an optical fiber 141. For time-domain OCT (TD-OCT) imaging, the reflector 140 can be implemented by a scanning mirror and an optical delay line (ODL). For frequency-domain OCT (FD-OCT) imaging, the reflector 140 can be implemented as a fixed mirror. The light of the reference beam reflected from the reflector 140 passes through the circulator 103 and is directed to the combiner 104. Thus, the sample beam and the reference beam are combined in the beam combiner 104 and then detected by the detector 121 to generate an interference signal according to known OCT principles.

[0035] The detector 121 (first detector) is implemented as an array of photodiodes, a photomultiplier tube (PMT), a multi-array camera, or other similar interference fringe detection equipment. In at least one embodiment, the detector 121 can be a balanced photodetector. The signal output from the first detector 121 is pre-processed (digitized) by the data acquisition electronics (DAQ1) 131 and transmitted to the computer 200. The computer 200 performs signal processing to generate an OCT image in a known manner. Interference fringes are generated only when the optical path length of the sample arm 10 matches the optical path length of the reference arm 20 within the coherence length of the OCT light source 110. Polarization-sensitive OCT measurements can be performed using polarization-maintaining (PM) optical fibers or through an in-line paddle-type polarization controller (PC).

[0036] In the fluorescence modality, the excitation light source 180 emits excitation light (radiation at a second wavelength) with a central wavelength of 633 nm. In other embodiments, the excitation light can have a different central wavelength (such as 485 nm) depending on the desired application. The excitation light is guided by the fiber 181, the FORJ 152, the double clad fiber 167, and the distal optics 168 to irradiate the sample 170. In response to irradiation with the excitation light, the sample 170 emits a near-infrared autofluorescence (NIRAF) signal or a near-infrared fluorescence (NIRF) signal in a broad band of wavelengths in the range higher than the excitation wavelength (radiation at a third wavelength (e.g., 633-800 nm)) according to known fluorescence emission principles. Fluorescence, as used herein, is an optical phenomenon in which energy in the form of photons is absorbed by a molecule, resulting in the immediate emission of fluorescence photons at a wavelength longer than the excitation light.

[0037] In one embodiment, the fluorescent light generated by the sample 170 may include autofluorescence, which is intrinsic fluorescence that occurs without the addition of dyes or agents. In other embodiments, the fluorescent light generated by the sample 170 may include fluorescence caused by exogenous fluorescence of dyes or contrast agents added to the sample (e.g., during luminal clearance). The autofluorescent (or fluorescent) light is collected by the distal optics 168 of the catheter 160 and sent back to the PIU 150, where the FORJ 152 and a beam combiner / splitter (not shown) send the fluorescent signal to the detector 183 via optical fiber 189. The signal (fluorescence intensity signal) output from the detector 183 is digitized by the data acquisition unit (DAQ2) 132 and sent to the computer 200 for image processing. The OCT interference fringes of the OCT modality and the fluorescent signal of the fluorescence modality are preferably sent to the computer 200 simultaneously.

[0038] The second detector 122 detects a portion of the sample beam transmitted from the fiber coupler 108 via the multimode fiber 109. The second detector 122 outputs an analog signal corresponding to the intensity of the backscattered light (backscattered signal). The backscattered signal returning from the sample 170 and detected by the second detector 122 is not an interference signal. The signal output from the detector 122 is converted to digital data by the data acquisition electronics (DAQ2) 132. In particular, as will be described in more detail below, the digital signal corresponding to the intensity of the backscattered light can be used to calculate the distance and / or angle at which the light from the catheter enters the sample 170. The intensity of the backscattered light can also be used as a trigger signal to start and / or end the pullback and image recording operations. Thus, the signal output from the detector 122 and converted to digital data by the data acquisition electronics (DAQ2) 132 can be used directly as a trigger signal or can be transferred to the computer 200 for controlling the process.

[0039] As shown in FIG. 1, the computer 200 includes a central processing unit (CPU) 191, a storage memory (ROM / RAM) 192, a user input / output (I / O) interface 193, and a system interface 194. The various functional components of the computer 200 are operatively connected and communicate with each other via physical and logical data lines (data bus) 195. The storage memory 192 includes one or more computer-readable and / or computer-writable media, and may include, for example, a magnetic disk (e.g., a hard disk drive HHD or a solid-state drive SSD), an optical disk (e.g., a DVD, Blu-ray, or wire), a magneto-optical disk, a semiconductor memory (e.g., a non-volatile memory card, a Flash memory, a solid-state drive, a SRAM, a DRAM), an EPROM, an EEPROM, etc. The storage memory 192 may store computer-readable data and / or computer-executable instructions, such as an operating system (OS) program, a control program, and a processing program.

[0040] The user interface 193 provides a communication interface (electrical connections) for input / output (I / O) devices, including a keyboard, a display device 300 (such as an LCD or OLED display screen), a mouse, a printing device, a touch screen, a light pen, an external optical storage device, a scanner, a microphone, a camera, a drive, a communication cable, and a network (either wired or wireless). The system interface 194 also provides an electronic interface (electronic connection circuitry) for one or more of the light source 110 of the OCT modality, the excitation light source 180 of the fluorescence modality, one or more detectors 121 of the OCT modality, the detector 183 of the fluorescence modality, and the data acquisition (DAQ2) circuitry 132, and the patient interface unit (PIU) 150. The system interface 194 may include other components, including programmable logic used in conjunction with a programmable logic device (PDL) (such as a field programmable gate array (FPGA) or other PLD), discrete components, integrated circuits (e.g., application specific integrated circuits (ASICs)), or any combination thereof. The functionality of the user interface 193 and the system interface 194 may be realized, at least in part, by computer executable instructions (e.g., one or more programs) stored in the storage device 192 and executed by the CPU 191, or by programs executed remotely over a network (e.g., cloud computing). Furthermore, the computer 200 may include one or more additional devices, such as components such as communication means and network interfaces for communicating with other medical devices (displays, printers, PACS 1400, etc.). The functional operations of the imaging system 100 shown in FIG. 1 are realized by the CPU 191 of the computer 200 executing executable instructions or programs stored in the ROM / RAM of the storage device 192.

[0041] 2A and 2B show example longitudinal and axial views of the distal optics 168 (optical probe) of the catheter 160 at successive positions during a pullback motion, respectively. As shown in FIG. 2A, the distal optics 168 (or optical probe) includes a double clad fiber (DCF) 167, a lens 166 (focusing component), such as a GRIN lens or a ball lens, and a reflective or diffractive surface 165 (beam directing component). At its proximal end, the catheter 160 is connected to the PIU 150 via a connector 161 (shown in FIG. 1). A torque coil 163 transfers a rotational torque from a rotation motor (not shown) located in the PIU 150 to the distal end of the catheter 160. At the distal end of the catheter 160, a reflective or diffractive surface 165 (mirror, prism, grating, etc.) radially deflects the illumination light (illumination light beam 11) toward a sample 170 (wall of the lumen) located at a working distance (Wd). 2A, the imaging probe 168 is configured for side-view imaging, where the illumination light beam 11 exits the catheter at an angle θ with respect to the catheter axis Ox and is incident on the sample surface 171 along a line transverse to the catheter axis Ox. Here, since the OCT and fluorescence data are acquired simultaneously by a common probe 168, the illumination light beam 11 refers to the light incident on the sample 170 (whether the beam 11 is light emitted by the OCT light source 110 and / or the excitation light source 180).

[0042] The catheter 160 rotates or oscillates (as indicated by arrow R) about the catheter axis Ox while the optical probe is pulled back within a protective sheath 162 that remains stationary while the illumination beam 11 (including OCT light and excitation light) illuminates the body cavity (e.g., blood vessel wall). In this way, the catheter 160 can continuously scan the sample 170 with the illumination beam 11 in a rotational manner (as shown in FIG. 2B) through successive radial positions. The distal optics or probe 168 collects the return light 12 (sample beam) including backscattered light and fluorescence from the sample 170. The return light 12 conveys information about the inner surface 171 (e.g., blood vessel wall). An interference signal (interference fringes) is obtained by combining the interference light beam (not shown) with the backscattered OCT light collected from the sample 170. At the same time, the fluorescence signal is collected, detected and stored. As discussed above with reference to Figure 1, the interferometric OCT and fluorescence signals are converted to electronic signals that are digitized, stored and / or processed to analyze the condition of the sample under examination. Combining the backscattered light from the sample beam 12 with the reference light from a reference beam (not shown) results in an interference signal only if the light in both the sample and reference beams has traveled approximately the same optical distance ("approximately the same optical distance" refers to a difference of no more than the coherence length of the light source).

[0043] Areas of the sample 170 that reflect more light will produce a stronger interference signal than areas that reflect less light. Light beyond the coherence length will not contribute to the interference signal. The intensity profile of the backscattered light, also called an A-scan or A-line, contains information about the spatial dimensions and location of characteristic features, such as the depth tissue layer 172, the location of calcified plaques 173 formed within the wall or layer of the lumen (sample surface) 171, and the location and size of the stent 174. An OCT image (i.e., a tomogram, commonly referred to as a B-scan) can be formed by combining multiple A-scans acquired at different locations along the length of the lumen wall.

[0044] The diagram in Figure 2A shows the catheter position (multiple longitudinal positions T1, T2, T3, T4) at corresponding times t1, t2, t3, t4 etc. along the pullback path, where different A-scans (A1, A2, A3, A4...) are collected while scanning a sample 170 at a fixed angle θ with an illumination light beam 11. Delta (δ) is the distance the catheter travels during pullback between successive one rotation measurements (or A-scans). In the example pullback of Figure 2A, a first A-line scan (A1) at pullback position T1 can provide information about tissue layers and artifact formation (neointima formation), a second A-line scan (A2) at position T2 can provide information about the stent 174 (stent thrombosis, stent crimping, stent incomplete crimping, stent edge dissection, stent strut coverage or detection, etc.), a third A-line scan (A3) at position T3 may include information from only the tissue layers of the sample 170, and a fourth A-line scan (A4) at position T4 may include information about plaque 175 accumulated on the surface of the sample 170 (plaque type, plaque depth, etc.). Information obtained from the A-line scans can be correlated with information obtained from the fluorescent signal to improve diagnostic results.

[0045] FIG. 2B shows an axial view (XY plan view) of the distal end of the catheter 160, with an exemplary illumination light 11 incident on a sample 170 (e.g., a blood vessel wall) at multiple rotational positions R1, R2, R3, 4 along a helical path. Measurements at each rotational position R1-R4 are performed while scanning the sample 170 with the light beam 11 at a fixed (identical) angle θ. Each of the positions R1, R2, R3, R4 represents a different rotational position on the intraluminal surface at which measurements are taken while the catheter 160 rotates. It should therefore be appreciated that the resulting signal detected at each rotational position may have different characteristics, depending, for example, on the distance (Wd) between the catheter 160 and the inner surface 171 (luminal edge) of the sample 170. In this regard, it is known that due to the tortuous nature of the luminal sample 170, the catheter may bend and acquire images while touching the luminal edge. In this case, the acquired signals may be subject to image artifacts that may lead to misinterpretation of the image data. For example, if the catheter 160 is bent such that the catheter axis is off-center in the lumen and an image is acquired while the catheter is in contact with a stent 174 or plaque 175 (as shown in FIG. 2A), the signals collected may include erroneous fluorescence intensity signals (e.g., false positive signals). It is therefore important to provide the user with the option to verify the true characteristics of such signals in real time.

[0046] A-lines can be generated multiple times by helically scanning the inner surface 171 of the sample 170 by combining the pullback motion in the -z direction of the catheter 160 with the rotation motion R. Multiple A-line scans can be combined to generate a 2D image (cross section) of the sample 170. Each 2D image of the vessel cross section can be formed by combining, for example, approximately 500 or more lines corresponding to a full circumferential (360 degree) scan of the catheter 160. Such a full circumferential scan can be referred to as a "frame." Three-dimensional (3D) imaging of the inner surface 171 can be achieved by combining multiple 2D image frames acquired during the longitudinal translation of the pullback motion as the catheter rotates. The resulting catheter scan is a helical path of successive A-lines to form a complete 3D data set (3D volume) of the inner surface 171 of the sample 170. The data collected from the successive A-line scans is processed (e.g., by fast Fourier transform or other known algorithms) to generate an OCT image of the sample 170 in a known manner. Simultaneously, the fluorescence signal is collected, processed, stored / displayed, and analyzed corresponding to the OCT image.

[0047] Determining the location of potential false positives, false negatives and / or blind spots The results of the pullback and image recording operations are typically displayed "as detected." That is, the fluorescence information and OCT data are displayed together as detected, along with all available information regarding the luminal sample and / or catheter parameters. Display of the fluorescence information (NIRAF or NIRF) may include displaying the fluorescence data in spatial association with the OCT data and / or displaying the particular catheter parameters for which the data was acquired. In this regard, it is common to display a ring of fluorescence data around a tomographic image of the OCT data centered on the catheter axis. It is also known to display the OCT data and the fluorescence data side-by-side in a longitudinal section to show a longitudinal view of the vessel as the catheter imaging core performs the pullback operation. Other systems may display the OCT data and the fluorescence data superimposed on each other in either a tomographic or longitudinal view, in either a 2D or 3D representation. The present disclosure is not limited to a particular display format, so long as the image data of both the fluorescence imaging data and the OCT imaging data are displayed in spatial registration, and the location of potential false positives, false negatives, and / or blind spots can be identified by the user. Examples of various methods for displaying multi-modality OCT-fluorescence data are described in some of the applicant's published patent and patent application publications, including US2019 / 0339850, US2019 / 0029624, US2018 / 0271614 (now U.S. Pat. No. 10,842,589, etc.), and U.S. Pat. No. 10,109,058, all of which are incorporated by reference herein for all purposes.

[0048] Advantageously, when catheter-based NIRAF is correlated with intraplaque features and other characteristic features determined from OCT data, it can not only identify morphologically high-risk plaques, but also assess pathologically high-risk plaques with sufficient accuracy to improve clinical outcomes and provide better diagnostic results for patient care.

[0049] However, detection of fluorescence data acquired through a catheter may be affected by the power of the excitation light, the position of the catheter relative to the luminal sample, the tissue structure of the sample, etc. For example, in some embodiments, the catheter may be configured to provide optical power per unit area (e.g., W / ma 2 or W / cm 2 In some medical applications, excitation lasers may emit excitation light at low output powers, measured as 1 mW. In such cases, the detection sensitivity of the system may not be optimal. However, such low excitation light output is not necessarily a negative constraint, as it can be designed to meet safety concerns that are deemed desirable for patient safety.

[0050] To optimize detection sensitivity, according to various embodiments of the present disclosure, proper detection and calibration of the catheter-lumen distance and accurate detection of the fluorescent signal are based on analysis of OCT data, catheter parameters, and other factors related to the procedure and the target sample. This allows the system to identify potential false positive, false negative, and / or blind spot areas from non-NIRAF information. To identify "false positive" cases, the system can utilize the lumen detection results and evaluate the distance from the lumen to the catheter sheath. To identify "blind spot" cases, the system can utilize plaque type characterization (which can be performed by image processing of intravascular OCT and / or spectroscopic OCT (S-OCT)). S-OCT is a signal processing technique that uses raw interferograms generated by OCT interferometry to examine the depth-resolved spectroscopic profile of a sample. The spectroscopic information provided by S-OCT can be used to enhance the contrast of OCT images and overcome the limitations of grayscale OCT images that depict only morphology. See Hyeong Soo Nam and Hongki Yoo, “Spectroscopic optical coherence tomography: A review of concepts and biomedical applications,” Applied Spectroscopy Reviews, 53:2-4, 91-111, 2018, DOI:10.1080 / 05704928.2017.1324876.

[0051] When processing multi-modality data of samples using fluorescent signals, signal attenuation (attenuation coefficient) by calcified tissue has the greatest impact on NIRAF detection. Other information of characteristic properties can include, but is not limited to, lumen diameter, external elastic lamina (EEL), lumen or catheter geometric properties (ovalization, collapse, etc.), lumen flushing efficiency, etc., as described below with reference to the image processing algorithms in Figures 4, 5, and 6.

[0052] An example of suitable detection and calibration of the catheter-lumen distance is disclosed in the applicant's previously published US Patent Application Publication No. 2019 / 0298174. An example of image processing for plaque type characterization includes, but is not limited to, US Patent Application Publication No. 2017 / 0209049. Both of these publications are incorporated herein by reference in their entirety. Briefly, the acquired OCT-NIRAF data was post-processed for distance calibration. This post-processing was performed after histological processing (described below). First, the background signal was subtracted from the detected NIRAF signal. The background signal was estimated by averaging the NIRAF signal levels from the luminal samples imaged under phosphate buffered saline (PBS) solution. The background subtracted signal was then calibrated to remove the effect of the distance between the luminal edge and the catheter. The distance between the luminal edge and the catheter was estimated from the OCT images without the use of fluorescence data.

[0053] The system may be configured to display warnings of potential false positive, false negative and / or blind spot cases (locations) in the acquired multi-modality images. The warnings may be programmed to be switched on / off according to user preferences. When considering the catheter output, the system may be configured to automatically mark areas around potential false positive, false negative and / or blind spot cases based on comparing the fluorescence to a minimum detection threshold. For example, the system limit of detection (LOD) value can be used as the threshold. Then, during image processing, if the fluorescence signal does not meet the threshold, the system marks the area of ​​the multi-modality image as a potential false positive, false negative and / or blind spot. Furthermore, if the system finds an area that does not meet the set threshold intensity, the system can also generate a warning taking into account the distance from the catheter to the lumen in this area. For example, if the system detects an area of ​​weak signal, and the distance from the lumen to the catheter in that area is greater than threshold A and the signal is less than threshold B, this will be the warning area. To simplify the user's evaluation and decision-making process, the system may be configured to indicate potential false positives, false negatives and / or blind spot areas using different display patterns, colors and marks, as described below with respect to Figures 9, 10A-10D and 11.

[0054] According to one or more embodiments, the system 100 of FIG. 1 is configured to acquire combined OCT-fluorescence data, analyze the OCT data, detect the location of potential false positives, false negatives, and / or blind spots in the fluorescence data based on the analysis of the OCT data, and display the OCT-fluorescence data along with the detected locations of potential false positives, false negatives, and / or blind spots. Acquiring the OCT-fluorescence data may include simultaneously acquiring OCT-fluorescence images by a multi-modality catheter and performing an in situ and real-time OCT analysis. In some embodiments, the system may import already pre-acquired multi-modality OCT-fluorescence data for OCT analysis. An important aspect of the process includes using the OCT data (non-fluorescence data) and / or the analysis results of the non-fluorescence data to identify the locations of potential false positives, false negatives, and / or blind spots of certain features of the fluorescence data. The fluorescence data may include near-infrared autofluorescence (NIRAF) or near-infrared fluorescence (NIRF) data.

[0055] FIG. 3 illustrates an exemplary graphical user interface (GUI) 321 displayed on the screen of the display device 300 to provide a user with information regarding the location of potential false positives, false negatives, and / or blind spots in the multi-modality image data of the luminal sample 170. The multi-modality image data is acquired by the MMOCT system 100 of FIG. 1 according to the exemplary process shown in FIG. 2A and FIG. 2B. In FIG. 3, the GUI 321 shown on the display device 300 is controlled by the CPU 291 of the computer 200 according to a customized algorithm (FIGS. 4-8B) to display the coregistered multi-modality image data (e.g., fluorescence image data and OCT image data) on multiple separate display panels (a first panel 321-1, a second panel 321-2, a third panel 321-3, etc.). The first panel 321-1 can display an angiogram 340 and a multi-modality image consisting of a tomographic image of the OCT data 310 combined with the fluorescence data 320. The second panel 321-2 can display a longitudinal image of the luminal sample, which shows only the OCT data 310 for evaluating the structural properties of the luminal sample. The third panel 321-3 is configured to display a longitudinal image of the fluorescent data. The longitudinal images of the OCT data 310 and the fluorescent data 320 are co-registered to show a longitudinal correspondence of the fluorescent data with respect to the OCT data along the length of the luminal sample. The tomographic image displayed in the first panel 321-1 shows a circular (polar) correspondence of the co-registered fluorescent data 320 and the OCT data 310 along the circumference of the luminal sample. The position of the luminal sample where the tomographic image is acquired is identified in the longitudinal image by a line indicator 315 shown in the second panel 321-2 and a box frame 335 shown in the third panel 321-3. The line indicator 315 may be a movable component of the GUI 321 and may be interactively controlled by a user to move bidirectionally along the length of the longitudinal image using a pointer 316, such as a mouse or joystick, or by touching the screen of the display 320.In this manner, the user can interactively select the longitudinal location of the luminal sample where the characteristic property needs to be evaluated. As the user moves the pointer 316, the line indicator 315 moves along with the bounding box 335, and the tomogram changes depending on the position of the line indicator 315.

[0056] In the first panel 321-1, the tomographic image of the OCT data 310 shows an axial view (first image) of the luminal sample, and the fluorescence data 320 is shown as an arc or partial ring located at the edge of the tomographic image. However, if the fluorescence data 320 is suspected to be a potential false positive, false negative, and / or blind spot, the system adds a marker 325. The system may determine that the fluorescence data is not a true positive signal for a variety of reasons, such as, for example, the structural characteristics of the OCT data 310 do not confirm that the fluorescence signal originates from a fluorescent source. In other words, as will be described in more detail below, the system cannot confirm that the fluorescence data 320 is a positive fluorescence signal. Thus, the system displays a marker 325, shown as a dotted line, around the fluorescence data 320. The marker 325 should be displayed in a distinctive and conspicuous manner to alert the user to the presence of potential false positive, false negative, and / or blind spot (artifact) data in the multi-modality image.

[0057] In the second panel 321-2 and the third panel 321-3, the positions of the fluorescence data 320 relative to the OCT data 310 are shown in the longitudinal image (second image) alongside each other. In other words, the same fluorescence data 320 displayed near the edge of the tomographic image is also shown as the position of potential false positives, false negatives and / or blind spots, for example by a bounding box 335 in the longitudinal image of the third panel 321-3. It should be noted here that the OCT data 310 and the fluorescence data 320 do not need to be displayed separately from each other. In at least some embodiments, the OCT data 310 and the fluorescence data 320 can be displayed superimposed on each other, as long as the positions of the suspected false positives, false negatives and / or blind spots are clearly identifiable by the user.

[0058] In at least some embodiments, the display of the potential false positive, false negative, and / or blind spot information is configured to be turned on / off according to user preference so as not to interfere with the user's ability to examine the displayed information. To that end, the GUI 321 may include one or more toggle buttons 350 (or toggle switches) that the user may operate to execute one or more of the algorithms described herein to identify the locations of potential false positives, false negatives, and / or blind spots. Additionally, the computer 200 may prompt the user to select whether to display the locations of potential false positives, false negatives, and / or blind spots either before or after displaying the imaging results of a given procedure. The system may prompt the user to confirm whether they prefer to check the locations of suspicious data every image frame or every number of image frames of collected data. In at least one embodiment, the system 100 may be configured to detect all possible suspicious locations in a single frame or multiple frames to determine whether cases of false positives, false negatives, and / or blind spots exist. If at least one potential case exists, the system may display markers 325 and / or bounding boxes 335 around each of the detected potential false positives, false negatives, and / or blind spot locations.

[0059] Alternatively, in at least some embodiments, if the user suspects that the image data may contain locations of false positives, false negatives, and / or blind spots, the system may be configured to receive input from the user to mark any such locations along the entire pullback length of the luminal sample. After acquisition of the image data, if the user suspects a location of a potential false positive, false negative, or blind spot, the user may move the location of the line indicator 315 along the length of the luminal sample displayed in the second panel, and when the user clicks or drops the line indicator 315 at a particular location, the system may add a bounding box 335 to the corresponding fluorescence data to mark the suspect location.

[0060] <Algorithmic analysis of multi-modality OCT-fluorescence images> FIG. 4 illustrates an exemplary algorithm (process 400) of the overall workflow for image processing of OCT-fluorescence data to detect the location of false positives, false negatives, and / or blind spots, according to an embodiment of the present disclosure. In an embodiment, the process 400 of the imaging system 100 is realized by the CPU 191 of the computer 200 executing executable instructions or programs stored in the ROM / RAM of the storage device 192. In other embodiments, the process 400 may be executed by any suitable computing device other than the CPU 191 (e.g., remote cloud computing). As used herein, the term process or "algorithm" refers to a finite sequence of steps for solving a logical or mathematical problem or for performing an organizational process. The algorithm may be described in any understandable terms (e.g., as a formula, prose, or flow chart) or in any other manner that provides a sufficient structure for implementing various aspects of the block diagrams and flow charts of the present disclosure, together with the hardware structures described above. Some of the detailed description below is presented in terms of algorithms and symbolic representations of the operation of a processor (e.g., a CPU or MPU) on a data set stored in a memory. These algorithmic descriptions and representations can be used by those skilled in the art in implementing the various embodiments disclosed herein. In this disclosure, each flowchart describes a sequence of steps that produces a well-defined and unique result. The steps may include a combination of physical manipulations of hardware structures and software manipulations and / or mathematical operations of image data sets. Usually, but not necessarily, the physical quantities take the form of electrical, optical, or magnetic signals that can be detected, stored, transferred, combined, compared, or otherwise manipulated to achieve a tangible, useful result. Throughout the description of this disclosure, terms such as processing, computing, calculation, determination, etc. refer to the actions and processes of a computer system, logic circuitry, processor, or other similar electronic device that manipulates and transforms information as a data set and outputs a result that represents a physical (electronic) output (image, text, sound, motion, etc.).

[0061] Referring again to FIG. 4, a workflow process 400 for detecting false positives, false negatives, and / or blind spots includes a series of steps S402-S417. For purposes of brevity, the process 400 is illustrated and described as a series of steps, but it should be understood that such a process is not limited by the order in which the steps are presented. Some steps may occur in different orders and / or simultaneously with one another, which may differ from those illustrated and described. In some embodiments, fewer than all of the illustrated blocks or steps may be required to perform a given process. In other embodiments, steps or blocks of a process may be combined or divided into multiple sub-steps or subcomponents. Additionally, additional unillustrated blocks may be employed in alternative but equivalent steps or processes, as inferred from the entire disclosure. In the process 400, the workflow begins at step S401, where the computer 200 controls the OCT and fluorescence modalities of FIG. 1 to initially initiate a pullback operation. As known to those skilled in the art, before pullback, the system first performs a clearance process to "wash away" blood around the catheter tip. Next, in step S402, the system uses a trigger signal to simultaneously acquire OCT and fluorescence image data. OCT-fluorescence image data can be acquired simultaneously (i.e., at approximately the same time) and co-registered (i.e., from the same location); however, each type of data can be collected separately by the two separate imaging modalities shown in FIG. 1. The fluorescence image data can include NIRAF or NIRF data sets (fluorescence intensity), and the OCT data can include 2D cross-sectional images (pixel data) consisting of multiple A-scans, or 3D image volumes (voxel data) formed by multiple 2D images (B-scans).

[0062] In step S404, computer 200 processes the acquired image data to detect lumen edges from the OCT image data. Lumen edge detection can be performed by analyzing each A-line of OCT data, analyzing each frame of OCT data, and / or slicing and analyzing a 3D volume of OCT data. Various examples of lumen edge detection are described in commonly-disclosed U.S. Pregrant Patent Application Publication No. 2019 / 0374109, the disclosure of which is incorporated herein by reference in its entirety.

[0063] In step S406, the computer 200 performs a NIRAF distance calibration. In the NIRAF distance calibration, a section of OCT-fluorescence data is selected from the OCT-fluorescence data set acquired in step S402. The data section may be selected from a single frame of helical scan data or from multiple data frames. Preferably, before selecting the data section, outlier data points in the fluorescence data may be removed, for example, by applying a smoothing filter to the OCT-fluorescence data set acquired in step S402. From the selected data section, a maximum fluorescence value is calculated and compared to a fluorescence detection threshold. In some embodiments, a system limit of detection (LOD) value may be used as the fluorescence detection threshold. The LOD value may be based on system parameters, or may be experimentally determined by a user, or may be determined using system calibration data (e.g., based on a measurement of a background signal). Based on the comparison result between the maximum fluorescence value and the LOD value, various algorithms may be applied to the selected data section to perform the NIRAF distance calibration. See, e.g., Applicant's previously filed U.S. Provisional Patent Application No. 62 / 925655 (now published as U.S. Patent Application No. 17 / 077894, U.S. Pregrant Patent Publication No. 2021 / 0121132). See also Applicant's U.S. Pregrant Patent Application Publication No. 2019 / 0298174, both of which are incorporated by reference herein for all purposes. Further examples of distance calibration are described in Ughi et al., "Dual modality intravascular optical coherence tomography (OCT) and near-infrared fluorescence (NIRF) imaging: a fully automated algorithm for the distance-calibration of NIRF signal intensity for quantitative molecular imaging," the International Journal of Cardiovascular Imaging volume 31, pp. 259-268 (2015).

[0064] In step S408, the computer 200 analyzes the OCT data and detects characteristic features of the luminal sample and / or catheter from the OCT image. The OCT characteristic features include plaque type, external elastic lamina (EEL), geometric features (e.g., presence of side vessels), flushing adequacy (e.g., determining whether residual blood remains in the lumen), sheath diameter (e.g., determining whether there is a deformation in the catheter sheath diameter), etc. The OCT data analysis can be performed frame by frame, for each A-line, or for every 10 A-lines (or other predetermined number), or for the entire tomographic image. In addition, a frame-to-frame analysis can be performed to improve the accuracy of the analysis. The frame-to-frame analysis can include comparing nearby frames (e.g., 10 frames (or a predetermined number of frames)) before and after the target frame. Further improvement of the detection of characteristic features can be achieved by comparing the OCT data with corresponding histological and / or confocal images, as described below with reference to Figures 7A, 8A, and 8B.

[0065] In step S410, computer 200 detects potential false positive cases or potential false negative cases based on the characteristic features detected in step S408, and labels and stores information associated with each potential case. Similarly, in step S412, computer 200 detects potential blind spot cases, and labels and stores each potential blind spot case. Exemplary processes for detecting potential false positive, false negative and / or blind spot cases are described below with reference to Figures 7, 8A and 8B.

[0066] In one embodiment, a blind spot is defined as a spot where the MMOCT catheter cannot detect a fluorescent signal for biological, pathological and / or technical reasons. For example, the optical properties of lesions in the luminal wall may vary, so that certain plaque lesion types may not send a fluorescent signal to the catheter compared to other plaque types. In this case, no fluorescence will be detected through the imaging catheter. However, in places where no fluorescent signal is detected, the OCT signal may be positive for identifying changes in tissue structure. Therefore, the system is configured to determine from the OCT image data whether the image contains at least one region with a potential false positive, false negative and / or blind spot location based on characteristic properties such as plaque type. If a region larger than a predetermined size (e.g., more than a predetermined number of pixels or voxels) is determined as a potential false positive, false negative and / or blind spot location, the system is configured to record the location of the luminal sample 170 and later display the location to the user so that the presence or absence of fluorescence can be confirmed.

[0067] Thus, in step S414, computer 200 stores and / or outputs the detection results to display device 300 and displays the results in a graphical user interface (GUI) (e.g., GUI 321). Specifically, as shown in Fig. 3, computer 200 may be programmed to display a bounding box 335 in the longitudinal image indicating the longitudinal location of the potential undefined data, and may also display a marker 325 shown as a dotted partial ring around the edge of the tomographic image indicating the radial location of the potential undefined data (e.g., a vessel layer).

[0068] In step S416, computer 200 controls display device 300 to prompt the user to review and determine whether the displayed location is a true false positive or true false negative signal. In this step 416, computer 200 may be configured, among other things, to prompt the user to review the blind spot location and visually determine whether the location is a true positive, true negative, or otherwise (e.g., an image artifact).

[0069] In step S417, computer 200 is configured to again prompt the user (or to automatically determine whether to repeat) to decide whether to repeat the process of steps S404 to S416 until all frames, or a predetermined number of frames, or a complete tomographic image have been processed.

[0070] FIG. 5 illustrates a workflow process 500 for detecting the location of false positives, false negatives, and / or blind spots according to another embodiment of the present disclosure. The process 500 of the imaging system 100 is realized by the CPU 191 of the computer 200 executing executable instructions or programs stored in the ROM / RAM of the storage device 192. Steps S502, S504, and S506 are similar to steps S402-S406, respectively. In step S508, the computer 200 detects whether the luminal edge in the OCT image is located within a certain distance from the catheter sheath. Here, the distance should be a short distance, such as 10 microns, which depends on the OCT resolution. If the result of step S508 is YES (i.e., the luminal edge is within a predetermined distance from the catheter sheath), the system checks the fluorescence level (fluorescence intensity value) in step S510. If the fluorescence signal is positive, for example, if the fluorescence signal is above the system limit of detection (LOD) and / or above a predefined threshold set by the user or the system, the system determines that the location is a potential false positive. In other words, in step S510, the reason for the positive fluorescence detection is most likely due to the catheter touching the lumen edge. In this case, the system saves the location as a potential false positive case in step S512.

[0071] On the other hand, if the result of step S508 or step S510 is NO, the system proceeds to the remaining steps to determine whether there is a potential blind spot case (steps S514-S518). In detecting a blind spot case, the system first checks whether there is a reason for the fluorescent signal not being detected. The reason for the potential blind spot may be one or more of biological, pathological and / or technical reasons. The biological reasons may be, for example, that the molecular fluorescent source is located farther away than the reach of the excitation light, or that the amount of the molecular fluorescent source is low. That is, there is a biological reason for the fluorescent signal not being detected. The pathological reasons may be, for example, that there is a calcified plaque in some areas of the luminal sample, that there is a thrombus, or that there are other structures that attenuate the fluorescence or prevent it from reaching the catheter. The technical reasons may be, for example, that the system detection limit is high, that the threshold for considering the fluorescent signal as positive is high, that the power of the excitation light is low, or that the distance from the catheter to the luminal edge is too far.

[0072] Thus, in step S514, the system checks the OCT signal to determine whether calcified plaque is detected. In alternative embodiments, the system may be configured to detect any other factors. If the system identifies a possible reason for a blind spot in the OCT image data (e.g., calcified plaque is detected), the system checks the fluorescence signal level in step S516. If the fluorescence signal is negative (i.e., no fluorescence signal is detected), for example, by the fluorescence signal being below the system detection limit and / or below a predefined threshold set by the user or the system, the system proceeds to step S518 and marks the location as a potential blind spot in step S518. The process of analysis in steps S508-S518 may be performed continuously in real time throughout the pullback operation, and may be performed every A-line scan, every predefined number of A-lines (e.g., every 10 A-line scans), or for the entire tomographic image. In step S520, the computer 200 outputs the detection result to the display device 300 and displays the result in a graphical user interface 321 as shown in FIG. 3. Again, computer 200 can actively prompt the user to confirm each marked location based on the user's own experience and knowledge of the presented data, similar to step 416. That is, the user can confirm whether a potential blind spot case should be a true false positive or a true false negative case.

[0073] FIG. 6 illustrates a workflow process 600 for detecting the location of false positives, false negatives, and / or blind spots according to yet another embodiment of the present disclosure. The process 600 of the imaging system 100 is realized by the CPU 191 of the computer 200 executing executable instructions or programs stored in the ROM / RAM of the storage device 192. The workflow process 600 mainly focuses on detecting "false positives". Steps S602, S604, and S606 are similar to steps S402-S406 of FIG. 4, respectively. Step S608 is similar to step S508. That is, the computer 200 determines whether the lumen edge is within a predetermined distance from the catheter sheath. The predetermined distance should be small, for example, approximately 10 microns, and depends mainly on the OCT maximum resolution of the system. If the catheter is within the predetermined distance from the lumen edge (YES in S608), the process proceeds to step S610. In step S610, computer 200 determines whether the signal of "calibration NIRAF" is positive. Here, the NIRAF signal is "positive" if the actual NIRAF signal is collected by the catheter and detected by the system. If the NIRAF signal is detected (YES in S610), the process proceeds to step S612. If the NIRAF signal is not detected (NO in S610), the process proceeds directly to a stop or end step. That is, the process of detecting potential false positive cases ends as soon as computer 200 determines that the lumen edge is not within a predetermined distance from the catheter sheath. Again, system computer 200 may consider the lumen edge distance in combination with other parameters, such as a signal intensity threshold for a particular region of interest.

[0074] In step S612, to avoid outlier cases, computer 200 can also check the size (angle) of the NIRAF arc in the tomogram. That is, the system determines whether the NIRAF signal in the tomogram is greater than a minimum (MIN) angle. If the arc of the NIRAF signal is greater than a certain degree (e.g., greater than 10 degrees) (YES in S612), the NIRAF signal is confirmed and does not need to be further processed, so the process proceeds to stop or end the process. If the NIRAF signal is positive (YES in S610), after the outliers are removed (NO in S612), computer 200 checks the lumen diameter in step S614. If the lumen diameter is small (smaller than a minimum threshold), the current OCT image is likely to be acquired from an area with a lesion in the lumen sample. If there is a lesion on the wall of the lumen sample, fluorescence is likely to be observed. Thus, if the result of step S614 is YES (ie, the lumen diameter is smaller than expected), computer 200 verifies that the signal is adequate (validates the data) and ends the process.

[0075] In step S616, if the lumen diameter is not less than the minimum, the system computer 200 checks whether nearby frames show NIRAF signals (frame-to-frame analysis). If many of the nearby frames show NIRAF signals, then it is likely that the current frame shows a true fluorescent signal. Thus, in step S616 (YES), the flow proceeds to stop or end the process. If in step S616 it is determined that there are no positive NIRAF signals in the nearby frames, then it is likely that the current frame (or A-scan) is a potential false positive (i.e., not a true fluorescent signal). The order of steps S612-S616 may be changed, swapped, or combined, and one or more steps may be selected or skipped. In step S618, the computer 200 marks the location of the current frame as a potential "false positive" case. In step S620, similar to step S520, the system saves and displays the potential false positive location, or prompts the user to confirm the display of the location. For example, the system may be configured (programmed) to inform the user (display a message) that there are one or more locations with potential "false positive" information in the processed data. The user can then flip switch 350 to manually observe and confirm the status of each false positive location.

[0076] Continuing to refer to FIG. 6, in step S608, the distance between the lumen edge and the catheter sheath (imaging probe) should be small, such as approximately 10 μm. This distance generally depends on OCT resolution, beam spot diameter, light source power, etc., so the distance between the lumen edge and the catheter may be manually set by the user or may be pre-stored in the system. In step S612, the threshold arc angle may be a predetermined arc of approximately 5 degrees, 10 degrees, 20 degrees, etc., or may be different for each type of lumen being imaged. The purpose of determining the threshold arc angle of the tomogram is to avoid small signal spikes due to catheter adhesion to the lumen or other external causes of artifacts. In step S614, the lumen diameter size may be larger than the minimum threshold diameter, for example, in some cases a minimum lumen diameter of 2 mm may be expected. Alternatively, it may be changed by the user depending on the lumen to be inspected and the size / type of catheter used. Once the reference OCT frame is determined, the minimum threshold diameter can be determined based on the lumen diameter in the reference frame. For example, 80% of the lumen diameter of the reference frame can be used as the minimum threshold diameter. In step S616, the number and / or distance of the "nearby frames" can be predetermined by the system or set by the user. For example, the number of frames within a distance of about 1 mm of the selected or target frame can be set by the user.

[0077] To determine the location of true autofluorescence and / or potential blind spots in high-risk coronary plaques, this disclosure proposes to confirm the source of near-infrared autofluorescence (NIRAF) by histopathological correlation analysis. This analysis is based on a previous study that showed a positive correlation between morphological features associated with high-risk plaques and high NIRAF areas obtained via catheter-based multimodality OCT-NIRAF imaging. Subsequent ex vivo studies suggested that NIRAF is a marker of intraplaque hemorrhage (IPH), one of the indicators of high-risk plaques. This study aims to clarify the source of high NIRAF and / or its blind spots using tissue sections. Serially sectioned tissue slides made from formalin-fixed paraffin-embedded samples from human cadaveric coronary arteries were stained with glycophorin A (IPH marker) and Sudan Black (ceroid marker) and compared with confocal NIRAF images obtained from adjacent unstained slides with 635 nm excitation light and 655–755 nm fluorescence emission. Each image was divided into eight 45° sections centered around the luminal centroid. Each section was visually analyzed for NIRAF or staining positivity, defined as >10% of the intima within the section containing NIRAF, glycophorin A, or Sudan Black staining.

[0078] FIG. 7 illustrates example results of an exemplary histopathology correlation case study performed by the present inventors in accordance with the principles disclosed herein. The results show instances where true fluorescence was detected as well as instances where potential false positives, false negatives, and / or blind spots of the fluorescence source were detected. The study compares multiple multi-modality catheter-based images 906 of a luminal sample with corresponding hematoxylin and eosin (H&E) images 902, Masson's Trichrome images 904, confocal NIRAF images 908, and confocal luminal images 910. The results 901 are tabulated in rows A-D in the left-most column of FIG. 7. The H&E images 902 include a first image 902A, a second image 902B, a third image 902C, and a fourth image 902D. The Trichrome images 904 include a first image 904A, a second image 904B, a third image 904C, and a fourth image 904D. The OCT-NIRAF images (catheter images) 906 include a first image 906A, a second image 906B, a third image 906C, and a fourth image 906D. The confocal NIRAF images 908 include a first image 908A, a second image 908B, a third image 908C, and a fourth image 908D. The confocal luminal images 910 include a first image 910A, a second image 910B, a third image 910C, and a fourth image 910D.

[0079] The H&E image 902 and Masson Trichrome image 904 are both histological images, revealing lesions classified as ruptured plaque, fibroatheroma, fibrocalcified plaque, pathological intimal thickening, fibrous plaque, and intimal hyperplasia. The confocal NIRAF image 908 and confocal luminal image 910 are images acquired from luminal sections close (e.g., within a few microns) to the H&E and Trichrome sections by confocal fluorescence microscopy to evaluate the microlocalization of NIRAF in the coronary artery wall of the luminal sample. The confocal NIRAF and confocal luminal images were acquired using two sets of excitation and emission wavelengths. 405 nm excitation and 425–475 nm emission were used to assess the overall tissue shape and orientation (confocal luminal image 910), while 635 nm excitation and 655–755 nm emission were used to identify the microscopic NIRAF distribution near the luminal edge (confocal NIRAF image 908).

[0080] To determine the localization of fluorescence in confocal NIRAF and catheter-based NIRAF, we performed registration of the confocal NIRAF images 908-910 with the OCT-NIRAF image 906. The registration involves two steps: (1) matching the confocal NIRAF image with the OCT-NIRAF image, and (2) transforming the confocal NIRAF image to align it with the OCT-NIRAF image. Figure 8A shows an example process for registering the OCT-NIRAF image 906 with the confocal luminal image 910 and the confocal NIRAF image 908. In matching (step 1), H&E and Masson's Trichrome images (tissue structure images) were used to identify characteristic properties of the lumen. First, we selected tissue structure images containing fibrocalcified plaques and / or side branches, and found the corresponding OCT-NIRAF images based on their morphological properties. Other morphological characteristics include plaque type characteristics, such as one or more of luminal disruption, fibroatheroma, fibrocalcified plaque, pathological intimal thickening, fibrous plaque, and intimal hyperplasia. For the remaining histological images (those without identifiable morphological characteristics (e.g., fibrocalcified plaque and / or side branches)), the corresponding OCT-NIRAF images were found based on matching pairs based on luminal diameter and pullback information (e.g., scanning speed and direction). After matching, in transformation (step 2), the confocal NIRAF image was transformed to align with the OCT-NIRAF image by matching luminal edges.

[0081] In FIG. 8A, the confocal NIRAF image 908 is transformed by an affine transformation 808 to be aligned with the OCT-NIRAF image by matching the luminal edges. Since the confocal luminal image and the confocal NIRAF image were acquired from approximately the same portion of the lumen, the luminal edge detected in the confocal luminal image was used as the luminal edge from the confocal NIRAF image, as described above. In this process, the luminal edge was first identified in both the confocal and OCT-NIRAF images, and feature points of the luminal edge were identified from both images based on plaque type and / or side branches. Then, an affine transformation 808 was applied to the confocal NIRAF image 908 to match the detected luminal edges and to match the same feature points identified in the OCT-NIRAF image 906 and the confocal luminal image 910. The resulting registered image 810 is shown at the bottom of FIG. 8A.

[0082] To determine the fluorescence positivity of the confocal NIRAF and catheter-based NIRAF images, the registered confocal NIRAF and OCT-NIRAF images (registered image 810) are divided into multiple segments of equal dimensions. Figure 8B shows an example of dividing the registered OCT-NIRAF and confocal NIRAF image 810 into multiple segments (S). In this example, the registered image 810 was divided into eight segments (S1, S2, S3, S4, S5, S6, S7, S8) of 45 degrees each, centered on the catheter position in the OCT-NIRAF image.

[0083] For each 45-degree segment, confocal NIRAF and catheter-based NIRAF were assessed for positivity as follows: For confocal NIRAF images, the intensity of the image Z-stack was averaged, and then the images were binarized using a threshold of the top 50% of 8-bit grayscale levels. This binarization threshold was deemed acceptable by a qualified pathologist comparing the original and binarized images. After binarization, the area of ​​confocal NIRAF positive signal was assessed for each 45-degree segment at 0.1 mm depth increments from the luminal edge. Each segment was considered confocal NIRAF positive if the size of the binarized area was greater than 5% of the vessel wall area. The vessel wall area was calculated from the average of the actual luminal diameter within the 45-degree segment, assuming an intima and media thickness of approximately 1.0 mm. Catheter-based NIRAF was considered positive within a 45-degree segment if more than 60% of the A-lines within the 45-degree segment had a calibrated NIRAF intensity higher than the system limit of detection (LOD). Of course, the above parameters (number and size of segments, size of binarized region, thickness of intima and media, etc.) in confocal NIRAF images and / or parameters of catheter-based NIRAF may be modified according to application and / or user preference. For example, in some embodiments where the thickness of the intima and media is about 0.5 mm, a 45-degree segment may be considered positive if at least 5% of the 45-degree arc is above the system limit of detection (LOD). That is, the signal positivity threshold may be modified and the data may be adjusted without departing from the general concept of data analysis to determine potential false positive, false negative and / or blind spot cases.

[0084] After evaluating the positivity of confocal NIRAF and catheter-based NIRAF for each 45-degree segment, the correlation between confocal NIRAF and catheter-based NIRAF was evaluated in terms of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) using confocal NIRAF as ground truth. Here, sensitivity is the ratio of true positives (TP) divided by the sum of true positives + false negatives, i.e., sensitivity = TP / (TP + FN). Specificity is the ratio of true negatives (TN) divided by the sum of true negatives + false positives, i.e., specificity = TN / (TN + FP). PPV = TP / (TP + FP) and NPV = TN / (TN + FN).

[0085] The results of the comparison of histology (H&E, Masson's Trichrome) images with OCT-NIRAF images and confocal microscopy images are shown in column 901 of FIG. 7. Row A: Catheter-based NIRAF, indicated by marker 320, was observed in the first OCT-NIRAF image 906A in the same area indicated by marker 321 (where confocal NIRAF was observed in the first confocal NIRAF image 908A). That is, the fibrous atheroma observed in the confocal NIRAF image 908A confirmed the NIRAF data in the OCT-NIRAF image 906A. Therefore, this is a true positive case. Row B: Catheter-based NIRAF was not observed in the second OCT-NIRAF image 906B, as confocal NIRAF was not confirmed in the second confocal NIRAF image 908B. Therefore, this is a true negative case. Row C: No confocal NIRAF was observed in the third confocal NIRAF image 908C, but catheter-based NIRAF was positive in the OCT-NIRAF image 906C, as indicated by marker 320. This catheter-based NIRAF is likely due to the catheter 160 attaching (contacting) to the luminal wall. Therefore, this case is marked as a potential false-positive case. Row D: No catheter-based NIRAF was observed in the fourth OCT-NIRAF image 906D, but confocal NIRAF was observed in the confocal images 908D, 910D, as indicated by marker 321. Furthermore, the histological images 902D, 904D also show fibrocalcified plaques, as indicated by marker 319. Therefore, this case is determined as a potential false-negative case.

[0086] More specifically, the leftmost column of FIG. 7 shows detection results 901 from an automated data analysis performed, for example, by computer 200. In row A, the image analysis results in a "true positive" detection of fluorescence. Specifically, row A represents a case where the presence of true NIRAF data was determined by analysis of the OCT data of the OCT-NIRAF image (catheter NIRAF) by comparing the OCT-NIRAF data with the corresponding confocal microscope image (confocal NIRAF). As explained above, the histological image can be used to confirm morphological characteristic features detected by the OCT data (or suspected based on the OCT data). Row B represents a case where the presence of NIRAF data is not detected by analysis of the OCT data of the OCT-NIRAF image. This is confirmed by analysis of the histological and confocal images when no NIRAF is observed in the corresponding confocal NIRAF image. Row C represents a case where the analysis of the OCT data of the OCT-NIRAF image appears to indicate the presence of NIRAF data. However, in the corresponding confocal NIRAF image, NIRAF is not clearly observed. Furthermore, the histological images (H&E and Trichrome) do not show morphological characteristics indicative of plaque or similar structures. Therefore, this result is likely due to the catheter 160 being in near contact with the luminal surface (e.g., the distal tip of the catheter touching the sample surface). Therefore, the OCT-NIRAF image 906C needs to be confirmed by the user. The system can therefore mark the image frame as a potential false positive and then present this information to the user for confirmation. Meanwhile, in row D, NIRAF is observed in the confocal NIRAF image 908D and the confocal luminal image 910D, but analysis of the OCT data in the OCT-NIRAF image 906D does not show any NIRAF signal. In row D, the histological H&E and Trichrome images also seem to indicate a reduced luminal diameter due to calcification of the vessel wall. Nevertheless, the OCT-NIRAF data does not show any fluorescence data.The reason for no fluorescence being detected in the OCT-NIRAF image 906D may be that areas of the blood vessel with calcified plaque attenuate or prevent the fluorescence signal from traveling from the blood vessel to the catheter. Thus, row D may represent a potential blind spot or false negative case where the MMOCT catheter is unable to detect the fluorescence signal for biological, pathological and / or technical reasons. In this case, the OCT-NIRAF image 906D still needs to be reviewed by the user. However, to ease the user's decision-making process, the system automatically registers and / or displays the location of such potential false positive, false negative and / or blind spot cases.

[0087] Referring to FIG. 7, the accuracy of catheter-based NIRAF detection can be more precisely established by applying one or more of the image processing algorithms described in this disclosure with respect to FIG. 4, FIG. 5, FIG. 6, and FIG. 8A-FIG. 8B. Catheter-based NIRAF is present when a certain amount of confocal NIRAF is observed in an area relatively close to the luminal edge. However, it is difficult to confirm whether catheter-based NIRAF is present by simply observing OCT-NIRAF images or automatically classifying each OCT-NIRAF image according to morphological characteristics (from the tissue structure image). Therefore, it is advantageous to configure a multi-modality imaging system with a graphical user interface so that a user can manually check the actual status of potential false positives, false negatives, and / or blind spot locations in a given image.

[0088] <Easily view the location of potential false positives, false negatives and / or blind spots> 9A-9F illustrate an exemplary embodiment of a method for displaying the location of potential false positives, false negatives, and / or blind spots in a multi-modality OCT-fluorescence image. According to FIGS. 9A, 9B, 9C, 9D, 9E, and 9F, the identification of the location of potential false positives, false negatives, and / or blind spots in the fluorescence data can be displayed on one or more of the tomographic and longitudinal images of the multi-modality image. As previously mentioned, the tomographic image can provide a rotational or circumferential position indicating the location where the fluorescence signal is positive or indicating the location where the potential false positives, false negatives, and / or blind spots exist. The locations of the false positives, false negatives, and / or blind spots can be displayed in various ways. For example, if a potential false positive location exists in the tomographic image, a line indicator, color-coded or with a particular pattern, can be displayed on or near the detected or suspected NIRAF location. If the system is programmed to identify specific parameters (e.g., biological reasons, pathological reasons, technical reasons, etc.) for potential blind spot, false negative, or false positive results, the locations of such results can be displayed according to the specific cause, reason, or parameter. For example, each type of location (false positive, false negative, or blind spot) can be displayed using different colored lines, different patterns, different values ​​of grayscale levels, etc., representing each type of case. Similarly, the reasons (e.g., biological, pathological, or technical reasons) for designating a location as a false positive, false negative, or blind spot can be indicated using a particular color or grayscale value or pattern, respectively.

[0089] FIG. 9A illustrates a first case (Case 1), where the system can confirm positive detection of a fluorescent signal and display the result in the cross-sectional and / or longitudinal images of the multi-modality image. FIG. 9B illustrates a second case (Case 2), where the system can confirm positive detection (or absence) of a fluorescent signal and display the result in the cross-sectional and / or longitudinal images of the multi-modality image. Methods or processes for confirming positive detection (or lack of detection) of a fluorescent signal are known, for example, as described in the above-referenced publication US2014 / 01276107. As described in this publication, the presence or absence of a fluorescent signal can be based on signal intensity levels corresponding to a type of tissue, a concentration of a particular substance in the tissue, a level or stage of malignancy prior to cancer development, etc. In the present disclosure, the MMOCT system is configured to determine the positive (or lack thereof) fluorescence according to the process described above with reference to FIG. 7 and FIG. 8A-8B. Furthermore, the system is configured to inform the user of the location of the fluorescent signal with respect to the OCT data of the multi-modality image of the luminal sample. To minimize user confusion, the system displays simplified versions (eg, reduced-size images) of the multi-modality images along with a representation (marker) indicating the location of the fluorescence data.

[0090] In FIG. 9A, the system displays the multi-modality image 906 as a tomographic image, with the OCT data 310 partially surrounded by a marker 720 (first marker) around the edge of the tomographic image. The marker 720 informs the user that a positive fluorescent signal was detected in the area between 12 o'clock and 3 o'clock on the wall of the lumen sample. The marker 720 is also shown in the longitudinal image, overlaid or aligned with the longitudinal image of the OCT data 310. In the longitudinal image, a line indicator 315 indicates the location along the length of the lumen where the tomographic image was taken. The tomographic and longitudinal images of the OCT data are displayed with the first marker 720 using the same color, same shading, and same pattern in both images.

[0091] In Fig. 9B, the system displays the cross-sectional images of the OCT data without any kind of marker for the fluorescence data, i.e., in the second case, when the system determines that there is no fluorescence signal at the position of the line indicator 315, both the longitudinal and cross-sectional images are displayed without the marker 720, since this case indicates a true negative result (i.e., the multi-modality image 906 does not contain a fluorescence signal).

[0092] FIG. 9C illustrates a third case (case 3) in which the system determines that the multi-modality image contains a fluorescent signal, but that the fluorescent signal may be a potential false positive result. In this case, the system determines that the fluorescent signal is most likely a “false positive” result based on an analysis of the OCT data. For example, the longitudinal view of the OCT data 310 does not show any structural features at the location of the line indicator 315 that may generate a fluorescent signal. Nevertheless, a positive fluorescent signal is detected. The reason for the presence of fluorescence may be that the distance from the catheter 160 to the lumen edge is within a predetermined distance. This potential false positive case is similar to the case of the previous image 906D shown in FIG. 7. In this case, the marker 720 is displayed to indicate the presence of fluorescence. Additionally, to inform the user that the fluorescent signal is a potential false positive, the system displays an additional marker (second marker) 722. The second marker 722 is added in a distinct color, shade, or pattern that is different from the first marker 720. To be unique and distinctive, the second marker 722 on at least one of the tomographic and longitudinal images may be configured to intermittently blink, intermittently increase or decrease in size, or change color to attract the user's attention.

[0093] FIG. 9D illustrates a fourth case (Case 4), where the multi-modality image 906 shows a cross-sectional view of the OCT data with no fluorescence signal. However, system analysis of the OCT data 310 reveals characteristic structural features that may be a source of fluorescence. Such characteristic structural features (indicated by indicator 324) include intimal thickening, abnormal adventitia, and damaged per-adventitial tissue. Thus, in this case, the system displays a second marker 722 on both the cross-sectional and longitudinal views of the OCT data. In this case, the second marker 722 indicates a potential false negative result (i.e., no fluorescence signal is shown, but data analysis of the OCT image reveals the presence of a fluorescent source). In Case 4, the system displays the second marker 722 in a clear and prominent color and pattern specific to a “false negative” result. This prompts the user to analyze and confirm whether the location indicated by the line indicator 315 contains fluorescence. The color, pattern, or line used to display the second marker corresponding to the false negative value is preferably displayed in a different color, pattern, or grayscale value than the first marker 720.

[0094] FIG. 9E illustrates a further case (Case 5), where the multi-modality image 906 shows a cross-section of the OCT data without any fluorescence signal. However, the system's analysis of the OCT data reveals that the characteristic features of the luminal sample include one or more artifacts 326 that may be potential sources of fluorescence. In this case, the artifacts 326 may include, for example, accumulated plaque (atherosclerotic plaque or atheroma), fibrous plaque, fibrocalcified plaque, and necrotic core of the luminal sample. Case 5 is similar to Case 4 in that an area of ​​the luminal sample exhibits a potential source of fluorescence, but no fluorescence signal is detected. However, in Case 5, the area with the artifact 326 may contain calcified plaque or other unknown components. Thus, the system marks the area with the artifact 326 with a specific color or pattern that is different from the first marker 720 or the second marker 722. Additionally, the system displays the multi-modality image with a second marker 722 as a circular arc surrounding the edge of the tomographic image corresponding to the circumferential location of the artifact 326. The system also displays the second marker 722 overlaid (or side-by-side) with the OCT data 310 in the longitudinal image to correspond to the location of the line indicator 315. In case 5, if no fluorescent signal is detected by the system but the artifact 326 is observed in the OCT image, the system classifies the artifact 326 as a potential blind spot case (e.g., a fluorescent source may be present but the fluorescent excitation light does not reach the plaque area or the fluorescent signal does not reach the catheter). Thus, the system displays the second marker 722 and the artifact 326 in the tomographic image and along the length of the longitudinal image to show the correspondence between the potential fluorescent signal and the OCT data. Even in this case, the second marker 722 may be prominently displayed to prompt the user to check whether the artifact 326 is a false positive or a false negative or a blind spot (undefined) location.

[0095] FIG. 9F illustrates an example where the system is configured to display multiple cases of potential false positives, false negatives, and / or blind spot locations. In FIG. 9F, a first marker 720 indicates detection of a fluorescent signal, and a second marker 722 indicates that the fluorescent signal may be a potential false positive, as shown in FIG. 9C. However, when the user moves the line indicator horizontally along the length of the longitudinal image to the position shown in FIG. 9F, the tomographic image changes to show a different cross-section of the luminal sample. Thus, in FIG. 9F, the system displays the second marker 722 overlaid on the OCT image data 310, and adds a third marker 730 to indicate the new location of the fluorescent data indicated by the first marker 720A, or the new location of the potential false positive, false negative, and / or blind spot indicated by the third marker 730. That is, the difference between Figures 9C and 9F is that a second marker 722 is overlaid on the OCT image of the luminal sample and a third marker 730 is added to indicate the location of additional false positives, false negatives and / or blind spots.

[0096] 10A, 10B, 10C and 10D illustrate various examples of further embodiments of a graphical user interface displaying the location of potential false positives, false negatives and / or blind spots of fluorescence in a multi-modality OCT-fluorescence image. According to this embodiment, the fluorescence image data (NIRAF profile image) can be displayed as a simple binary display or histogram. A loose definition of a histogram is a graphical representation of data. Here, the term "histogram" is used to refer to a pictorial representation of fluorescence intensity data. A conventional histogram can show the frequency of each intensity value from 0 to 255 in a grayscale equivalent of a photograph. In a histogram, each intensity value is plotted against the number of pixels with that value. Thus, a histogram can provide a simple view of the distribution of various intensity values. In this case, a basic binary display (e.g., two colors, two grayscale values, two patterns) can be used to simplify the display of the presence or absence of fluorescence data (NIRAF data). Furthermore, a few more colors, greyscale values ​​or patterns may be sufficient to indicate the presence or absence of fluorescent data as well as potential false positives, potential false negatives, and / or blind spot cases.

[0097] A binary histogram can be displayed along the length of a vessel length image (longitudinal view) by discarding the angular information of the tomographic image and simplifying the display of fluorescence intensity to high (H) and low (L) values. To that end, each multimodality image can be defined by a small number of values ​​corresponding to signal intensity. For example, NIRAF intensity can be defined around a median value (e.g., ±25% of the median) between a high and a low threshold. In one embodiment, the histogram can have low and high values ​​to indicate the presence (value 1) or absence (value 0) of fluorescence along the length of the lumen. In other embodiments, the histogram can have more than two values ​​such that each value can represent an intensity level at each location along the length of the lumen.

[0098] To facilitate a quick evaluation of the signal by the user, the fluorescence signal (each high or low value of the histogram) may be color-coded or shaded to indicate its correspondence with various possibilities of the presence or absence of fluorescence. Again, the analysis of the image data may be performed for each acquired OCT-NIRAF image, frame by frame, or A-line by A-line, or every 10 A-lines (or other predefined number), or, for tomographic images, by dividing the image into a predefined number of sections. Furthermore, to improve the accuracy of the analysis, an inter-frame analysis (comparison of nearby frames (e.g. 10 frames before and after the target frame)) may be performed.

[0099] 10A and 10B illustrate an example of how the third panel 321-3 of the display 320 can be configured to display the fluorescence signal in a binary format instead of (or in addition to) the carpet image shown in FIG. 3. Similar to the concept of an OCT lumen profile (or lumenogram), the fluorescence signal can be displayed as a NIRAF profile (or Nirafogram) to simplify the image of the fluorescence signal in the longitudinal image for ease of user evaluation. In the longitudinal image, the carpet image is replaced by a histogram that discards the angle information to show only the presence of NIRAF in terms of the length of the pullback. The actions that the user can then take based on this simplified NIRAF profile image are (i) to go to each slice with NIRAF and look more closely at the OCT to see what happened at that location, and (ii) to decide where to perform the procedure (e.g., stent placement) by intentionally avoiding (or targeting) sample areas where NIRAF signals were identified.

[0100] Possible algorithms for displaying NIRAF include: When displaying NIRAF intensity: (a) display the sum of NIRAF intensities present in the slice; (b) display the peak NIRAF intensity present in the slice; (c) use (a) or (b), but display the intensity in units (e.g. high, medium, low) defined by a threshold (e.g. low=2.5V, high=5V, medium=average of high to low); When displaying the presence of NIRAF (binary NIRAF): (d) if NIRAF is present anywhere in the slice and is above a threshold (e.g. 5V), display YES NIRAF for that slice; (e) if the sum of NIRAF intensities present in the slice is above a threshold (e.g. 5V), display YES NIRAF for that slice; (f) use (c) or (d), but display NO NIRAF or BLIND SPOT if below threshold.

[0101] Advantages and benefits of using a binary or histogram display include, but are not limited to, the saving of real estate on the display screen (NIRAF carpet images take up a lot of vertical space on the screen, whereas histogram or binary images use minimal space). For example, a carpet image requires a height of 2πr to provide angular information, whereas a binary longitudinal image can be reduced in height to a single line (a few pixels wide) using different colors, shading, or patterns since it simply indicates the presence or absence of NIRAF. The smaller the NIRAF display in the longitudinal vessel image, the more space there is for other images, and the less confusing the user is. This is particularly advantageous when several types of images and / or characteristic features are used (as shown in FIG. 7) to determine whether the fluorescent signal is false positive, false negative, and / or undefined (blind spot). In other embodiments, the binary or histogram display also uses fewer colors. That is, the NIRAF profile image can be displayed in one color to simply indicate the presence of NIRAF, reserving other colors to indicate other features and / or minimizing user confusion. Furthermore, by displaying NIRAF in one or a few colors with minimal usable screen real estate, the system can be configured to display additional applications (e.g., spectroscopic OCT) in future updates. Also of note, the simplicity of this modified display embodiment reduces the time required for user interpretation.

[0102] In FIG. 10A, the patterns can be used to identify and confirm the specific locations along the length of the pullback where true positive signals 820, true negative signals 822, false positive signals 824, and false negative (or blind spot) signals 826 were detected. FIG. 10A displays the presence of signals 820-826 along the length of the vessel, regardless of the reason for such classification. Note that not all cases need be displayed simultaneously. In some embodiments, the GUI may be controlled to selectively display only the presence of one or more of true positive signals 820, true negative signals 822, false positive signals 824, and false negative (or blind spot) signals 826. This can be displayed with or without an estimated location along the length of the vessel. If the location of a given signal along the length of the vessel is not displayed on the display, it will only be made available (displayed to the user) if the user selects (clicks) the type of information they wish to see. For example, after viewing the histogram of FIG. 10A, the user can select "false positive". After such selection, the system then displays longitudinal locations indicating where potential false positive signals were detected along the length of the vessel image. These patterns can be used to identify where the true positives, true negatives, false positives, and false negatives (or blind spots) signals are.

[0103] However, in another embodiment shown in FIG. 10B, the GUI may be controlled to display the presence of one or more false positive, false negative, and / or blind spot locations along with a specific reason for marking each location as such. FIG. 10B illustrates a display showing a positive signal 830 (confirmed fluorescent signal), a negative signal 832 (confirmed absence of fluorescent signal), a false positive signal 834 (a potential "false positive" case), and a false negative signal 836 (a potential "false negative" case). These signals (particularly the false positive and false negative cases) can be stored in memory and / or displayed on the display along with the reason for marking the location as such. In FIG. 1B, certain colors or patterns can be used to identify which are true positive, true negative, false positive, and false negative (or blind spot) cases. For example, if the entire OCT-NIRAF image shows positive fluorescence, but OCT data analysis reveals the presence of "calcified plaque" or "low laser power", the false negative signal 836 can be displayed in a specific pattern consistent with the reason for such a decision. In this case, each of the positive signal 830, the negative signal 832, the false positive signal 834, and the false negative signal 836 may be assigned a unique color or pattern so that the user can easily identify the presence of false positives, false negatives, and / or blind spot locations. The type of pattern or color may be selectable by the user or may be pre-assigned depending on the reason (biological, pathological, or technical) for designating a given location as either a false positive, false negative, or blind spot (undefined), etc. Figure 10B is an example that can be used when the specific reason for a false positive or false negative is known.

[0104] FIG 10C shows an example of a display screen configured to display a multi-level histogram to inform the user of the presence of a fluorescent signal (positive NIRAF) and potential false positive cases. In FIG 10C, the presence of a fluorescent signal is indicated by a first positive signal 830, a second positive signal 832, and a third positive signal 836, each with different intensities represented by the size of the marker. Additionally, FIG 10C shows a false positive signal 844. The horizontal direction indicates the position along the length of the lumen (vascular image) and the vertical direction indicates the intensity level.

[0105] 10D shows an example of a graphical user interface configured to display a binary histogram on a display panel using centered values. In this example, the GUI can be configured to display fluorescent signals 850 (positive NIRAF), true negative signals 852 (no NIRAF), potential false positive signals 854, and false negative signals 856. As noted with respect to other embodiments, it is not necessary for all signals to be displayed on the display panel at the same time. However, to make user interaction with the GUI easier and more efficient, it may be advantageous to display each type of signal in a particular color or pattern.

[0106] FIG. 11 illustrates a display device 300 with a modified graphical user interface (GUI) 321 to display the locations of potential false positives, false negatives and / or blind spots in a simplified binarized manner. In FIG. 11, the GUI 321 shown on the display device 300 is controlled by the CPU 291 of the computer 200 according to the algorithm for displaying multi-modality image information previously described. In this embodiment, the display screen 320 is divided into a number of separate display panels, including a first panel 321-1, a second panel 321-2 and a third panel 1121. The first panel 321-1 is configured to display an angiogram 340 and a cross-sectional image of the multi-modality image 906 showing the OCT data 310 and the fluorescence data 320. The second panel 321-2 is configured to display a longitudinal image (angiogram) of the luminal sample, which shows only the OCT data 310 for visually assessing the structural properties of the luminal sample. That is, the first panel 321-1 and the second panel 321-2 function in the same manner as the GUI 321 described above with reference to FIG.

[0107] The third panel 1121 is configured to display the presence or absence of fluorescence data as well as the location of potential false positives, false negatives, and / or blind spots in a simplified manner. In FIG. 11, the second panel 321-2 displays a line indicator 315 to indicate the longitudinal position of the luminal sample displayed in the tomographic image shown in the first panel 321-1. The line indicator 315 may be a movable component of the GUI 321 and may be interactively controlled by a user to move horizontally using a pointer 316, such as a mouse or joystick, or by touching the screen of the display 320. As the user slides the pointer 316 along the longitudinal image of the OCT data, the tomographic image changes continuously to display the OCT data and fluorescence data corresponding to the position of the line indicator 315. In this manner, the user may select a position along the length of the luminal sample for which structural properties should be evaluated.

[0108] To reduce the amount of data a user needs to view and analyze, the third display panel 1121 is configured to display the fluorescence information in a binary fashion. The fluorescence information is displayed as one or more of a true positive marker 1321, a false positive marker 1322, a true negative marker 1323, a blind spot marker 1324, and a false negative marker 1325. The true positive marker 1321 indicates the true presence of a fluorescence signal. The false positive marker 1322 indicates the presence of a fluorescence signal that the system has determined to be a potential false positive. The true negative marker 1323 indicates the true absence of a fluorescence signal. The blind spot marker 1324 indicates data for which the system cannot determine the true configuration (such as data representing an image artifact). The false negative marker 1325 indicates data that may be a source of fluorescence but where the system did not detect a fluorescence signal. Thus, the false positive markers 1322, false negative markers 1325, and blind spot markers 1324 indicate the locations of potential false positives, false negatives, and / or blind spots in the multi-modality images that need to be manually confirmed by a user.

[0109] Potential false positive, false negative and / or blind spot information should be able to be switched on / off according to user preference so as not to interfere with the user's ability to examine the displayed information. To that end, the GUI 321 may include one or more toggle buttons 350 (or toggle switches) that the user can operate to execute one or more of the display algorithms described above. In FIG. 11, the toggle button 350 may be configured to switch only the third display panel 1121 to show either a fluorescent longitudinal section as shown in FIG. 3 or one or more markers in a simplified binary display (as shown in FIG. 11). Further, in FIG. 11, the system may be configured to toggle each of the markers 1321-1325 when the user slides the line indicator 315 along the longitudinal section of the OCT data 310 with the pointer 316. In this case, the toggle button 350 may optionally be removed (hidden), and switching of one or more of the markers 1321 , 1322 , 1323 , 1324 , 1325 may occur automatically when the user slides the line indicator 315 with the pointer 316 .

[0110] In one or more embodiments, to improve the accuracy of the system, the system may be programmed to learn from the markups made by the user during each procedure. For example, the user may add the location of false positive and / or false negative data after examining blind spot locations that were not identified by the system. To do this, the GUI 321 of the display device 300 should have a button or prompting function to prompt the user to optionally annotate or input specific information (e.g., biological and / or technical reasons that often cause blind spot detection). Once the user adds information about false positive and / or false negative cases and their causes, the system can identify patterns of such cases using OCT data (non-fluorescence data) and add the identified patterns for future detection. Blind spot (artifact) data identified by the user as false positive or false negative may be stored by the system in a database (table). The system can then use such database to further improve the accuracy of the detection of potential false positives, false negatives and / or blind spots using trained machine learning networks and / or artificial intelligence (AI) systems.

[0111] According to the present disclosure, it is advantageous to identify the location of potential false positives, false negatives and / or blind spots in a multi-modality image to improve diagnostic results. The above embodiment is described with the example of catheter-based intravascular imaging, where the luminal sample can be a blood vessel. As will be appreciated by those skilled in the art, the principles described herein apply equally to other luminal samples, such as respiratory or fire extinguisher lumens, which can be imaged by a multi-modality endoscope in obtaining coregistrated multi-modality image data including blind spot locations. After detecting such locations, the system disclosed herein is configured to display the results in a graphical user interface, and the user is prompted to manually review the results. However, the user is not required to do so, as relying solely on visual and manual evaluation may be overwhelmed with an excessive amount of data, leading to confusion and misinterpretation. Instead, the user can rely on predefined algorithms and a practical graphical user interface to allow the system to identify the locations of potential false positives, false negatives and blind spots. The user is then prompted to review the system's output results. In addition to easing the user's decision-making process, this technique can be particularly beneficial for inexperienced users (eg, for training purposes).

[0112] This disclosure has presented the identification of potential false positive, false negative and / or blind spot cases of fluorescence data in OCT-NIRAF image data by analyzing non-NIRAF information. In one embodiment, for “false positive” detection, the system utilizes lumen detection results to evaluate the distance from the lumen to the catheter sheath (or vice versa). In one embodiment, for “blind spot” detection, the system utilizes plaque type characterization, which can be performed by image processing of OCT and / or spectroscopic OCT that correlates with histological information. In one embodiment, for “false negative” detection, the system utilizes correlation of catheter NIRAF data with histological image data (e.g., H&E and Trichrome).

[0113] Based on the attenuation coefficient, calcified tissue has the greatest impact on NIRAF detection. Other information includes luminal diameter, EEL (external elastic lamina), geometrical properties, flushing efficiency (is the flushing sufficient?; is there residual blood in the OCT frame?, etc.). It is well known that blood cells are highly fluorescent, so residual blood can be an important indicator of false positive NIRAF detection. Identification can be performed on a frame-by-frame basis. To increase accuracy, a frame-to-frame analysis can be added. In the frame-to-frame analysis, the NIRAF / NIRF information can be utilized to determine relevant parameters (e.g. how much NIRAF / NIRF is present within a certain distance of the frame in the longitudinal direction).

[0114] The system is configured to display a warning indicating the detection of a potential false positive, false negative, and / or blind spot case. The warning can be turned on / off depending on the user's preference. Markings around the areas of potential false positive, false negative, and / or blind spot cases can be displayed in various configurations. In one configuration, the graphical user interface can utilize different patterns, different colors, and different markings to designate each type of image area. In one configuration, the imaging system can be configured to receive input from a user to manually add the locations of potential false positives, false negatives, and / or blind spots to the multi-modality image. This configuration is particularly advantageous when the user finds potential cases based on their own experience that were not identified by the system. In such cases, the user can add the locations of potential false positives, false negatives, and / or blind spots to the image along with the reason for adding such information. According to some embodiments, the system can identify the locations of potential false positives, false negatives, and / or blind spots based on non-NIRAF information and add labeling patterns to the identified locations for future detection (e.g., to compare the results of an initial analysis of the same sample with subsequent or later analyses).

[0115] In one or more embodiments, the NIRAF information regarding the location of false positives, false negatives and / or blind spots can be presented in a binary display format to reduce the amount of information displayed. For this purpose, the NIRAF data can be defined around a median (mean) intensity value and within a range of high and low threshold values. This avoids the need to display the full range of NIRAF intensity values ​​as in a "NIRAF carpet image" format that would arguably display more information than the clinician actually needs to make a decision. The resulting solution provides a NIRAF profile image with a simplified display of the NIRAF data in the angiogram by discarding the angle information and simplifying the display of intensity to only the position along the vessel length. In an alternative embodiment, the same concept of a binary display can be applied to NIRF. Assuming that both NIRF and NIRAF are displayed on the same system and are distinguishable from each other, the NIRF profile image is best achieved by using different colors, different shading, and different patterns for the NIRF profile image than for the NIRAF profile image. Both options are available in the angiogram menu, and these options can be switched between.

[0116] When referring to the description, specific details are set forth to provide a thorough understanding of the disclosed examples. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily lengthen the present disclosure. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. The breadth of the present invention is not limited by this specification, but rather only by the plain meaning of the claim terms employed.

[0117] In describing the exemplary embodiments shown in the drawings, specific terminology is used for clarity, however, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected, and it is understood that each specific element includes all technical equivalents that function in a similar manner.

[0118] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments, and the scope of the following claims should be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. 1. A computer-implemented method for detecting potential blind spots in data in multi-modality images of a luminal sample, comprising: acquiring a multi-modality image of the luminal sample, the multi-modality image including first and second image data simultaneously collected and co-registered by scanning an inner wall of the luminal sample with two or more wavelengths of light transmitted through a multi-modality imaging catheter; analyzing the first image data to identify a characteristic property of the luminal sample; detecting locations of potential blind spots of the second image data in the multi-modality image based on the characteristic properties of the luminal sample; displaying, on a display device, the multi-modality image in one or more of a plurality of image views along with markers indicating locations of the potential blind spots of the second image data relative to the first image data; Including, the luminal sample comprises a blood vessel; the first image data includes optical coherence tomography (OCT) image data of the blood vessel, and the second image data includes fluorescence data of the blood vessel; analyzing the first image data includes analyzing the OCT image data to identify the characteristic property of the blood vessel; and detecting locations of potential blind spots comprises processing the fluorescence data based on the characteristic properties of the blood vessel to detect locations of potential false positives in the fluorescence data and / or locations of potential false negatives in the fluorescence data in the multi-modality image of the blood vessel. method.

2. analyzing the OCT image data to detect characteristic properties of the blood vessel includes detecting one or more of plaque type, external elastic lamina, side branches, minimum diameter, and residual blood volume in the blood vessel; detecting locations of potential blind spots in the second image data includes analyzing the fluorescence data to detect locations of potential false positives in the fluorescence data corresponding to one or more of the plaque type, the external elastic lamina, the side branches, the minimum diameter, and the residual blood volume in the blood vessel, and / or locations of potential false negatives in the fluorescence data. The method of claim 1.

3. displaying the multi-modality image with the markers indicating locations of the potential blind spots in the second image data comprises displaying, on the display device, the OCT image data and the fluorescence data with the markers indicating locations of the potential false positives in the fluorescence data and / or locations of the potential false negatives in the fluorescence data in a manner corresponding to the OCT image data in a first view and a second view. The method of claim 1.

4. displaying the OCT image data on the display device includes displaying the OCT image data in a tomographic image, and displaying the fluorescence data includes displaying the locations of the potential false positives in the fluorescence data and / or the locations of the potential false negatives in the fluorescence data as arcs or rings disposed around an edge of the tomographic image. The method according to claim 3.

5. displaying the OCT image data on the display device includes displaying the OCT image data in a longitudinal section, and displaying the fluorescence data includes displaying the locations of the potential false positives in the fluorescence data and / or the locations of the potential false negatives in the fluorescence data in a longitudinal section substantially parallel to or superimposed on the longitudinal section of the OCT image data. The method according to claim 3.

6. displaying the OCT image data on the display device comprises displaying the OCT image data in a longitudinal section, and displaying the fluorescence data comprises displaying the locations of the potential false positives in the fluorescence data and / or the locations of the potential false negatives in the fluorescence data as a binary histogram superimposed on or substantially parallel to the longitudinal section of the OCT image data; displaying the binary histogram includes displaying a binary marker using a different color or a different pattern for each of the potential false positive locations in the fluorescence data and / or the potential false negative locations in the fluorescence data. The method according to claim 3.

7. 1. A computer-implemented method for detecting potential blind spots in data in multi-modality images of a luminal sample, comprising: acquiring a multi-modality image of the luminal sample, the multi-modality image including first and second image data simultaneously collected and co-registered by scanning an inner wall of the luminal sample with two or more wavelengths of light transmitted through a multi-modality imaging catheter; analyzing the first image data to identify a characteristic property of the luminal sample; detecting locations of potential blind spots of the second image data in the multi-modality image based on the characteristic properties of the luminal sample; displaying, on a display device, the multi-modality image in one or more of a plurality of image views along with markers indicating locations of the potential blind spots of the second image data relative to the first image data; Including, the luminal sample comprises a blood vessel; the first image data includes optical coherence tomography (OCT) image data of the blood vessel, and the second image data includes fluorescence data of the blood vessel; The method comprises: detecting a lumen edge of the blood vessel from the OCT image data; determining whether the catheter is located within a minimum distance from the lumen edge; The method further comprises:

8. if it is determined that the catheter is located within the minimum distance of the lumen edge, storing and / or displaying the fluorescence data as indicative of the location of the potential blind spot; The method of claim 7 further comprising:

9. 1. A computer-implemented method for detecting potential blind spots in data in multi-modality images of a luminal sample, comprising: acquiring a multi-modality image of the luminal sample, the multi-modality image including first and second image data simultaneously collected and co-registered by scanning an inner wall of the luminal sample with two or more wavelengths of light transmitted through a multi-modality imaging catheter; analyzing the first image data to identify a characteristic property of the luminal sample; detecting locations of potential blind spots of the second image data in the multi-modality image based on the characteristic properties of the luminal sample; displaying, on a display device, the multi-modality image in one or more of a plurality of image views along with markers indicating locations of the potential blind spots of the second image data relative to the first image data; Including, the luminal sample comprises a blood vessel; the first image data includes optical coherence tomography (OCT) image data of the blood vessel, and the second image data includes fluorescence data of the blood vessel; The method comprises: analyzing the fluorescence data to determine whether the blood vessel contains a calcified plaque in at least a particular region thereof; if the at least certain region of the blood vessel is determined to contain calcified plaque, storing and / or displaying the at least certain region of the blood vessel as a location of the potential blind spot; The method further comprises:

10. 1. A system for detecting blind spots in a multi-modality image of a blood vessel, the system comprising: a memory storing executable instructions; a processor configured to execute the executable instructions stored in the memory; Equipped with The processor, acquiring a multi-modality image of a blood vessel, the multi-modality image including first and second image data simultaneously collected and co-registered by scanning an interior wall of the blood vessel with two or more wavelengths of light transmitted through a multi-modality imaging catheter; analyzing the first image data to identify a characteristic property of the blood vessel; detecting potential blind spot locations in the second image data of the multi-modality images based on the characteristic properties of the blood vessels; displaying the multi-modality image on a display device along with markers indicating the potential blind spot locations of the second image data relative to the first image data; configured to run the processor displays the multi-modality image by outputting, to the display device, the first image data surrounded by the second image data in a first view and outputting, to the display device, the first image data aligned with the second image data in a second view; the processor displays the markers indicating the potential blind spot locations in the first view and the second view in the same color or pattern. system.

11. the potential blind spot locations include locations of potential false positives in the second image data and / or locations of potential false negatives in the second image data; The processor, receiving a request input from a user to display the locations of potential false positives and / or potential false negatives in the second image data corresponding to the one or more characteristic features in the displayed image of at least one of a plurality of imaging modalities; Further configured as follows: The system of claim 10.

12. The processor, rotating and pulling back an imaging probe to scan a region of the body cavity with at least two different wavelengths simultaneously irradiated onto a wall of the body cavity to generate optical coherence tomography (OCT) image data and fluorescence data; and displaying the OCT image data and the fluorescence data on the display device as images of each of a plurality of imaging modalities; The system of claim 10 , further configured to control:

13. the blind spot locations include locations of potential false positives in the fluorescence data and / or locations of potential false negatives in the fluorescence data; The processor, Detecting the characteristic feature of the body cavity based on the OCT image data; Detecting the locations of the potential false positives in the fluorescence data and / or the locations of the potential false negatives in the fluorescence data based on the characteristic property of the body cavity. The system of claim 12 further configured to:

14. The processor, displaying the OCT image data and the fluorescence data on the display device by displaying the OCT image data as a tomographic image and displaying the fluorescence data as arcs or rings arranged around an edge of the tomographic image; or displaying the OCT image data as a longitudinal section according to a pullback distance and a scan angle of the imaging probe, and displaying the fluorescence data as a binary histogram superimposed on or parallel to the longitudinal section of the OCT image data, thereby displaying the OCT image data and the fluorescence data on the display device. The system of claim 12 further configured to:

15. The processor, Detecting a lumen edge from the OCT image data; Determining if the lumen edge is within a certain distance of the imaging probe The system of claim 12 further configured to:

16. If the lumen edge is within the specified distance, the processor updates the displayed image by marking the location of the lumen edge as a location of a potential false positive in the fluorescence data. The system of claim 15.

17. The processor, analyzing the fluorescence data to determine whether the body cavity contains a calcified plaque in at least a particular region thereof; If the at least certain region of the body cavity includes a calcified plaque, the processor updates the displayed image by marking the at least certain region of the body cavity as a location of a potential false negative in the fluorescence data. The system of claim 12 further configured to:

Images