Object detection and measurement in multimodal imaging

Abstract

Systems and methods are provided for detecting features, outputting feature representations, determining measurements, and combinations thereof using machine learning algorithms. In some embodiments, first sample data from a first characterization modality and second sample data from a second characterization modality are provided to the machine learning algorithm. Improved feature detection, feature representations, and measurements may be achieved by using multi-modal data with the machine learning algorithm. In some embodiments, the first characterization modality is an interferometry modality and the second characterization modality is a spectroscopy modality. The first characterization modality may be optical coherence tomography and the second characterization modality may be a diffuse spectroscopy modality such as near infrared spectroscopy. The sample data may be luminal and / or vascular data useful for characterizing the vasculature of a subject, such as a human.

Description

[Technical field]

[0001] Priority Application This application claims the benefit of U.S. Provisional Patent Application No. 63 / 301,486, filed January 20, 2022, the disclosure of which is incorporated by reference in its entirety herein.

[0002] The present disclosure generally relates to methods for detecting and characterizing objects in images of a body cavity using more than one discriminative data source. [Background technology]

[0003] Multimodal imaging systems have been used to characterize lumens such as arteries. One example of such a system combines spectroscopy and interferometry imaging. Typically, feature detection and measurement, when performed, uses data from one or the other mode. For example, spectroscopy data provides information about the presence or absence of plaque within a location, while interferometry data provides structural information. Images may be spatially registered to present information to a clinician, but in some cases, it remains difficult to make a determination regarding the presence or absence of a particular feature of interest (e.g., distinguishing between types of plaque) and / or to make measurements related to those features. Summary of the Invention [Means for solving the problem]

[0004] Aspects of the present disclosure relate to data from a discriminative characterization modality being used as input to a machine learning algorithm (e.g., a multimodal machine learning feature detector), for example, for endoluminal image segmentation. The first sample data may be from a first characterization modality. The second sample data may be from a second characterization modality. The first sample data and the second sample data may be pre-processed or processed (e.g., via a feature extractor) before being provided (e.g., as input) to the machine learning algorithm. The machine learning algorithm may then output the detected (interesting) feature(s) (e.g., selected by a user during training and / or upon execution of the algorithm with test data), transformed (e.g., enhanced) feature representation(s), measurement(s) regarding the feature representation(s) of the detected feature(s), or a combination thereof. The first data source, acquired, for example, at a first characterization subsystem, may be interferometry. For example, the first sample data may be or include coherence-gated depth-resolved imaging (e.g., as in optical coherence tomography (OCT)). The second data source, acquired, for example, in a second characterization subsystem, may be spectroscopy-based. For example, the second sample data may be or include molecular information (e.g., from diffuse spectroscopy, such as near-infrared spectroscopy (NIRS)). The interferometric data source and the spectroscopy-based data source together may be used as input to a machine learning algorithm, such as, for example, a multimodal machine learning feature detector. Multiple data sets provide discriminatory sources of information, but image segmentation of the depth-resolved imaging modality may be improved by using, for example, a machine learning feature detector.

[0005] Sample characterization (e.g., material characterization) can be performed using electromagnetic radiation (e.g., light). In some cases, characterization can be improved when performing more than one characterization (e.g., multimodal characterization) method. However, challenges exist with multimodal characterization. For example, building a system capable of multimodal characterization is one challenge. Another is maximizing the utility of the collected multimodal information. Each modality can provide similar or differential data sources, which, at least in some embodiments, can be utilized in either parallel or sequence to utilize one another. For example, one data source can encode spatial information and provide a depth-resolved image (e.g., optical coherence tomography (OCT)), while another data source can encode molecular information based on a spectrum from a single point (e.g., diffuse reflectance spectroscopy (DRS), such as near-infrared spectroscopy (NIRS)). Each data source may detect light from a different sample volume, for example due to the optical properties of the sample being characterized and / or the wavelength(s) of light being used by the different characterization modalities. Furthermore, each data source may be processed and used to detect a unique output, but each may provide information that can enhance the other detection algorithms. Manual sequential approaches to provide compensation information between multiple algorithms rarely take full advantage of all the information stored in each data source. By using modern feature detection algorithms (e.g., machine learning, such as deep learning and / or neural networks) with inputs from each data source, higher order information can be utilized. Disclosed herein, among other things, is a method for optimizing detection of endoluminal features using a multimodal system that provides discriminatory characterization data sources by providing both data sources to a machine learning algorithm (e.g., feature detector) that has simultaneous access to both modalities.

[0006] Intraluminal imaging of tissue can utilize and be affected by many optical phenomena. For example, the scattering properties of tissue can affect the amount and direction of light scattering that occurs when the tissue is illuminated. Rayleigh scattering and Mie scattering are elastic scattering sources that cause photons to change direction without a change in energy. Elastic scattering can provide information about the refractive index, molecular density, molecular orientation, and / or structure of a sample, among others. Raman scattering is an inelastic scattering source that causes photons to change energy and propagation direction. In inelastic scattering, the amount of energy change depends on the vibration of intrinsic molecular bonds, and this energy shift (e.g., wavelength shift) can be used to probe the molecular composition of tissue. Inelastic scattering is a much less efficient optical phenomenon than elastic scattering, occurring approximately once every 10,000,000 photons.

[0007] The absorption properties of tissue can affect the amount of energy at a given wavelength that is deposited (e.g., absorbed) in tissue when it is illuminated. Due to the wavelength dependence of molecular absorption, the absorption properties of tissue can provide information about its molecular composition. Importantly, absorption is a very efficient process. When light is absorbed, the primary energy transfer is from light to heat, but other light interactions can occur. For example, some tissues have fluorescent molecules that can be excited when the energy of the illumination wavelength matches a specific band gap of the molecule, so that the molecule absorbs light and then re-emits it at a lower energy. Thus, fluorescence can be used to probe the molecular composition of tissues based on the amount and wavelength-dependent shape of the re-emitted light spectrum. Fluorescence can be intrinsic to natural tissue molecules, known as autofluorescence. Fluorescence can also be exploited by conjugating (e.g., labeling, tagging) natural tissue molecules with fluorescent probes (e.g., tags, molecules, labels) (e.g., by covalently binding fluorescent dyes to biomolecules).

[0008] Scattering and absorption occur simultaneously and can be difficult to characterize separately. However, there are modalities that rely more on one effect than the other. For example, interferometric imaging (e.g., OCT) relies on depth-dependent scatterers in tissue to generate images, whereas diffuse spectroscopy (e.g., fluorescence, Raman, diffuse reflectance spectroscopy (DRS)) relies on both scattering and absorption. Thus, while OCT and diffuse spectroscopy share fundamental information, they are not identical. For example, illumination at a focal point in tissue using OCT creates depth-resolved lines of the image, identifying its structure. Diffuse spectroscopy, on the other hand, is the integration of scattered light within the optical interrogation volume, which can give a wavelength-dependent spectrum that describes the molecular composition of the tissue, but generally does not provide depth-resolved data. In probe orientation, OCT generally measures backscattered light through a single-mode waveguide because coherence is important when performing interferometric imaging. Diffuse spectroscopy (e.g., fluorescence, Raman, DRS) on the other hand often uses multimode waveguides because collection efficiency is the primary parameter for optimization.

[0009] The efficiency of optical phenomena is a key attribute when developing multimodal systems. Some modalities, such as Raman spectroscopy, are inefficient and require long integration times (e.g., at least 100 ms). Other modalities, such as OCT, can image at very high sampling rates (e.g., 1-10 μs, e.g., 5 μs) due to the efficiency of the optical process. This difference is particularly important in some clinical contexts (e.g., intravascular imaging), where imaging can be performed over long distances (e.g., at least 100 mm) within short periods (e.g., 1-3 s, e.g., 2 s) at high rotational speeds (e.g., at least 10,000 rpm), for example imaging the lumen (e.g., coronary artery) while temporarily flushing blood out of the imaging path (e.g., using saline or radiopaque contrast). One approach may be to image with one modality and then image with another, but the optimal approach is to perform imaging nearly simultaneously (e.g., interwoven, parallel, simultaneous) to minimize acquisition time and registration artifacts (e.g., from cardiac motion) between the two modalities. Thus, first of all, the imaging system must carefully optimize the multimodal optics, detection scheme(s), and acquisition timing to provide high-fidelity co-registered data.

[0010] Optical design to optimize multiple modalities can be further complicated by the geometry of the optical system. In diffuse spectroscopy scenarios, such as DRS, where the same wavelength carries the detected signal as the illuminating light, specular reflections can obscure useful signals arising from light interacting with subsurface tissue. In other scenarios, such as fluorescence imaging, where the wavelength of the detected light is different from the illuminating wavelength, filters can be used to limit the illuminating light so collection efficiency is maximized. OCT measures backscattered light, but not in a diffuse manner, requiring focused illumination and detection. Thus, a multimodal system may measure light arising from different volumes of tissue by optimizing the geometry of the optical system.

[0011] The ability to perform multimodal imaging at high speeds has only recently become possible at commercially viable costs. The combination of high speed light sources, detectors, and rotation systems now makes it possible to acquire co-registered multi-modal information from small form factor rotating probes. Advances in acquisition hardware have shifted focus to innovations in data analysis methods that can intelligently optimize the information (e.g., spectroscopy and tomography information) from co-registered endoluminal data sets.

[0012] Imaging modalities often provide complementary information for feature detection purposes (e.g., segmentation and / or classification). For example, segmenting objects in OCT images often relies on the contrast and morphology of the structure. While it may be possible to segment objects, object classification can be challenging for similarly scattering objects (e.g., coronary plaques) that are similar in shape (e.g., fibrous plaques vs. lipid-rich plaques). Diffuse spectroscopy detects molecular composition (e.g., using near-infrared spectroscopy (NIRS)) and provides information about the type of tissue in the segmented image by angle. As disclosed herein, these discriminatory sources of information can be combined to improve the accuracy and specificity of feature detection (e.g., classification) in OCT images.

[0013] Multimodal imaging algorithms can also improve single-modality segmentation. For example, some objects in an OCT image may not be fully resolved (e.g., due to artifacts such as non-uniform rotational distortion (NURD)) and therefore may have less certain (e.g., less confident) regions of segmentation on the class probability map (e.g., due to attenuation) (e.g., due to poor axial resolution). In these cases, the diffuse spectroscopy input can provide information to the spatial segmentation algorithm to improve the confidence of the segmentation of a particular segmentation class. For example, a stent may have high diffuse reflectance at a given location because different volumes of tissue are imaged by the two modalities, even though it is poorly resolved by OCT, leaving only a visible shadow.

[0014] Multimodal imaging algorithms can also improve regression. For example, diffuse spectroscopy may be able to detect and / or classify objects based on their spectra, but may have challenges quantifying molecules (e.g., lipids) due to geometrical optical factors (e.g., the distance of the object from the imaging system) (e.g., the thickness of the object) or confounding other objects in the object's line of sight for quantification. In these cases, information from structural imaging modalities, such as OCT, may enable the algorithm to limit the effect of confounding factors and improve quantification.

[0015] Machine learning algorithms (e.g., multimodal imaging algorithms) can also transform (e.g., enhance) the output of a modality. For example, an OCT image may have artifacts or may decay rapidly. A machine learning algorithm with information about tissue fluorescence may enable enhancement of the OCT image by increasing low contrast regions or decreasing high contrast regions. As another example, the enhanced image may be an important input to another algorithm for feature detection and / or measurement. In some embodiments, the machine learning algorithm may transform (e.g., enhance) one or more feature representations of one or more features of interest, e.g., detected by the machine learning algorithm.

[0016] Machine learning algorithms (e.g., multimodal imaging algorithms) can also transform the visual representation of modalities. For example, DRS can provide information about the "color" of tissue. Thus, a multimodal algorithm with information from both OCT images and DRS spectra can assist in color transformation. For example, an image transformation algorithm may be trained on co-registered white light (e.g., visible spectrum) images and OCT images to transform the OCT images into pseudocolor images. Input to such an exemplary image transformation algorithm may enable improved image transformation into a color image (e.g., showing plaque with one or more specific colors).

[0017] The present disclosure recognizes that improvements in detection [e.g., semantic segmentation, 1D (e.g., line-based) segmentation, bounding box segmentation], transformation (e.g., enhancement), and measurement (e.g., quantification, regression) algorithms (e.g., separate or multi-feature algorithms) may require multi-modal, and possibly discriminatory, sources of data to significantly improve the performance of automated results and enable effective clinical decision making. Indeed, while clinicians do not necessarily have the time or expertise to understand and correlate multiple data sources such as spectral and / or structural information, machines can learn to do so with high performance, speed, and accuracy using the machine learning techniques disclosed herein. As such, the present disclosure describes and enables prior single modalities (e.g., in terms of intraluminal imaging) to be improved upon by describing and enabling multi-modal feature detection (e.g., segmentation) through the combination of discriminatory data sources of overlapping and complementary information, including, among other things, structural and spectroscopic data.

[0018] In some aspects, the present disclosure is directed to a method for detecting a feature of interest. The method may include receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality. The method may further include detecting the feature of interest (e.g., an external or internal structure of the subject (e.g., a physiological structure)) by providing, by the processor, the first sample data and the second sample data to a machine learning algorithm. The method may further include outputting (e.g., from the algorithm) by the processor, a feature representation of the feature of interest directed to the first characterization modality.

[0019] In some embodiments, the machine learning algorithm is trained to detect two or more features of interest. In some embodiments, the first sample data and the second sample data are both from an intraluminal characterization modality. In some embodiments, the first characterization modality is an interferometry modality (e.g., OCT) and the second characterization modality is an intensity measurement modality (e.g., a fluorescence modality). In some embodiments, the first characterization modality is a depth-dependent imaging modality (e.g., OCT) and the second characterization modality is a wavelength-dependent measurement modality (e.g., NIRS). In some embodiments, one or both of the first characterization modality and the second characterization modality are processed (e.g., formatted) before being input to the machine learning algorithm. In some embodiments, one or both of the first characterization modality and the second characterization modality are registered (e.g., with each other) before being input to the machine learning algorithm.

[0020] In some embodiments, the method includes registering, by the processor, one or both of the first sample data and the second sample data (e.g., with each other) prior to inputting the first sample data and the second sample data to the machine learning processor. In some embodiments, the machine learning algorithm is trained to detect features using labels from only the first characterization modality or only the second characterization modality. In some embodiments, the machine learning algorithm outputs the detected features with reference to the first characterization modality (e.g., only the first characterization modality). In some embodiments, the first sample data is generated from a first characterization modality detected at a first region having a first tissue volume within the body cavity, and the second sample data is generated from a second characterization modality detected at a second region having a second volume within the body cavity.

[0021] In some embodiments, the endoluminal characterization volumes of each modality do not completely overlap. In some embodiments, the first region and the second region do not completely overlap. In some embodiments, the first sample data is generated from detection by the first characterization modality at time t1, and the second sample data is generated from detection by the second characterization modality at time t2, where t2-t1<1 ms. In some embodiments, the first sample data and the second sample data are combined into combined sample data, and the combined sample data is input to the machine learning algorithm during the detection step. In some embodiments, the combination of the first sample data and the second sample data includes (e.g., consists of) appending the first sample data to the second sample data. In some embodiments, the appending includes merging the first sample data and the second sample data together.

[0022] In some embodiments, the machine learning algorithm has multiple stages into which data can be input. In some embodiments, information from a first characterization modality and from a second characterization modality are input to the machine learning algorithm as two unique inputs at different stages. In some embodiments, the first sample data and the second sample data are input separately to the machine learning algorithm at different stages of the multiple stages. In some embodiments, each sample data is subjected to feature extraction, and the output of the feature extraction is used as an input to the machine learning algorithm.

[0023] In some embodiments, the method includes inputting, by a processor, the first sample data and the second sample data to one or more feature extractors and generating, by the processor, output from the one or more feature extractors, and the detecting includes inputting the output from the one or more feature extractors to a machine learning algorithm.

[0024] In some embodiments, the method includes segmenting and classifying, by the processor, one or more extraneous objects (e.g., one or more optical fibers, one or more sheaths, one or more stent struts, one or more balloons). In some embodiments, the method includes segmenting and classifying, by the processor, one or more extraneous objects (e.g., the detecting step includes segmenting and classifying) using a machine learning algorithm (e.g., using the first sample data and the second sample data as inputs to the algorithm). In some embodiments, the method includes segmenting and classifying, by the processor, one or more vascular structures (e.g., lumen, intima, tunica media, external elastic tunica, bifurcation). In some embodiments, the method includes segmenting and classifying, by the processor, one or more vascular structures (e.g., using the first sample data and the second sample data as inputs to the algorithm) (e.g., the detecting step includes segmenting and classifying).

[0025] In some embodiments, the method includes segmenting and classifying, by the processor, the plaque morphology based on the plaque content (e.g., calcium, macrophages, lipids, collagen, or fibrous tissue). In some embodiments, the method includes segmenting and classifying, by the processor, the plaque morphology (e.g., the detecting step includes segmenting and classifying) using a machine learning algorithm (e.g., using the first sample data and the second sample data as inputs to the algorithm). In some embodiments, the method includes segmenting and classifying, by the processor, one or more necrotic cores or thin-capped fibrous atheromas (TCFAs). In some embodiments, the method includes segmenting and classifying, by the processor, one or more necrotic cores or TCFAs (e.g., the detecting step includes segmenting and classifying) using a machine learning algorithm (e.g., using the first sample data and the second sample data as inputs to the algorithm).

[0026] In some embodiments, the method includes detecting (e.g., segmenting and classifying), by the processor, one or more arterial pathologies. In some embodiments, the method includes detecting (e.g., segmenting and classifying), by the processor, one or more arterial pathologies using a machine learning algorithm (e.g., using the first sample data and the second sample data as inputs to the algorithm) (e.g., the detecting step includes segmenting and classifying).

[0027] In some embodiments, the method includes detecting (e.g., segmenting and classifying), by the processor, one or more spectroscopy-sensitive markers. In some embodiments, the method includes detecting (e.g., segmenting and classifying), by the processor, one or more spectroscopy-sensitive markers (e.g., one or more indices) using a machine learning algorithm (e.g., using the first sample data and / or the second sample data as inputs to the algorithm) (e.g., the detecting step includes segmenting and classifying). In some embodiments, the method includes adjusting (e.g., correcting) an image (e.g., an image generated by an interferometric technique, e.g., an OCT image) based on the detection of the one or more spectroscopy-sensitive markers (e.g., thereby mitigating effects of non-uniform rotational distortion of the probe (e.g., imaging catheter) observable in the image).

[0028] In some embodiments, the feature of interest includes (e.g., is) one or more foreign objects, one or more vascular structures, plaque morphology, one or more arterial pathologies, one or more necrotic cores or thin-capped fibrous atheromas, or any combination thereof.

[0029] In some embodiments, detecting, by the processor, the feature of interest includes segmenting and classifying the feature of interest. In some embodiments, the method includes determining, by the processor, one or more measurements based on the feature of interest (e.g., using a machine learning algorithm). In some embodiments, the one or more measurements include geometric measurements (e.g., angle, thickness, distance). In some embodiments, the one or more measurements include image-based measurements (e.g., contrast, brightness, histogram).

[0030] In some embodiments, the method includes determining, by the processor, the bad frame, insufficient blood flushing, contrast injection detection, or a combination thereof using a machine learning algorithm (e.g., using the first sample data and the second sample data as inputs to the algorithm).

[0031] In some embodiments, the method includes automatically (e.g., by the processor) initiating pullback of the imaging catheter and / or scanning of the probe based on the features of interest detected using the machine learning algorithm. In some embodiments, the method includes determining, by the processor, breakage of the optical probe based on the features of interest detected using the machine learning algorithm.

[0032] In some embodiments, the method includes detecting, by the processor, insufficient penetration based on features of interest detected using a machine learning algorithm. In some embodiments, the method includes correcting, by the processor, an image (e.g., an image generated by an interferometric technique, e.g., an OCT image) based on features of interest detected using a machine learning algorithm (e.g., thereby mitigating effects of non-uniform rotational distortion of a probe (e.g., an imaging catheter) observable in the image).

[0033] In some embodiments, the method includes generating first sample data using a first characterization modality and second sample data using a second characterization modality, hi some embodiments, generating the first sample data and the second sample data includes performing a catheter pullback.

[0034] In some embodiments, the method includes enhancing, by the processor (e.g., automatically using a machine learning algorithm), feature representations with respect to the first characterization modality based on the second sample data and / or with respect to the second characterization modality based on the first sample data (e.g., the enhanced feature representations are output from the machine learning algorithm).

[0035] In some embodiments, the feature representation is registered to the first sample data, the second sample data, or both the first and second sample data. In some embodiments, the method includes outputting (e.g., displaying), by the processor, the feature representation overlaid on an image (e.g., an OCT image) derived from the first sample data or the second sample data.

[0036] In some embodiments, the method is performed after pullback of the catheter (e.g., automatically upon completion of pullback) during which the first sample data and the second sample data are acquired. In some embodiments, the feature representation of the feature of interest is also directed with respect to a second characterization modality.

[0037] In some embodiments, the outputting includes displaying (e.g., on a display included in a system, such as a catheter system). In some embodiments, the outputting is via one or more graphical user interfaces (GUIs). In some embodiments, the outputting includes storing (e.g., in a non-transitory computer readable medium).

[0038] In some aspects, the present disclosure is directed to a multi-modal system for feature detection including a processor and a non-transitory computer-readable medium having stored thereon instructions that, when automatically executed by the processor upon initiation of a characterization session, cause the processor to receive first sample data from a first characterization modality and second sample data from a second characterization modality, cause the processor to detect features of interest by providing the first sample data and the second sample data to a machine learning algorithm, and output, by the processor, a feature representation of the feature of interest directed to the first characterization modality.

[0039] In some embodiments, the system further includes a display. In some embodiments, the instructions, when executed automatically by the processor at the start of a characterization session, cause the processor to output the feature representation using the display. In some embodiments, the system further includes a first characterization subsystem for a first characterization modality and a second characterization subsystem for a second characterization modality. In some embodiments, the instructions, when executed automatically by the processor at the start of a characterization session, cause the processor to perform the methods described herein. In some embodiments, the system is a catheter system. In some embodiments, the system includes a probe operable to collect sample data of one or more characterization modalities (e.g., two characterization modalities (e.g., an interferometry modality (e.g., OCT) and a spectroscopy modality (e.g., NIRS)). The probe may be sized and shaped to collect light from inside a body cavity (e.g., an artery) and transmit it to one or more detectors. The one or more detectors may be included in the system.

[0040] In some aspects, the present disclosure is directed to a method for measuring a feature of interest, the method including receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality, detecting, by the processor, the feature of interest by providing the first sample data and the second sample data to a machine learning algorithm, and determining, by the processor, one or more measures of a feature representation of the feature of interest.

[0041] In some embodiments, the method further includes displaying, by the processor, the measurements on a display. In some embodiments, both the first sample data and the second sample data are from an endoluminal characterization modality. In some embodiments, the measurements include geometric measurements (e.g., angle, thickness, distance, depth). In some embodiments, the measurements include image-based measurements (e.g., contrast, brightness, histogram). In some embodiments, the measurements quantify aspects of intraluminal extraneous objects (e.g., one or more optical fibers, one or more sheaths, one or more stent struts, one or more balloons). In some embodiments, the measurements relate to positioning of intraluminal extraneous objects (e.g., stent placement in an artery). In some embodiments, the measurements quantify aspects of vascular structure (e.g., lumen, intima, tunica media, external elastic membrane, bifurcation). In some embodiments, the measurements quantify aspects of plaque morphology (e.g., amount of calcium, macrophages, lipids, fibrous tissue, or necrotic core in an area). In some embodiments, the measurements quantify a risk associated with detected plaque (e.g., detected TCFA). In some embodiments, the measurements include lipid pool or thickness of the cap on the necrotic core. In some embodiments, the measurements include plaque volume or lipid core volume (e.g., maximum volume over distance). In some embodiments, the measurements include plaque instability. In some embodiments, the measurements include calcium measurements (e.g., arc, thickness, size, area, volume, calcium to other ratios). In some embodiments, the measurements include lipid measurements (e.g., arc, thickness, size, area, volume, lipid to other ratios). In some embodiments, the measurements include stent malapposition, stent length, or planned stent position.

[0042] In some embodiments, the method includes automatically determining, by a processor (e.g., using one or more machine learning algorithms, e.g., the machine learning algorithm), a location for stent placement based on one or more measurements (e.g., by optimization). In some embodiments, the one or more measurements include lumen area. In some embodiments, the measurements include measurements related to the external elastic membrane or external elastic lamina. In some embodiments, the machine learning algorithm outputs the measurements.

[0043] In some embodiments, the method includes generating first sample data using a first characterization modality and second sample data using a second characterization modality, hi some embodiments, generating the first sample data and the second sample data includes performing a catheter pullback.

[0044] In some aspects, the present disclosure is directed to a method for enhancing data acquired from a body cavity, the method including receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality, detecting features of interest by the processor by providing the first sample data and the second sample data to a machine learning algorithm, and outputting, by the processor, a transformed representation of either the first sample data, the second sample data, or both, from the algorithm based on the detected features.

[0045] In some embodiments, the method includes displaying (e.g., by a processor) the transformed representation (e.g., outputting includes displaying the transformed representation). In some embodiments, the method includes inputting, by the processor, the transformed representation to another machine learning algorithm for feature detection. In some embodiments, the method includes detecting, by the processor, features of interest using a machine learning algorithm for feature detection based on the transformed representation input. In some embodiments, the transformed representation is an enhanced OCT image. In some embodiments, the transformed representation uses the detected features to correct for non-uniform rotational distortion of the probe. In some embodiments, the transformed representation is enhanced reflectance data. In some embodiments, the transformed representation is enhanced spectroscopy data. In some embodiments, the transformed representation is in a new image space or color scheme. In some embodiments, the method includes determining, by the processor, a specular to diffuse reflectance ratio based on one of the first sample data and the second sample data (e.g., using a machine learning algorithm) and improving or enhancing attenuation correction in the transformed representation, where the transformed representation is another of either the first sample data and the second sample data. In some embodiments, the transformed representation is an attenuation corrected representation.

[0046] In some aspects, the present disclosure is directed to a method for detecting features of interest, the method including receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality, and detecting, by the processor, one or more features of interest by providing the first sample data and the second sample data to a machine learning algorithm. In some embodiments, the method includes automatically (e.g., by the processor) initiating (i) pullback of the imaging catheter and / or (ii) a scan of the probe based on the one or more features of interest detected using the machine learning algorithm.

[0047] In some aspects, the present disclosure is directed to a method of compensating for non-uniform rotational distortion (NURD), the method including receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality; evaluating, by the processor, NURD by providing at least one of the first sample data and the second sample data to a machine learning algorithm; and correcting, by the processor (e.g., using the machine learning algorithm), at least one of the first sample data and the second sample data based on the evaluation to accommodate NURD.

[0048] In some embodiments, the evaluating includes providing only the first sample data to the machine learning algorithm, and the correcting is of the second sample data, hi some embodiments, the first sample data and / or the second sample data are images.

[0049] In some aspects, the present disclosure is directed to a method for determining an improved physiological measurement, the method including receiving, by a processor, first sample data from a first characterization modality; receiving, by the processor, information regarding a feature of interest (e.g., a location and / or composition of the feature of interest), where at least a portion of the first sample data corresponds to the feature of interest (e.g., a feature representation of the feature of interest is included in the first sample data); and determining, by the processor, a physiological measurement using the first sample data and the information.

[0050] In some embodiments, the feature of interest is a plaque or a curvature of a blood vessel (e.g., an artery). In some embodiments, the physiological measurement corresponds to a flow rate, a pressure drop, or a resistance of a vascular structure (e.g., an artery) (e.g., the physiological measurement is a measurement of flow velocity, fractional flow reserve, pressure drop, absolute or relative coronary flow (CF), fractional flow reserve (FFR), instantaneous fractional flow reserve / total cycle resting coronary pressure ratio (iFR / RFR), index of coronary microvascular resistance (IMR), maximum hyperemic coronary microvascular resistance (HMR), maximum hyperemic stenosis resistance (HSR), coronary flow reserve (CFR), or a combination thereof).

[0051] In some embodiments, the first characterization modality is an interferometry modality (e.g., OCT). In some embodiments, the method includes determining, by the processor, a location and / or composition of the feature of interest using second sample data from the second characterization modality (e.g., as well as the first sample data). In some embodiments, the second characterization modality is a spectroscopy modality (e.g., NIRS). In some embodiments, the method includes determining, by the processor, information about the feature of interest using the first sample data.

[0052] In some embodiments, the method includes determining, by the processor, information about the feature of interest using a machine learning algorithm (e.g., by providing the first sample data (e.g., and / or the second sample data) to the machine learning algorithm). In some embodiments, the method includes detecting, by the processor, using a machine learning algorithm (e.g., as described above) (e.g., by providing the first sample data (e.g., and / or the second sample data) to the machine learning algorithm).

[0053] In some aspects, the present disclosure is directed to a method for performing a physiological measurement, the method including receiving, by a processor of a computing device, first sample data from a first characterization modality and second sample data from a second characterization modality, and determining, by the processor, a physiological measurement value by providing the first sample data and the second sample data to a machine learning algorithm.

[0054] In some aspects, the present disclosure is directed to a method of training a machine learning algorithm, the method including providing, by a processor of a computing device, training data to the machine learning algorithm, the training data being labeled with training labels derived from data from a second characterization modality that is different from the first characterization modality.

[0055] In some embodiments, the first characterization modality is an interferometry modality (e.g., OCT) and the second characterization modality is a spectroscopy modality (e.g., NIRS). In some embodiments, the training data does not include data from the second characterization modality.

[0056] In some aspects, the present disclosure is directed to a method for detecting and / or determining measurements of a feature of interest, the method including receiving, by a processor of a computing device, first sample data from a first characterization modality (e.g., an interferometry modality (e.g., OCT)); detecting the feature of interest by providing, by the processor, the first sample data to a machine learning algorithm trained with training data from the first characterization modality, where the training data is labeled with training labels derived from data from a second characterization modality (e.g., a spectroscopy modality (e.g., NIRS)); and outputting, by the processor (e.g., from the algorithm), (i) a feature representation of the feature of interest directed to at least the first characterization modality; (ii) one or more measurements (e.g., including the feature representation and / or a physiological measurement); or (iii) both (i) and (ii).

[0057] In some embodiments, the machine learning algorithm does not accept as input data from the second characterization modality. In some embodiments, the machine learning algorithm does not accept as input data from any other characterization modality than the first characterization modality. In some embodiments, the method includes outputting, by the processor, the feature representation and / or at least one measure of the feature representation, wherein the feature of interest is plaque or a portion thereof (e.g., a lipid core of the plaque).

[0058] In some aspects, the present disclosure is directed to a system that includes a processor and a non-transitory computer-readable medium having instructions stored thereon that, when executed by the processor (e.g., automatically upon initiation of a characterization session), cause the processor to perform the methods disclosed herein.

[0059] In some embodiments, the system further includes a display. In some embodiments, the instructions, when executed automatically by the processor at the start of a characterization session, cause the processor to output the feature representation using the display. In some embodiments, the system further includes a first characterization subsystem for a first characterization modality and a second characterization subsystem for a second characterization modality. In some embodiments, the instructions, when executed automatically by the processor at the start of a characterization session, cause the processor to perform the methods described herein. In some embodiments, the system is a catheter system. In some embodiments, the system includes a probe operable to collect sample data of one or more characterization modalities (e.g., two characterization modalities (e.g., an interferometry modality (e.g., OCT) and a spectroscopy modality (e.g., NIRS)). The probe may be sized and shaped to collect light from inside a body cavity (e.g., an artery) and transmit it to one or more detectors. The one or more detectors may be included in the system.

[0060] In some aspects, the present disclosure is directed to a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the methods disclosed herein.

[0061] Any two or more of the features described in this specification, including in this summary section, may be combined to form embodiments of the present disclosure, whether or not specifically and explicitly described in separate combinations herein.

[0062] The drawings herein are presented for purposes of illustration and not limitation. The drawings are not necessarily to scale. The foregoing and other objects, aspects, features, and advantages of the present disclosure will become more apparent and may be better understood by referring to the following description in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0063] [Figure 1] FIG. 1 shows a diagram of a multi-modal characterization system according to an exemplary embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of a method of using a multi-modal feature detection system according to an exemplary embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of a method of using a multi-modal feature enhancement system according to an exemplary embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method of using a multi-modal feature detection and measurement system according to an exemplary embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of a method of using a multi-modal feature detection system according to an exemplary embodiment of the present disclosure. [Figure 6A] 1 illustrates a method for appending or combining one-dimensional multi-modal sample data into a single feature detector for multi-modal feature detection, according to an exemplary embodiment of the present disclosure. [Figure 6B] 1 illustrates a method for appending or combining two-dimensional multi-modal sample data into a single feature detector for multi-modal feature detection, according to an exemplary embodiment of the present disclosure. [Figure 6C] 1 illustrates a method for appending or combining N-dimensional multi-modal sample data into a single feature detector for multi-modal feature detection, according to an exemplary embodiment of the present disclosure. [Figure 7A] FIG. 1 is a block diagram of a multi-modal feature detection system according to an exemplary embodiment of the present disclosure. [Figure 7B] FIG. 1 is a block diagram of a multi-modal feature detection system having a dedicated feature extractor for each sample data source prior to a machine learning feature detection algorithm, in accordance with an exemplary embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram of a multi-modal feature detection system having a dedicated pre-processor for each sample data source prior to a machine learning feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 9]FIG. 1 is a block diagram of a multi-modal feature detection system having a dedicated feature extractor for each sample data source prior to a machine learning feature detection algorithm, in accordance with an exemplary embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram of a multi-modal feature enhancement system according to an exemplary embodiment of the present disclosure. [Figure 11A] FIG. 1 is a block diagram of a multi-modal feature measurement system according to an exemplary embodiment of the present disclosure. [Figure 11B] FIG. 1 is a block diagram of a multi-modal feature detection and measurement system according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram of a multi-modal feature detection algorithm according to an exemplary embodiment of the present disclosure. [Figure 13A] 1 illustrates an exemplary 1D sample data source for a multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 13B] 1 illustrates an exemplary 1D sample data source for a multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 14A] 1 illustrates an exemplary 1D sample data source for an appended multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 14B] 1 illustrates an exemplary 2D sample data source for a multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 15] 1 shows an exemplary segmentation output overlaid on and directed against an input 2D arterial OCT image, where the exemplary output has been generated (e.g., predicted) using a pre-trained multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 16] 1 illustrates an exemplary line-based segmentation output directed at an input 2D arterial OCT image, where the exemplary output has been generated (e.g., predicted) using a pre-trained multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 17] 1 illustrates an exemplary line-based segmentation output directed at an input 2D arterial OCT image, where the exemplary output has been generated (e.g., predicted) using a pre-trained multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 18] 1 illustrates an exemplary bounding box-based segmentation output directed at an input 2D arterial OCT image, where the exemplary output has been generated (e.g., predicted) using a pre-trained multi-modal feature detection algorithm, according to an exemplary embodiment of the present disclosure. [Figure 19] 1 illustrates an exemplary output from a pre-trained multi-modal feature enhancement algorithm, according to an exemplary embodiment of the present disclosure. [Figure 20] 1 illustrates an exemplary output from a pre-trained multi-modal feature enhancement algorithm, according to an exemplary embodiment of the present disclosure. [Figure 21] 1 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm, according to an example embodiment of the present disclosure. [Figure 22] 1 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm, according to an example embodiment of the present disclosure. [Figure 23] 1 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm, according to an example embodiment of the present disclosure. [Figure 24] 1 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm, according to an example embodiment of the present disclosure. [Diagram 25] 1 illustrates an exemplary display (e.g., user interface) of output from a pre-trained multi-modal feature detection and measurement algorithm, according to an exemplary embodiment of the present disclosure. [Figure 26] FIG. 1 is a block diagram of an exemplary network environment for use in the methods and systems described herein, in accordance with an exemplary embodiment of the present disclosure. [Figure 27] 1A-1C are block diagrams of an exemplary computing device and an exemplary mobile computing device for use in exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] Provided herein are systems and methods for, among other things, using data from different characterization modalities to improve feature detection and / or extraction, feature representation, feature transformation, data (e.g., image) transformation, measurement (e.g., assessment), and combinations thereof. Thus, sample data from one characterization modality may be used to improve data (e.g., images) from another characterization modality (and vice versa). In various embodiments, such improvement is achieved using a machine learning algorithm. The machine learning algorithm may receive sample data as input from multiple characterization modalities, either alone or in addition to other inputs. In some embodiments, first sample data from a first characterization modality and second sample data from a second characterization modality are provided to the machine learning algorithm. The machine learning algorithm may have been trained on such data. In some embodiments, the algorithm is trained on the data to generate the machine learning algorithm. Features of interest may be detected using a machine learning algorithm. Detection of features may include semantic segmentation, line-based segmentation, or frame-based segmentation (e.g., bifurcation or low quality frames).

[0065] Using the methods disclosed herein, the spectroscopic modality can enhance the interferometric modality (e.g., OCT) (e.g., contrast, brightness, structure, sharpening, or a combination thereof). Using the methods disclosed herein, the interferometric modality (e.g., OCT) can enhance the spectroscopic modality. For example, the spectroscopic data can be enhanced using scaling, calibration or normalization, based on distance from the lumen wall, etc. Using the methods disclosed herein, the detected features (e.g., lipids) in the image (e.g., OCT image) can be selectively enhanced, for example, based on sample data from a spectroscopic characterization modality. Using the methods disclosed herein, the detected features in the spectroscopic data can be selectively enhanced (e.g., to identify lipid types) using data from an interferometric characterization modality such as OCT.

[0066] In some embodiments, the methods and / or systems disclosed herein employ machine learning algorithms such as neural networks, random decision forests, support vector machines, or other machine learning models. The machine learning algorithm may use (e.g., perform) segmentation and / or classification, for example, to detect one or more features of interest, determine one or more measurements (e.g., of one or more feature representations of one or more features of interest), transform (e.g., enhance) the feature representation of one or more features of interest, or a combination thereof. The machine learning algorithm may perform multiple such functions in sequence or simultaneously. The machine learning algorithm may use a single stage or multiple stages, for example, using different stages to perform different such functions (e.g., based on different inputs), for example, a first stage performing feature extraction / detection, and a second stage determining one or more measurements. The machine learning algorithm may be or include a feature detector and / or a feature extractor.

[0067] In some embodiments, the machine learning algorithm includes (e.g., is) a classifier trained using a training dataset using supervised or semi-supervised learning. A training dataset for such a classifier algorithm typically includes multiple training examples, each training example being test data for a sample and a ground truth value for the class of the sample. The training examples may be obtained empirically and / or may be synthetic training examples calculated using software simulation. In some embodiments, the empirical training examples are generated from one or more catheter insertion procedures imaging one or more portions of one or more subjects (e.g., human(s)), e.g., from a population of subjects exhibiting a range of physiology. A range of different samples may be obtained, such as from human subjects from different demographic groups, to avoid any unintended bias in the performance of the resulting machine learning model. For each sample used to form the training data examples, the class of the sample (e.g., healthy tissue, plaque morphology, and / or calcium morphology) may be known from the medical records of the human subject from which the sample originates. The machine learning algorithm may then be trained using any suitable training algorithm and an objective function that considers the difference between the predicted value of the algorithm and the ground truth class of the training examples. In some embodiments where the machine learning model is a neural network such as a multi-layer perceptron or other type of neural network, the neural network may be trained using backpropagation. The machine learning algorithm may then be trained using the available training data items or until little change in the parameters of the machine learning model is observed. Once trained, the machine learning algorithm may be deployed to any suitable computing device, such as a computing device included in an imaging catheter system, a hospital desktop computer, or a web server in the cloud.The deployed machine learning algorithm may receive test data items from previously unused samples to train the algorithm, for example, to detect one or more features of interest, to determine one or more measurements, to transform a representation of one or more features of interest, or a combination thereof. In some embodiments, the machine learning algorithm processes the test data items to calculate a prediction value indicating which of a plurality of classes the test data will be classified into. In some cases, the machine learning algorithm provides a confidence value indicating the uncertainty associated with the prediction value. In some embodiments, the machine learning algorithm performs segmentation in addition to classification, and such algorithms may be trained to perform segmentation using an approach similar to classification.

[0068] In general, any suitable training approach may be used to form the machine learning algorithm. In some embodiments, a predicate system is used to train the multimodal system as disclosed herein. In some embodiments, the training is based on at least manually annotated data. For example, a specialist, such as a suitable physician (e.g., a cardiologist), may annotate data (e.g., images) that are then input as training data for forming the machine learning algorithm. For example, the cardiologist may annotate the location, size, or other parameters of plaque within a larger data set. Such annotations may be made, for example, on data oriented in a polar or Cartesian coordinate system. In some embodiments, an output (e.g., co-registered output) from another device may be used to train the algorithm. In some embodiments, the registered ground truth histology data is used to train the algorithm. Also, any combination of these (manual annotation, co-registered output from another device, registered ground truth) may be used to train the algorithm to the machine learning algorithm. Any one or combination of these approaches (manual annotation, co-registered output from another device, registered ground truth) may be used to generate training labels, which are then associated with the training data. The training labels may be derived from data from one characterization modality, but the training data for which the labels are used is data from a different characterization modality. For example, training labels derived from data from a spectroscopy modality may be used with training data from an interferometry modality (e.g., a co-registered dataset from two modalities may be used to generate labeled training data).

[0069] In some embodiments, an algorithm is trained or has been trained to detect one or more objects (e.g., plaque). In some embodiments, such detection is based solely on a first characterization modality (e.g., an interferometry modality (e.g., OCT)). Training of such an algorithm may be performed using training labels derived from data from a second characterization modality (e.g., a spectroscopy modality (e.g., NIRS)). In some embodiments, the use of co-registered multi-modal data allows such algorithm training, for example, where training data is derived from one modality and training labels for the training data are derived from another modality. Thus, in some embodiments, detection and / or transformation (e.g., enhancement) of one or more features of interest and / or measurements of the features may use data from only one characterization modality as input to a machine learning algorithm trained using data labeled based on another characterization modality. One example of such an algorithm uses only interferometry modality (e.g., OCT) data to detect and / or transform (e.g., enhance) objects (e.g., lipid core plaques) yet uses spectroscopy (e.g., NIRS) data for training labels. In some embodiments, the training labels are also derived from a first characterization modality that corresponds to the modality used to generate the training data (e.g., the training labels are derived from two modalities and the training data is derived from only one of the two modalities).

[0070] Thus, in some embodiments, improved feature detection (e.g., segmentation and / or classification) and / or transformation (e.g., enhancement) and / or its measurement can be realized from sample data from only one characterization modality by using a machine learning algorithm trained based on information derived from more than one characterization modality. The advantage(s) of the second characterization modality can be realized when only sample data from the first characterization modality is available using some such machine learning algorithms. For example, a spectroscopy modality may be better suited to characterize the composition of a particular feature of interest compared to an interferometry modality, so feature detection and / or transformation and / or its measurement may be improved using training data from an interferometry modality that is labeled based on data from the spectroscopy modality. Approaches such as these may be useful, for example, when collecting sample data using a device such as a catheter that is a single modality device, or when multimodal data is poorly co-registered or cannot be co-registered for some reason.

[0071] In some embodiments, the machine learning algorithm outputs a feature representation (e.g., a transformed (e.g., enhanced) feature representation) and / or one or more measurements. In some embodiments, the outputting includes displaying the feature representation, e.g., the software may be programmed to automatically display the data output from the machine learning algorithm for viewing by the user. In some embodiments, the outputting includes saving (e.g., storing) the feature representation and / or transmitting it to another computing device (e.g., a hospital PACS). The one or more measurements may be displayed near (e.g., adjacent to) and / or overlaid on one or more images. The image may be a combined representation of multiple characterization modalities, e.g., corresponding to a combination of an interferometry modality and a spectroscopy modality. As disclosed herein, a machine learning algorithm may be used to transform a representation of a feature of interest corresponding to one modality using data from another modality (e.g., vice versa). The feature representation generated using the machine learning algorithm may be combined with image data that has not been modified using the machine learning algorithm and displayed to the user. For example, a machine learning algorithm may be used to detect features of interest, generate a feature representation, optionally a transformed feature representation, and output the representation (e.g., the transformed representation) as a composite image including the feature representation and any unaffected surrounding data in a single image. For example, the morphology of plaque and / or calcium may be generated as a feature representation output from the machine learning algorithm, which is then used to image including a representation of the surrounding tissue derived from the original sample data unaltered by the machine learning algorithm (e.g., not input to the machine learning algorithm). In some embodiments, the entire image (e.g., displayed) is generated as an output from the machine learning algorithm, and only one or more (e.g., detected) features of interest are depicted as the transformed representation, for example. In some embodiments, the feature representation is registered to the first sample data, the second sample data, or both the first sample data and the second sample data.For example, the first sample data may be an OCT image or may be otherwise usable to derive an OCT image, and the feature representation output of the machine learning algorithm may be registered to the first sample data.

[0072] The machine learning algorithm may use, for example, a regression-based model (e.g., a logistic regression model), a regularization-based model (e.g., an elastic net model or a ridge regression model), an instance-based model (e.g., a support vector machine or a k-nearest neighbor model), a Bayesian-based model (e.g., a naive-based model or a Gaussian naive-based model), a clustering-based model (e.g., an expectation-maximization model), an ensemble-based model (e.g., an adaptive boosting model, a random forest model, a bootstrap agglomeration model, or a gradient boosting machine model), or a neural network-based model (e.g., a convolutional neural network, a recurrent neural network, an autoencoder, a backpropagation network, or a stochastic gradient descent network). In embodiments, the machine learning model is trained using a supervised learning algorithm, an unsupervised learning algorithm, a semi-supervised learning algorithm (e.g., partially supervised), weak supervision, transfer, multi-task learning, or any combination thereof. In some embodiments, the machine learning algorithm uses a model that includes parameters (e.g., weights) that are adjusted during training of the model. For example, parameters can be tuned to improve the predictive ability of a machine learning algorithm by minimizing a loss function.

[0073] In some embodiments, the machine learning algorithm uses multi-modal data (e.g., including interferometry data and spectroscopy (e.g., diffuse spectroscopy) data, e.g., appended to one another) to detect one or more features (e.g., one or more images) of interest in the sample data. In general, the feature of interest can be any feature that is of interest to a particular user for characterization. In some embodiments, the systems and methods disclosed herein are used to characterize a subject (e.g., a human). For example, the system may be an imaging catheter system, such as a system for intravascular imaging. In some embodiments, the feature of interest can be an internal or external structure (e.g., physiological structure) of the subject. The systems disclosed herein (e.g., imaging catheter systems) may be used to image, for example, one or more blood vessels (e.g., arteries) and / or one or more organs (e.g., eye(s)). The feature of interest can be a structure in and / or on such blood vessel or organ. The following are non-limiting examples of objects that can be detected (e.g., segmented and / or classified) (e.g., in one or more images) using the methods disclosed herein (e.g., using interferometry and spectroscopy data). In some embodiments, any one or more of these objects can be features of interest. One or more arterial wall structures can be detected. For example, lumen, external elastic membrane (EEM), external elastic lamina (EEL), intima, media, adventitia, collaterals, calcium, lipids, lipid subtypes, calcium subtypes, collagen, cholesterol, or combinations thereof can be features of interest (e.g., detected and / or representations thereof measured). Capsular thickness over lipid pools / necrotic cores can also be detected and / or measured. Arterial pathology can be one or more features of interest (e.g., detected), such as pathological intimal thickening, intimal xanthomas, early and late fibrous atheroma, thin-capped fibrous atheroma, one or more fibrolipid plaques, or one or more fibrocalcified plaques.The extraneous object may be one or more features of interest (e.g., detected), such as a catheter, a probe lens, a probe reflector, a probe (e.g., catheter) sheath, a guidewire, a stent(s), or a combination thereof. The curvature of a blood vessel may be a feature of interest.

[0074] In some embodiments, the machine learning algorithm outputs a feature representation of the feature of interest. The algorithm may output different feature representations for different features of the object of interest, or a common representation that includes multiple features of the object of interest. The feature representation may be generated from the features of the object of interest detected by the algorithm. In general, the feature representation is data. The data defining the feature representation may be sufficient to be displayed as an image, for example on a display included in the multimodal characterization system. The feature representation may simply be saved (e.g., stored) and / or transmitted, as opposed to being displayed, for example. The data defining the feature representation may define only a portion of the image, for example an image that represents a portion of the object, such as a blood vessel. For example, the data defining the feature representation may be integrated (e.g., appended to) other data to form a complete image of the portion of the object. The feature representation may include a representation of the feature of interest (e.g., sample data) in addition to other information that does not correspond to the feature of interest (e.g., other structure(s) surrounding the feature of interest in or on the object). The feature representation may correspond to one or more characterization modalities, e.g., may represent data from an interferometry modality and a spectroscopy modality. The feature representation may be generated using a machine learning algorithm based on sample data from multiple characterization modalities. The machine learning algorithm may output one or more transformed (e.g., enhanced) feature representations, e.g., of one or more features of interest. In some embodiments, multiple feature representations of a single feature of interest are output, e.g., to provide a user (e.g., physician) with a more holistic assessment of the feature of interest.

[0075] The one or more measurements may be determined (e.g., automatically) (e.g., using a machine learning algorithm) based on the feature representation data output from the machine learning algorithm. For example, the feature of interest may be a lipid core, the machine learning algorithm may detect the presence of the lipid core and generate data defining a feature representation of the lipid core, and one or more measurements, e.g., core area, volume, and / or thickness, may be determined using the feature representation data. Generating the data defining the feature representation may include processing a first sample data from a first characterization modality, a second sample data from a second characterization modality, or both such first sample data and such second sample data. Generating the data defining the feature representation may include selectively segmenting / extracting a first sample data from a first characterization modality, a second sample data from a second characterization modality, or both such first sample data and such second sample data that is identified as corresponding to the feature of interest. An image or a portion thereof may be transformed (e.g., enhanced) using machine learning algorithms. For example, a feature representation included in an image may be transformed (e.g., enhanced) using machine learning algorithms. Enhancement of image data may include one or more of processing, pre-processing, improving (e.g., resolution), contrast, scale, and ratio. Transformation of at least a portion of an image (e.g., its feature representation) (e.g., the entire image) may result in a new imaging space or color scheme. In some embodiments, an image (e.g., an OCT image) (e.g., derived from sample data from an interferometry modality) is transformed into another image space (e.g., white light image, pseudotissue stain) based on spectroscopy data (e.g., visible diffuse reflectance spectroscopy, or infrared spectroscopy or autofluorescence). Using the systems and methods disclosed herein, the specular to diffuse reflectance ratio can be measured using one data source (from a first characterization modality), and the measurement data can be used to improve or enhance the attenuation correction of a second data source (from a second characterization modality). For example, spectroscopy may be used to assess such ratios, which may be used to provide more accurate attenuation correction of interferometric (eg, OCT) data.

[0076] The following are non-limiting examples of measurements that can be made (e.g., on one or more images) using the methods disclosed herein (e.g., using interferometry and spectroscopy data): stent malapposition, stent length, stent position planning, EEM measurements, EEL measurements, lumen area, plaque volume, lipid measurement(s) (e.g., arc, lipid core volume, maximum lipid core volume over distance, thickness, size, area, volume, lipid to others ratio), calcium measurement(s) (e.g., arc, thickness, size, area, volume, lipid to others ratio), necrotic core cap thickness, plaque instability, lipid core volume, and combinations thereof. In some embodiments, a machine learning algorithm performs the measurements.

[0077] FIG. 1 illustrates a block diagram of a multimodal characterization (e.g., feature detection) system according to an exemplary embodiment of the present disclosure. The system 100 includes a physician display 102, a technician display 104, a tray 106, a computing device 108 including a processor, memory, and one or more machine learning algorithms disclosed herein, a stationary unit 110, a fiber 112, a rotating unit 114, and a probe 116. As illustrated, the system 100 is an imaging catheter system, and the probe 116 is an imaging catheter, e.g., a cardiac catheter. In some embodiments, the multimodal characterization system includes a probe, but is not an imaging catheter. Not all of the fiber 112 (shown in diagonal lines) is shown to illustrate that the fiber 112 can generally be of any suitable length. The stationary unit 110, the rotating unit 114, the fiber 112, and the probe 116 together are operable to perform (e.g., simultaneously) multiple characterization modalities (e.g., during pullback of the probe 116). For example, the stationary unit 110 and / or the rotational unit 114 may include one or more detectors for such purposes.

[0078] In some embodiments, at least one characterization modality in a multi-modality system is an interferometric technique (e.g., OCT). In some embodiments, at least one characterization modality in a multi-modality system is a spectroscopic technique (e.g., NIRS). In general, a system may include one or more characterization modality subsystems (e.g., two different characterization modality subsystems). A characterization modality subsystem may be an interferometric modality (e.g., OCT) subsystem or a spectroscopic modality (e.g., DRS, such as NIRS) subsystem. Different characterization modality subsystems may share components, such as optics and / or fibers. Different characterization modality subsystems may have at least some different components, such as detectors and / or light sources.

[0079] An example of a multi-modality characterization system that may be used, adapted to, or modified to be the multi-modality system disclosed herein is disclosed in International (PCT) Patent Application No. PCT / US22 / 40409, filed August 16, 2022, the disclosure of which is incorporated herein by reference in its entirety. An example of a probe that may be used in the multi-modality characterization system disclosed herein is disclosed in International (PCT) Patent Application No. PCT / US22 / 50460, filed November 18, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0080] Output from the machine learning algorithm(s) (e.g., one or more feature representations of one or more detected features and / or one or more measurements of one or more detected features) may be displayed on the display 102. In some embodiments, the display 102 is used by a user (e.g., a physician) to control the system 100, but not to display, store, and / or transmit the output from the machine learning algorithm(s) elsewhere (e.g., a hospital PACS). For example, a user may be able to select whether the output should be displayed, or simply stored and / or transmitted for future display elsewhere. The output may also be displayed on the display 102, or may be stored and / or transmitted elsewhere. The system 100 may be located in or near a procedure room, such as a catheterization lab.

[0081] A machine learning algorithm as disclosed herein may perform one or more functions including, for example, feature detection, feature transformation (e.g., enhancement) (e.g., of detected features), feature measurement (e.g., of detected features), or a combination thereof. Figures 2-5 show an example method of utilizing a system including a machine learning algorithm. The system may include one or more characterization modality subsystems used to generate sample data and / or may receive sample data from elsewhere (e.g., transmitted by one or more separate systems used for data acquisition).

[0082] FIG. 2 is a flowchart of a method 200 of using a multimodal feature detection system according to an exemplary embodiment of the present disclosure. The method 200 begins at step 202. In optional step 202, the characterization system acquires multimodal data from a characterization modality subsystem. The multimodal data typically includes at least a first sample data from a first characterization modality and a second sample data from a second characterization modality. In step 204, the processor receives the multimodal sample data including the first sample data and the second sample data. In step 206, the first sample data and the second sample data are provided to a machine learning algorithm, including the processor inputting a portion of the sample data to the machine learning algorithm, in this case a machine learning feature detector. Generally, when the sample data is provided to the machine learning algorithm, not all of the data is necessarily input to the machine learning algorithm. In some embodiments, the sample data provided to the machine learning algorithm is first processed (e.g., performed by a feature extractor) and / or pre-processed (e.g., filtered) before actually being input to the machine learning algorithm. Referring back to FIG. 2, in step 208, the processor executes a machine learning feature detector based on input, which may include other data in addition to the first and second sample data. The machine learning feature detector outputs, for example, one or more feature representations of one or more features of interest detected by the algorithm. In step 210, the display displays at least a portion of the detection results, which may be directed against the endoluminal image. For example, the feature representation(s) of the detected feature(s) may be overlaid on an image derived from the original first sample data or the original second sample data. FIGS. 15-25 are further described subsequently and show examples of such overlay representations.

[0083] FIG. 3 is a flowchart of a method 300 of using a multimodal feature enhancement system according to an exemplary embodiment of the present disclosure. In optional step 302, the characterization system obtains multimodal data from a characterization modality subsystem. The multimodal data typically includes at least a first sample data from a first characterization modality and a second sample data from a second characterization modality. In step 304, the processor receives the multimodal sample data including the first sample data and the second sample data. In step 306, the first sample data and the second sample data are provided to a machine learning algorithm, which includes the processor inputting a portion of the sample data to the machine learning algorithm, which is a machine learning feature transformer, in this case a machine learning feature highlighter. In step 308, the processor executes the machine learning enhancement algorithm based on the input, which may include other data in addition to the first sample data and the second sample data. The machine learning feature detector outputs, for example, one or more transformed (in this case enhanced) feature representations of one or more features of interest detected by the algorithm. The machine learning algorithm may first detect the feature(s) of interest before or simultaneously with the enhancement. In step 210, the display displays at least a portion of the enhancement result, which may be directed to the endoluminal image. For example, the enhanced feature representation(s) of the detected feature(s) may be overlaid on an image derived from the original first sample data or the original second sample data.

[0084] FIG. 4 is a flow chart of a method 400 of using a multimodal feature detection and measurement system, according to an exemplary embodiment of the present disclosure. In optional step 402, the characterization system acquires multimodal data from a characterization modality subsystem. The multimodal data typically includes at least a first sample data from a first characterization modality and a second sample data from a second characterization modality. In step 404, the first sample data and the second sample data are (co)registered. In some embodiments, the data are automatically co-registered depending on the way they are acquired. Some characterization systems, for example, certain multimodal intravascular catheter systems, acquire data that is fully co-registered based on simultaneous data acquisition of multiple modalities. In step 406, processing, in this case feature extraction, is performed on the first sample data and the second sample data before inputting them into a machine learning algorithm. The feature extraction in this step may include the use of a machine learning algorithm or may include other known feature extraction techniques. In step 408, the first sample data and the second sample data are provided to a machine learning algorithm, a feature detection and measurement algorithm, and one or more detected feature(s) are output, e.g., in the form of one or more feature representations. In particular, the extracted features from step 406 are fed as input to the algorithm. In step 410, one or more measurements are determined at least in part from the feature representation output. The measurements may be determined by a machine learning algorithm, such as a machine learning feature detector and measurement algorithm. In some embodiments, an assessment is provided based on the measurement(s) made, e.g., in optional step 412. The assessment itself may be a measurement in certain embodiments, e.g., where the assessment is an index such as a plaque volume index.The assessment may be performed in any suitable manner, such as automatically by a processor and / or using a machine learning algorithm (e.g., the machine learning algorithm uses one or more measurements, first sample data from a first characterization modality, second sample data from a second characterization modality, or some combination thereof).

[0085] FIG. 5 is a flow chart of a method 500 of using a multimodal feature detection system according to an exemplary embodiment of the present disclosure. In optional step 502, the characterization system acquires multimodal data from a characterization modality subsystem. The multimodal data typically includes at least a first sample data from a first characterization modality and a second sample data from a second characterization modality. In step 504, the first sample data and the second sample data are (co)registered. In step 506, processing, in this case feature extraction, is performed on the first sample data and the second sample data before inputting them into a machine learning algorithm. The feature extraction in this step may include the use of a machine learning algorithm or may include other known feature extraction techniques. In step 508, the first sample data and the second sample data are provided to a machine learning algorithm, a feature detection and measurement algorithm, and one or more detected feature(s) are output, for example in the form of one or more feature representations. In particular, the extracted features from step 506 are fed as input to the algorithm. In step 510, at least a portion of the detection results are automatically selected (e.g., corresponding to one or more features of interest). The output of the detected feature(s) in step 508 may be in the form of one or more feature representations, and / or the automatic selection in step 510 may include selecting one or more feature representations (e.g., including generating representation(s) from the particular detected feature(s). In optional step 512, the selected feature representation(s) are output on a display.

[0086] In some embodiments, at least two data sets are provided to the machine learning algorithm. For example, at least a first sample data from a first characterization modality and a second sample data from a second characterization modality are provided to the machine learning algorithm. In general, the set of sample data can be a vector that can be in any suitable form, such as a scalar value (i.e., a 1×1 dimensional vector), in any suitable dimension. The data sets may be appended to each other to serve as input to the machine learning feature detector. For example, the machine learning algorithm may accept a single input that includes data from multiple characterization modalities instead of two separate inputs. In some embodiments, the input to the machine learning algorithm has a reduced dimensional input compared to the initial first sample data and / or second sample data, so processing may be required to prepare the first sample data and the second sample data for input to the machine learning algorithm. In some embodiments, appending includes merging two or more data sources into one in any dimension. The sample data may include data from one or more characterization subsystems, such as an interferometry modality subsystem (e.g., OCT) and / or a spectroscopy modality subsystem (e.g., diffuse spectroscopy, e.g., NIRS). The sample data may include interferometry data and / or spectroscopy data.

[0087] The first sample data from the first characterization modality and the second sample data from the second characterization modality may be (co)registered, for example, either automatically by their acquisition method or by post-acquisition processing. For example, as is the case in certain systems, such as multimodal imaging catheter systems (e.g., intraluminal catheter systems), the two characterization modalities may be performed simultaneously. Nevertheless, co-registering sample data from different modalities does not imply that the sample data necessarily correspond to instantaneous simultaneous acquisitions and / or that the sample data necessarily correspond to coextensive sample volumes.

[0088] In some embodiments, the first sample data is generated from a first characterization modality detected in a first region having a first volume (e.g., a first tissue volume in a body cavity), and the second sample data is generated from a second characterization modality detected in a second region having a second volume (e.g., a second tissue volume in a body cavity). The first sample data and the second sample data can then be provided to a machine learning algorithm. The first and second regions can be only a portion of the entire sample volume to be characterized. For example, in an imaging catheter system, sample data acquisition is typically performed at high frequencies (e.g., >1 kHz) to fully image the sample (e.g., an intraluminal volume (e.g., an artery)), and the first and second regions can refer to only a single data acquisition instance, or to multiple ones (e.g., all) of the instances. In some embodiments, the first and second regions do not completely overlap. In some embodiments, the characterization volumes (e.g., intraluminal characterization volumes) of each of the multiple characterization modalities do not completely overlap, and sample data corresponding to each of the characterization volumes is provided to the machine learning algorithm. The different sample regions may result, for example, from different optical properties of the light used for the different characterization modalities (e.g., provided as illumination light and / or detected) with respect to the sample being characterized. For example, OCT and NIRS applied to the same sample using a common probe generally detect light from different sample volumes due to differences in the optical properties associated with the two modalities. In some embodiments, machine learning algorithms operate during data acquisition.

[0089] While the previous paragraph described how the first sample data and the second sample data may correspond to a coextensive region that is not the entire space within the sample, the first sample data and the second sample data may alternatively or additionally correspond to a coextensive period that is not the entire time. For example, in some embodiments, the first sample data is generated from detection by a first characterization modality at time t1, and the second sample data is generated from detection by a second characterization modality at time t2, where t2-t1>0. In some embodiments, t2-t1<5ms (e.g., <3ms, <2ms, or <1ms).

[0090] 6A-6C illustrate certain exemplary data sets, represented by their dimensions, and the inputs to the machine learning algorithms (referred to as "feature detectors" in the figures). FIG. 6A illustrates a method of appending or combining 1-dimensional multimodal sample data (including first sample data from a first characterization modality and second sample data from a second characterization modality) together into a single feature detector for multimodal feature detection, according to an exemplary embodiment of the present disclosure. FIG. 6B illustrates a method of appending or combining 2-dimensional multimodal sample data (including first sample data from a first characterization modality and second sample data from a second characterization modality) into a single feature detector for multimodal feature detection, according to an exemplary embodiment of the present disclosure. FIG. 6C illustrates a method of appending or combining N-dimensional multimodal sample data (including first sample data from a first characterization modality and second sample data from a second characterization modality) into a single feature detector for multimodal feature detection, according to an exemplary embodiment of the present disclosure.

[0091] 7A-11B are block diagrams of an exemplary multimodal characterization system including a machine learning algorithm that performs feature detection, feature representation transformation (e.g., enhancement), measurement (e.g., on the feature representation of a feature of interest), or a combination thereof. FIG. 7A is a block diagram of an exemplary multimodal feature detection system according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes image data and spectroscopy data, and the output of the machine learning algorithm is an image registration prediction. FIG. 7B is a block diagram of an exemplary multimodal feature detection system having a dedicated feature extractor for each sample data source before the machine learning feature detection algorithm, according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes image data (e.g., OCT data) and spectroscopy data, and the output of the machine learning algorithm is an image registration prediction. The feature extractor processes the image sample data and the spectroscopy sample data before they are provided to the machine learning algorithm. FIG. 8 is a block diagram of an exemplary multimodal feature detection system having a dedicated pre-processor for each sample data source before the machine learning feature detection algorithm. In this example, the sample data input includes depth-dependent image data (e.g., OCT data) and reflectance measurement data, and the output of the machine learning algorithm is an image registration prediction. FIG. 9 is a block diagram of a multimodal feature detection system with a dedicated feature extractor for each sample data source prior to the machine learning feature detection algorithm. In this example, the sample data input includes depth-dependent image data (e.g., OCT data) and fluorescence data, and the output of the machine learning algorithm is an image registration prediction. FIG. 10 is a block diagram of a multimodal feature transformation (e.g., enhancer) system according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes image data (e.g., OCT data) and spectroscopy data, and the output of the machine learning algorithm is enhanced spectroscopy data. In some embodiments, the output of the machine learning algorithm is enhanced image data (e.g., enhanced OCT image). FIG. 11A is a block diagram of a multimodal feature measurement system according to an exemplary embodiment of the present disclosure.In this example, the sample data input includes image data (e.g., OCT data) and spectroscopy data, and the output of the machine learning algorithm is a feature measurement. Figure 11B is a block diagram of a multimodal feature detection and measurement system, according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes image data (e.g., OCT data) and spectroscopy data, and the output of the machine learning algorithm is a detected feature and a subsequent measurement of the detected feature.

[0092] FIG. 12 is a block diagram of a multimodal feature detection algorithm according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes image data (e.g., OCT data) and spectroscopy data, and the output of the machine learning algorithm is the detection features (e.g., in the form of a feature representation). As shown, the spectroscopy sample data may be input to different layers 1202 of the multi-stage neural network, including an input layer 1204 (e.g., a convolutional layer), an intermediate stage layer 1206 (e.g., a latent / encoding / downsampling layer), a final stage layer 1208 (e.g., an upsampling layer), or even a loss function 1210, or a combination thereof. This example is not limiting. For example, the spectroscopy sample data may be input to the input layer, and the image data (e.g., OCT data) may be input to the input layer, the intermediate stage layer (e.g., a latent / encoding / downsampling layer), the final stage layer (e.g., an upsampling layer), the loss function, or a combination thereof. As another example, both image data (e.g., OCT data) and spectroscopy data may be input to any one or more of the different stages of a multi-stage machine learning algorithm. In general, data from one or more characterization modalities may be input to any one or more of the different stages of a multi-stage machine learning algorithm.

[0093] FIG. 13A illustrates an exemplary 1D sample data source for a multimodal feature detection algorithm according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent line (e.g., A-line) from an interferometry dataset (e.g., OCT) and a diffuse reflectance spectrum from a spectroscopy dataset (e.g., NIRS). Similarly, the spectroscopy dataset may be integrated over the detector bandwidth and provided by a single scalar value. The interferometry dataset and the spectroscopy dataset may be input to the machine learning algorithm as a single dataset (i.e., appended to each other before input), or may be used as separate inputs. In either case, the interferometry dataset and the spectroscopy dataset are considered to be provided to the machine learning algorithm.

[0094] FIG. 13B illustrates an exemplary 1D sample data source for a multimodal feature detection algorithm, according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes depth-dependent lines (e.g., A-lines) from an interferometry dataset (e.g., OCT) and a fluorescence spectrum from a fluorescence spectroscopy dataset. Similarly, the spectroscopy dataset may be integrated over the detector bandwidth and provided by a single scalar value. The interferometry dataset and the fluorescence dataset may be input to the machine learning algorithm as a single dataset (i.e., appended to each other before input), or may be used as separate inputs. In either case, the interferometry dataset and the fluorescence dataset are considered to be provided to the machine learning algorithm.

[0095] 14A illustrates an exemplary 1D sample data source for a jointly appended multi-modal feature detection algorithm according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes depth-dependent lines (e.g., A-lines) from an interferometry dataset (e.g., OCT) and reflectance spectra from a diffuse reflectance spectroscopy dataset. The dashed vertical lines represent the transitions (append points) between the different sets. Similarly, the spectroscopy dataset may be integrated (e.g., over the detector bandwidth) and provided as a single scalar value (e.g., appended to the A-line scan data from the interferometry dataset).

[0096] FIG. 14B illustrates an exemplary 2D sample data source for a multimodal feature detection algorithm according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an interferometry dataset (e.g., OCT), along with an array of spectra from diffuse reflectance spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the first interferometry dataset. Similarly, data may be input to the detection algorithm in polar space or other coordinate systems. The interferometry dataset and the spectroscopy dataset may be input to the machine learning algorithm as a single dataset (i.e., appended to each other before input), or may be used as separate inputs.

[0097] 15-25 depict potential outputs from a machine learning algorithm illustrating various embodiments of the systems and methods disclosed herein. Different examples include detection features, feature representations, feature transformations, measurements, and combinations thereof. These examples are illustrative and are not intended to be conceptually or literally limiting as to what the output(s) of the machine learning algorithm may look like.

[0098] 15 illustrates an exemplary segmentation output overlaid on and directed against an input 2D arterial OCT image, where the exemplary output has been generated (e.g., predicted) using a pre-trained multimodal feature detection algorithm (an exemplary machine learning algorithm), according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent, rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from the OCT dataset, along with an array of spectra from diffuse reflectance spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and spectroscopy sample data are provided (e.g., as inputs) to the machine learning algorithm (e.g., after being appended to each other). Exemplary detected output from the illustrated machine learning algorithm includes feature representations of the guidewire 1502, the external elastic membrane (EEM) 1504, lipid pools 1506, calcium deposits 1508, catheter sheath location 1510, and the arterial lumen 1512 overlaid on an OCT image of the arterial wall derived from the OCT dataset. The exemplary output may be displayed on a display via a graphical user interface.

[0099] FIG. 16 illustrates an exemplary line-based segmentation output directed against an input 2D arterial OCT image, where an exemplary output has been generated (e.g., predicted) using a pre-trained multimodal feature detection algorithm (an exemplary machine learning algorithm), according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent, rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from the OCT dataset, along with an array of spectra from diffuse reflectance spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and spectroscopy sample data are provided (e.g., after being appended to each other) to the machine learning algorithm (e.g., as inputs). The exemplary detected output from the machine learning algorithm shown includes feature representations of calcium deposits 1602, lipid pools 1604, and no pathology 1606 overlaid against the OCT image of the arterial wall derived from the OCT dataset. In some embodiments, a particular coloring of one or more feature representations may indicate one or more measurements determined using a machine learning algorithm (e.g., the machine learning algorithm may be a feature detection and measurement algorithm). For example, a shade of color (e.g., red and / or yellow) may indicate a measurement associated with a feature of interest represented by the feature representation. For example, different shades of yellow may indicate different coating thicknesses (e.g., based on bucketing values ​​into one of a number of ranges). An exemplary output may be displayed on a display via a graphical user interface.

[0100] FIG. 17 illustrates an exemplary line-based segmentation output directed against an input 2D arterial OCT image, where an exemplary output has been generated (e.g., predicted) using a pre-trained multimodal feature detection algorithm (an exemplary machine learning algorithm), according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent, rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from the OCT dataset, along with an array of spectra from NIRS measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and the spectroscopy sample data are provided (e.g., after being appended to each other) (e.g., as inputs) to the machine learning algorithm. Exemplary detected outputs from the illustrated machine learning algorithm include calcium deposits 1602, lipid pools with specific molecular composition 1604a, lipid pools with molecular composition different from the original 1604b, and no pathology 1606, overlaid against the OCT image of the arterial wall derived from the OCT dataset. The molecular composition may be determined using one or more measurements, for example, using a machine learning algorithm (e.g., the machine learning algorithm may be a feature detection and measurement algorithm). An exemplary output may be displayed on a display via a graphical user interface.

[0101] FIG. 18 illustrates an exemplary bounding box-based segmentation output directed to an input 2D arterial OCT image, where an exemplary output has been generated (e.g., predicted) using a pre-trained multi-modal feature detection algorithm (an exemplary machine learning algorithm) according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent, rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from the OCT dataset, along with integrated reflectance intensity measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. An exemplary detected output from the machine learning algorithm shown includes a feature representation of a calcification 1802 overlaid on an OCT image of the arterial wall derived from the OCT dataset. The exemplary output may be displayed on a display via a graphical user interface.

[0102] FIG. 19 depicts an exemplary output from a pre-trained multimodal feature enhancement algorithm (an exemplary machine learning algorithm that transforms feature representations) according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent, rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. The exemplary enhancement performed by the illustrated machine learning algorithm consists of a contrast-enhanced OCT image ("output"). An OCT image (shown in "input") otherwise derived from the OCT dataset has been contrast-enhanced by the spectroscopy data input (shown in the outer ring of "input"). For example, if the spectroscopy data indicates a feature of interest (e.g., a lipid pool), the presence of that feature in the spectroscopy data may be detected by an algorithm and then automatically transformed (in this case enhanced) the feature representation in an OCT image derived from the OCT data set. In this case, for example, the contrast of the lipid pool feature in the transformed OCT image has been enhanced by a machine learning algorithm based on the spectroscopy data used as part of the input to the algorithm. Exemplary output may be displayed on a display via a graphical user interface.

[0103] FIG. 20 represents an exemplary output from a pre-trained multimodal feature enhancement algorithm (an exemplary machine learning algorithm that transforms feature representations) according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes depth-dependent lines (e.g., A-lines) from an interferometry dataset (e.g., OCT) and a reflectance spectrum from a diffuse reflectance spectroscopy dataset. The exemplary enhancement performed by the illustrated machine learning algorithm consists of enhanced (e.g., contrast, visibility) depth-dependent lines as well as enhanced (e.g., calibrated, scaled, or normalized) reflectance spectra. The OCT sample data and the spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. That is, using the combined input of the interferometry dataset and the reflectance spectrum, the machine learning algorithm can enhance both datasets separately. The exemplary output can be displayed on a display via a graphical user interface. In this case, the transformation (e.g., enhancement) is performed in reduced dimensions compared to the example of FIG. 19.

[0104] FIG. 21 depicts an exemplary output from a pre-trained multimodal feature detection and measurement algorithm (an exemplary machine learning algorithm) according to an exemplary embodiment of the present disclosure. In this example, the sample data input includes a depth-dependent rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from diffuse reflectance spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. An exemplary feature detection output from the machine learning algorithm includes the lumen 2102 and EEM 2104, and an exemplary measurement output from the machine learning algorithm shown includes the plaque volume index (determined to be 70% in this case). The arrow indicates the spacing between the lumen and EEM (e.g., used by the machine learning algorithm in determining the plaque volume index measurement). The exemplary output may be displayed on a display via a graphical user interface.

[0105] FIG. 22 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm (an example machine learning algorithm) according to an example embodiment of the present disclosure. In this example, the example data input includes a depth-dependent rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from a diffuse spectroscopy measurement, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The co-registered OCT sample data and the spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. An example detection output includes a detected lipid pool 2202, and an example measurement output shown includes a lipid pool thickness. A multi-modality input (e.g., of an OCT dataset and a spectroscopy dataset) may produce a more accurate measurement. In this case, the lipid pool thickness may be determined to be greater when considering both the OCT data and the spectroscopy data ("multi-modality input") instead of only the OCT data ("single-modality input"). An exemplary output may be displayed on a display via a graphical user interface.

[0106] FIG. 23 represents an example output from a pre-trained multimodal feature detection and measurement algorithm (an example machine learning algorithm) according to an example embodiment of the present disclosure. In this example, the example data input includes a depth-dependent rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from a diffuse spectroscopy measurement, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The OCT sample data and the spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. An example measurement output from the machine learning algorithm includes a plaque instability (e.g., risk of rupture) index. In this example, no feature representation (transformed or untransformed) is output, only the plaque instability measurement is output. An OCT image derived from the original OCT dataset is shown. The example output may be displayed on a display via a graphical user interface.

[0107] FIG. 24 illustrates an example output from a pre-trained multi-modal feature detection and measurement algorithm (an example machine learning algorithm) according to an example embodiment of the present disclosure. In this example, the example data input includes a depth-dependent rotationally acquired image of the arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from a diffuse spectroscopy measurement, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The OCT sample data and the spectroscopy sample data (e.g., after being appended to each other) are provided (e.g., as inputs) to the machine learning algorithm. An example measurement output from the machine learning algorithm includes an improved automatic stent placement proposal prediction with a longer stentable region (L2>L1) (the larger grey box represents the artery) compared to the single-modality algorithm, for example, due to improved detection of necrotic core plaque size and based on knowledge on which the algorithm is trained that it is undesirable to place a stent edge on the necrotic core. In some embodiments, the visualization of the stent placement proposal may include visual indicators useful for understanding the physical location along the artery. An exemplary output may be displayed on a display via a graphical user interface.

[0108] FIG. 25 illustrates an exemplary display (e.g., graphical user interface(s)) of output from a pre-trained multi-modal feature detection and measurement algorithm (an exemplary machine learning algorithm) according to an exemplary embodiment of the present disclosure. In this example, sample data input may include a depth-dependent, rotationally acquired image of an arterial lumen wall displayed in Cartesian space (e.g., A-line versus rotational position) from an OCT dataset, along with an array of spectra from diffuse spectroscopy measurements, also rotationally acquired from the arterial lumen wall, co-registered to the OCT dataset. The OCT sample data and the spectroscopy sample data are provided (e.g., after being appended to each other) (e.g., as inputs) to the machine learning algorithm. A multi-panel UI 2500 shows an arterial OCT image 2502, a 2D angiography projection 2504, and a longitudinal pullback representation 2506 of multiple 2D imaging positions along the catheter pullback. Exemplary output from the feature detection algorithm includes feature representations detected along the lipid arc 2508 and EEM arc 2510 overlaid on an OCT image derived from the OCT dataset, frame-based lumen 2512, EEM 2514, calcium 2516, lipid 2518, and imaging pullback representation of side branches 2520. Finally, an automatic prediction of a measurement of optimal stent placement location 2522 based on the multi-modal feature detection algorithm.

[0109] The methods disclosed herein can be used for automatic stent placement planning. In some embodiments, EEM cannot be placed because there is too much diseased tissue to place a stent edge. In some embodiments, it is not desirable for the stent to be placed directly on the lipid pool / necrotic core. In some embodiments, the presence of calcium means that the area needs to be prepared before stent placement, but the presence of calcium does not necessarily determine the stent location. Stent placement planning can be improved in some embodiments because the accuracy of EEM measurements, lipid pool / necrotic core measurements, and / or calcium detection can potentially be improved using the methods disclosed herein. Because feature detection can be performed automatically, stent placement planning can also be performed automatically (e.g., based on optimizing one or more measurements using the detected feature(s).

[0110] In some embodiments, the physiological measurements are performed, for example, by machine learning algorithms. Examples of such physiological measurements that may be performed include flow velocity, fractional flow reserve, pressure drop, absolute or relative coronary flow (CF), fractional flow reserve (FFR), instantaneous fractional flow reserve / total cycle resting coronary pressure ratio (iFR / RFR), index of coronary microvascular resistance (IMR), coronary microvascular resistance under maximum hyperemia (HMR), stenosis resistance under maximum hyperemia (HSR), and coronary flow reserve (CFR). Combinations of such physiological measurements may also be performed. In some embodiments, the physiological measurements may be based at least in part on measurements (e.g., measurements of one or more images from an interferometry modality) of geometric measurements and / or image dynamics (e.g., speckle variance). In some embodiments, data from a first characterization modality (e.g., an interferometry modality (e.g., OCT)) may be used to calculate measurements that are physiological measurements, for example, by using the data as input to a machine learning algorithm where the measurements are the output. In some embodiments, the physiological measurements are determined automatically (e.g., by a processor) from data from a first characterization modality (e.g., OCT), e.g., using machine learning algorithms. In some embodiments, data from a second modality [e.g., a spectroscopy modality (e.g., NIRS)] may be used to improve (e.g., more accurately reflect physical reality) the physiological measurements made with data from at least the first characterization modality. Such improvements may be made, for example, by detecting and / or characterizing objects present in human physiology. For example, in some embodiments, the physiological measurements are improved by detecting and / or characterizing objects present in an arterial wall. For example, the physiological measurements may be different when lipid-rich plaque is detected compared to when calcified plaque is detected at that location. The physiological measurements may be affected by properties of the arterial wall (e.g., at the location of the detected plaque) (e.g., wall strength, shear stress, friction). Thus, detecting and incorporating information about one or more objects in an arterial wall may improve the physiological measurements associated with that artery.In some embodiments, data from the two modalities may be used by machine learning algorithms to calculate physiological measurements (e.g., with greater accuracy) that may be improved compared to the same measurements made using sample data from the subject's characterization modality without taking into account the presence and / or properties of objects present in the subject's physiology.

[0111] In some embodiments, data from at least a first characterization modality (e.g., an interferometry modality (e.g., OCT)) is used to make a physiological measurement (e.g., in combination with data from a second characterization modality (e.g., a spectroscopy modality (e.g., NIRS)). Data from at least one of the first and second characterization modalities may be used to improve the measurement. Such data used for improvement may correspond to a feature different from the feature for which the physiological measurement is made. For example, data from the first characterization modality may be used to make a physiological measurement, which may be improved using one or more other measurements and / or information regarding one or more detected features, such as, for example, the location and / or characterization of plaque. The one or more other measurements and / or the one or more detected features may be determined using data (e.g., other data) from the first characterization modality and / or the second characterization modality. Such physiological measurements and / or the one or more other measurements and / or the one or more detected features may be determined using machine learning algorithms disclosed herein.

[0112] Thus, in some embodiments, co-registered data from at least two characterization modalities (e.g., an interferometry modality and a spectroscopy modality (e.g., OCT and NIRS)) are collected during catheter pullback and fed as input to a machine learning algorithm from which physiological measurements are output. The physiological measurements can be improved compared to the same measurements made using only data from the first characterization modality. In some embodiments, data from only one characterization modality is input to a machine learning algorithm trained using training labels derived from a second characterization modality, and physiological measurements (e.g., improved compared to the same measurements made using only data from the first characterization modality) are output. Improvements in physiological measurements, such as, for example, flow rate, pressure drop, resistance, etc., can be achieved by considering plaque location and / or composition and / or vessel (e.g., artery) curvature. Considering plaque can improve the final physiological measurements, since plaque location and / or composition can have an effect on, for example, wall strength and / or wall friction, which affect physiology. The machine learning algorithm can be trained to learn the relationship between physiology (e.g., plaque location and / or composition) and physiological measurements for use in making further physiological measurements using the sample data.

[0113] Using the methods disclosed herein, data from one modality can be used to assess NURD of the catheter, and then the measurement output can be used to correct for NURD in data of a second modality (e.g., image(s)). For example, a machine learning algorithm can take as input a first sample data from a first characterization modality and a second sample data from a second characterization modality. The machine learning algorithm can be trained to assess NURD with respect to the first characterization modality. Of course, this involves using the first sample data, but the first sample data may (or may not) be transformed (e.g., enhanced) by the machine learning algorithm using the second sample data before assessing NURD. The measurement output of the machine learning algorithm can be one or more discrete measurements that reflect the NURD of the catheter. The measurement output can then be used to transform (e.g., enhance) the second sample data to reduce the effect of the NURD that would otherwise be present. This process can all be done automatically as part of a machine learning algorithm, such that the first sample data and the second sample data are provided to the machine learning algorithm (along with any other necessary inputs, such as, for example, characterization of the catheter and / or pullback used to generate the data), and the machine learning algorithm outputs "NURD-corrected" images or feature representation(s) corresponding to one of the characterization modalities based on another of the characterization modalities.

[0114] Using the methods disclosed herein, spectroscopy sensitive markers on a probe sheath (e.g., a catheter) can be detected to improve (e.g., reduce) NURD on one or more images (e.g., OCT image(s)). For example, indicia on the sheath that produce a signal in the spectroscopy channel but are transparent in the OCT data can be detected using the methods disclosed herein. The indicia and / or spectroscopy data can be used to determine NURD and perform correction of the depth-dependent modality images (e.g., OCT images). In some embodiments, a method, such as a feature detection method, is performed prior to catheter pullback where a first sample data from a first characterization modality and a second sample data from a second characterization modality are acquired.

[0115] Bad frames, insufficient blood flushing, contrast injection detection, or combinations thereof may be detected using the methods disclosed herein, for example automatically using a machine learning algorithm. Blood flushing can be detected using the methods and systems disclosed herein to trigger (e.g., automatically) the initiation of probe (e.g., catheter) pullback and / or the initiation of a scan. For example, the first sample data from the first characterization modality and / or the second sample data from the second characterization modality may be provided to a machine learning algorithm trained to determine one or more measurements indicative of bad frames, insufficient blood flushing, contrast injection detection, or combinations thereof. In some embodiments, a simple threshold may be used on one or more measurements determined by the machine learning algorithm to determine bad frames, insufficient blood flushing, contrast injection detection, or combinations thereof.

[0116] The methods disclosed herein can be used to detect optical probe breakage or penetration failure. For example, the first sample data from the first characterization modality and / or the second sample data from the second characterization modality can be provided to a machine learning algorithm trained to determine one or more measurements indicative of optical probe breakage and / or penetration failure. The machine learning algorithm can output a (e.g., automatic) determination of optical probe breakage and / or penetration failure and / or one or more measurements for review by a user to assess whether optical probe breakage and / or penetration failure is or has occurred.

[0117] In some embodiments, one modality is used to dynamically measure the transfer function of the system or catheter and use that data to correct or improve the quality of a second modality. In some embodiments, one modality is used to identify structures or defects in the catheter and use that detection to unmask valid / invalid regions of a second modality based on the identified structures or defects.

[0118] "Imaged" refers to the detection of light (e.g., at a given spatial location). For example, detecting light from a specific location on a sample using a waveguide is referred to as "imaging" the sample at that location. Imaging may be performed using depth-dependent imaging, such as, for example, OCT, ultrasound, or variable confocal imaging. Spectroscopy modalities may be DRS, fluorescence, autofluorescence, spontaneous Raman spectroscopy, coherent Raman spectroscopy, hyperspectral imaging, or point measurements at specified wavelengths. Characterization modalities may use, for example, spectrally separated detectors or wide bandwidth integration.

[0119] Various embodiments of the present disclosure may use data from any suitable characterization modality. Generally, data from at least two characterization modalities are used, but in some embodiments, data from more than two characterization modalities may be used. The characterization modality (e.g., the first and / or second one / more modalities) may be a spatially resolved imaging modality, such as, for example, a depth-dependent imaging modality. In some embodiments, the characterization modality is OCT, ultrasound, or variable confocal imaging, and combinations thereof may be used, but are generally at least partially redundant. The characterization modality may be a tomography modality. The characterization modality may be a microscopy modality. The characterization modality may be an interferometry modality. The characterization modality may be a spectroscopy modality. The spectroscopy modality may be a diffuse spectroscopy modality. For example, the characterization modality may be DRS, fluorescence, autofluorescence, spontaneous Raman spectroscopy, coherent Raman spectroscopy, hyperspectral imaging, or point measurements at a specified wavelength. The characterization modality may be an intraluminal characterization modality suitable for characterizing the lumen of a subject. The characterization modality may be a vascular characterization modality suitable for characterizing the structure of the vasculature of a subject. In certain embodiments, data from the depth-dependent imaging modality and the spectroscopy modality are used together (e.g., provided to a machine learning algorithm).

[0120] The term "image" includes any visual representation, e.g., a photograph, a video frame, a streaming video, as well as any electronic, digital, or mathematical analog of a photograph, a video frame, or a streaming video, such as, for example, a two-dimensional or three-dimensional image of a sample. Thus, an "image" may refer to data that may be displayed, but is not necessarily displayed. Any system or device described herein, in certain embodiments, includes a display for displaying an image or any other result generated by the processor. Any method described herein, in some embodiments, includes a step of displaying an image or any other result generated by the method. Any system or device described herein, in certain embodiments, outputs an image to a remote receiving device (e.g., a cloud server, a remote monitor, or a hospital information system (e.g., a picture archiving and communication system (PACS))) or to an external storage device that can be connected to the system or device. In some embodiments, the image is generated using a multimodal catheter (e.g., including an interferometric imaging modality and a spectroscopic imaging modality). In some embodiments, the image is a two-dimensional (2D) image. In some embodiments, the image is a three-dimensional (3D) image. In some embodiments, the image is a reconstructed image. The image (eg, a 3D image) can be a single image or a set of images.

[0121] Tissue volume refers to the volume of tissue seen by the detection waveguide (e.g., by its aperture number). Depending on the optical geometry, tissue volume can vary based on the distance of the probe (e.g., catheter) to the sample. The probe may be a catheter, e.g., a cardiac catheter, etc.

[0122] Exemplary embodiments of the systems and methods disclosed herein have been described above with reference to computations performed locally by a computing device. However, computations performed over a network are also contemplated. FIG. 26 illustrates an exemplary network environment 2600 for use with the methods and systems described herein. In a brief overview, a block diagram of an exemplary cloud computing environment 2600 is shown and described with reference to FIG. 26 now. The cloud computing environment 2600 may include one or more resource providers 2602a, 2602b, 2602c (collectively 2602). Each resource provider 2602 may include computing resources. In some implementations, the computing resources may include any hardware and / or software used to process data. For example, the computing resources may include hardware and / or software capable of executing algorithms, computer programs, and / or computer applications. In some implementations, exemplary computing resources may include application servers and / or databases with storage and retrieval capabilities. Each resource provider 2602 may be connected to any other resource provider 2602 in the cloud computing environment 2600. In one implementation, the resource providers 2602 may be connected through a computer network 2608. Each resource provider 2602 may be connected through the computer network 2608 to one or more computing devices 2604a, 2604b, 2604c (collectively 2604).

[0123] The cloud computing environment 2600 may include a resource manager 2606. The resource manager 2606 may be connected to the resource providers 2602 and the computing devices 2604 through a computer network 2608. In an embodiment, the resource manager 2606 may facilitate the provision of computing resources by one or more resource providers 2602 to one or more computing devices 2604. The resource manager 2606 may receive a request for a computing resource from a particular computing device 2604. The resource manager 2606 may identify one or more resource providers 2602 that have the capability to provide the computing resource requested by the computing device 2604. The resource manager 2606 may select a resource provider 2602 that provides the computing resource. The resource manager 2606 may facilitate a connection between the resource provider 2602 and a particular computing device 2604. In an embodiment, the resource manager 2606 may establish a connection between a particular resource provider 2602 and a particular computing device 2604. In one implementation, the resource manager 2606 may redirect a particular computing device 2604 to a particular resource provider 2602 that has the requested computing resource.

[0124] 27 illustrates an example of a computing device 2700 and a mobile computing device 2750 that can be used in the methods and systems described in this disclosure. The computing device 2700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The mobile computing device 2750 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components illustrated, their connections and relationships, and their functions are meant to be exemplary only and not limiting.

[0125] The computing device 2700 comprises a processor 2702, a memory 2704, a storage device 2706, a high-speed interface 2708 connecting to the memory 2704 and multiple high-speed expansion ports 2710, and a low-speed interface 2712 connecting to the low-speed expansion port 2714 and the storage device 2706. Each of the processor 2702, the memory 2704, the storage device 2706, the high-speed interface 2708, the high-speed expansion port 2710, and the low-speed interface 2712 are interconnected by various buses, which may be mounted on a common motherboard or in other formats as appropriate. The processor 2702 is capable of processing instructions for execution within the computing device 2700. The instructions include instructions stored in the memory 2704 or the storage device 2706 that display graphical information for a GUI on an external input / output device (such as, for example, a display 2716 connected to the high-speed interface 2708). In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memories. Also, multiple computing devices may be connected (e.g., as a server bank, group of blade servers, or multiprocessor system) with each device providing a portion of the required computations. Also, multiple computing devices may be connected (e.g., as a server bank, group of blade servers, or multiprocessor system) with each device providing a portion of the required computations. Thus, as the term is used herein, when functions are described as being performed by a "processor," this encompasses embodiments in which the functions are performed by any number of processors (e.g., one or more processors) of any number of computing devices (e.g., one or more computing devices). Furthermore, when functions are described as being performed by a "processor," this encompasses embodiments in which the functions are performed by any number of processors (e.g., one or more processors) of any number of computing devices (e.g., one or more computing devices) (e.g., in a distributed computing system).

[0126] The memory 2704 stores information within the computing device 2700. In some implementations, the memory 2704 is a volatile memory unit or units. In some implementations, the memory 2704 is a non-volatile memory unit or units. The memory 2704 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0127] The storage device 2706 has the capability of mass storage for the computing device 2700. In an embodiment, the storage device 2706 may be or may include a computer-readable medium. The medium may be, for example, a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, an array of devices including devices in a storage area network or other configuration, and the like. The instructions may be stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., the processor 2702), perform one or more methods, such as those described above. The instructions may also be stored in one or more storage devices (e.g., the memory 2704, the storage device 2706, or the memory on the processor 2702), such as a computer-readable medium or a machine-readable medium.

[0128] The high-speed interface 2708 manages bandwidth-intensive operations for the computing device 2700, while the low-speed interface 2712 manages less bandwidth-intensive operations. Such an allocation of functions is merely an example. In one embodiment, the high-speed interface 2708 is connected to the memory 2704, the display 2716 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 2710 that may accept various expansion cards (not shown). In this embodiment, the low-speed interface 2712 is connected to the storage device 2706 and the low-speed expansion port 2714. The low-speed expansion port 2714, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected, for example, through a network adapter to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0129] Computing device 2700 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 2720, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer, such as a laptop computer 2722. It may also be implemented as part of a rack server system 2724. Alternatively, components from computing device 2700 may be combined with other components in a mobile device (not shown), such as mobile computing device 2750. Each such device may include one or more of computing device 2700 and mobile computing device 2750, and the entire system may be composed of multiple computing devices in communication with each other.

[0130] The mobile computing device 2750 includes, among other components, a processor 2752, a memory 2764, input / output devices such as a display 2754, a communication interface 2766, and a transceiver 2768. The mobile computing device 2750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor 2752, memory 2764, display 2754, communication interface 2766, and transceiver 2768 are interconnected by various buses, and some of the components may be mounted on a common motherboard or in other forms, as appropriate.

[0131] The processor 2752 can execute instructions within the mobile computing device 2750, including instructions stored in the memory 2764. The processor 2752 may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor 2752 may be responsible for coordinating other components of the mobile computing device 2750, such as controlling a user interface, applications run by the mobile computing device 2750, and wireless communication by the mobile computing device 2750.

[0132] The processor 2752 may communicate with a user through a control interface 2758 and a display interface 2756 connected to a display 2754. The display 2754 may be, for example, a TFT display (thin film-transistor liquid crystal display) or an OLED (organic light-emitting diode) display, or other suitable display technology. The display interface 2756 may include appropriate circuitry for driving the display 2754 to provide graphical and other information to the user. The control interface 2758 may receive commands from a user and convert them for sending to the processor 2752. In addition, an external interface 2762 may enable communication with the processor 2752 to enable short-range communication of the mobile computing device 2750 with other devices. The external interface 2762 may provide, for example, wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may also be used.

[0133] The memory 2764 stores information within the mobile computing device 2750. The memory 2764 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 2774 may also be provided, which may be connected to the mobile computing device 2750 through an expansion interface 2772, which may include, for example, a SIM (single in-line memory module) card interface. The expansion memory 2774 may provide additional storage space for the mobile computing device 2750 or may also store applications or other information for the mobile computing device 2750. In particular, the expansion memory 2774 may include instructions that perform or complement the processes described above, and may also include secure information. Thus, for example, the expansion memory 2774 may be provided as a security module for the mobile computing device 2750 and may be programmed with instructions that enable secure use of the mobile computing device 2750. Additionally, secure applications may be provided via the SIM card along with additional information, such as placing identifying information on the SIM card in a manner that cannot be hacked.

[0134] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as described below. In some embodiments, the instructions are stored on an information carrier and, when executed by one or more processing devices (e.g., processor 2752), perform one or more methods, such as those described above. The instructions may also be stored in one or more storage devices, such as one or more computer-readable or machine-readable media (e.g., memory 2764, expansion memory 2774, or memory on processor 2752). In some embodiments, the instructions may be received by propagated signals through transceiver 2768 or external interface 2762.

[0135] The mobile computing device 2750 may communicate wirelessly through a communication interface 2766, which may include digital signal processing circuitry, if necessary. The communication interface 2766 may support communication under various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, by radio frequencies through a transceiver 2768. Additionally, short-range communication may occur, such as by Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 2770 can transmit additional navigational and location-related radio data to the mobile computing device 2750, such data can be used by applications running on the mobile computing device 2750 as desired.

[0136] The mobile computing device 2750 may also communicate audibly using an audio codec 2760, which may receive spoken information from a user and convert it into usable digital information. The audio codec 2760 may also generate audible sounds for the user, such as through a speaker in a handset of the mobile computing device 2750. Such sounds may include sounds from voice telephone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on the mobile computing device 2750.

[0137] The mobile computing device 2750 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a mobile phone 2780. It may also be implemented as part of a smartphone 2782, a personal digital assistant, or other similar mobile device.

[0138] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0139] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0140] To provide user interaction, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be acoustic, verbal, or tactile input.

[0141] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., a data server, etc.), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0143] The systems and methods described herein are generally described with specific examples of use for endoluminal / vascular characterization. However, the disclosure is not limited to those specific examples, and other applications are also contemplated. In general, the multimodal characterization systems and / or methods disclosed herein may be used to characterize any sample desired. In some embodiments, the sample is in vivo. In some embodiments, the sample is in vitro. In some embodiments, the sample is a part of a living subject, such as a human (e.g., in vivo or excised). In some embodiments, the sample is not living. For example, various pipes, tunnels, tubes, and the like may be characterized using the systems and methods disclosed herein. In some embodiments, the first sample data from a first characterization modality and the second sample data from a second characterization modality are vascular data and / or endoluminal data. For example, OCT data and diffuse spectroscopy data from a cardiac imaging catheter system used to characterize a patient's arteries are vascular endoluminal data. The characterization modalities may be used to acquire data in any suitable manner, for example, the catheter system typically (but not necessarily) acquires data from the characterization modality during pullback. The data may be obtained prior to the pullback (eg, to determine when the pullback should be initiated (eg, automatically)).

[0144] At least some of the methods, systems, and techniques described herein may be controlled by executing instructions stored on one or more non-transitory machine-readable storage media on one or more processing devices. Examples of non-transitory machine-readable storage media include read-only memory, optical disk drives, memory disk drives, and random access memories. At least some of the methods, systems, and techniques described herein may be controlled using a computing system that includes one or more processing devices and a memory that stores instructions executable by the one or more processing devices to perform various control operations.

[0145] In this application, unless otherwise clear from the context or otherwise stated, (i) the term "a" may be understood to mean "at least one," (ii) the term "or" may be understood to mean "and / or," and (iii) the terms "comprising" and "including" may be understood to encompass the listed components or steps, whether presented alone or together with one or more additional components or steps.

[0146] At least some of the methods, systems, and techniques described herein may be controlled by executing instructions stored on one or more non-transitory machine-readable storage media on one or more processing devices. Examples of non-transitory machine-readable storage media include read-only memory, optical disk drives, memory disk drives, and random access memories. At least some of the methods, systems, and techniques described herein may be controlled using a computing system that includes one or more processing devices and a memory that stores instructions executable by the one or more processing devices to perform various control operations.

[0147] It is contemplated that the systems, devices, methods, and processes of the present disclosure encompass variations and modifications developed using information from the embodiments described herein. Modifications and / or variations of the systems, devices, methods, and processes described herein may be made by those skilled in the art.

[0148] Throughout the description, when articles, devices, and systems are described as having, including, or comprising particular components, or processes and methods are described as having, including, or comprising particular steps, it is believed that in addition there are articles, devices, and systems according to particular embodiments of the disclosure that consist essentially of or consist of the recited components, and there are processes and methods according to particular embodiments of the disclosure that consist essentially of or consist of the recited processing steps.

[0149] It should be understood that the order of steps or the order for performing certain actions is not important so long as operability is not lost. Furthermore, two or more steps or actions may be performed simultaneously. As will be appreciated by those skilled in the art, the terms "on," "under," "up," "down," "below," and "on" are relative terms and may be interchanged to refer to different orientations of layers, elements, and substrates included in the present disclosure. For example, in some embodiments, a first layer of a second layer means that the first layer is directly on and in contact with the second layer. In other embodiments, the first layer of a second layer may include another layer therebetween.

[0150] Specific embodiments of the present disclosure have been described above. However, it should be specifically noted that the present disclosure is not limited to these embodiments, but rather additions and modifications to those explicitly described in the present disclosure are also intended to be included within the scope of the present disclosure. Furthermore, it should be understood that the features of the various embodiments described in the present disclosure are not mutually exclusive and may exist in various combinations and permutations, which may be possible without departing from the spirit and scope of the present disclosure even if such combinations or permutations are not expressed. Although the present disclosure has been described in detail with particular reference to specific embodiments thereof, it will be understood that variations and modifications may occur within the spirit and scope of the invention as claimed.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, the processor receiving first sample data from a first characterization modality and second sample data from a second characterization modality. causing the processor to perform The instructions, when executed, the processor detecting features of interest by providing the first sample data and the second sample data to a machine learning algorithm; the processor outputting a feature representation of the feature of the object of interest directed to the first characterization modality; 20. A non-transitory computer-readable medium, further comprising:

2. The non-transitory computer-readable medium of claim 1 , wherein the machine learning algorithm is trained to detect two or more features of interest.

3. The non-transitory computer-readable medium of claim 1 , wherein the first sample data and the second sample data are both from an endoluminal characterization modality.

4. The non-transitory computer-readable medium of claim 1 , wherein the first characterization modality is an interferometry modality and the second characterization modality is an intensity measurement.

5. The non-transitory computer-readable medium of claim 1 , wherein the first characterization modality is a depth-dependent imaging modality and the second characterization modality is a wavelength-dependent measurement modality.

6. The non-transitory computer-readable medium of claim 1 , wherein one or both of the first characterization modality and the second characterization modality are processed before being input to the machine learning algorithm.

7. 2. The non-transitory computer-readable medium of claim 1, wherein one or both of the first characterization modality and the second characterization modality are enrolled prior to being input to the machine learning algorithm.

8. The processor: registering one or both of the first sample data and the second sample data before inputting the first sample data and the second sample data into the machine learning algorithm; inputting the first sample data and the second sample data into one or more feature extractors; generating an output from the one or more feature extractors, wherein the detecting includes inputting the output from the one or more feature extractors into the machine learning algorithm; The non-transitory computer-readable medium of claim 1 , further configured to:

9. 10. The non-transitory computer-readable medium of claim 1, wherein the machine learning algorithm is trained to detect features using labels from either the first characterization modality or the second characterization modality only.

10. The non-transitory computer-readable medium of claim 1 , wherein the machine learning algorithm outputs detected features with reference to the first characterization modality.

11. 2. The non-transitory computer-readable medium of claim 1, wherein the first sample data is generated from the first characterization modality detected in a first region having a first tissue volume within a body cavity, and the second sample data is generated from the second characterization modality detected in a second region having a second volume within the body cavity.

12. The first sample data is obtained at time t １ and the second sample data is generated from detection by the first characterization modality at time t ２ generated from detection by the second characterization modality at t ２ -t １ 10. The non-transitory computer-readable medium of claim 1, wherein the timeout period is < 1 ms.

13. 2. The non-transitory computer-readable medium of claim 1, wherein the first sample data and the second sample data are combined into combined sample data, and the combined sample data is input to the machine learning algorithm during the detecting step.

14. The non-transitory computer-readable medium of claim 1 , wherein the combining of the first sample data and the second sample data comprises appending the first sample data to the second sample data.

15. The non-transitory computer-readable medium of claim 14 , wherein the appending comprises merging the first sample data and the second sample data together.

16. The non-transitory computer-readable medium of claim 1 , wherein the machine learning algorithm has multiple stages into which data can be input.

17. 17. The non-transitory computer-readable medium of claim 16, wherein information from the first characterization modality and from the second characterization modality are input as two unique inputs to the machine learning algorithm at different stages.

18. 20. The non-transitory computer-readable medium of claim 17, wherein the first sample data and the second sample data are input separately to the machine learning algorithm at different stages of the plurality of stages.

19. The non-transitory computer-readable medium of claim 1 , wherein each sample data undergoes feature extraction, and an output of the feature extraction is used as an input to the machine learning algorithm.

Images