System and method for classification of arterial image regions and their characteristics

Abstract

The present disclosure relates in part to methods and systems suitable for evaluating image data from patients in real time or approximately real time using machine learning (ML). Systems and methods using machine learning techniques to improve diagnostic tools for end users (e.g., cardiologists and imaging specialists) are applied to specific problems associated with intravascular images with polar coordinate representations. Further, if OCT, IVUS image data or other imaging data is obtained using a rotating probe, processing two coordinate systems associated therewith poses a challenge. The present disclosure focuses on these and various other challenges associated with solving the problem of rapid imaging and diagnosis of patients so that stent placement and other procedures can be performed in a single session in the catheterization laboratory.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 791,876 (filed on January 13, 2019), the entire disclosure of which is hereby incorporated by reference. Technical Field

[0003] The present application relates in part to imaging arteries and segmenting and characterizing their component parts. In particular, in some embodiments, the present application relates to machine learning applications to characterize and / or classify arterial tissue and associated arterial regions and features of interest. Background Art

[0004] Optical coherence tomography (OCT) is an interferometric imaging technique with wide applications in ophthalmology, cardiology, gastroenterology and other medical fields. The ability to view subsurface structures with high resolution through a small diameter fiber optic probe makes OCT particularly useful for minimally invasive imaging of internal tissues and organs. OCT systems can generate up to 100 frames per second, making it possible to image coronary arteries in a beating heart artery within seconds. OCT can be implemented in both the time domain (TD-OCT) and the frequency domain (Fourier domain OCT or optical frequency domain imaging, OFDI). OCT can be used in combination with various other imaging techniques, such as intravascular ultrasound (IVUS), angiography, fluoroscopy, X-ray-based imaging systems, and other imaging techniques.

[0005] OCT imaging of a patient's body parts provides physicians with a useful tool to determine the best type and route of treatment. For example, imaging of coronary arteries with intravascular OCT can reveal the location of stenosis, the presence of vulnerable plaques, or the type of atherosclerotic plaque. This information helps cardiologists choose which treatment will be most effective for the patient - drug therapy (such as cholesterol-lowering medications), catheter-based therapies (such as angioplasty and stent placement), or invasive surgical procedures (such as coronary artery bypass surgery). In addition to its use in clinical medicine, OCT is also useful for drug development in animals and clinical trials.

[0006] Normal arteries have a consistent layer structure, consisting of an intima, a media, and an adventitia. As a result of the atherosclerotic process, the intima becomes pathologically thickened and may contain plaques, which are composed of different types of tissues, including fibers, proteoglycans, lipids, and calcium, as well as macrophages and other inflammatory cells. These tissue types have different optical properties, which can be measured by manual measurement and imaging techniques. It is believed that the most pathologically important plaque is the so-called vulnerable plaque, which has a fibrous cap and a lipid pool in the lower layer.

[0007] In a typical OCT imaging system, an optical probe mounted on a catheter is carefully maneuvered to the site of interest, such as within a coronary vessel. A beam of light is then emitted, and backscattered signals are received by interferometric detection using an interferometer. As the probe is swept across a predetermined line or area, many lines of data can be collected. An image (2D or 3D) is then reconstructed using known techniques. This image is then visually analyzed by a cardiologist to assess pathological features, such as vessel wall thickening and plaque composition.

[0008] Since tissue types are identified by their appearance on the screen, errors may occur during analysis because certain information (e.g., tissue type) is not easily recognizable. Various other components or areas of interest regarding a given patient's artery, organ, or other body part are difficult to accurately classify by visual inspection, a situation that occurs in almost all branches of medicine. Therefore, there is a need for a system and method for detecting various objects in image data and presenting them to an end user. The present disclosure addresses these and other challenges. Summary of the invention

[0009] The system of one or more computers may be configured to perform specific operations or actions using software, firmware, hardware, or a combination thereof installed on the system, which when operated causes the system to perform actions. One or more computer programs may be configured to perform specific operations or actions using instructions contained therein, which when executed by a data processing device causes the device to perform actions. One general aspect includes a method of evaluating coronary arteries using one or more machine learning systems.

[0010] In one embodiment, the present disclosure relates to various method steps. The method may include: acquiring a series of image data including polar coordinate image frames; annotating one or more regions of interest or features in each polar coordinate image of the series of images, so that each annotated region or feature is a ground truth annotation; training a neural network of a machine learning system using the series of annotated polar coordinate images, wherein each of a plurality of regions in each polar coordinate region is identified by a category; inputting the polar coordinate image data into the trained neural network; and displaying a predictive output image, wherein the predictive output image includes color-coded regions, wherein each color corresponds to a category.

[0011] In one embodiment, the system comprises: one or more AI processors (with associated memory), wherein one or more trained software-based neural networks are executed on the one or more AI processors. The machine learning system may include a convolutional neural network. The method may include: acquiring a series of image data, such as arterial image data. In one embodiment, the image data includes intravascular image data. In one embodiment, the image data includes polar coordinate images. The method may include: annotating one or more regions of interest or features in each polar coordinate image of the series of images, such that each annotated region or feature is a ground truth annotation. The ground truth annotations may be stored in the memory as a series of ground truth masks. The method may include: training a neural network of the machine learning system using a series of annotated polar coordinate images (e.g., via the series of ground truth masks). In one embodiment, a ground truth mask includes a region of interest corresponding to a particular feature or channel.

[0012] Thus, a first region of the ground truth mask may correspond to a first feature and a second region of the ground truth mask may correspond to a second feature. The above may apply to M features and P regions, where each feature corresponds to one or more regions. As an example, two or more regions of calcium, a region of lumen, and a region of intima may be part of one ground truth mask, and its classification by category or type would be calcium, lumen, and intima, where each region is so classified / segmented. In one embodiment, the ground truth mask is used to train a neural network to detect / predict which region in input image data from a user corresponds to a particular feature or channel. In one embodiment, the method includes: inputting image data (e.g., polar coordinate image data and / or ground truth mask) to the neural network to train the neural network. The neural network is trained in one or more full cycles until it is operable to detect multiple features / channels within an error threshold. In one embodiment, the method includes: inputting image data (e.g., polar coordinate image data) to the trained neural network and displaying a predictive output image from a machine learning system. In one embodiment, the ground truth image mask is generated in response to annotating a graphical representation of an image in a user interface. In various embodiments, the polar coordinate image is annotated in such a user interface. The neural network is implemented in one or more software applications, such as PyTorch, LibTorch, and other applications disclosed herein. Other embodiments of this aspect include: corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0013] Embodiments may include one or more of the following processes or steps. In one embodiment, each image includes: a plurality of image data elements regarding a coronary artery. In one embodiment, annotation is performed via a graphical user interface, the graphical user interface including a user control to select groups of pixels or two-dimensional boundaries for defining features of interest. In one embodiment, the neural network is repeatedly trained until the output of a cost function is at or below a threshold, wherein the cost function compares the predictive output of the MLS with a ground truth input. In various embodiments, a cross entropy evaluation is used to measure the measurement error associated with the cost function. The method may further include: classifying one or more regions of interest or features of each polar coordinate image into a type or category. In one embodiment, the type or category is selected from the group consisting of: intima, media, adventitia, lumen, external elastic layer (EEL), internal elastic layer (IEL), plaque, calcium, calcium plaque. In one embodiment, image data for the systems and methods disclosed herein include: carpet view images, scan lines, pixels, 2D images, 3D images, angiographic images, intravascular images, CT scan images, X-ray images, and other images of arteries, veins, organs, or other components of the circulatory system. The aforementioned features, regions, channels, categories, etc. can be detected using a neural network trained thereon.

[0014] In one embodiment, the features, areas, types and / or categories include one or more side branches, lumens, guidewires, stent ribs, stents, imprisoned stents, bioresorbable vascular scaffolds (BVS), drug eluting stents (DES), halo artifacts, pressure wires, guidewires, lipids, atherosclerotic plaques, stenosis, calcium, calcified plaques, calcium-containing tissue, lesions, fat, poorly placed stents; under-expanded stents; over-expanded stents; radio-opaque markers; arterial tree branching angles; probe calibration elements, doped membranes; light scattering particles, sheaths; doped sheaths; fiducial alignment points, diameter measurements, calcium arc measurements, area of ​​interest or feature thickness, radial measurements, guide catheters, shadow areas, guidewire segments, lengths, thicknesses, and other parameters as disclosed herein.

[0015] In one embodiment, each data element, image, and output is stored in a machine readable memory that is in electronic communication with the machine learning system. In one embodiment, the series of annotated polar images includes: images containing one or more imaging artifacts or undesirable imaging conditions. In one embodiment, the one or more imaging artifacts or undesirable imaging conditions are selected from the group consisting of: incomplete clearing of arteries prior to intravascular imaging; insufficient contrast; insufficient contrast solution; light intensity below average for intravascular imaging; contrast clouding; non-uniform rotation distortion (NURD); halo artifacts; imprisoned side branches; reflections from imaging probe components. In one embodiment, the methods and systems disclosed herein are operable to perform or execute: identifying one or more arc-based measures, measurements of similarity of Ca and EEL in the predictive output image; detected EEL diameter; detected Ca depth. In one embodiment, the neural network is a convolutional neural network, wherein the number of input channels for the first node or layer is four channels. In one embodiment, the methods and systems disclosed herein are operable to perform or execute: generating a carpet view using line projection, and filtering the carpet view to reduce noise in the predictive output image.

[0016] In one embodiment, each of the foregoing (and other examples disclosed herein with respect to identifiable elements in input image data) is a data channel that can be used as a region of interest (ROI) or feature of interest (FOI) to train an MLS and can be detected by the trained MLS. In one embodiment, each of the foregoing has an associated mask or data channel, or is an element in an image mask (e.g., a ground truth mask or an output mask). In one embodiment, the output mask includes a plurality of regions, wherein different regions correspond to different channels, such that a multi-channel segmentation representation is generated with respect to the input data. In one embodiment, a first frame of image data is processed by an MLS neural network to generate a first output mask corresponding to the first frame of image data, wherein the first mask is modified such that regions / features of interest are identified by markings (e.g., color coding, hatching, or other markings). Thus, the first output mask includes: input image data with overlays, changes (to identify their category / type) to the image data or mask regions, or other markings for pixel regions that are classified as particular features, categories, etc.

[0017] In one embodiment, the predictive output image includes: one or more markers that indicate the boundaries of the features being predicted or classified. The method further includes: converting the predictive output image from polar form to Cartesian form. In this way, the output polar image mask (which has markers corresponding to the detection ROI / FOI) can be converted to a Cartesian image, wherein the markers are converted and represented in the Cartesian image. In various embodiments, annotating and operating on Cartesian images to generate ground truth images / masks and train neural networks is avoided, and instead the ground truth polar images are operated on and used to train a given neural network. In one embodiment, a carpet view mask and a subset / subset thereof can be used. In one embodiment, the carpet view has a first axis corresponding to the number of frames and a second axis corresponding to the number of scan lines.

[0018] In one embodiment, the neural network is a conformal neural network. In one embodiment, the MLS includes: an artificial intelligence (AI) processor, wherein the AI ​​processor includes one or more parallel processing elements. In one embodiment, the AI ​​processor includes N parallel processing elements and further includes dedicated AI processor memory. In one embodiment, the dedicated AI processor memory ranges from about 8GB to about 64GB. In one embodiment, the dedicated AI processor memory ranges from about 64GB to about 128GB. In one embodiment, the AI ​​processor is a graphics processing unit. In one embodiment, the parallel processing element is selected from the group consisting of: a CUDA core processor, a core processor, a tensor core processor, a stream processor. In one embodiment, the AI ​​processor runs locally via an edge network facility or server. In one embodiment, an AI processor (e.g., a graphics processing unit) is used, which includes 8GB or more of dedicated memory combined with 32GB or more of self-loaded RAM (as part of a computing device disposed within the housing of the data collection / imaging system).

[0019] In one embodiment, the method further includes: when classifying user image data, reducing MLS processing time by flattening the image data before inputting into the neural network. This can be applied in the training phase and when classifying, detecting and / or identifying features / regions of interest with respect to patient image data. The method further includes: when classifying user image data, reducing MLS processing time by resizing or excluding regions before inputting into the neural network. The method further includes: for one or more of the polar coordinate images, performing 1, 2 or 3 circular rotations. The method further includes: for one or more of the polar coordinate images, performing left-right flipping. The method further includes: performing lumen detection using image processing methods or machine learning methods to generate a series of detected lumen boundary data. The method further includes: generating one or more image masks for each region of interest or feature identified in the patient image. Implementations of the technology may include: hardware, methods or processes or computer software on a computer accessible medium. The method and system may include various bus and interface components.

[0020] One general aspect includes: performing lumen detection to detect lumen boundaries. In one embodiment, the detected lumen boundary data (e.g., on a per-image basis) is also input into a neural network along with the patient polar coordinate image data. Other embodiments of this aspect include: corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0021] Embodiments may include one or more of the following features. In one embodiment, inputting detected lumen boundary data reduces latency for classifying regions of interest and features in patient polar coordinate image data. Embodiments of the technology may include: hardware, methods or processes, or computer software on a computer accessible medium. In one embodiment, each input image used to train or process patient data is converted into multiple versions, wherein the multiple versions are generated by left-right flipping and circular rotation. This provides an enhanced data set, which in turn reduces error and increases the accuracy of the neural network / MLS. In various embodiments, reference to MLS also includes neural networks and vice versa.

[0022] A general aspect includes: a data collection and / or imaging and region / feature characterization system. The system also includes a housing. The system also includes: a frame grabber for receiving one or more of the image data, such as polar coordinate data, ultrasound data, optical image data, X-ray image data, and intravascular image data. The intravascular system also includes a power supply. The intravascular system also includes: one or more electronic memory storage devices, which are electrically connected to the power supply. The intravascular system also includes: one or more image processing software modules, which can be executed on the processor and stored in the one or more electronic memory storage devices. The intravascular system also includes: a computing device, which includes a first processor, and the computing device is electronically connected to the power supply and the first processor. In one embodiment, one or more AI processors and dedicated AI processor memory are disposed in the housing or connected to the housing via one or more ports, buses or networks. In one embodiment, the MLS and its trained neural network are remotely operated, such as through a client / server implementation, an edge computing implementation, or a cloud or software as a service implementation.

[0023] In one embodiment, the system further comprises one or more software programs stored in the one or more electronic memory storage devices. The system further comprises: a machine learning system comprising a neural network, and the neural network comprises one or more machine learning software modules. The intravascular system further comprises one or more AI processors, wherein the one or more machine learning software modules are executable on the one or more AI processors, wherein the one or more AI processors comprise dedicated memory; a bus; an AI processor memory; an interface for transmitting and receiving image data relative to the first processor; the machine learning system, which is electrically connected to a power source, wherein the machine learning system, the computing device, and the one or more electronic memory storage devices are disposed in a housing. In one embodiment, the bus is a PCIe bus. Other embodiments of this aspect include: corresponding computer systems, devices, computer programs recorded on one or more computer storage devices, AI processors, application specific integrated circuits (ASICS), circuits, and circuit components, each of which is constructed to perform the actions of the method. In one embodiment, the bus connects the AI ​​processor and the self-contained memory and the processor of the diagnostic / imaging system.

[0024] Embodiments may include one or more of the following features. The system, wherein the housing is a housing of an optical coherence tomography system or an intravascular supermarket imaging system. The system, wherein the one or more image processing software modules include one or more of the following: intravascular polar coordinate image to Cartesian coordinate image conversion software; including intravascular Cartesian coordinate image to polar coordinate image conversion software; tissue classification overlay software that identifies areas of interest or features when displayed to an end user; lumen detection software module; image flattening preprocessing software module; image resizing software module; image annotation software with a GUI to identify or label training images with real data in the field; preprocessing software module and circular rotation software module. The system, wherein the one or more machine learning software modules include one or more of the following: neural network interface; lumen contour prediction; side branch prediction; image resizing module; user interface and input processing software module; MLS interface software module to control and set parameters of the neural network; MLS memory management software; preprocessing software module; stent rib prediction software module; imprisoned stent prediction software module; guidewire prediction software module; interface module for exchanging data with an imaging system. Implementations of the techniques may include: hardware, a method or process, or computer software on a computer-accessible medium.

[0025] The present disclosure relates in part to computer-based methods and systems suitable for evaluating image data from a patient in real time or on a substantially real time basis using machine learning (ML) methods and systems. In various embodiments, a series of image data (e.g., a pullback of intravascular data) is classified on a substantially real time basis using a trained neural network (e.g., a convolutional neural network). In various embodiments, the series of image data includes between about 400 frames and about 600 frames, obtained from a memory or obtained by imaging a patient using an imaging system. In one embodiment, a series of image data (including between about 400 frames and about 600 frames) is classified. In one embodiment, the range of the substantially real time basis is from about 1 second to about 60 seconds. In one embodiment, the range of the substantially real time basis is from about 1 second to about 30 seconds. In one embodiment, the range of the substantially real time basis is from about 1 second to about 20 seconds. In one embodiment, the range of the substantially real time basis is from about 1 second to about 15 seconds. In one embodiment, the range of the substantially real time basis is from about 1 second to about 10 seconds. In one embodiment, the substantially real time basis is less than about 10 seconds. The present disclosure relates, in part, to improving diagnostic tools for end users (e.g., cardiologists and imaging specialists) using machine learning techniques applied to specific problems associated with intravascular images having Cartesian and polar coordinate representations. Further, if OCT, IVUS image data, or other imaging data is acquired using a rotating probe, processing the two coordinate systems associated therewith poses a challenge. The present disclosure focuses on these and various other challenges associated with addressing the problem of rapid imaging and diagnosis of patients so that stent placement and other procedures can be performed in a single session in the catheterization laboratory. The ability to segment an image into multiple features or regions of interest reduces the time a patient spends in the initial diagnostic process and subsequent treatment process by providing clinicians with diagnostic information to inform stent planning, bypass evaluation, cardiovascular surgery, and other surgical options, and to assess changes in the patient's condition over time.

[0026] In one embodiment, MLS system training is performed using polar images or polar image elements annotated by an expert. The annotated polar images are used to train the MLS. The MLS operates on new polar images from the patient to generate outputs of classified image regions that are still in polar form. After using the trained MLS predictions and inferences, the polar form predictive output from the MLS is then converted to Cartesian form, and the image with the classified tissue regions (lumen, endothelium, side branches, guidewires, stents, plaques, calcium, etc.) is then displayed in Cartesian form. In one embodiment, the coordinates may be annotated as an image in Cartesian form, and then finally converted to polar form after processing.

[0027] In various embodiments, probability maps and tissue maps are generated to provide user interface feedback for various workflows. In addition, probability maps and tissue maps can be combined, compared, convolved, and otherwise used to generate region of interest and feature classification outputs using trained neural networks. In various embodiments, a given neural network is preferably trained using annotated polar coordinate images.

[0028] The present disclosure relates in part to user interface design that facilitates improved information and time management based on characterizing tissue of a body lumen (e.g., a coronary artery) using one or more tissue atlas representations. In various embodiments, various detected ROIs / FOIs can be registered with angiographic data and displayed using one or more user interfaces as part of an imaging system or other diagnostic system.

[0029] The present disclosure relates in part to methods for identifying regions of interest in a blood vessel, which regions of interest may include tissue type and other features, such as side branches, stents, guidewires, and other features, properties, and materials of the vessel, wherein the aforementioned features of interest are detected using an imaging processing pipeline and a neural network is used to detect other ROIs / FOIs, such as calcium, lumen, media, intima, lipids, and other features disclosed herein.

[0030] In one embodiment, the tissue type or tissue characteristics, regions of interest (ROIs), features of interest, categories or types, or vascular features selected for segmentation and / or detection and represented in one or more masks, images, or outputs include tissue maps, including one or more of: cholesterol, fiber, lipid pools, lipids, fibrofat, calcifications, calcium nodules, calcium plates, intima, thrombus, foam cells, proteoglycans, and other parameters as disclosed herein. The various systems disclosed herein are operable to perform all methods and processes disclosed herein using dedicated circuits, controllers, FPGAs, AI processors, and other components as disclosed herein.

[0031] Although the present disclosure relates to different aspects and embodiments, it should be understood that the different aspects and embodiments disclosed herein can be integrated, combined or used together as a combined system or partially as discrete components, devices, and systems when appropriate. Therefore, each embodiment disclosed herein can be incorporated into each aspect, changing to a certain extent as appropriate for a given embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles as a whole. The drawings are to be considered illustrative in all respects and are not intended to be limiting of the disclosure, the scope of which is defined solely by the claims.

[0033] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0034] Figure 1 is a schematic diagram of a system according to an exemplary embodiment of the present disclosure, the system being adapted to train a neural network using ground truth masks obtained relative to arterial image data (e.g., image frames), and adapted to use, for example, the network to generate a predictive output for classifying a particular feature / region of interest.

[0035] Figure 1A is a schematic diagram of a machine learning system according to an exemplary embodiment of the present disclosure, wherein the machine learning system is suitable for using image data, such as image data obtained for an artery, in a training phase.

[0036] Figure 1B is a schematic diagram of a machine learning system according to an exemplary embodiment of the present disclosure, wherein the machine learning system is suitable for using user image data, such as image data obtained for an artery, to detect, predict and / or classify features of interest in the user image data.

[0037] Figure 1C is a schematic diagram of a machine learning system according to an exemplary embodiment of the present disclosure, wherein the machine learning system is suitable for using user image data showing input polar coordinate image data (e.g., OCT image data), which is operated to generate real field image data for training MLS and predictive image data obtained using the trained MLS.

[0038] Figure 1D is a schematic diagram of a machine learning system according to an exemplary embodiment of the present disclosure, wherein the machine learning system is suitable for using user image data, which generates features / regions of interest for classification / characterization of the user image data.

[0039] Figure 1E is a schematic diagram of a machine learning system according to an exemplary embodiment of the present disclosure, wherein the machine learning system is suitable for using user image data, which generates features / regions of interest for classification / characterization of the user image data, including a lumen detection system and a preprocessing system.

[0040] Figure 1F is an exemplary process flow for the training process and subsequent prediction process for a given illustrative embodiment of the present disclosure.

[0041] Figure 1G Image data for patient imaging for a given illustrative embodiment of the present disclosure is shown, along with three output probability maps corresponding to detection of the media, calcium, and lumen in separate channels.

[0042] Figure 2 is a multi-layer neural network architecture according to an exemplary embodiment of the present disclosure, suitable for training and generating predictive results using real-world annotations.

[0043] Figure 2A is a schematic diagram of an exemplary embodiment according to the present disclosure, showing a series of outputs from MLS (including tokens corresponding to various color-coded categories), and using the tokens to obtain projections and generate a carpet view (and optional tissue atlas).

[0044] Figure 2B is a graph of an MLS system according to an exemplary embodiment of the present disclosure, showing loss function values ​​for test data and training data.

[0045] Figure 2C is a graph of an MLS system according to an exemplary embodiment of the present disclosure, showing mismatch percentage values ​​of test data and training data.

[0046] Figure 2D Shown are intravascular polar images of a patient's artery, polar images annotated or enhanced with classified ground truth regions, and polar images with features and / or regions of interest classified using MLS according to illustrative embodiments of the present disclosure.

[0047] Figures 2E to 2H Shown from left to right are polar coordinate image data of an artery according to an exemplary embodiment of the present disclosure, a polar coordinate image annotated with actual true lumen data, lumen detection results performed by MLS with a two-dimensional neural network, and lumen detection results performed by MLS with a three-dimensional neural network.

[0048] Figure 3A A graphical user interface of an MLS system according to an exemplary embodiment of the present disclosure is shown, suitable for browsing and annotating image data for training the MLS or components thereof.

[0049] Figure 3B ] is a polar coordinate frame of image data from a pullback of an imaging probe through an artery (annotated using MLS's graphical user interface based training tool) according to an exemplary embodiment of the present disclosure.

[0050] Figure 3C is a schematic diagram of a Cartesian form of a segmented artery image according to an exemplary embodiment of the present disclosure, wherein each tissue region is classified by a predictive operation of MLS.

[0051] Figure 3D and 3Eis an output image from MLS according to an exemplary embodiment of the present disclosure, converted to Cartesian form, showing the input images and semantic segmentation of each image to show the MLS classified tunica media, lumen, and Ca regions / masks.

[0052] FIG. 4A to FIG. 4C is an exemplary imaging and diagnostic system that includes one or more machine learning systems that implement or can be integrated or otherwise combined with an imaging system.

[0053] Figure 5A is a schematic diagram of a system suitable for performing guided cardiovascular surgery using an optical imaging system (eg, an OCT imaging system) according to an exemplary embodiment of the present disclosure.

[0054] Figure 5B is a schematic diagram of a method according to an exemplary embodiment of the present disclosure, the method being suitable for performing guided cardiovascular surgery using an optical imaging system (eg, an OCT imaging system).

[0055] Figure 5C and 5D is an OCT image segmented using an MLS system (eg, a deep learning-based system) to highlight calcium plaque areas according to an exemplary embodiment of the present disclosure.

[0056] Figure 5E is a user interface display on a three-dimensional, cross-sectional, longitudinal view of a calcium plaque according to an exemplary embodiment of the present disclosure.

[0057] Fig. 5F is a calcium map according to an exemplary embodiment of the present disclosure, formed using angiographic co-registration on an angiographic system.

[0058] Figure 5G and 5H An annotated Cartesian OCT image is shown according to an illustrative embodiment of the present disclosure.

[0059] Fig.5I A magnified Cartesian OCT image identifying arterial layers is shown according to an illustrative embodiment of the present disclosure.

[0060] Figure 5J Showing an exemplary embodiment according to the present disclosure Figure 5C A histological image of an artery in which various layers of the artery are identified is part of a ground truth or training set review of image data suitable for annotating for a given MLS embodiment.

[0061] Fig. 6A and 6BAn exemplary embodiment according to the present disclosure displays a Cartesian OCT image and a predictive output of an MLS identifying the lumen, media, and calcium plaques, and displays exemplary diameter measurements accordingly.

[0062] Figures 6C to 6F Additional OCT images, ground truth masks, and output masks are shown indicating tissue classification and prediction using MLS according to an illustrative embodiment of the present disclosure.

[0063] Fig. 7A An initial exemplary OCT image in Cartesian form obtained using a combined imaging and MLS based system according to an exemplary embodiment of the present disclosure is shown.

[0064] Figure 7B , 7C 7D shows a grayscale polar coordinate image, a ground truth mask and a Figure 7B The deep learning output of the MLS embodiment is generated by annotating images as input, training, and then using the trained MLS prediction.

[0065] Fig. 8A Pre-processing operations performed on a polar coordinate OCT image prior to annotation with expert data / ground truth data according to an exemplary embodiment of the present disclosure are shown.

[0066] Figures 8B to 8D Various exemplary polar coordinate image masks are shown for detecting lumen, media, and calcium according to illustrative embodiments of the present disclosure.

[0067] Figures 9A to 9C is a polar OCT image showing a circular rotation and left-right flipping applied to an initial OCT polar image as part of a pre-processing step according to an illustrative embodiment of the present disclosure.

[0068] Figures 10A to 10B is an image showing a ground truth annotated image and a predicted output according to an illustrative embodiment of the present disclosure.

[0069] Figures 11A to 11D is an image showing various pre-processing steps that have been developed to increase MLS processing speed to reduce patient wait time in accordance with an illustrative embodiment of the present disclosure.

[0070] Fig. 12A and 12B is a flow chart of exemplary processing steps whereby raw image data is routed and processed along with ground truth data annotations, according to an illustrative embodiment of the present disclosure.

[0071] Fig.13A and13B An exemplary user interface for a diagnostic tool to support stent planning through virtual stent placement, including tissue classification using MLS systems and methods, is shown according to an exemplary embodiment of the present disclosure.

[0072] Fig.14 A representation of an artery according to an exemplary embodiment of the present disclosure is shown, including various slices on the border from an OCT scan of a subject's artery and a tissue atlas generated using various series of image data acquired during a pullback process.

[0073] Fig.15 is a schematic representation of an artery according to an exemplary embodiment of the present disclosure, obtained using one or more imaging modalities and different slices suitable for generating a tissue atlas.

[0074] Figures 16 to 18 Various tissue atlas representations are shown according to an exemplary embodiment of the present disclosure, which are generated using OCT imaging pullback of an artery, wherein various markers are integrated into a user interface displaying the various tissue atlases to support diagnosis and treatment planning, such as stent placement and cardiovascular surgery. DETAILED DESCRIPTION

[0075] Various data collection and analysis systems can be used to obtain information related to the coronary system. Data obtained using a device from a blood vessel or data obtained from intravascular or extravascular measurements associated therewith can be analyzed or displayed to auxiliary researchers and clinicians. Optical coherence tomography (OCT) is an imaging modality that uses an interferometer to obtain distance measurements relative to a blood vessel or an object placed therein. Intravascular ultrasound (IVUS) can also be used in a probe to image portions of a blood vessel. Angiographic systems and fluoroscopic angiography systems are also commonly used to image patients, enabling diagnostic decisions to be made and various possible treatment options to be performed, such as stent placement. These and other imaging systems can be used to image patients externally or internally to obtain raw data (which may include various types of image data). The present disclosure relates to various machine learning system (MLS) embodiments, including one or more networks (e.g., neural networks) to provide improved classification of components of medical imaging data. An exemplary MLS-based system is provided by Figure 1 , 1A , 1B, 1C, 1D, 1E, 4A, 4B and other systems disclosed and illustrated in this document are displayed.

[0076] In particular, image data obtained about body lumens (e.g., coronary arteries) is of particular interest. Further, in view of diagnostic intravascular imaging, flow measurement, stent planning, and other benefits, there is great value in using MLS systems to obtain timely image analysis and tissue classification and classification. These systems often require a large amount of time to perform image processing tasks. Therefore, using an MLS system to reduce the time required to process a series of patient image data is a feature of various embodiments of the present disclosure.

[0077] In general, the present disclosure may be applied to any intravascular data collection device that can be used to generate and receive signals including diagnostic information (e.g., image data) about the blood vessel in which the device is used. Such devices may include, but are not limited to, imaging devices (e.g., optical or ultrasound probes), pressure sensor devices, and other devices suitable for collecting data related to a blood vessel or other component of the cardiovascular system.

[0078] Figure 1 A system is included that is suitable for performing the various steps of the present disclosure, such as collecting image data for annotation and generating ground truth masks and for classification using a trained neural network and displaying such predictive results to a user. Various user interface features are described herein to view and evaluate visual representations of arterial information. These user interfaces may include one or more movable elements that may be controlled by a user with a mouse, joystick, or other controller and may be operated using one or more processors and memory storage elements.

[0079] During the stent delivery planning process, the user can refer to the OCT and annotated angiography for the extent and location of the attachment to further expand and mobilize the stent as part of the delivery plan. These system features and methods can be used Figure 1 The system shown in 3 is implemented.

[0080] Figure 1 The system 3 is shown to include various data collection subsystems suitable for collecting data or detecting features of the object 4 or sensing the condition of the object 4 or otherwise diagnosing the object 4. In one embodiment, the object is placed on a suitable support 19 (e.g., a table, a bed, or even a chair, or other suitable support). Typically, the object 4 is a person or other animal with a specific area of ​​interest 25.

[0081] The data collection system 3 includes a non-invasive imaging system, such as magnetic resonance imaging, X-ray, computer-assisted tomography, or other suitable non-invasive imaging technology. As shown, as a non-limiting example of such a non-invasive imaging system, an angiography system 20 is shown, which is suitable for generating movies, for example. The angiography system 20 may include a fluoroscopic angiography system. The angiography system 20 is configured to non-invasively image the object 4, thereby generating frames of angiography data (typically in the form of image data frames), and the pullback process is performed using the probe 30, so that the blood vessels in the area 25 of the object 4 are imaged using angiography of one or more imaging technologies (such as OCT or IVUS).

[0082] The angiography system 20 is in communication with an angiography data storage and image management system 22 (which, in one embodiment, may be implemented as a workstation or server). In one embodiment, data processing associated with the collected angiography signals is performed directly on the detector of the angiography system 20. Images from the system 20 are stored and managed by the angiography data storage and image management system 22.

[0083] In one embodiment, the system server 50 or workstation 85 handles the functions of the system 22. In one embodiment, the entire system 20 generates electromagnetic radiation (e.g., X-rays). The system 20 also receives such radiation after passing through the object 4. In turn, the data processing system 22 uses the signals from the angiography system 20 to image one or more regions of the object 4 (including the region 25).

[0084] As shown, in this particular example, the region of interest 25 is a subset of a blood vessel or peripheral vascular system, such as a specific blood vessel. This can be imaged using OCT. A catheter-based data collection probe 30 is introduced into the object 4 and disposed in the lumen of a specific blood vessel (e.g., a coronary artery). The probe 30 can be various types of data collection probes, such as an OCT probe, an FFR probe, an IVUS probe, a probe combining features of two or more of the foregoing, and other probes suitable for intravascular imaging. The probe 30 typically includes: a probe tip, one or more radiation-opaque markers, an optical fiber, a torque wire. In addition, the probe tip includes: one or more data collection subsystems, such as: a beam guide, an acoustic beam guide, a pressure detection sensor, other transducers or detectors, and combinations of the foregoing.

[0085] For probes that include a beam guide, the optical fiber 33 is in optical communication with the probe through the beam guide. The torque wire defines a hole in which the optical fiber is disposed. Figure 1In the figure, the optical fiber 33 is shown without the torque wire surrounding it. In addition, the probe 30 also includes a sheath, such as a polymer sheath (not shown), which forms a part of the catheter. The optical fiber 33 (part of the sample arm of the interferometer in the application environment of the OCT system) is optically coupled / connected to the patient interface unit (PIU) 35, as shown.

[0086] The patient interface unit 35 includes: a probe connector, which is suitable for receiving the end of the probe 30 and optically coupling / connecting with it. Typically, the data collection probe 30 is disposable. The PIU 35 includes: appropriate connectors and components based on the type of data collection probe used. For example, a combined OCT and IVUS data collection probe requires a PIU for OCT and IVUS. The PIU 35 typically also includes: a motor, which is suitable for pulling back the torque line, sheath and optical fiber 33 disposed therein as part of the pulling back process. In addition to pulling back, the probe tip is typically rotated by the PIU 35. In this way, the blood vessels of the subject 4 can be imaged longitudinally or in cross-section. The probe 30 can also be used to measure specific parameters, such as blood flow reserve fraction (FFR) or other pressure measurements.

[0087] Furthermore, the PIU 35 is connected to one or more intravascular data collection systems 40. The intravascular data collection system 40 may be: an OCT system, an IVUS system, another imaging system, and a combination of the foregoing. For example, in an application environment where the probe 30 is an OCT probe, the system 40 may include: a sample arm of an interferometer, a reference arm of an interferometer, a photodiode, a control system, and a patient interface unit. Similarly, as another example, in an application environment of an IVUS system, the intravascular data collection system 40 may include: an ultrasound signal generation and processing circuit, a noise filter, a rotatable joint, a motor, and an interface unit. In one embodiment, the data collection system 40 and the angiography system 20 have a shared clock or other timing signal (constructed to synchronize the angiography video frame timestamp with the OCT image frame timestamp).

[0088] Apart from Figure 1 In addition to the invasive and non-invasive image data collection systems and devices of the present invention, various other types of data may be collected about the region 25 of the subject and other parameters of interest of the subject. For example, the data collection probe 30 may include: one or more pressure sensors, such as pressure wires. The pressure wires may be used without an attached OCT or ultrasound component. Pressure readings may be obtained along a segment of a blood vessel in the region 25 of the subject 4,

[0089] Such readings may be forwarded via a wired connection or via a wireless connection. As shown, in a fractional flow reserve (FFR) data collection system, a wireless transceiver 47 is configured to receive pressure readings from the probe 30 and transmit them to the system to generate FFR measurements or more locations along the measured vessel. One or more displays 82, 83 may also be used to display angiographic data frames, OCT frames, user interfaces for OCT and angiographic data, and other control parameters and features of interest.

[0090] Intravascular image data (e.g., frames of intravascular data) generated using the data collection probe 30 may be routed to a data collection processing system 40, which is coupled to the probe via the PIU 35. Non-invasive image data generated using the angiography system 22 may be sent to, stored in, and processed by one or more servers or workstations (e.g., registration server 50, workstation 85). A video frame grabber 55 (e.g., a computing board) configured to capture angiography image data from the system 22 may be used in various embodiments.

[0091] In one embodiment, the server 50 includes: one or more registration software modules 67, which are stored in the memory 70 and executed by the processor 80. The server may include a training neural network 52 suitable for implementing various embodiments of the present disclosure. In one embodiment, an AI processor (e.g., a graphics processing unit) 53 is included in the server 50 and is electrically connected / communicated with the memory 70. The computing device / server 50 may include other typical components for a processor-based computing server. Alternatively, more databases (e.g., database 90) may be constructed to receive generated image data, parameters of the object, and other generated, received, or executed by Figure 1 Information transmitted to database 90 by one or more of the systems, devices or components shown in FIG.

[0092] Although database 90 is shown as being connected to server 50 when stored in memory at workstation 85, this is only an exemplary configuration. For example, software module 67 may be run on a processor at workstation 85 and database 90 may be located in memory of server 50. Devices or systems for running various software modules are provided as examples. In various combinations, the hardware and software described herein may be used to obtain frames of image data, process such image data, and register such image data. The various software modules may also include tissue atlas generation software suitable for generating one or more tissue atlases displaying one or more regions of interest (ROIs) and / or detected or characterized tissue or arterial material (e.g., calcium).

[0093] As described elsewhere herein, the software modules 67 may include software such as pre-processing software, transforms, matrices, and other software-based components for processing image data or responding to patient triggers (to facilitate registration of different types of image data through other software-based components 67 or otherwise perform annotation of image data to generate ground truth values), as well as other software, modules, and functions suitable for implementing various embodiments of the present disclosure. The modules may include: lumen detection (using scan-line-based or image-based approaches); stent detection (using scan-line-based or image-based approaches); indicator generation; attachment strip generation (for stent planning); guidewire shadow indicators (to prevent confusion with foreign matter); side branch and missing data; and others.

[0094] The database 90 may be configured to receive and store angiography image data 92, such as image data generated by the angiography system 20 and obtained by the frame grabber 55 of the server 50. The database 90 may be configured to receive and store OCT image data 95, such as image data generated by the OCT system 40 and obtained by the frame grabber 55 of the server 50.

[0095] Additionally, subject 4 may be electrically coupled via one or more electrodes to one or more monitors, such as monitor 49. Monitor 49 may include, but is not limited to, an electrocardiogram monitor configured to generate data related to cardiac function and display various states of the subject, such as diastole and systole.

[0096] The use or absence of arrows showing directionality in a given figure is not intended to limit or require the direction along which information can flow. For a given connection, e.g. Figure 1 The arrows and lines connecting the elements shown in the figure may flow in one or more directions, or only in one direction, as appropriate for a given embodiment. The connection may include various suitable data transmission connections, such as optical, wired, power, wireless or electrical connections.

[0097] One or more software modules may be used to process data from an angiography system (e.g. Figure 1 Various software modules (which may include but are not limited to software, components thereof, or one or more steps of a method based on software or executed by a processor) may be used in a given embodiment of the present disclosure.

[0098] The present disclosure relates in part to intravascular data collection systems and related methods, whereby intravascular data collected by an intravascular probe can be transformed or analyzed by a processor-based system. The results of such analysis and transformation can be displayed to an end user in various presentation modes, such as by a display that communicates / is connected to a given MLS (which has a neural network to classify components of a medical image). In one embodiment, a given imaging system (e.g., OCT, IVUS, an X-ray-based imaging system) is in electronic communication / communication with the MLS and is able to display a modified version of the image data when the image data was acquired in the same session using a given type of imaging system. Various neural network architectures can be used for image segmentation, such as V-net, U-net, CUMedVision1, CUMedVision2, VGGNet, multi-stage multi-recursive input fully convolutional network (M2FCN), coarse-to-fine stacked fully convolutional network, deep active learning framework, ResNet, combinations thereof, and other neural network and software-based machine learning frameworks suitable for image segmentation.

[0099] In one embodiment, the MLS includes a dedicated hardware system to handle the necessary machine learning operations and their training process so that results can be obtained in an accelerated manner (e.g., within about 2 seconds to about 30 seconds). In one embodiment, the results are obtained in less than about 45 seconds. The dedicated hardware system of a given MLS embodiment may include multiple processors, such as AI / ML processors. The machine learning system can be implemented by training classifiers to segment or manipulate images to constitute tissues, tissue types, and other areas of interest that are detected and characterized based on type or another parameter. In one embodiment, the lumen, endothelium, media, and plaque are detected and identified as having boundaries corresponding to these different tissues. Training a given MLS / neural network includes teaching the network using known inputs and known outputs.

[0100] The present disclosure relates to an advanced machine learning system, which includes one or more AI processors, the AI ​​processors including: more memory allocated based on each processor. The advanced machine learning system is designed to support multi-channel segmentation. For a given embodiment, each channel can be selected according to different areas of interest and characteristics. For example, in one embodiment, the first channel, the second channel, the third channel, and the fourth channel are defined so that one of each of the aforementioned channels is associated with a lumen, an intima, a media, or a plaque. Other categories / types can be associated with different channels to facilitate segmentation.

[0101] In one embodiment, plaque types are classified. In some embodiments, plaque types may be classified based on calcification. Additionally, if the presence of plaque and other detectable features in a given region of an artery may indicate the presence of a narrowing (e.g., due to a stenosis), another feature of the present disclosure enables rapid and automatic retrieval of one or more scores associated with a given plaque or stenosis to facilitate decision making by an end user. For example, a given score determined using image data and machine learning-based analysis thereof may help determine whether immediate action is not recommended, whether a stent should be placed for the stenosis, or whether cardiovascular surgery or other surgery (e.g., bypass surgery) is necessary.

[0102] For healthy patients, arteries have layers arranged in a consistent structure, including the intima, the middle membrane, and the outer membrane. As a result of the atherosclerotic process, the intima becomes pathologically thickened and may contain plaques composed of different types of tissues (including fibers, proteoglycans, lipids and calcium, as well as macrophages and other inflammatory cells). When imaged using various imaging systems, these tissue types have different characteristics, which can be used to establish a series of training data for one or more machine learning systems of the present disclosure. It is believed that the most important plaque in pathology is the so-called vulnerable plaque, which is composed of a fibrous cap with an underlying lipid pool. Different atherosclerotic plaques have different geometric shapes. For example, foam cells often form band-like features on the shoulders of large lipid pools; the middle membrane presents a ring around the blood vessel, etc. Shape information is currently used for qualitative evaluation of OCT images. In one embodiment, a neural network is trained to identify a fibrous cap and / or a fibrous cap with an underlying lipid pool. In various embodiments, reference to calcium herein also includes, but is not limited to: calcified plaques and other calcium-containing tissues.

[0103] While the patient is still catheterized and preparing to receive a stent or other treatment option, the imaging procedure can be quickly performed on the patient and an arterial image obtained and then processed using a machine learning system, resulting in significant time savings and improved patient outcomes.

[0104] Physicians use the tunica media and the outer edge of the tunica media (called the external elastic lamina, EEL) to size their stents during interventional procedures. Finding the tunica media and measuring the diameter in locally diseased tissue is time consuming and difficult. It also requires training in image interpretation. Automated detection and measurement of EEL diameter addresses these technical issues when making a diagnosis or otherwise evaluating a patient’s treatment options. An example of this diameter measurement is shown in Figure 6B middle. Figure 3C A series of identification masks and ROI masks are displayed in a Cartesian view of the artery.

[0105] ROIs are shown as generic examples and may correspond to calcium or another feature of interest, such as an area containing side branches or stent ligaments. Each region / feature corresponding to lumen L, intima I, plaque Q, adventitia ADV, imaging probe P, media M, and other structures may be generated by MLS using a trained neural network (NN) such as a CNN. Figure 3C The image 385 can be used as a diagnostic tool to help guide and inform clinicians. In various embodiments, Figure 3C The detection of various features in the Fig. 5F In some embodiments, one or more features, regions, types, categories, predictive results are registered with the angiographic data. Fig. 5F In some embodiments, the tissue atlas and other image data representations are used to support diagnosis and treatment proposals. In one embodiment, each feature or category identified (e.g., ADV, EEL, IEL, L, P, I, Q) can be generated as a mask or predictive mask using one or more of the trained neural networks disclosed herein. The markers corresponding to the output results can be displayed using color-coded markers and other markers.

[0106] In one embodiment, each annotated mask region corresponding to a series or group of pixels defines a ground truth mask for training one or more neural networks disclosed herein. Once the neural network is trained, a predictive or detection mask is generated that includes a series of pixels that correspond to user data regions and identifiers of features or categories of the regions, such as whether they are luminal, calcium, EEL, or other categories or features disclosed herein. In one embodiment, predictive results are generated based on each category, and then all predictive results for a given image data input (e.g., an input frame of OCT, IVUS, or other image data) are compared at a pixel level to generate a final predictive result for all categories. In one embodiment, the predictive results are displayed as an output image mask with regions corresponding to specific categories indicated by markers (e.g., colors and one or more legends summarizing which marker reflects which category).

[0107] Detecting plaques and classifying plaque types helps physicians choose their interventional strategies. They can choose to perform cardiovascular surgery when calcium burden is too high or choose to place stents in different areas based on the underlying plaque composition. Automating the process of plaque identification and characterization makes image interpretation easier for physicians and improves their workflow.

[0108] The MLS disclosed herein can be implemented by various neural networks and integrated into various imaging and diagnostic systems. In one embodiment, the system and method are implemented using a deep learning framework. In various embodiments, the MLS includes one or more of the following: a neural network, a rule engine, a fuzzy logic system, a comparator, an image processing module (e.g., a flattening, rotation / shifting, and resizing module, such as a heuristic system), a pattern matching and pattern recognition system, and the software implementation of the above.

[0109] In one embodiment, the MLS uses a neural network (NN) (e.g., a convolutional neural network, CNN). The CNN includes a plurality of nodes or neurons and can receive and output image data, data derived from the image data and / or changes to the image data and / or classification information about components or segmented portions of the image data. In one embodiment, the CNN performs semantic image segmentation. In one embodiment, semantic segmentation using a given MLS embodiment can be used to: detect whether an image has calcium and EEL and identify pixels with calcium and EEL. This helps doctors solve various problems related to selecting treatment options and guiding specific treatments. In one embodiment, the MLS uses a three-dimensional CNN, such as a PyTorch three-dimensional CNN or V-NET. The neural networks described herein can be implemented using various frameworks, including: PyTorch, Microsoft Cognitive Toolkit (CNTK), TensorRT, TensorFlow, and other similar proprietary and open source ML frameworks and tools.

[0110] In one embodiment, the tool selected for data annotation allows the user to select, move pixels and / or draw boundaries. Such an exemplary user interface tool 305 is shown in Figure 3A and 3B GUI usage supports consistency of frame-by-frame annotation (especially in 3D). If standardized tools (such as GUI-based annotation tools) are used, the constraints on the tools and the series of annotation actions support consistent training data. This is different from different users manually annotating images and then scanning or processing the images. In one embodiment, a useful feature of the GUI is the ability to copy data transmitted from a previous frame. Further, in one embodiment, each point is annotated with polar coordinates, eliminating the need for Cartesian to polar conversion during pre-processing.

[0111] In one embodiment, the input to the MLS includes a training set of about 450 expert annotated images, where the tissue type of each image pixel is annotated by one or more expert users. In one embodiment, before using the image data as a training set, the training set can be preprocessed by the same or another MLS to perform lumen detection. Thus, in one embodiment, the first series of training data is preprocessed using one or more preprocessing techniques. The preprocessing techniques may include: using a previously trained MLS (implemented using a training set with annotated lumen regions or segments) to perform lumen detection. The preprocessing techniques may also be selected to speed up network training and / or the prediction speed of the trained network during the back propagation process. Therefore, the preprocessing techniques may also include: image data flattening, periodic rotation processes, circular rotation processes (wherein part of the image data is excluded, such as depth data below the noise floor, data removal may be performed in an alternating manner so that every other scan line of the image is removed or every other column is removed, and pixels may be filtered to remove noise and increase regional uniformity) and other preprocessing steps.

[0112] In one embodiment, the training set includes between about 400 and about 600 image data elements, such as individual images, or groups, groups or subsets of images, scan lines or pixels (image data elements). In one embodiment, the training set includes: between about 500 and about 700 images or image data elements. In one embodiment, the training set includes: between about 700 and about 800 images or image data elements. In one embodiment, the training set includes: between about 800 and about 900 images or image data elements. In one embodiment, the training set includes: between about 900 and about 1000 images or image data elements. In one embodiment, the training set includes: between about 1000 and about 1100 images or image data elements. In one embodiment, the training set includes: between about 1000 and about 5000 images or image data elements.

[0113] In one embodiment, a given CNN for a particular MLS includes a plurality of nodes or neurons. Such a network may include various layers, including hidden layers. Each element of the network may have weight values, filter values, or bias values ​​that vary over time as the network is trained to learn input image data (e.g., OCT, IVUS, angiography, CT scan, or other image data). In one embodiment, a network for a given MLS implementation includes: a plurality of filters that vary over time in response to a training set (which includes polar and / or Cartesian image data input).

[0114] In one embodiment, a CNN interface (which may include one or more graphical user interface components) is used to assist in batch input of image data. A given image is an example of image data. The image data may also include: scan lines from intravascular pullback maps and other sources of medical imaging data suitable for segmentation and classification. In one embodiment, each image that constitutes the image data (ground truth) is divided into different parts, categories or types to support the use of such data to train the network to detect such parts, categories or types when operating on new patient image data. In one embodiment, the ground truth training set is a mask generated for the area of ​​interest or feature (of interest for classification and generation of trained MLS) to predict or detect them automatically (without human intervention) when the MLS is running in various embodiments. Human or machine learning can be used to generate training sets / ground truth. Each of these can be implemented as a mask or data channel. Each data channel is color-coded in various embodiments.

[0115] like Figure 1A As shown in , a schematic representation of a system 5 is shown for implementing the training phase of MLS (suitable for classifying tissue and other data types for a given series of medical image data). Various features, regions, types and / or categories of tissue and regions, pixels, contours and boundaries in the image can be marked or identified relative to the image data to obtain labeled image data (e.g., ground truth masks). In turn, the foregoing can be used to train a neural network so that the network can operate on the image data to generate image data output using the trained network, which has markings suitable for classifying and visualizing features, regions, types and / or categories of tissue and regions, pixels, contours and boundaries. A given ground truth match can include: a region corresponding to a specific channel or feature, or a group of regions corresponding to a specific channel or feature. A given region includes: a series of pixels in a given ground truth mask. In one embodiment, the ground truth mask can be annotated to include regions or features corresponding to specific types or categories (e.g., calcium, plaque, EEL boundaries and various other categories, types and classifications disclosed herein).

[0116] In one embodiment, data augmentation is performed by increasing the cardinality of the image data set by performing one or more transformations on the image data, the annotated image data, and / or the ground truth mask. The transformations may include one or more of the following: circular rotation, left flip, right flip, flattening, resizing, cropping, filtering, binarization, normalization. When the image data is OCT data, IVUS data, or other data obtained using one or more rotation elements, the data augmentation is performed under the condition of avoiding transformations inconsistent with the imaging performed.

[0117] In one embodiment, the MLS includes a CNN that includes one or more inputs to receive image data and generates outputs based on the MLS operating on the image data. In one embodiment, the MLS and / or the CNN include a deep learning classifier. In one embodiment, about 400 to about 2000 image data elements (e.g., OCT, IVUS, or angiography images) are annotated by an expert or a first MLS and provided to the MLS as image data. Such image data includes image data elements and real-world annotations or even such data elements obtained from one or more experts and / or the first MLS for image preprocessing (e.g., lumen detection). In one embodiment, the pixels of each image data element (or a subset / subset thereof) are annotated by an expert of the first MLS.

[0118] Therefore, the image data (including the image data elements and the ground truth annotations thereto) constitute a training set, which is provided as input to the MLS. In one embodiment, the training set is input to the MLS including a CNN and / or deep learning classifier. Figure 1A As shown in , image data / image data elements and ground truth annotations are input to MLS. The output from MLS is compared with the ground truth annotations and / or with the image data elements combined with the ground truth annotations, and an error value is generated. In one embodiment, the error value is the output of the cost function. In one embodiment, the error value is a training error value. Various metrics, scores, and tests can be used to evaluate the difference in training errors relative to an allowable threshold and the convergence over time. In various embodiments, Dice, Jaccard, cosine similarity, and overlap distance, metrics, scores, or coefficients may be outputs, inputs, or components of the cost function. More details related to the cost function are described herein. Figure 1A The system of and other systems disclosed herein may include various pre-processing steps to further improve training and / or processing time.

[0119] Figure 1F80 of the present disclosure, comprising: a training process and a subsequent prediction process suitable for various regions of interest / features, such as lumen, calcium, media, and intima. In one embodiment, the trained neural network is trained using ground truth annotations, including calcium, intima, lumen, media. In one embodiment, the trained neural network is trained using ground truth annotations, which include or specifically relate to one or more of the following: calcium, intima, lumen, media. The method comprises: acquiring image data, which may include polar coordinate image data, such as intravascular data. Step 100. Initially, a series of ground truth data (e.g., ground truth masks) is established by reviewing and annotating the series of image data. In one embodiment, the method comprises: annotating one or more regions of interest or features in each polar coordinate image of the series of images, such that each annotated region or feature is a ground truth annotation. Step 102. Pixels corresponding to the ground truth annotations may be stored to a persistent electronic memory (e.g., Figure 1 All data and annotations described herein may be stored in a database or other data structure permitted by the neural network and the image processing software module of a given imaging system.

[0120] In one embodiment, the ground truth annotations can be made available through a user interface as shown and discussed in more detail herein (e.g., see Figure 3A ). In one embodiment, the annotations are stored as ground truth masks and may include a series of regions of interest or features identified by a user and the boundaries or pixels that make up the regions or features. In one embodiment, the mask includes a stack of pixels combined with an initial image (e.g., a polar coordinate image). In one embodiment, the mask includes addresses / lists of pixels that correspond to specific ground truth annotations or predictive results.

[0121] A neural network of a machine learning system is trained using a series of annotated polar images. Step 104. The training may be performed using a ground truth mask and image data annotated to obtain the ground truth mask. The ground truth mask may be enhanced using circular flipping, right flipping, left flipping, and other enhancement transformations that may be used for rotational imaging modalities. The present disclosure relates in part to enhancing polar images in which a ground truth mask is modified to increase the number of ground truth masks. In one embodiment, lumen detection is performed as an initial detection step such that the ground truth mask includes lumen boundaries or lumen features or lumen regions. In one embodiment, training is performed until one or more metrics (e.g., a cost function or other error measurement) are reduced to an acceptable level. As the error decreases, the detection / prediction accuracy of the MLS increases.

[0122] Once the MLS is trained, a series of image data is fed into the neural network to generate various features / regions of interest (with predictions, detections, classifications, etc.). Step 105. Generate K probability masks (for each of the K categories / types). Step 106. When K is 3 (three different categories), an example of the probability map output / probability mask is shown in Figure 1G In particular, Figure 1G Image data of patient imaging and three output probability maps / masks for different classes are shown. An illustrative embodiment of the present disclosure is given corresponding to the output masks / maps for detecting the media M, calcium Ca, and lumen L as separate channels. As shown, for each corresponding channel, the detected class is shown in red. Thus, in Figure 1G In the figure, the media M is red in the media channel, the two detected calcium regions Ca are red in the calcium channel, and the lumen L is shown in red in the lumen channel. Other markers can be used to color code or represent various detection / prediction categories corresponding to the region / feature of interest. In one embodiment, each category or type of feature of interest is assigned an associated channel for tracking masks, atlases, and final output (which has predictions associated with the classification of each part of the image used for network training).

[0123] If calcium, media, lumen, and intima are the categories / types used for neural network training to classify features / regions, then K is 4. In one embodiment, the output of the neural network includes: K probability maps, such that there is one probability map for each category / type. The method may include: generating a final predictive output for each frame of input image data. Step 107. In one embodiment, each of the K probability maps (for each of the different K categories / types) is compared and evaluated on a per-pixel basis, such that for each pixel of a given image frame, a final predictive result is then assigned to each pixel. In this manner, each frame of image data is processed to generate a final predictive output. In one embodiment, the final predictive result is a predictive output mask that includes one or more tokens corresponding to a type / category. In one embodiment, the method includes: displaying the final predictive output image from the neural network / machine learning system with the category / type tokens. Step 108.

[0124] In one embodiment, various markers are used to color code or otherwise visualize or display the segmentation between different regions of interest and features identified using the trained MLS. In one embodiment, output image data is displayed that has been modified to include features / regions of interest identified / predicted using the neural network / machine learning system. The foregoing steps can be performed using one or more of the computer-based and AI processor-based systems disclosed herein.

[0125] Various neural network architectures can be used with the embodiments disclosed herein. For example, V-net, U-net, CUMedVision1, CUMedVision2, VGGNet, multi-stage multi-recurrent input fully convolutional network (M2FCN), coarse-to-fine stacked fully convolutional network, deep active learning framework, ResNet, combinations thereof, and other neural network and software-based machine learning frameworks can be suitable for performing feature / region of interest image segmentation and classification as disclosed herein. Figure 2 is an example of an architecture for a convolutional neural network suitable for training using annotated ground truth masks and generating probability maps suitable for evaluating predictive outputs and displaying classified image data with classified regions / features of interest. Figure 2 The multi-layer neural network architecture 115 is suitable for training and generating predictive results using real-world annotations. Figure 2 The neural network architecture of can be trained with ground truth masks for various types / classes (calcium, lumen, media, intima, and various other groups disclosed herein). Network 115 can be any network architecture disclosed herein, or a modified version thereof, or a combination thereof. In one embodiment, network 115 is a convolutional network.

[0126] The neural network 115 includes an input 111 and an output 117. The output 117 is K probability maps for each image data input (when K categories are specified to classify the features and regions of interest). The K probability maps are evaluated using a scoring or weighting system (whereby the output probability maps are compared for each frame of the image data 111 and used to determine which pixels have a higher relative probability of being one of the K categories). As a result of the evaluation, a final predictive output is generated that displays these K categories and associated features and regions with symbols and legends to distinguish the categories.

[0127] Each node is N1-N9 shown in the network 115, which has an associated series of operations and transformations "OP" that can be applied to the image data channels. In various embodiments of the network 115, the number of channels T is typically 16 or more channels. In an effort to increase the processing speed of the network 115, the applicant has found that setting T to 4, T=1, is suitable for classifying and training the network to evaluate arterial image data. Thus, in various embodiments, T channels correspond to 4 channels, 2T, 4T, 8T, 16T channels correspond to 8 channels, 16 channels, 32 channels, 64 channels. These various channels are set at each layer / hidden layer of the network to specify how the input data situation should be operated for each function OP of each node of the network. In one embodiment, the network 115 is trained using a function / operator (e.g., an adaptive learning algorithm) through a ground truth mask. In one embodiment, a gradient descent-based method is used to train the network 115 and the annotated ground truth mask.

[0128] Each node of the network 115 includes a series of operations and transformations OP for the nine nodes N1-N9 shown: OP1, OP2, OP3, OP4, OP5, OP6, OP7, OP8, OP9. Additional nodes and layers may be added between nodes N2, N3 and nodes N7, N8, as shown by network elements 128a, 128b. The left side of the network 115 (including nodes N1, N2, N3, N4) and the input arrow from node N4 to N5 perform one or more downsampling / downconversion operations DC. In contrast, on the right side of the network 115 (including the output arrow from node N5 to N6 and nodes N6-N9), this portion of the network performs one or more upsampling / upconversion operations UC between the nodes. In one embodiment, the neural network architecture includes an encoder (capturing scene information) and a symmetrical decoder path (enabling precise positioning).

[0129] The left portion of the network 115 is the downsampling encoder portion, where convolution blocks and downsampling operations are used to encode the input image into feature representations at multiple different levels. The right portion of the network includes upsampling convolution operations and cascade operations. The right portion of the network operates to output an image that will have the same scale as the input image. In one embodiment, the network architecture has 4 cases of downsampling operations or upsampling operations, so the scale of the input image will need to be divided by 16 to avoid scale mismatch. In this document, the image data has M×N pixel scale.

[0130] In one embodiment, the M×N scale is 912×512, although other specified scales may also be used. In one embodiment, images that are different from the specified M×N size are stretched or cropped to match the specified scale. In various embodiments, the OP operations and functions performed at each node (OP1-OP9) are selected from the group of the following processes: convolution, deconvolution, additive process, cascade process, up-conversion process, down-conversion process, Softmax process and PReLu process. In one embodiment, the Softmax and parameterized rectified linear unit (PReLu) processes are performed as layers between one or more nodes, converting network parameters transmitted between nodes from image data into probabilities included in the output probability map. In one embodiment, the output of the neural network includes a probability assignment layer, which can be constructed using a Softmax function, for example. The probabilities are provided as multiple outputs 117 for each of the K categories based on pixels or according to another grouping method (e.g., by scan line or by mask area), based on a series of training categories (also called types or identifications). In one embodiment, each node can operate as a layer, or a layer can be defined by two horizontal nodes and arrows between them. In one embodiment, transmissions between nodes (shown by arrows 130a-130d from the left side of the network to the right side) correspond to combining or concatenating one or more channels from the left side of the fabric to the right side of the fabric.

[0131] In one embodiment, Figure 2 The network architecture is constructed to characterize three or more types / categories of pixel identifiers, such as calcium, media, lumen, and other categories and types corresponding to imaged features and regions of interest. In one embodiment, network 115 reduces the channel count so that T is 4 (for all T channels shown) so that the processing speed of the network is approximately real-time (between about 400 to about 600 frames of image data). If the goal is to reduce the time a patient spends in the catheterization laboratory for imaging and other procedures, a composite neural network (e.g., a network with T greater than 4) causes a slowdown in inference time when generating predictive results / probabilities, and leads to other problems such as overfitting during training. In various embodiments, in Figure 2 In the network 115, T is set to 4 so that the network accuracy is improved while reducing the inference time. In one embodiment, for T equal to 4, Figure 2 The number of channels is 4, 8, 16, 32, and 64, corresponding to the T channels, 2T channels, 4T channels, 8T channels, and 16T channels shown.

[0132] In one embodiment, each horizontal layer (such as a given node or two nodes linked by transfer operations 130a-130d) is a convolutional layer that performs one or more of the convolution operations in OP1-OP9 and / or vertical DC operations between nodes. These layers extract or learn some features of the input real data. For the convolutional layers on the left, various spans can be set for each node. In the convolution operation, a given span provides control over the reduction of the image size during the convolution operation. For example, if the image size is 256×256, the output size after convolution will be 128×128 (256 / span). The deconvolution on the right side of the network performs coarsening on each of the various UC operations.

[0133] Figure 2A The schematic diagram shows a series of outputs from MLS according to an exemplary embodiment of the present disclosure, including markers corresponding to each color-coded category (Ca (red), lumen (blue), media (green)), and showing the use of the aforementioned method to obtain axial projections and generate carpet views (and optional tissue atlases).

[0134] In various embodiments, the neural network architecture may be a two-dimensional (2D) or three-dimensional (3D) network, and thus operable to process both 2D and 3D data. In one embodiment, the 3D for F-frame pullback (where each frame is a 2D polar image) is displayed using a carpet view representation. In various embodiments, filters may be applied to the 2D carpet view to remove noise or other undesirable features, resulting in longer processing times than when using 3D operators. For additional details on using the carpet view, see Figure 2A supply.

[0135] After the media and calcium detection process, each frame in polar coordinate space will have a corresponding frame / mask for the media M and calcium Ca regions / features of interest. These frames / masks may include lumen L and other categories used to train the neural network for ROI / FOI detection. A series of four output image masks / frames 190 are shown as an example. In various embodiments, the series of frames will include between about 400 and about 600 frames. Color coded notations have been used, which are shown in Figure 2AIn the frames and images in , blue is lumen, red is calcium, and green is media. When using OCT or IVUS data, the output image mask can cover the entire pullback map. In one embodiment, all frames of the media and calcium masks (also including the lumen in this example) are projected (as an OR operation) along an axis (e.g., x-axis or y-axis) to obtain a series of masks. The lines resulting from the projection operation are displayed as four lines 193. The color coding of the pixels in the projected lines is visible: green is media, red is calcium. In one embodiment, all lines are combined from left to right into a binary / binary media mask and a binary / binary calcium mask based on the order of frames from near to far.

[0136] This line projection combination 193 is displayed as a carpet view 195. Optionally, in some embodiments, the carpet view or line projection 192 is used to form the tissue atlas 198 shown. The outermost ring of the tissue atlas corresponds to the proximal direction, while the innermost ring shows the distal direction. In one embodiment, the tissue atlas 198 is formed by performing a polar coordinate transformation on the carpet view. The carpet view includes: 3D data (substantially including all frames from the pullback). In one embodiment, a binary / binary topographic reconstruction filter is applied to the medial membrane and calcium carpet view image 198 to remove noise and small structures. In one embodiment, such a carpet view filtering step, relative to a carpet view based on a predictive output frame / mask from a neural network, advantageously removes small detection areas in the carpet view image, while large detection areas remain unchanged.

[0137] In one embodiment, the processed carpet view image can then be applied to the median and calcium masks (e.g., by convolution, addition, or comparison processes) to remove noise in the 3D polar coordinate space. Limiting the operation on the 2D carpet view to 2D filters improves processing time relative to using 3D operators to operate on the 3D data space. Figure 2A The various processes and operations shown in can be implemented using computer-based systems and software modules (e.g., image processing pipelines and neural network-based feature / region detection and classification). In one embodiment, a method of detecting ROI / FOI (e.g., calcium, media, lumen, and intima) using a trained neural network includes: using an image processing pipeline instead of using a trained neural network to perform one or more of side branch detection, guidewire detection, and stent detection to increase the processing rate of a series of image frames obtained during an OCT pullback process.

[0138] In one embodiment, the carpet view is a 3D representation of the image frames of the intravascular pullback view. The intravascular image data obtained using the probe during the pullback process can be displayed to the user by unfolding the cross-section or polar view along the longitudinal view to form a representation of the scan line. The carpet view is a two-dimensional data representation. In one embodiment, the carpet view shows the cross-sectional OCT image, but unfolded or expanded in a manner similar to the unfolding of a rolled column of carpet.

[0139] The carpet view may be used to display each set of polar coordinate image data and its underlying components in one or more ways. For example, in one embodiment, the carpet view compresses the radial offset dimensions of the 3D intravascular data set into a single intensity. In this way, the data may be presented in (Z,θ) coordinates. In one embodiment, line projection may be used to generate the carpet view. In one embodiment, the radial offset dimensions are compressed by summing the intensity values ​​between the near and far offset estimates. This sum of intensity values ​​produces a detectable increase relative to a particular region in the carpet view / OCT image. These may be used to improve resolution and to obtain additional information from the output predictive mask (using, e.g., Figure 5E , 13A , one or more panels of the graphical user interface shown in 13B) to remove noise.

[0140] In one embodiment, the carpet view or OCT data may be used to generate a carpet view mask. A carpet view mask of intravascular data may be generated to facilitate filtering of the final predictive output (displayed via a user interface, such as displayed on a display). Figure 5E , 13A , 13B). The binarization process performed on the carpet view image is advantageous. In one embodiment, the carpet view is used as input to a subsequent intravascular image processing phase (wherein undesirable pixel debris or other noise is removed).

[0141] In one embodiment, the carpet view is a two-dimensional data set generated from a pulled-back scan line, where the offset scale along the scan line is removed or reduced. In the carpet view, shadow intensity values ​​are low and tissue intensity values ​​are high. The carpet view is typically a grayscale image, although a color version may be displayed in some embodiments.

[0142] In one embodiment, training and testing are performed using 3 or more AI processors (e.g., graphics processing units). In one embodiment, the training coding is implemented in Python using the PyTorch framework. Before the OCT images and masks are input into the neural network for training, all images are normalized to the range of (0.0 to 1.0). All images and masks are sequentially randomized and further divided into two parts. The first part is 90% of the total dataset and is used in training the model. During the training phase, the images and masks are vertically randomly offset between (-256, +256) to enhance the training samples for each full cycle. The second part is used to evaluate the performance of the trained model after each full cycle. In one embodiment, the cross entropy loss is calculated using a pixel-level operable Softmax function in the model output. The training and testing results are shown in Figure 2B and 2C The loss function refers to the actual cross entropy loss calculated by the predicted and ground truth values, such as Figure 2B The mismatch percentage is the percentage of misclassified pixels in each image compared to the ground truth. Figure 2B and 2C Both show a well-converged training process after 300 full cycles, without any overfitting problem.

[0143] As described herein, lumen detection is performed based on ground truth data, image data, and user image data (features / regions of interest to be detected / classified). In some embodiments, lumen detection is performed by analyzing scan line discontinuities (e.g., pairs of starts and stops). In other embodiments, lumen detection is implemented using a 2D or 3D neural network trained with annotated images (e.g., ground truth masks with identified lumen boundaries). Figure 2D Polar coordinate image data of the artery 230 is displayed, in which the lumen L and tissue T are identified. Figure 2E Shown is polar image data 233 annotated with ground truth lumen data and a mask having lumen L and tissue T corresponding to two different regions / channels of interest. Figure 2F Output image data 235 is shown as a result, lumen detection is performed by MLS with a 2D neural network. Figure 2G Output image data 240 is shown as a result, lumen detection is performed by MLS with a 3D neural network.

[0144] Figure 3A and 3B A user interface suitable for a system for browsing frames of image data and annotating the image data to generate ground truth is shown. Figure 3B350 is shown in FIG. 350 , where regions 357 have been defined as point-based ground truth values, for example by moving boundary points 360a, 360b, 360c (having pixels at the edges of the regions). In one embodiment, the raw image data can be annotated using various drawing and editing tools. These annotated images can be used to generate categories or features with annotated regions (using Figure 3A The actual mask of the interface 305 defined and stored in the memory. Figure 3A In , 271 frames of intravascular imaging pullback images are available for annotation. Frames 146 have been selected for annotation to provide ground truth data for training MLS. Figure 3B The user-selected area for annotation in corresponds to Figure 3A . In this way, any feature / class can be selected for identification and stored in memory along with annotations. The annotations include: pixel boundaries / regions defining a mask that is matched to a specific class or feature, such as lumen, media, calcium, etc., without limitation, and can include any class, feature, or region disclosed herein.

[0145] As an example, Figure 2C The graph shown is suitable for evaluating the change in training error over a certain period of time (zero to more than 600 full cycles). Figure 2C In the graph, the x-axis shows the number of full cycles of training, while the y-axis shows the dice coefficient (which is a measure of error). The blue and orange curves indicate the network performance on the training and test data.

[0146] In one embodiment, training data (such as real data) and image data elements (which may include various image formats, such as raw grayscale images) are provided as input to the CNN of MLS. MLS is run in periods of full cycles until the training error is reduced, minimized, or otherwise below a threshold. In one embodiment, the period ranges from about 100 to about 1000 full cycles.

[0147] In one embodiment, the training data (including the original image and the ground truth annotations) are in polar coordinates. The ground truth annotations can be enhanced by existing software algorithms that perform lumen detection. The lumen detection output from the software is combined with expert user annotations for media, plaques, and plaque types to generate a ground truth image. In one embodiment, the input to the MLS is the ground truth image and the output is the classified image. When the classified image is identical to the ground truth image, the error is zero.

[0148] Figure 1BThe use of the MLS 15 after it has been trained (performed using one or more training processes described herein or otherwise suitable for use with MLS and CNNs) is shown. After a given MLS / CNN has been trained, it can then process raw images and output image segmentations including classified or characterized tissue. In one embodiment, polar coordinate images from pullback maps of intravascular imaging probes are run through a trained MLS to generate a classified image. Figure 1B The system may include various pre-processing steps / operations 16 to further improve training and / or processing time. In one embodiment, the pre-processing includes a normalization step. In one embodiment, the normalization step includes normalizing the intensities of one or more or all of the image data, ground truth annotations, masks, frames, scan lines, neural network outputs, and other intensity-based data so that the intensities are normalized within a range (e.g., from about 0 to about 1, or another useful range).

[0149] An example of an unclassified / uncharacterized OCT image in polar coordinate format (corresponding to a cross-sectional image of a coronary artery) is shown in Figure 2D in the first image. Figure 2D The second image is the polar coordinate image plus a ground truth mask indicating the media ROI / FOI, Ca and lumen. Figure 2D The third image of is a polar coordinate OCT image including the inferred or predicted results, showing color-coded markers: green for media, red for calcium, and blue for lumen. An example of user image data obtained using an optical coherence tomography image that has been classified or characterized by the trained MLS is shown in Figure 3D and 3E And other figures. Figure 3D and Figure 3C By comparison, it is clear that both images are Cartesian images. In one embodiment, MLS training and prediction are performed on polar coordinate images. After classification occurs, the polar coordinate view is converted to a Cartesian view, and the annotations generated by the trained MLS are displayed on the image using color coding or other suitable notation or visualization. Figure 3D As shown in , the inner lumen area is shown in blue, the calcium-containing plaque area at the 9 o'clock position is shown in red, and the media area is roughly shown as the thin curved area on the right side of the image, which extends through the midpoint of the top and bottom of the lumen.

[0150] A given cost function provides a measure for evaluating the output of a machine learning system by comparing the real-world input / training set with the expected output when operating on patient data. The goodness of fit between the training data and the output data can be measured by a cost function. The output of the cost function can be a value corresponding to an error measure. This output is a comparative measure (e.g., a difference or gap) to summarize the success of the machine learning system or neural network or its other operational components in accurately predicting given the predictive results and the real-world values ​​used for system training. If the output of the cost function is zero, the system will work perfectly and effectively. In this way, iterative changes to the system can be used to reduce the cost or error function of the system and improve its prediction accuracy.

[0151] In addition, a pixel-based cost function is defined to measure the gap between the predicted and actual values ​​or another suitable measure or score. In one embodiment, back propagation updates each weight in the network-based value derived from the cost function. In one embodiment, the partial derivatives of the cost function are used to update the weights during the back propagation process. This weight update process has the benefit that the actual predictive results are closer to the actual values. This has the benefit of reducing or minimizing the error for each output neuron / node of the neural network. In one embodiment, the neural network is a convolutional neural network (CNN).

[0152] Figure 1C is a schematic diagram of a machine learning system 30, which includes a trained neural network 34, such as a network trained with an augmented data set of annotated ground truth masks. As shown, input image data (e.g., OCT polar coordinate image data in the example shown) is operated through the trained neural network 34. The input polar coordinate OCT image is operated to generate a prediction / predictive output in the form of a polar coordinate image, which includes color-coded markers corresponding to various arterial features / regions, such as media (green), calcium (red), and lumen (blue), as shown. An example of a ground truth image is also shown, which takes the form of annotated ground truth image masks, used to train the neural network 34 of MLS. In various embodiments, the output of the MLS system includes one or more of the following: arc-based metrics / measurements of similarity of Ca and EEL; detected EEL diameter; detected Ca depth. In some embodiments, these values ​​are measured with respect to the image data after classification of EEL, media, calcium, lumen, and other areas / features of interest.

[0153] Still see Figure 1C, once the MLS is trained, back propagation can be run from time zero to back propagation time BPT. In this way, predictions corresponding to annotated images or masks are generated by inputting raw images from the patient (e.g., from OCT pullback images) into the MLS and propagating the data through the network to generate predictive outputs. In one embodiment, the BPT time ranges from greater than zero to about 60 seconds. In one embodiment, the BPT time is less than about 180 seconds. In one embodiment, the BPT time is less than about 90 seconds. In one embodiment, the BPT time is approximately real time in various examples, as disclosed herein. Various neural networks 34 can be used to implement the systems and methods disclosed herein. These networks may include various hidden layers, functions, operators, and combined operators that can operate on multiple channels. In one embodiment, the channel corresponds to input image data, such as a series of pixels from a given frame or a sub-series of pixels from a given frame.

[0154] Figure 1D The MLS 55a is shown working in conjunction with the image processing components of the imaging system to generate results that are sent to the imaging system. In one embodiment, the image processing is OCT image processing. The MLS performs AI detection using a neural network. This can be performed by the AIP / GPU. The processing time ranges from about 20 to about 40 seconds in one embodiment. In one embodiment, the AIP is a GeForce GPU of model 1080 or higher. The AI ​​detection can be configured to detect lumen, calcium, EEL, and any ROI / FOI disclosed herein. Figure 1E The system 65a divides lumen detection into discrete processing stages (with processing time T3). Lumen AI detection can be performed using one or more AIP / GPUs. Figure 1E The system also includes various pre-processing steps to reduce downstream processing time when the patient is in the catheterization laboratory, and tissue detection / prediction is performed in preparation for surgery. Pre-processing can have a processing time. In one embodiment, pre-processing includes a data flattening step.

[0155] Finally, AI detection can be performed on the image data that has completed lumen detection and pre-processing. The AI ​​detection process can take time T4. In one embodiment, T3 ranges from about 1 to about 5 seconds. In one embodiment, T4 ranges from about 5 to about 12 seconds. In one embodiment, T4 ranges from about 5 to about 12 seconds. In one embodiment, the total processing time T2 ranges from about 8 to about 20 seconds. In one embodiment, T2 ranges from about 5 seconds to about 15 seconds.

[0156] In one embodiment, MLS and the method steps and operations performed may include semantic image segmentation, which processes OCT images using an end-to-end trained neural network. The training data includes: approximately 900 OCT images in polar coordinate space (with manually marked regions of interest), where each region of interest corresponds to a given ground truth value. Each given ground truth value can be mapped to a channel to be detected in the image data using the trained network. In one embodiment, each input image and its corresponding channel-specific mask are used to train the network using one or more algorithms. In one embodiment, a stochastic gradient descent optimization algorithm is used.

[0157] The present disclosure relates in part to methods for classifying tissue and different component layers and materials (each of the exemplary tissue types or tissue characteristics described above) and changes over time as described above. An exemplary output Cartesian image of a patient's artery that has been classified by MLS and one or more related methods is shown in Figure 3D and 3E In images B and D.

[0158] Figure 2D to Figure 2G Shown are: a polar coordinate image of an artery, a polar coordinate image annotated with real-world lumen data, lumen detection results performed by MLS with a 2D neural network, and lumen detection results performed by MLS with a 3D neural network.

[0159] The various layers of the coronary artery are displayed. The inner region P (corresponding to the shadow or reflection from the intravascular imaging probe P) is displayed in the lumen of the blood vessel L. In particular, the present disclosure relates to detecting and identifying tissue types and areas of interest and / or features about the artery, and displaying markers or other visualizations of such tissue types. Such predictions or inferences from the MLS, which are illustrated in the tissue characterization / classification artery image, help the end user facilitate diagnosis and treatment decisions. Some non-limiting examples of tissue types that can be detected using the methods and systems disclosed herein include: the inner region (where blood flows), the lumen, the inner surface, the media, the outer elastic lamina (EEL) (also known as the outer elastic membrane), the internal elastic lamina (IEL), the adventitia, plaque, calcium or calcified tissue, and others. The media is bounded by the IEL and the EEL. The intima is bounded by the lumen and the IEL.

[0160] The present disclosure relates to various embodiments in which one or more machine learning or artificial intelligence (AI) systems are used to detect or segment an arterial image or other structure into various component tissue types or regions of interest. The machine learning system is designed in part to enable it to be installed or combined with an imaging system, for example, an intravascular imaging system, an ultrasound system, or an X-ray system, such as an angiography or fluoroscopy system. In one embodiment, the present disclosure relates to using MLS to perform tissue characterization to detect one or more of the following: lumen, EEL, media, calcium / calcium plaques. Additional details related to these arterial layers and calcium plaques are shown in Figure 5G -5J.

[0161] In one embodiment, image data (which may be raw image data from one or more imaging systems (OCT, IVUS, angiography, X-ray, fluoroscopy, and others)) and ground truth annotations are in one or more coordinate systems. Each of the ground truth, ROI, FOI, and predictive results may correspond to a channel and have an associated image mask in a given training or patient image. Processing data in polar coordinate form (e.g., OCT data) is difficult for various systems.

[0162] Timely processing of image data using MLS is a practical necessity if real-time requirements are associated with generating classified serial image data immediately after acquiring such data from a patient who is still catheterized. To allow the use of machine learning and AI techniques, various pre-processing or MLS design options may be implemented to facilitate data processing times ranging from about 1 second to about 20 seconds.

[0163] In general, the systems and methods disclosed herein provide for various automated diagnostic tools to assist physicians in determining whether they should treat a given patient, and if so, which lesion / stenosis should be treated. In one embodiment, the system provides guidance based on physiology to treat the most severe lesions. In addition, details related to plaque type and other MLS detection features can be used to select the shortest stent that provides maximum blood flow restoration. In addition, virtual stent placement can be implemented, which provides interactive planning that allows stents to be modulated for placement in an artery (subject to tissue classification and other measurements as disclosed herein).

[0164] The systems and methods disclosed herein provide guidance to enable clinicians to access various stent placement areas. This can be supported by virtual stent placement, where virtual blood flow reserve values ​​help select lesion placement areas. This is further supported by registration of angiography and OCT data. Following the use of MLS detection, normal frames (frames with healthy tissue) will have fuller media / EEL coverage. In this way, MLS as implemented herein solves the problem of guiding stent planning and helping to improve patient outcomes.

[0165] Further, the automated EEL measurements obtained using the MLS described herein influence what stent size to consider and what type of stent to use. Calcium detection by MLS provides information for injury preparation and treatment options (e.g., choosing cardiovascular surgery instead of stent placement). In addition, calcium detection provides an input parameter in multiple choice decision making, and calcium detection will help decide whether to use BVS or DES.

[0166] FIG. 4A to FIG. 4C is an exemplary imaging and diagnostic system, including one or more machine learning systems, which are integrated or can be integrated or otherwise combined with the imaging system. Exemplary imaging systems may include: imaging systems based on OCT, IVUS, angiography, fluoroscopic contrast X-ray, etc.

[0167] See also Figure 4A , showing a data collection / imaging probe 407 and an imaging system 410 (including an MLS system 442), suitable for use as an intravascular diagnostic system / data collection system. System 442 may include: an AIP (e.g., one or more GPUs in series or parallel) and an AI memory AIP. The system may be used to implement one or more AI / ML-based techniques, systems, and methods described herein. System 410 may be used as a stent planning system for suggesting stent placement options and providing a graphical user interface for simulation or virtual stent planning based on flow measurements (e.g., VFR (volume flow), IFR (instantaneous wave-free rate), FFR, etc.). Some exemplary graphical user interfaces for such a system are shown in Fig.13A and 13B middle. Fig.14 B shows a cross-sectional view of the artery with measured and predicted VFR values. In addition, a mask has been generated using the trained neural network to show calcium (Ca / red), media (M / green), and lumen (L / blue). A simulated stent is also shown in Fig. 13B The longitudinal view of the artery representation in the user interface is shown in FIG.

[0168] In various embodiments, the intravascular probe 407 can be used to image the artery in the presence of an X-ray imaging system such as an angiogram. Other imaging systems such as computed tomography (CT) scans, magnetic resonance imaging (MRI), X-ray based imaging systems, and other 2D and 3D imaging systems can be used to generate and / or store image data in one or more memory storage devices. The probe 407 can include other imaging modalities such as OCT, intravascular ultrasound (IVUS), etc. in various embodiments. The probe 407 is in optical communication / connection with the intravascular diagnostic system / data collection system 410. The OCT optical system or subsystem 431 (connected to the probe 407 via optical fiber 414) includes: a light source (e.g., a laser), an interferometer (having a sample arm and a reference arm), various optical paths, a clock generator, a photodiode and other OCT system components.

[0169] The system 410 further includes one or more diagnostic software tools or modules 412 associated with MLS-based graphical detection. This software can be stored as non-temporary instructions on one or more memory devices (e.g., memory device 445) and executed by one or more computing devices (e.g., computing device 440) or MLS 442. The MLS includes one or more AI processors (AIPs) and dedicated memory AIPs. In one embodiment, the stent planning software tool may include: one or more vascular configurations (e.g., target configurations) generated by a user, a comparator or other comparison software routine (for comparing anterior and posterior stent configurations or other configurations). Generally, the software 412 can process a series of intravascular data and perform various method steps described herein, such as regarding Figure 1F , 2 , 2A, 2B, 4B, 4C, 12A, 12B and other figures illustrated and disclosed herein. In one embodiment, software 412 is stored in and executable by the AIP.

[0170] Software 412 is designed to operate on intravascular data sets and other related data from intravascular probes or other detectors or data sources (e.g., angiography systems). In one embodiment, vascular data can be recorded during the pullback process and stored in an electronic memory device. Training, pre-processing, field truth mask generation, detection / prediction tissue classification, neural networks and other features and software components can be run on the AIP or computing device 440. The software includes: various MLS training, pre-processing and prediction modules as shown. These may include:

[0171] Lumen contour prediction 12A, side branch prediction 12B, size adjustment 12C, image flattening 12D, lumen flattening, stent rib prediction 12E, user interface and input processing 12F, preprocessing 12G, MLS interface 12H, MLS memory manager 12I, GUI training module 412J for annotating image data (to generate ground truth masks), intensity normalization module 412K, etc.

[0172] In one embodiment, the software module is designed to operate on the intravascular data to characterize the tissue and identify areas of interest, such as calcium areas, tapered areas, lipid pools, and other tissue features. The software 412 may also compare the fractional flow reserve (FFR), intravascular resistance ratio (VRR), and other measured and calculated parameters of the intravascular data collection. To the extent such parameters vary from the stented state to the unstented state, such parameters may be used to generate one or more metrics.

[0173] In one embodiment, an OCT system 431 may be used. The system includes an optical receiver (e.g., a balanced photodiode based system) that receives light returned by the probe 407. The computing device 440 is, for example, a computer, a processor, an application specific integrated circuit (ASIC), or other device that is included as part of the system 410 or as a discrete subsystem (which is in electrical or optical communication / communication with the system 410) and receives electronic signals from the probe 7. The computing device 440 includes, in various embodiments: local memory, a bus, and other components suitable for processing data and utilizing software 444 (e.g., image data processing configured for stent visualization and stent malapposition detection). In one embodiment, a peripheral component interconnect express (PCIe) bus or other high bandwidth, low latency bus is used to connect the various components of a given imaging system, MLS, or a combination system including both.

[0174] Stent deployment planning tools 412 may be part of or exchange data with software 444. These tools may be used to place a virtual stent into a luminal region where probe 407 is positioned relative to the vessel wall. Fig. 13B An exemplary area of ​​a section of a pullback diagram is shown where one or more virtual stents may be deployed and displayed on a user interface.

[0175] As shown in 4A, a display 446 may also be part of the system 410 for displaying information 447, such as cross-sectional and longitudinal views of a vessel generated using the collected intravascular data. Once the intravascular data is acquired by the probe 407 and stored in the memory 445, it may be processed to generate and display information 447, such as cross-sectional, longitudinal, and / or three-dimensional views of the vessel along the length of the pullback region or a sub-region thereof. Two-dimensional or three-dimensional image masks may be used to display or store real-world data and predictive results. These views may be illustrated as part of the user interface shown and described below and in subsequent figures. The vessel images generated using the distance measurements obtained from the system 410, when processed using the tools and software modules described herein, provide information about the vessel, including lumen contours, vessel diameter, vessel cross-sectional area, placement area, and virtual stents defined by the placement area. In one embodiment, the MLS 442 includes one or more computing devices and one or more software programs or modules. The various devices, components / components, systems and subsystems disclosed herein are operable to perform the tasks, methods, steps, processes and other features described herein for the foregoing.

[0176] The MLS 442 may include: one or more AI processors and / or GPUs and / or processing cores and / or stream processors and dedicated memory for performing MLS training and prediction / inference when operating on patient image data (e.g., image data elements). More details of the MLS-based system are shown in Figure 4B middle.

[0177] like Figure 4BAs shown in, a combined imaging and MLS system 450 is shown. The system includes: a display 420, which can support annotation of raw image data with points and contours to store real data in the field to train the MLS. The display can be connected to the MLS425 or the imaging subsystem 430. The MLS 425 and the imaging system 430 have: a data transmission interface 465. In one embodiment, the MLS 425 includes one or more buses / motherboards 468 to provide communication channels and connections for various AIPs, memories and other components / components. In one embodiment, the MLS 425 includes multiple AI processors (AIPs). These can be arranged in a server configuration. As shown, three AI processors AIP1, AIP2, AIP3 are connected in series, in parallel or in a combination thereof. In one embodiment, the MLS and the imaging system are placed in a housing. The housing includes a cooling system CS to manage the high temperatures generated by the AIP. In one embodiment, the AIP includes one or more graphics processing units (GPUs). In one embodiment, the housing includes: a temperature sensor TS, which regulates the cooling system CS to ensure continuous operation of the AIP in the housing. In one embodiment, one or more AIP networks are connected to the imaging system and / or MLS and transmit the processed data outside the housing.

[0178] In one embodiment, the system includes a motherboard connected to the AI ​​processor. The motherboard is placed in the system housing. The housing can be a cart, including wheels to move the housing and its imaging and MLS system components in a catheterization room or another location. The system includes a probe interface unit 470, which includes a connector / coupler for an optical imaging probe (e.g., an OCT probe or an IVUS probe). This probe interface unit (PIU) is also referred to as a base in one embodiment.

[0179] In various embodiments, the trained neural network is executed on an AI / ML processor (e.g., a graphics processor / graphics processing unit) disposed in the housing of the imaging / data collection system. The AI / ML processor includes N parallel processors. In one embodiment, N ranges from about 2000 to about 2500. In one embodiment, N ranges from about 2000 to about 3000. In one embodiment, N ranges from about 3000 to about 4000. In one embodiment, N ranges from about 4000 to about 5000. In one embodiment, N ranges from about 5000 to about 6000. Multiple parallel processors may be arranged in groups on various hardware elements referred to as computing units. In one embodiment, the range of computing units is: for a given AI processor, about 20 to about 80 computing units are arranged. Each computing unit may have multiple parallel processors. In one embodiment, a unified computing device architecture (CUDA) is used, wherein the range of CUDA cores is from about 2000 to about 10000 cores. In one embodiment, a unified computing device architecture (CUDA) is used, wherein the range of CUDA cores is from about 2000 to about 2500 cores. Examples of suitable parallel processors include, but are not limited to, CUDA core processors and tensor core processors from Nvidia, Inc., stream processors from AMD, Inc. In one embodiment, between about 200 and about 300 tensor cores are included in the AI ​​processor / graphics processor used.

[0180] In one embodiment, the AI ​​processor memory (AIP) is greater than about 8 GB. In one embodiment, the AI ​​processor memory is greater than about 16 GB. In one embodiment, the AI ​​processor memory is greater than about 32 GB. In one embodiment, the AI ​​processor memory is greater than about 64 GB. In one embodiment, the AI ​​processor memory is greater than about 128 GB. In one embodiment, the AI ​​processor memory ranges from about 4 GB to about 256 GB. In one embodiment, the AI ​​processor memory ranges from about 8 GB to about 128 GB. In one embodiment, the AI ​​processor memory ranges from about 16 GB to about 64 GB. In one embodiment, the AI ​​processor memory ranges from about 8 GB to about 32 GB. In one embodiment, the AI ​​processor memory ranges from about 32 GB to about 64 GB. In one embodiment, one or more of the electronic memory storage devices include NVMe TM The interface is used to increase processing speed to reduce data analysis time for MLS operations while the patient is in the catheterization lab. In one embodiment, 16GB or more of on-board RAM is used on the motherboard of the imaging system / data collection system. In one embodiment, 32GB or more of on-board RAM is used on the motherboard of the imaging system / data collection system.

[0181] In one embodiment, the AI ​​processor memory ranges from about 1 GB to about 2 GB. In one embodiment, the AI ​​processor memory ranges from about 2 GB to about 4 GB. In one embodiment, the AI ​​processor memory ranges from about 4 GB to about 6 GB. In one embodiment, the AI ​​processor memory ranges from about 6 GB to about 8 GB. In one embodiment, the AI ​​processor memory ranges from about 8 GB to about 10 GB. In one embodiment, the AI ​​processor memory ranges from about 10 GB to about 12 GB. In one embodiment, the AI ​​processor memory ranges from about 12 GB to about 14 GB. In one embodiment, the AI ​​processor memory ranges from about 14 GB to about 16 GB.

[0182] Figure 4C A system 475 is shown having software modules adapted to operate on image data via two alternative process flows to identify / segment features / regions of interest with respect to the image data. In one embodiment, the image data is arterial image data, such as polar coordinate OCT image data, although other image data from different image sources may also be used. In the above process flow alternative, artificial intelligence-based lumen detection 487 is performed, such as by using a trained NN on lumen boundaries / lumen masks (as ground truth). The AI ​​lumen detection runs on an AI processor (e.g., a GPU or other processor with dedicated memory). The image data and masks or other representations of detected lumen boundaries for the image data frame are then processed by the NN using forward and back propagation to segment / classify the image to generate categories, features of interest, regions of interest, and other detections / predictions for which the NN was trained. As a result, a predictive result (e.g., an output predictive image mask) is generated 495.

[0183] In contrast, in other embodiments, other detections / predictions are performed in a different order and sequence prior to using any initial detections (e.g., side branches 479, guidewires 481, and lumens 482, as well as arterial image data, as inputs to the MLS for processing the arterial data) and the results of previous detections (using the trained MLS). In this way, the MLS-based detection / prediction of the class (FOI, ROI, etc.) of a given series of pixels is determined with respect to the image data and the previous series of detections. Previous detections (e.g., side branches, guidewires, lumens, etc.) may be stored as image masks. In one embodiment, the MLS is trained on a field-real image data set that includes the implemented previous detections (e.g., side branches, guidewires, and lumens in the exemplary embodiment shown).

[0184] The present disclosure relates in part to automated methods for detecting calcium-containing tissue (e.g., calcified tissue or calcium plaques or other areas of interest) and identifying associated nodules / areas of interest for cardiovascular surgery, and to substantially real-time guidance methods for performing cardiovascular surgery using lasers. In various embodiments, an OCT imaging probe is in optical communication / connection with an imaging laser (e.g., a swept source laser). As a result, there is an optical path extending through the probe to a light directing element (e.g., a single lens, a gradient refractive index (GRIN) lens, or a beam guide) whereby light is received from the imaging laser, transmitted to the tissue through one or more of the aforementioned optical elements, and then the light from the tissue is received and transmitted back to the OCT imaging system, where it interferes with the light generated by the imaging laser.

[0185] Figure 5A is a schematic diagram of an exemplary system for performing OCT-guided cardiovascular surgery. Figure 5B is a schematic diagram of an exemplary method for performing OCT-guided cardiovascular surgery. Figure 5A , a system 500 is shown for detecting features of interest via an OCT system 505, including one or more MLS systems (including feature detection software, such as a trained NN). An imaging probe 510 includes an optical fiber that is part of a sample arm of an interferometer. A reference arm of the interferometer is included in or communicates with the OCT system 500. A patient interface unit 470 includes a swivel coupling or other mechanism to couple the imaging probe and its optical fiber to the OCT system 505.

[0186] One or more optical switches 520 may be in optical communication / connection with the PIU 520. The optical switches 520 may be controlled by control software or a controller. Further, the OCT system includes an imaging laser 527 and includes or communicates / connects with an ablation laser. The control software or controller 535 allows the optical switch to switch between the imaging laser (for OCT imaging) and the ablation laser (for target ablation), using GRIN lenses, beam guides, microlenses, or other optical elements to guide light in the probe 510. Further, the MLS and its detection software (e.g., calcium detection software 525) can be used to identify areas of interest containing calcium or other materials, thereby making ablation a preferred treatment option.

[0187] The present disclosure relates in part to calcium detection using an MLS system (e.g., a deep learning based artificial intelligence system) in conjunction with OCT software or other imaging software. MLS highlights areas where calcium plaques or other ablation targets are present. The deep learning network / MLS is trained using 450 to 100 annotated OCT images marked with calcium areas by expert users. This trained network is then fed with raw OCT images that have been pixel-wise segmented by the deep learning network.

[0188] In one embodiment, calcium detection using a trained MLS system (e.g., deep learning-based artificial intelligence) in conjunction with OCT software is an aspect of the present disclosure. MLS highlights areas where calcium plaques or other ablative tissues are present. The deep learning network is trained using 450 annotated OCT images marked with calcium areas by expert users. This trained network is then fed with raw OCT images that have been pixel-wise segmented by the deep learning network.

[0189] Figure 5C and 5D Shown is an OCT image segmented using the MLS system to highlight areas of calcium plaque. Figure 5E It is a user interface that displays the detected calcium (e.g., calcium plaque) area in three-dimensional, cross-sectional, and longitudinal views. Fig. 5F It is a calcium map, which is formed by registering the OCT data with the angiography data, and then displays the frames containing calcium relative to the angiography data. This reflects the calcium plaques detected in the OCT to the angiography graphical user interface for review and planning. The calcium areas detected in the OCT pullback image are also reflected on the angiography data for the physician to review. The physician then identifies the calcium nodule areas of concern for candidates for cardiovascular surgery or ablative surgery.

[0190] like Figure 5B , a method 560 for detecting and ablating tissue of interest using an OCT imaging probe is described. Initially, a first OCT pullback 562 is performed. Calcium detection is performed 565 using the initial series of OCT data using an MLS embodiment or other calcium detection method. OCT data is collected in an angiographic imaging session. As a result, the OCT image and the angiographic image are registered 573. A calcium map is generated (e.g., in Figure 5D ) 575. Next, the clinician reviews the image data and calcium detection and identifies plaques that warrant ablation 577. Once the area for cardiovascular surgery has been identified, a second OCT pullback is performed 579. During this acquisition, calcium detection is also performed 581. The imaging / ablation system uses the calcium map and target calcium nodules to align the high power laser used for cardiovascular surgery as part of software-based control 583. It also performs calcium detection in real time to correct for any alignment changes between the first OCT pullback image and the current image. This is achieved by narrowing the frame search window to be considered. In one embodiment, when the catheter tip reaches the target calcium plaque nodule, the high power ablation laser is turned on 584 to provide a short energy pulse to evaporate and remove the calcium nodule. The ablation procedure is controlled by the user in various embodiments to perform cardiovascular surgery 587.

[0191] Figure 5G and5H Annotated Cartesian OCT images according to an exemplary embodiment of the present disclosure are shown. They show the location of the imaging probe P in the lumen, vessel wall W, adventitia AD, and details related to calcium plaques and calcium angles. In one embodiment, calcium angle measurements can be detected and output using MLS. Fig.5I A magnified Cartesian OCT image identifying arterial layers is shown according to an illustrative embodiment of the present disclosure.

[0192] Figure 5J Showing an exemplary embodiment according to the present disclosure Figure 5C A histological image of an artery in which the arterial layers are identified is part of a ground truth or training set review suitable for annotating image data for a given MLS embodiment. Figure 5G and 5H , a cross-sectional image of an artery obtained by using an intravascular probe P is shown in a living environment relative to a blood vessel B having a blood vessel wall VW defining a lumen L. The elliptical / circular reflection and shadow of the blood vessel B also include side branches SB.

[0193] Fig. 6A and 6B A Cartesian OCT image and the predictive output of MLS identifying the lumen (blue markers), tunica media (green markers), and Ca / calcium plaques (red markers) are displayed according to one or more embodiments disclosed herein, and an exemplary diameter measurement of 3.31 mm (obtained using an MLS-based system with a trained neural network (NN)) is shown. Figure 6C and 6D An initial Cartesian OCT image and a polar coordinate image generated by using scan lines collected by the imaging probe are shown. Fig. 6E Solid-state true masks for Ca, lumen, intima, and media are shown. Fig. 6F Shown are the predictive results output from the deep learning MLS of the present disclosure.

[0194] Fig. 7A An initial exemplary OCT image in Cartesian format obtained using a combined imaging and MLS based system is shown. The imaging probe is shown in the upper right quadrant of the image. The black lumen area is in the middle. Figure 7B Shows the corresponding Fig. 7A Grayscale polar image of the Cartesian image shown. Figure 7C and 7D Displays the ground truth mask, and Figure 7BThe deep learning output of an MLS embodiment generated by annotating the image as input, training, and then using the trained MLS for prediction. Each mask or detection corresponds to a channel. The present disclosure relates in part to a multi-channel MLS designed for tissue classification using image masks corresponding to ground truth values ​​and predictive results.

[0195] In one embodiment, semantic segmentation is performed on multiple channels corresponding to multiple arterial features (e.g., tissue type, calcium, side branches, lumen, guidewire, intima, media, calcium, fibrous, stent, stent ribs, stenosis, and other arterial features). Ground truth values ​​for various features are used to train a convolutional neural network. In one embodiment, two neural networks are used, with the first neural network used for lumen detection and the second neural network used to detect other arterial features after lumen detection has been performed by the second neural network. In one embodiment, the first, second, or both networks are convolutional neural networks. In some embodiments, lumen detection is performed on image data before detecting other features of interest (e.g., calcium, media, intima, and other features disclosed herein). Lumen detection can be implemented using various systems and methods, including those disclosed in U.S. Patent No. 9,138,147, entitled "Lumen Topography Image Reconstruction Based on OCT Scan Line Data," filed on September 22, 2010, the details of which are incorporated herein by reference in their entirety.

[0196] In one embodiment, a first neural network and a first image processing pipeline are used. Advantageously, it is desirable to reduce the time that a patient is catheterized in an imaging session, thereby reducing the time to generate an output from a classification system for a region / feature of interest. Therefore, in some embodiments, a neural network is trained using frames of image data annotated as M and / or M or more types or categories of features / regions of interest. In some embodiments, M ranges from 2 to 3. In some embodiments, M ranges from 2 to 4. In some embodiments, M ranges from 2 to 5. In some embodiments, M ranges from 2 to 6. In some embodiments, M ranges from 2 to 7. In some embodiments, M ranges from 2 to 8. In some embodiments, M ranges from 2 to 9. In some embodiments, M ranges from 2 to 10. In some embodiments, M ranges from 3 to 4. In some embodiments, M ranges from 3 to 5. In one embodiment, M is 3 and the types / categories used to train the neural network are calcium, lumen, and media. The ground truth annotation / ground truth mask used to train the neural network includes a mask having a set of annotated regions, wherein at least three of the annotated regions on one of the ground truth masks include media, calcium, and lumen annotations. In various embodiments, any set of types and categories of features / regions of interest may include any of the types and categories disclosed herein.

[0197] In various embodiments, the use of an image processing pipeline and a trained neural network supports accelerated processing of image data through each of the above. In addition, training a neural network to predict features / regions of interest through annotations of all possible detectable features in an image leads to one or more of the following problems: excessive network training time, excessive network complexity, and excessive processing time when using the network prediction results.

[0198] Fig. 8A Preprocessing operations performed on a polar coordinate OCT image 800 prior to annotation with expert data / ground truth data are shown. As shown, as part of the preprocessing step, depth pixels that do not encode or capture useful imaging data or detectable tissue or ROI or FOI are effectively excluded by reducing or constraining the depth scale to generate reduced image data elements 805, which can be used for iterative enhancement or direct annotation with expert ground truth data. In one embodiment, multiple data enhancement processes are used to transform an image or image data element into multiple versions thereof, such as by circular rotation, left-right flipping, and other image-based transformations.

[0199] For example, if the original image is rotated circumferentially by multiple phases, such as 90 degrees, 180 degrees, and 270 degrees, then these new versions and the original image or image data elements constitute 4 versions. Each of these 4 versions can be flipped left and right to produce 8 versions or enhanced forms. In turn, each of these can be annotated 810 with ground truth annotations, such as by using Figure 3A and 3B The user interface software shown in is implemented, whereby the user can select series of pixels, pixel region boundaries and other geometric elements to annotate the image data with real-world information (e.g. calcium classification, lumen, side branches, etc.). In turn, these can also be evaluated by post-training MLS to measure detection accuracy through a cost function or other metric. Statistical analysis of the average of each result or multiple versions / enhanced forms can be used to improve detection accuracy.

[0200] Figures 8B to 8DVarious exemplary image masks are shown for detecting lumen, media, and calcium according to illustrative embodiments of the present disclosure. Various image masks may be used, such as lumen, intima, media, calcium, fibrous, and masks for any ROI or FOI disclosed herein. The mask may be a binary mask in one embodiment. Further, in one embodiment, each mask may be used to reflect ground truth detection. In some embodiments, a given ground truth mask includes: one or more regions corresponding to ground truth categories (e.g., calcium, lumen, intima, etc.); and other mask regions that do not correspond to one or more of the categories. In one embodiment, the ground truth mask includes: a first series of pixels corresponding to one or more categories; and a second series of pixels that do not correspond to the one or more categories. The ground truth mask may be used to train a neural network for a given MLS, for example Figure 1C and 2 . In one embodiment, resizing and other steps (e.g., flattening and other steps described herein) are performed prior to generating an enhanced form or additional version of a given original image. A given mask may include a subset of pixels corresponding to a particular channel, wherein the channel corresponds to a feature of interest, such as calcium, intima, and other features disclosed herein. The mask may identify pixels for a given channel that are used to train ground truth values ​​or as a result of operating on image data using a trained MLS to produce an output. In one embodiment, multiple masks are associated with multiple channels on a one-to-one basis, and processing of a given image assigns pixels to a given channel, such as a given region of interest in a mask. In one embodiment, the ground truth annotation includes multiple channels of information. For example, a given ground truth annotation / mask used to train a neural network includes: calcium region, media region, lumen region, and intima region.

[0201] Figures 9A to 9C is a polar OCT image showing the circular rotation, left flip, and right flip applied to the original OCT polar image as part of the preprocessing steps performed prior to training the MLS neural network. Figures 9A to 9C In the present invention, various pre-processing steps are performed on the image data (e.g., the original polar coordinate OCT image). The result of these processing steps or transformations is that one original polar coordinate image is transformed into 7 additional variants or versions. This data enhancement process can be implemented in various ways. Fig. 9B A circular rotation of the original image of 9A is shown. Fig. 9C The left-right flipping of the transformation rotation / shift map of 9B is shown. A given augmentation such as this can result in 8 image variants of a single frame. This includes 1 initial map and 3 circular rotation maps (90 degrees each). Each of the 4 copies is flipped left-right, resulting in a total of 8. These operations, additional operations, or a subset of them can be used to preprocess data for training and / or prediction using MLS.

[0202] Figures 10A to 10B The images show the ground truth annotated images and the predicted output respectively. Fig. 10A The inner blue solid state real math is bounded by a lighter middle membrane layer. The yellow layer may be encoded as another channel or classification, such as lipid LP. The orange region corresponds to calcium. In some embodiments, it is desirable to implement a penalty or constraint system to avoid "false positives". Fig. 10B , background channel B is shown. Identified as background in the prediction results, this can be used to effectively prevent the background from being misclassified as one of the other categories or types of interest. In various embodiments, channels can be established using features corresponding to imaging artifacts, background, non-uniform rotation distortion effects, etc., which can be generated using real-world annotations. Thus, in some embodiments, a neural network can be trained to identify artifacts, imaging system errors, false positives, and other imaging phenomena that are desired to be screened out or avoided from being displayed in a given user interface imaging data representation. A mask showing the prediction results is shown in Fig. 10B middle.

[0203] Figures 11A to 11D The images show the various pre-processing steps that have been developed to increase the speed of MLS processing and reduce patient waiting time. Fig.11A As shown in , an OCT image with ground truth values ​​displayed by a color-coded mask (green for media, red for Ca, blue for lumen) shows that along the 976 pixels in the vertical direction, there are regions, such as region 300 along the depth scale, where there is a lack of useful data. The image can be flattened, such as Fig. 11B The white lumen outlines in the middle and top regions of the polar coordinate image are shown in Fig. 11B This flattening allows MLS / NN to operate on image regions in a transformed state that may contain relevant tissue data. Similarly, Fig. 11C In the example, half-resolution resizing is performed, for example by skipping A lines (scan lines) to generate an intermediate image. Further, depth pixels in the image are skipped or excluded to further resize the image to obtain Fig. 11C These steps improve the MLS procedure and can be performed in real time or substantially real time while the patient is catheterized following OCT or IVUS images. In this way, these procedures can save time and improve patient outcomes. Fig.11D The flattened OCT image 244 is shown, along with the same flattened OCT image with the inferred or predicted results (in the form of an image mask showing the red calcium plaque areas, the blue lumen areas, and the green media areas). Flattening may be performed as a pre-processing step in some embodiments. This polar coordinate image may be converted to Cartesian form and displayed to the end user in a diagram (which will be similar to Figure 6BCartesian image shown).

[0204] Fig. 12A and 12B is a flow chart of exemplary processing steps whereby raw image data is routed and processed together with ground truth data annotations. Fig. 12A In the embodiment of the present invention, one or more of steps A1 to A8 may be performed using a system based on one or more computing devices disclosed herein. Fig. 12A As shown in , according to one embodiment, the system and method may perform one or more of the following schemes. Through a computing device or AIP, the imaging system and / or MLS may perform multi-point annotation of image data or image data elements. The annotation GUI helps to place and connect the points to define boundaries that can be converted into an image mask (which effectively identifies a given tissue type in the image). The method may include: detecting / updating lumen detection conditions, such as lumen boundaries or contours. The method may include: transmitting annotations, lumen detection conditions, raw image data or data elements from a system (such as an imaging system) to an interface or intermediate system (providing input to the MLS / neural network). In one embodiment, resizing of image data elements is performed to remove useless image data to improve processing time. In one embodiment, the method includes: generating a binary / binary mask for tissue identification / annotation. Further, the data may be formatted into an array or another format for input. Finally, the input data (which may include the image mask and the resized raw image data or data elements) is input to the MLS / NN for processing, such as training or prediction.

[0205] Fig. 12B is another embodiment, including various steps that can be performed by various systems disclosed herein. Fig. 12B In the embodiment of the present invention, one or more of steps A1 to A8 can be performed using a system based on one or more computing devices disclosed herein. The steps may include one or more of the following. One or more system components, such as an imaging system (OCT, IVUS, X-ray, angiography, etc.) acquire a raw image data set. Various pre-processing steps may be performed. Flattening of the lumen and / or image region may be performed on the raw image and the mask image with real-world data. In one embodiment, every other scan line is removed. In one embodiment, the lumen outline or boundary is flattened to emphasize meaningful ROI / FOI data, such as in Fig. 11B and 11D In other embodiments, the images may be rotated in a circle, flipped left to right, or transformed in different ways. This may increase the number of items in a given training set or form different versions for MLS / NN operations so that ML results can be monitored and tested.

[0206] The following steps may be performed: cropping, skipping or removing scan lines / data / pixels from the original image and the annotated mask image to adjust the rows (in polar or Cartesian form). Fig.11A As shown in , pixels corresponding to depth along the vertical axis can be removed to adjust the image size, as shown in Fig. 11B and 11C In addition, it may be useful to add scan lines / data / pixels to normalize the columns of the original image and the annotated mask image for MLS / NN processing. The method may include: flattening the original image data and the mask image data and inputting them to the MLS / neural network for predictive analysis or additional training as needed.

[0207] Fig.13A and 13B Exemplary user interfaces are shown for diagnostic tools to support stent planning through virtual stent placement, including tissue classification. These user interfaces help illustrate the MLS analysis (which can identify the most severe injury based on physiology and / or the shortest stent that provides maximum blood flow restoration). In one embodiment, the white stent area (VS) formed by the semicircle shown can be positioned relative to the luminal representation to perform interactive virtual stent placement. Interactive virtual stent placement allows the stent to be modulated for the injury. The user is provided with predicted VFR values ​​and current VFR values. The minimum lumen area (MLA) graph in the bottom panel shows the percentage diameter stenosis and the minimum lumen area, and also shows the VFRp and VFR values ​​that vary based on stenosis and / or virtual stent selection. The angiographic image is displayed in the upper right panel, registered with the Cartesian or polar coordinate views or longitudinal views of the artery in the other panels. In Fig. 13B In FIG. 5 , the predicted VFRp and the current VFR are displayed relative to the virtual stent (VS) and the Cartesian artery image (which has been masked or characterized by various markers corresponding to the lumen, calcium plaques and media ROI / FOI).

[0208] The methods and systems disclosed herein provide diagnostic and planning tools to the user. For example, the methods and systems include tools that enable virtual stent placement into an artery to be automatically performed based on image data from a pullback map. More details on such diagnostic graphical user interface-based tools can be found at Fig.13A and 13B An exemplary user interface for a combined imaging and machine learning system suitable for use in such diagnostic planning is shown in FIG. Fig.13A and 13B Further, the automated placement of such stents includes processes, user interfaces, and associated software-based features to display such stents in optimized locations and at sizes suitable for stents identified by an end user.

[0209] The present disclosure includes various embodiments of stent planning software for placing stents in optimized positions or otherwise in positions that optimize specific parameters. In one embodiment, the parameters optimized to facilitate stent planning include: blood flow, which can be achieved by deploying a stent of a specific length. The distal and proximal placement area positions and stent sizes of the stent are provided to the end user. These are determined by optimizing blood flow improvements (which can be achieved using a range of possible stents and stent deployment positions).

[0210] In one embodiment, the training data is formed using real expert guidance in the field or guidance from MLS (e.g., for lumen detection). The training data is separated into masks. In one embodiment, the mask is one or more independent channels. They are used to train the training set. When the training set is run through the neural network, each of these channels will basically contribute different weights and filters of the network. In this way, the NN adapts and changes in response to the mask / training data. Furthermore, when the patient sample data is input into the trained network, the original image data is marked or identified with different channels, requiring these channels to correspond to the different features used in the training set, such as images of lumen, intima, intima, media, adventitia, lumen, EEL, IEL, plaque, calcium, calcium plaque, stent, calcium, guidewire, etc.

[0211] Generally, the MLS system disclosed herein involves a multi-channel segmentation process where each tissue type, region of interest, arterial layer, etc. is processed as a separate data channel with its own image mask for generating training sets and predictive outputs.

[0212] As an exemplary method to assess blood flow restoration due to stent deployment, the method described in U.S. Patent Application No. 14 / 115,527 (entitled "Method and Apparatus for Automated Determination of Lumen Contour of Stented Vessels," the contents of which are incorporated herein by reference in their entirety) may be used. Other methods may be used, including those otherwise described herein. In order to understand some aspects of blood flow changes and behavior in an artery, it is helpful to consider Fig.13A and 13B , which displays stenosis and various features related to virtual stent selection and position based on the identified landing zone and stent length.

[0213] In various aspects, the present disclosure relates to training one or more of a machine learning system, a neural network, and a convolutional neural network using polar images of coronary arteries. In various aspects, the present disclosure relates to training one or more of a machine learning system, a neural network, and a convolutional neural network using field-truth annotations of polar images of coronary arteries. In various aspects, the present disclosure relates to training one or more of a machine learning system, a neural network, and a convolutional neural network using field-truth annotations of polar images of coronary arteries, wherein various tissue types and features of interest are annotated in a training set.

[0214] In various aspects, the present disclosure relates to training one or more of a machine learning system, a neural network, and a convolutional neural network using ground truth annotations of Cartesian or non-polar images of coronary arteries, where various tissue types and features / regions of interest are annotated in a training set. In various aspects, the present disclosure relates to classifying / characterizing input patient data in polar or non-polar or Cartesian form using a trained machine learning system, neural network, and / or convolutional neural network, and combinations thereof, to identify various tissue types and features / regions of interest using ground truth annotations of polar images of coronary arteries, where various tissue types and features of interest are annotated in a training set.

[0215] The present disclosure relates to the use of real-world tissue types and features / regions of interest, and the classification, identification and / or characterization of various tissue types and features / regions of interest in patient image data and image data elements. In various embodiments, aspects, and for one or more MLS embodiments, tissue types and features of interest (FOI) / regions of interest (ROI) may include one or more of the following Cartesian, polar, non-polar coordinate images or local images: coronary arteries, coronary arteries, OCT images, IVUS images, X-ray images, ultrasound images, angiographic images, lines or curves of arterial trees, side branches, lumens, guidewires, stents, imprisoned stents, bioresorbable vascular scaffolds (BVS), drug eluting stents (DES), fibrous , halo artifacts, pressure wires, lipids, calcium, atherosclerotic plaques, stenosis, plaques, calcium, calcified plaques, calcium-containing tissue, lesions, fat, poorly placed stents; under-expanded stents; over-expanded stents; radio-opaque markers; branching angles of the arterial tree; calibration elements (e.g., polyethylene terephthalate (PET) doped membranes); sheaths; doped sheaths; fiducial alignment points; changes in arterial diameter; changes in arterial radial direction; flow measurements obtained using imaging data; FFR values ​​for images; branching models; combinations of the foregoing and classifications or types of the foregoing.

[0216] In one embodiment, the probe includes: a probe tip that includes an optical fiber or is in optical communication / communication with an optical fiber. The optical fiber and the probe tip are disposed in one or more sheaths (e.g., a catheter sheath). The probe tip may include various elements, such as an angled beam guide or a lens cap and a transducer / sensor for other imaging modalities. The optical fiber of the probe 407 may also include a torque wire disposed around the optical fiber. The probe emits light in the lumen L and receives light scattered from the vessel wall VW.

[0217] In one embodiment, the optical fiber is part of the sample arm of the interferometer. A data collection probe 407 (e.g., an OCT probe) can be used to collect depth information suitable for imaging a sample (e.g., a blood vessel). For example, a series of image data frames are, for example, polar coordinates or Cartesian images shown in 3D. Each image is generated as shown based on the optical signals sent and received by such a probe 407. A cross-sectional image of a blood vessel is formed by scanning lines collected when the probe rotates (see Cartesian images shown in 3D, 3E, 6A, 6B). The cross-sectional images (e.g., images 6A, 6B) are Cartesian images. Figure 7B and 8A An example of a polar view or polar form of a medical image is shown. Cartesian images and polar images may be referred to as frames in various embodiments.

[0218] OCT images (e.g. Figure 3B , 3C , 3D cross-sectional images) are typically acquired one scan line at a time. In one embodiment, a series of samples along a line starting from the center of the catheter toward the maximum imaging depth is called a scan line. In one embodiment, the smallest data unit in an OCT image is called a sample. A series of samples along a line starting from the center of the probe toward the maximum imaging depth is called a scan line. OCT images are typically acquired one scan line at a time. Cross-sectional images can be formed from a series of scan lines collected when the probe rotates. Further, in order to image a segment of an artery or other blood vessel, the catheter moves longitudinally during rotation. In this way, the probe acquires a series of cross-sectional images by rotation. The image originates from each scan line associated with a slice of the blood vessel or artery of interest. The scan lines are arranged at angles between them, similar to spokes on a wheel. In one embodiment, the scan lines are acquired in polar coordinate format.

[0219] User interface related to tissue feature maps

[0220] The present disclosure relates in part to medical diagnostic systems, including: image browsing, user interface design, time-saving enhancements, and other design considerations. The foregoing features are disclosures developed to improve time and information management issues during time-critical medical procedures (e.g., procedures performed in a catheterization laboratory). The present disclosure describes ways to convert tissue characterization results into tissue atlas displays. Tissue characterization can be performed using a variety of techniques, including pathology, artificial intelligence techniques, machine learning techniques, attenuation and backscatter-based techniques, image processing detection and enhancement techniques, and other techniques that may exist, are being developed, or have been developed.

[0221] The tissue atlas embodiments disclosed herein provide a user interface to enhance viewing of image data including various characterizations of tissues and boundaries and their relative arrangement with respect to one or more portions of an object (e.g., an artery, blood vessel, or other body lumen). Physicians, researchers, technicians, and other end users can engage in planning decisions more quickly and make rational decisions based on diagnostic information given a series of images with tissue characterization regions having more rational correlation factors than would otherwise be possible.

[0222] Tissue characterization typically includes one or more methods to determine tissue type or other tissue properties for a given series of patient tissue. Thus, if an artery is imaged, multiple images form a series of image data, and the tissue in each image can be characterized to determine whether it is a particular type, such as fat, muscle, at a high level or at a more granular level (e.g., calcification), endothelium, EEL, and other types as disclosed herein or other types present in the imaged subject. Typically, one or more detection processes are used to detect tissue type at the tissue in a region of the image. The calcium plaque mask and the media mask can be applied to a series of image frames after a machine learning inference process or another detection process (e.g., an image processing pipeline-based process detects calcium and media in the image data). By applying each mask along the same lines as previously described with respect to Figure 2A A-line projection can obtain a ring structure. Figures 16 to 18 Various tissue atlas representations are shown according to an exemplary embodiment of the present disclosure, generated using OCT imaging pullback of an artery, wherein various markers are integrated into a user interface displaying the various tissue atlases to support diagnosis and treatment planning, such as stent placement and cardiovascular surgery.

[0223] exist Fig.14 , 15 In , 16, 17, and 18, each ring on the plane indicates a tissue characterization result frame from the pullback OCT data. Fig.14In FIG. 1 , a three-dimensional representation of an artery 1405 is shown with a guidewire GW, areas of calcium that have been detected using MLS or another method, and three cross-sectional boundaries R1, R2, R3, as shown, which are projected into a plane to generate a tissue atlas TM. Another version of the tissue atlas is also shown in FIG. Figure 2A Neutralization Fig.16 , 17 middle. Fig.15 Schematic representations of various slices / rings / boundaries R1 to Rn are shown, which are displayed along a 3D artery representation 1425 (extending in the proximal-distal direction). In one embodiment, imaging data (such as OCT, IVUS, X-ray or other imaging modalities) can be used to generate such a 3D representation of the artery 1425.

[0224] Intravascular optical coherence tomography (OCT) images provide high resolution visualization of coronary artery morphology. The present disclosure relates in part to automatic detection and / or classification of internal coronary plaque (calcium, lipid, fibrosis, and thrombus). In one embodiment, internal and external calcification boundaries are also detected and displayed. The process of detection and classification can enhance the interpretation of OCT images and provide targeted information to the diagnostician. The present disclosure relates in part to systems and methods for displaying the results of data analysis (applied to intravascular data sets) to the user in a manner that is clear, easy to interpret, and useful for subject diagnosis (e.g., Tissue Atlas™ and Figures 14 to 18 ).

[0225] The present disclosure describes, in part, a graphical user interface (GUI) that provides a user interface and graphical data representation that can generate a total tissue atlas from a series of imaging data for a given artery or another body lumen (bowel, esophagus, etc.). The tissue atlas is responsive to user selections, such as user selections of a particular region of the tissue atlas. In response to a user clicking or otherwise selecting a region of the tissue atlas, a user-interactive software module of an available imaging system can redirect information displayed to the user so that a base image or image data frame specific to the use selection can be displayed.

[0226] In this way, the tissue atlas is presented to the user without flipping through a long series of images or frames of image data. The interaction and selection of the tissue atlas reduces diagnosis and analysis by speeding up the guidance of the end user to the area of ​​interest (e.g., calcium plaque or lipid in the artery). Furthermore, this can speed up stent planning when the patient is catheterized on the table and the blood flow has been temporarily stopped to image a portion of the blood vessel. In this way, faster decisions can be made to improve patient outcomes. The selection of stent placement areas is also improved by avoiding calcified areas (where the proper stent expansion is limited due to the presence of calcium or other undesirable arterial conditions). The favorable and unfavorable areas of the artery can be marked as suitable or unsuitable for stent placement using colors, graphics, visual cues, or other marks or user interface features (e.g., animations or other cues).

[0227] The present disclosure relates in part to a data collection system, such as an intravascular data collection system suitable for use in a catheterization laboratory, such as an optical coherence tomography system. The present disclosure relates in part to a data collection system, which includes a processor suitable for displaying intravascular image data. The displayed image data includes: data or images generated based on depth measurements. In one embodiment, the image data is generated using optical coherence tomography. The system can also display a user interface that displays intravascular information, such as data related to intravascular plaque.

[0228] The diagnostic systems, methods, browsing tools, and tissue atlases and related features disclosed herein provide improvements in the form of better tools for interpretation and decision making based on plaque composition, lesion properties, tissue type, and related tissue and vascular information after one or more tissue characterization processes and image data detection and analysis software modules have been operated. The tissue atlas interface and the organized characterization of tissue data from the underlying image data obtained during a given object scan validate various improved image views. For example, in the context of OCT and other intravascular imaging modalities, a given tissue atlas can provide a volumetric map of the complete imaging probe pullback in a single image.

[0229] In this case, the imaging probe is pulled back through the blood vessel as it rotates to obtain imaging data. The end user can use a given tissue map to quickly recognize the arc extension and thickness of calcium plaques and healthy placement areas for stent implantation. Further, other details related to tissue type can be evaluated. In addition, as for a given tissue map showing excessive calcium or other undesirable tissue states, this information can help the end user select alternative treatment options, such as bypass or cardiovascular surgery. A given tissue map can be generated using different methods. A given method can vary based on the source of the image data obtained by scanning a patient with an imaging system.

[0230] During the pre-interventional evaluation phase, physicians can assess plaque composition. Fig.16As shown in , the tissue atlas displays 1425 pixels, where the green G indicates areas where the media has been detected in the tissue characterization, and the red indicates the presence of calcium plaques Ca. The colors referenced in a given tissue atlas user interface may vary and may be replaced with hatching in some cases or other markings.

[0231] The dotted lines in the organizational map are shown to Fig.14 The rings R1, R2, and R3 along the length of the artery representation provide a reference. In one embodiment, a given tissue map can display the extent of the plaque. The intensity value corresponds to the thickness of the plaque (in mm, as shown in the legend), ranging from about 0.5 mm to about 1.5 mm. Any category or type of ROI / FOI detected using the methods disclosed herein can be displayed using a tissue map representation (e.g., tissue map 1425). In addition, other information (e.g., fibrous and lipid-containing plaques) can also be added to the tissue map, such as displayed in Fig.18 In the exemplary tissue map in . During the stent deployment phase, the doctor can easily confirm the placement area through this line. The intensity of green indicates the thickness of the tunica media, but may vary to indicate the distance between the opposite end points of the two EELs at 180 degrees. In turn, this can be used for useful information about the size of the stent.

[0232] Fig.17 Show Fig.16 The tissue atlas of FIG. 1 shows cross-sectional frames R1, R2, and R3 showing the identified media M, lumen L, and calcium C. In one embodiment, these regions of interest are color-coded, such as lumen in blue, calcium in red, and media in green. Fig.18 A tissue atlas 1490 is shown, which includes markers for guidewire GW (grey), media M (green), calcium Ca (red), lipids LP (blue), and side branches SB (borders shown). These various representations can be used to support workflow and diagnosis.

[0233] The present disclosure relates in part to a diagnostic system and an interface thereof, which facilitates browsing of vascular representations to which one or more imaging and tissue detection methods / classifications have been applied. With respect to a given vessel, such as a coronary artery or other body lumen, one or more tissue types or other areas of interest may be identified using various techniques. In particular, calcium nodules, calcified tissues, and other calcium-associated tissues may be represented as calcified areas in a vessel. One or more tissue atlas representations may be generated and used to display representative tissues and areas of interest to a user. Representative tissues and / or areas of interest suitable for detection and inclusion in one or more tissue atlases may include one or more of the following: lipid regions, lumen regions, stent tendons, side branches, guidewires, external elastic layers (EEL), internal elastic layers (IEL), boundaries and volumes associated with the foregoing, and other arterial features and tissue types as disclosed herein.

[0234] The present disclosure relates in part to user interface design that facilitates improved information and time management based on characterizing tissue of a body lumen (e.g., a coronary artery) using one or more tissue atlas representations. In each tissue atlas shown, the paper is moved in a direction away from where the OCT image probe is located and then pulled back. The length of the pullback is between R1 and RN, where RN is the frame count of the pullback. The lowest ring count is the closest. In one embodiment, the high ring count is the farthest. This can be seen in Fig.15 in a ring arrangement.

[0235] The present disclosure is based in part on the discovery that calcium and other tissue of blood vessels can be detected and characterized, image data is obtained about blood vessels, such as OCT image data, IVUS image data, CT scan image data, MRI image data, angiography image data and other image data sources. In some imaging modalities, calcified areas appear as discrete dark shapes. This is the case with OCT images, where calcium appears as darker areas relative to the brighter vascular tissue background of the OCT image.

[0236] The present disclosure relates in part to a method for identifying regions of interest in a blood vessel, which may include tissue type and other features, such as lumen, side branches, stents, guidewires, and other features, properties, and materials of the vessel.

[0237] In one embodiment, the representation of the vessel that has undergone tissue type analysis and / or tissue type segmentation displays one or more two-dimensional cross-sections of the vessel or a three-dimensional longitudinal representation of the vessel. In one embodiment, the vessel representation or the underlying tissue characterization image data obtained about the vessel is converted into a tissue atlas. In one embodiment, various colors, shapes, hatches, masks, borders and other graphical elements or overlays are used to identify or segment detected tissue types and / or regions of interest in the tissue atlas.

[0238] The present disclosure relates in part to a system for identifying an area of ​​interest in a blood vessel, the system comprising: a processor in communication / connected with a memory, the memory containing instructions that, when executed, cause the processor to: acquire blood vessel image data; apply multiple filters to the image data to generate characteristics or types, such as tissue types. In one embodiment, the image data is a plurality of scan lines. In one embodiment, the image data is X-ray based data. In one embodiment, the image data is a polar coordinate image. In one embodiment, one or more polar coordinate images are sampled. The samples are combined to generate a tissue representation of the blood vessel. The tissue representation of the blood vessel takes the form of polar coordinates. In one embodiment, a tissue atlas is obtained by converting the polar coordinate tissue representation to a Cartesian representation. In one embodiment, the tissue atlas is a series of rings, circles, ellipses, which are arranged in sequence along an axis from proximal to distal.

[0239] One embodiment of the present disclosure relates in part to an intravascular data collection system and one or more software-based graphical user interfaces and software modules to perform one or more detection and display processes as described herein. In one embodiment, intravascular data is collected while angiographic data is collected. In other embodiments, angiography, CT scan, X-ray-based imaging, photography or other imaging modalities are used to obtain imaging data for generating tissue atlases.

[0240] The present disclosure relates in part to systems and methods for performing treatment assessment (including stent planning and surgical options) by visualizing a subject's blood vessel (e.g., one or more coronary arteries). Image data can be obtained using an intravascular data collection probe. The probe can be pulled back through the blood vessel, for which data can be collected. Such a pullback and associated data collection are used to plan stent deployment or evaluate deployed stents. The resulting intravascular data from the pullback can be used in various ways, such as to visualize various vascular regions, features, and deployed stents associated therewith. Image data used to generate a tissue atlas can be registered with corresponding angiographic data. In this way, a user can select an area of ​​a tissue atlas and see the underlying image data (OCT, IVUS, X-ray, etc.) used to generate the atlas and see the angiographic data, where the selected vascular area on the tissue atlas is highlighted or otherwise indicated.

[0241] The stent can be visualized relative to the side branches in various embodiments of the present disclosure. This is an important feature because it is typically desirable to avoid stent involvement with side branches during stent deployment. In this way, the tissue atlas can display side branches and frames containing them can be marked as unsuitable for use as stent placement areas. The systems and methods described herein facilitate visualization of stent placement areas relative to different tissue types and areas of interest. The tissue atlas can be aligned with angiographic data, various user interfaces, and representations of stent ribs and side branches based on detection of these features in the collected intravascular data.

[0242] The present disclosure relates in part to the generation and visualization of intravascular data collection systems (e.g., OCT, IVUS, and other imaging modalities) and diagnostic information (e.g., stent placement areas, side branches, regions of interest, and intravascular characterized tissue regions as part of a tissue atlas. Graphical elements suitable for indicating diagnostic information of interest such as the foregoing are used as user-selectable elements in a tissue atlas that facilitate movement to a base image summarized in the tissue atlas.

[0243] Also disclosed herein are systems and methods for visualizing stents, tissue types, tissue volumes, and tissue boundaries. One or more software modules can be used to detect side branch locations, lumen contours, and stent rib locations, generate vascular representations, generate tissue maps, and control image browsing based on user selections of tissue maps. The systems and methods disclosed herein also include: automated measurement systems and related features that can measure angles, thicknesses, volumes, widths, frame counts, and relative proximity of tissues to the lumen of various tissue types (including calcium, lipids, fibers, and others).

[0244] In various embodiments, such measurement tools can be used to measure the aforementioned parameters and any geometric properties of a given region of interest for a particular tissue type. These measurements can be used to generate various ratings and scores suitable for consideration by the end user. For example, if the calcium burden in a particular vascular region is evident in the tissue atlas, but the surface calcium is only trace in general, then measurements of this can help guide the user to not exclude such regions as candidate placement areas.

[0245] It should be appreciated that, for clarity, the present disclosure details embodiments of various aspects of the applicant's teachings, while omitting certain specific details (where convenient or appropriate). For example, the discussion of similar or similar features in alternative embodiments may be reduced. For brevity, well-known ideas or ideas may not be discussed in any detail. Those skilled in the art will recognize that some embodiments of the applicant's teachings may not require specific details in the details specifically described in each embodiment, and the details are proposed here only to provide a complete understanding of the embodiments. Similarly, it should be apparent that the embodiments may be easily changed or varied according to common common sense without departing from the scope of the present disclosure. The detailed description of each embodiment is not considered to limit the scope of the applicant's teachings in any way.

[0246] As used herein, the words "about" and "substantially the same" refer to quantitative changes that may occur (e.g., due to real-world measurement or processing processes, due to inadvertent errors in these processes, due to differences / failures in the manufacture of electrical components, due to electrical losses) and changes that would be considered equivalent by a person skilled in the art, as long as such changes do not cover known values ​​of prior art practice. Typically, the word "about" means 1 / 10 of the value that is larger or smaller than the value or range of values, such as ±10%. For example, applying a voltage of about +3V direct current (DC) to an element may mean that the voltage is between +2.7VDC and +3.3V DC. Similarly, if the values ​​are referred to as "substantially the same", the difference between the values ​​may be as high as 5%. Whether or not modified by the words "about" or "substantially the same", the values ​​described in the claims include equivalents of the values, such as quantitative changes in the values ​​that may occur but would be considered equivalent by a person skilled in the art.

[0247] Non-Limiting Software Features and Examples for MLS and Tissue Characterization / Classification Systems and Methods

[0248] The following description is intended to provide an overview of the device hardware and other operating components suitable for performing the methods of the disclosure described herein. This description is not intended to limit the scope of the applicable environment or the disclosure. Similarly, the hardware and other operating components may be suitable as a part of the above-mentioned equipment. The disclosure may be implemented by other system configurations, including personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, network personal computers, microcomputers, mainframe computers, etc. The disclosure may also be implemented in a distributed computing environment, wherein each task is performed by a remote processing device (e.g., in different rooms of a catheterization room) linked via a communication network.

[0249] The detailed description of some parts is presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations can be used by technicians in computer and software related fields. In one embodiment, an algorithm is generally considered to be a series of self-consistent operations that lead to a desired result. The operations performed as method steps or otherwise described in this article require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, converted, compared, and otherwise manipulated.

[0250] Unless otherwise expressly noted as will become apparent from the following discussion, it should be recognized that throughout the specification, discussions employing terms such as “processing” or “computing” or “classifying” or “characterizing” or “correlating” or “detecting” or “evaluating” or “convolution” or “deconvolution” or “classification” or “segmentation” or “training” or “annotation” or “registering” or “measuring” or “calculating” or “comparing” or “generating” or “sensing” or “determining” or “displaying”, or Boolean logic or other set-related operations or similar operations, refer to the actions and processes of a trained MLS, computer system, AI processor, GPU, or electronic device that manipulates and transforms data represented as physical (electronic) quantities within registers and memories of a computer system or electronic device into other data represented as similar physical quantities within electronic memories or registers or other such information storage, transmission, or display devices.

[0251] The present disclosure also relates in some embodiments to an apparatus for performing the operations herein. Such an apparatus may be specially constructed for the desired purpose, or it may include a general-purpose computer (selectively activated or reconfigured by a computer program stored in the computer). Its various circuits and components may be used to perform some of the data collection and transformation and processing described herein.

[0252] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems can be used according to the procedures taught herein, or they can prove convenient for building more specialized equipment to perform the required method steps. The required structure for various of these systems will be apparent from the following description. In addition, the present disclosure is not described with reference to any particular programming language, and various embodiments can thus be implemented using various programming languages.

[0253] Embodiments of the present disclosure may be implemented in a variety of different forms, including but not limited to: computer program logic for a processor (e.g., a microprocessor, a microcontroller, a digital signal processor, or a general purpose computer), programmable logic for a programmable logic device (e.g., a field programmable gate array (FPGA) or other programmable logic device), discrete components, integrated circuits (e.g., an application specific integrated circuit (ASIC)), or any other manner including combinations thereof. In a typical embodiment of the present disclosure, some or all of the processing of data collected using an OCT probe, a 2D imaging or 3D imaging system, and a processor-based system is implemented as a series of computer program instructions, which are converted into a computer executable form, stored in a computer readable medium, and executed by a microprocessor under the control of an operating system. In this way, query responses and input data are converted into instructions that the processor can understand, suitable for generating training sets, image masks, and other inputs and outputs disclosed herein. The computer program logic that implements all or part of the functions previously described herein may be implemented in various forms, including but not limited to: source code form, computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator). Source code may include: a series of computer program instructions for use in any of a variety of programming languages ​​(e.g., object code, assembly language, or high-level languages ​​such as Python, Perl, Go, FORTRAN, C, C++, JAVA, or HTML) for use in any of a variety of operating systems or operating environments. Source code may define and use a variety of data structures and communication messages. Source code may be in a computer executable form (e.g., implemented by an interpreter), or source code may be converted (e.g., implemented by a converter, assembler, compiler) into a computer executable form.

[0254] The various embodiments described herein or their components or parts can be implemented in a variety of different embodiments of software, firmware, and / or hardware, or its modules. The software code or dedicated control hardware used to implement some embodiments of the present application does not limit the present invention. For example, the embodiments described above can be implemented in computer software using any suitable computer programming language (e.g., .NET, SQL, or MySQL) using, for example, traditional or object-oriented techniques.

[0255] Programming languages ​​used for computer software and other computer-implemented instructions may be translated into machine language by a compiler or assembler prior to execution, and / or may be translated directly by an interpreter at runtime. Examples of assembly languages ​​include ARM, MIPS, and x86; examples of high-level languages ​​include: Ada, BASIC, C, C++, C#, COBOL, Fortran, LUA, Clojure, Java, Lisp, Pascal, Object Pascal; examples of scripting languages ​​include: Bourne script, JavaScript, Python, Ruby, PHP, and Perl.

[0256] The operation and behavior of the embodiments are described without specific reference to actual software code or dedicated hardware components. The lack of such specific reference is feasible because it is clearly understood that, based on the description herein, with only reasonable efforts and without undue experimentation, one of ordinary skill in the art will be able to design software and control hardware to implement the embodiments of the present disclosure.

[0257] Various machine learning systems and associated neural networks, such as deep learning neural networks, 3D neural networks, convolutional neural networks, 2D neural networks, N-layer neural networks, feedforward neural networks, feedforward networks, feedback networks, radial basis function neural networks, Korhonen self-organizing neural networks, recurrent neural networks (RNNs), modular neural networks, deep learning networks, artificial intelligence-based systems and frameworks, and combinations of the foregoing.

[0258] Software for the various machine learning systems described herein and other computer functions described herein may be implemented in computer software using any suitable computer programming language (e.g., .NET, C, C++, Python, C#, Matlab programming modules and tools) and using traditional, functional, or object-oriented techniques. For example, the various machine learning systems may be implemented by software modules stored or otherwise maintained in a computer-readable medium (e.g., RAM, ROM, secondary storage, etc.). One or more processing cores (e.g., CPU, GPU, and / or AI acceleration core) of the machine learning system may then execute the software modules to implement the functions of the corresponding machine learning system (e.g., network 107, encoders 111-114, learning coach 110, etc.).

[0259] A computer program may be permanently or temporarily fixed in any form (e.g., source code form, computer executable form, or intermediate form) to a physical storage medium, such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a magnetic disk or fixed hard disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., a PCMCIA card), or other memory device. A computer program may be fixed in any form to a signal that can be transmitted to a computer using any of a variety of communication technologies (including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), network technology, and Internet technology). A computer program may be distributed in any form as a removable storage medium with an accompanying printed or electronic record (e.g., a compressed package), pre-loaded with a computer system (e.g., on a system ROM or fixed hard disk), or distributed from a server or electronic bulletin board on a communication system (e.g., the Internet or the World Wide Web).

[0260] The hardware logic that implements all or part of the functionality previously described in this document (including programmable logic for a programmable logic device) can be designed using traditional manual methods, or can be designed, captured, simulated, or electronically recorded using various tools such as computer-aided design (CAD), hardware description languages ​​such as VHDL or AHDL, or PLD programming languages ​​such as PALASM, ABEL, or CUPL.

[0261] Programmable logic may be permanently or temporarily fixed to a physical storage medium, such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a magnetic disk or fixed hard disk), an optical memory device (e.g., a CD-ROM), or other memory device. Programmable logic may be fixed to a signal that can be transmitted to a computer using any of a variety of communication technologies (including, but not limited to, analog, digital, optical, wireless (e.g., Bluetooth), networking, and Internet technologies). Programmable logic may be distributed as a removable storage medium with accompanying printed or electronic records (e.g., a compressed package), preloaded with a computer system (e.g., on a system ROM or fixed hard disk), or distributed from a server or electronic bulletin board on a communication system (e.g., the Internet or the World Wide Web).

[0262] Various examples of suitable processing modules are discussed in more detail below. As used herein, a module refers to software, hardware, or firmware suitable for performing a specific data processing or data transmission task. Typically, in a preferred embodiment, a module refers to a software routine, program, or other memory-resident application suitable for receiving, converting, routing, and processing instructions, or various types of data (e.g., resistance changes, voltage changes, current changes, guidewire-based probe data, intravascular pressure data, ratios, indices, and other information of interest).

[0263] Computers and computer systems described herein may include: operatively associated computer-readable media (e.g., memory for storing software applications) for obtaining, processing, storing and / or communicating data. It will be appreciated that such memory may be internal, external, remote, or local relative to the computer or computer system to which it is operatively associated.

[0264] The memory may also include any structure for storing software or other instructions, such as, but not limited to: a hard disk, an optical disk, a floppy disk, a DVD (digital versatile disk), a CD (compact disk), a memory stick, a flash memory, a ROM (read-only memory), a RAM (random access memory), a DRAM (dynamic random access memory), a PROM (programmable ROM), an EEPROM (extended erasable PROM), and / or other similar computer-readable media.

[0265] Typically, the computer-readable memory medium used in association with the embodiments of the present disclosure described herein may include any memory medium capable of storing instructions executed by a programmable device. If applicable, the method steps described herein may be implemented or executed as instructions stored on a computer-readable memory medium or memory medium. According to embodiments of the present disclosure, these instructions may be software implemented in various programming languages ​​(e.g., C++, C, Java, and / or various other types of software programming languages ​​that may be applied to form instructions).

[0266] The storage medium may be non-transitory or include a non-transitory device. Thus, a non-transitory storage medium or a non-transitory device may include a tangible device, meaning that the device has a physical, tangible form, but the device can change its physical state. Thus, for example, non-transitory means that the device remains tangible even if its state changes.

[0267] The aspects, embodiments, features, and examples of the present disclosure should be considered to be illustrative in all respects and are not intended to limit the present disclosure, the scope of which is limited only by the claims. Other embodiments, modifications, and uses will be apparent to those skilled in the art without departing from the spirit and scope of the present disclosure as claimed.

[0268] The use of headings and sections in this application is not meant to limit the disclosure; each section may apply to any aspect, embodiment, or feature of the disclosure. Only those claims that use the wording "means" are intended to be interpreted in accordance with 35 USC 112, P6. In the absence of the statement "means" in a claim, such claim should be deemed not to be required in accordance with 35 USC 112. No limitations from the specification are intended to be read into any claim unless such limitations are expressly included in the claim.

[0269] When a value or range of values ​​is given, each value and the endpoints of the given range and the values ​​therebetween may be increased or decreased by 20% and still be within the teachings of the present disclosure unless some different range is explicitly stated.

[0270] Throughout this application, if an ingredient is described as having, including, or comprising specific components, or if a process is described as having, including, or comprising specific process steps, it can be envisioned that the ingredient of the present teaching also consists essentially or does consist of the listed ingredients, and the process of the present teaching also consists essentially or does consist of the listed process steps.

[0271] In this application, if an element or component is said to be included in and / or selected from a list of listed elements or components, it should be understood that the element or component can be any one of the listed elements or components and can be selected from a group consisting of two or more listed elements or components. Further, it should be understood that the elements and / or features of the compositions, devices, or methods described herein can be combined in various ways without departing from the spirit and scope of the present teachings, whether expressly or implicitly herein.

[0272] Use of the expressions "including" and variations thereof and "having" and variations thereof should generally be construed as open ended and non-limiting unless expressly stated otherwise.

[0273] As used herein, the singular includes the plural (and vice versa) unless otherwise expressly indicated. In addition, the singular forms "a", "an", and "the" include the plural forms unless the context clearly indicates otherwise. Furthermore, if the expression "about" is used before a quantitative value, the present teachings also include the specific quantitative value itself, unless otherwise expressly indicated.

[0274] It should be understood that the order of the various steps or the order in which a particular action is performed is not important, as long as the present teaching is still operable. In addition, two or more steps or actions can be performed simultaneously.

[0275] If a range or list of values ​​is provided, each intermediate value between the upper and lower limits of the range or list of values ​​is independently contemplated and is included within the scope of the present disclosure, as if each value were specifically listed herein. In addition, smaller ranges between and including the upper and lower limits of a given range are contemplated and are included within the scope of the present disclosure. The listing of exemplary values ​​or ranges does not disclaim other values ​​or ranges between and including the upper and lower limits of a given range.

[0276] It should be understood that, in order to clearly understand the present disclosure, the figures and descriptions of the present disclosure have been simplified to illustrate relevant elements, while removing other elements for clarity purposes. However, it will be appreciated by those of ordinary skill in the art that these and other elements may be desired. However, because such elements are well known in the art, and because they are not conducive to a better understanding of the present disclosure, discussion of such elements is not provided herein. It should be appreciated that the presentation of each figure is for illustrative purposes, rather than as a construction diagram. The details and modifications or alternative embodiments omitted are within the scope of those of ordinary skill in the art.

[0277] It can be appreciated that in certain aspects of the present disclosure, a single component can be replaced with multiple components, and multiple components can be replaced with a single component to provide an element or structure or to perform a given one or more functions. Unless such a replacement is not operational to implement a specific embodiment of the present disclosure, such a replacement is considered to be within the scope of the present disclosure.

[0278] The examples presented herein are intended to illustrate possible specific embodiments of the present disclosure. It will be appreciated that each example is primarily intended to illustrate the purpose of the disclosure for those skilled in the art. There may be variations in these schematic diagrams or the operations described herein without departing from the spirit of the present disclosure. For example, in certain circumstances, method steps or operations may be performed or executed in a different order, or operations may be added, deleted, or modified.

Claims

1. A method for evaluating a coronary artery, comprising: acquiring a series of image data including polar coordinate image frames; annotating one or more regions of interest or features of interest in each polar coordinate image of the series of image data, such that each annotated region of interest or feature of interest is a ground truth annotation; training a neural network of a machine learning system using a series of annotated polar images, wherein a plurality of regions in each polar image are identified by category; Input polar coordinate image data into the trained neural network; Generating a predictive output image by propagating the polar coordinate image data through a trained neural network; as well as The predicted output image is displayed, wherein the predicted output image includes color-coded regions, wherein each color corresponds to a category.

2. The method according to claim 1, wherein: Each image of the series of image data includes a plurality of image data elements regarding a coronary artery.

3. The method according to claim 1, wherein: The annotation is performed via a graphical user interface that includes user controls to select groups of pixels or two-dimensional boundaries for defining the feature of interest.

4. The method according to claim 1, wherein: The neural network is repeatedly trained until an output of a cost function that compares the predictive output of the machine learning system to ground-truth inputs is at or below a threshold.

5. The method according to claim 1, further comprising: One or more annotated regions of interest or features of each polar coordinate image are classified into a type or category, wherein training the neural network of the machine learning system further includes: classifying each of the annotated regions of interest or features of interest by the type or category.

6. The method according to claim 5, wherein: The type or category is selected from the group consisting of: intima, media, adventitia, lumen, external elastic lamina (EEL), internal elastic lamina (IEL), plaque, calcium, calcium plaque.

7. The method according to claim 5, wherein: The type or category is selected from the group consisting of: side branch, lumen, guidewire, stent rib, stent, imprisoned stent, bioresorbable vascular scaffold (BVS), drug eluting stent (DES), fibrous, halo artifact, pressure wire, lipid, atherosclerotic plaque, stenosis, calcium, calcified plaque, calcium-containing tissue, lesion, fat, poorly set stent, under-expanded stent, over-expanded stent, radio-opaque marker, arterial tree branching angle, probe calibration element, doped membrane, light scattering particles, sheath; Doped sheath, fiducial alignment points, diameter measurement, radial measurement, guide catheter, shadow area, guidewire segment, length, and thickness.

8. The method according to claim 2, wherein: Each data element, image, and predictive output image is stored in a machine-readable memory that is in electronic communication with the machine learning system.

9. The method according to claim 1, further comprising: Identified in the predictive output image are: one or more of: an arc-based measure or measurement of similarity of Ca and EEL, a detected external elastic layer (EEL) diameter, and a detected calcium (Ca) depth.

10. The method according to claim 1, wherein: The neural network is a convolutional neural network and wherein some of the input channels for a first node or layer are four channels.

11. The method according to claim 1, wherein: The predictive output image includes one or more markers indicating boundaries of features being predicted or classified.

12. The method according to claim 1, further comprising: A carpet view is generated using line projection, and the carpet view is filtered to reduce noise in the predictive output image.

13. The method according to claim 1, wherein: The neural network is a conformal neural network.

14. The method according to claim 1, wherein: The machine learning system includes an artificial intelligence (AI) processor, and wherein the AI ​​processor includes one or more parallel processing elements.

15. The method according to claim 14, wherein: The AI ​​processor includes N parallel processing elements, and wherein the AI ​​processor further includes dedicated AI processor memory.

16. The method according to claim 14, wherein: The AI ​​processor is a graphics processing unit.

17. The method according to claim 14, wherein: The N parallel processing elements are selected from the group consisting of: a Compute Unified Device Architecture core processor, a Tensor Core processor, and a stream processor.

18. The method according to claim 15, wherein: The dedicated AI processor memory ranges from about 8 GB to about 64 GB.

19. The method according to claim 15, wherein: The dedicated AI processor memory ranges from about 64 GB to about 128 GB.

20. The method of claim 1, further comprising: When classifying user image data, the processing time of the machine learning system is reduced by flattening the user image data before inputting it into the neural network.

21. The method of claim 1, further comprising: When classifying user image data, processing time of the machine learning system is reduced by resizing or excluding at least one region of one or more of the polar coordinate images prior to input into the neural network.

22. The method of claim 1, further comprising: For one or more of the polar images, the training data is augmented by performing 1, 2 or 3 circular rotations.

23. The method of claim 1, further comprising: For one or more of the polar coordinate images, a left-right flip is performed.

24. The method of claim 1, further comprising: Prior to inputting the patient polar image data into the neural network, a sub-series of scan lines are removed from the patient polar image.

25. The method of claim 1, further comprising: Lumen detection is performed using an image processing method or a machine learning method to generate a series of detected lumen boundary data.

26. The method of claim 1, further comprising: An image processing pipeline is used instead of using the trained neural network to perform one or more of side branch detection, guidewire detection, and stent detection to increase the processing rate of a series of image frames obtained during an optical coherence tomography pullback process.

27. The method according to claim 26, wherein: The input of the detected lumen boundary data enables reduced latency in classifying one or more regions of interest and features in the patient polar image data.

28. The method of claim 1, further comprising: One or more image masks are generated for each region of interest or feature identified in the one or more patient images.

29. The method of claim 1, wherein: Each input image used for training or processing patient data is transformed into multiple versions, wherein the multiple versions are generated by at least one of left flipping, right flipping, and circular rotation.

30. An intravascular imaging and tissue characterization system comprising: case; a frame grabber for receiving one or more of the X-ray image data and the intravascular image data; power supply; one or more electronic memory storage devices in electrical communication with the power source; one or more image processing software modules executable on the one or more processors and stored in the one or more electronic memory storage devices; a computing device including a first processor, the computing device being in electronic communication with the power source and the first processor; one or more software programs stored in the one or more electronic memory storage devices; a machine learning system comprising a neural network comprising one or more machine learning software modules, wherein the neural network is trained using ground truth annotations of polar coordinate images, wherein the ground truth annotations include calcium and tunica media; one or more artificial intelligence (AI) processors, wherein the one or more machine learning software modules are executable on the one or more AI processors, wherein the one or more AI processors include dedicated memory; and an interface for sending and receiving image data from the first processor, the machine learning system being in electronic communication with the power source, wherein the machine learning system, the computing device, and the one or more electronic memory storage devices are disposed in the housing, wherein the trained neural network is operable to classify input image data on a substantially real-time basis.

31. The system of claim 30, wherein: The housing is a housing of an optical coherence tomography system or an intravascular ultrasound imaging system, and wherein the substantially real-time reference is less than about 10 seconds.

32. The system of claim 30, wherein: The one or more image processing software modules include one or more of the following: intravascular polar coordinate image to Cartesian coordinate image conversion software, intravascular Cartesian coordinate image to polar coordinate image conversion software, tissue classification overlay software that identifies areas or features of interest when displayed to an end user, a lumen detection software module, an image flattening preprocessing software module, an image resizing software module, an image annotation software with a graphical user interface to identify or label training images with real-world data, a preprocessing software module, and a circular rotation software module.

33. The system of claim 30, wherein: The one or more machine learning software modules include one or more of the following: a neural network interface, lumen contour prediction, side branch prediction, an image resizing module, a user interface and input processing software module, a machine learning system interface software module for controlling and setting parameters of the neural network, a machine learning system memory management software, a preprocessing software module, a stent rib prediction software module, a captured stent prediction software module, a guidewire prediction software module, and an interface module for exchanging data with an imaging system.

Images