Method and apparatus for analyzing intracoronary images

Abstract

[0006] Embodiments of the present technique provide an apparatus and method for analyzing intracoronary images, e.g., to predict disease likelihood, clinical features or events, and / or to track the success of drugs or other treatments. The method includes, for each image in a set of coronary images, using a first neural network to classify the image for the presence or absence of diseased tissue, and when the image is classified as having diseased tissue present, using a second neural network to classify the image for the presence or absence of artifacts, determining whether to analyze the image based on the classification step, and when the image is classified, analyzing the image by identifying one or more features of interest of the coronary tissue using a third neural network, and measuring each identified feature of interest.

Description

Technical Field

[0001] The present application relates to methods and apparatuses for analyzing intravascular coronary images, for example, to predict the likelihood of a disease, disease presentation or event and / or to track the outcome of a drug or other treatment.

Background Art

[0002] Despite its prevalence, prediction of clinical events in coronary artery disease (CAD) is generally based on demographics. More recently, such predictions may have an overlapping genetic risk score, as described, for example, in "Multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses" by Ripattis et al., published in Lancet 376, 1393-1400 (2010), or in "A genetic risk score is associated with incident cardiovascular disease and coronary artery calcium: the Framingham Heart Study" by Thanassoulis et al., published in Circ. Cardiovasc. Genet. 5, 113-21 (2012).

[0003] High-resolution intracoronary imaging may be used to incorporate patient-specific features for individualized risk prediction, but it requires clinician interpretation. For example, as described in "Clinical outcome of nonculprit plaque ruptures in patients with acute coronary syndrome in the PROSPECT study" by Xie et al., published in JACC Cardiovasc Imaging 7, 397-405 (2014), intracoronary imaging has been used to provide patient-specific data on the extent and type of atherosclerosis. However, as reported, less than 20% of the "high-risk" atherosclerotic plaques identified in natural history studies led to patient events over 5 years. The poor outcomes may be due to multiple factors, including failure to identify high-risk imaging features, clinician interpretation with inadequate intraobserver reproducibility, imaging artifacts, and low spatial resolution.

[0004] For other uses of high-resolution imaging such as optical coherence tomography (OCT), see "Tissue characterization of coronary plaques and assessment of thickness of fibrous cap using integrated backscatter intravascular ultrasound. Comparison with histology and optical coherence tomography" by Kawasaki et al., published in Circ J 74, 2641-2648 (2010). OCT is used to visualize tissue components and identify different plaque types. However, large amounts of imaging data sets are generated, and only small areas of the disease are selected for detailed clinical interpretation. As described in "Quantification of fibrous cap thickness in intracoronary optical coherence tomography with a contour segmentation method based on dynamic programming" by Zahnd et al., published in Int J Comput Assist Radiol Surg 10, 1383-1394 (2015), or "Contour segmentation of the intima, media, and adventitia layers in intracoronary OCT images: application to fully automatic detection of healthy wall regions" by Zahnd et al., published in Int J Comput Assist Radiol Surg (2017). doi:10.1007 / s11548-017-1657-7, attempts have been made towards automation.However, as described in these articles, both the frequency and variation of artifacts seen in clinical practice mean that even semi-automated analysis requires region selection and interpretation by a clinician, or limited analysis regarding the complete coronary artery structure.

[0005] Background information can be found in U.S. Patent Application Publication No. 2014 / 276011, which describes methods and apparatus for automatically identifying the lumen boundary of an in-vessel position in an image of a blood vessel and then measuring the diameter of the blood vessel; U.S. Patent Application Publication No. 2017 / 309018, which describes methods and apparatus for automatically classifying intravascular plaque using features extracted from intravascular OCT images; CN108961229, which describes methods and systems for detecting vulnerable plaque from cardiovascular OCT images based on deep learning; and CN109091167, which describes methods for predicting the growth of coronary atherosclerotic plaque. Further background information can be found in U.S. Patent Application Publication No. 2017 / 0148158, which describes systems and techniques for analyzing angiographic image data to extract vasculature structure information, where angiography includes X-ray images of a contrast agent flowing through a vasculature structure to visualize the inside or lumen of blood vessels and organs.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 10

Non-Patent Document 11

Summary of the Invention

Problems to be Solved by the Invention

[0008] The applicant has recognized the need for a new automated method for analyzing high-resolution intracoronary imaging.

Means for Solving the Problems

[0009] According to the present invention, an apparatus and a method described in the appended claims are provided. Other features of the present invention will become apparent from the dependent claims and the following description.

[0010] An automated computer-implemented method for analyzing a set of coronary artery images is described, the method comprising, for each set of three images in the set of images, using a first neural network to classify the set of images as to the presence or absence of diseased tissue; using a second neural network to classify the set as to the presence or absence of artifacts when the set is classified as having diseased tissue; determining whether to analyze the images based on the classification; when the images are classified, using a third neural network to classify the images by identifying one or more features of interest of the coronary artery tissue; and measuring each identified feature of interest.

[0011] Also described is an apparatus for analyzing a set of coronary artery images, the apparatus comprising an imaging device for capturing a set of coronary artery images and at least one processor coupled to a memory and configured to, for each image in the set of images, use a first neural network to classify the image as to the presence or absence of diseased tissue; use a second neural network to classify the image as to the presence or absence of artifacts when the image is classified as having diseased tissue; determine whether to analyze the image based on the classification; when the image is analyzed, use a third neural network to analyze the image to identify one or more features of interest of the coronary artery tissue; and measure each identified feature of interest.

[0012] Such an automated method or apparatus can be freed from operator involvement after image acquisition and can thus be applicable to a wide range of datasets. The set of images may be a complete or partial pullback.

[0013] Before classifying the image, the coronary artery in the image may be sampled into a plurality of samples. A set of samples, for example three samples, is input into a second version of the neural network for classifying the image, and the samples may also be trained. Each sample may be a rectangular segment, and the upper edge may be aligned with the lumen edge of the artery.

[0014] The step of analyzing the image using the second neural network may include determining the ratio of coronary artery tissue detectable in each sample in the input set and classifying the image based on the determined ratio. The second neural network may classify the image by analyzing a plurality of training dataset samples using a supervised learning algorithm. The step of classifying the image may include, for example, classifying any existing artifacts as correctable or not based on whether the underlying tissue is detected. Correctable artifacts may be generated by lesion artifacts (e.g., residual blood in the lesion lumen and resulting lesion tissue shadows) or structural components of the artery (e.g., when arterial tissue appears between the imaging device (e.g., OCT catheter) and the region of interest, i.e., when the OCT catheter is partially within a branch / branch), and non-correctable artifacts may be generated by medical structural components (e.g., stents, guide catheters). For training, there is rich data from repeated patient pullbacks that generate both a comparison state rich in artifacts and a comparison state without artifacts. For example, the image may be classified as having correctable artifacts when the determined ratio of observable tissue is greater than 50% but less than 100%, i.e., when at least half of the samples show underlying tissue. When the determined ratio is approximately 0%, i.e., when there are few samples showing underlying tissue, the image may be classified as having non-correctable artifacts.

[0015] When the image contains correctable artifacts, the method may further include, before analyzing the image, correcting the image classified as having correctable artifacts. The step of correcting the image may include applying an adversarial generation network of a variational autoencoder to the identified image to restore details of the coronary tissue underlying the identified artifacts. For example, the network may generate exchange samples to be inserted into the image. If there is a possibility that no exchange samples are generated, the samples may be classified as containing non-correctable artifacts.

[0016] When the image is classified as having no diseased tissue, the image may be replaced with blank information. Thus, the same number of inputs as the set of images is maintained.

[0017] Measuring each identified feature of interest may include measuring any one or more of fibrous capsule / tissue thickness, fibrous tissue pixel intensity and thickness, plaque composition, lumen area, and the area surrounding the diseased lumen.

[0018] This method may further include, before analyzing the image, sampling the coronary artery in the image into a plurality of samples around the lumen, and constructing the samples in a linear representation of the coronary artery. The sampling may be the same as the above-described sampling. The step of analyzing the processed image may further include identifying, using a regression bounding box technique, the interface between fibrous tissue and necrotic or calcified tissue in each of the plurality of samples. The step of measuring the identified feature of interest may include measuring the distance between the interface and the lumen edge, i.e., measuring the distance between the interface and the edge of the sample and the distance between the lumen edge and the edge of the sample to determine the thickness of the fibrous tissue. The fibrous capsule may be defined as fibrous tissue covering the necrotic / lipid tissue. For this subset of fibrous tissue, the measurement may be the thickness of the fibrous capsule.

[0019] The coronary artery image may be an optical coherence tomography image.

[0020] There can be many applications for the analysis. For example, the method may further include determining, using measurements of each identified feature of interest, which may be necrotic core / lipid or fibrous tissue covering calcium, whether a patient exhibits specific signs of coronary artery disease or has a likelihood of having an acute cardiac event (e.g., a heart attack) in the near future, for example using a fourth neural network. The coronary artery disease may be stable angina, non-ST elevation myocardial infarction (NSTEMI) or ST elevation myocardial infarction (STEMI). Such a fourth neural network may be an independent series classifier. The fourth neural network may receive, as input, measurements of each identified feature of interest for each analyzed image and blank information for each unanalyzed image. The only inputs are the measurements and any blank information, and thus the determination identifies the patient presentation independent of the original patient or pullback context or post-event findings. This produces a bias-free data-based prediction, for example, by removing bias due to operator measurements and / or artifacts related to the presentation, such as bias due to artifacts from the context of the arterial input.

[0021] Another application example is to determine changes in the coronary arteries over time, for example in response to drugs and / or other treatments, or following the study of coronary arteries (natural course). The image or set of images may include a first set or subset of images of the coronary arteries of a patient at a first stage (e.g., the first stage of treatment), and at least a second set or subset of images of the coronary arteries of the patient at at least a second subsequent stage. The patient's images may be acquired / captured at three or more stages or time points, and thus it will be understood that the image or set of images may include images of the patient at a first stage, a second stage, a third stage, a fourth stage, etc. The first subset of images of the patient at the first stage may be captured at a first time (e.g., the first stage of treatment or the first time point), and the second subset of images may be captured at at least a second time (e.g., the second, third, fourth, or subsequent stages of treatment, or the second, third, fourth, or subsequent time points).

[0022] The method may further include measuring a first set of measurements for each identified feature of interest for a first set (or subset) of images, measuring at least a second set of measurements for each identified feature of interest for at least a second set (or subset) of images, and using the first and at least second sets of measurements to determine any changes in the coronary arteries that may be related to the effectiveness of treatment, for example, if treatment is applied. Thus, the first time is the first stage of treatment and the second time is the second stage of treatment. It will be understood that the determining step may be performed using images taken at any time / stage and using any number of sets of measurements. For example, a set of images of a patient's coronary arteries may be taken at the start of the treatment process, at one or more intermediate points during the treatment process, and at the end of the treatment process. Any two or more of these sets of images (or sets of measurements associated with the images) may be compared to determine changes in the coronary arteries. For example, the set of images at the start and end of treatment may be compared, or the start and intermediate points, or intermediate points and end, or between intermediate points, or start and each intermediate point and end may be compared. It will be understood that any two or more sets or subsets of images taken at any two or more time points may be used to determine changes in the coronary arteries.

[0023] Another application example is a method for diagnosing a patient, the method comprising the steps of: receiving a plurality of images of the patient's coronary arteries; for each of the received images, using a first neural network to classify the image as to the presence or absence of diseased tissue; when the image is classified as having diseased tissue, using a second neural network to classify the image as to the presence or absence of artifacts; determining whether to analyze the image based on the classification; when the image is analyzed, using a third neural network to analyze the processed image to identify the interface between fibrous tissue and necrotic, lipid, or calcified tissue; measuring the fibrous tissue or the capsule thickness; and diagnosing the patient based on the measured fibrous tissue / capsule thickness. For example, the diagnosing step may be performed using a fourth neural network, for example, a neural network trained with measurements from patients having one or more heart diseases.

[0024] Another application example is a method for treating a patient, the method comprising the steps of diagnosing a patient having one or more heart diseases (such as stable angina, non-ST elevation myocardial infarction (NSTEMI) or ST elevation myocardial infarction (STEMI)), and treating the patient based on the diagnosis. Similarly, another application example is a method for treating a patient predicted to be likely to have a particular heart disease, which is to treat the patient based on the prediction.

[0025] As will be appreciated by those skilled in the art, the present invention may be embodied as a system, method, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software aspects and hardware aspects.

[0026] Furthermore, the present invention can take the form of a computer program product embodied in a computer-readable medium including computer-readable program code. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0027] The computer program code for carrying out the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages and traditional procedural programming languages. The code components may be included as procedures, methods, etc., and may include sub-components in the form of instructions or sequences of instructions at any level of abstraction, from direct machine instructions of native instruction sets to high-level compiled or interpreted language constructs.

[0028] Embodiments of the present invention also provide a non-transitory data carrier carrying code that causes a processor to execute any of the methods described herein when implemented on the processor.

[0029] The present invention further provides processor control code for implementing the methods described herein, for example, in a general-purpose computer system or in a digital signal processor (DSP). The present invention also provides a carrier for carrying, at runtime, processor control code for implementing any of the methods described herein, specifically on a non-transitory data carrier. The code may be provided on a carrier such as a programmed memory such as a disk, a microprocessor, a CD-ROM or a DVD-ROM, a non-volatile memory (e.g., flash) or a read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. The code (and / or data) for implementing embodiments of the techniques described herein may include source, object, or executable code in a conventional programming language (interpreted or compiled) such as python, C, or assembly code, code for configuring or controlling an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As will be appreciated by those skilled in the art, such code and / or data may be distributed among a plurality of coupled components communicating with each other. The present invention may include a controller including a microprocessor, a working memory, and a program memory coupled to one or more of the components of the system.

[0030] All or part of the logical method according to embodiments of the present invention may be preferably embodied in a logical device comprising logical elements for performing the steps of the above-described method, and it will also be apparent to those skilled in the art that such logical elements may comprise components such as logic gates in, for example, a programmable logic array or an application-specific integrated circuit. Such a logical configuration may also be embodied in an enabling element for temporarily or permanently establishing a logical structure in such an array or circuit, for example using a virtual hardware descriptor language, which may be stored and transmitted using a fixed or transportable carrier medium.

[0031] In one embodiment, the present invention may be implemented using a plurality of processors or control circuits. The present invention may be configured to operate on or be incorporated into an operating system of a device.

[0032] In one embodiment, the present invention may be realized in the form of a data carrier having functional data thereon, and the above functional data includes a functional computer data structure that enables the computer system to perform all the steps of the method described herein when loaded onto and operated by the computer system or network.

[0033] For better understanding and to show how embodiments may be put into practice, reference will now be made, by way of example only, to the accompanying schematic drawings.

Brief Description of the Drawings

[0034]

Figure 1a

Figure 1b

Figure 1c

Figure 2a

Figure 2b

Figure 2c

Figure 2d

Figure 2e

Figure 2fg

Figure 3a

Figure 3b

Figure 3c

Figure 3d

Figure 3e

Figure 3f

Figure 3g

Figure 3h

Figure 4a

Figure 4b

Figure 4c

Figure 4d

Figure 5a

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 5f

Figure 6a

Figure 6b

Figure 6c

Figure 6d

Figure 6e

Figure 6f

Figure 6g

Figure 6h

Figure 7a

Figure 7b

Figure 8

Figure 9a

Figure 9b

Figure 9c

Figure 9d

DETAILED DESCRIPTION OF THE INVENTION

[0035] Figure 1a shows an artery 12 into which a plurality of images 10 (also called a set of images or an image frame) are captured. As shown in the figure, the images 10 are captured at regular intervals along the length of the artery 12. The number of images is merely illustrative, and it should be understood that the images may be captured along all or part of the length of the artery. The set may thus represent an overall or partial pullback along the length of the artery. Further, the interval between the images may be varied as needed.

[0036] Figure 1b schematically shows a system for capturing and processing an image captured from an artery. Such an image may be captured using an optical coherence tomography (OCT) device 14 inserted into the lumen 16 of the artery, and the image is captured as the device 14 is pulled back through the artery. For example, a pullback at a rate of 20 mm / s continues for about 2.5 seconds, enabling imaging of about 72 mm of the blood vessel. As will be described in more detail below, the system may be used to analyze the thickness of the plaque region 18 covering the inner wall of the artery. As schematically shown, atherosclerotic plaques are generally eccentric, with varying composition, which can make visualization of tissue features difficult. The OCT device 14 includes a guide catheter 42, which prevents OCT information from being captured in the shadow region 44 of the catheter 42. Further, as will be described in more detail below, there may be one or more artifacts 46 that similarly prevent OCT information from being captured in the shadow region 48. The OCT frames may be captured using standard techniques, for example, using near-infrared light having a wavelength of 1250 - 1350 nm, as described in, for example, "Consensus standards for acquisition, measurement, and reporting of intravascular optical coherence tomography studies: a report from the International Working Group for Intravascular Optical Coherence Tomography Standardization and Validation" by Tearney et al., published in J. Am. Coll. Cardiol. 59, 1058 - 1072 (2012).

[0037] The captured image may be processed or analyzed in a separate analysis device 20 that is remote from the OCT device 14, for example, located in a different location, or at least partially local to the OCT device 14 (for example, located adjacent to the patient). Using a remote analysis device 20 may allow access to more powerful resources and may enable the raw data to be processed more quickly. A local processor has the advantage of being usable in an area without a wireless connection. Thus, both local and remote processing can be performed. The analysis device 20 may be implemented in hardware as a computing device such as a server, or may be implemented in the cloud.

[0038] The analysis device 20 may include standard components such as a processor 22, a user interface 24, a storage 26, and a communication interface 28. The user interface 24 may be any standard interface including, for example, a touch sensor type display screen, voice input, keyboard input, and the like. Similarly, the storage 26 may be a conventional memory that may include RAM and ROM. Communication may be by any suitable protocol, such as Wi-Fi, Bluetooth, or a wired connection.

[0039] The analysis device 20 further includes a storage 32 for storing modules that will be executed on the device. For example, the storage 32 may include an operating system module 34 that enables the device to operate. The storage 32 may also include an artifact module 36 that can be used to remove artifacts from an image as described below. Also, as described below, there may be a measurement module 38 for determining or obtaining the interface between a plaque region (e.g., a fibrous tissue layer) and the underlying layer. Also, as described below, there is a prediction module 40 that uses measurement values from the measurement module to make various predictions. Also, as described below, there may be a module 41 related to frames that is used to classify each frame. Each of the artifact module 36, the measurement module 38, the prediction module 40, and the module related to frames may be implemented as a neural network or other suitable artificial intelligence system, each of which can operate independently of the others and given appropriate inputs. In other words, the input need not be from a peer neural network, may be other products, or may generate an independent useful final output.

[0040] The system of FIG. 1b may be considered a fully independent modular multi-metric automatic analysis system for OCT-based coronary atherosclerotic plaque analysis using deep neural networks. The three tools used to analyze OCT images are artifacts (sometimes called artifact identifiers), measurement modules, and prediction modules (sometimes called presentation predictors). As will be described in more detail below, the input includes a whole-artery pullback that is analyzed to generate micron-level measurements and context-free classifications. FIG. 1c shows a method for processing or analyzing an image that can be implemented using the system described above. In a first step (S100), an input pullback set of images is received. The images can be obtained from an image system within a patient's artery as described above. The set of images may include a standard number of images taken along a cross-section of the patient's artery, for example, in some known devices this is 270 images, although it will be understood that this is only an exemplary number. A single frame, such as the frame shown in FIG. 2a, showing an image of the wall 10 of the artery 12 captured by the OCT device 14, can thus be extracted from the set of images (step S102). Generally, OCT images are captured as grayscale, although the images may be pseudo-colored to make it easier for the clinician to examine the images. For example, in this image, the image is colored orange, but this is not standard and is only the choice of a particular OCT provider. Other OCT providers always generate gray OCT frames.

[0041] Next, for positioning the image into one of three categories, namely diseased, disease-free (i.e., healthy), or gross artifact (i.e., an artifact spanning the entire frame), there may be a frame wise classification of the first frame unit of the input image. Gross artifacts may result from large "structural" artifacts such as stents and / or guide catheters. The classification of the first frame unit may be performed by a suitably trained neural network. Such a classification of the first frame unit may serve as a screening step, find dominant frame components that may appear to be diseased and, if included, may cause incorrect measurements in subsequent processing. Such gross artifacts may be more distinguishable in the larger context of the entire frame rather than after applying the sampling described below. The frame unit classifier may accept three groups of frames.

[0042] The frame unit classification may also be related to the nature of the disease. For example, a sample may be classified as representing a disease of no interest, such as pathological intimal hyperplasia without an overlying discrete fibrous tissue. Frames classified in this way may be under the umbrella of the "healthy" classification. Alternatively, such frames may be identified when attempting to measure the information within the frame, as described below. For example, if the attempt fails due to the absence of a tissue-tissue interface (i.e., fibrous and necrotic tissue), the frame (or the sample within the frame) may be classified under this classification.

[0043] The frame-level classification is then used to determine whether the frame should be further processed (step S106). If no further processing of the frame is required, the method may loop back to step S102 to consider another input frame in the set of frames. For example, a frame classified as healthy or a frame classified as having visible artifacts may both be considered to have no useful information and may not need to be further processed. However, there may be a requirement that three consecutive frames be classified as healthy before such frames are no longer processed. Such frames that are no longer processed may be considered to be ignored or discarded. However, these discarded frames may be represented as blanks when the set of frames is input into the final prediction stage. Thus, the input to the prediction module is all of the same vertical dimension regardless of the range of the disease. Identifying which frames do not contain disease can be useful for efficiency / speed, i.e., subsequent processing such as feature measurement can be undertaken only on the series of frames with disease rather than the entire pullback. These gross classifications may also be used incidentally elsewhere, such as labeling the frames presented to the clinician. Additionally, images identified as having dominant structural features such as stents, catheters, or bifurcations may help to align pullbacks from the same artery taken at different times, e.g., before and three months after stent insertion.

[0044] Frames classified as diseased may have local (finer) artifacts that are amenable to correction. These frames are therefore processed further. The next step S108 is to reconstruct the received frame by virtually expanding the wall of the vessel, as schematically shown in Figure 2b. As shown in the figure, the appearance is similar to that of an artery with a single longitudinal incision opened and flattened out. This reconstruction results in a series of homogeneous aligned flat strips, as shown in Figure 2c. Thus, the reconstruction can be considered a sampling step that samples or extracts the received frame into a plurality of samples (or strips or segments - these terms can be used interchangeably). The samples are consistently placed along an aligned baseline that can be the lumen up to the tissue interface (i.e., the lumen edge). Each segment may be micron-sized, for example having a width of about 80 μm. Such adjustment ensures that when the neural network receives the samples as input, the network can focus on the task (i.e., learning arterial disease) rather than irrelevant features such as the OCT photoreceptor and its movement. Then at least one of the samples can be classified (step S110), as described in more detail with respect to Figures 3a through 3h. This local classification may include determining whether there is an artifact in each sample. The samples may be classified as having no artifact, having a correctable artifact, or having an uncorrectable artifact. The next step may be to determine whether to process these frames further, i.e., to determine whether to measure features within the image (step S112).

[0045] These steps of removing unusable frames and / or samples can be considered preprocessing to clean up the image. Next, the micron-sized tissue regions of each OCT frame may be processed as described below, for example, to classify cell walls and / or to measure the thickness of fibroblasts. As shown, the irrelevant lumen space can be effectively discarded, but sufficient basic OCT resolution can be used to analyze the arterial wall.

[0046] For each segment, an interface 50 between the fibrous capsule tissue and the underlying layer (such as a necrotic core or calcium) may be measured. The interface can be a feature of one of the individual measurement vectors of each tissue component. Identification of tissue planes has already been attempted using edge detectors. As shown in FIG. 2d, the left segment has a clearer edge than the right segment that appears blurred in the OCT image. Even in the case of a clearer edge, calcium immediately below the fibrous tissue shares edge gradient characteristics with the internal elastic lamina, guide wire, and other artifacts, and thus the edge may be inaccurately identified. Further, edge detection may not be effective in the case of a segment having an interface between fibrous tissue and a lipid-rich core without a typical edge. In this method, a neural network may be used to determine the interface and other features of the vector.

[0047] A neural network that uses a regression bounding box method, such as the method described in Fast R-CNN published by Girschick in 2015, may be used to identify the interface between fibrous tissue and necrotic tissue (corresponding to soft-edge gradients or hard-edge gradients, respectively), or between fibrous tissue and calcareous tissue. As described above, all samples (i.e., network inputs) are the same size, and the use of such image segments that are simplified compared to the original image means that a simpler neural network can be used. For example, the neural network may have about 15 layers.

[0048] Referring to FIG. 2e, the distance of interest is the thickness d of the fibrous tissue below the lumen edge 10, between the fibrous tissue and the underlying layer. The fibrous tissue (i.e., the plaque region) is bounded by the lumen 10 and the interface 50 (or the abluminal edge). An effective thickness was recognized for a scan line that intersects both the lumen edge and the abluminal edge around the lumen passage and starts from the index position of the center of the OCT device 14 (i.e., the center of the light source). Such measurements mimic those of a general clinician.

[0049] The regression network generally generates measurements for fixed landmarks that can be selected as the upper edge of each sample, i.e., the short edge of each sample closest to the lumen edge. Returning to FIG. 2d, the regression network may then measure the edge distance A from the upper edge of the sample to the lumen edge, and the interface distance B from the upper edge of the sample to the interface 50 (or the abluminal edge) between the fibrous tissue and the underlying layer. The fibrous capsule thickness in pixel units is estimated as the difference between the edge distance A and the interface distance B.

[0050] The pixel distances were converted to micron distances using the clinician calibration of each pullback in order for each device manufacturer to standardize the output. The average coating pixel intensity was taken as the average of the scan line vectors between the lumen and the tissue - interface points (i.e., distance d), and the edge type was an indication of the classification of each sample. The number of lesion samples (i.e., the number of segments) identified in the frame by the dimensions of each segment, which could also be, for example, a standard 20 - pixel minimum dimension, could be calculated by using the product of the dimensions of the lesion lumen perimeter. It will be appreciated that even a slight pixel error in calculating the difference between two measurements can significantly affect the measurement of fibrotic thickening. Using a neural network can increase the accuracy of these measurements when compared to other known techniques.

[0051] In this way, an accurate regression - measurement tool that combines tissue sample classification with extrapolative mapping of the tissue plane of interest may be used. The method may thus include the step of measuring the feature of interest (step S110). The extracted regression - measurement values and derived metrics may include fibrotic capsule thickness (FCT), fibrotic tissue strength, plaque composition, lumen area, and lesion lumen perimeter. As shown in Figure 1c, after the interface (or other feature of interest) is measured, another input frame is extracted and the process returns to repeat the extraction and correction steps for the next frame. This may be repeated for all or only some of the frames in the complete pullback set. The frame measurement vectors can thus be concatenated for each type of pullback, either alone or in combination, to generate classifiable features.

[0052] Figure 2f shows the average of the registered samples that averages the samples of each frame over the area of the pullback corresponding to one lesion. Figure 2f presents an expanded average similar to that shown in Figure 2c with the lumen edge at the top of the graph. The arrow indicates the shadow area of the guide wire.

[0053] FIG. 2g shows an attention map of the same region as FIG. 2f. As shown by the plot in FIG. 2f, the attention map may be generated by a neural network that performs the measurement step (step S114) shown in FIG. 1c. Both FIG. 2f and FIG. 2g are examples of concatenated measurement vectors.

[0054] Once all the measurements for each frame are determined, the padded measurement value vectors for each frame may be concatenated to produce a single input array (or pullback measurement value sequence) representing the valid measurement values for the pullback. Each final dimension (channel) may be thought of as providing a map of each individual measurement type (e.g., film thickness, film interface type, pixel intensity). As described above, blank inputs may be included to represent unmeasured images (e.g., clean healthy images). As an example, a recurrent neural network (RNN) was used with this input, which is significantly smaller (compared to a full 3D pullback) and explicitly benefits from dealing with measurements sequentially. The RNN may include long short-term memory (LSTM) cells stacked three cells deep and may have an architecture with the same number of time steps as the number of images in the original pullback (e.g., 270 time steps for 270 images). There may also be an intervening cell dropout wrapper layer, and the output may pass through a single dense layer.

[0055] The performance of the predictor module is most likely to be affected by proximal lesions. This is because these are the cause of 70% of acute coronary events, and distal disease is not of the same significance. This may be explained by the fact that proximal disease means a larger heart attack as more myocardium necroses, and proximal LAD lesions are thought to account for the majority of fatal coronary events. An exemplary paper describing this problem is "Assessment of Thin-Cap Fibroatheroma Distribution in Native Coronary Arteries" by Fiji et al., published in JACC Cardiovasc. Imaging 3, 168-175 (2010).

[0056] Figure 3a shows both a cross-sectional image and a side view of artery 112 with three locations of cross-sectional OCT images shown. The first image 110 was taken at the location where artery 112 is diseased and is shown in Figure 3b. The second image 210 was taken at a location with correctable artifacts. For example, between the 6 o'clock and 10 o'clock positions of the image shown in Figure 3c, a median lumen residual blood is seen as a swirl that attenuates most of the light. Correctable artifacts may be due to a ruptured intimal surface that attenuates the light source, lesion residual lumen blood, or a vessel wall that has entered through a branch and collateral, and the shadow adjacent to the tissue. Examples of such correctable artifacts are shown in Figures 4a through 4d. Such correctable artifacts generally constitute 0.90% of the total frames in the lesion segment.

[0057] The third image 310 was taken at locations with non-correctable artifacts and is shown in FIG. 3d. In this example, the non-correctable artifacts include acquisitions within the guide catheter, and as shown, all light is attenuated by the catheter tube. Information about the tissue outside the catheter cannot be obtained, and thus correction is not possible as there is no data for the basis of correction. Images of non-correctable artifacts and healthy, i.e., non-diseased, blood vessels are excluded as a first step when correcting the artifacts.

[0058] An artifact module including a neural network may be used to identify and correct artifacts. The input may be a low-resolution 256x256x3 frame triplet that can be classified for the presence or absence of dominant structural components (stents, guide catheters, branches / divisions), plaque types (lipid, calcified or normal vessels including intimal hyperplasia), and lumen artifacts (visible lumen residual blood and thrombus). As an example, a 6-layer convolutional architecture with a 7x7x3 kernel followed by a 3x3x3 for every other layer may be used. The architecture may have a stride of 1, 2, 2, 1, 1 and an initial 64 filters, doubling the number of filters while repeating spatial reduction every other layer and ending with a fully connected layer of 1000 units. The network may be trained by supervised training on some sample frames, e.g., 300, and then used itself to tentatively classify the frames, with corrections made to the tentative classification and subsequent re-training on those frames.

[0059] As described above, individual frames may be reconstructed as a plurality of samples. This is shown in FIG. 3e, which shows how a plurality of samples (also called windows) where the longest edge extends into the arterial wall from the lumen outward are acquired around the lumen. In this example, a window of 20x144 pixels may be used. These samples provide a generalized training target without loss of spatial information for subsequent measurements. FIG. 3f shows three of these individuals, labeled 31, 32, and 33 in FIG. 3e.

[0060] Sample artifacts are considered to be either correctable or uncorrectable based on whether the underlying tissue was detectable. For example, uncorrectable samples included in these are those where blood fills the lumen entirely, attenuating all OCT signals, and focal blood shadowing or hypo-illumination generally includes samples where less than half of the sample could be restored. Uncorrectable samples often result from the presence of a stent or guide catheter at the point where the image is captured. Such dominant structural features generally create uncorrectable artifacts except in limited situations such as when only a part of the guide catheter is within the frame, i.e., the guide catheter tip is visible. To identify the type of artifact, one of 14 visually prominent subclasses (identified by the experimenter with low feature correlation) that generate a consistent autoencoder latent space representation may be used for each sample classified as having an artifact present. These 14 classes correspond to the adversarial generation networks used to repair those samples as described below. As an example, the classes may be named, for example, "including partial guide wire shadow" and "ray shadow", or may not be named, but can be consistently distinguished from each other.

[0061] If there are correctable artifacts, the details of the sample are restored in the generative arm of a variational-autoencoder generative adversarial network (VA-GAN), as described, for example, in "Autoencoding beyond pixels using a learned similarity metric" by Larsen et al., published in 2015. Visual attribute vectors for each training sample class are generated to be added to the generator latent space, and the vectors are calculated as the difference between the mean of all samples of the class with artifacts and the mean of all samples of all other classes. The GAN was used to correctly reconstruct the diseased target for measurement while maintaining detail and spatial resolution.

[0062] Figure 3g shows the correction of the three segments of Figure 3f. Artifacts that are not correctable may be identified from failed attempts during correction. The failure can be inferred by the near absence of a continuous horizontal fibrous tissue region (e.g., the gray area in Figure 3g) in the output of the attempted correction. Thus, each class may also have an attached binary classifier that can confirm whether there are correctable artifacts present, i.e., determine whether the correction was successful. The samples for which correction was performed are then substituted with the original image instead of the incorrect samples, as shown in Figure 3h. Thus, the output is a complete image with all the samples. Multiple tissue samples, such as the sample shown in Figure 3e with small individual dimensions, means that the recovery results can be favorable, in some cases nearly perfect.

[0063] Figures 4a through 4d show different types of artifacts. Figure 4a shows an artifact resulting from blood, with a shadow caused by very small droplets of stagnant blood within the OCT catheter. Figure 4b shows an artifact caused by a protrusion of the arterial wall within a side branch casting a shadow stripe at the 4 to 5 o'clock position below it. Figure 4c is an artifact that is a combination of residual blood and the bifurcation of the left main trunk. A swirling of remaining unclear blood labeled (i) attenuates the signal and impedes good visualization of other structures. The measurable portion of the diseased artery is labeled (ii). Figure 4d is another structural artifact with light attenuation at the mouth of the bifurcation between the left anterior descending coronary artery, the intermediate artery, and the circumflex artery. Removing or correcting such artifacts helps ensure minimal feature learning based on artifacts for training while allowing the most likely representation of the tissue for prediction.

[0064] Figures 5a through 5f show how the extracted features can be used in the final step of FIG. 1c to distinguish between patients with stable angina and patients with non-ST elevation myocardial infarction (NSTEMI) or ST elevation myocardial infarction (STEMI). Figure 5a is a violin plot showing some fibrous cap thickness (FCT) measurements for patients showing NSTEMI and STEMI, with a random sample of 1 million (1e6) FCT measurements from each group for comparison. As shown, all FCT intervals appear in all patient groups. However, STEMI patients have a smaller median and mean FCT measurement for STEMI throughout the vessel. A comparison is shown in the following table.

[0065] [Table 1]

[0066] Figure 5b is an image of a cross-section of an artery of a patient showing NSTEMI, and Figure 5c is an image of a cross-section of an artery of a patient showing STEMI. Figure 5d is a plot of an FCT heat map aligned to the central baseline along the length of the artery for a patient showing NSTEMI. The mark indicates the location where the image of Figure 5b is captured. Figure 5e is a plot of an FCT heat map along the length of the artery of a patient showing STEMI with an inset of a sub-region breakdown. Again, the mark indicates the location where the image of Figure 5c is captured. Figure 5f is a plot of an FCT heat map aligned to the central baseline along the length of the artery for a patient with stable angina. The scale bar is in microns (μm).

[0067] As shown in FIGS. 5d, 5e, and 5f, there are FCT patterns characteristic of each type of patient. For example, the STEMI pattern is characterized by a sudden transition between a clear boundary of thin and thick regions that may be the site of plaque rupture. In contrast, in both NSTEMI patients and angina patients, there are more gradual or randomly distributed FCT transitions. Generally, STEMI patients also show lower FCT not only in highly localized regions but also along the entire artery compared to NSTEMI patients. It should be understood that these patterns are only simplified examples to show how a predictor module can identify the type of cardiac event / disease. The RNN network selects the unique features of the measurement series that best distinguish the type of cardiac event / disease. When a patient is diagnosed with having a particular type of heart disease, a treatment method may be recommended for the patient.

[0068] Another prediction that can be made is to predict whether a patient is at high risk of a sudden cardiac event. When the likelihood of death is predicted, it will be appreciated that the prediction module needs to be trained on an appropriate dataset that includes images from patients who died due to a sudden cardiac event. For example, as reported in "Vulnerable Plaque: The Pathology of Unstable Coronary Lesions" by Farb et al., published in J. Interv. Cardiol. 15, 439-446 (2002), a fibrous atherosclerotic cap thickness of less than 65 μm is associated with sudden cardiac events in histopathological case series. Figures 6a through 6h compare the determined performance of FCT measurements with FCT measurements from OCT by clinicians. The latter is challenging due to large datasets for a single artery, artifacts, and the difficulty of distinguishing plaque from noise at the limits of resolution. Figure 6a shows an ex vivo OCT image obtained postmortem at a point of eccentric fibrous atheroma. Figure 6b is a tissue fragment of the fibrous atheroma co-registered with Figure 6a, and Figure 6c is a high-magnification view of the area outlined in Figure 6b.

[0069] Figure 6d plots the ten smallest FCT measurements calculated by the method described above, compared to manually measured FCT for tissue structure on co-registered slices of OCT images acquired ex vivo post-mortem. In this example, the post-mortem dataset was collected from the left anterior descending coronary artery of 14 humans. The mean patient age was 71.1 ± 11.8 years, with non-obstructive cardiovascular disease being the cause of death in 57.1% of cases and the remainder being non-cardiovascular disease. The study protocol was approved by the Cambridgeshire Research and Ethics Committee 07 / H0306 / 123, together with informed consent from next of kin. OCT was collected with 40 mm of surrounding material for structural integrity and performed within 48 hours of excision on excised tissue stored in phosphate buffered saline at 4°C. Collaterals were ligated, the vessel was warmed again, and the guiding catheter was sutured into the left coronary ostium. A 0.014" Abbott Vascular BMW Universal or Pilot 50 guidewire was used for catheter delivery on a custom-built rig. OCT imaging was initiated at a physiological saline flow pressure of 100 mmHg using a DragonFly C7 catheter (St. Jude Medical) with an automatic pullback of 25.0 mm / s.

[0070] After imaging, the arteries were stored in 10% buffered formalin for more than 24 hours, and then regions of interest were defined every 5 mm along the artery, and 5-μm sections were taken every 400 μm and labeled while maintaining the longitudinal orientation with respect to the inlet. The tissue sections were stained with hematoxylin-eosin and Van Gieson and then measured by two clinicians, and each measurement was randomly reviewed by an experienced cardiac pathologist. The fibrous cap thickness was determined by the thickness of the layer of smooth muscle cells infiltrated to some extent by lymphocytes or macrophages, as described, for example, in Virmani et al., "Lessons from sudden coronary death: a comprehensive morphological classification scheme for atherosclerotic lesions," published in Arterioscler. Thromb. Vasc. Biol. 20, 1262-1275 (2000). Measurements were first made at 200-μm intervals of circumferential distance across the lipid-rich core. Along the exact same plane, a heterogeneous smooth muscle plane containing the (de)calcified area in contact with the lipid core was perceptible. An area representing calcification in contact with the lipid was included when 30-35% of the ruptures showed scattered or fragmented calcified plates, as described in Virmani et al., "Vulnerable Plaque: The Pathology of Unstable Coronary Lesions," published in J. Interv. Cardiol. 15, 439-446 (2002).

[0071] The tissue structure was initially co-registered with OCT at 5 mm intervals from the guide catheter to the exactly corresponding distance in OCT, and then at 400 μm intervals from that point. Each OCT frame formed a triplet with two frames on either side for comparison with the tissue structure as described below, and the system was carefully observed for co-registration regardless of artifacts specific to either modality. One hundred and fifty-five (75%) randomly selected and co-registered tissue fragments were used to calibrate the software measurement of fibrous cap thickness and were then excluded, and the remainder were used to form part of the validation study. In vitro data were not used to train a specific region-of-interest classifier.

[0072] Figure 6e is a Bland–Altman plot with variable mean FCT showing the difference between FCT measurements calculated by the method described above compared to manually measured FCT for tissue structure. FCT was similar using both techniques (86.6 ± 65.12 μm vs. 91.6 ± 53.2 μm, p = 0.355), and there was a significant correlation between the measurements (r = 0.90, p < 0.001).

[0073] Figure 6f is a Bland–Altman plot with variable mean FCT showing the difference between FCT measurements calculated by the method described above compared to manually measured FCT by independent cardiologists from different centers experienced in research OCT measurements using the same postmortem frames and 80 frames randomly selected from clinical trial patients (intraclass correlation coefficient for clinicians = 0.84 [0.69–0.92]). Again, automated and manually measured FCT were similar (121.8 ± 70.4 μm vs. 124.2 ± 74.9 μm, p = 0.2621). FCT measurements were validated by OCT pullbacks from 100 separate arteries in addition to 30 OCT frames from postmortem lesions not seen by the algorithm for the clinicians.

[0074] For comparison, FIG. 6g plots FCT measurements from two different clinicians. There is a significant correlation, but the FCT automatically obtained using the method described above was closer to the tissue structure FCT in 86.9% of the cases.

[0075] Another prediction that can be provided by this method is the likelihood of myocardial infarction. This can occur due to the rupture or erosion of atherosclerotic plaques, which results in intracavitary features after events such as thrombosis and standing columns of blood. The responsible vessel is often stented, but plaques upstream and downstream of the stented portion can cause further events. Since the method used above simply analyzes the thickness of the fibrous capsule, it is situation-independent, and thus, the situation-independent features used by the multi-classifier can correct for any confounding factors.

[0076] FIG. 6h is a plot showing the variation of the mean FCT when FCT measurements are taken at different times on the same day. Analysis of repeated pullbacks in 25 patients performed on the same day showed no significant difference between the mean arterial whole FCT of the IVI2M-derived (154.1 ± 22.3 to 158.3 ± 25.08 p = 0.41).

[0077] Figures 7a and 7b examine the performance of the multi-classifier in the most difficult clinical situations, including STEMI before and after stent implantation. In the case of repeat studies performed before and after stent implantation, the stent was an everolimus-eluting bioresorbable vascular scaffold. Repeat studies of the same artery were performed both before and after stent implantation, and after stent implantation and during follow-up. For this comparison, 63 total pullbacks from 18 patients (54.7 ± 7.8 years old, 88.8% male) planned for a multi-stage OCT study (both before and after PCI, and again 3 months later) were exported as individual lossless TIFF files for the repeat study. Each patient had three pullbacks, namely two on the same day (before and after PCI) and one 3 months later. Only the images showing the guiding catheter and the matching part of the vessel containing the stent were excluded. The latter was manually started by counting the number of frames showing the stent implantation segment in the post-PCI pullback, then the number of frames to the nearest reference point, the number of frames from the reference point in the pre-PCI pullback to the start of the stent, and the exact number of frames to be excluded. No image exclusion was performed outside the stent implantation segment and the guiding catheter, regardless of the degree of artifact.

[0078] Figure 7a plots the average FCT measurements after and before the stent. Figure 7b plots the average FCT measured 3 months after stent implantation and immediately after stent implantation. The results are presented in the table below and similarly show the reproducibility of the results using the automated method described above.

[0079]

Table 2

[0080]

Table 3

[0081] As described above, a neural network is used for the analysis of OCT images. FIG. 8 compares the performance of different systems that similarly incorporate a neural network using the method described above in FIG. 1c. The first comparison system, called the baseline classifier, uses a current state-of-the-art 3D convolutional residual neural network to generate a classifier for patient presentations from raw volume data of the coronary artery. The second comparison system, called the centralized baseline classifier, was a two-stage network. Both classifiers were validated for patients with stable angina (SA), ST-elevation myocardial infarction (STEMI), and non-ST-elevation myocardial infarction (NSTEMI). The multi-stage classifier was further validated postmortem against two clinicians and for a repeatable dataset. In the first stage, artifacts were identified as described above in the same manner as the described method. Correctable artifacts were corrected and non-correctable artifacts were removed. In the second stage, the remaining images were sampled and aligned as described above for consistent measurements or classification.

[0082] For each of the three methods, the training dataset (and associated 10% hold-out test ratio) was from anonymized patients who had received OCT for some reason at the Cardiothoracic Centre based at Papworth, Brighton, and Swansea hospitals, UK. 293,490 images were taken from 1087 arteries. The multi-stage classifier described in Figure 1c was further trained using an additional 47,520 images from 176 arteries that were of a different format (Terumo). These images could only be used for the multi-stage classifier due to the independence of its modules at each stage. In other words, the final stage (predictor) would not have been format-agnostic or unbiased. Training iterations were done on 90% of the training dataset along with parameter tuning, and tested and re-trained on the 10% hold-out set. Artifacts, regions of interest, and measurement algorithms were developed from 100 pull-backs (each having 270 frames) from the training dataset and updated where appropriate with histological ground truths from 14 patients who had received post-mortem OCT.

[0083] Subsequent validation was done on the non-ST elevation myocardial infarction (NSTEMI) / stable angina trial dataset from Ashford and St Peters Hospitals, UK, and the ST elevation myocardial infarction (STEMI) / stable angina dataset from Papworth. Stable angina patients at Ashford and St Peters Hospitals had typical symptoms on at least two anti-ischemic drug therapies and had >70% stenosis lesions on a separate angiogram dating back before the OCT procedure. NSTEMI was diagnosed by typical symptoms and troponin values exceeding the hospital's 99th percentile criteria. The increment was approved by the Research Ethics Service Committee South East Coast - Surrey, UK (REC Reference: 13 / LO / 0238) and registered as clinical trial NCT02335086. The patient demographics of both validation datasets are presented below.

[0084]

Table 4

[0085] The results for each of the training test dataset (i.e., 10%) and the two validation datasets are compared below and in Figure 8.

[0086]

Table 5

[0087] Accuracy may be calculated as (TP + TN) / (Real positive + Real negative), where TP is true positive and TN is true negative.

[0088]

Table 6

[0089] Excellent baseline performance was observed with the raw baseline classifier (AUC 0.86, ACC 0.79, sensitivity 0.73). However, the analysis required non - practical computing resources to train considering the instance size (512x512x256). Furthermore, focusing the classifier on the uncorrected raw frames meant a lack of the ability to ignore the range of OCT artifacts. Such major artifacts (defined as diffusion lumen artifacts that attenuate the signal) were considerably higher in myocardial infarction than in angina patients (40.2% vs 38.1% p = 0.0333).

[0090] The centralized baseline classifier improved the sensitivity (0.88) in an independent NSTEMI / stable angina dataset compared to the baseline classifier, while reducing the need for training data augmentation (2943 vs. 9783 arterial cases generated from standard distortions, i.e., rotation or inversion, brightness adjustment). For the multi-stage classifier described in Figure 1c, inputs containing fibrous cap thickness were found to be best in the NSTEMI validation dataset (AUC 0.87, ACC 0.79, and sensitivity 0.83).

[0091] The performance of the multi-stage deep learning-based classifier was also compared with the performance of conventional machine learning algorithms including the k-nearest neighbor method, random forest, and support vector machine using both individual features and frame images and panels of attributes in a random search for optimal hyperparameters. The best conventional algorithm (random forest) performed only moderately well (AUC 0.74, sensitivity 0.74, specificity 0.6) for the first (NSTEMI / SA) validation dataset, and the best estimator was born with inputs in the FCT interval of 140 μm to 220 μm.

[0092] For completeness, the details of each of the training dataset and the validation dataset are summarized in the following table (where n is the number of arteries and i is the number of OCT images). As described above, both classifiers were validated for patients with stable angina (SA), ST-elevation myocardial infarction (STEMI), and non-ST-elevation myocardial infarction (NSTEMI). The multi-stage classifier was further validated using ex vivo postmortem pullbacks correlated with tissue structure, comparison with two clinical observers, and same-day repeatability studies.

[0093]

Table 7

[0094] As described above, intracoronary images can be analyzed to predict the likelihood of disease or events and / or to track the outcomes of drugs or other treatments. Coronary artery disease (CAD) imaging has the potential to identify high-risk plaques and is a widely used surrogate efficacy marker for phase 2 / 3 drug studies. However, the event rates of putative "high-risk" lesions identified using different imaging modalities are below the event rates required to change the treatment of individual plaques. Intracoronary optical coherence tomography (OCT) generates a very high-resolution sequence that provides extremely detailed plaque images. Furthermore, several OCT parameters are associated with high-risk lesions, including a minimum fibrous cap thickness FCT (<75 μm), a minimum lumen area (MLA) < 3.5 mm 2 , a lipid pool spread within the plaque > 180°, and the presence of macrophages, calcified nodules, neovascularization, or cholesterol crystals. Some of these OCT features change with treatment using high-dose statins or ezetimibe, suggesting that they may indicate plaque stabilization. However, OCT pullbacks are rich datasets containing hundreds of images and tens of thousands of candidate measurements per artery. As a result, currently, OCT analysis requires time-consuming offline manual frame selection and measurement at specialized core laboratories and is limited by inter- and intra-observer variability. Fully automated measurements are also limited by the high frequency of artifacts and the fact that artifacts resemble diseases. Therefore, this OCT automatic analysis system / method can be used by clinicians and industry to measure disease progression / instability and distinguish different presentations in CAD to enable more cost-effective patient management.

[0095] This OCT automatic analysis system / method was used to analyze the effectiveness of a specific treatment, specifically the treatment with rosuvastatin. Images were captured at baseline (before treatment) and 13 months after treatment with 40 mg of rosuvastatin.

[0096] Currently, optical coherence tomography (OCT) is rarely used to examine the effectiveness of anti-atherosclerotic drugs due to its partial manual selection and offline analysis over a wide range of baseline and follow-up images, and it limits the analysis to a small number of frames. Symmetrically, the fully automated analysis of this OCT automatic analysis method can generate a whole plaque circumferential FCT map containing thousands of individual RCT measurements (2.0x10 3 ~1.0x10 6 measurements / artery, median 4.1x10 4 ), and based on the pixel gradients between fibrous and underlying tissues, other features related to plaque stability such as lipid content or calcification can be examined.

[0097] The study examined non-culprit artery lesions at baseline and after 13 months of treatment with 40 mg / day of rosuvastatin. FCT was measured from manually selected frames and semi-automatically measured in 31 lesions from 27 patients. In the original analysis, the minimum FCT increased, the macrophage arc and mean lipid arc decreased, and 9 / 13 thin cap fibroatheromas (TCFAs) regressed to non-TCFA morphologies. This OCT automatic analysis method enabled examination of the appearance of both plaques and the entire vessel without manual selection of frames at baseline and follow-up in 83 patients. Despite no change in the lumen surface area covered by lesions (181.68 ± 163.72 mm 2 vs 223.37 ± 199.86 mm 2 , p = 0.163), there was a significant increase in the overall plaque mean FCT at follow-up (161.98 ± 44.91 μm vs 183.99 ± 46.18 μm p = 0.0038). The minimum FCT / frame also showed a significant increase at follow-up (83.39 ± 29.87 μm vs 98.29 ± 41.00 μm, p = 0.0113), and 63.37% (95% confidence interval (CI) 59.88 - 67.58) of individual frames showed an increase in FCT at follow-up.

[0098] To determine the effect of high-intensity statin at the minimum FCT, the total arterial FCT count / artery was examined at different FCT thresholds.

[0099] Figure 9a shows box plots of the frequency of FCT measurements at different thickness thresholds (75 μm and 65 μm) between the pre-treatment baseline and the post-treatment (at 13 months) follow-up. The data are the median, and the error bars represent the 25% and 75% quartiles. The total arterial count of FCT, lipids, or calcium varies considerably between arteries due to different plaque burdens for the most part, but there is a significant decrease in the number of points where the cap thickness was minimum (<65 μm) (85,540 vs. 87,731 (number of points) or 1.9% vs. 2.1% (among all measurements), p < 0.0001), suggesting that rosuvastatin has the greatest effect on the thinnest fibrous caps.

[0100] Figure 9b is a diagram of a single patient's frame of FCT between the pre-treatment baseline and the post-treatment follow-up. The changing line represents the average FCT for each frame, and the constant horizontal line represents the average FCT for the entire pullback. The arrows indicate the same points along the patient's artery and show the regression of the thin cap area (i.e., the fibrous cap is getting thicker). It can be seen from Figure 9b that the multi-classifier also enabled a direct comparison of the FCT plots along the entire plaque at baseline and follow-up, revealing plaque regions where the FCT increased or decreased.

[0101] Figures 9c and 9d show, respectively, a box-and-whisker plot of the frequency of lipids between the baseline before treatment and the follow-up after treatment, and a box-and-whisker plot of the frequency of calcification between the baseline before treatment and the follow-up after treatment. It can also be seen from Figures 9c and 9d that the proportion of lipids decreased slightly but significantly from 20.00% at baseline to 18.06% at follow-up (p<0.0025), and calcium increased across the arterial level (24588 vs 38105 (number of points) or 3.1% vs 4.9% (as a percentage of all measurements), p = 0.016). Therefore, it can be seen that this OCT automatic analysis method can be used to monitor the effectiveness of anti-atherosclerotic treatment and to predict patient events.

[0102] At least some of the exemplary embodiments described herein may be configured, in part or whole, using dedicated special-purpose hardware. As used herein, terms such as "component," "module," or "unit" include, but are not limited to, hardware devices that perform or provide related functions in the form of discrete or integrated components, field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs). In some embodiments, the elements described may be configured to exist on a tangible, persistent, addressable storage medium and may be configured to operate on one or more processors. These functional elements may, in some embodiments, include, by way of example, components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Although described herein with respect to the components, modules, and units of the exemplary embodiments, such functional elements may be combined with fewer elements or divided into additional elements in some cases. While various combinations of select features have been described herein, it will be understood that the described features may be combined in any suitable combination. Specifically, the features of any one exemplary embodiment may be combined with the features of any other embodiment as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the terms "comprising" or "comprises" mean including the specified (one or more) components, but do not exclude the presence of others.

[0103] Note all documents and papers filed in connection with this application and published simultaneously with or before this specification and made available for general viewing with this specification. Also, the content of all such documents and papers is hereby incorporated by reference into this specification.

[0104] All features disclosed in this specification (including the appended claims, abstract, and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive.

[0105] Unless otherwise specified, each feature disclosed in this specification (including the appended claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose. Accordingly, unless otherwise specified, each feature disclosed is only an example of a generic series of equivalent or similar features.

[0106] The present invention is not limited to the details of the above (one or more) embodiments. The present invention extends to any novel single one, or any novel combination, of the features disclosed in this specification (including the appended claims, abstract, and drawings), and / or to any novel single one, or any novel combination, of the steps of any method or process so disclosed.

Description of Reference Numerals

[0107] 10 Image 12 Artery 14 Optical Coherence Tomography (OCT) Device 16 Lumen 18 Plaque Region 20 Analysis Device 22 Processor 24 User Interface 26 Storage 28 Communication Interface 32 Storage 34 Operating System Module 36 Artifact Module 38 Measurement Module 40 Prediction Module 42 Catheter 44 Shadow Region 46 Artifact 48 Shadow area 50 Interface surface

Claims

1. A method for analyzing a set of coronary artery images by a computer, comprising: For each image in the set of images, Classifying the image for the presence or absence of diseased tissue using a first neural network; When the image is classified as having diseased tissue, classifying the image for the presence or absence of artifacts using a second neural network; Determining whether to analyze the image based on the step of classifying using the first neural network and the step of classifying using the second neural network; When the image is to be analyzed, analyzing the image by identifying one or more features of interest of the coronary artery tissue using a third neural network; Measuring each identified feature of interest; A method comprising: The set of images includes a first subset of coronary artery images of a patient taken at a first time and a second subset of coronary artery images of the patient at a second, subsequent time, Measuring a first set of measurements for each identified feature of interest for the first subset of images; Measuring a second set of measurements for each identified feature of interest for the second subset of images; Further comprising determining a change in the coronary artery using the first and second sets of measurements.

2. The method of claim 1, further comprising sampling the coronary artery in the image into a plurality of samples and inputting a set of samples into the second neural network before classifying the image using the second neural network.

3. The step of classifying the image comprises determining, for each sample in the input set, a detectable ratio of coronary artery tissue; classifying the image based on the determined ratio; The method according to claim 2, comprising:

4. training the second neural network to classify the image by analyzing a plurality of training dataset samples; The method according to claim 2 or 3, further comprising:

5. The method according to any one of claims 2 to 4, further classifying the artifact as correctable or non-correctable when the image is classified as having an artifact.

6. The method according to claim 3, further classifying the artifact as correctable or non-correctable when the image is classified as having an artifact, and classifying the image as having a correctable artifact when the determined ratio exceeds 50%.

7. The method according to claim 5 or 6, comprising correcting the image classified as having a correctable artifact before analyzing the image.

8. The method according to claim 7, wherein the step of correcting the image classified as having a correctable artifact comprises applying an adversarial generation network of a variational autoencoder to the identified image to restore details of the coronary artery tissue underlying the identified artifact.

9. The method according to any one of claims 1 to 8, replacing the image with blank information when the image is classified as having no diseased tissue.

10. The method according to any one of claims 1 to 9, wherein the step of measuring each identified feature of interest includes the step of measuring any one or more of fibrous tissue thickness, plaque composition, lumen area, and lesion lumen circumference.

11. The method according to any one of claims 1 to 10, further comprising the steps of sampling the coronary artery in the held image into a plurality of samples and constructing the samples in a linear representation of the coronary artery before analyzing the image.

12. The step of analyzing the image further includes the step of identifying the interface between fibrous tissue and necrotic or calcified tissue in each of the plurality of samples using a regression bounding box technique The method according to claim 11.

13. The method according to claim 12, wherein the step of measuring the identified feature of interest includes the step of measuring the distance between the interface and the edge of the sample.

14. The method according to any one of claims 1 to 13, wherein the set of images of the coronary artery is an optical coherence tomography image.

15. Each image is of a patient's coronary artery, and the method further includes the step of using the measurement of each identified feature of interest to determine the likelihood that the patient exhibits specific signs of coronary artery disease The method according to any one of claims 1 to 14.

16. The method according to claim 15, wherein the step of determining the likelihood that the patient exhibits specific signs of coronary artery disease includes the step of using a fourth neural network.

17. The method according to claim 16, wherein the fourth neural network receives, as input, the measured values of each identified feature of interest for each analyzed image and blank information for each unanalyzed image.

18. The method according to any one of claims 15 to 17, wherein the coronary artery disease is angina pectoris or myocardial infarction, and the identified feature of interest is fibrous tissue thickness.

19. The method according to claim 1, wherein the first time is a first stage in the treatment, the second time is a second stage in the treatment, and further comprising the step of determining the effectiveness of the treatment based on any determined changes.

20. An apparatus for analyzing a set of coronary artery images, an imaging device for capturing a set of coronary artery images and at least one processor coupled to a memory, for each image in the set of images, classifying the image using a first neural network for the presence or absence of diseased tissue; when the image is classified as having diseased tissue, classifying the image using a second neural network for the presence or absence of artifacts; determining whether to analyze the image based on the classification, and when the image is analyzed, analyzing the retained image by using a third neural network to identify one or more features of interest within the coronary artery tissue and measuring each identified feature of interest; at least one processor configured to perform An apparatus comprising wherein the set of images includes a first subset of images of a patient's coronary artery captured at a first time and a second subset of images of the patient's coronary artery at a second, subsequent time, for the first subset of images, the at least one processor measuring a first set of measurements for each identified feature of interest; Measuring a second set of measurements for each identified feature of interest for said second subset of images; An apparatus further configured to determine changes in said coronary artery using said first and second sets of measurements. **Claim 21** The apparatus according to claim 20, wherein said imaging device is an optical coherence tomography device.

Images