A deep learning based approach for OCT image quality assurance

Abstract

Aspects of the present disclosure relate to systems, methods, and algorithms for training a machine learning model or neural network to classify OCT images. The neural network or machine learning model may receive annotated OCT images indicating which portions of the OCT image are occluded and which portions are clear, and a classification of the OCT image as clear or occluded. After training, the neural network may be used to classify one or more new OCT images. A user interface may be provided to output the results of the classification and summarize the analysis of the one or more OCT images.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of vasculature imaging and data collection systems and methods, and more particularly to methods for improving image quality detection and image categorization in Optical Coherence Tomography (OCT) systems.

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 220,722, filed July 12, 2021, the disclosure of which is incorporated herein by reference. [Background technology]

[0003] Optical coherence tomography (OCT) is an imaging technique that uses light to capture cross-sectional images of tissues at the micron scale. OCT can be a catheter-based imaging modality that uses light to peer into the walls of coronary or other arteries and generate images of them for study. Utilizing coherent light, interferometry, and micro-optics, OCT enables video-rate in vivo tomography within diseased blood vessels with micrometer-level resolution. Seeing subsurface structures at high resolution using a fiber-optic probe makes OCT particularly useful for minimally invasive imaging of internal tissues and organs. This level of detail possible with OCT allows physicians to diagnose and monitor the progression of coronary artery disease.

[0004] OCT images can be corrupted for a variety of reasons. For example, an OCT image can be corrupted due to the presence of blood in a blood vessel when an OCT image of that vessel is acquired. The presence of blood can prevent proper identification of the vessel boundary during an intravascular procedure. A corrupted image can be unuseful for interpretation or diagnosis. For example, during a "pull-back," a procedure in which an OCT device is used to scan the length of a blood vessel, thousands of images can be acquired, some of which can be corrupted, inaccurate, or unuseful for analysis due to the presence of blood obscuring the lumen contour during the OCT pull-back.

[0005] Identifying which OCT images are corrupted requires manual, frame-by-frame or image-by-image analysis of hundreds or thousands of images acquired during an OCT scan of the blood vessel, and furthermore, this analysis is performed after the OCT procedure is completed, potentially requiring additional OCT scans to obtain better quality images of the portion of the blood vessel that corresponds to the corrupted image.

[0006] The additional equipment required to detect the presence of blood may alter typical clinical workflow, degrade image quality, or otherwise increase the complexity in clinical implementation. Other tools developed to detect potentially inaccurate lumen detection have been shown to be unreliable and do not directly detect whether the captured OCT image is occluded by blood and therefore are not useful for interpretation. Summary of the Invention

[0007] Real-time or near real-time identification directly from the images of which images or groups of images are degraded allows those images to be ignored when interpreting the OCT scan, allowing the obstructed blood vessel portions to be rescanned while the OCT device is still in situ.

[0008] Aspects of the disclosed technology allow for calculation of the clear image length (CIL) of the OCT pullback, which can be a measure of the contiguous sections of the OCT pullback that are not obstructed by, for example, blood.

[0009] Aspects of the disclosed technology include a method of classifying a diagnostic medical image. The method may include receiving a diagnostic medical image, analyzing the diagnostic medical image in real time or near real time using a trained machine learning model, the trained machine learning model being trained on a set of annotated diagnostic medical images, determining an image quality of the diagnostic medical image based on the analysis, and outputting an indication of the determined image quality in real time or near real time for display on a user interface. The diagnostic medical image may be a single image of a series of diagnostic medical images. The series of diagnostic medical images are acquired through optical coherence tomography pullback. The diagnostic medical image may be classified as a first classification or a second classification. An alert or notification may be provided if the diagnostic medical image is classified as the second classification. The set of annotated diagnostic medical images may include annotations including "clear", "blood", or "guide catheter". The diagnostic medical image may be an optical coherence tomography image. The diagnostic medical image may be classified as a clear medical image or a blood medical image. A probability indicating whether the diagnostic medical image is acceptable or unacceptable may be calculated. A threshold method may be used to convert the calculated probability into a classification of the diagnostic medical image. A graph cut may be used to convert the calculated probability into a classification of the diagnostic medical image. A morphological classification may be used to convert the calculated probability into a classification of the diagnostic medical image. "Acceptable" may mean that the diagnostic medical image is above a predefined threshold quality that allows for evaluation of a characteristic of the human tissue with a higher accuracy or confidence than a threshold level. A clear image length or a clear image length indicator may be displayed or output.

[0010] Aspects of the disclosed technology may include a system comprising a processing device coupled to a memory storing instructions that cause the processing device to receive a diagnostic medical image, analyze the diagnostic medical image in real time or near real time using a trained machine learning model, the trained machine learning model being trained on a set of annotated diagnostic medical images, determine an image quality of the diagnostic medical image based on the analysis, and output an indication of the determined image quality in real time or near real time for display on a user interface. The diagnostic medical image may be an optical coherence tomography (OCT) image. The instructions may be configured to display a plurality of OCT images with an indicator associated with a classification of each image of the plurality of OCT images. The series of diagnostic medical images may be acquired through an optical coherence tomography pullback.

[0011] Aspects of the disclosed technology may include a non-transitory computer readable medium including program instructions that, when executed, perform the steps of receiving a diagnostic medical image, analyzing the diagnostic medical image in real time or near real time with a trained machine learning model, the trained machine learning model being trained on a set of annotated diagnostic medical images, determining image quality of the diagnostic medical image based on the analysis, and outputting an indication of the determined image quality in real time or near real time for display on a user interface. The diagnostic medical image may be a single image in a series of diagnostic medical images. The series of diagnostic medical images may be acquired through optical coherence tomography pullback. The diagnostic medical image may be classified as a first classification or a second classification. An alert or notification may be provided if the diagnostic medical image is classified into the second classification. The set of annotated diagnostic medical images may include annotations including "clear", "blood", or "guide catheter". The diagnostic medical image may be classified as a clear medical image or a blood medical image. A probability indicating whether the diagnostic medical image is acceptable or unacceptable may be calculated. A threshold method may be used to convert the calculated probability into a classification of the diagnostic medical image. A graph cut may be used to convert the calculated probability into a classification of the diagnostic medical image. A morphological classification may be used to convert the calculated probability into a classification of the diagnostic medical image. "Acceptable" may mean that the diagnostic medical image is above a predefined threshold quality that allows for evaluation of the characteristics of the human tissue with a higher accuracy or confidence than the threshold level. A clear image length or a clear image length indicator may be displayed or output. The unclassifiable images may be saved to retrain the trained machine learning model. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an imaging and data collection system according to an aspect of the present disclosure. [Figure 2A] 1 is a "clear" OCT image according to an embodiment of the present disclosure. [Figure 2B] 1 is an annotated "clear" OCT image according to an embodiment of the present disclosure. [Figure 3A] 1 is an "occluded" OCT image according to an embodiment of the present disclosure. [Figure 3B] 1 is an annotated "occluded" OCT image according to an embodiment of the present disclosure. [Figure 4] 1 is a histogram associated with a training set of data, according to an aspect of the present disclosure. [Diagram 5] FIG. 13 is a flow diagram of a training process according to an aspect of the present disclosure. [Figure 6] FIG. 1 is a flow diagram relating to an aspect of classifying OCT images, according to an aspect of the present disclosure. [Figure 7] 1A-1C illustrate aspects of techniques that may be used to classify or group a sequence of OCT images, according to aspects of the present disclosure. [Figure 8] 13A-13C illustrate user interfaces related to aspects of lumen contour confidence and image quality, according to aspects of the present disclosure. [Figure 9] FIG. 1 illustrates a method that may be used to generate or calculate a clear image length (CIL) for OCT pullback, according to an embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates an example CIL cost matrix, according to an aspect of the present disclosure. [Figure 11] FIG. 1 illustrates an example OCT pullback having a CIL incorporated therein, according to an embodiment of the present disclosure. [Figure 12] FIG. 1 is a flow diagram illustrating an example method for ensuring image quality using a machine learning task-based approach, according to aspects of the present disclosure. [Figure 13A] 13 is an example image from a lumen detection task, according to an aspect of the present disclosure. [Figure 13B] FIG. 13B illustrates an example task result for the image of FIG. 13A, according to an embodiment of the present disclosure. [Figure 13C] FIG. 13 illustrates an example confidence result for a lumen detection task, according to aspects of the present disclosure. [Figure 14A] 13D is an example graph illustrating confidence values ​​associated with FIG. 13C, according to an embodiment of the present disclosure. [Figure 14B] 13D is an example graph illustrating confidence values ​​associated with FIG. 13C, according to an embodiment of the present disclosure. [Figure 15A] 13 is another example image from a lumen detection task, according to an aspect of the present disclosure. [Figure 15B] FIG. 15B illustrates another example task result for the image of FIG. 15A, according to an embodiment of the present disclosure. [Figure 15C] FIG. 13 illustrates another example confidence result for a lumen detection task, according to aspects of the present disclosure. [Figure 16A] 15D is an example graph illustrating confidence values ​​associated with FIG. 15C, according to an embodiment of the present disclosure. [Figure 16B] 15D is an example graph illustrating confidence values ​​associated with FIG. 15C, according to an embodiment of the present disclosure. [Figure 17A] FIG. 13 illustrates an example confidence tally per A-line frame, according to aspects of the present disclosure. [Figure 17B] FIG. 13 illustrates an example confidence tally per frame pullback, according to aspects of the present disclosure. [Figure 18] 1 is a screenshot of an example user interface according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present disclosure relates to a system, method, and non-transitory computer readable medium for identifying poor quality medical diagnostic images in real time through the use of machine learning based techniques. Non-limiting examples of medical diagnostic images include OCT images, intravascular ultrasound (IVUS) images, CT scans, or MRI scans. For example, an OCT image is received and analyzed using a trained machine learning model. In some examples, the trained machine learning model may output a probability after analyzing the image. In some examples, the output probability may relate to the probability that the image belongs to a particular category or classification or not. For example, the classification may relate to the quality of the acquired image and / or whether the quality is sufficient to perform further processing or analysis. In some examples, the classification may be a binary classification such as "acceptable / unacceptable", "clear / blocked", etc.

[0014] The machine learning model can be trained based on an annotated or marked set of data. The annotated or marked set of data may include classifications or identifications of portions of an image. According to some examples, the set of training data can be marked or classified as "blood occluded" or "not blood occluded". In some examples, the training data can be marked as acceptable or unacceptable / occluded. In some examples, the set of data may include OCT images acquired during one or more OCT pullbacks. In some examples, one or more sets of training data may be selected or stratified such that each set of training data has a similar distribution of classifications of data.

[0015] The training set of data may be manipulated, such as by augmenting, modifying, or altering the set of training data. Training of the machine learning model may also be performed on the manipulated set of training data. In some examples, the use of augmented, modified, or altered training data may generalize the machine learning model and prevent overfitting of the machine learning model.

[0016] After categorizing the OCT images by the trained machine learning model, or obtaining the probability that the image belongs to a particular category, post-processing techniques may be used on the images before displaying information related to the images to a user. In some examples, post-processing techniques may include rounding techniques, graph cuts, erosion, dilation, or other morphological methods. Additional information related to the analyzed OCT images may also be generated and used when displaying output related to the OCT images to a user, such as information indicating which OCT images were unacceptable or occluded.

[0017] As used in this disclosure, OCT image or OCT frame may be used interchangeably. Additionally, as used in this disclosure, an "unacceptable" or "obstructed" OCT image is one in which the lumen and vessel walls are not clearly imaged due to the presence of blood or other fluids, as will be understood by those skilled in the art.

[0018] Although the examples provided herein are primarily described with respect to OCT images, those skilled in the art will appreciate that the techniques described herein may be applied to other imaging modalities.

[0019] FIG. 1 illustrates a data collection system 100 for use in collecting intravascular data. The system may include a data collection probe 104 that may be used to image a blood vessel 102. A guidewire, not shown, may be used to introduce the probe 104 into the blood vessel 102. The probe 104 may be introduced and retracted along the length of the blood vessel while collecting data. As the probe 104 is retracted, i.e., pulled back, multiple scans or OCT and / or IVUS data sets may be collected. The data sets, or frames of image data, may be used to identify characteristics such as blood vessel dimensions, pressure and flow characteristics.

[0020] The probe 104 may be connected to a subsystem 108 via an optical fiber 106. The subsystem 108 may include a light source such as a laser, an interferometer having a sample arm and a reference arm, various optical paths, a clock generator, photodiodes, and other OCT and / or IVUS components.

[0021] The probe 104 may be connected to an optical receiver 110. According to some examples, the optical receiver 110 may be a balanced photodiode-based system. The optical receiver 110 may be configured to receive light collected by the probe 104.

[0022] The subsystems may include a computing device 112. The computing device may include one or more processors 113, memory 114, instructions 115, data 116, and one or more modules 117.

[0023] The one or more processors 113 may be any conventional processor, such as a commercially available microprocessor. Alternatively, the one or more processors may be dedicated devices, such as application specific integrated circuits (ASICs) or other hardware-based processors. Although FIG. 1 functionally depicts the processor, memory, and other elements of device 112 as being in the same block, those skilled in the art will appreciate that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be housed in the same physical housing. Similarly, the memory may be a hard drive or other storage medium located in a housing different from that of device 112. Thus, it will be understood that reference to a processor or computing device includes reference to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0024] The memory 114 may store information accessible by the processor, including instructions 115 that may be executed by the processor 113, and data 116. The memory 114 may be any type of memory operable to store information accessible by the processor 113, including non-transitory computer-readable media or other media that store data that may be read with the aid of electronic devices, such as hard drives, memory cards, read-only memory ("ROM"), random access memory ("RAM"), optical disks, and other writable and read-only memories. The subject matter disclosed herein may include various combinations of the foregoing, such that different portions of the instructions 115 and data 116 are stored in different types of media.

[0025] The memory 114 may be retrieved, stored, or modified by the processor 113 according to the instructions 115. For example, the data 116 may be stored in a computer register, a relational database as a table with multiple different fields and records, an XML document, or a flat file, although the disclosure is not limited by any particular data structure. The data 116 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. By way of mere further example, the data 116 may be stored as a bitmap composed of pixels stored compressed or uncompressed, or various image formats (e.g., JPEG), vector-based formats (e.g., SVG), or computer instructions for drawing graphics. Additionally, the data 116 may include sufficient information to identify relevant information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memories (including other network locations), or information used by a function to calculate relevant data. The memory 114 may also include or store a set of training data, such as OCT images, that are used with the machine learning model to train the machine learning model to analyze OCT images that are not included in the set of training data.

[0026] The instructions 115 may be any set of instructions, such as machine code, that are executed directly by the processor 113, or instructions, such as scripts, that are executed indirectly. In this regard, the terms "instructions," "application," "steps," and "programs" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language, including scripts or a collection of independent source code modules that are interpreted on demand or pre-compiled. The functions, methods, and routines of the instructions are described in more detail below.

[0027] Modules 117 may include a display module. In some examples, additional types of modules may be included, such as modules for calculating other vessel characteristics. According to some examples, the modules may include an image data processing pipeline or its component modules. The image processing pipeline may be used to convert the collected OCT data into two-dimensional ("2D") and / or three-dimensional ("3D") views and / or representations of the vessels, stents, and / or detected regions. Modules 117 may also include image recognition and image processing modules for identifying and classifying one or more elements of the image.

[0028] The module 117 may include a machine learning module. The machine learning module may include machine learning algorithms and models, including neural networks and neural nets. The machine learning module may include a machine learning model, which may be trained using a set of training data. In some examples, without limitation, the machine learning module or machine learning algorithm may include or be made of any combination of a convolution neural network, a perceptron network, a radial basis network, a deep feed forward network, a recurrent neural network, an autoencoder network, a gated recurrent unit network, a deep convolution network, a deconvolution network, or a support vector machine network. In some examples, the machine learning algorithm or model may be configured to take a medical diagnostic image, such as an OCT image, as an input and provide as an output a probability that the image belongs to a particular classification or category.

[0029] The subsystem 108 may include a display 118 for outputting content to a user. As illustrated, the display 118 is separate from the computing device 112, although according to some examples, the display 118 may be part of the computing device 112. The display 118 may output image data related to one or more features detected in the blood vessel. For example, the output may include, but is not limited to, cross-sectional scan data, longitudinal scans, diameter graphs, image masks, and the like. The output may further include visual indicators of lesions and blood vessel characteristics or characteristics of the lesion, such as calculated pressure values, or blood vessel size and shape. The output may further include information related to the collected OCT image, such as areas where the acquired OCT image was not "clear," or summary information about the OCT scan, such as the overall quality of the scan. The display 118 may identify features using text, arrows, color coding, highlighting, outlines, or other suitable human or machine readable indicia.

[0030] According to some examples, the display 118 may include a graphic user interface ("GUI"). According to other examples, a user may use other forms of input, such as a mouse, keyboard, trackpad, microphone, gesture sensor, or any other type of user input device, to interact with the computing device 112 such that certain content is output on the display 118. One or more steps may be performed automatically or without user input, such as to navigate an image, input information, select an input, and / or interact with an input. The display 118 and input device, along with the computing device 112, may enable transitions between different stages in a workflow, different viewing modes, and the like. For example, a user may select a segment of a vessel for viewing an OCT image and associated analysis of the OCT image, such as whether the image is deemed acceptable / clear or unacceptable / occluded, as described further below.

[0031] FIG. 2A shows a "clear" OCT image. Shown in FIG. 2A is a clear OCT image 200. The OCT image 200 is a cross-sectional representation of a portion of vascular tissue. The OCT image may be non-uniform and may be of varying degrees, intensity, and shape and may include artifacts such as bright concentric rings or bright structures emerging from the guidewire. Shown in image 200 is a lumen 205 and a centrally located OCT guidewire 210 contained within the periphery of an OCT guide catheter 215. The OCT image 200 is clear because there is no obstruction to viewing the lumen or artifacts in the image other than the guide catheter. In the OCT image 200, the outline of the lumen is visible in the image and the presence of blood, if any, is minimal or below a predefined threshold. The OCT image 200 is thus "clear."

[0032] FIG. 2B shows an annotated clear OCT image 250, which is an annotated version of image 200. For reference, OCT guidewire 210 and OCT guide catheter 215 are labeled in FIG. 2B. Annotated clear image 250 is an annotated or marked version of clear OCT image 200, marking the lumen with lumen annotation 251. Similar to lumen annotation 251, the guide catheter may also be annotated, as depicted by the dashed line in FIG. 2B. In some examples, a particular set of annotations may be used to train a machine learning model, while other annotations may be ignored or not used for training. For example, the guide catheter may be predicted to be present in all OCT images and therefore not used to train the machine learning model or to later categorize new images. Image 250 may be categorized as "clear" since the lumen is generally visible and there are no major obstructions to viewing the lumen.

[0033] Image 250 may also be associated with a tag, metadata, or placed into a category such as "clear" to indicate that the image is considered clear when used to train a machine learning model. The machine learning model may be configured to perform classification of new images. Classification is a technique for determining which class a dependent variable belongs to based on one or more independent variables. Thus, classification takes one or more independent variables as input and outputs a classification or a probability associated with a classification. For example, image 250 may be part of a set of machine learning training data used to train a machine learning model to classify new images. By using the categorization of images in a set of data used to train a machine learning model, including images such as image 250 and their associated categories, a machine learning algorithm may be trained to evaluate which features or combinations of features lead to a particular image being categorized as "clear" or in another category.

[0034] Figure 3A shows an "occluded" OCT image. Shown in Figure 3 is an occluded OCT image 300. As can be seen in OCT image 300, a portion of the image is occluded by blood 301 surrounding the central guide catheter 315 and guidewire 310 in the upper left portion of the image.

[0035] In some instances, the degree of occlusion that is considered "occluded" or "unacceptable" may be configurable by the user or may be pre-set during manufacture. By way of example only, an image in which 25% or more of the lumen is obstructed by blood may be considered to be an "occluded" image.

[0036] 3B shows an annotated "occluded" OCT image. Annotated occluded OCT image 350 shows an annotated version of occluded OCT image 300. Annotation 351 (solid circular line) illustrates the lumen portion, annotation 352 indicates the portion of the lumen that is occluded by blood 301, and annotation 353 (convex closed shape in the upper left) indicates the portion of the OCT image that is blood. Similar to the clear annotated image 250, the occluded annotated image 350 may also be associated with a tag, metadata, or placed into a category such as "not clear," "occluded," or "blood" to indicate that the image is considered unacceptable or unclear when used for training in a machine learning model.

[0037] FIG. 4 illustrates a histogram 400 associated with a training set of data. The training data may include OCT images with various degrees of clearness or occlusion, such as those described above in connection with FIGS. 2 and 3. The training set of data may further include additional information, such as annotations, metadata, measurements, or other information corresponding to the OCT images that can be used to classify the OCT images. The training set of data may include any number of images, with a large number of images providing improved accuracy of the machine learning model. For example, hundreds or thousands of OCT images may be used, which may be obtained from various OCT pullbacks or other OCT measurements. The relative proportions of images that have already been categorized or that consist of a guide catheter, are occluded by blood, or are clear are shown or visualized in the histogram 400. The relative proportions of images may be adjusted or adjusted to tune the training of the machine learning model being trained. The training set of data may be adjusted to have an appropriate proportion of these various categories to ensure proper training. For example, if a training set is "unbalanced," such as by including more images that are clear, the machine learning model may not be well trained to distinguish features that make an image not "clear," and may be biased toward artificially boosting performance by simply classifying a majority of images as "clear." By using a more "balanced" training set, this problem may be avoided.

[0038] FIG. 5 illustrates a flow diagram of a method 500. The method 500 may be used to train a neural net, neural network, or other machine learning model. The neural network or neural network may be comprised of a collection of simulated neurons. Training the neural network may include weighting various connections between neurons or connections of the neural network. Training the neural network may be performed in epochs in which an error associated with the network may be observed until the error sufficiently converges. In some examples, without limitation, the neural net or neural network may be a convolutional neural network, a perceptron network, a radial basis network, a deep forward propagation network, a recurrent neural network, an autoencoder network, a gated recurrent unit network, a deep convolutional network, a deconvolutional network, a support vector machine network, or any combination of these or other types of networks.

[0039] At block 505, a set of medical diagnostic images may be acquired. In some examples, the set of medical diagnostic images may be acquired from OCT pullback or other intravascular imaging techniques. In other examples, the set of medical diagnostic images may be randomized or taken from various samples, specimens, or vascular tissues to provide a large sample size of images. The set of medical diagnostic images may be similar to OCT image 200 or OCT image 300.

[0040] A set of medical diagnostic images may be prepared for use as a data set for training a machine learning model in block 510. In this block, one or more techniques may be used to prepare the set of medical diagnostic images for use as training data.

[0041] For example, the medical diagnostic images may be annotated. A portion of each medical diagnostic image from a set of medical diagnostic images may be annotated to create an image similar to, for example, the annotated clear OCT image 250 or the annotated occluded OCT image. For example, each image may have a portion of the image annotated as "clear" or "blood" to represent the portion of the image that represents the image. For example, a set of medical diagnostic images that may be used for training may be annotated or categorized to create images similar to the annotated clear OCT image 250 and the annotated occluded OCT image 350. In other examples, annotations may be digitally drawn on the images to identify portions of the image that correspond to particular features, such as a lumen, blood, or a guide catheter. In some examples, the annotation data may be represented as a portion of an image or a set of pixels.

[0042] The medical diagnostic images may also be categorized or separated into several categories. In some examples, the categorization may be performed by a human operator. For example, the medical diagnostic images may be classified between a binary set of values, such as [unacceptable, acceptable], [not clear, clear], [occluded, not occluded] or [unuseful, useful]. In some examples, a non-binary classification may be used, such as a set of classifications that may indicate the percentage of occlusion, for example [0% occluded, 20% occluded, 40% occluded, 60% occluded, 80% occluded, or 100% occluded]. Each medical diagnostic image may be placed into the category that best represents the medical diagnostic image.

[0043] In some examples, multiple types of classifications may be used for a medical diagnostic image. A medical diagnostic image may be associated with multiple sets of categories. For example, if a medical diagnostic image has a stent and there is a high probability that blood is obstructed, the classification for the image may be <stent, obstructed>. Another example may be whether a frame contains a guide catheter or not, and the classification for the image may be <catheter, obstructed>. Multiple classifications may be used together during training of a machine learning model or classification of data.

[0044] In some examples, the training data set may be pruned or adjusted to include a desired distribution of occluded and clear images.

[0045] A set of medical diagnostic images may be reworked, manipulated, modified, corrected, or generalized prior to use in training. Manipulation of medical diagnostic images allows training of machine learning models to be balanced with respect to one or more characteristics, rather than overfitted to a particular characteristic. For example, medical diagnostic images may be resized, transformed using a random Fourier series, flipped in polar coordinates, randomly rotated, adjusted for contrast, brightness, intensity, noise, grayscale, scale, or other adjustments or modifications may be applied. In another example, an arbitrary linear mapping represented by a matrix may be applied to an OCT image. Underfitting may occur if the model is too simple, such as having only two features, and does not accurately represent the complexity required to categorize or analyze new images. Overfitting may occur if the trained model does not generalize sufficiently to solve the general problem intended to be represented by the training set of data. For example, if a trained model categorizes images more accurately in a training set of data, but has lower accuracy for a test set of data, the trained model is said to be overfitted. Thus, for example, if all images are of one orientation or have a particular contrast, the model may be overfitted and not be able to accurately categorize images that have different contrast ratios or are oriented differently.

[0046] In block 515, a neural network, neural net, or machine learning model may be trained using the categorized data set. In some examples, training of the machine learning model may proceed in epochs until an error associated with the machine learning model has sufficiently converged or stabilized. In some examples, the neural network is trained to classify images, such as with a binary set of images. For example, the neural network may be trained based on a set of training data that includes clear and occluded images, and may be trained to output either "clear" or "occluded" as an output.

[0047] In block 520, the trained neural net, neural network, or machine learning model may be tested. In some examples, a neural network may not be used to train the network, but may be tested based on images whose classification is otherwise known. In some examples, images that are deemed to be "edge cases" when analyzed, such as images that cannot be clearly classified, may be used to retrain the neural network after manual classification of the image. For example, if the determination of whether a particular image depicts a blood-filled or clear vessel cross-section has low confidence, the particular image may be saved for analysis by a human operator. Once categorized by the human operator, the image may be added to the set of data used to train the machine learning model, and the model may be updated with the new edge case images.

[0048] In block 525, a learning curve, such as a loss curve or an error rate curve, for various epochs of training of the machine learning model may be displayed. In some examples, each epoch may be associated with a unique set of OCT images used to train the machine learning model. The learning curve may be used to evaluate the effect of each update during training, and measuring aspects of the model during each epoch or update and plotting the performance may provide information about the characteristics and performance of the trained model. In some examples, a model may be selected such that it has the smallest validation loss, such that the validation loss training curve is most significant. Blocks 515 and 520 may be repeated until the machine learning model is sufficiently trained such that the trained model has the desired performance characteristics. As an example, the computation time or computation intensity of the trained model may be a performance characteristic below a certain threshold.

[0049] The model may be saved at the epoch with the lowest validation loss, and this model with trained characteristics may be used to evaluate performance metrics against a test set that may not have been used in training. If the performance of such a model exceeds a threshold, the model may be considered to be sufficiently trained. Other characteristics associated with the machine learning model may also be examined. For example, a receiver operating characteristic curve or a confusion matrix may be used to evaluate the performance of a trained machine learning model.

[0050] FIG. 6 provides a flow diagram illustrating a method 600 for classifying images in a medical diagnostic procedure. The method 600 may be used to characterize an OCT image or a series of OCT images. For example, the method 600 may be used to characterize a series of OCT images associated with an OCT pullback in which OCT images corresponding to a particular length of vascular tissue, such as an artery, are acquired. Such characterization may be used to indicate to a physician in real time whether an image having a predefined threshold image quality was acquired. In this regard, if the image quality of the OCT pullback was not sufficient, the physician may perform another pullback within the same medical procedure while the OCT probe and catheter are still in the patient's blood vessel, rather than requiring a follow-up procedure in which the OCT catheter and probe would need to be reinserted.

[0051] One or more unlabeled OCT images may be received in block 605. The received OCT images may be associated with specific locations within the vascular tissue, which locations may later be used to generate various representations of the data acquired during OCT.

[0052] In block 610, the received OCT image may be analyzed or classified using a trained neural network, trained neural net, or trained machine learning model. The trained neural network, trained neural net, or trained machine learning model has been trained and tuned to identify various features, such as lumen or blood, from a training set of data. These parameters may be identified using image or object recognition techniques. In another example, a set of characteristics may be collected from the image or image data that may be known or hidden variables during training of the machine learning model or neural network. For example, the relative color, contrast, or roundness of elements of the image may be known variables. Other hidden variables may be derived during the training process and may not be directly identified, but are related to the image provided. Other variables may be related to image metadata, such as which OCT system took the image. In another example, the trained neural network may have weights between various neurons or connections of the network based on the training of the network. These weighted connections may take an input image and weight different parts of the image or features contained within the image to generate a final result, such as a probability or classification. In some examples, each input image has associated manual annotations, so the training may be considered supervised.

[0053] The trained neural network, trained neural net, or trained machine learning model may take an OCT image as an input and provide a classification of the image as an output. For example, the output may be whether the image is "clear" or "occluded." In some examples, the neural network, neural net, or machine learning model may provide a probability associated with a received OCT image, such as whether the OCT image is "clear" or "occluded."

[0054] In some examples, such as that described with respect to FIG. 7, additional methods may be used to classify or group a sequence of OCT images.

[0055] In other examples, multiple neural networks or machine learning models may be used to process the OCT images. For example, any number of models may be used and the probability results of the models may be averaged to provide a more robust prediction or classification. The use of multiple models may optionally be used when a particular image is difficult to classify or is an edge case where one model is not able to clearly classify the results of the OCT image.

[0056] In block 615, the output received from block 610 may be appended or otherwise associated with the received OCT image. This information may be used when displaying the OCT image to a user.

[0057] In block 620, information about the OCT images and / or information about the OCT image quality may be provided to the user on a user interface. Additional examples of user interfaces are provided with respect to FIG. 8. For example, information may be displayed with each OCT image or a summary of the OCT scan or OCT pullback. In some examples, a longitudinal view of the vessel, such as that shown in FIG. 8, may be created from a combination of OCT images, and information about which portions of the vessel were not imaged due to "occluded" images may be displayed with the longitudinal view.

[0058] In another example, summary information about the scan may be provided to a user for viewing on a display. The summary information may include information such as the number of frames or OCT images that were deemed occluded, or the overall percentage of OCT images that were deemed clear, and may identify areas where clusters of OCT images were occluded. In another example, the summary information or notification may provide additional information about why a particular frame was occluded, such as that the OCT pullback was performed too quickly.

[0059] FIG. 7 illustrates an aspect of a technique that may be used to classify or group a sequence of OCT images from a probability. Shown in FIG. 7 is a graph 710 that represents the probability that a particular image is "clear" or "occluded" on a scale of 0 to 1. Graph 710 is a raw probability value that may be obtained from a trained machine learning model or neural network. A probability of 0 means that the image is considered completely clear, and a probability of 1 means that the image is considered occluded. Values ​​between 0 and 1 may represent the likelihood that the image is clear or occluded. The horizontal x-axis in graph 710 may represent a frame number of a sequence of OCT images or OCT frames, such as those acquired during OCT pullback. The horizontal x-axis may also relate to the proximal or distal location of the vascular tissue imaged to create the OCT image.

[0060] Graph 720 illustrates the use of a "threshold" approach to classify the probability distribution of graph 710 into binary classifications. In the threshold approach, OCT images with probability values ​​above a certain threshold may be considered "occluded" and those with probability values ​​below the same threshold may be considered "clear." Thus, graph 710 may be used as input and graph 720 may be obtained as output.

[0061] Graph 730 illustrates the use of a graph cut approach to classify the probability distributions of graph 710. For example, a graph cut algorithm may be used to classify the probabilities as either "cleared" or "occluded."

[0062] Graph 740 illustrates the use of a morphological approach to classify the probability distribution of graph 710. Morphological approaches apply a structuring element to an input image to create an output image of the same size. In a morphological operation, the value of each pixel in the output image is based on a comparison of the corresponding pixel in the input image with its neighboring pixels. The probability values ​​of graph 710 may be compared in this manner to create graph 740.

[0063] FIG. 8 illustrates an example user interface 800 illustrating aspects of lumen contour confidence and image quality. User interface 800 illustrates a linear representation of a series of OCT images in component 810, with the horizontal axis indicating location or depth within the vascular tissue. Indicator 811 in component 810 may represent the current location within the vascular tissue or depth within the vascular tissue represented by OCT image 820. Indicator 812 may be a colored indicator corresponding to the horizontal axis. Indicator 812 may be colored, such as red, to represent the probability or confidence that the OCT image associated with that location is "occluded" or "clear." In some examples, there may be a white or transparent overlay over the portion of the image corresponding to indicator 812 to further indicate that the region is of low confidence. Image 820 may be an OCT image at the location represented by indicator 812. Image 820 may also include coloring or other indicators to indicate portions of the lumen that are regions of low confidence. The user interface 800 may include an option to rerun the OCT pullback or to accept the results of the OCT pullback.

[0064] In some examples, additional metadata related to the image 820 can be displayed on the user interface 800. For example, additional information about the image is available, such as the image resolution, the imaging wavelength used, granularity, suspected diameter of the OCT frame, or other metadata related to the OCT pullback that can assist the physician in evaluating the OCT frame.

[0065] As shown in FIG. 8, the interface may further provide prompts to the physician in response to notifications or other information related to the machine learning evaluation of the images. For example, the prompts may provide the physician with the option to accept the collected images and proceed to the next step of the procedure, or to repeat the image collection step, such as by performing another OCT pullback. For example, the user interface 800 may include a prompt 830 that allows the OCT pullback to be repeated. Upon selecting or interacting with the prompt 830, the computing device may configure the OCT device to receive additional OCT frames. The interface 800 may also include a prompt 831 that allows the OCT results to be accepted. Upon interacting with the prompt 831, additional OCT frames will not be accepted. In addition, as will be further described with reference to FIGS. 9-11, the user interface 800 may display a clear image length (CIL) of the OCT pullback. In some examples, the user interface 800 may suggest or request that the OCT pullback be performed again if the CIL is less than a predetermined length.

[0066] FIG. 9 illustrates a method 900. The method 900 may be used to generate or calculate a clear image length (CIL) of an OCT pullback. The clear image length or CIL may be an indicator or information related to a continuous section of an OCT pullback that is determined to be unobstructed or clear, such as not obstructed by blood or not considered a blood frame. A CIL vector score for a pullback of "n" frames may be calculated with a value between 0 and n. A score of 0 may represent a complete mismatch, and a score of n means a complete match. With reference to FIG. 10, an example of a CIL vector score is given. Match refers to a classification that matches the CIL classification. In some examples, within the CIL classification, everything in the "exclusion zone" may be a "0" and everything outside the exclusion zone may be a "1". If the CIL classification matches the frame-by-frame classification, a "1" may be added to the score, and if not, a "0" may be added to the score. The CIL with the highest score may be selected.

[0067] For a given OCT pullback, a frame-by-frame quality assurance classification may be performed for each OCT image in the pullback in block 905. In some examples, a binary classifier may be used, resulting in a score of 0 or 1 for each OCT frame. In other examples, a value ranging between 0 and 1 may be generated for each OCT frame, such as through the use of an assembly technique.

[0068] In block 910, an exhaustive search of marker locations is performed, such as marker x1 and marker x2. In some examples, x1 may correspond to a blood marker and x2 may correspond to a clear marker. For example, referring to FIG. 8, markers 840 and 841 may correspond to x1 and x2, respectively. By varying markers 840 and 841, all combinations may be evaluated. After performing the search for each location, permutations for each x1 and x2 location such that x2>x1 are calculated, and approximately N 2This may lead to a computational complexity of 1 / 2.

[0069] In block 915, for each permutation, a cost associated with that permutation may be calculated and a global optimum or maximum of the cost may be determined. In some examples, the cost may be calculated by summing the number of matches between the score vectors of the automatic image quality vector and the corresponding CIL score vector. An example of a calculated score is given with reference to FIG. 10. The maximum point in FIG. 10 may correspond to the longest or largest CIL in the OCT pullback. The location of the maximum value in this cost matrix is ​​the optimum x1 and x2 location for the resulting CIL. In some examples, the CIL is the "best" possible continuous range of non-blood frames, but may still include some blood frames. In some examples, the CIL may be a measure of the location of the bolus of contrast agent in the pullback. In other examples, it may be possible to have some blood frames within this bolus due to side branches and mixing of the bolus with blood.

[0070] In some examples, the CIL may be calculated automatically during OCT pullback. In some examples, information related to the CIL may be used by downstream algorithms to avoid processing images obstructed by blood to improve the performance of the OCT imaging system and improve the computational efficiency of the OCT system.

[0071] In block 920, a CIL indicator may be plotted on the OCT image based on the calculated optimal or maximum CIL. For example, the CIL may be plotted between colored dashed lines. If there are OCT frames detected or classified as "blood" frames outside of the CIL, those frames may be overlaid with a transparent red color to indicate that they are "blood" frames. If there are frames detected as blood within the CIL, those frames may be visually smoothed and displayed as a transparent red color.

[0072] FIG. 10 illustrates an example CIL cost matrix 1000. The cost matrix 1000 may be the matrix in the top right corner for values ​​of x2≧x1. Region 1005 may be the region of allowable or feasible values ​​of x1 and x2. Also illustrated in the cost matrix 1000 is point 1010, which is the maximum discussed with reference to block 910. Point 1010 may be calculated from the values ​​of x1 and x2 within region 1005. Point 1010 may correspond to a maximum of a cost function. In some examples, region 1005 may be colored with gradients to indicate strength and cost in a 2D format, and point 1010 may be chosen to be the maximum of the cost function.

[0073] 11 shows an example OCT pullback 1100 with a CIL incorporated in the OCT pullback. The CIL incorporated in the OCT pullback can also be seen with respect to FIG. 8. For example, with reference to FIG. 8, markers 840 and 841 can correspond to x1 and x2, respectively. The CIL can be for the length between markers 840 and 841.

[0074] The OCT pullback 1100 may be displayed on a graphical user interface or user interface, such as user interface 800 (FIG. 8). The horizontal axis of the OCT pullback 1100 may indicate an OCT frame number, a location within the vascular tissue, or a depth within the vascular tissue. Shown in FIG. 11 are various indicia included on the OCT pullback 1100. Dashed lines 1105 and 1106 may indicate a boundary of the CIL. Shown within the boundary of the CIL are blood region 1115 and blood region 1116, shown as blurred regions. Region 1120 to the left of dashed line 1105 indicates a region outside the boundary of the CIL. In some examples, region 1120 may include an overlaid translucent, transparent, or semi-transparent image to visually provide a visual indication to the user that the region is outside the CIL. Location indicator 1130 may indicate a location within the OCT pullback 1100 that corresponds to OCT frame 1135.

[0075] The technology may provide real-time or near real-time notifications including information related to image quality as the OCT procedure is being performed based on the trained machine learning model or trained neural network. For example, the notification may be an icon, text, an audible indicator, or other form of notification that alerts the physician regarding the classification made by the machine learning model. For example, the notification may identify the image as "clear" or "occluded." According to some examples, the notification may include a quantification of how much blood obstruction is blocking the vessel in a particular image frame or vessel segment. This allows the physician to have an immediate indication of whether the data and images being acquired are clear enough for diagnostic or other purposes, eliminating the need to manually check hundreds or thousands of images after the procedure is performed. Since it may be impractical to manually check every OCT image, the technology prevents improper interpretation of inadequate or not sufficiently clear OCT scans.

[0076] In addition, because the analysis may be performed in real time, notifications or alerts related to the OCT images may indicate which portions of the OCT scan or OCT pullback were not of sufficiently clear quality (or were obstructed) and allow those portions of the OCT scan or OCT pullback to be performed. This allows the physician to perform another OCT scan or OCT pullback of the portions that were not sufficiently clear while the OCT device is still in place, avoiding the need for the patient to return for another procedure. Additionally, the computing device may replace the portions of the scan that were deemed defective or obstructed with a new set of OCT images and "stitch" or combine the images to provide a single longitudinal view of the vessels acquired in the OCT pullback.

[0077] Additionally, identification of portions of the OCT scan or OCT pullback that are not deemed acceptable or clear may be evaluated by a physician to determine whether the regions corresponding to the occluded OCT images are of concern to the physician.

[0078] Additionally, a summary of the OCT scan or OCT pullback may be provided to the user. For example, the summary information may include information about the overall percentage or number of frames that were deemed acceptable, whether a second scan is likely to improve that percentage of frames, etc. In other examples, the summary information or notification may provide additional information about why a particular frame was occluded, such as the OCT pullback was performed too quickly, or blood was not displaced, etc.

[0079] In some examples, a user or physician can define whether an image is clear or obstructed, such as by setting a threshold used in detecting image quality, and in other examples, a confidence level of a computational task can be used to determine whether an image is sufficiently clear. For example, a task-based image quality assessment method is described herein. A task-based image quality assessment method can be beneficial in that it does not require a human operator to select high and low quality image frames to train a predictive model. Rather, image quality is determined by the confidence level of the task being accomplished. Image quality assurance methods can accommodate the evolution of the technology used in the computational task. For example, as technology for accomplishing a task improves, the image quality assurance results can evolve with it to more realistically reflect image quality. Task-based image quality assurance can help users retain as many OCT frames as possible while ensuring the clinical usefulness of these frames.

[0080] 12 is a flow diagram illustrating an example method 1200 of ensuring image quality using a machine learning task-based approach. The task can be any of a variety of tasks, such as lumen contour detection, calcium detection, or detection of any other characteristic. Lumen contour detection can include, for example, geometric measurements, vessel wall or border detection, hole or opening detection, curve detection, etc. Such detection can be used in assessing the severity of vascular stenosis, identifying side branches, identifying stent struts, identifying plaque, EEL or other media, or other types of vascular evaluation.

[0081] In block 1210, data is collected for the task. The data may be, for example, intravascular images, such as OCT images, ultrasound images, near-infrared spectroscopy (NIRS), micro-OCT images, or any other type of image. In some examples, the data may also include information, such as patient information, image capture information (e.g., date, time, image capture device, operator, etc.), or any other type of information. The data may be collected from one or more patients using one or more imaging probes. According to some examples, the data may be retrieved from a database that stores multiple images captured from multiple patients over a period of time. In some examples, the data may be presented in a polar coordinate system. According to some examples, the data may be manually annotated, such as to indicate the presence and location of a lumen contour if the task is to identify a lumen contour. Furthermore, the data may be divided into a first subset used for training and a second subset used for validation.

[0082] In block 1220, a machine learning model is trained using the collected data. The machine learning model can be configured according to a task. For example, the model can be configured to detect lumen contours. Training the model can include, for example, inputting the collected data that matches the task. For lumen detection, training the model can include inputting an image depicting a lumen contour.

[0083] At block 1230, the machine learning model is optimized based on the training data. In the example of a lumen contour detection task, the model input can be a series of gray-level OCT images, which may be in the form of 3D patches. A 3D patch is a stack of consecutive OCT images, the size of the stack depends on the computational resources such as the memory of a graphical processing unit (GPU). The model output during training can include a binary mask of each corresponding stack, which is manually annotated by a human operator. Since manual annotation of the 3D patches is time-consuming, a data augmentation pre-processing step can be included before optimizing the machine learning model. Data augmentation can be performed on the annotated data using variations such as random rotation, cropping, flipping, geometric transformation of the 3D patches of both the OCT images and the annotations, so that a sufficient training data set is generated. The data augmentation process can vary depending on the type of task. Once the data augmentation step is determined, the loss function and optimizer are specified as cross-entropy and Adam optimizer. Similarly, the loss and optimizer (as well as other hyperparameters in the training process) can vary depending on the type of task and image data. A machine learning model is optimized until a loss function value, which is a measure of the discrepancy between the model's calculated output and the predicted output, is minimized within a given number of iterations or epochs.

[0084] In block 1240, the validation set of data can be used to assess the accuracy of the machine learning model. For example, the machine learning model can be run using the validation data to determine whether the machine learning model produced the predicted results for the validation data. For example, the annotated validation image can be compared to the output of the machine learning model to determine the degree of overlap between the annotated validation image and the machine learning output image. The degree of overlap can be expressed as a number, a ratio, an image, or any other mechanism for assessing the degree of similarity or difference. The machine learning model may be further optimized by making adjustments that account for discrepancies between the predicted results for the validation data and the output results for the validation data. The assessment of accuracy and optimization of the machine learning can be repeated until the machine learning model outputs results with a sufficient degree of accuracy.

[0085] At block 1250, the optimized machine learning model can provide an output for the task along with a confidence value corresponding to the output. For example, for the task of detecting a lumen contour, the confidence value can indicate how likely it is that a portion of an image does or does not contain a contour.

[0086] Although the method 1200 is described above with respect to one task, in other examples, a confidence value may be obtained based on multiple tasks by integrating information from each task. In any example, the confidence value may be output along with the image frame being assessed. For example, the confidence value may be output as a numerical value on a display. In other examples, the confidence value may be output as a visual, audio, tactile, or other indicator. For example, the indicator may be a color, a shade, an icon, text, or the like. In some examples, the visual indicator may designate a particular portion of the image to which the confidence value corresponds, and a single image may have multiple confidence values ​​corresponding to different portions of the image. In further examples, the indicator may be provided only if the confidence value is above or below a certain threshold. For example, if the confidence value is below a threshold, indicating a low quality image, the indicator may inform the physician that the image is not clear enough. If the confidence is above a threshold, the indicator may inform the physician that the image is acceptable. Such a threshold may be automatically determined through the machine learning optimization described above. Image quality indicators not only capture the clarity of the image itself, but also provide reliable image characterization results throughout the analysis pipeline, such as for the assessment of disease states using diagnostic medical imaging systems.

[0087] 13A-13C show images processed using the machine learning model described above in relation to FIG. 12. In each of FIG. 13A-13C, the horizontal axis shows the pixels of the A-line and the vertical axis represents the A-line of the image frame. The A-line may be, for example, a scan line. If the imaging probe rotates as it passes through the blood vessel, each rotation may include multiple A-lines, for example, hundreds of A-lines.

[0088] FIG. 13A is an intravascular image, such as an OCT image. FIG. 13B is the output of a machine learning model. For example, for a machine learning model for a lumen detection task, the model output can be a binary mask. White pixels in the binary mask represent the detected lumen and black pixels represent the background. FIG. 13C is a confidence map for lumen detection. Each pixel is represented by a floating point number between 0 and 1, with 0 indicating no confidence and 1 indicating full confidence. The visualization in FIG. 13C inverts the values ​​by (1-confidence value) to represent uncertainty. As shown in FIG. 13C, part of the lumen is outside the field of view, resulting in an A-line with low confidence.

[0089] To assess the quality of an image frame, the information embedded in the confidence map can be converted into a binary decision as a high quality frame or a low quality frame. Given the confidence maps of all OCT frames, for each frame i, the confidence values ​​of the pixels on each A-line are converted into a single confidence value that represents the quality of the entire A-line.

[0090] 14A and 14B provide histograms showing the difference between A-lines of high and low confidence quality. If the lumen detection task identifies a clear segmentation between lumen and non-lumen for an A-line, the computational model used in the task will classify pixels on the A-line as either lumen or background with high confidence. Therefore, the histogram will show confidence values ​​mostly 0 and 1. However, if the image quality along the A-line is low, the model will be less confident in determining pixels as lumen or background. The corresponding histogram will then clearly visualize this, and several probability values ​​between 0 and 1 will be presented. The difference of such histograms can be calculated by using the entropy defined in the following formula:

number

[0091] Ei,j represents the entropy of the quality of the i-th A-line in frame j, a is the index of the pixel on the i-th A-line, n is the number of pixels on the i-th A-line, and p is the probability of the confidence value of the pixel at location (i, a).

[0092] Figure 14A shows an example of the entropy on a high-confidence A-line. In this example, the entropy according to the above formula is 0.48. Figure 14B shows an example of the entropy on a low-confidence A-line, where the entropy is 22.64.

[0093] The jth frame quality can be determined by the following formula:

number

[0094] T 2 The value of can be determined, for example, based on a receiver operating characteristic (ROC) analysis. For example, T 2 The value of T may depend on factors or settings that can be defined by the user, such as sensitivity, specificity, positive predictive value, etc. As an example, if the user prefers to capture all low quality images, the sensitivity may be set close to 100% and T 2 may be set relatively low, such as 0-10%. This may result in a high number of false positives, where an image frame is classified as "bad" even if only a few pixels are not clear. In another example, T may be set to "0" to classify fewer image frames as "bad". 2 may be set higher. By way of example only, T 2 may be set to approximately 70%, 50%, 30%, 20% or any other value.

[0095] 15A-15C show another example of image quality detection using a machine learning model. In this example, the acquired image frame shown in FIG. 15A is an image with blood artifacts. Despite the blood spreading throughout the lumen, the segmentation task is accomplished properly. Therefore, the mask in FIG. 15B depicts a clear boundary between the white pixels representing the lumen and the black pixels representing the background. Moreover, the output in FIG. 15C shows a high confidence for the detected contours. Since the model used in this task is robust against blood artifacts, the A-line histograms in FIG. 16A and FIG. 16B show that the confidence values ​​mostly fall into the buckets of 0 and 1. The entropy values ​​are low, 0.44 and 2.08. As a result, the frame in FIG. 15A is classified as good quality.

[0096] 17A and 17B show the aggregated output of the confidence assessment. FIG. 17A shows the quality of all A-lines of all frames during pullback, with pixel intensity indicating the quality of the A-lines. Using frame qualities such as those determined using the above formula, the OCT image quality can be determined as shown in FIG. 17B, where 0 indicates low quality and 1 indicates high quality. Certain post-processing may be applied to the result to ensure that the longest clear image length with minimal uncertainty is provided to the user.

[0097] Although the above formula is for a metric of entropy, other metrics can be used. By way of example, such other metrics may include randomness or variance of the data sequence. Confidence or uncertainty metrics can be calculated from various types of statistics, such as standard deviation, variance, or various forms of entropy, such as Shannon entropy or computational entropy. The above thresholds may be determined either by receiver operating characteristic (ROC) analysis or empirical determination.

[0098] According to some examples, an indicator of image quality consistent with the task-based quality metric can be output. The quality indicator can be, for example, visual, audio, tactile, and / or other types of indicators. For example, the system can play a distinctive audio tone if the captured image meets a threshold quality. As another example, the system can place a visual indicator on a display that outputs images acquired during an imaging procedure. In this regard, the physician performing the procedure will know immediately whether a sufficient image has been acquired, thereby reducing the potential need for a subsequent procedure to acquire a clearer image. The reduced need for a subsequent procedure results in improved patient safety.

[0099] 18 is a screenshot of an example user interface for an imaging system that provides a visual indicator of the quality of an image frame. The imaging system can be, for example, an intravascular imaging system such as OCT, ultrasound, NIRS, micro-OCT, etc. In other examples, real-time quality assessment and indications can be provided for other types of medical or non-medical imaging.

[0100] The example of Figure 18 includes a frame view 1810 and a segment view 1820. The frame view 1810 can be a single image of multiple images in the segment view 1820. For example, a frame indicator 1821 in the segment view 1820 can identify which frame corresponds to the frame currently being rendered in the frame view 1810 relative to other frames in the segment. In the example of an intravascular imaging procedure, the frame view 1810 may render a cross-sectional view of the blood vessel being imaged, and the segment view 1820 renders a longitudinal view of a segment or portion of the blood vessel being imaged.

[0101] The example of FIG. 18 is for OCT pullback where the task is to detect lumen contour. The task may be identified by the physician prior to initiating pullback, such as by selecting an input option through a user interface. The quality indicators may be specific to the selected task. For example, for a task of detecting lumen contour, the indicators may identify where the image or portions of the image depicting the lumen contour are clear or not clear. For a task of detecting calcium, the indicators may identify where calcium is shown in the image relative to a threshold degree of certainty. According to some examples, multiple tasks may be selected such that the user interface portrays quality indicators associated with the multiple tasks. For example, a first indicator may be provided associated with lumen contour and a second indicator is provided associated with calcium. The first and second indicators may be the same or different types of color, shade, text, annotation, alphanumeric value, etc.

[0102] As seen in the frame view 1810, the lumen contour is clearly imaged in a first portion 1812 of the image at the lower right side of the image. The lumen contour is less clearly imaged in a second portion 1814 of the image at the upper left side of the image. The first portion 1812 clearly shows the boundary between the lumen wall and the lumen, while the second portion 1814 shows the boundary less clearly. In this example, the frame view indicator 1815 corresponds to the second portion 1814 where the lumen contour is not clearly depicted. The frame view indicator 1815 is shown as a colored arc that extends partially around the circumference of the lumen cross section. The angular distance covered by the arc corresponds to the angular distance of the second portion 1814 where the lumen contour is not clearly imaged. For example, the frame can be evaluated pixel by pixel such that image quality is assessed for each pixel and an image quality indicator can correspond to a particular pixel. Thus, the frame view indicator 1815 may identify particular portions of an image where the image quality falls below a particular threshold.

[0103] Although the frame view indicator 1815 is shown as a colored arc, it should be understood that any of a variety of other types of indicators may be used. By way of example only, such other types of indicators may include, but are not limited to, overlays, annotations, shading, text, and the like. According to some examples, the indicators may depict degrees of quality for various portions of the image. For example, the arcs in FIG. 18 may be gradations, such as degrees of color, shading, or transparency, with one end of the spectrum corresponding to low quality and the other end of the spectrum corresponding to high quality.

[0104] The segment view 1820 may also include an indicator of image quality. As shown, the segment quality indicator 1825 may indicate the quality of each image frame along the imaged vessel segment. In the example of FIG. 18, the segment quality indicator 1825 is a colored bar that extends along the length of the segment view. The colored bar includes a first color that indicates where the frame quality is above a threshold and a second color that indicates where the frame quality is below the threshold. For example, the threshold may correspond to a portion or percentage of each frame where the image according to the task was captured with sufficient clarity. Such a threshold may be, for example, a threshold T described in connection with the frame quality equation above. 2 In this example, a first portion 1827 of the segment quality indicator 1825 is a first color that corresponds to frames in the segment that have sufficient quality above a threshold. A second portion 1829 of the segment quality indicator 1825 is a second color that corresponds to frames in the segment that have a low quality below a threshold, such as the frames shown in the frame view 1810. In this example, the segment quality indicator 1825 uses color to distinguish the quality of each frame along the segment, although in other examples, the segment quality indicator 1825 may use other indicia, such as shading, gradients, annotations, etc. Additionally, while the segment quality indicator 1825 is illustrated as a bar, it should be understood that indicia of any other shape, size, or format may be used.

[0105] Although some of the above examples are described in the context of OCT images, the above described automatic real-time quality detection techniques using direct deep learning or lumen confidence can be applied to any of a variety of medical imaging modalities, including, but not limited to, IVUS, NIRS, micro-OCT, etc. For example, a machine learning model for lumen detection can be trained using lumen-annotated IVUS images. A confidence signal from the model can be used to assess image quality. As another example, IVUS frames can be annotated as high or low quality, and a direct deep learning approach to detect image quality can be applied in real-time image acquisition during an IVUS procedure. As yet another example, when using high-definition intravascular ultrasound (HD-IVUS), a saline flush can be used to clear blood to provide improved IVUS image quality. In such cases, a quality detection technique can be applied to distinguish between flushed and unflushed regions of the vessel. In a further example, the quality detection technique can be based on IVUS parameters such as grayscale or axial / lateral resolution. For example, a machine learning model can be trained to detect whether an image is acquired at a threshold resolution. It should be understood that any of a variety of further applications of the techniques described herein are also possible.

[0106] Aspects of the disclosed technology may include the following combinations of features. (Feature 1) 1. A method for classifying diagnostic medical images, comprising: receiving a diagnostic medical image; analyzing diagnostic medical images in real time or near real time using a trained machine learning model, the trained machine learning model being trained on a set of annotated diagnostic medical images; determining an image quality of the diagnostic medical image based on the analysis; outputting the determined image quality indicators in real time or near real time for display on a user interface; The method comprising: (Feature 2) 2. The method of claim 1, wherein the diagnostic medical image is a single image of a series of diagnostic medical images. (Feature 3) 3. The method of claim 2, wherein the series of diagnostic medical images are acquired through optical coherence tomography pullback. (Feature 4) 2. The method of claim 1, further comprising classifying the diagnostic medical image as a first classification or a second classification. (Feature 5) 5. The method of any of features 1-4, further comprising providing an alert or notification if the diagnostic medical image is classified into the second classification. (Feature 6) 2. The method of claim 1, wherein the set of annotated diagnostic medical images includes annotations including "clear," "blood," or "guide catheter." (Feature 7) 2. The method of claim 1, wherein the diagnostic medical image is an optical coherence tomography image. (Feature 8) 2. The method of claim 1, further comprising classifying the diagnostic medical image as a clear medical image or a blood medical image. (Feature 9) 2. The method of claim 1, further comprising the step of calculating a probability indicating whether the diagnostic medical image is acceptable or unacceptable. (Feature 10) 10. The method of claim 9, further comprising using a thresholding method to convert the calculated probability into a classification of the diagnostic medical image. (Feature 11) 10. The method of claim 9, further comprising using graph cuts to convert the calculated probabilities into classifications of diagnostic medical images. (Feature 12) 10. The method of any of features 1-9, further comprising using morphological classification to convert the calculated probabilities into classifications of diagnostic medical images. (Feature 13) 10. The method of any one of features 1 to 9, wherein acceptable means that the diagnostic medical image is of higher than a predefined threshold quality that allows assessment of the characteristics of human tissue with a higher accuracy or confidence than the threshold level. (Feature 14) 1. A system comprising a processing device coupled to a memory storing instructions, the instructions causing the processing device to: receiving a diagnostic medical image; Analyzing diagnostic medical images in real time or near real time with a trained machine learning model, the trained machine learning model being trained on a set of annotated diagnostic medical images; determining image quality of the diagnostic medical image based on the analysis; outputting the determined image quality indicators in real time or near real time for display on a user interface; A system that allows the above to be performed. (Feature 15) 15. The system of feature 14, wherein the diagnostic medical image is an optical coherence tomography (OCT) image. (Feature 16) 16. The system of feature 15, wherein the instructions are configured to display the plurality of OCT images along with an indicator associated with a classification of each image of the plurality of OCT images. (Feature 17) 17. The system of any of features 14-16, wherein the series of diagnostic medical images are acquired through optical coherence tomography pullback. (Feature 18) A non-transitory computer readable medium containing program instructions that, when executed, receiving diagnostic medical images; analyzing diagnostic medical images in real time or near real time using a trained machine learning model, where the trained machine learning model is trained on a set of annotated diagnostic medical images; Based on the analysis, identify the image quality of diagnostic medical images; Outputting the determined image quality metrics in real time or near real time for display on a user interface. A non-transitory computer-readable medium for carrying out the steps of: (Feature 19) 20. The non-transitory computer-readable medium of feature 18, wherein the diagnostic medical image is a single image of a series of diagnostic medical images. (Feature 20) 20. The non-transitory computer-readable medium of feature 19, wherein the series of diagnostic medical images are acquired through optical coherence tomography pullback. (Feature 21) 21. The non-transitory computer readable medium of any of Features 18-20, further comprising classifying the diagnostic medical image as the first classification or the second classification. (Feature 22) 22. The non-transitory computer readable medium of any of Features 18-21, further comprising providing an alert or notification if the diagnostic medical image is classified into the second classification. (Feature 23) 23. The non-transitory computer-readable medium of any of features 18-22, wherein the set of annotated diagnostic medical images includes annotations including "clear," "blood," or "guide catheter." (Feature 24) 23. The non-transitory computer readable medium of any of Features 18 to 22, wherein the diagnostic medical image is an optical coherence tomography image. (Feature 25) 25. The non-transitory computer readable medium of any of Features 18-24, further comprising classifying the diagnostic medical image as a clear medical image or a blood medical image. (Feature 26) 20. The non-transitory computer readable medium of feature 18, further comprising calculating a probability indicating whether the diagnostic medical image is acceptable or unacceptable. (Feature 27) 27. The non-transitory computer readable medium of any of Features 18-26, further comprising using the non-transitory computer readable medium of the threshold to convert the calculated probability into a classification of the diagnostic medical image. (Feature 28) 30. The non-transitory computer-readable medium of feature 27, further comprising storing the unclassifiable images for retraining a trained machine learning model. (Feature 29) 20. The non-transitory computer-readable medium of feature 18, further comprising outputting a clear image length or a clear image length indicator. (Feature 30) 15. The system of feature 14, wherein the instructions are configured to display a clear image length or a clear image length indicator. (Feature 31) 2. The method of claim 1, further comprising displaying or outputting a clear image length or a clear image length indicator.

[0107] The aspects, embodiments, features, and examples of the present disclosure are considered in all respects to be illustrative and are not intended to be limiting of the disclosure, the scope of which is defined solely by the claims. Other embodiments, modifications, and uses will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure as claimed.

[0108] The use of headings and paragraphs in this application is not meant to limit the disclosure; each paragraph may apply to any aspect, embodiment, or feature of the disclosure.

[0109] Throughout this application, when a composition is described as having, including, or comprising particular components, or a process is described as having, including, or comprising particular process steps, it is intended that the composition of the present teachings consists essentially of or consists of the recited components, and that the process of the present teachings consists essentially of or consists of the recited process steps.

[0110] When an element or component is referred to in this application as being included in and / or selected from a list of enumerated elements or components, it should be understood that the element or component can be any one of the enumerated elements or components, or can be selected from a group consisting of two or more of the enumerated elements or components. Furthermore, it should be understood that the elements and / or features of the compositions, devices, or methods described herein, whether expressly or implicitly stated herein, can be combined in various ways without departing from the spirit and scope of the present teachings.

[0111] Use of the terms "include," "includes," "including," "have," "has," or "having" should generally be understood to be open-ended and non-limiting, unless otherwise specified.

[0112] The use of the singular herein includes the plural (and vice versa) unless otherwise specified. Additionally, the singular forms "a," "an," and "the" include the plural unless the context clearly dictates otherwise. Additionally, when the term "about" or "substantially" is used before a quantitative value, the present teachings also include the specific quantitative value itself unless otherwise specified. The term "about" or "substantially" as used herein refers to variations in quantity that may occur, for example, through real-world measuring or handling procedures, through accidental errors in these procedures, through differences / faults in the manufacture of materials such as composite tapes, through defects, as well as variations that would be recognized as equivalents by those skilled in the art, unless such variations encompass known values ​​implemented by the prior art. Typically, the term "about" or "substantially" means greater or smaller than the stated value or range of values ​​by 1 / 10, e.g., ±10%, of the stated value.

[0113] It should be understood that the order of steps or order for performing certain actions is immaterial so long as the present teachings remain operable. Moreover, two or more steps or actions may be conducted simultaneously.

[0114] When a range or list of values ​​is provided, each intervening value between the upper and lower limits of that range or list of values ​​is individually contemplated and encompassed within the disclosure as if each value were specifically recited herein. Additionally, smaller ranges between and including the upper and lower limits of a given range are contemplated and encompassed within the disclosure. A list of exemplary values ​​or ranges does not exclude other values ​​or ranges between and including the upper and lower limits of a given range.

[0115] It is understood that the figures and descriptions of the present disclosure have been simplified to show elements that are relevant for a clear understanding of the present disclosure, while omitting other elements for clarity. Those skilled in the art will recognize, however, that these and other elements may be desirable. However, because such elements are well known in the art, and because they do not facilitate a better understanding of the present disclosure, descriptions of such elements are not provided herein. It should be understood that the figures are presented for illustrative purposes, and not as structural diagrams. The omitted details and modifications or alternative examples are within the knowledge of those skilled in the art.

[0116] It can be recognized that in certain aspects of the present disclosure, multiple components can be substituted for single components, and multiple components can be substituted for single components, to provide an element or structure or to perform a given function or functions. Except to the extent that such substitutions cannot be used to practice a particular embodiment of the present disclosure, such substitutions are deemed to be within the scope of the present disclosure.

[0117] The examples presented herein are intended to illustrate possible and specific implementations of the present disclosure. It can be appreciated that the examples are intended primarily for illustration of the present disclosure for those skilled in the art. There may be variations to these diagrams or the operations described herein without departing from the spirit of the present disclosure. For example, in certain cases, method steps or operations may be performed or executed in a different order, or operations may be added, deleted, or modified.

Claims

**Claim 1** A method for classifying diagnostic medical images, comprising: receiving the diagnostic medical images; analyzing the diagnostic medical images in real-time or near real-time using a trained machine learning model, wherein the trained machine learning model is trained on a set of annotated diagnostic medical images with respect to obstacles when viewing lumens in the annotated diagnostic medical images; identifying the image quality of the diagnostic medical images based on the analysis; outputting, in real-time or near real-time, an indicator of the identified image quality for display on a user interface, at least partially based on the degree of obstacles when viewing lumens; and a method comprising the steps of. **Claim 2** The method according to claim 1, further comprising classifying the diagnostic medical images as a first classification or a second classification. **Claim 3** The method according to claim 2, further comprising providing an alert or notification when the diagnostic medical images are classified as the second classification. **Claim 4** The method according to claim 1, wherein the set of annotated diagnostic medical images includes annotations including "clear", "blood", or "guide catheter". **Claim 5** The method according to claim 1, wherein the diagnostic medical images are optical coherence tomography images. **Claim 6** The method according to claim 1, further comprising classifying the diagnostic medical images as clear medical images or blood medical images. **Claim 7** The method according to claim 1, further comprising calculating a probability indicating whether the diagnostic medical images are acceptable or unacceptable. **Claim 8** The method according to claim 7, further comprising using a threshold method to convert the calculated probability into a classification of the diagnostic medical images. **Claim 9** The method according to claim 7, further comprising using graph cuts to convert the calculated probability into a classification of the diagnostic medical images. **Claim 10** The method according to claim 7, further comprising using morphological classification to convert the calculated probability into a classification of the diagnostic medical images. **Claim 11** As used herein, "acceptable" means that the diagnostic medical images are of a predefined threshold quality that allows for the evaluation of human tissue characteristics with a higher accuracy or reliability than a threshold level. Claim 12 The method according to claim 11, wherein the value of the predefined threshold quality is determined by optimizing a machine learning model. Claim 13 A system comprising a processing device coupled to a memory for storing instructions, the instructions causing the processing device to receive a diagnostic medical image, analyze the diagnostic medical image in real time or substantially in real time using a trained machine learning model, wherein the trained machine learning model is trained on a set of annotated diagnostic medical images with respect to obstacles in viewing a lumen in the annotated diagnostic medical images, identify the image quality of the diagnostic medical image based on the analysis, output, in real time or substantially in real time, an indicator of the identified image quality, at least partially based on the degree of obstacles in viewing a lumen, for display on a user interface The system is configured to perform the above operations. Claim 14 The system according to claim 13, wherein the diagnostic medical image is an optical coherence tomography (OCT) image. Claim 15 The system according to claim 14, wherein the instructions are configured to display a plurality of OCT images together with indicators related to the classification of each of the plurality of OCT images.

Images