Detection of fibrous caps in medical images

Abstract

Aspects of the present disclosure provide a method, system, and apparatus, including a computer-readable storage medium, for lipid detection by identifying a fibrous cap in a medical image of a blood vessel. The method includes accepting one or more input images of a blood vessel and processing the one or more input images with a machine learning model trained to identify the location of the fibrous cap of the blood vessel. The machine learning model is trained using a plurality of training images, each of which is annotated with the location of one or more fibrous caps. The method includes identifying and characterizing the fibrous cap of a lipid pool based on differences in radial signal intensity measured at different locations of the input images. The system can generate one or more output images having visually annotated segments representing predicted locations of the fibrous cap covering the lipid plaque.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 217,527, entitled "FIBROTIC CAP DETECTION IN MEDICAL IMAGES," filed on July 1, 2021, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] Optical coherence tomography (OCT) is an imaging technique that has widespread applications in ophthalmology, cardiology, gastroenterology, and other medical disciplines and scientific research. OCT can be used in conjunction with a variety of other imaging techniques, such as intravascular ultrasound (IVUS), near-infrared spectroscopy (NIRS), angiography, fluoroscopy, and x-ray-based imaging.

[0003] To perform imaging, an imaging probe can be attached to a catheter and manipulated through a point or region of interest, such as a patient's blood vessel. The imaging probe can return multiple image frames of the point of interest, which can be further processed or analyzed, for example, to diagnose a patient with a medical condition or as part of a scientific study. A normal artery has a layered structure that includes the tunica intima, tunica media, and tunica adventitia. As a result of some medical conditions, such as atherosclerosis, the intima or other portions of an artery may contain plaque, which may be formed from various types of fibers, proteoglycans, lipids, or calcium.

[0004] A neural network is a machine learning model that includes one or more layers that perform nonlinear operations to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an input layer and an output layer. The output of each hidden layer can be input to another hidden layer or to an output layer of the neural network. Each layer of a neural network can generate a respective output from a received input according to the values ​​of one or more model parameters of that layer. The model parameters can be weights or biases. The model parameter values ​​and biases are determined by a training algorithm to cause the neural network to generate accurate outputs. Summary of the Invention

[0005] Aspects of the present disclosure provide for automated detection and characterization of fibrotic caps adjacent regions or pools of lipids depicted in vascular images. A system with one or more processors can receive images of blood vessels annotated with segments representing the fibrotic cap of lipid pools surrounding the imaged blood vessel. The system can process these images to further annotate with segments representing background, the lumen of the vessel, the tunica media, and / or portions of the image that are indicative of calcium. From these processed images, the system can train one or more machine learning models to identify one or more segments of the fibrotic cap depicted in the input image of the blood vessel that are indicative of lipid pools in the tissue surrounding the imaged blood vessel.

[0006] Additionally or alternatively, the system can detect and characterize the fibrous cap of the lipid pool based on measuring the rate of attenuation of the imaging signal intensity through the edge of the imaged lumen into the surrounding tissue. Based on comparison of the attenuation rate with known samples, the system can predict the location of the fibrous cap of the lipid pool and estimate properties of the fibrous cap. Example properties can include its thickness and / or the boundary between the fibrous cap and the lipid pool, etc.

[0007] Aspects of the present disclosure provide a method, system, and apparatus, including a computer-readable storage medium, for lipid detection by identifying a fibrous cap in a medical image of a blood vessel. The method includes accepting one or more input images of a blood vessel and processing the one or more input images using a machine learning model trained to identify the location of the fibrous cap of the blood vessel. The machine learning model is trained using a plurality of training images, each training image annotated with the location of one or more fibrous caps. The method includes identifying and characterizing the fibrous cap of a lipid pool based on differences in radial signal intensity measured at different locations of the input images. The system can generate one or more output images having visually annotated segments representing the predicted location of the fibrous cap.

[0008] A fibrous cap may overlie the lipid plaque. Aspects of the present disclosure provide for identifying the fibrous cap to identify the underlying lipid plaque.

[0009] One aspect of the disclosure includes a method for identifying a fibrous cap of a blood vessel, comprising: one or more processors receiving one or more input images of a blood vessel; the one or more processors processing the one or more input images using a machine learning model trained to identify the location of a fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images annotated with one or more locations of one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; and the one or more processors receiving as output from the machine learning model one or more output images having visually annotated segments representing predicted locations of the fibrous cap.

[0010] One aspect of the present disclosure includes a system comprising one or more processors configured to accept one or more input images of a blood vessel; process the one or more input images using a machine learning model trained to identify a location of a fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images annotated with the location of one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; and accept as output from the machine learning model one or more output images having segments visually annotated to represent or illustrate a predicted location of the fibrous cap.

[0011] One aspect of the disclosure includes one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including accepting one or more input images of a blood vessel; processing the one or more input images using a machine learning model trained to identify a location of a fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images each annotated with a location of one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; and accepting as output from the machine learning model one or more output images having visually annotated segments representing a predicted location of the fibrous cap.

[0012] One aspect of the present disclosure provides a method of identifying a fibrous cap of a blood vessel, the method comprising: one or more processors receiving one or more input images of a blood vessel; processing the one or more input images with a machine learning model trained to identify a location of a fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images annotated with one or more locations of the one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; receiving one or more output images as output from the machine learning model, the output images having visually annotated segments representing predicted locations of the fibrous cap; and generating, using the one or more processors, an updated boundary of the fibrous cap relative to the adjacent lipid pools from the one or more output images based on signal intensities of a plurality of points in the one or more input images. The plurality of points can be along one or more arc lines surrounding the fibrous cap.

[0013] Generating the updated boundary can include measuring signal intensities at a plurality of points along one or more arc lines that encircle the fibrous cap, and determining a boundary between the fibrous cap and an adjacent lipid pool based on a comparison of the measured decay rates of the signal intensities at the plurality of points to a predetermined decay rate of the signal intensity through the lipid fibrous cap. The signal intensities can be stored as metadata comprising a profile of the signal intensities.

[0014] The method can further include identifying lipid plaques based on the location of one or more fibrous caps. Generating an updated boundary can include updating the boundary based on a radial intensity profile that includes the measured signal intensity and that is associated with the one or more input images.

[0015] Other aspects of the above include a system comprising one or more processors configured to perform a method of fibrous capsule detection. Other aspects of the above include one or more computer readable storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of fibrous capsule detection.

[0016] One aspect of the present disclosure provides a method of identifying a fibrous cap of a blood vessel, the method including: one or more processors receiving one or more input images of the blood vessel; generating, using the one or more processors, one or more first output images including a boundary of the fibrous cap relative to adjacent lipid pools, the generating based on signal intensities of a plurality of points in the one or more input images; processing the one or more input images using a machine learning model trained to identify a location of the fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images annotated with one or more locations of the one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; receiving, as an output from the machine learning model, one or more second output images having visually annotated segments representing predicted locations of the fibrous cap; and updating the boundary of the fibrous cap in the one or more first output images using the one or more second output images. The plurality of points can be along one or more arc lines surrounding the fibrous cap.

[0017] One aspect of the present disclosure provides a method for identifying a fibrous cap of a blood vessel, the method including: one or more processors accepting one or more input images of the blood vessel; generating, using the one or more processors, one or more first output images including a boundary of the fibrous cap relative to adjacent lipid pools, where the generating is based on signal intensities of a plurality of points in the one or more input images; processing, by the one or more processors, the one or more first output images using a machine learning model trained to identify a location of the fibrous cap of the blood vessel, where the machine learning model is trained using a plurality of training images annotated with one or more locations of the one or more fibrous caps, each fibrous cap being adjacent to a respective lipid pool; and accepting, by the one or more processors as output from the machine learning model, one or more updated output images having visually annotated segments representing predicted locations of the fibrous cap.

[0018] The method can further include identifying one or more lipid plaques based on the identified one or more fibrous cap locations. The method can further include storing the measured signal intensities as a profile in metadata corresponding to the one or more output images. The one or more updated output images can be updated using the profile of the measured signal intensities, where updating includes modifying the one or more fibrous cap boundaries generated using the machine learning model.

[0019] These and other aspects of the disclosure can include one or more of the following features. In some implementations, an aspect of the disclosure can include all of the following features in combination.

[0020] The one or more input images may be further annotated with segments corresponding to the location of at least one of calcium, the lumen of a blood vessel, or the tunica media.

[0021] The one or more input images include annotated segments representing one or more regions of the tunica media, the one or more input images being images received from the imaging probe during pullback of the imaging probe through the blood vessel, and the method or operations further include the one or more processors estimating an average signal-to-noise ratio (SNR) of the one or more input images based on a comparison of the predicted annotations of the regions of the tunica media in the one or more input images to one or more ground truth annotations of the regions of the tunica media in the one or more input images, and in response, the one or more processors flagging one or more output images corresponding to the one or more input images in response to a determination that the average SNR is below a predetermined threshold.

[0022] The imaging probe may be an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, a micro-OCT (μOCT) imaging probe, etc. In some examples, the imaging probe may be configured to generate images according to a combination of the above and other imaging techniques.

[0023] The imaging probe may be an optical coherence tomography (OCT) imaging probe.

[0024] Receiving the one or more output images can include receiving, for each input image, a respective visually annotated segment of the input image that represents a predicted location of the fibrous cap.

[0025] The method or operation may further include the one or more processors accepting, for each of the one or more output images, one or more measurements of thickness of each fibrous cap whose location is predicted in the output image.

[0026] The method or operation may further include generating, using one or more processors, an updated boundary of the fibrous cap relative to an adjacent lipid pool from the one or more output images, the generating including the one or more processors measuring signal intensities of a plurality of points in the one or more input images, and the one or more processors determining a boundary between the fibrous cap and an adjacent lipid pool or lipid plaque based on a comparison of the measured attenuation rates of the signal intensities of the plurality of points to a predetermined attenuation rate of the signal intensity through the lipid fibrous cap. The plurality of points may be along one or more arc lines surrounding the fibrous cap.

[0027] Determining the boundary between the fibrous cap and the adjacent lipid pool can include identifying a point among the plurality of points having a measured signal intensity proportional to a peak signal intensity of the plurality of points within a predetermined threshold.

[0028] The system may further include an imaging probe communicatively connected to the one or more processors, and receiving one or more input images of the blood vessel may include receiving image data corresponding to the one or more input images from the imaging probe while the imaging probe is inside the blood vessel.

[0029] The system may further comprise one or more display devices configured to display the image data, the one or more processors further configured to display the one or more output images on the one or more display devices.

[0030] One aspect of the present disclosure includes a method of training a machine learning model specific to a fibrous cap of a blood vessel, the method including: one or more processors receiving a plurality of training images, each training image annotated with one or more locations of one or more fibrous caps in the training image, each fibrous cap adjacent to a respective lipid pool; one or more processors processing the plurality of training images to annotate each training image with a location of at least one of calcium, a lumen of the blood vessel, or a tunica media; and one or more processors training a machine learning model using the processed plurality of training images. Processing the plurality of training images further includes processing the plurality of training images through one or more machine learning models trained to identify segments of an input image corresponding to the location of at least one of calcium, a lumen of the blood vessel, and a tunica media.

[0031] One aspect of the present disclosure provides a system that includes one or more processors configured to accept a plurality of training images, each training image annotated with a location of one or more fibrous caps in the training image, each fibrous cap adjacent to a respective lipid pool; process the plurality of training images to annotate each training image with a respective one or more segments corresponding to a location of at least one of calcium, a lumen of a blood vessel, or a tunica media; and train a machine learning model using the processed plurality of training images.

[0032] One aspect of the present disclosure provides one or more temporary or non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including accepting a plurality of training images, each training image annotated with a location of one or more fibrous caps in the training image, each fibrous cap adjacent to a respective lipid pool; processing the plurality of training images to annotate each training image with a respective one or more segments corresponding to a location of at least one of calcium, a lumen of a blood vessel, or a tunica media; and training a machine learning model using the processed plurality of training images.

[0033] The machine learning model may be a first machine learning model, and the method may further include accepting a second machine learning model including a plurality of model parameter values ​​and trained to identify segments of the input image corresponding to a location of at least one of calcium, the lumen of the blood vessel, and the tunica media in an image of the blood vessel, and training the first machine learning model includes initializing the training with at least a portion of the model parameter values ​​from the second machine learning model.

[0034] The second machine learning model may be a convolutional neural network including a plurality of layers, including an output layer, each layer including one or more respective model parameter values, and training the first machine learning model may further include replacing the output layer of the second machine learning model with a new layer configured to (i) accept an input to the output layer of the second machine learning model and (ii) generate as an output a segmentation map of the input image, the segmentation map including a plurality of channels including a channel that identifies a segment of the input image that represents a predicted location of the one or more fibrous capsules in the input image, and training the second machine learning model having the replaced output layer using the processed plurality of training images.

[0035] The multiple channels of the segmentation map may further include one or more channels that identify segments of the input image that represent predicted locations of at least one of calcium, the lumen of the blood vessel, and the tunica media.

[0036] Training the first machine learning model with the replaced output layer may further include updating only model parameter values ​​of the new layer of the first machine learning model.

[0037] Processing the plurality of training images may further include processing the plurality of training images with a second machine learning model.

[0038] Training the machine learning model can include training the machine learning model to output visually annotated segments of an image of a blood vessel that represent a predicted location of the fibrous cap.

[0039] Training the machine learning model can include training the machine learning model to output, from an image of a blood vessel, one or more measurements of at least one of a thickness and a length of each fibrous cap identified in the image.

[0040] The imaging probe can be an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, a micro-OCT (μOCT) imaging probe, etc. In some examples, the imaging probe can be configured for multi-modal imaging, e.g., imaging using a combination of OCT, NIRS, OCT-NIRS, μOCT, etc.

[0041] The multiple training images are images captured using optical coherence tomography (OCT).

[0042] One aspect of the disclosure provides a method that includes one or more processors receiving an input image of a blood vessel, the one or more processors calculating respective signal intensities of imaging signals at each of a plurality of points in the input image for an arc line referenced to a blood vessel reference point in the input image, and the one or more processors identifying a fibrous cap adjacent to a lipid pool depicted in the input image from the respective signal intensities of the plurality of points referenced to the reference point, where the plurality of points can be along one or more arc lines that surround the fibrous cap.

[0043] One aspect of the disclosure provides a system including one or more processors configured to: receive an input image of a blood vessel; calculate a signal intensity of each of imaging signals at each of a plurality of points in the input image for an arc line referenced to a blood vessel reference point in the input image; and identify a fibrous cap adjacent to a lipid pool depicted in the input image from the signal intensities of the plurality of points referenced to the reference point, where the plurality of points can be along one or more arc lines surrounding the fibrous cap.

[0044] One aspect of the disclosure provides one or more transitory or non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: the one or more processors accepting an input image of a blood vessel, the one or more processors calculating respective signal intensities of imaging signals for each of a plurality of points in the input image for an arc line referenced to a blood vessel fiducial point in the input image, and the one or more processors identifying a fibrous cap adjacent to a lipid pool depicted in the input image from the respective signal intensities of the plurality of points referenced to the fiducial point, where the plurality of points can be along one or more arc lines surrounding the fibrous cap.

[0045] These and other aspects of the disclosure can include one or more of the following features.

[0046] The reference point may be the center of the lumen of the blood vessel.

[0047] The input image can be annotated with one or more arc lines that correspond to the fibrous cap.

[0048] The input image can be annotated with a segment corresponding to the fibrous cap, and the identifying can include identifying an updated boundary between the fibrous cap and the lipid pool. The input image can be annotated with a visual representation of the fibrous cap shown in the input image, and the method or operations can further include identifying one or more arc lines based on a coverage angle characterizing the fibrous cap relative to a reference point.

[0049] The multiple points can form a sequence of points of increasing distance relative to the center of the lumen, where a first point in the sequence is closest to the reference point and a last point in the sequence is furthest from the reference point, and identifying the updated boundary between the fibrous cap and the lipid pool can include calculating a signal intensity decay rate for each of two or more points in the sequence of points along the arc line, determining that the calculated attenuation rate is within a predetermined threshold of signal intensity decay rate through the fibrous cap and the lipid pool, and in response to determining that the calculated attenuation rate is within the threshold, identifying a segment of the input image between the two or more points as the fibrous cap.

[0050] Calculating a rate of decay of the signal strength may further include calculating a rate of decay of the signal strength for each signal strength of an arc line that is farther from the center of the lumen than a second point along the one or more arc lines.

[0051] Identifying a segment of the input image as a fibrous cap can further include annotating a boundary between the fibrous cap and a lipid pool adjacent to the fibrous cap.

[0052] Determining that the calculated attenuation rate is within a predetermined attenuation rate threshold includes measuring a fitting error between the calculated attenuation rate and a curve that at least partially includes the predetermined attenuation rate over two or more points along one or more arc lines and points at the same distance relative to the center of the lumen.

[0053] The method or operation may further include, in response to determining that the calculated attenuation rate is not within a predetermined attenuation rate threshold of the fibrous cap of the lipid pool, determining that the attenuation rate is within a respective threshold of one or more other predetermined attenuation rates, each of the other predetermined attenuation rates corresponding to a respective measured attenuation rate of the imaging signal through a respective non-lipid region of the plaque or media.

[0054] The method or operation may further include identifying a first point of the plurality of points as corresponding to a peak signal intensity referenced to respective signal intensities of the plurality of points, identifying a second point of the plurality of points as corresponding to a respective signal intensity equal to a threshold intensity referenced to the peak signal intensity, and measuring a thickness of the fibrous cap as a distance between an edge of a lumen of the imaged blood vessel and the second arc line.

[0055] The threshold intensity relative to the peak signal intensity may be 80 percent.

[0056] The method or operation may further include comparing one or both of the identified fibrous cap thickness measurements and attenuation rates of a plurality of points along one or more arc lines of the input image frame to one or more predetermined thresholds, and flagging the input image if one or both of the thickness measurements and attenuation rates are within the one or more predetermined thresholds.

[0057] The method or operations may further include displaying the input image annotated with the location of the fibrous cap.

[0058] The input image may be captured using an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, a micro-OCT (μOCT) imaging probe, or the like.

[0059] The multiple training images are images captured using optical coherence tomography (OCT), intravascular ultrasound (IVUS), near-infrared spectroscopy (NIRS), OCT-NIRS, micro-OCT (μOCT), or the like.

[0060] The input image may be captured using optical coherence tomography (OCT).

[0061] One aspect of the present disclosure provides a system including one or more processors, the one or more processors being configured to: receive an input image of a blood vessel; process the input image using a machine learning model trained to identify a location of a fibrous cap of the blood vessel, the machine learning model being trained using a plurality of training images annotated with the locations of one or more fibrous caps, each fibrous cap adjacent to a respective lipid pool; receive as output from the machine learning model an output image including visually annotated segments representing predicted locations of the fibrous cap including a boundary between the fibrous cap and the lipid pool; identify one or more arc lines corresponding to the segments in the output image, the one or more arc lines originating from a reference point in the output image; and identify an updated boundary between the fibrous cap and the lipid pool based on a difference in signal intensity in the output image measured across one or more points of the one or more arc lines.

[0062] Other aspects of the disclosure include methods, apparatus, and non-transitory computer-readable storage media storing one or more computer program instructions that, when executed, cause one or more processors to perform operations of the methods. [Brief description of the drawings]

[0063] [Figure 1] FIG. 1 is a block diagram of an example image segmentation system according to aspects of the present disclosure. [Diagram 2] FIG. 1 shows an example of a labeled fibrous cap shown in an example image frame. [Diagram 3] 1 is a flowchart of an example process for training a fibrosis detection model according to an aspect of the present disclosure. [Figure 4A] 1A-1C illustrate an example input image and a corresponding output image, expressed in polar coordinates, generated by an image segmentation system in accordance with aspects of the present disclosure. [Figure 4B]FIG. 4B illustrates the example input image of FIG. 4A and the corresponding output image expressed in Cartesian coordinates. [Diagram 5] 1 is a flowchart of an example process for training a fibrous cap detection model using a pre-trained model, according to an aspect of the present disclosure. [Figure 6A] 1 is a flow chart of an example process for detecting a fibrous cap of lipid pools in tissue surrounding a blood vessel, according to aspects of the present disclosure. [Figure 6B] 13 is a flowchart of an example process for flagging output images of a fibrous cap detection model having a low signal-to-noise ratio, according to aspects of the present disclosure. [Figure 7A] FIG. 13 illustrates multiple arc lines from the center of the lumen shown in the image frame. [Figure 7B] 1 is a flowchart of an example process for identifying a fibrous cap based on radial signal intensity of arc lines measured from an input image of a blood vessel, according to aspects of the present disclosure. [Figure 8] 1 is a flow chart of an example process for identifying a fibrous cap based on the rate of attenuation of radial signal intensity of an arc line measured from an input image of a blood vessel. [Figure 9] 1 is a graph of peak signal intensity and attenuation rate through different tissues and plaques. [Figure 10] 1 is a flowchart of an example process for measuring fibrous cap thickness using peak radial signal intensity in a series of measurement arc lines measured from an input image of a blood vessel, according to aspects of the present disclosure. [Figure 11] FIG. 1 is a block diagram of an example computing environment for implementing an image segmentation system according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] <Summary> Aspects of the present disclosure provide for automated detection of fibrous caps of lipid pools in images of blood vessels. Images of blood vessels can be captured by an imaging probe, for example, an imaging probe in an imaging device such as a catheter that is maneuvered through a blood vessel or other region of interest within a patient's body. Segments of the image can then be analyzed and labeled as corresponding to various tissues or plaques. Lipid accumulations within imaged blood vessels are of particular interest because the presence and characteristics of lipids in and around blood vessels can be used, for example, as part of screening patients for various cardiovascular diseases.

[0065] One problem with OCT captured images is that lipid pools or lipid plaques, unlike other tissues or plaques such as calcium or the tunica media, often do not image as clearly as other segments of the imaged vessel. One reason for this is that the imaging signal from the imaging probe is attenuated as it propagates through the fibrous cap and into the adjacent lipid pools. On the other hand, other plaques such as calcium may be relatively easy to identify, at least because those types of plaques do not have lipid-like physical properties that cause the imaging signal to attenuate rapidly.

[0066] As a result, lipids are difficult to distinguish and often require manual review by trained experts to identify. Even in such cases, experts are often unable to directly characterize lipids, for example, by characterizing the depth or width of the pool in tissue surrounding a blood vessel, because the pool often does not appear clearly in the OCT-captured image. Another challenge is properly identifying the boundary between the fibrous cap and the lipid pool itself. Identifying the boundary is difficult, at least because measuring the thickness of the fibrous cap requires knowing where the cap ends and where the lipid pool is. As a result, manually labeled images of lipids and fibrous caps are time-consuming to create and often inaccurate. Furthermore, annotations by experts often may not be consistent from image to image, and may not be consistent across different experts annotating the same image. These problems in accurately identifying lipids can also occur for images taken according to other modalities, such as when images are captured from an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, a micro-OCT (μOCT) imaging probe, or any other of a variety of probes or devices implementing any of a variety of different imaging technologies.

[0067] Aspects of the present disclosure provide techniques for identifying lipid pools surrounding an imaged blood vessel by training a model to identify the fibrous cap. The fibrous cap of a lipid pool may refer to tissue that encapsulates the lipid pool in the tissue surrounding the blood vessel. A system as described herein can predict the fibrous cap of a lipid pool in an input image, as well as predict the presence of other regions of interest such as calcium or the tunica media.

[0068] A system as provided herein can augment training data labeled with the fibrous cap covering the lipid pool with labels of other regions of interest. Because automatic and manual annotation of segments of interest such as calcium is generally more accurate and faster than identifying lipids, the system can take advantage of existing labels of other regions of interest in the image to identify the fibrous cap more accurately than without those additional labels. The system can be configured to predict one or more channels of visible regions of interest from the input image, which can be used to annotate the input image of the blood vessel. These additional channels can correspond, by way of example, to the lumen, calcium, media, lipids, and / or background of the input image of the blood vessel.

[0069] Additionally, systems as provided herein can take advantage of differences in physical properties of lipid and non-lipid regions of interest, such as by signal decay rates of the imaging probe through different regions of interest. By identifying and comparing signal decay rates of regions relative to distance from the lumen edge, the system can predict whether the region corresponds to lipid or non-lipid. By comparing the signal intensity decay rate of the imaging signal to previously accepted samples of signal intensity decay rates through different tissues and plaques, the system can predict physical properties of the fibrous cap, such as fibrous cap thickness.

[0070] Additionally, the system as provided herein can more accurately identify the boundary between the fibrous capsule and the lipid pool by measuring and comparing the signal intensity of the OCT image along the arc line points of the arc that includes the fibrous capsule or the lipid pool. In some examples, the system can accept as input one or more manually annotated images of the fibrous capsule or images annotated according to the aspects of the present disclosure provided herein. The system can annotate the image with the boundary between the fibrous capsule and the adjacent lipid pool. As part of accepting the input image annotated with the fibrous capsule, the system can calculate one or more arc lines based on the coverage angle of the annotated capsule. In some embodiments, the system can process the images taken using NIRS, IVUS, OCT-NIRS, and / or μOCT, etc., to measure and compare the signal intensity of the images and identify the fibrous capsule and / or the boundary between the capsule and the lipid pool or lipid plaque.

[0071] In some examples, the system may only accept images annotated with arc lines corresponding to angles that include the fibrous capsule and identify the boundary between the fibrous capsule enclosed by those arc lines and the adjacent lipid pool, in which examples, the system may be configured to perform a signal intensity analysis of points along the arc lines to predict the location of the fibrous capsule, including the boundary between the capsule and the lipid pool.

[0072] The physical properties of the fibrous cap can be used to improve the diagnosis of coronary conditions such as thin cap fibroatheroma (TCFA) in the imaged vessel. By improving the accuracy of the measurement of the fibrous cap thickness over other techniques, the system can provide data that can be used with greater confidence in diagnosing a patient. Accurate measurement can be especially important when differences in fibrous cap thickness can be the difference in diagnosing a patient between a relatively benign thick-cap fibroatheroma and a more dangerous condition such as TCFA. In addition, the systems described herein can identify boundaries based on adjustable thresholds that can be adjusted according to indications or standards for diagnosing or assessing increased risk of plaque disruption such as fibrous cap or TCFA for cardiovascular disease, or based on observation of previously analyzed samples of the fibrous cap in, for example, OCT-captured images, IVUS-captured images, NIRS-captured images, OCT-NIRS-captured images, μOCT-captured images, and / or images captured using any one of a variety of different imaging techniques.

[0073] <System example> 1 is a block diagram of an image segmentation system 100 according to one aspect of the disclosure. Image segmentation system 100 can include one or more processors and memory devices in one or more locations and across one or more devices, such as a server computing device or computing devices connected to imaging equipment and / or other tools in a catheter lab. Image segmentation system 100 can include a training engine 105, an annotation engine 110, and a fibrous capsule detection engine 115.

[0074] In general, the image segmentation system 100 accepts an input image, such as an input image 120 of a blood vessel, such as blood vessel 102, captured by an imaging device 107. The system 100 can generate as output one or more output images 125, including a fibrous cap annotated image 125A and, optionally, one or more annotated images 126B-126N. The fibrous cap annotated image 125A can visually annotate segments of the fibrous cap in the input image 120 predicted by the system 100. For example, the fibrous cap annotated image 125A can include an overlay of highlighted or otherwise visually distinct portions of the predicted fibrous cap in the input image 120. From the fibrous cap annotated image 125A, the system can measure or estimate one or more physical properties of the identified cap, such as the thickness of the cap.

[0075] As described in more detail with reference to Figures 7A-10, the image segmentation system 100 can be configured to refine the boundary by analyzing the attenuation of the imaging signal measured from the input image 120 and comparing the measured attenuation rate of the imaging signal through the capsule with previously obtained attenuation rates of the imaging signal through various plaques, including lipids.

[0076] System 100 may output other annotated images 125B-125N, in some examples, that represent predicted locations of other types of regions of interest, such as calcium, tunica media, background, and the size and shape of the lumen of blood vessel 102. Annotated images 125B-125N may each be associated with a particular type of region of interest, such as calcium or tunica media. In some examples, system 100 generates a segmentation map or other data structure that maps pixels in input image 120 to one or more channels. Each channel corresponds to one region of interest.

[0077] The segmentation map can include multiple elements. For example, the multiple elements are in an array, with each element corresponding to a pixel in the input image 120. Each element in the array can include a different value, e.g., an integer value, with each value corresponding to a different channel of the predicted region of interest. For example, the output segmentation map can include elements having a value of "1" for corresponding pixels predicted to be fibrous cap. The output segmentation map can include other values ​​for other regions of interest, such as a value of "2" for corresponding pixels predicted to be calcium. The system 100 can be configured to output a segmentation map with some or all of the channels represented in the map.

[0078] In some examples, instead of generating an output image 125, the system can generate a segmentation map of one or more channels, and the user computing device 135 can be configured to apply the segmentation map to the input image 120. As an example, the user computing device 135 can be configured to process the input image 120 with one or more channels of the segmentation map to generate corresponding images of one or more channels, e.g., a fibrous cap annotated output image, a calcium annotated output image, etc. Multiple channels can be combined to generate an output image annotated for regions of calcium and lipids, for example.

[0079] Predictions of various regions of interest, such as the fibrous cap of a lipid pool, can be annotated in many ways. For example, the segmentation map can be one or more masks of one or more channels that can be applied as an overlay to the input image 120. As another example, the system 100 can copy and modify pixels of the input image 120 that correspond to the locations of the predicted regions of interest. For example, the system 100 can generate a fibrous cap annotated image 125A having pixels of the input image 120 where the fibrous cap is predicted to be located that are shaded or modified to appear visually different from other parts of the input image 120. In other examples, the system can modify pixels corresponding to the predicted fibrous cap in other ways, such as by the contour of the predicted cap, a visually different pattern, shading or thatching.

[0080] The user computing device 135 can be configured to accept input images 120 from an imaging device 107 having an imaging probe 104. The imaging probe 104 can be, by way of example, an OCT probe and / or an IVUS catheter. Although the examples provided herein relate to an OCT probe, the use of an OCT probe or a particular OCT imaging technique is not intended to be limiting. For example, an IVUS catheter can be used in conjunction with or in place of an OCT probe. The probe 104 can be introduced into the blood vessel 102 using a guidewire, not shown. The probe 104 can be introduced and retracted along the length of the lumen of the blood vessel 102 while data is collected, for example, as a sequence of image frames. According to some examples, the probe 104 can be held stationary during the pullback so that multiple scans of an OCT data set and / or an IVUS data set can be collected. The data set, which can include image frames or other image data, can be used to identify lipid pools and fibrous caps of other regions of interest. According to some examples, the probe 104 can be configured for micro-optical coherence tomography (μOCT). Other imaging techniques such as near infrared spectroscopy and near infrared imaging spectroscopy (NIRS) can also be used.

[0081] The probe 104 can be connected to a user computing device 135 through an optical fiber 106. The user computing device 135 can include a light source such as a laser, an interferometer with a sample arm and a reference arm, various optical paths, a clock generator, a photodiode, and other OCT and / or IVUS components. In some examples, the user computing device 135 is connected to one or more other devices and / or equipment (not shown) that perform medical imaging using the imaging device 107. As an example, the user computing device 135 and the imaging device 107 can be part of a catheterization lab. In other examples, the system 100, the user computing device 135, the display 109, and the imaging device 107 are part of a larger system for medical imaging implemented, for example, as part of a catheterization lab.

[0082] The system 100 can be configured to accept and process the input images 120 in real time, for example, while the imaging device 107 is being operated imaging the blood vessel 102. In another example, the system 100 accepts one or more image frames after a procedure to image the blood vessel 102 has occurred, for example, by accepting input data stored on the user computing device 135 or another device. In another example, the system 100 accepts the input images from an entirely different source, for example, from one or more devices over a network. In this latter example, the system 100 can be configured, for example, on one or more server computing devices that process the received input images according to the techniques described herein.

[0083] As shown, the display 109 is separate from the user computing device 135, although according to some examples the display 109 can be part of the computing device 135. The display 109 can output image data, such as the output image 125. The display 109 can display the output through a display viewport, such as a circular display viewport, in some examples.

[0084] The display 109 may show one or more image frames, for example, as two-dimensional cross-sections of the blood vessel 102 and surrounding tissue. The display 109 may also include one or more other views showing different perspectives of the imaged blood vessel or another region of interest in the patient's body. As an example, the display 109 may include a longitudinal view of the length of the blood vessel 102 from start to finish. In some examples, the display 109 may highlight certain portions of the blood vessel 102 along the longitudinal view and at least partially occlude other portions that are not currently selected for viewing. In some examples, the display 109 accepts input to scrub portions of the lumen 201 to be shown in the longitudinal view.

[0085] The output may be displayed, for example, in real time during the procedure as the imaging probe 104 is maneuvered through the blood vessel 102. For example, other data that may be output in combination with the output image 125 includes cross-sectional scan data, longitudinal scans, diameter graphs, lumen boundaries, plaque size, plaque perimeter, visual indicia of plaque location, visual indicia of risk presented for stent expansion, flow velocity, etc. The display 109 may identify features using text, arrows, color coding, highlighting, contours, or other suitable human or machine readable indicia.

[0086] According to some examples, the display 109 may be a graphic user interface ("GUI"). One or more steps of the processes described herein may be performed automatically, i.e., without user input, such as navigating an image, entering information, selecting an input, and / or interacting with an input. The display 109, alone or in combination with the user computing device 135, may enable switching between one or more viewing modes in response to user input. For example, a user may be able to switch between different side branches on the display 109, such as by selecting a particular side branch and / or selecting a view associated with a particular side branch.

[0087] In some examples, the display 109, alone or in combination with the user computing device 135, can include a menu. The menu can allow the user to show or hide various features. There can be more than one menu. For example, there can be a menu to select which vascular features to display, such as toggling on or off one or more masks to overlay on top of the input image 120. Additionally or alternatively, there can be a menu to select a virtual camera angle for the display. In some examples, the display 109 can be configured to accept input. For example, the display 109 can include a touch screen configured to accept touch input to interact with the menus or other interactable elements displayed on the display 109.

[0088] The output image frames 125 may be used, for example, for medical analysis, diagnosis, and / or general research as part of downstream processes. For example, the output image frames 125 may be displayed for review and analysis by a user, such as a medical professional, or may be used as input to an automated process of medical diagnosis and evaluation, such as an expert system or other downstream process implemented on one or more computing devices, such as one or more computing devices implementing the system 100. From the annotated segments of the output image frames 125, the system 100 may estimate the thickness of one or more identified fibrous caps adjacent to respective lipid pools in the tissue surrounding the blood vessel 102, for example, according to the techniques described herein with reference to Figures 3 and 7A-10.

[0089] For example, based on the estimated physical properties of the predicted fibrous cap, the output image frames 125 may be used, at least in part, to diagnose TCFA and / or other coronary conditions. TCFA can be difficult to diagnose by visual inspection alone due at least to the aforementioned physical properties of lipid pools that attenuate OCT or other imaging signals, such as in images taken using NIRS, OCT-NIRS, IVUS, μOCT, etc., which can make it difficult to ascertain the boundary between the fibrous cap and its corresponding lipid pool.

[0090] Turning to engines 105, 110, and 115, fibrous cap detection engine 115 is generally configured to predict the presence of a fibrous cap in input image 120, for example, in the form of highlighting or other visible indicia. Fibrous cap detection engine 115 may be configured to detect the fibrous cap of a lipid pool, for example, through one or more machine learning models trained as described herein with reference to FIGS. 3-6. In some examples, fibrous cap detection engine 115 may be configured to identify and characterize the fibrous cap by analyzing radial signal intensity of the input image of the blood vessel against known samples of fibrous cap of the lipid pool, as described herein with reference to FIGS. 7-10. In some implementations, fibrous cap detection engine 115 may be configured to identify and characterize the fibrous cap using a combination of techniques described herein. Characterization of the fibrous cap of a plaque or other region in the input image may refer to the generation or measurement of quantitative or qualitative features of the plaque cap or region. Exemplary features that may form part of the characterization may include the length, width, or overall geometry of the plaque coating or region.

[0091] The training engine 105 is configured to accept received training image data 145 and train one or more machine learning models implemented as part of the fibrous cap detection engine 115. The training image data 145 can be image frames of a blood vessel annotated with a fibrous cap. As described herein with reference to FIGS. 3-6B, the training engine 105 can be configured to train one or more machine learning models implemented as part of the fibrous cap detection engine 115. The training engine 105 can use training image data 145 that has been annotated by the annotation engine 110 with the location of the fibrous cap in the corresponding image frames, and further annotated with the locations of other regions of interest.

[0092] FIG 2 shows an example of a labeled fibrous cap 201 in an exemplary image frame 200. In FIG 2, the labeled fibrous cap 201 is outlined by a series of dots, although the fibrous cap 201 can be represented in other ways, such as by shading an area of ​​the cap relative to other portions of the image frame 200. The labels can be generated, for example, manually, following instructions to evaluate the image frame of the fibrous cap. The image frame 200 can also be additionally labeled, for example, with the location of the center of the lumen 204, the center 202 of the imaging device at the time the image frame was captured by the device, and a region 203 indicative of a lipid pool adjacent to the fibrous cap 201.

[0093] In some examples, the training engine 105 accepts training image frames, such as image frame 200 of FIG. 2, that have not been annotated with one or more other regions of interest, such as the contours or areas covered by calcium, tunica media, or the lumen of the imaged blood vessel. The annotation engine 110 may be configured to accept training image data 145 and further annotate the image frames of data with annotations corresponding to regions of calcium, tunica media, lumen, background, etc. To do this, the annotation engine 110 may be implemented using any of a variety of different techniques, for example, using one or more appropriately trained machine learning models to identify and classify these regions of interest.

[0094] For example, the annotation engine 110 can implement one or more machine learning models trained to identify and classify regions of calcium shown in the image frames. In some examples, the annotation engine 110 generates modified image frames having a visual indication of a predicted region of interest, such as a contour surrounding the region of calcium or other identified non-lipid plaque. In other examples, the annotation engine 110 can generate data that the training engine 105 is configured to process in addition to the corresponding training image frames. The generated data can be, for example, data defining a mask over the pixels of the training image frames, where each pixel of the training image frames corresponds to a pixel in the mask and indicates whether the pixel partially represents a region of interest shown in the training image.

[0095] The generated data, in some examples, can include coordinate data corresponding to pixels of the processed image frames that at least partially depict the region of interest. For example, the generated data can include spatial or Cartesian coordinates of each pixel of a mask that corresponds to a region of detected non-lipid plaque. The annotation engine can also be configured to convert coordinate data from one system (such as Cartesian coordinates) to another coordinate system (such as polar coordinates referenced to a reference point such as the center of the lumen) as needed.

[0096] Similarly, the training image data 145 may also include data defining the locations of pixels annotated as corresponding to regions of the fibrous cap of the lipid pool. The system 100 may convert this data depending on the input requirements of the training engine 105 and / or the fibrous cap detection engine 115. One reason for using different coordinate systems is that the training image data 145 is manually labeled with a number of individual points that collectively define the perimeter of the annotated fibrous cap, but the fibrous cap detection engine 115 is configured to process the same image data with different point locations expressed in polar coordinates. The system 100 may be configured to generate an output image 125 having pixel locations arranged according to a polar coordinate system, and, if necessary, convert the output image 125 back to Cartesian coordinates for display on the display 109.

[0097] Although the training engine 105 is shown as part of the image segmentation system 100, in some examples the training engine 105 is implemented on one or more devices that are distinct from one or more devices that implement other portions of the system 100. Additionally, the system 100 may train or fine-tune one or more of the machine learning models described, or may not train or fine-tune, but instead accept a model that has been pre-trained in accordance with aspects of the present disclosure.

[0098] <Example of method> 3 is a flow chart of an example process 300 for training a fibrous cap detection model according to an embodiment of the present disclosure. Process 300 can be performed by a system comprising one or more processors located at one or more locations and suitably configured according to an embodiment of the present disclosure.

[0099] The fibrous capsule detection model can include one or more machine learning models, such as a neural network, that can be trained using labeled image training data, such as training image data 140 as described with reference to Figure 1. A fibrous capsule detection engine, such as fibrous capsule detection engine 115 of system 100, can implement one or more mathematical models, such as one or more machine learning models described above with reference to Figures 1 and 2, collectively referred to as a fibrous capsule detection model, and can be configured to identify and characterize the fibrous capsule of a lipid pool of an image frame, as described herein.

[0100] The system accepts a plurality of training images, per block 310, each of which has been annotated with the location of one or more fibrous capsules in the training image. As described herein with reference to Figures 1 and 2, the system may accept training image data labeled with the location of the fibrous capsule detected in each image. As part of accepting the plurality of training images, the system may separate the data into a plurality of sets, such as training, testing, and validation image frames.

[0101] The system processes a plurality of training images to annotate each image with one or more non-lipid segments of the input image, per block 320. For example, as described with reference to FIG. 1, the system can include an annotation engine that annotates training image frames with annotations that identify predicted regions of non-lipid plaque, such as calcium, tunica media, lumen of blood vessels, and background.

[0102] The system trains a fibrous cap detection model using the processed training images, per block 330. In some examples, the model generates as output a segmentation map representing one or more predicted regions of interest that include portions of the input images predicted to be fibrous cap.

[0103] The fibrous capsule detection model can be trained according to any technique of supervised learning, generally according to techniques for training machine learning models, using a dataset in which at least some of the training examples are labeled. For example, the fibrous capsule detection model can be, by way of example, one or more neural networks having model parameter values ​​that are updated as part of the training process using backpropagation with gradient descent, either for individual image frames or for batches of image frames.

[0104] In some examples, the fibrous capsule detection model can be one or more convolutional neural networks that accept as input pixels corresponding to an input image and generate as output segmentation maps corresponding to one or more channels of a region of interest in the input image. As an example, the fibrous capsule detection model can be a neural network that includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. Each layer can include one or more model parameter values. Each layer can accept one or more inputs, such as from a previous layer in the case of a hidden layer, or can accept a network input, such as an input image in the case of an input layer. Each layer that accepts an input can process the input with one or more activation functions that are weighted and / or biased according to the model parameter values ​​of the layer. A layer can pass an output to a subsequent layer in the network, or in the case of an output layer, can be configured to output a segmentation map and / or one or more output images, e.g., as described herein with reference to Figures 1 and 2.

[0105] The fibrous cap detection model as a neural network can include a variety of different types of layers, such as pooling layers, convolutional layers, and fully connected layers, which can be arranged and sized according to any of a variety of different configurations for receiving and processing input image data.

[0106] Examples of fibrous capsule detection models are provided as neural networks and convolutional neural networks. However, a machine learning model may refer to any model or system that is configured to receive an input and generate an output according to the input, and that can be trained to generate an accurate output using the input training image data and / or data extracted from the input training image data. When the fibrous capsule detection model includes two or more machine learning models, the machine learning models may be trained and run end-to-end such that the output of one model may be the input of a subsequent model until the output of the final model is reached.

[0107] In some examples, the fibrous capsule detection model includes, at least in part, an encoder-decoder architecture with skip connections. As another example, the fibrous capsule detection model can include, as part of the autoencoder, one or more neural network layers trained to learn compact (encoded) representations of images, such as images taken using OCT, IVUS, NIRS, OCT-NIRS, μOCT, or any other of a variety of other imaging techniques, from unlabeled training data. The neural network layers can be further trained on input training images as described herein, and the fibrous capsule detection model can benefit from a broader training set by being at least in part trained using unlabeled training images.

[0108] In some embodiments, the fibrous cap detection model can be trained to predict arc lines of coverage angles that include one or more lipid capsules depicted in an input image. The fibrous cap detection model can generate output images annotated with the arc lines that can be used as input to refine or identify boundaries between lipid capsules and adjacent lipid pools, as described herein with reference to Figures 7A-10.

[0109] In some embodiments, the fibrous capsule detection model can be trained for lipid capsule classification in addition to or as an alternative to image segmentation as described herein, in which the fibrous capsule detection model can be trained using the same input training image data that has been annotated with the locations of lipid capsules and non-lipid regions of interest.

[0110] Training can be performed using a loss function that quantifies the difference between the location predicted by the system for the fibrous cap of the lipid pool and the ground truth location of the fibrous cap that was accepted as part of the training data for the input images.

[0111] The loss function can be, for example, the distance between the predicted and ground truth locations of the fibrous cap measured at one or more pairs of points on the predicted and ground truth locations. In general, any loss function that compares each pixel between the training images with the predicted annotations and the training images with the ground truth annotations can be used. An example loss function can include calculating a Jaccard similarity coefficient or score between the training image frames with the predicted location of the fibrous cap and the training image frames with the ground truth location of the fibrous cap. Another example loss function can be a pixel-wise cross entropy loss, although any loss function used to train a model to perform an image segmentation task can be applied.

[0112] A cost function used as part of training the system may be an average of the values ​​of the loss function values ​​over at least a portion of the training image frames processed. For example, the cost function may be an average Jaccard score over the set of training image frames. The system may train the fibrous capsule detection model until it meets one or more training criteria, which may specify, for example, a maximum error threshold when processing a validation test set of training images. Other training criteria may include, for example, a minimum or maximum number of training iterations measured as a number of epochs (i.e., a complete pass through all of the training data), batches and / or individual training examples processed, or a minimum or maximum amount of time spent performing the process 300 of training the fibrous capsule detection model.

[0113] The system may train until one or more stopping criteria are determined to be met, for example, a preset number of epochs, a minimum improvement of the system between epochs as measured using a loss function, the passage of a predetermined amount of real time, and / or until a computational budget, e.g., a predetermined number of processing cycles, is exhausted.

[0114] As described in more detail with reference to Figure 5, an existing model can be augmented with an existing machine learning model that has been trained to predict non-lipid regions of interest and also to predict the location of the fibrous cap of the lipid pool as one of multiple output channels. As also described herein with respect to Figures 1 and 2, a fibrous cap detection model can be trained to output multiple channels, each channel corresponding to a respective region of interest. These channels can be represented, for example, as one or more output images and / or as segmentation maps or other data that characterize regions of the input image as corresponding to different regions of interest, both lipid and non-lipid.

[0115] By including additional annotations of non-lipid regions of interest, the fibrous cap detection model can utilize additional information from more easily identifiable regions, such as regions of calcium, to more accurately identify the fibrous cap of the lipid pool. This is because, at least, different regions of interest, such as regions of calcium versus lipid, have different physical properties that can be compared within the same image annotated with both types of plaque. As described in more detail herein with reference to Figures 7-10, the attenuation rates of OCT signals or other imaging signals, such as imaging signals generated using IVUS, NIRS, OCT-NIRS, μOCT, etc., through different fibrous caps can be analyzed by the system to predict the fibrous cap of the lipid pool versus the cap of other regions of non-lipid plaque, such as calcium. In addition, the fibrous cap detection model, due to the nature of the processing of image data annotated with non-lipid regions of interest, can additionally output segmentation maps with multiple channels, which allows for more data to be generated together for analysis rather than separately.

[0116] FIG. 4A illustrates an exemplary input image 401A and corresponding output image 403A, expressed in polar coordinates, generated by an image segmentation system according to an embodiment of the present disclosure. FIG. 4 also illustrates an image 402A with ground truth annotations 402B of an area of ​​fibrous cap of a lipid pool. Image 403A is shown with model generated annotations 403B, generated using a model trained as described herein with reference to FIG. 3, for example. Images 401A-403A are shown in polar coordinates referenced to a reference point, for example, the center of the lumen of the imaged blood vessel. In some examples, the system can post-process the output image, for example, by applying a MAX filter with a kernel size of (15,2). Other types of filters with different sizes can be used instead or in combination. One reason for post-processing can be to smooth the boundaries of annotations before the annotations are displayed.

[0117] Figure 4B shows the example input image 401A of Figure 4A expressed in Cartesian coordinates and the corresponding output image 403A. Image 402A is also shown. Figures 4A and 4B also show that the system can be configured to output images in a variety of different formats and according to different coordinate systems.

[0118] 5 is a flow chart of an example process 500 for training a fibrous capsule detection model using a pre-trained model, according to an embodiment of the present disclosure. In some examples, process 500 can be performed as part of a transfer learning procedure to fine-tune or train the fibrous capsule detection model using another machine learning model trained to perform a different task.

[0119] The system receives a machine learning model according to block 510. The machine learning model can include a number of model parameter values ​​and can be trained to identify input images that correspond to non-lipid regions of interest, such as calcium, lumen, or media, in an image of a blood vessel.

[0120] The system replaces the output layer of the machine learning model with a new layer that receives inputs to the output layer and generates a segmentation map of the input image, according to block 520. For example, the original output layer can be replaced with a new neural network layer that receives inputs from the original output layer and outputs a segmentation map having one or more channels. Initially, the new neural network layer can include randomly initialized model parameter values.

[0121] The system uses the processed training images to train a machine learning model with a new output layer. The training images can be processed according to block 320 of process 300 in FIG. 3, and each can include a respective annotation of the location of the non-lipid region of interest. During training, the system can freeze (defer) the model parameter values ​​of each layer in the machine learning model except for the new output layer. The system can train this partially frozen model according to any technique of supervised learning, for example, using the technique described herein with reference to FIG. 3 and the training image data. The system can be configured to train the machine learning model for a certain number of epochs, for example 50 epochs, and then save the best set of model parameter values ​​of the machine learning model identified after processing a validation set of training images through the machine learning model.

[0122] The system can load the best model parameter values ​​of the new output layer, e.g., the model parameter values ​​that result in the smallest amount of loss of the loss function for the set of training images for validation. The system can then train until one or more other training criteria are met, e.g., until 200 epochs are met, but this time after unfreezing the rest of the model parameter values ​​of the model. In other words, the machine learning model can be trained and its model parameter values ​​updated, e.g., according to the process 300 described with reference to FIG. 3.

[0123] FIG. 6A is a flow chart of an example process 600A for detecting a fibrous cap of lipid pools in tissue surrounding a blood vessel, according to an embodiment of the present disclosure.

[0124] The system accepts one or more input images of a blood vessel, according to block 610. For example, the system may accept input image 120 and / or one or more additional images, as described herein with reference to FIG.

[0125] The system processes one or more input images using a fibrous cap detection model that has been trained to identify the location of the fibrous cap, according to block 620. The fibrous cap detection model can be trained using training images that are each annotated with the location of one or more fibrous caps. In some examples, the training images are further annotated with the locations of non-lipid regions of interest, such as calcium, media, and lumen of the blood vessel, that are distinct from the fibrous cap. These annotations can include visual boundaries overlaid on the output image, or can be provided as isolated data, such as a mask of pixels.

[0126] As used herein, identification and annotation can be approximate, for example, within a predetermined margin of error or threshold. Approximate annotation can under-annotate or over-annotate the fibrous cap within a predetermined margin of error or threshold. Similarly, identification of the fibrous cap according to aspects of the present disclosure can under-identify or over-identify a portion of the input image as corresponding to the identified fibrous cap within a predetermined margin of error or threshold.

[0127] In some embodiments, the system is configured to use the prediction of the non-lipid segments of the output images as part of an estimation of the signal-to-noise ratio of the output image sequence. As described herein, due to the physical properties of at least the lipid regions relative to the non-lipid regions when interacting with the imaging signal, non-lipid regions such as calcium or media can be identified with greater accuracy relative to lipid regions. While the system can use the annotation of the non-lipid regions as described herein to more accurately predict the location of the fibrous cap adjacent to the lipid pool, the system can also utilize the confidence of identifying the non-lipid regions to estimate the signal-to-noise ratio of the output image sequence.

[0128] It has been determined that a low signal-to-noise ratio (SNR) in an image frame sequence may correspond to reduced accuracy in identifying and characterizing the fibrous cap within that sequence. The system may estimate the SNR by comparing ground truth annotations corresponding to non-lipid regions, such as the tunica media, with predicted annotations of locations within the imaged vessel. If the estimated SNR is low, e.g., below a threshold, the system may flag the sequence as potentially having reduced accuracy in detecting the fibrous cap of the lipid pool.

[0129] FIG. 6B is a flowchart of an example process for flagging an output image of a fibrous cap detection model with a low signal-to-noise ratio, according to an embodiment of the present disclosure.

[0130] The system accepts an output image of the fibrous capsule detection model according to block 610B. The output image may be generated, for example, by a fibrous capsule detection model trained as described herein with reference to Figures 3 and 5. The output image may correspond, for example, to a sequence of input images captured by an imaging device during pullback as part of an imaging procedure.

[0131] In addition to detecting the location of the fibrous cap of the lipid pool, the fibrous cap detection model is also trained to detect at least one type of non-lipid region in the input image, such as the tunica media or calcium, as described in process 600B. Although the following description describes the use of regions of the tunica media in estimating the SNR, other types of plaque or tissue, such as calcium or background, may also be used.

[0132] The system estimates an average signal-to-noise ratio of the image based on a comparison of annotations of predicted regions of the tunica media in the output image to ground truth annotations of those regions of the tunica media. The system can accept the ground truth annotations, for example, as part of a validation set of the output image, or from another source configured to identify and characterize regions of interest, for example, annotation engine 110 of system 100 of FIG. 1.

[0133] The system can estimate the noise in the image by calculating the standard deviation between a region of the image that should have zero signal, e.g., the lumen of a blood vessel, and the actual value of the signal in that region in the image frame. The standard deviation between the actual and expected value of the signal in that region can be the estimated noise value of the image. Other regions of the image can be selected in addition to or instead of the region corresponding to the lumen of the imaged blood vessel. For example, the region can be a point far away from the position of the catheter as shown in the image, since a region at a point far enough away will record a signal value of zero without noise. Another example region can be the space behind the guidewire of the catheter, since the guidewire blocks all signal behind it.

[0134] In some embodiments, the system can estimate noise by measuring the highest unnormalized intensity from the refractive effects of the tissue shown in the image frame, ignoring the refractive effects from wires, stents, and catheters. In some examples, the system can estimate the SNR by fitting a two-mode Gaussian mixture model to the intensity histogram of the image frame excluding the signal intensity in the region showing the wire, stent, or catheter. The system can divide the average of the two mixture models, where the lower intensity model represents the noise and the higher intensity model represents the signal. In some examples, the system can estimate the SNR by dividing the brightest tissue pixel by the darkest lumen pixel.

[0135] The system flags the output image if the estimated SNR is below a predetermined threshold, per block 630B, and the flagged output image can be set aside and / or manually reviewed, for example. In some examples, the system performs process 600B only as part of processing a validation set to train a fibrous capsule detection model.

[0136] The image segmentation system 100 can be configured to identify and / or characterize the fibrous cap of a lipid pool by measuring radial signal intensity across an arc line in an input image of a blood vessel. As described herein, different tissues and plaques have different physical properties. Exemplary properties include peak intensity and decay rate of an imaging signal, such as an OCT signal, an IVUS signal, a NIRS signal, an OCT-NIRS signal, a μOCT signal, etc., passing through the tissue or plaque.

[0137] The decay rate of a signal refers to the change in signal intensity with increasing distance from a reference point. The reference point can be, for example, the center of the lumen of the imaged blood vessel when viewed as a two-dimensional cross section, or the center of a catheter with an imaging probe from which the signal originates. As the signal propagates through the lumen, tissue, and / or plaque, the signal becomes weaker and weaker from a peak signal intensity that typically occurs at or near the edge of the lumen, for example, when the edge of the lumen has a fibrous cap. The peak signal intensity can occur, for example, as a result of the signal at least partially reflecting off the edge of the lumen. After the edge of the lumen, the signal generally decays until the signal intensity is zero or sufficiently low that the signal can no longer be detected.

[0138] As also described herein, one challenge with accurately identifying and characterizing lipids in tissues surrounding blood vessels arises from the property of lipids to rapidly attenuate the signal transmitted through them. However, it has been determined that the attenuation rate is relatively consistent across different images of different lipid pools and fibrous caps across multiple blood vessels. Aspects of the present disclosure provide techniques for identifying signal attenuation rates and comparing the rates to known rates for various tissues and plaques to identify the fibrous cap depicted in the input image frame.

[0139] 7A shows multiple arc lines 750A and 750B from a center 760 of the lumen shown in image frame 700A. Image frame 700A also shows a center 770 of the imaging probe from which the imaging signal originates that captures image frame 700A. The arc lines 750A and 750B form a coverage angle 790 that corresponds to the portion of the lumen wall occupied by the fibrous cap. Although the arc lines 750A and 750B are shown referenced to the center 760 of the lumen as an example, the arc lines 750A and 750B can be referenced to any reference point, for example, the center 770 of the catheter with the imaging probe that captures image frame 700A. The system can be configured to measure signal strength in the image frames along different points of the arc lines 750A and 750B.

[0140] Although arc lines 750A and 750B are shown in FIG. 7A for illustrative purposes, the system does not render or draw the arc lines for display as part of execution of process 700B. In some implementations, the system can be configured to additionally transmit data for display corresponding to one or more measured arc lines and their corresponding signal strengths. Also shown in FIG. 7A is a boundary 780 between a fibrous capsule 785 and a region of lipid 795. As described herein with reference to FIGS. 7B-9, an image segmentation system can be configured to identify the fibrous capsule and to identify a boundary between the fibrous capsule and a lipid pool. In some examples, the image segmentation system is configured to receive an input image having a fibrous capsule annotation and refine the annotation to represent a more accurate boundary between the annotated capsule and an adjacent lipid pool.

[0141] FIG. 7B is a flow chart of an example process 700B for identifying a fibrous cap based on radial signal intensity of an arc line in an input image of a blood vessel, according to an embodiment of the present disclosure.

[0142] The system accepts an input image frame of a blood vessel according to block 710B. For example, the input image frame may be an image frame accepted by a user computing device through an imaging device as described herein with reference to FIG. 1. The input image may include one or more annotations of a fibrous cap surrounding a lumen of the blood vessel. These annotations may be, for example, manually labeled or may be generated by an image segmentation system as described herein with reference to FIG. 1. In some examples, when an input image is annotated with a fibrous cap but not annotated with an arc line corresponding to a coverage angle of the cap, the system may be configured to calculate an arc line referenced to a reference point such as the center of the lumen shown in the image frame. As part of the arc line generation, the system may identify the center of the lumen.

[0143] In some examples, the system accepts input image frames that have been annotated using only one or more fibrous cap arc lines shown in the input image, in which the input image frames can be annotated manually or using one or more machine learning models trained to predict arc lines defined by fibrous cap coverage angles shown in the input image frames.

[0144] The system calculates a respective radial signal strength for each arc line referenced to the center of the lumen of the blood vessel shown in the image frame, according to block 720B. Radial signal strength (or "signal strength") refers to the strength of the imaging signal in a region of the image frame corresponding to each arc that is defined relative to a reference point, such as the center of the lumen of the imaged blood vessel. The system may measure the radial signal strength along multiple points on each arc line that vary in distance relative to the reference point.

[0145] The system can convert the signal into a numerical value for each pixel of the input image frame. For each pixel, each numerical value can correspond to the amount of light reflected from the imaging probe at that point. In some examples, the system can normalize each image frame so that the highest value is 1 and the lowest value is 0. The normalized values ​​within a pixel neighborhood can be averaged to provide a less noisy signal. The pixel neighborhood can be all pixels adjacent to the target pixel, as an example, although the pixel neighborhood can be defined across other pixels relative to the target pixel, depending on the implementation.

[0146] The signal strength measurements of each arc line can be smoothed to remove noise from the various collected measurements. For example, multiple samples, e.g., 12 samples at a time, can be averaged in an angular dimension (e.g., the angular dimension formed by a line intersecting a point and a reference point, referenced to a common origin). As another example, the system can apply a Bartlett or triangular window function, e.g., with a window size of 13 pixels, to reduce noise along the radial dimension of the end points (e.g., the radial dimension of a point along an arc line represented in polar coordinates). The signal strength measurements of each arc line can be normalized by dividing each value by the signal value at the edge of the lumen.

[0147] The system identifies one or more fibrous caps shown in the input image from a plurality of radial signal intensities at points along the arc line, according to block 730B. As described herein with reference to Figures 8 and 9, the system calculates a decay rate of the plurality of radial signal intensities and compares the decay rate to other known decay rates through different tissues or plaques. For example, it has been found that an arc line through lipids has a high signal intensity after the edge of the lumen, after which the intensity of the arc line decreases according to a decay curve. The decay curve may be, for example, exponential with respect to distance from the reference point.

[0148] As part of identifying the fibrous cap, the system may receive an image annotated with the fibrous cap and identify a boundary between the fibrous cap and an adjacent lipid pool. The system may then update the annotation of the fibrous cap to more accurately reflect the boundary between the cap and the pool. The system may be configured to identify a relative radial intensity of a point along the arc line as a point corresponding to the boundary. The relative radial intensity may be proportional to a peak radial intensity measured for another point along the arc line that corresponds to a wall of the lumen. The relative signal intensity corresponding to the boundary between the fibrous cap and an adjacent lipid pool may be based on a signal intensity decay curve of a known sample of images annotated with the fibrous cap, as described below.

[0149] In some examples, the final boundary of the initially generated boundary can be determined by the system taking an average position of pixels between the annotated fibrous cap boundaries identified in different output images, e.g., obtained by both the fibrous cap detection model and the radial intensity analysis, as described herein, and the average position over each of the pixels can form part of the updated boundary.

[0150] In some examples, the system can determine the degree of overlap of annotated pixels between the separately generated output images and can include in the updated boundary pixels that represent locations where the separately annotated output images overlap. The system can interpolate the remaining pixels of the updated boundary, for example, at locations where the separately generated output images do not overlap in the annotation.

[0151] In some examples, the system may be configured to first identify the fibrous cap based on the decay rate of radial signal intensity of multiple arc lines measured from an input image of a vessel, and then update the boundary of the fibrous cap annotation using a fibrous cap detection model. In this manner, any approach implemented may be enhanced through additional processing by performing complementary techniques described herein.

[0152] In some examples, for example, either the approach using a fibrous cap detection model as described herein or the approach using radial intensity analysis can be used to detect false positives or conflicting identifications produced by either approach for the same input image. For example, if an input image is processed by a fibrous cap detection model, the same input image can also be processed to identify a fibrous cap based on the decay rates of radial signal intensity of different arc lines measured from the input image, for example, by performing processes 700 and 800 described with reference to Figures 7 and 8, respectively.

[0153] If the output images from the different techniques do not match, e.g., exactly or within a predefined similarity threshold between the annotated output images, the system can take one or more actions. For example, the system can flag the mismatch, e.g., for further review by the user. Additionally or alternatively, the system can suggest that one of the two generated output images is the "true" output of the system, or automatically select one to be the "true" output of the system. The system can make the decision, e.g., based on predefined preferences. In some examples, generating two or more generated output images using different techniques for the same input as described herein can be a feature that can be enabled or disabled by the user, providing error checking and output validation as needed.

[0154] FIG. 8 is a flow chart of an example process 800 for identifying a fibrous cap based on the rate of attenuation of radial signal intensity of a plurality of arc lines measured from an input image of a blood vessel.

[0155] The system calculates the rate of attenuation along two or more points of the arc line according to block 810. For example, the system calculates the rate of change of signal strength from different points of the arc line starting at a reference point and extending outward away from the reference point. The system can plot a curve of the rate of attenuation across two or more points as a function of distance from the reference point, as shown in FIG. 9 described herein.

[0156] The two or more points may be a subset of the plurality of points selected based on the distance of the two or more arc lines relative to a reference point. For example, based on different observations of a location where the signal strength decay rate may begin to decrease, the system may calculate the decay rate of the arc line starting from that location. This location may be, for example, 1 millimeter from the reference point, but the location may be adjusted closer to the reference point or further away from the reference point. In some examples, the signal strength curve may represent the signal strength across all of the measured arc lines. In those examples, the measured decay rate starts from a first point of the two or more points that is closest to the reference point and ends at a last point of the two or more points that is farthest from the reference point.

[0157] The system determines whether the attenuation rate is within a predetermined attenuation rate threshold according to block 820. For example, the system determines whether the attenuation rate is within a predetermined attenuation rate threshold by fitting the measured curve of the attenuation rate to a known curve, for example, a known curve of the attenuation rate of the signal intensity of the arc line propagating through the fibrous cap and lipid pool. The system can calculate the difference or error between the curves using any statistical error measurement technique, such as root-mean-square error (RMSE). The RMSE or other technique can generate an error value that the system compares to a predetermined threshold, which can be, for example, 0.02. The predetermined threshold can vary from embodiment to embodiment, for example, based on an acceptable range of error in fitting the measured curve to the known curve.

[0158] As an example of accepting a known curve for the fibrous cap of a lipid pool, the system can compare the measured curve to a curve calculated over a sample set of images labeled with one or more fibrous caps of the lipid pool. For example, the sample set can include training image data as described herein with reference to FIG. 1 and / or expert annotated images from one or more other sources. In some examples, the sample set can include image frames that have been annotated with a fibrous cap detection model trained to predict fibrous caps in input image frames, as described herein with reference to FIGS. 1-6.

[0159] If the system determines that the attenuation rate is not within the predetermined attenuation rate threshold ("NO"), process 800 ends. Otherwise, the system identifies a segment of the input frame between two or more points and the edge of the lumen as indicative of a fibrous cap, according to block 830.

[0160] In identifying the fibrous cap of a lipid pool, the attenuation rate corresponds to the attenuation of the signal intensity as the signal passes through the lipid pool. The first of the two or more points at which the attenuation rate is identified can represent the first point at which the signal passes through the lipid pool. The system can identify the space between the point at the lumen edge and the point at the lipid boundary (indicated by the first of the two or more points at which the compared attenuation rate begins on the curve) as the location of the fibrous cap. By identifying the location of the fibrous cap, the system also determines the boundary between the cap and the adjacent lipid pool.

[0161] The system can be configured to identify the edge or periphery of the lumen, for example, using a fibrous cap detection model as described herein that has been trained to output a channel that corresponds to the lumen of the imaged blood vessel.

[0162] In some examples, the system can refine the training data for training the fibrous cap detection model by processing the training data using radial intensity analysis as described herein to determine whether each image in the training data exhibits a fibrous cap. The system can sample some or all of the accepted training data accepted for training the fibrous cap detection model and determine, within a predetermined confidence level, whether the sampled training data includes images that do not exhibit a fibrous cap. The system can flag these images for further review, e.g., by manual inspection, to determine whether the images should be discarded or remain as training data. In some examples, the system can be configured to automatically add and remove flags.

[0163] In some examples, the system can pre-label the training data for training the fibrous capsule detection model by processing the training data using radial intensity analysis as described herein. The pre-labels can be used as labels for the training data to train the fibrous capsule detection model. In other examples, the pre-labels can be provided for manual inspection to facilitate manual labeling of the fibrous capsule in the training data. In some examples, the system provides at least some of the pre-labels as labels for the accepted training data and provides at least some other pre-labels for manual inspection.

[0164] In some examples, the system accepts input image frames that are annotated with only the coverage angles of the corresponding fibrous capsules, from which the system can identify arc lines that include the fibrous capsule at the annotated angles and identify the fibrous capsule as described herein with reference to Figure 8. The image data can be provided as additional training data for training a fibrous capsule detection model as described herein.

[0165] Manual annotation with coverage angles can be easier than annotating the location of the fibrous cap itself, which can allow more training data to be generated in the same amount of time. The system can be trained on more data, which can allow a greater variety of training data to be used to train the system. In addition, generating training data from images annotated with the coverage angles of the fibrous cap can improve manual annotation by standardizing annotations across the training data. For example, manual annotators can be more consistent with each other while annotating the coverage angles as opposed to annotating the location of the fibrous cap itself. Annotation of the location of the fibrous cap itself can be more susceptible to variation, for example, different annotators may estimate different thicknesses for the same fibrous cap.

[0166] 9 shows graphs 900A-D of peak signal intensity and attenuation rates through different tissues and plaques. Graph 900A plots relative signal intensity 901A (referenced to the highest and lowest detected signal intensity) versus distance 902A from a reference point, e.g., measured in pixels. Solid curve 903A represents a curve measured from a sample set of image frames that share a common characteristic, e.g., all exhibiting a fibrous cap of lipid pools. Dotted curve 904A represents at least a portion of a curve calculated from measured arc lines of input image frames, e.g., two or more arc lines as described herein with reference to FIG. 900.

[0167] Region 905A corresponds to a region of the image frame where the attenuation rate is measured from two or more points of the arc line corresponding to positions within the region. As described herein, the attenuation rates of the two or more points are fitted to the solid curve 903A previously accepted by the system. In this example, the degree of fit between the curves has an error of 0.02. Region 906A corresponds to a region of the image frame predicted to show the fibrous cap of the lipid pool. As described herein with reference to FIG. 10, the system can estimate the thickness of the fibrous cap of the lipid pool. Line 950 corresponds to the end of region 906A, which also represents the boundary between the fibrous cap and the lipid pool. In other words, line 950 corresponds to the point where curve 903A begins to attenuate at a rate corresponding to the previously measured signal intensity attenuation when passing through lipid.

[0168] Graphs 900B-900D show dotted curve 904A corresponding to the measured attenuation rate, along with solid curves 904B-904D previously generated from sets of image frames with different characteristics, e.g., different plaques around the imaged vessel. Solid curve 904B of graph 900B is a curve measured from a set of image frames showing a fibrous cap of lipid pools, as described with reference to graph 900A. Solid curve 904B shows that the signal intensity is usually higher (brighter) after the luminal edge.

[0169] The solid curve 904C in graph 900C is a curve measured from a set of image frames showing the visible tunica media in the imaged vascular tissue. The solid curve 904C shows the signal intensity as being generally higher (brighter) after the lumen edge, but the decay rate of curve 904C is not fitted to the dotted curve 904A, nor is it fitted to the solid curves 903A or 904B. In graph 900C, the fitting error between the curves is measured as 0.05.

[0170] The solid curve 904D of graph 900D is a curve measured from a set of image frames that show no visible tunica media, calcium, or lipids. The solid curve 904D exhibits a lower peak intensity relative to the luminal edge of the vessel measured to generate the solid curves 904A-904C, and the fitting error between curve 904D and dotted curve 904A is also higher (0.03) than the fitting error between dotted curve 904A and solid curve 903A. In some examples, a predetermined threshold can be generated based on a comparison of the curves of a known sample set, e.g., the solid curves 904A-904D, and a comparison of the difference in fitting errors of the different curves.

[0171] FIG. 10 is a flow chart of an example process 1000 for measuring fibrous cap thickness using peak radial signal intensity in a series of measurement arcs of an imaged blood vessel, according to an embodiment of the present disclosure.

[0172] The system identifies a first point on the arc line that corresponds to a peak radial signal intensity, according to block 1010. For example, the system may plot the radial signal intensity of multiple points along the arc line and identify the point having the highest signal intensity value. As described herein with reference to Figures 8 and 9, the peak signal intensity may generally occur after the edge of the lumen of the imaged blood vessel.

[0173] The system identifies a second point of the arc line corresponding to a radial signal intensity that meets a threshold intensity value relative to the peak radial signal intensity, per block 1020. For example, the threshold intensity value can be set to 80% of the peak signal intensity. The threshold intensity value can be varied from embodiment to embodiment, for example, based on analysis of a sample set of image frames of annotated fibrous cap and a comparison of signal intensities of points along the arc line at both ends of the fibrous cap.

[0174] The system measures the thickness of the fibrous cap as the distance between a first point and a second point on the arc line, according to block 1030. The system can repeat process 1000 for multiple arc lines originating from the same reference point, and can repeat process 1000 for one or more lines originating from the same reference point between the arc lines that define a coverage angle corresponding to the fibrous cap. For example, because the fibrous cap may have different thicknesses at different points, the system can measure thickness along different lines, according to block 1030, to identify areas of greater or lesser fibrous cap thickness.

[0175] After estimating the location and thickness of the fibrous cap in the received input image frames, the system can be configured to output data defining these estimates, e.g., on a display or as part of a downstream process for diagnosis and / or analysis, as described herein with reference to Figures 1 and 2. In some examples, the system can be configured to flag image frames with a predicted fibrous cap of lipid pool thinner than a thickness threshold, e.g., through a prompt or some visual indicator on the display. For example, a thickness threshold can be set such that image frames are flagged for possible further review and analysis, since image frames exhibiting a fibrous cap thinner than a thickness threshold can be an indicator of an increased risk of plaque disruption, such as TCFA, in the imaged vessel.

[0176] <Example of a computing environment> 11 is a block diagram of an example computing environment for implementing an image segmentation system 100 according to aspects of the disclosure. The system 100 can be implemented on one or more devices having one or more processors at one or more locations, such as a server computing device 1115. The user computing device 1112 and the server computing device 1115 can be communicatively coupled to one or more storage devices 1130 via a network 1160. The storage device(s) 1130 can be a combination of volatile and non-volatile memory and can be in the same physical location as the computing devices 1112, 1115 or in a different physical location. For example, the storage device(s) 1130 can include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, solid-state drive, tape drive, optical storage device, memory card, ROM, RAM, DVD, CD-ROM, writeable memory, and read-only memory.

[0177] The server computing device 1115 may include one or more processors 1113 and memory 1114. The memory 1114 may store information accessible by the processor(s) 1113, including instructions 1121 that the processor(s) 1113 may execute. The memory 1114 may also include data 1123 that the processor(s) 1113 may retrieve, manipulate, or store. The memory 1114 may be any type of non-transitory computer-readable medium capable of storing information accessible by the processor(s) 1113, such as volatile and non-volatile memory. The processor(s) 1113 may include one or more central processing units (CPUs), graphic processing units (GPUs), field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs).

[0178] The instructions 1121 may include one or more instructions that, when executed by the processor(s) 1113, cause the one or more processors to perform operations defined by the instructions. The instructions 1121 may be stored in an object code format that is processed directly by the processor(s) 1113, or may be stored in other formats including an interpretable script or a collection of separate source code modules that are interpreted on demand or pre-compiled. The instructions 1121 may include instructions that implement the system 100 consistent with aspects of the present disclosure. The system 100 may be executed using the processor(s) 1113 and / or other processors located remotely from the server computing device 1115.

[0179] The data 1123 may be retrieved, stored, or modified by the processor(s) 1113 according to the instructions 1121. The data 1123 may be stored in computer registers or in a relational or non-relational database as a table with multiple different fields and records or as a JSON, YAML, proto, or XML document. The data 1123 may also be formatted into a computer readable format such as, but not limited to, binary values, ASCII, or Unicode. Moreover, the data 1123 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory including other network locations, or information used by a function to calculate the related data.

[0180] The user computing device 1112 may be configured similarly to the server computing device 1115, with one or more processors 1116, memory 1117, instructions 1118, and data 1119. The user computing device 1112 may also include a user output 1126 and a user input 1124. The user input 1124 may include any suitable mechanism or technique for accepting input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0181] The server computing device 1115 may be configured to transmit data to the user computing device 1112, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 1126. The user output 1126 may also be used to display an interface between the user computing device 1112 and the server computing device 1115. The user output 1126 may alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non-visual and non-audible information to a platform user of the user computing device 1112.

[0182] 11 illustrates the processors 1113, 1116 and memories 1114, 1117 as being present within computing devices 1115, 1112, but the components described herein, including the processors 1113, 1116 and memories 1114, 1117, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 1121, 1118 and data 1123, 1119 may be stored on a removable SD card and others in a read-only computer chip. Some or all of the instructions and data may be stored in locations physically remote from the processors 1113, 1116 but still accessible by the processors 1113, 1116. Similarly, the processors 1113, 1116 may include a collection of processors that may perform parallel and / or sequential operations. The computing devices 1115 , 1112 may each include one or more internal clocks that provide timing information that can be used to time operations and programs executed by the computing devices 1115 , 1112 .

[0183] The devices 1112, 1115 may be capable of direct and indirect communication over the network 1160. For example, using network sockets, the user computing device 1112 may connect to services running in the data center 1150 through the Internet Protocol. The devices 1115, 1112 may set up listening sockets that may accept initiating connections for sending and receiving information. The network 1160 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using communication protocols proprietary to one or more companies. The network 1160 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth™ standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi™ communication protocol), or using a variety of communication standards, such as the LTE™ standard for wireless broadband communication. The network 1160 may additionally or alternatively support wired connections between the devices 1112, 1115, including via various types of Ethernet connections.

[0184] 11, it will be appreciated that aspects of the disclosure may be implemented according to a variety of different configurations and quantities of computing devices, including in serial or parallel processing paradigms, or across a distributed network of multiple devices. In some implementations, aspects of the disclosure may be performed in a single device and any combination thereof.

[0185] Although the operations depicted in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed more than once, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring separation of those components. The components, modules, programs, and engines described may be integrated together as a single system or may be part of multiple systems. Additionally, as described herein, an image segmentation system, such as the image segmentation system 100 of FIG. 1, may perform the processes described herein.

[0186] Unless otherwise indicated, the alternative examples described above are not mutually exclusive and can be implemented in various combinations to achieve their own advantages. These features and other variations and combinations described above can be utilized without departing from the subject matter defined by the claims, and the description of the above-described embodiments should be construed as illustrative, rather than limiting, the subject matter defined by the claims. In addition, the provision of examples described herein, and clauses such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to specific examples, but rather, such examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in various figures may identify the same or similar elements.

Claims

1. One or more processors receive one or more input images of a blood vessel; the one or more processors process the one or more input images using a machine learning model trained to identify the location of a fibrous capsule of the blood vessel, the machine learning model being trained with a plurality of training images annotated with the location of one or more fibrous capsules, each fibrous capsule being adjacent to a respective lipid pool; the one or more processors receive, as an output of the machine learning model, one or more output images having segments visually annotated such that the predicted location of the fibrous capsule is represented; generating, using the one or more processors, an updated boundary of the fibrous capsule relative to adjacent lipid pools from the one or more output images based on signal intensities of a plurality of points in the one or more input images; A method for identifying a fibrous capsule of a blood vessel, comprising:

2. The method of claim 1, wherein the one or more input images are further annotated with segments corresponding to at least one of calcium, the lumen of the blood vessel, and the media.

3. The one or more input images include annotated segments representing one or more regions of the media; the one or more input images are images received from an imaging probe during retraction of the imaging probe in the blood vessel; the method further comprises: estimating, by the one or more processors, an average signal-to-noise ratio (SNR) of the one or more input images based on a comparison between a predicted annotation of a region of the media in the one or more input images and one or more ground truth annotations of the region of the media in the one or more input images; responsive thereto, if it is determined that the average SNR is below a predetermined threshold, flagging, by the one or more processors, the one or more output images corresponding to the one or more input images; The method of claim 2, further comprising:

4. The method according to claim 3, wherein the imaging probe is any one of an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, and a micro-OCT (μOCT) imaging probe.

5. The method according to claim 1, wherein the plurality of points are along one or more arc lines surrounding the fibrous capsule.

6. Receiving one or more input images of a blood vessel; Processing the one or more input images using a machine learning model trained to identify the location of a fibrous capsule of the blood vessel, the machine learning model being trained with a plurality of training images annotated with the location of one or more fibrous capsules, each fibrous capsule being adjacent to a respective lipid pool; Receiving, as an output of the machine learning model, one or more output images visually annotated with segments representing the predicted locations of the fibrous capsules; Generating, from the one or more output images, an updated boundary of the fibrous capsule relative to adjacent lipid pools based on the signal intensities of a plurality of points in the one or more input images; A system comprising one or more processors for performing the above.

7. The system according to claim 6, wherein the one or more input images are further annotated with segments corresponding to at least one of calcium, the lumen of the blood vessel, and the media.

8. The one or more input images include annotated segments representing one or more regions of the media; The one or more input images are images received from the imaging probe during retraction of the imaging probe in the blood vessel; The one or more processors further Estimating an average signal-to-noise ratio (SNR) of the one or more input images based on a comparison between the predicted annotation of the region of the media in the one or more input images and one or more ground truth annotations of the region of the media in the one or more input images; Accordingly, when it is determined that the average SNR is below a predetermined threshold, flagging the one or more output images corresponding to the one or more input images; The system according to claim 7, which performs the above.

9. The system according to claim 8, wherein the imaging probe is any one of an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) imaging probe, a near-infrared spectroscopy (NIRS) imaging probe, an OCT-NIRS imaging probe, and a micro-OCT (μOCT) imaging probe.

10. The system according to claim 6, wherein the plurality of points are along one or more arc lines surrounding the fibrous capsule.

11. The system according to claim 7, wherein the step of receiving the one or more output images includes receiving, for each input image, each visually annotated segment of the input image representing the predicted position of the fibrous capsule.

12. The system according to claim 7, wherein the one or more processors further perform a step of receiving, for each of the one or more output images, a measurement value of one or more thicknesses of each fibrous capsule whose position is predicted in the output image.

13. The system further comprises an imaging probe communicably connected to the one or more processors, wherein the one or more processors further receive, when the imaging probe is inside the blood vessel, image data corresponding to the one or more input images from the imaging probe in order to receive the one or more input images of the blood vessel. The system according to claim 7.

14. In the step of generating the updated boundary, the one or more processors further measure the signal intensity of a plurality of points along one or more arc lines surrounding the fibrous capsule, and determine the boundary between the fibrous capsule and the adjacent lipid pool based on a comparison between the measured attenuation rate of the signal intensity of the plurality of points and a predetermined attenuation rate of the signal intensity through the fibrous capsule of the lipid. The system according to claim 7.

15. The system according to claim 14, wherein, in order to determine the boundary between the fibrous capsule and the adjacent lipid pool, the one or more processors further perform a step of identifying, among the plurality of points, points having a measured value of signal intensity proportional within a predetermined threshold range to the peak signal intensity among the plurality of points.

Images