Resolution and Manipulation Decision Focus in Machine Learning-Based Vascular Imaging

Abstract

A system (SY) for determining the relative importance of each of a plurality of image features (F n ) in a vascular medical image that affects an overall diagnostic metric, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule. A medical image database (MIDB) includes a plurality of vascular medical images (M 1..k ). A rule generation unit (RGU) analyzes the plurality of vascular medical images and automatically generates at least one diagnostic rule corresponding to a common diagnosis of a subset of the plurality of vascular medical images based on a plurality of image features common to the subset of vascular medical images. An image providing unit (IPU) provides a current vascular medical image (CVMI) including a plurality of image features. A diagnostic metric calculation unit (DMCU) calculates an overall diagnostic metric for the current vascular medical image by applying at least one automatically generated diagnostic rule to the current vascular medical image. A decision propagation unit (DPU) identifies the relative importance of each of the plurality of image features in the current vascular medical image to the calculated overall diagnostic metric.

Description

Technical Field

[0001] The present invention relates to the field of image-guided therapy. More specifically, it relates to decision-making in the field of vascular imaging. Background Art

[0002] Vascular imaging modalities such as intravascular ultrasound (i.e., IVUS) or optical coherence tomography (i.e., OCT) play an increasingly important role in imaging coronary and peripheral blood vessels. Applications include evaluating coronary artery plaque and calcification burden, stenosis level characterization, or imaging support in the context of stent placement.

[0003] Data-driven decision support systems are also evolving in this field. There is an increasing need for clinicians to understand where automatically derived decisions originate and to interact with the decision-finding process when necessary. This application addresses this need and others.

[0004] The following documents are known in the present invention and related fields:

[0005] [1] Dmitry Nemirovsky, Imaging of High-Risk Plaque, Cardiology 100:160–175; 2003.

[0006] [2] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller: Methods for Interpreting and Understanding Deep Neural Networks Digital Signal Processing, 73:1-15, 2017.

[0007] [3] Zeiler, Matthew D., and Rob Fergus. Visualizing and understanding convolutional networks. Computer Vision–ECCV 2014. Springer. International Publishing, 818-833, 2014.

[0008] [4]R.R. Salakhutdinov and G.E. Hinton. Deep Boltzmann machines. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 12, 2009.

[0009] Another document WO2015095282 discloses systems and methods for predicting coronary plaque vulnerability. A method includes acquiring anatomical image data of at least a portion of a patient's vascular system; performing, using a processor, one or more of image feature analysis, geometric analysis, computational fluid dynamics analysis, and structural mechanics analysis on the anatomical image data; predicting, using the processor, the vulnerability of coronary plaques present in the patient's vascular system, wherein predicting the vulnerability of coronary plaques includes calculating adverse plaque characteristics based on results of the image feature analysis, geometric analysis, computational fluid dynamics analysis, and structural mechanics analysis of the anatomical image data; and reporting, using the processor, the calculated adverse plaque characteristics. Summary of the Invention

[0010] Vascular imaging techniques such as intravascular ultrasound (IVUS) or (coronary) computed tomography angiography (i.e., (C)CTA) play an increasingly important role in imaging the coronary arteries and surrounding blood vessels. Applications include evaluating coronary plaques and calcification burden, stenosis level characterization, or imaging support in the context of stent placement.

[0011] The emergence of data-driven decision support systems in this field creates the need for clinicians to understand where automatically derived decisions originate and to interact with the decision-finding process when necessary. The higher and more abstract the decision level, the greater the need for the understanding process.

[0012] In a specific example of plaque detection and risk scoring (e.g., for acute coronary syndrome, i.e., ACS), algorithms can be trained based on data to detect certain image features, such as soft plaques and calcium, from intravascular images along the vessel of interest and incorporate them into the risk score. In this case, image artifacts, such as those caused by motion such as heart motion, contrast agent variations, existing stents, or other implants, can occur, remain unrecognized, and incorrectly influence the decision.

[0013] The present invention provides a system for determining the relative importance of each of a plurality of image features of a vascular medical image that affect an overall diagnostic metric, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule. The system includes a medical image database, a rule generation unit, an image providing unit, a diagnostic metric calculation unit, and a decision propagation unit. The medical image database includes a plurality of vascular medical images. The rule generation unit analyzes the plurality of vascular medical images and automatically generates at least one diagnostic rule corresponding to a common diagnosis of a subset of the plurality of vascular medical images based on a plurality of image features common to the subset of the plurality of vascular medical images. The image providing unit provides a current vascular medical image including a plurality of image features. The diagnostic metric calculation unit calculates an overall diagnostic metric for the current vascular medical image by applying at least one automatically generated diagnostic rule to the current vascular medical image. The decision propagation unit identifies the relative importance of each of the plurality of image features in the current vascular medical image to the calculated overall diagnostic metric.

[0014] For example, an automatically generated rule for analyzing a vascular image to provide an overall diagnostic metric such as a calcification burden can be generated by a machine learning process. Typically, the machine learning process weights various image features in order to arrive at a rule and then calculates the overall diagnostic metric. However, the weighting of each feature is generally a hidden parameter, and thus the relative importance of each of the image features that affect the diagnostic metric is not transparent. Known systems either only calculate an overall diagnostic parameter for an image or indicate high-risk regions in the image, but do not indicate the relative importance of each feature in the image with respect to the overall diagnostic parameter. In some cases, such automatically generated rules can incorrectly identify an image feature with a high weighting, resulting in a high overall risk score for the image. Image artifacts from, for example, movement or contrast agent variations due to cardiac motion, existing stents, implants, or bifurcations can be incorrectly automatically identified in this manner. By identifying the relative importance of each of the plurality of image features in a vascular medical image to the calculated overall diagnostic metric, the system advantageously allows a user to verify the underlying assumptions of the automatically generated rule. The user will easily know from their own experience whether such motion artifacts, stents, etc. should indeed have the identified relative impact on the calculated overall diagnostic metric.

[0015] According to one aspect, the diagnostic metric calculation unit is further configured to: receive user input from a user input device indicating at least one of the plurality of image features, and i) recalculate the overall diagnostic metric for the current vascular medical image by changing the relative importance of the at least one of the plurality of image features and reapplying the at least one diagnostic rule to the current vascular medical image or ii) change the relative importance of the at least one of the plurality of image features and cause the rule generation unit to re-analyze the plurality of vascular medical images and automatically generate a revised diagnostic rule corresponding to the selected common diagnosis of the vascular medical images based on the plurality of image features common to the selection of the vascular medical images, and cause the diagnostic metric calculation unit to calculate a revised overall diagnostic metric for the current vascular medical image by applying the automatically generated revised diagnostic rule to the current vascular medical image, and cause the decision propagation unit to identify the relative importance of each of the plurality of image features in the current vascular medical image to the calculated revised overall diagnostic metric.

[0016] Thus, according to this aspect, the system can allow a user to change, for example, the relative importance of a region of an image currently being analyzed, and reapply the same diagnostic rule and recalculate the overall diagnostic metric for the image, or change, for example, the relative importance of a region of an image currently being analyzed, and recalculate the diagnostic rule and recalculate the overall diagnostic metric for the image. For example, the user can indicate to the system that a region in an image corresponding to a stent that has been mis-identified as having that impact on the originally calculated diagnostic metric should not contribute to the metric at all, and in response thereto, cause the system to recalculate the overall diagnostic metric by ignoring any contribution of the stent to the metric.

[0017] According to other related aspects, corresponding methods and computer program products are also provided.

[0018] Thus, the present invention addresses one or more of the aforementioned needs by propagating automated decisions back to relevance scores within the input image space. The relevance scores can be indicated by means of heatmaps. These heatmap-like structures can be used to understand the decisions, validate or reject them, or most importantly, provide a way to give feedback to the clinician. The latter is achieved by image manipulations that reduce or increase the impact of specific image features on the decision. The heatmap can be used to directly modulate the input image, or can be used as a separate additional feedback channel for the system. In addition to a standard feed-forward decision system, a generative model with bidirectional inference constitutes another way to input masked or modulated inputs and recalculate the required outputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Illustrated is a system SY according to some aspects of the present invention, the system SY being for determining the relative importance of a plurality of image features F of a vascular medical image that affect an overall diagnostic metric, the overall diagnostic metric being computed for the image according to an automatically generated diagnostic rule. n for each of which.

[0020] Figure 2 Illustrated is a method MET according to some aspects of the present invention.

[0021] Figure 3 Illustrated is an exemplary system, where a network uses IVUS image data for prediction and interpretation and outputs it to a user in the form of a calcium score.

[0022] Figure 4 Illustrated is an exemplary correlation heatmap that is used to feed information back into the system to potentially correct decisions after re-running the evaluation of the images on (a-c) IVUS images and (d-f) coronary CTA images. DETAILED DESCRIPTION

[0023] To illustrate the principles of the present invention, a system for determining the relative importance of each of a plurality of image features of a vascular medical image is described with particular reference to IVUS images. However, it should be appreciated that the system can alternatively be used with images from other imaging modalities, including but not limited to OCT images, (C)CTA images, or angiography images.

[0024] As noted above, vascular imaging modalities such as IVUS or OCT are playing an increasingly important role in imaging coronary as well as peripheral vessels. Applications include evaluating coronary plaque and calcification burden, stenosis level characterization, or imaging support in the context of stent placement.

[0025] The emergence of data-driven decision support systems in this field creates a need for clinicians to understand where automatically derived decisions originate and to interact with the decision-finding process when necessary. The higher the level of decision, the more abstract it is, and the greater the need for an understanding process. The present invention can also help meet the increasing requirements for formal documentation of diagnostic and treatment decision-making.

[0026] In the specific example of plaque detection and risk scoring for, e.g., ACS, algorithms trained on data to detect specific image features from intravascular images along vessels of interest can add this local evidence to the overall risk score. Specifically, the detection and assessment of soft plaque patterns, as described in reference document [1], whose extent and distribution on the vessel tree will substantially affect the ACS score. In this case, it may happen that image artifacts (e.g., caused by motion such as cardiac motion, contrast agent changes, existing stents or other implants) remain unrecognized and erroneously influence the decision system. Additional deterministic measures on the output often do not address the problem, and no module for interaction between the clinician and the support system is provided. In other cases, the support system may not be sufficiently trained to cope with special data occurrences, such as contrast loss, motion artifacts, unusual views of inexperienced users. These situations often require repeated acquisitions or abandonment of the support system, which is time-consuming or impossible in the case of post hoc processing.

[0027] Figure 1 A system SY for determining a plurality of image features F of a vascular medical image that influence an overall diagnostic metric according to some aspects of the present invention is illustrated. n The relative importance of each of the above items is determined by the automatic generation of diagnostic rules, wherein the overall diagnostic metric is calculated for the image according to the automatically generated diagnostic rules. The system SY comprises a medical image database MIDB, a rule generation unit RGU, an image providing unit IPU, a diagnostic metric calculation unit DMCU and a decision propagation unit DPU. The functions of these items can generally be provided by computer-related products. More specifically, the functions of the items RGU, DMCU and DPU can be implemented by one or more processors, and the functions of the units MIDB and IPU can be implemented by one or more memories. The (one or more) memories can be supported or controlled by one or more processors.

[0028] Continue to refer Figure 1 , the medical image database MIDB includes multiple vascular medical images M 1..k In an embodiment consistent with the present invention, the image may be, for example, i) an IVUS image, ii) an OCT image, iii) a (C) CTA image, or iv) an angiography image. However, the present invention is not limited to these examples. The rule generation unit RGU is configured to analyze the plurality of vascular medical images M 1..k Based on multiple image features F shared by subsets of vascular medical images nAutomatically generate at least one diagnostic rule corresponding to a common diagnosis of a subset of multiple vascular medical images. Suitable techniques for analyzing such images are generally known from the above-cited documents [1-4] and can generally include providing at least one diagnostic rule by performing, for example, machine learning algorithms, deep learning algorithms, or automated intelligence algorithms. Image feature F n Can generally correspond to image intensity, or more specifically, by way of non-limiting example, to the presence of plaque, calcium, ruptured plaque, thrombus in a lesion, the presence of fat in plaque adjacent to the lumen, and / or include one or more of a calculated lumen area, a calculated plaque area, a calculated lesion size, a distribution or proportion of calcium in a lesion.

[0029] Continuing to refer to Figure 1 , the image providing unit IPU is configured to provide a current vascular medical image CVMI including a plurality of image features F n The current vascular medical image CVMI can be an image received from a medical image database MIDB, or in fact an image received from another database such as a database within the image providing unit IPU or a web server, or in fact an image received from a catheter CA within the vasculature VA. The current vascular medical image CVMI can be a pre-recorded image or a live image. The vasculature VA can be part of the vasculature of a human or animal body. Figure 1 The portion illustrated in includes a lumen LU defining an axis A-A', and the lumen LU has a corresponding vascular wall VW. In the illustrated example, the catheter CA can be an IVUS or OCT imaging catheter that provides OCT or IVUS image data that is measured rotationally about the axis A-A', as indicated by the angle θ. In the case of IVUS, the intensity of the data corresponds to the ultrasonic reflectivity of the surrounding material. In the case of OCT, the data corresponds to the interference between an optical irradiation beam and the reflected portion of the beam. Rotation refers to measuring the data around a portion in the angular direction θ. The data can be acquired, for example, from a fixed array of IVUS / OCT detector elements arranged around the catheter CA with respect to the angular direction θ, or the data can be acquired, for example, from a scanning or rotating IVUS / OCT detector, where one or more detector elements or corresponding optical elements rotate about the catheter CA with respect to the angular direction θ. The data is typically acquired during a so-called "pullback" procedure, where the catheter CA, which has typically been inserted into the vasculature VA with the aid of a guide wire, is subsequently pulled back along the vasculature while the data is being acquired.

[0030] As illustrated by the dashed line interconnecting the catheter CA and the medical image database MIDB, the catheter CA can also be used in a similar manner as described above to obtain multiple vascular medical images M 1..kProvided to the medical image database MIDB. In one of the above embodiments, the current vascular medical image CVMI can be multiple vascular medical images M 1..k One of them. In another embodiment, the vascular medical image M can be acquired at a time point earlier than the current vascular medical image CVMI 1..k . In some embodiments, the vascular medical image M can be acquired from the same vasculature as the current vascular medical image CVMI 1..k , while in other embodiments, the vascular medical image M can be acquired from a vasculature different from the current vascular medical image CVMI 1..k . The vascular medical image M can be acquired using the same or a different catheter as the catheter that provided the current vascular medical image CVMI 1..k . The vascular medical image M can be acquired before the current vascular medical image CVMI 1..k .

[0031] When the system SY is used with alternative image types (such as images provided by (C)CTA or angiography), the sources of these images may also be as described above.

[0032] Continuing to refer to Figure 1 , the diagnostic metric calculation unit DMCU calculates an overall diagnostic metric for the current vascular medical image CVMI by applying at least one automatically generated diagnostic rule to the current vascular medical image CVMI. Overall, it is intended that multiple image features have an impact on the diagnostic metric, rather than simply one image feature that affects the metric. Non-limiting examples of the overall diagnostic metric include: risk scores corresponding to risks of acute coronary events such as ACS, calcification burden, risks of peripheral vascular diseases, risks of coronary plaques, risks of coronary vascular diseases, or stenosis level characterizations. Other overall diagnostic metrics can be calculated in a similar manner.

[0033] Subsequently, the decision propagation unit DPU identifies multiple image features F in the current vascular medical image CVMI nThe relative importance of each in the calculated overall diagnostic metric. Thus, relative to simply identifying, for example, regions of high risk in a diagnostic image, the relative importance of each in the image region to the calculated high-risk score is identified. This allows the user to determine the relative importance of the key regions or aspects of the current image to the calculated overall diagnostic metric and thus validate its relative importance. In one example implementation, the impact of each pixel in the current cardiovascular medical image (CVMI) on the overall diagnostic metric for that image can be rated on a scale from 1 to 100, where 100 is the sole factor influencing the calculated overall diagnostic metric and is indicated, for example, by color. Groups of pixels can be represented in a similar manner. This is highly beneficial when automatically generating such diagnostic metrics, as the transparency of the decision-making supports the user's confidence in the automated decision-making and allows for the identification of results that do not conform to the user's own experience.

[0034] A preferred way is where such relative importance can be indicated in the form of a heat map superimposed on the current cardiovascular medical image. In this case, the image regions having a relatively high impact on the overall diagnostic metric can be colored, for example, red, while the image regions having a relatively low impact on the overall diagnostic metric can be transparent. Image regions having an intermediate impact can also be assigned various colors. By indicating the relative importance in this way, the user can easily validate the calculated overall diagnostic metric. The relative importance of calculated image features (such as, for example, lumen diameter, etc.) can also be indicated in a similar manner. Clearly, other forms of relative importance indication can alternatively be used, such as text and table indications on a display.

[0035] In one implementation, the relative importance of each of a plurality of image features F n in the calculated overall diagnostic metric can be determined based on the sensitivity of the change of each of the plurality of image features. The sensitivity of the overall diagnostic metric can be calculated, for example, by varying the intensity of each image feature by a predetermined percentage and thus used to identify the relative importance of regions of the current cardiovascular medical image. The relative importance of calculated image features (such as, for example, lumen diameter, etc.) can also be changed by a predetermined percentage, for example, by deforming the image using known image deformation techniques in order to indicate its relative importance to the calculated overall diagnostic metric in a similar manner.

[0036] In one implementation, a change in the entropy of the calculated overall diagnostic metric can be calculated as a result of changing a plurality of image features F n This change in entropy (or more specifically, the change in information gain) as a result of changing the relative importance of the image features can provide an alternative indication of the relative importance of the image features to the calculated overall diagnostic metric.

[0037] In one embodiment, the diagnostic metric calculation unit DMCU is further configured to calculate a measure of the uncertainty of the overall diagnostic metric for the current vascular medical image. One way to calculate uncertainty is by training an additional network to predict the error of its own predictions. The document "Learning uncertainty in regression tasks by artificial neural networks" by Pavel Gurevich and Hannes Stuke (January 20, 2018, arXiv:1707.07287, published online at https: / / arxiv.org / abs / 1707.07287 ) describes a technique applicable to this. The document "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning" by Yarin Gal and Zoubin Ghahramani (October 4, 2016, arXiv:1506.02142, published online at https: / / arxiv.org / abs / 1506.02142 ) describes another way to calculate uncertainty. In this latter method, dropout and Monte Carlo estimation of the mean prediction and its variance are used to calculate uncertainty. This method uses an ensemble of weak rules. In a manner similar to the identification of the relative importance of each image feature pair in the overall diagnostic metric calculated for multiple image features F n , the decision propagation unit DPU can then identify the relative importance of each image feature pair in the current vascular medical image CVMI to the uncertainty of the overall diagnostic metric. This uncertainty can also support the user in validating the results of the automatically generated rules. For example, the user can see that a particular image feature has a high impact on the calculated overall diagnostic metric, but its uncertainty is very high. This can alert the user, as described below, to consider changing or even ignoring the contribution of this image feature to the calculated overall diagnostic metric and recalculating the metric. As described below, the system can alternatively perform this recalculation automatically.

[0038] As described above, in one embodiment, the system SY can additionally receive user input. The user input can be used to revise the diagnostic rules and / or recalculate the overall diagnostic metric. Thus, this embodiment can allow the user to guide the creation of improved diagnostic rules and / or correct the diagnostic metric based on the user's own experience. In one instance, the diagnostic metric calculation unit DMCU of the system SY is further configured to receive the user input from the user input device UID. The user input indicates at least one of the multiple image features F n , and the diagnostic metric calculation unit is configured to, based on the user input, by changing the multiple image features Fn the relative importance of at least one of the foregoing and reapplying at least one diagnostic rule to the current vascular medical image (CVMI) to recalculate the overall diagnostic metric for the current vascular medical image (CVMI). The user input may, for example, indicate image features F corresponding to one or more image artifacts n . Such image artifacts may include, for example, cardiac motion, contrast agent variations, existing stents, implants, bifurcations, etc. Via the automatically generated rules, such image features may be erroneously determined to have a high relative importance for the calculated overall diagnostic metric. Thus, the user may manually change the relative importance of such features based on their own experience and thereby correct the diagnostic metric.

[0039] The user may provide the user input via Figure 1 a user input device UID in (such as a pointer, mouse, keyboard, etc.). The user input device UID may be used to define a region of the current vascular medical image (CVMI) provided by the image providing unit (IPU). The user may, for example, identify the contour of a high relative importance image feature in the current vascular medical image. The system SY may facilitate this, for example, by automatically providing a contour around an image feature having a relative importance exceeding a predetermined threshold, where the user may select the image contour to provide the user input.

[0040] Changing the relative importance of at least one of the plurality of image features F n in the foregoing may, for example, mean increasing or decreasing or even ignoring the relative importance of at least one of the plurality of image features F n in the foregoing. When the rule generation unit (RGU) automatically generates at least one diagnostic rule by weighting each of the plurality of image features F n in the foregoing; changing the relative importance of at least one of the plurality of image features F n in the foregoing may include changing the weight of at least one of the plurality of image features F n in the foregoing.

[0041] In another example, the user input may be used to change the relative importance of at least one of the plurality of image features F n in the foregoing and cause the rule generation unit (RGU) to re-analyze a plurality of vascular medical images M 1..k and automatically generate, based on the plurality of image features common to the selection of the vascular medical images M 1..k a diagnostic rule associated with the vascular medical images M 1..ka revised diagnostic rule corresponding to the selected common diagnosis, and cause the diagnostic metric calculation unit to calculate a revised overall diagnostic metric for the current vascular medical image CVMI by applying the revised automatically generated diagnostic rule to the current vascular medical image CVMI, and cause the decision propagation unit DPU to identify a plurality of image features F in the current vascular medical image CVMI n The relative importance of each of the calculated revised overall diagnostic metrics in. By revising the diagnostic rule and recalculating the diagnostic metric in this way, even more accurate diagnostic metrics can be obtained because errors in the automatically generated rules can be reduced.

[0042] In another embodiment, there is provided a method for determining the relative importance of a plurality of image features F of a vascular medical image that affect an overall diagnostic metric n for each of the image features, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule. Refer to Figure 2 , which illustrates a method MET according to certain aspects of the present invention, the method MET including:

[0043] Analyze a plurality of vascular medical images M of APVMI 1..k , and automatically generate at least one diagnostic rule corresponding to the common diagnosis of the subset of vascular medical images based on a plurality of image features common to the subset of vascular medical images;

[0044] Provide a current vascular medical image CVMI including a plurality of image features F n ;

[0045] Calculate an overall diagnostic metric CODM for the current vascular medical image CVMI by applying at least one automatically generated diagnostic rule to the current vascular medical image CVMI; and

[0046] Identify the relative importance of each of a plurality of image features F of IRI in the current vascular medical image CVMI n to the calculated overall diagnostic metric.

[0047] The method MET may include performing additional steps of one or more of the functions described above with respect to Figure 1 the items described. Additionally, the method MET may be stored as instructions on a computer program product, the instructions causing a computer to perform the method MET when executed on the computer.

[0048] Accordingly, the present invention addresses one or more of the foregoing needs by first propagating the automated decision back to a relevance score within the input image space. A simple example of such a relevance score would be a measure of sensitivity in a deterministic setting or an entropy difference in a probabilistic setting. Heatmaps can be used to visualize, convey, record, and understand the decision, and then to validate, reshape, or reject it, or most importantly to provide a way of feedback to the clinician - see Figure 3 .

[0049] The latter is achieved by image manipulation that reduces or increases the influence of certain image features on the decision - see Figure 4 . Using relevance backpropagation, the decision focus in the image domain can be identified, with the bottom image having a red focus. Although the right focus is a correctly detected calcium patch, the left focus has been incorrectly detected and wrongly drives the calcium score to be high. For focus manipulation via the feedback interface, the user can modify the input based on this relevance, such as intensity modulation or erasing image features, and rerun the analysis with the redirected focus, potentially deriving a different overall estimate - Figure 3 Adapted from reference document [1].

[0050] Main components:

[0051] Vascular imaging: The input information is generated by one or more vascular imaging methods (such as IVUS, OCT, CCTA, angiography, etc.). This information is provided as an N - dimensional image, which can theoretically be inspected manually or automatically to evaluate the risk for acute coronary events, calcification burden, peripheral vascular diseases, etc.

[0052] Decision support system: The raw data is input into a decision support system, which takes the above - mentioned image information as input and provides a disease risk score and possibly a measure of uncertainty as output. Although the present invention can mainly focus on such high - level support with more complex reasoning, the application scope can also be extended to lower - level systems. An example would be the detection of specific plaque patterns, such as for example the napkin - ring pattern for high - risk coronary plaques, where the system will look internally for specific textures in the data to identify the pattern. Generally, the system acts as a black box, whose behavior has been learned in an earlier data - driven manner.

[0053] Relevance propagation module: Once a decision is received in the output, the relevance propagation module can be triggered to propagate the decision back to the original image space. In an embodiment of the present invention, this will produce an image such as a heatmap in the same format as the input data. Since the decision of the support system is the only input to this module, the heatmap will only contain an indication of which information in which regions has been used to arrive at the decision - see Figure 3Possible implementations for this module are system - specific and presented in reference document [2] or reference document [3].

[0054] Figure 4 An exemplary correlation heatmap is illustrated, which is used to feed information back into the system to potentially correct decisions after image evaluation of re - run pairs of (a - c) IVUS images and (d - f) coronary CTA images. Refer to Figure 4 (a - c), The system is trained on IVUS images to output a risk score based on the overall calcification burden in the vessel (a). The system generates a high - risk score, e.g., 3, and provides a heatmap that highlights the decision - making focus within the image, mainly marked by a very bright segment (b). The user believes that the first segment should not be identified as calcium and thus wrongly increases the score. Therefore, the importance of the segment is reduced by using the heatmap (e.g., reducing the intensity of this focus area). This does result in a lower total score (c). Refer to Figure 4 (d - f), A similar scenario using coronary CTA data is described. The system predicts a high ACS score (d) based on high - risk plaque and calcium features. The system misidentifies the first high - focus segment, which is actually a bifurcation (e). The user decides to mask this area as not part of the region. The masking can be applied to the input data or can be used as an additional input channel to the system, which will inform the system of the user - defined low priority in this area (f).

[0055] Feedback module: An optional feedback module closes the loop to complete the interactive system. Given the heatmap, the feedback module provides an interface for the clinician to manipulate the input. By manipulating the heatmap, the attention focus of the decision - support system can be manipulated. A human observer can downgrade regions that have been wrongly assigned high correlations, e.g., weakening the focus. This can lead to directly masking the input or modulating its intensity, or it can be achieved using an additional input channel of the decision system, which indirectly informs the system of the user - defined correlations of changes in the input image. Examples of false - high focus regions in the plaque scenario are: stents or other implants, which can be interpreted as calcium, abnormally high concentrations of contrast agents, motion, or other artifacts. Similarly, upgrading of certain regions is also possible, e.g., generating a focus. Examples include poorly considered regions with poor contrast, shadows in ultrasound applications, etc.

[0056] After manipulation, the transformed heatmap is used to directly or indirectly re - weight the input and resubmit the input to the decision system.

[0057] The decision - support system is typically implemented by a feed - forward architecture. Alternatively, a (probabilistic) generative model with bidirectional inference constitutes another way for the input to be masked or for modulation to form prior information for a repeated inference process, as discussed in reference document [4].

[0058] For interventional applications, the feedback module interface can be implemented via, for example, touch input, where a clinician can use user-generated commands (such as gestures) to manipulate the heat map and maneuver the focus. Re-focusing and additional user input can also be used to automatically improve the decision support system to avoid long-term similar re-focusing situations.

[0059] The present invention relates to applications involving vascular imaging such as IVUS, CCTA, or OCT. It can be used for interventional and post hoc applications. It is applicable to coronary and peripheral vessels.

[0060] Any of the method steps disclosed herein can be recorded in the form of instructions that, when executed on a processor, cause the processor to perform such method steps. The instructions can be stored on a computer program product. The computer program product can be provided by dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Additionally, the explicit use of the term "processor" or "controller" should not be construed to specifically refer to hardware capable of executing software and can implicitly include, but is not limited to, digital signal processor "DSP" hardware, read-only memory "ROM" for storing software, random access memory "RAM", non-volatile storage, etc. Further, embodiments of the present invention can take the form of a computer program product accessible from a computer-usable or computer-readable storage medium that provides program code, which is used by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable storage medium can be any device that can include, store, communicate, propagate, or transport a program used by or in conjunction with an instruction execution system, apparatus, or device. The medium can be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, or devices or apparatuses, or a propagation medium. Examples of computer-readable media include semiconductor or solid state memories, magnetic tape, removable computer floppy disks, random access memory "RAM", read-only memory "ROM", rigid disks, and optical disks. Current examples of optical disks include compact disk – read-only memory "CD-ROM", compact disk – read / write "CD-R / W", Blu-RayTM, and DVD.

[0061] In summary, a system has been provided for determining the relative importance of each of a plurality of image features of a vascular medical image that affect an overall diagnostic metric, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule. The system includes a medical image database, a rule generation unit, an image providing unit, a diagnostic metric calculation unit, and a decision propagation unit. The medical image database includes a plurality of vascular medical images. The rule generation unit analyzes the plurality of vascular medical images and automatically generates at least one diagnostic rule corresponding to a common diagnosis of a subset of the plurality of vascular medical images based on a plurality of image features common to the subset of vascular medical images. The image providing unit provides a current vascular medical image including a plurality of image features. The diagnostic metric calculation unit calculates an overall diagnostic metric for the current vascular medical image by applying at least one automatically generated diagnostic rule to the current vascular medical image. The decision propagation unit identifies the relative importance of each of the plurality of image features in the current vascular medical image to the calculated overall diagnostic metric.

[0062] Various embodiments and selections have been described relative to system SY, and it should be noted that various embodiments can be combined to achieve further beneficial effects.

Claims

1. A system for determining the relative importance of each of a plurality of image features of a vascular medical image that affect an overall diagnostic metric, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule, the system comprising: A medical image database that includes a plurality of vascular medical images; A rule generation unit configured to analyze the plurality of vascular medical images and automatically generate at least one diagnostic rule corresponding to a common diagnosis of the subset of the plurality of vascular medical images based on a plurality of image features common to the subset of the plurality of vascular medical images; An image providing unit configured to provide a current vascular medical image including the plurality of image features; A diagnostic metric calculation unit configured to calculate an overall diagnostic metric for the current vascular medical image by applying at least one automatically generated diagnostic rule to the current vascular medical image; A decision propagation unit configured to identify the relative importance of each of the plurality of image features in the current vascular medical image to the calculated overall diagnostic metric.

2. The system according to claim 1, wherein The rule generation unit is configured to provide the at least one diagnostic rule by performing a machine learning algorithm, a deep learning algorithm, or an automatic intelligence algorithm.

3. The system according to claim 1, wherein The overall diagnostic metric is a risk score corresponding to the risk of an acute coronary event, a calcification burden, the risk of peripheral vascular disease, the risk of coronary artery plaque, the risk of coronary artery disease, or a stenosis level characterization, such as ACS for the acute coronary event.

4. The system according to claim 1, wherein, The decision propagation unit is configured to identify the relative importance of each of the plurality of image features in the current vascular medical image to the calculated overall diagnostic metric in the form of a heat map.

5. The system according to claim 1, wherein, The diagnostic metric calculation unit is further configured to: receive user input indicating at least one of the plurality of image features from a user input device, and i) recalculate the overall diagnostic metric for the current vascular medical image by changing the relative importance of the at least one of the plurality of image features and reapplying the at least one diagnostic rule to the current vascular medical image or ii) change the relative importance of the at least one of the plurality of image features and cause the rule generation unit to re-analyze the plurality of vascular medical images and automatically generate a revised diagnostic rule corresponding to a common diagnosis of a selection of the plurality of vascular medical images based on a plurality of image features common to the selection of the plurality of vascular medical images, and cause the diagnostic metric calculation unit to calculate a revised overall diagnostic metric for the current vascular medical image by applying the revised automatically generated diagnostic rule to the current vascular medical image, and cause the decision propagation unit to identify the relative importance of each of the plurality of image features in the current vascular medical image to the calculated revised overall diagnostic metric.

6. The system according to claim 5, wherein Changing the relative importance of the at least one image feature among the plurality of image features includes increasing or decreasing or ignoring the relative importance of the at least one image feature among the plurality of image features.

7. The system according to claim 5, wherein The rule generation unit is configured to automatically generate the at least one diagnostic rule by weighting each of the plurality of image features, and wherein changing the relative importance of the at least one image feature among the plurality of image features includes changing the weight of the at least one image feature among the plurality of image features.

8. The system according to claim 5, wherein The user input device is configured to define a region of the current vascular medical image provided by the image providing unit.

9. The system according to claim 5, wherein At least one of the plurality of image features corresponds to one or more image artifacts.

10. The system according to claim 1, wherein, The decision propagation unit is configured to identify, in the current vascular medical image, the relative importance of each of the plurality of image features to the calculated overall diagnostic metric based on: i) the sensitivity of the calculated overall diagnostic metric to changes in each of the plurality of image features; or ii) the change in the entropy of the calculated overall diagnostic metric caused by changes in each of the plurality of image features.

11. The system according to claim 1, wherein, The plurality of image features correspond to one or more of the following: plaque, calcium, presence of ruptured plaque, thrombus in the lesion, presence of fat in the plaque region adjacent to the lumen; and / or include one or more of the following: calculated lumen area, calculated plaque area, calculated lesion size, calculated distribution or proportion of calcium in the lesion.

12. The system according to claim 1, wherein, The plurality of vascular medical images and the current vascular medical image are i) IVUS images, ii) OCT images, iii) coronary CTA images or iv) angiography images.

13. The system according to claim 1, wherein, The diagnostic metric calculation unit is further configured to calculate a measure of the uncertainty of the overall diagnostic metric for the current vascular medical image; and wherein the decision propagation unit is further configured to identify the uncertainty in the current vascular medical image.

14. A method for determining the relative importance of each of a plurality of image features of a vascular medical image that affect an overall diagnostic metric, the overall diagnostic metric being calculated for the image according to an automatically generated diagnostic rule; the method comprising: Analyzing a plurality of vascular medical images and automatically generating at least one diagnostic rule corresponding to a common diagnosis of the subset of the plurality of vascular medical images based on a plurality of image features common to the subset of the plurality of vascular medical images; Providing a current vascular medical image including the plurality of image features; Calculating an overall diagnostic metric for the current vascular medical image by applying at least one automatically generated diagnostic rule to the current vascular medical image; and Identifying, in the current vascular medical image, the relative importance of each of the plurality of image features to the calculated overall diagnostic metric.

15. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method according to claim 14.

Images