Visual explanation of classification, method and system

Abstract

A framework for visually explaining classifications, the framework can train (204) a generative model to generate new images that resemble an input image but are classified by a classifier as belonging to one or more alternative classes, and generate (206) at least one explanation mask by performing an optimization based on the current input image and the new images generated by the trained generative model from the current input image.

Description

[Technical field]

[0001] The present invention relates generally to digital medical data processing, and more particularly to visual interpretation of classifications. [Background technology]

[0002] In recent years, artificial intelligence (AI) systems have dramatically improved their accuracy in a variety of tasks and domains. However, these systems are essentially black boxes, and the increased accuracy comes at the cost of reduced transparency, meaning that these algorithms cannot explain their decisions. Lack of transparency is particularly problematic in the medical field, where humans must be able to understand how decisions are made in order to trust an AI system. Increased transparency would allow human operators to know when they can trust an AI decision and when they should discard it.

[0003] Explainable AI (referred to in the literature as XAI) is an emerging field with many techniques being presented. The goal of XAI is to provide key elements that lead to a classification. These techniques can be classified into the following categories: (1) symbolic, (2) saliency-based, and (3) attention-based. Summary of the Invention [Problem to be solved by the invention]

[0004] Symbolic reasoning systems with built-in explanation capabilities were developed in the 70s through the 90s. However, these systems do not perform well on non-categorical tasks such as image interpretation. Saliency-based methods require the classifier to be differentiable in its output with respect to its input. Numerous techniques have been proposed in the literature, such as guided backpropagation, Grad-CAM, integrated gradient, etc. See, for example, Springenberg, Jost Tobias et al. "Striving for Simplicity: The All Convolutional Net". CoRR abs / 1412.6806 (2015); Selvaraju, RR, Cogswell, M., Das, A. et al. Grad-CAM: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vis 128, 336-359 (2020); and Sundararajan, M., Taly, A., and Yan, Q., "Axiomatic Attribution for Deep Networks", 2017, respectively. Saliency-based methods focus primarily on the influence of the input by computing the derivative of the input with respect to the output of the neural network (NN). A well-trained neural network projects its input onto a low-dimensional manifold and classifies it. However, because images are inherently noisy, the NN may not be able to accurately project the input onto the manifold. Differentiation of the input with respect to the output exacerbates noise, resulting in noisy patterns in the saliency map that are difficult to interpret. This effect is amplified in medical imaging applications, where the number of training samples is typically small and the relative similarity of the samples is high (i.e., they tend to fall outside the manifold).

[0005] Attention-based methods use a trainable attention mechanism added to a neural network to identify relevant locations in the image. See, for example, K. Li, Z. Wu, K. Peng, J. Ernst and Y. Fu, "Tell Me Where to Look: Guided Attention Inference Network," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp.9215-9223. Attention-based methods do not "explain" the classification, but rather point out relevant regions that require further interpretation. This makes them less suitable for medical applications. [Means for solving the problem]

[0006] Described herein is a framework for visually explaining classification. According to one aspect, the framework trains a generative model to generate new images that resemble an input image but are classified by a classifier as belonging to one or more alternative classes. At least one explanation mask can then be generated by performing an optimization based on the current input image and the new images generated by the trained generative model from the current input image.

[0007] A more complete understanding of the present disclosure and many of its attendant aspects will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an exemplary system. [Diagram 2] FIG. 2 illustrates an exemplary method for generating an explanation mask. [Diagram 3] FIG. 3 shows an exemplary cGAN architecture. [Figure 4A] FIG. 4A illustrates an exemplary optimization architecture. [Figure 4B] FIG. 4B illustrates an exemplary process for generating multiple description masks. [Diagram 5] FIG. 5 shows an exemplary comparison of the results. [Figure 6] FIG. 6 shows another exemplary comparison of results. [Figure 7] Figure 7 shows the results produced by our framework. [Figure 8] Figure 8 shows additional results generated by our framework. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following description, numerous specific details are given, such as examples of specific components, devices, methods, etc., to fully understand the implementation of the framework. However, it will be apparent to one skilled in the art that these specific details need not be employed to practice the implementation of the framework. In other instances, well-known materials or methods have not been described in detail to avoid unnecessarily obscuring the implementation of the framework. While the framework is susceptible to various modifications and alternative forms, specific embodiments thereof are illustratively shown in the drawings and described in detail herein. However, it is to be understood that there is no intention to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Furthermore, although certain method steps are separated into separate steps for ease of understanding, these separate separated steps should not be construed as necessarily order-dependent in their execution.

[0010] The term "X-ray image" as used herein may refer to a visible X-ray image (e.g., displayed on a video screen) or a digital representation of an X-ray image (e.g., a file corresponding to the pixel output of an X-ray detector). The term "intra-treatment X-ray image" as used herein may refer to an image captured at any time during the treatment delivery phase of an interventional or therapeutic procedure, which may include times when the radiation source is either on or off. At times, for convenience of explanation, CT imaging data (e.g., cone-beam CT imaging data) may be used herein as an exemplary imaging modality. However, it will be understood that data from any type of imaging modality may also be used in various embodiments, including, but not limited to, X-ray radiographs, MRI, PET (positron emission tomography), PET-CT, SPECT, SPECT-CT, MR-PET, 3D ultrasound images, and the like.

[0011] Unless otherwise noted, as will become apparent from the discussion below, terms such as "partitioning," "generating," "registering," "determining," "aligning," "locating," "processing," "calculating," "selecting," "estimating," "detecting," "tracking," and the like, may refer to operations and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical (e.g., electronic) quantities in the registers or memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system, or other information storage, transmission, or display device. Embodiments of the methods described herein may be implemented using computer software. When written in a programming language that conforms to a recognized standard, instruction sequences designed to implement the methods may be compiled to run on a variety of hardware platforms and for interfacing with a variety of operating systems. Further, implementations of the present framework have not been described with reference to a particular programming language. It will be understood that a variety of programming languages ​​may be used.

[0012] As used herein, the term "image" refers to multi-dimensional data composed of discrete image elements (e.g., pixels in a 2D image, voxels in a 3D image). An image may be, for example, a medical image of a subject acquired by computed tomography, magnetic resonance imaging, ultrasound, or other medical imaging system known to those skilled in the art. Images may also be provided from non-medical contexts, for example, from remote sensing systems, electron microscopes, etc. Images may be provided by a 3D imager, such as a 3D imager, a 3D imager, or a 4D imager. 3 to R function, or R 3 , but the method of the present invention is not limited to such images and can be applied to images of any dimension, for example, a two-dimensional image or a three-dimensional volume. For two- or three-dimensional images, the domain of the image is typically a two- or three-dimensional rectangular array, and each pixel or voxel can be addressed with reference to a set of two or three mutually orthogonal axes. As used herein, the terms "digital" and "digitized" refer to an image or volume in a digital or digitized format, as appropriate, acquired via a digital acquisition system or via conversion from an analog image.

[0013] The terms "pixel" to denote an image element, as conventionally used in connection with 2D imaging and image display, and "voxel" to denote a volumetric image element, as often used in connection with 3D imaging, may be used interchangeably. It should be noted that a 3D volumetric image is itself synthesized from image data obtained as pixels on a 2D sensor array and displayed as a 2D image from an angle. Thus, 2D image processing and image analysis techniques may be applied to the 3D volumetric image data. In the following description, techniques described as manipulating pixels may also be described as manipulating 3D voxel data, which is stored and represented in the form of 2D pixel data for display. Similarly, techniques manipulating voxel data may also be described as manipulating pixels. In the following description, the terms "new input image", "fake image", "output image" and "new image" may be used interchangeably.

[0014] One aspect of the framework provides explanations for anomalies detected by any classifier whose task is to determine normality or anomaly, by training a generative model. The generative model is trained to generate new images that resemble the input image but are classified by the classifier as belonging to one or more alternative classes. The generative model constrains the explanation to remove noise from the explanation mask (or map). The level of noise in the generated explanation mask is advantageously several tens of times lower than existing methods, thereby making the explanation mask easier to interpret. The trained generative model can generate new input images x′, which can be used to understand what the classifier considers to be the class with the highest probability. This is very useful for understanding the bias of the classifier (e.g., if an expert saw t′, would they make the same classification?). These and other features and advantages are described in detail herein.

[0015] FIG. 1 is a block diagram illustrating an exemplary system 100. The system 100 includes a computer system 101 for implementing the framework described herein. In some embodiments, the computer system 101 operates as a stand-alone device. In other implementations, the computer system 101 may be connected (e.g., using a network) to other machines, such as an imaging device 102 or a workstation 103. In a network deployment, the computer system 101 may operate as a server (e.g., a thin-client server), a cloud computing platform, a client user machine in a server-client user network environment, or a peer machine in a peer-to-peer (or distributed) network environment.

[0016] In some embodiments, computer system 101 comprises a processor or central processing unit (CPU) 104 coupled to one or more non-transitory computer-readable media 105 (e.g., computer storage or memory), a display device 110 (e.g., a monitor), and various input devices 111 (e.g., a mouse or keyboard) via an input / output interface 121. Computer system 101 may further include supporting circuits such as cache, power supplies, clock circuits, communications buses, etc. Various other peripheral devices may also be connected to computer system 101, such as additional data storage devices and printing devices.

[0017] The technique can be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof, as part of microinstruction code, or as part of an application program or software product executed via an operating system, or as a combination thereof. In some implementations, the technique described herein is implemented as computer readable program code embodied on a non-transitory computer readable medium 105. In particular, the technique can be implemented by an explanation module 106 and a database 109. The explanation module 106 can include a training unit 102 and an optimizer 103.

[0018] The non-transitory computer readable medium 105 may include random access memory (RAM), read only memory (ROM), magnetic floppy disk, flash memory, and other types of memory, or a combination thereof. The computer readable program code is executed by the CPU 104 to process, for example, medical data acquired from the imaging device 102. In this manner, the computer system 101 is a general-purpose computer system that becomes a special-purpose computer system when the computer readable program code is executed. The computer readable program code is not intended to be limited to a particular programming language and its implementation. It will be understood that a variety of programming languages ​​and coding thereof may be used to implement the teachings of the disclosure contained herein.

[0019] The same or a different computer readable medium 105 may be used to store a database (or data set) 109 (e.g., medical images). Such data may also be stored on an external storage device or other memory. External storage may be implemented using a database management system (DBMS) managed by the CPU 104 and residing on a memory such as a hard disk, RAM, or removable media. External storage may be implemented on one or more additional computer systems. For example, external storage may include a data warehouse system, a cloud platform or system, a picture archiving and communication system (PACS), or other hospital, medical institution, medical office, laboratory, pharmacy, or other medical patient record storage system residing on another computer system.

[0020] The Acquisition Modality 102 acquires medical image data 120 related to at least one patient. Such medical image data 120 may be processed and stored in a database 109. The Acquisition Modality 102 may be a radiology scanner (e.g., X-ray, MR or CT scanner) and / or suitable peripheral devices (e.g., keyboard and display devices) for acquiring, collecting and / or storing such medical image data 120.

[0021] The workstation 103 may include a computer and suitable peripherals, such as a keyboard and a display device, and may operate in conjunction with the overall system 100. For example, the workstation 103 may communicate directly or indirectly with the imager 102 such that medical image data acquired by the imager 102 may be rendered by the workstation 103 and viewed on a display device. The workstation 103 may also provide other types of medical data 122 for a given patient. The workstation 103 may include a graphical user interface for receiving user input via an input device (e.g., a keyboard, a mouse, a touch screen voice or video recognition interface, etc.) for inputting the medical data 122.

[0022] Moreover, it should be understood that some of the constituent system components and method steps depicted in the accompanying figures may be implemented in software, and therefore the actual connections between the system components (or process steps) may vary depending on how the framework is programmed. Given the teachings provided herein, one of ordinary skill in the relevant art will be able to contemplate these and similar embodiments or configurations of the framework.

[0023] 2 illustrates an exemplary method 200 for generating an explanation mask. It should be understood that the steps of method 200 may be performed in the order shown or in a different order. Also, additional, different, or fewer steps may be provided. Furthermore, method 200 may be implemented in system 101 of FIG. 1, a different system, or a combination thereof.

[0024] At 202, the training input images may be medical images obtained directly or indirectly using medical imaging techniques such as high resolution computed tomography (HRCT), magnetic resonance (MR) imaging, computed tomography (CT), helical CT, x-ray, angiography, positron emission tomography (PET), fluoroscopy, ultrasound, single photon emission computed tomography (SPECT), or combinations thereof. The training input images may include normal and abnormal images for evaluating one or more types of diseases. For example, the training input images may include normal and abnormal dopamine transporter scan (DaTscan) SPECT images for evaluating Parkinson's disease. As another example, the training input images may include amyloid-positive and amyloid-negative PET images for evaluating amyloidosis. The abnormal images include at least one abnormality (e.g., abnormal accumulation of alpha-synuclein protein in brain cells, amyloid deposits, lesions), while the normal images do not include any abnormality.

[0025] In some embodiments, the classifier f is a binary classifier that is trained to classify training input images as normal or abnormal images. It should be understood that in other embodiments, the classifier f may be a non-binary classifier. The classifier f takes an input x and returns an output О that represents the classification probability among N classes, where c is the class with the highest probability. The classifier f can be implemented using machine learning techniques, including but not limited to neural networks, decision trees, random forests, and support vector machines, co-evolutionary neural networks, or combinations thereof.

[0026] At 204, the training unit 102 trains a generative model with the training input images to generate new high-quality fake images x′. The generative model is trained to generate new input images x′ that resemble (or are as close as possible to) the training input images x, but are classified by the classifier as belonging to one or more alternative classes (i.e., one or more different classes from the respective training input images). Generative models are a class of statistical models that can generate new data instances. Generative models include the distribution of the data itself, indicating how likely a given example is. In some embodiments, the generative model is a conditional generative model, where the input image x is conditioned to generate the corresponding output image x′. The generative model may be a deep generative model, for example, formed by a combination of a generative model and a deep neural network. Examples of deep generative models include, but are not limited to, variational autoencoders (VAEs), generative inverse networks (GANs), autoregressive models, and the like.

[0027] In one embodiment, the generative model includes a generative adversarial network (GAN). A generative adversarial network (GAN) is a machine learning framework that includes two neural networks—a generator G and a discriminator D—that compete against each other in a minimax game. The generative model may be a conditional GAN. A conditional GAN ​​(cGAN) learns a conditional generative model of data, where an input image x is conditioned to generate a corresponding output image x′ that is used as an input to the discriminator for learning. See, for example, Isola, Phillip et al., “Image-to-Image Translation with Conditional Adversarial Networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017):5967-5976. The goal of the generator G is to generate a fake input x′ that is indistinguishable from the real input x, and the goal of the discriminator D is to recognize the real input from the fake inputs generated by G. The generator G and the discriminator D are sequentially optimized, and the minimax game ideally converges to a generator G that produces high-quality new input images x′ that mimic the distribution of x, while the discriminator D cannot guess whether the new input image x′ is real or fake. Unlike unconditional GANs, in cGANs both G and D observe the input image x.

[0028] FIG. 3 illustrates an exemplary cGAN architecture 300. The cGAN includes a generator 302 and a classifier 304. The generator 302 can be trained to take in as input image x (306) and generate a new input image x′ (308) that is as close as possible to the input image x (306) but with the constraint of being classified into an alternative class by a classifier f. For example, if the input image x (306) is classified as “abnormal” by the classifier, the generator 302 can be trained to generate a new image x′ (308) to be a “normal” image and closely resemble the input image x (306). As another example, if the input image x (306) is classified as “normal” by the classifier, the generator 302 can be trained to generate a new image x′ (308) to be an “abnormal” image that closely resembles the input image x (306). The classifier 304 can be trained to recognize the real input image (310) from the fake input image x′ (308) generated by the generator 302.

[0029] The objective of cGAN can be formulated as follows: JPEG0007673209000001.jpg27161Here, x is the observed input image, c is the most probable class, G(x,c) is the new image x′ in G, D(x,c) is the output of D, and Loss cGAN (G,D) is the loss function of G and D. G is the loss function Loss against an adversarial D that tries to maximize the loss function. cGAN We try to minimize (G,D). Loss function cGAN (G,D) is the expected value E x,c [log D(x,c)] and E x,c It is the sum of [log(1-D(G(x,c),c)], where x and c are sampled from the possible images and classes, respectively.

[0030] In another embodiment, the cGAN objective is formulated as follows: JPEG0007673209000002.jpg40161where x is the observed input image, c is the class with the highest probability, G(x,c) is the new image x′ in G, D(x,c) is the output of D, and LosscGAN (G,D) is the loss function of G and D, α is a parameter, and L1(G) is the distance between the observed input image x and the new image x′ generated by G (i.e., x′=G(x,c)). In this case, the task of D remains the same, but G is trained not only to fool D, but also to be close to the ground truth output in the sense of L1. In other words, G is penalized if it generates a new image x′ that is different (or dissimilar) to the input image x.

[0031] In yet another embodiment, the cGAN objective is formulated as follows: JPEG0007673209000003.jpg37161where x is the observed input image, c is the class with the highest probability, G(x,c) is the new image x′ in G, D(x,c) is the output of D, and Loss cGAN (G,D) is the loss function of G and D, α and β are parameters, f is the classification function (or classifier), L1(G) is the distance between the observed input x and the new image x′ generated by G (i.e., x′=G(x,c)), and L is a loss term that penalizes the generator G if it generates an image that is recognized by the classifier as belonging to an incorrect class. Exemplary values ​​of parameters α and β are, for example, 0.0002 and (0.5, 0.999), respectively. In the objective function (6), the disease classifier is linked to the generator 302 by a loss term L such that the generator 302 is penalized if it generates a new image that is deemed by the classifier to belong to an incorrect class. For example, if G tries to generate a “normal” image from an “abnormal” input image, and the disease classifier classifies this generated new image as “abnormal” (instead of normal), L is assigned a non-zero value to penalize G.

[0032] Training of the cGAN may be performed using various techniques, such as conditional autoencoders, conditional variational autoencoders, and / or other GAN variations. Additionally, the trained cGAN may be optimized using, for example, the ADAM optimizer - an adaptive gradient descent algorithm (ADAM). See, for example, Kingma, DP and Ba, J. (2014), Adam: A Method for Stochastic Optimization. Other types of optimization algorithms may also be used.

[0033] The trained generator G can generate new input images x′ that can be used to understand what the classifier considers to be class c. This can be very useful to understand the bias of the classifier (e.g., would an expert see x′ and make the same classification) and also, from a designer's perspective, to ensure that the classifier is properly trained and mimics what an expert knows about the disease.

[0034] Returning to FIG. 2, in 206, the optimizer 103 generates an explanation mask by performing an optimization based on the current input image x and a new image x′ generated by the trained generative model from the current input image. The current input image x may be acquired from the patient by the imaging device 102, for example, using the same modality (e.g., SPECT scanner or PET scanner) as that used to acquire the training image. The explanation mask is then generated by performing an optimization based on the current input image x and the new image x′ to reduce the classification probability of the classifier to a predetermined value. The explanation mask represents voxels that need to be changed in the current input image x to change the decision of the classifier, so that these voxels may explain the classification by the classifier. Each value in the explanation mask represents a blending coefficient between x and x′.

[0035] FIG. 4A illustrates an exemplary optimization architecture 400. A new image x′ (408) is generated by G, a trained cGAN, based on an observed input image x (404) and a class c. The mask (401) represents the portion of the input image x (404) that is blended with the new input image x′ (408). The classification function f (402) takes as input the combination of the input image x (404) and the new fake image x′ (408) blended by the mask (401) and returns an output O representing the classification probability among N classes, where c is the most probable class. The gradient of the mask (401) with respect to the classifier output O is constrained to be a blend of x and x′, while x′ is designed to be similar to x, thus limiting unrealistic noise sources and increasing robustness to noise.

[0036] The optimization tries to find a smaller Mask′ that reduces the classifier probability of class c to 1 / N probability. The optimization problem may be formulated as follows: JPEG0007673209000004.jpg39161α (e.g., 0.05) represents a scaling factor to control the sparsity of the description mask (411), JPEG0007673209000005.jpg98 represents the element-by-element product of two terms. The combined input x" represents the sum of the current input image x and the new image x' blended with the previous mask. Optimization is performed over the combined input x" that minimizes the probability of class c of the classifier (402). Thus, by construction, the combined input x" is in the same domain as the input x (404) and can be interpreted as such.

[0037] The optimization can be implemented using a backpropagation algorithm by computing the partial derivative of the classifier output O(c) with respect to the mask (401). See, for example, Le Cun Y. (1986), "Learning Process in an Asymmetric Threshold Network", Disordered Systems and Biological Organization, NATO ASI Series (Series F: Computer and Systems Sciences), vol. 20. Springer, Berlin, Heidelberg. The optimization stops when the classifier f(x")(c) reaches a certain probability (e.g., 1 / N), because the mask may be noisy and 1 / N has been found to be a good compromise between noise and explanation. In some embodiments, backpropagation is applied up to 200 times with a learning rate of 0.1.

[0038] The framework can be extended to support multiple modes of normality or anomalies. This can be done by sampling multiple new input images x′ from the generator G and aggregating the explanation masks for each x′ generated by the architecture 400. Multiple explanation masks may be generated from a single current input image x. FIG. 4B shows an exemplary process 410 for generating multiple explanation masks. Multiple distinct explanation masks 412 can be generated by passing the same single input image x (414) through the trained generative model multiple times (e.g., 100 times) to generate multiple new input images x′, which are then passed through the architecture 400 to generate multiple explanation masks 412. Multiple distinct explanation masks 412 may be optionally aggregated to improve the robustness of the explanations. Aggregation can be performed, for example, by averaging or clustering the explanation masks.

[0039] As shown in the exemplary process 410, clustering may be performed to generate multiple clusters 416a-b. Although only two clusters (cluster 1 and cluster 2) are shown, it should be understood that other numbers of clusters may be generated. The different clusters may represent, for example, different pathologies or other abnormalities. The clustering algorithm may include, for example, density-based spatial clustering of noisy applications (DBSCAN) or other suitable techniques. A representative explanatory mask may be selected for each cluster (e.g., cluster centers) and presented to the user. The size of the cluster may be used to order the explanatory masks for the user or to characterize the importance of the explanation.

[0040] Returning to FIG. 2, at 208, the explanation module 106 presents an explanation mask. The explanation mask is displayed, for example, in a graphical user interface displayed on the workstation 103. The explanation mask provides a visual explanation of the classification (e.g., anomaly classification) generated by the classifier. The level of noise in the explanation mask is advantageously much lower than the level of noise generated by existing methods, thereby making the interpretation of the explanation mask more straightforward. Additionally, new input images x′ generated by the trained generative model can also be displayed in the graphical user interface. The new input images x′ can be used to understand what the classifier considers to be the class with the highest probability.

[0041] The framework was implemented in the context of Parkinson's disease. A classifier was trained on DaTscan images to classify normal and abnormal images. The classifier was trained on 1356 images and tested on 148 images, achieving an accuracy of 97% on the test data.

[0042] FIG. 5 shows an exemplary comparison of results obtained using different conventional algorithms and the inventive framework to explain the classification of a classifier. Conventional algorithms include Grad-CAM, backpropagation, guided backpropagation, and integrated gradient algorithms. Column 502 shows a randomly selected abnormal input DaTscan image. Columns 504, 506, 508, and 510 display explanation maps generated using standard algorithms based on the input image in column 502. Column 512 displays the explanation map generated by the inventive framework. The explanation map generated by the conventional method shows extreme noise that makes it difficult to interpret. In contrast, the explanation map generated by the inventive framework is easy to interpret because it has very little noise.

[0043] 6 shows another exemplary comparison of results obtained using different conventional algorithms and the inventive framework to explain the classification of a classifier. Columns 604, 606, 608 and 610 display explanation maps generated based on the input images in column 602 using conventional algorithms. Column 612 displays explanation maps generated by the inventive framework. Compared to the inventive framework, the conventional methods generate explanation maps that exhibit more extreme noise, making them more difficult to interpret.

[0044] FIG. 7 shows the results generated by the framework. Column 702 shows an input DaTscan image x from the test data. Column 706 shows the explanation mask generated by the framework. Column 704 overlays the input image x with the explanation mask to allow better visualization of the spatial pattern relative to the input image. The pattern of the computed mask correlates extremely well with the asymmetry or bilateral reduction in putamen uptake, providing a reasonable explanation for why these scans were classified as abnormal. Column 712 shows new input images x′ for the trained cGAN that transform the abnormal input image 702 into a normal image that closely matches the input image 702. Finally, column 710 shows the combined input x″, which represents the sum of the current input image x and the new image x′ blended with the mask. These images x″ closely match the input image 702, reducing the classifier's probability to below 50%.

[0045] FIG. 8 shows additional results generated by the framework. Column 802 shows an input DaTscan image x from the test data. Column 806 is the explanation mask generated by the framework. Column 804 overlays the input image x with the explanation mask to allow better visualization of the spatial pattern relative to the input image. The pattern of the computed mask correlates extremely well with the asymmetry or bilateral reduction in putamen uptake, providing a reasonable explanation for why these scans were classified as abnormal. Column 812 shows new input images x′ of the trained cGAN that transform the abnormal input image 802 into a normal image that closely matches the input image 802. Finally, column 810 shows the combined input x″, which represents the sum of the current input image x and the new image x′ blended by the mask. These images x″ closely match the input image 802, reducing the probability of the classifier to below 50%.

[0046] Although the framework has been described in detail with reference to exemplary embodiments, those skilled in the art will appreciate that various modifications and substitutions can be made without departing from the spirit and scope of the invention as set forth in the appended claims. For example, elements and / or features of different exemplary embodiments can be combined with each other and / or substituted for each other within the scope of this disclosure and the appended claims.

Claims

1. Description: One or more non-transitory computer-readable media embodying a program of instructions executable by a machine to perform operations for mask generation, comprising: The operation is receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by performing an optimization based on a current input image and a new image generated by the trained generative model from the current input image; and presenting the description mask; Each value of the explanation mask represents a mixing factor between the current input image and the new image generated from the current input image by the trained generative model. Non-transitory computer-readable medium.

2. One or more non-transitory computer-readable media embodying a program of instructions executable by a machine to perform operations for generating a mask, comprising: The operation is receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating a plurality of different explanation masks by performing optimization based on a current input image and a new image generated by the trained generative model from the current input image; aggregating the plurality of different explanation masks; and presenting the aggregated explanation mask; A non-transitory computer readable medium comprising:

3. One or more non-transitory computer-readable media embodying a program of instructions executable by a machine to perform operations for generating a mask, comprising: The operation is receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by optimizing a mixture of a current input image and a new image generated from the current input image by the trained generative model; and presenting said explanation mask; A non-transitory computer readable medium comprising:

4. 1. A system comprising: a non-transitory storage device for storing computer readable program code; a processor in communication with the storage device; The processor, using the computer readable program code, receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by performing an optimization based on a current input image and a new image generated by the trained generative model from the current input image; and presenting said explanation mask; and operable to perform operations including: Each value of the explanation mask represents a mixing factor between the current input image and the new image generated from the current input image by the trained generative model.

5. A system comprising: a non-transitory storage device for storing computer readable program code; a processor in communication with the storage device; The processor, using the computer readable program code, receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating a plurality of different explanation masks by performing optimization based on a current input image and a new image generated by the trained generative model from the current input image; aggregating the plurality of different explanation masks; and presenting the aggregated explanation mask; The system is operable to perform operations including:

6. A system comprising: a non-transitory storage device for storing computer readable program code; a processor in communication with the storage device; The processor, using the computer readable program code, receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by optimizing a mixture of a current input image and a new image generated from the current input image by the trained generative model; and presenting said explanation mask; The system is operable to perform operations including:

7. The system of claim 4 , wherein the classifier comprises a binary classifier trained to classify the training input images as normal or abnormal images.

8. 5. The system of claim 4, wherein the processor is operative to train a deep generative model by training a generative adversarial network (GAN) with the computer readable program code.

9. 5. The system of claim 4, wherein the processor is operative to train a deep generative model by training a conditional generative adversarial network (cGAN) with the computer readable program code.

10. 5. The system of claim 4, wherein the processor is operative with the computer readable program code to train the generative model by penalizing the generative model in response to the generative model generating the new image deemed by the classifier to belong to an incorrect class.

11. 5. The system of claim 4, wherein the processor is operative with the computer readable program code to train the generative model by penalizing the generative model in response to the generative model generating the new images that are dissimilar to the training input images.

12. 5. The system of claim 4, wherein the processor is operative with the computer readable program code to train the generative model in response to receiving a first input image classified by the classifier as “abnormal” by training the generative model to generate a first new image that is “normal” and resembles the first input image.

13. 5. The system of claim 4, wherein the processor is operative with the computer readable program code to train the generative model by training the generative model to generate second new images that are “abnormal” and similar to the second input image, in response to receiving a second input image classified as “normal” by the classifier.

14. The system of claim 5 , wherein the processor is operative with the computer readable program code to aggregate the plurality of different explanation masks by performing a clustering technique.

15. 5. The system of claim 4, wherein the processor is operative with the computer readable program code to generate the at least one explanation mask by performing the optimization to reduce a classification probability of the classifier to a predetermined value.

16. 16. The system of claim 15, wherein the predetermined value comprises 1 / N, where N is a total number of classes determined by the classifier.

17. The system of claim 15 , wherein the processor is operative with the computer readable program code to perform the optimization by finding a smaller mask that reduces the classification probability.

18. 1. A method comprising: receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by performing an optimization based on a current input image and a new image generated by the trained generative model from the current input image; and presenting the description mask; Each value of the explanation mask represents a mixing factor between the current input image and the new image generated from the current input image by the trained generative model. method.

19. A method comprising: receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating a plurality of different explanation masks by performing optimization based on a current input image and a new image generated by the trained generative model from the current input image; aggregating the plurality of different explanation masks; and presenting the aggregated explanation mask; The method includes:

20. A method comprising: receiving a training input image and a classifier; training a generative model based on the training input images to generate new images that resemble the training input images but are classified by the classifier as belonging to one or more alternative classes; generating at least one explanation mask by optimizing a mixture of a current input image and a new image generated from the current input image by the trained generative model; and presenting said explanation mask; The method includes:

Images