Domain-Constrained Regularized Adversarial Samples for Model Interpretability
By masking input regions during backpropagation and using a gradient-based approach to generate a saliency mask, the method enhances model accuracy and explanation quality by focusing on continuous regions, addressing the limitations of conventional techniques.
Patent Information
- Application Number
- JP2022526490
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-11-13
- Filing Date
- 2020-10-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-10-27
AI Technical Summary
Conventional machine learning techniques fail to exclude specific input regions during optimization, leading to inaccurate and incomplete explanations, particularly in cases where certain regions dominate the decision-making process.
The method involves masking portions of input data during backpropagation and iteratively modifying a perturbed version of the input using a gradient of a loss function to generate a saliency mask, focusing on continuous regions over individual elements.
Improves the accuracy and explanatory quality of model outputs by excluding irrelevant input regions, allowing for more precise localization and segmentation of influential features.
Smart Images

Figure 0007586606000001 
Figure 0007586606000002 
Figure 0007586606000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for eliminating parts of input data by ignoring parts of the input during the optimization process by masking them during backpropagation in order to improve the accuracy and explanatory quality of the output of a machine learning model. [Background technology]
[0002] As machine learning algorithms continue to improve, there is an increasing need to explain why a given model generated a certain prediction. The benefits of such explanations fall roughly into two categories: explaining the end result to a user, and analyzing the network by a researcher. Many traditional methods of explainability focus on identifying parts of the input that affect the score that a machine learning process, such as a neural network, assigns to a given class. In this context, it is useful to be able to restrict the choice of domains of inputs that are included in or presented as explanations. One example of such a need is a radiologist analyzing medical images containing findings and hoping to find incidental, e.g., additional, potentially malignant, findings and to exclude major findings from consideration.
[0003] One such use of explainability is the use of unsupervised segmentation, such as identifying regions that contain objects of a given class despite not having access to location information. Often, the segmented regions may contain only a small portion of the object in question, such as only the portion that most strongly influences the decision.
[0004] Conventional techniques for finding parts of inputs that are responsible for increasing the prediction of a given model typically work by minimizing the score on the inputs, subject to additional constraints. However, conventional techniques cannot remove certain regions from consideration. There are cases where a user wants to exclude some of the inputs from the explanation, for example, a radiologist may find a region suspicious and want to analyze the contribution of another region. Another example is when a small region has such a strong contribution that it completely dominates the optimization process, resulting in convergence to only a partial explanation (as is often the case in practice).
[0005] Therefore, there is a need for techniques that provide the ability to exclude parts of the input in order to improve the accuracy and explanation quality of the output. Summary of the Invention
[0006] Embodiments may provide techniques that provide the ability to exclude parts of the input to improve the accuracy and explanation quality of the output. For example, embodiments may ignore parts of the input during the optimization process by masking them during backpropagation. For example, embodiments may incrementally grow the explanation by masking previously discovered explanation regions.
[0007] For example, in one embodiment, a method may be implemented in a computer system including a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, and the method may include receiving, at the computer system, input data and a machine learning model for generating predictions based on the input data; generating, at the computer system, a mask indicating portions of the input data to ignore during backpropagation of the machine learning model; and modifying, at the computer system, the generated mask to improve predictions of the machine learning model.
[0008] In an embodiment, the step of correcting the input data may include, in the computer system, iteratively correcting a perturbed or adversarial version of the input data using a gradient of a loss function with respect to the input data while keeping the machine learning model frozen, where the perturbed or adversarial version of the input data is invariant in a portion defined by the mask, and generating, in the computer system, a saliency mask that approximates the support of the perturbation to smoothly approximate the step function. The step of correcting the mask may further include, in the computer system, expanding the mask by adding the generated saliency mask to the mask to generate a corrected mask. The loss function may be based on a first term that reduces the class value of the class of the machine learning model to a predetermined value, a second term that approximates the size of the support of the perturbation, and a third term that promotes smoothness of the support of the perturbation, and may favor continuous regions of non-zero values over interspersed individual elements. The input data may be an image. The mask may be a portion of pixels of the image.
[0009] In one embodiment, a system may include a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the system comprising: receiving input data and a machine learning model for generating a prediction based on the input data; generating a mask indicating portions of the input data to ignore during backpropagation of the machine learning model; and modifying the generated mask to improve a prediction of the machine learning model.
[0010] In one embodiment, a computer program product may include non-transitory computer readable storage having program instructions embodied therein, the program instructions being executable by a computer system to cause the computer system to perform a method including receiving, at the computer system, input data and a machine learning model for generating predictions based on the input data; generating, at the computer system, a mask indicating portions of the input data to ignore during backpropagation of the machine learning model; and modifying, at the computer system, the generated mask to improve predictions of the machine learning model.
[0011] The details of the present invention, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numbers and designations refer to like elements and in which: [Brief description of the drawings]
[0012] [Figure 1] 1 shows an example block diagram of a system in which embodiments of the present systems and methods may be implemented. [Diagram 2] FIG. 2 is an exemplary flow diagram of one embodiment of a process for explaining or interpreting a “masked” model, according to embodiments of the present system and method. [Diagram 3] FIG. 2 is an exemplary flow diagram of one embodiment of a process for unsupervised segmentation according to an embodiment of the present systems and methods; [Figure 4] FIG. 1 is an exemplary block diagram of a computer system capable of implementing the processes included in the embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Embodiments may provide techniques that provide the ability to exclude parts of the input to improve the accuracy and explanatory quality of the output. For example, embodiments may ignore parts of the input during the optimization process by masking them during backpropagation. For example, embodiments may incrementally grow the explanation by masking previously discovered explanation regions.
[0014] An exemplary block diagram of a system 100 is shown in FIG. 1. In this example, input data 102, such as one or more images, may include multiple features 104, 106. In this example, portions of the input data 102 that include the feature 106 may be excluded to improve the accuracy and explanation quality of the output. A mask 108 may be used to exclude portions of the input data 102 that include the feature 106, and thus include portions of the input data 102 that include the feature 108. The input data 102 is input to the model to generate predictions based on the input data 102. A mask generation 112 may perform an optimization process by masking portions of the input data 102 using the mask 108 during backpropagation. An embodiment may, for example, incrementally grow explanations by masking previously discovered explanation regions.
[0015] The embodiments can provide explanations for the model (network) decisions as the parts of the inputs that most influence the decision. For example, in the case of an image, the regions of the image that most influence the decision on the image. In the case of vector inputs, the subset of vectors that most influence the decision on the vector. The embodiments can utilize mechanisms to exclude parts of the inputs from the explanation. The embodiments can be applied to use cases for such mechanisms. For example, the embodiments can allow a user to exclude regions that are obvious to them. Similarly, the embodiments can improve unsupervised segmentation based on explainability by incrementally growing an explanation mask, filtering out previously found explanations and augmenting these with newly discovered ones. Additionally or alternatively, modifying or growing the explanation mask can improve the localization of objects classified by the learning model.
[0016] FIG. 2 illustrates an exemplary flow diagram of one embodiment of a process 200 for “masked” model explanation or interpretation. In this example, inputs to the process 200 may include an existing “ignored” mask K, a model M, an input image I, and a class C. An image variable Î may represent a perturbed or adversarial version of the image. The process 200 may begin at 202, where an image variable Î may be initialized to the input image I. In step 204, while the model M is frozen, the variable Î may be iteratively modified using the gradient of a loss function E with respect to the input. The loss function E may be defined to include (1) a first term that reduces the class value of class I to a given value s, (2) a second term that approximates the magnitude of the support of the perturbation, and (3) a third term that promotes smoothness of the support of the perturbation, where continuous regions of non-zero values are favored over interspersed individual elements (pixels in the case of images). Embodiments may use any gradient-based optimizer, including, for example, SDG, Adam, AdaDelta, etc.
[0017] In step 206, the image variable I^ may be constrained to remain unchanged across the "ignore" mask, i.e., equal to I in the region defined by K (the "ignore" region). The image variable I^ may be constrained to remain in the range of applicable values from which the input is sampled, for example by clipping its value. If less than a predetermined number of iterations have been performed and convergence has not been reached in step 208, the process 200 may go to 204 to perform additional iterations. If a predetermined number of iterations have been performed or convergence has been achieved in step 208, a saliency mask may be derived in step 210 by thresholding S at zero. The saliency mask defines the region of the image to be used for the description or interpretation of the model. S may be defined as an approximation of the support of the perturbation in a manner that smoothly approximates a step function. S may drop below zero at small values, which may promote sparsity. A perturbation may be defined to be the delta between the original input I and the perturbed or adversarial version in the image variables Î.
[0018] FIG. 3 illustrates an exemplary flow diagram of an embodiment of a process 300 of unsupervised segmentation with explanations or interpretations. In this example, inputs to the process 300 may include a model M, an input image I, and a class C. The process 300 may begin at 302, where an ignore mask K may be set to zero. In step 304, an explanation or explanation may be calculated using the process 200 with inputs K=0, M, I, and C to generate a Masked Explanation, resulting in a saliency mask Ks. In 306, Ks may be added to K according to K:=K ∪ Ks. In step 308, if fewer than a predetermined number of iterations have been performed and convergence has not been reached, the process 300 may proceed to 304 and perform additional iterations of 304 and 306. In step 308, if a predetermined number of iterations have been performed or convergence has been achieved, K may be returned as a new segmentation mask in step 310.
[0019] FIG. 4 illustrates an exemplary block diagram of a computer system 400 in which the processes included in the embodiments described herein may be implemented. The computer system 400 may be implemented in an embedded processor, a system on a chip, a personal computer, a workstation, a server system, a minicomputer, a mainframe computer, or in a distributed, networked computing environment. The computer system 400 may include one or more processors (CPUs) 402A-402N, an input / output circuit 404, a network adapter 406, and a memory 408. The CPUs 402A-402N execute program instructions to implement the functions of the present communication system and method. Typically, the CPUs 402A-402N are one or more microprocessors, such as Intel® Core® processors. 4 illustrates an embodiment in which computer system 400 is implemented as a single multi-processor computer system, where multiple processors 402A-402N share system resources, such as memory 408, input / output circuitry 404, and network adapter 406. However, the present communication systems and methods include embodiments in which computer system 400 is implemented as multiple networked computer systems, which may be a single processor computer system, a multi-processor computer system, or a mixture thereof.
[0020] The input / output circuitry 404 provides the ability to input data to or output data from the computer system 400. For example, the input / output circuitry may include input devices such as a keyboard, mouse, touchpad, trackball, scanner, analog-to-digital converter, etc., output devices such as a video adapter, monitor, printer, modem, etc. The network adapter 406 can interface the device 400 to a network 410. The network 410 can be any public or private LAN or WAN, including but not limited to the Internet.
[0021] Memory 408 stores program instructions that are executed and data that is used and processed by CPU 402 to perform the functions of computer system 400. Memory 408 may include, for example, electrical memory devices such as random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), electrically erasable programmable read only memory (EEPROM), flash memory, and electromechanical memory such as magnetic disk drives, tape drives, optical disk drives, and the like, including an Integrated Drive Electronics (IDE) interface, or IDE variants or extensions such as Enhanced IDE (EIDE), Ultra Direct Memory Access (UDMA), or a Small Computer System Interface (SCSI)-based interface or variants or extensions such as Fast SCSI, Wide SCSI, Fast Wide SCSI, or a Serial Advanced Technology Attachment (SATA), or a variant or extension of SATA, or a Fiber Channel Arbitrated Loop (FC-AL) interface.
[0022] The contents of memory 408 may vary depending on the functions that computer system 400 is programmed to execute. In the example shown in FIG. 4, exemplary memory contents are shown illustrating the routines and data of an embodiment of the process described above. However, one skilled in the art will recognize that these routines, along with the memory contents associated with these routines, need not be contained in one system or device, but rather may be distributed among multiple systems or devices based on well-known engineering considerations. The present systems and methods may include any and all of such arrangements.
[0023] In the example shown in FIG. 4, memory 408 may include masked model routines 412, segmentation routines 414, models 416, input data 418, output masks 420, and an operating system 422. Message collection routines 412 may include software routines for generating masked model saliency masks, which may be included in output masks 420, as described above. Segmentation routines 414 may include software routines for generating unsupervised segmentation masks, which may be included in output masks 420, as described above. Models 416 may include software routines and data for performing machine learning modeling, such as neural networks, as described above. Input data 418 may include data, such as images or other data, that is processed by models 416 using output masks 420, as described above. Operating system 422 may provide overall system functionality.
[0024] As shown in FIG. 4, the communication system and method may include implementation on one or more systems that provide multiprocessor, multitasking, multiprocessing, or multithreaded computing, or a combination thereof, as well as implementation on systems that provide only single processor, single threaded computing. Multiprocessor computing involves performing computing using more than one processor. Multitasking computing involves performing computing using more than one operating system task. A task is an operating system concept that refers to the combination of a program being executed and bookkeeping information used by the operating system. Each time a program is executed, the operating system creates a new task for it. A task is like an envelope for a program, identifying the program with a task number and attaching other bookkeeping information to it. Many operating systems, including Linux, UNIX, OS / 2, and Windows, are capable of executing many tasks simultaneously and are referred to as multitasking operating systems. Multitasking is the ability of an operating system to run more than one executable file at a time. Each executable runs in its own address space, which means that executables have no way to share any of their memory. This has an advantage because it is not possible for a program to harm the execution of any of the other programs running on the system. However, the programs have no way to exchange any information except through the operating system (or by reading files stored in the file system). Multiprocess computing is similar to multitasking computing because the terms task and process are often used interchangeably, but some operating systems distinguish between the two.
[0025] The present invention may be a system, method or computer program product or combination thereof at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention. The computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device.
[0026] The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Illustrative enumeration of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having recorded instructions, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.
[0027] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.
[0028] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented languages such as Smalltalk®, C++ or the like, procedural languages such as the C programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of the invention.
[0029] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.
[0030] These computer readable program instructions may be provided to a general purpose computer, special purpose computer processor or other programmable data processing apparatus to produce a machine such that the instructions, executed via the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in the block or blocks of the flowchart diagrams and / or blocks. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device or combination thereof to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts specified in the block or blocks of the flowchart diagrams and / or blocks.
[0031] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart and / or blocks.
[0032] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special purpose hardware-based system that performs a particular function or action or implements a combination of special purpose hardware and computer instructions.
[0033] Although specific embodiments of the present invention have been described, it will be understood by those skilled in the art that there are other embodiments which are equivalent to those described. It is therefore to be understood that the present invention is not limited to the specific illustrated embodiments, but is limited only by the scope of the appended claims.
Claims
1. 1. A method implemented in a computer system including a processor, a memory accessible by said processor, and computer program instructions stored in said memory and executable by said processor, said method comprising: receiving, at the computer system, input data and a machine learning model for generating predictions based on the input data; generating, at the computer system, a mask indicating portions of input data to ignore during backpropagation of the machine learning model; modifying the generated mask in the computer system to at least one of improve the explanation of the machine learning model or improve the localization of objects classified by the machine learning model; and modifying the mask comprises: while keeping the machine learning model frozen, iteratively modifying, in the computer system, a perturbed or adversarial version of the input data using gradients of a loss function with respect to the input data, where the perturbed or adversarial version of the input data is invariant in portions defined by the mask; generating, in the computer system, a saliency mask that approximates the support of the perturbation to smoothly approximate a step function; A method comprising:
2. The step of modifying the mask includes: extending the mask by adding the generated saliency mask to the mask in the computer system to generate a modified mask. The method of claim 1 further comprising:
3. 3. The method of claim 2, wherein the loss of the loss function is based on a first term that reduces class values of the classes of the machine learning model to a predetermined value, a second term that approximates a size of the support of the perturbation, and a third term that promotes smoothness of the support of the perturbation, favoring continuous regions of non-zero values over scattered individual elements.
4. The method of claim 3 , wherein the input data is an image.
5. The method of claim 4 , wherein the mask is a portion of pixels of the image.
6. 1. A system including a processor, a memory accessible by said processor, and computer program instructions stored in said memory and executable by said processor, comprising: receiving input data and a machine learning model for generating predictions based on the input data; generating a mask indicating portions of input data to ignore during backpropagation of the machine learning model; modifying the generated mask to at least one of improve the explanation of the machine learning model or improve the localization of objects classified by the machine learning model; and modifying the mask comprises: while keeping the machine learning model frozen, iteratively modifying a perturbed or adversarial version of the input data using gradients of a loss function with respect to the input data, where the perturbed or adversarial version of the input data is invariant in portions defined by the mask; generating a saliency mask that approximates the support of the perturbation to smoothly approximate the step function; Including, the system.
7. The step of modifying the mask includes: extending the mask by adding the generated saliency mask to the mask to generate a modified mask. The system of claim 6 further comprising:
8. 8. The system of claim 7, wherein the loss of the loss function is based on a first term that reduces a class value of a class of the machine learning model to a predetermined value, a second term that approximates a size of a support of the perturbation, and a third term that promotes smoothness of the support of the perturbation, favoring continuous regions of non-zero values over scattered individual elements.
9. The system of claim 8 , wherein the input data is an image.
10. The system of claim 9 , wherein the mask is a portion of pixels of the image.
11. A computer program for a computer system, comprising: receiving, at the computer system, input data and a machine learning model for generating predictions based on the input data; generating, at the computer system, a mask indicating portions of input data to ignore during backpropagation of the machine learning model; modifying the generated mask in the computer system to at least one of improve the explanation of the machine learning model or improve the localization of objects classified by the machine learning model; and the step of modifying the mask comprises: while keeping the machine learning model frozen, iteratively modifying, in the computer system, a perturbed or adversarial version of the input data using gradients of a loss function with respect to the input data, where the perturbed or adversarial version of the input data is invariant in portions defined by the mask; generating, in the computer system, a saliency mask that approximates the support of the perturbation to smoothly approximate a step function; A computer program comprising:
12. The step of modifying the mask includes:
12. The computer program product of claim 11, further comprising: expanding the mask by adding the generated saliency mask to the mask in the computer system to generate a modified mask.
13. 13. The computer program product of claim 12, wherein the loss of the loss function is based on a first term that reduces class values of the classes of the machine learning model to a predetermined value, a second term that approximates a size of the support of the perturbation, and a third term that promotes smoothness of the support of the perturbation, favoring continuous regions of non-zero values over scattered individual elements.
14. The computer program product of claim 13 , wherein the input data is an image.
15. The computer program product of claim 14 , wherein the mask is a portion of pixels of the image.
Citation Information
Patent Citations
Advanced analytical infrastructure for machine learning
JP2017004509A
Image processing device, image processing method, image processing program and teacher data creating method
JP2018173814A
System for training networks for semantic segmentation
US20170109625A1
Resolving and steering decision foci in machine learning-based vascular imaging
WO2019170561A1
