Deep neural network visualization

Abstract

Aspects and embodiments relate to a method, computer program product, and apparatus configured to perform the method of providing a representation of features identified by a deep neural network as being relevant to an outcome. The method includes providing a training library for the deep neural network, the training library including a plurality of samples associated with an outcome; training the deep neural network to identify features in the plurality of samples associated with the outcome; creating a feature identification library from an input library by identifying, in each sample of the plurality of samples in the input library, one or more elements that trigger the deep neural network to identify the features; synthesizing, using the feature identification library, a synthesized plurality of one or more elements for a sample having a characteristic that triggers the deep neural network to identify the features; and providing, using the synthesized plurality of one or more elements, a representation of the features identified by the deep neural network in the plurality of samples associated with the outcome. Thus, instead of visualizing a single instance of one or more elements in a sample that trigger a feature associated with an outcome, a series of samples that include elements that would trigger the feature associated with the outcome can be visualized, enabling a more comprehensive understanding of the operation of the deep neural network with respect to the particular feature.

Description

Technical Field

[0001] This invention relates to the field of neural networks, and more particularly, to providing representations of features identified by deep neural networks. Background Technology

[0002] Deep learning methods can be used to generate or identify useful high-level features from very low-level data, such as the color values ​​of individual pixels in medical images. Deep learning methods reduce the need for manually creating such useful high-level features and allow for the automatic definition of "unknown" features. Features generated or learned through deep learning methods are defined via a large number of weights in a deep neural network.

[0003] Although deep neural networks configured to make decisions or label outcomes based on a set of input samples generally perform well, they appear as black boxes to the user. That is, it is difficult to explain or visualize how a deep neural network works and why it operates to make the decisions it makes.

[0004] There are techniques for visualizing regions (or patches) in large sample or example images that activate generated or learned features (i.e., indicate the presence of features in the image). One such technique is called Grad-CAM (Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization [Selvaraju et al, arXiv:1610.02391]), which provides a way to highlight pixels in an input image that help, for example, image classification. In one example, when an image is classified as “dog” by a deep neural network, pixels in the image corresponding to one or more features that help the deep neural network classify the image as “dog” are highlighted. Grad-CAM can be applied to image analysis using deep convolutional neural networks. This technique allows users to examine specific image examples that trigger the network to recognize features. Summary of the Invention

[0005] A first aspect of the invention relates to a method for providing a representation of features identified by a deep neural network as being relevant to an outcome, the method comprising: providing a training library for the deep neural network, the training library including a plurality of samples associated with the outcome; training the deep neural network to identify features in the plurality of samples associated with the outcome; creating a feature recognition library from the input library by identifying one or more elements in each of the plurality of samples in the input library that trigger the deep neural network to recognize features; using the feature recognition library to synthesize a plurality of one or more elements of samples having properties that trigger the deep neural network to recognize features; and using the synthesized plurality of one or more elements to provide a representation of the features identified by the deep neural network in the plurality of samples associated with the outcome.

[0006] A second aspect of the invention provides a computer program product that, when executed on a computer, is operable to perform the method of the first aspect.

[0007] A third aspect of the invention provides an apparatus for providing a representation of features identified by a deep neural network as being related to a result. The apparatus includes: a deep neural network trained with a training library comprising a plurality of samples associated with the result, and the deep neural network being trained and configured to recognize features associated with the result in the plurality of samples; library creation logic configured to create a feature recognition library from the input library by identifying one or more elements in each of the plurality of samples forming the input library that trigger the deep neural network to recognize features; synthesis logic configured to synthesize a plurality of one or more elements of samples having the elements that trigger the deep neural network to recognize features using the feature recognition library; and feature visualization logic configured to provide a representation of the features identified by the deep neural network in the plurality of samples associated with the result using the synthesized plurality of one or more elements.

[0008] Further specific and preferred aspects are set forth in the appended independent and dependent claims. Features of the dependent claims may be appropriately combined with features of the independent claims, and may also be combined with features other than those expressly set forth in the claims.

[0009] One technical advantage of embodiments of the present invention may include: a mechanism for improving the model created by a deep neural network. Another advantage of embodiments of the present invention may include: a mechanism for identifying data that may be missing from the training library provided to the deep neural network, relating to features related to the outcome. Yet another advantage of embodiments of the present invention may include: a mechanism for visualizing features identified by different layers in the deep neural network. Another advantage of embodiments of the present invention may include: a mechanism for providing domain experts with features created by the model related to the outcome created by the deep neural network. A further advantage of embodiments of the present invention may include: a mechanism for helping domain experts understand the operation of the deep neural network. More technical advantages may be further described in the description of the embodiments. The foregoing technical advantages(s) are intended only as non-limiting examples. Those skilled in the art will recognize that other technical advantages not explicitly described may also be applied, and different advantages may be applicable to different embodiments.

[0010] When a device feature is described as operable to provide a function, it should be understood that this includes device features that provide the function or are adapted or configured to provide the function. Attached Figure Description

[0011] Now, embodiments of the present invention will be further described with reference to the accompanying drawings, wherein:

[0012] Figure 1a The illustration shows the arrangement of an image processing apparatus according to one embodiment;

[0013] Figure 1b A typical contour plot of a deep neural network used for image classification is illustrated schematically.

[0014] Figure 2a This is a schematic diagram illustrating possible steps of a method according to an illustrative general example of an embodiment of the present invention;

[0015] Figure 2b It is a schematic diagram that illustrates in detail the possible steps of the method in the illustrative example;

[0016] Figure 2c The illustration shows an example of GradCAM output for network features;

[0017] Figure 2d The illustration shows an example of an image portion that can form part of a feature recognition library according to some embodiments; and

[0018] Figure 3 This illustrates a possible example of a feature-exploration graphical user interface. Detailed Implementation

[0019] Before describing specific illustrative examples in detail, a general overview is provided.

[0020] Typically, embodiments of the present invention relate to a method and system that can provide representations of features identified by a deep neural network as relevant to an outcome. The method of the present invention typically includes: providing a training library to a deep neural network, the training library comprising a plurality of samples associated with an outcome; and using the deep neural network to identify features in the plurality of samples associated with the outcome. It is then possible to create a feature recognition library from the input library by identifying one or more elements in each of the plurality of samples in the input library that trigger the deep neural network to recognize features. This feature recognition library can be used to synthesize a plurality of one or more elements of samples having properties that trigger the deep neural network to recognize features. The synthesized plurality of one or more elements can then be used to provide representations of features identified by the deep neural network in the plurality of samples associated with the outcome. This visualization can be used to improve the operation of the deep neural network. For example, features of interest known to be relevant to the outcome but not present in those features identified by the deep neural network can be identified as missing from the training library, which can be updated to include training material containing those missing features and a model created by the improved deep neural network.

[0021] It has been recognized that methods and apparatuses for providing users with representations of features identified by deep neural networks as relevant to outcomes can be useful. That is, providing users with visual or otherwise human-readable or human-understandable indications of one or more features of input data that lead to the identification or labeling of an “outcome” can be a useful means of understanding how deep neural networks operate and, for example, whether the connections or associations established by the deep neural network about the training data are reasonable or correspond to information that may be known about the training data. In other words, representations of features identified by deep neural networks can aid in understanding the models created by deep neural networks.

[0022] Some embodiments of the present invention recognize that one possible way to provide representations of features identified by deep neural networks as relevant to the outcome, and thus to provide human-understandable information to allow the operation of the system to be understood, is to perform a multi-stage process in which various information bases are used.

[0023] For example, a typical approach to training a deep neural network using images could be to provide the deep neural network with a training library that includes multiple samples (images) associated with the results; and to use the deep neural network to identify features in the multiple samples associated with the results. For example, with medical images, the results could be a disease, syndrome, or similar diagnosis, and the features could include, for example, the identification of organs of a particular shape, or a specific image pattern indicating a tissue type.

[0024] In the illustrated example of the invention, after completing the standard deep neural network training process, the obtained information can be further processed. For example, it is possible to create a feature recognition library from the input library for each feature identified by the deep neural network as relevant to the result. The feature recognition library can be created by identifying one or more elements in each of a plurality of samples that trigger the deep neural network to recognize features. In the case of images, elements in the samples can be specific pixels, patches, or combinations of pixels and / or patches from images in the input library. Relevant elements of the input image (rather than the entire input image) form entries in the feature recognition library.

[0025] The feature recognition library itself can then be used as a tool to enhance the understanding of deep neural network operations. For each feature, multiple elements of a sample can be synthesized using the relevant feature recognition library, these elements possessing properties that trigger the deep neural network to recognize that feature. One implementation approach could be to use a GAN based on the feature recognition library. Conceptually, a properly trained GAN (or similar model) allows users to explore a set of parameters representing a sample that will trigger the deep neural network to recognize features.

[0026] Finally, in some illustrative examples, it should be understood that it may be useful for users to use synthesized multiple elements (one or more) to provide representations of features identified by a deep neural network across multiple samples related to the outcome. In particular, users can explore a range of parameters characterizing the samples that will trigger the deep neural network to recognize features.

[0027] Some embodiments of the present invention recognize that a more comprehensive understanding of the operation of a deep neural network with respect to a particular feature can be obtained by visualizing a series of samples that include elements that trigger features associated with the result, rather than by visualizing a single instance of one or more elements in the samples that trigger features associated with the result.

[0028] Some embodiments of the present invention recognize that visualization tools that allow experts in artificial intelligence (AI) and / or application fields (e.g., medicine) to analyze features identified by deep neural networks as relevant to outcomes or decisions can be helpful. In particular, visualization tools can help cultivate user trust in the operation of deep neural networks, allowing the identification or generation of features to be recognized or expressed in general terms, such as medical terminology related to medical imaging. Furthermore, visualization tools may allow some features identified by deep neural networks to be discarded as unwanted features, such as overfitting or hospital-specific features, and allow for correction of those features within the system, or appropriate modification of the training data to consider or otherwise remove those “unwanted features,” thereby providing a more accurate and precise system for feature identification.

[0029] Visualization tools according to some embodiments of the invention can help recognize that features identified by deep neural networks can provide new insights for improved understanding. It should also be understood that if feature visualization is effective, domain experts can gain a more comprehensive understanding of how deep neural networks operate and, for example, identify any possible "missing" features. Input from the domain expert can then be used to expand the training examples given to the neural network, thereby allowing the deep neural network to perform improved generalization.

[0030] Some embodiments of the present invention may provide representations or visualizations of features identified by a deep neural network as relevant to an outcome. This representation may include human-interpretable representations. For example, a visualization may include one or more images comprising one or more elements that can be successfully interpreted by humans. Since deep neural networks can also be used, for example, to classify samples other than images (e.g., trajectory signals, sounds, etc.), the visualization or representation may include one or more elements of such human-interpretable trajectories or sounds. The method may include providing a deep neural network with a training library comprising multiple samples associated with an outcome. The method may include several steps of using the deep neural network to identify features in the multiple samples associated with the outcome. Thus, features relevant to the outcome are constructed by the deep neural network, and analysis of those features can be performed. The method may include the step of creating a feature recognition library from an input library, which may include a training library, a combination of portions of the training library and some domain samples, a subset of the training library, or a set of samples in the same domain as those samples in the training library. The feature recognition library may be created by identifying one or more elements in each of the multiple samples in the input library that trigger the deep neural network to recognize features. This method may include the step of synthesizing one or more elements of a sample using a feature recognition library, these elements having properties that trigger a deep neural network to recognize features. The synthesis process can enable the generation of a series of one or more elements for any specific sample in the domain, each element having properties that trigger a deep neural network to recognize features. This method can allow the use of the synthesized one or more elements to provide a representation of features identified by a deep neural network in multiple samples associated with the result.

[0031] According to one embodiment of the invention, the representation of a feature includes: a continuous library of synthetic examples that possess properties that trigger a deep neural network to recognize the feature. In other words, it is possible to create multiple synthetic examples that are infinitely variable or smooth within an identified range, which still triggers the deep neural network to recognize a given feature. This feature allows domain experts to understand the full range of visual features, such as those in an image, that trigger the neural network to recognize the feature. Imaging a continuous range can better help domain experts understand the parameters of features, such as spurs or tissue abnormalities, that will trigger the deep neural network to recognize the feature.

[0032] According to one embodiment of the present invention, identifying one or more elements in each of a plurality of samples that trigger feature recognition by a deep neural network comprises: using a gradient-based method to identify one or more elements in each of the plurality of samples that trigger feature recognition. This gradient-based method allows the generation of heatmaps or weight maps of the elements of the samples, which indicate which parts of the samples are relevant to or "critical" to the deep neural network's recognition of a given feature that is relevant to the outcome.

[0033] According to one embodiment of the present invention, gradient-based methods include one or more of the following: Grad-CAM, CAM, deconvolution, or guided back-propagation. As described above, Grad-CAM provides a method to highlight pixels in an input image that are helpful, for example, for image classification. CAM, deconvolution, and guided back-propagation all include similar methods for identifying elements in a sample that are related to the features identified by a deep neural network in that sample. Using those methods allows the system to focus only on those portions or elements of the image that are related to the features that trigger the recognition and are relevant to the result.

[0034] According to one embodiment of the invention, entries in the feature recognition library include a combination of one or more elements of samples in the input library and an indication obtained from a gradient-based method, indicating the relative importance of one or more elements of the samples in the input library when triggering a deep neural network to recognize features. This allows the feature recognition library to be formed only from those portions of the samples in the input library that are relevant to the triggered feature. Therefore, the feature recognition library associated with a given feature can be a more concise version of the complete training library. This more concise library can facilitate efficient data processing and allows for more effective extraction of relevant information about the recognized features associated with the results.

[0035] According to one embodiment of the invention, a plurality of one or more elements of a synthetic sample having characteristics that trigger deep neural network recognition of features include: using a generative modeling method guided by a feature recognition library. This method helps ensure that entries in the feature recognition library are intelligently used by the generative modeling method to create or simulate input samples that trigger deep neural network recognition of features relevant to the outcome. The ability to generate portions of “fake” images that trigger the system to recognize features allows the system operation to be understood and allows for system improvement. For example, if the system identifies a feature as relevant to the outcome, generating a fake version of the image (partially) that triggers the recognition of the feature can allow analysis of whether the feature has a true relevance to the outcome as assessed by domain experts. Fake or “generated” images can identify aspects of the image previously unknown to domain experts as relevant to the outcome, and this information can be used to improve or enhance the recognition of that feature for other systems seeking to report or identify specific outcomes. Fake or “generated” images can identify aspects of the image to domain experts that can be evaluated as artifacts captured by images in the training library and therefore irrelevant to the outcome, and the training library can be updated to remove images containing such artifacts, thereby providing a more accurate and precise system.

[0036] According to one embodiment of the invention, the generative modeling method includes one or more of the following: generative adversarial networks or variational autoencoders. One technique that can be used with respect to features identified by deep neural networks is a generalized adversarial network (GAN) (see, for example, Generative Adversarial Networks [Ian J. Goodfellow, Generative adversarial networks, Jun 2014]). GANs typically operate to simultaneously train a “generator” and a “discriminator”, for example, based on multiple example images. The generator operates to transform an input seed from a random number generator into an image of the same size and shape as the example images. In other words, the generator operates to create fake images. The discriminator is optimized to distinguish between “real” images (examples) and “simulated,” “fake,” or “synthetic” images (images created by the generator). The generator is optimized to produce images that “fool” the discriminator, i.e., are mistakenly identified as real images. The result of a trained GAN is a generator that transforms a continuous space of random numbers into a continuous space of images that appear to be images from multiple example images. This method allows the generated “mathematical” results to be transformed into images that domain experts can consider, thereby allowing the assessment of the applicability of the trained system to recognize specific outputs.

[0037] According to one embodiment of the invention, the method includes: selecting features related to the outcome to be represented. Features are linked to nodes in selected layers of a deep neural network. A sample is said to contain a corresponding feature when a node is activated (i.e., outputs a high value) in response to a given sample input. Typically, the selected layer from which the features will be represented is one of the last layers before the neural network output, because such a layer contains the highest-level features that are combined into the model's decision or output. However, it should be understood that deeper layers can be selected depending on the visualization needs. The layer selection can be changed after viewing the initial visualization. For example, if the initial visualization shows feature levels that are too high / too low, deeper / shallower layers can be selected. In some examples, all features or nodes in a layer can be represented; in some arrangements, as an alternative to all features or nodes in the selected layers for visualization, the user can select specific nodes in the network that can be in different layers. Selecting higher-level layers allows for the generation of larger and more complex "features" to evaluate the applicability of the system.

[0038] According to one embodiment of the present invention, the method further includes: using a deep neural network to identify candidate features in a plurality of samples associated with the result; creating a candidate feature recognition library from the input library by identifying one or more elements in each of the plurality of samples in the input library that trigger the deep neural network to identify candidate features; using the candidate feature recognition library to synthesize a plurality of one or more elements of a sample having properties that trigger the deep neural network to identify candidate features; and using the synthesized plurality of one or more elements to provide a representation of the candidate features identified by the deep neural network in the plurality of samples associated with the result. Therefore, as mentioned above, more than one feature constructed by the deep neural network can be visualized. In some examples, all features or nodes in a layer can be represented; in some arrangements, as an alternative to visualizing all features or nodes in a selected layer, the user can select specific nodes in the network that may be located in different layers. This approach allows for a method in which the system can identify results from input that is transparent to the user.

[0039] According to one embodiment of the invention, candidate features identified by a deep neural network are selected such that the difference between one or more elements triggering the deep neural network to identify the feature and the candidate features is maximized in each of a plurality of samples. Therefore, features represented or visualized by means of the method of the first aspect can be identified as “distinctive” by a user attempting to understand the operation of the deep neural network. In other words, several steps can be taken to try to keep the feature recognition library operating on the distinctive “features” identified by the network. In some embodiments, using a deep neural network to identify outcome-related features based on a training example set may involve several steps to ensure that the identified features are distinguishable from each other.

[0040] According to one embodiment of the invention, the sample includes: an image, sound, or a signal trace. Such an image may include medical imaging images, such as X-ray images, MRI images, ultrasound images, etc.; sound may include appropriate medical records; and the signal trace may include a trace such as an EEG or ECG trace. Each sample may exhibit sample-specific "features".

[0041] Now let's turn to a more specific illustrative example, where a deep neural network is used for images:

[0042] General Architecture

[0043] Figure 1a The illustration shows an arrangement of an image processing apparatus according to one embodiment. A data processor 10 is provided, coupled to a user display 20. Image data 40 is provided to the data processor 10 in real time or from a storage device 50. The image data can be provided from a wide variety of different sources in any of many different formats. The image data may include individual 2D image frames, 3D image data, audio files, track recordings, etc. It should be understood that any of a variety of different types of image data can be used.

[0044] Machine learning algorithm architecture

[0045] Figure 1b A typical contour plot of a deep neural network used for image classification is illustrated schematically. Embodiments of the present invention utilize a deep neural network (DNN) or model executed on a data processor 10 to perform feature recognition of image data. Figure 1b A schematic diagram of a typical deep neural network used for image classification is shown. Figure 1b The network illustrated in the diagram is such that the input image enters from the left, and the deep neural network creates a model so that subsequent model layers generate more and more abstract features as the image progresses through the deep neural network.

[0046] exist Figure 1b In the example deep neural network 100 shown, a training library of input images 110 can be provided, in this case, a color image of a face. The input layer 120 of the deep neural network can be operated to identify features 130, including local contrast maps. The "hidden" layer 140 of the deep neural network can be operated to identify features 150, which include combinations of local contrast maps representing facial features, such as the nose, eyes, etc. The hidden layer 160 of the deep neural network can be operated to identify features 170, which include combinations of facial features representing the face. The output layer 180 of the deep neural network can be operated to identify specific types of facial outcomes, such as a female face or a child's face, depending on the outcome the deep neural network is trained to recognize.

[0047] It should be understood that the method according to the invention can be implemented such that layers in a deep neural network at a desired level of abstraction can be selected for feature visualization. In other words, in the given example, the invention can be used to visualize facial feature 150, or face 170, or both. Similarly, when implementing the method according to the invention, layer units can be activated for a specific input image.

[0048] An example of a deep neural network that can be used for image classification is the standard convolutional neural network. Assuming the input image has dimensions of 256×256×3 (the final dimension describing the RGB color information of the image pixels (3)), its architecture might be as follows:

[0049]

[0050]

[0051] In this example network, convolutional layers perform convolutions within a 3x3 pixel region around each pixel. The third dimension of a convolutional layer is the number of output features for that layer. Each convolutional layer has a Rectified Linear Unit (ReLU) output activation function. When computing the convolutional output, convolutions around pixels at image boundaries are assumed to have zeros added around the image (the convolution pattern is "same"). Max-pooling layers return the maximum value in a 2x2 pixel block with a stride of 2 in both the horizontal and vertical directions. This model assumes a 5-class classification network, where the softmax layer ensures that the sum of the model's 5 outputs is 1 (output normalization).

[0052] An illustrative example is that image features can be constructed by a deep neural network in a conventional way, i.e., by training a deep network for the specific image analysis task mentioned above before taking any further steps. Figure 2a This is a schematic diagram illustrating the main steps of a method according to an illustrative general example of the present invention.

[0053] The method of this invention typically includes the following steps:

[0054] S1: Provide a deep neural network with a training library as described above, the training library comprising multiple samples associated with the results. The deep neural network is used to identify one or more features in the multiple samples in the training library that are associated with the results of interest.

[0055] S2: Select features identified by the deep neural network as being associated with the results of interest for further investigation. For example, in the example network given above, such features could be selected as the output nodes of layer 12, which number 7. The output nodes of this layer represent the 16 highest-level features of the example network.

[0056] S3: Next, the images are fed into the trained deep neural network. These images can come from a training library or be separate image sets. The images fed into the deep neural network identify those that activate the recognition of the selected features (from step S2). The term "activation" can be defined in various ways: at the most sensitive level, a feature can be considered activated if the corresponding node takes a non-zero value (note that ReLU activation sets all negative outputs to zero). Alternatively, the highest percentile of a node's activation value can be used as the activation threshold, or a node can be defined as activated if its output value is the maximum output value of all nodes in that layer.

[0057] S4: Then, the data processor (see...) Figure 1a The algorithm is configured to identify one or more elements in each of multiple samples fed into the deep neural network that trigger the deep neural network to recognize features relevant to the outcome. Specifically, image elements in the input library that are identified as causing the deep neural network to "activate" and recognize selected features relevant to the outcome are highlighted and identified, for example, by using a gradient-type method.

[0058] A clear explanation of gradient-type methods and their implementations can be found in "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" (RRSelvarajuetal et al., 2017, arXiv:1610.02391v3).

[0059] Figure 2c The illustration shows an example of network features in GradCAM output that appear to be related to "dog". Typically, the GradCAM method is used to take the final output features of a network as examples to be highlighted in the input image. However, the visualization features according to the invention can include "deeper" features. For example, in a network, "having a dog" is one of the features that contributes to the model output / result (e.g., "inhabited house" vs. "abandoned house").

[0060] Figure 2a (A) shows the input image. Figure 2a (B) shows a heatmap indicating the relative contribution of each pixel to activating the selected feature (dog). Figure 2a (C) shows a visualization of “guided GradCAM”, as described by Selvaraju et al. in 2017.

[0061] S5: The system operates to create a "feature recognition library" from the information collected in step S4. This library is a collection of elements in the image that cause the deep neural network to recognize features. These elements can be extracted from the output of the gradient-type method in S4 as follows:

[0062] A feature image size (width and height) smaller than the original image size is selected. A moving window method is used to identify one or more locations where the pixel contributions (as indicated in the heatmap) falling within the window form a local maximum. In effect, a bounding box is drawn around each high-contribution blob within the heatmap. The coordinates of the selected window location are then used to crop the input image to create feature images, which together form a "feature recognition library."

[0063] Figure 2d The image shown is a partial or “cropped” image that can be found as part of a visualization of features recognized by a deep neural network and formed into a feature recognition library configured to classify CT images as “cancer” or “no cancer”.

[0064] S6: The data processor 10 can use the generated feature recognition library, for example, by using appropriate iterative computation techniques, to synthesize a sample of one or more elements that have the properties to trigger a deep neural network to recognize selected features.

[0065] S7: Multiple or more synthesized elements can be used as needed. For example, being able to synthesize elements in an image that trigger the deep neural network to recognize features can lead to a better understanding of the operation of the deep neural network. It is possible to provide a representation of the features identified by the deep neural network in multiple samples associated with the result. This representation can be visual, and the visualization can be easily presented to domain experts.

[0066] S8: The information obtained via step S7 can be used to improve the operation of the deep neural network. For example, features known to be relevant to the outcome but not present in those features identified by the deep neural network can be identified as missing from the training library. Therefore, the training library can be updated and fed into the deep neural network during the training phase S1, thereby improving the system.

[0067] In other words, the system according to the invention is such that, once the deep neural network is trained using a training library, a larger set of images and an input library can be used to activate the constructed features of the deep neural network. The system operates such that image regions causing activation of the constructed features can be identified and stored on a per-feature basis. It should be understood that annotation is unnecessary as long as the images are relevant to the domain.

[0068] The stored information represents a feature-specific database of visual examples associated with each constructed feature. In the next step, the system uses this created feature-specific database and, with respect to each feature, trains an appropriate generative technique, such as a GAN, to generate images that trigger the trained deep neural network to recognize the features. It should be understood that any suitable mechanism can be used to train the system to recognize / generate elements of an image that cause the deep neural network to activate image-related features; this mechanism does not have to be a GAN.

[0069] Each GAN generator can be manipulated to interpolate between the generated visual examples to form a feature-specific library, thus providing a more comprehensive view of the learned features. In some arrangements, GAN generators can be used in conjunction with GUI tools such as sliders, allowing AI and medical experts to analyze the visual features constructed from deep neural networks and express them in domain-specific terminology where appropriate.

[0070] In some illustrative examples, it is recognized that maximizing the differences between features constructed by a deep neural network and fed into a GAN can be useful. In other words, maximizing the differences between GANs for different features. Thus, the overlap between different features and the different feature views generated can be minimized, and those skilled in the art can more clearly visualize each feature (distinguished from any other constructed features identified by the deep neural network).

[0071] Figure 2b This is a schematic diagram illustrating the method steps according to an illustrative example. As shown in the figure, such as... Figure 1a The system shown can be operated to perform a method in a series of sequential steps as described below.

[0072] Step 1 (S1). The convolutional deep neural network is trained in the usual way on an image classification or segmentation task (e.g., see ImageNet Classification with Deep Convolutional Neural Networks [Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, Advances in Neural Information Processing Systems 25 (NIPS 2012)], etc.).

[0073] Step 2 (S2). You can select a layer in the deep neural network that is close to the output (classification, segmentation) to visualize its features. The selected layer allows all its features to be visualized. Features are linked to nodes in the selected layer. When a node is activated (i.e., outputs a high value) for a given image input, the image is said to contain the corresponding feature.

[0074] Typically, the selected layer is one of the last layers before the network output, as this layer contains the highest-level features that are combined to form the model's decision or output. However, it should be understood that deeper layers can be chosen depending on the visualization needs. The layer selection can be changed after viewing the initial visualization. For example, if the initial visualization shows feature levels that are too high / too low, deeper / shallower layers can be selected. In some examples, as an alternative to visualizing all features or nodes in the selected layer, the user can choose specific nodes in different layers of the network.

[0075] Step 3 (S3). A number of example images (e.g., from a training or validation set relevant to a domain, on which the model created by the deep neural network is relevantly optimized) are used as input to the model. These images do not need to be annotated. Each image input to the model can activate one or more feature nodes in the selected layer.

[0076] Step 4 (S4). For example, Grad-CAM visualization methods can be used to highlight one or more elements of a specific activated feature node for a particular image (for each image and each node in the selected layer). It should be understood that for image classification, pixels across the entire (potentially large) input image might be highlighted to account for activation of a single node. For segmentation, activated nodes typically correspond to sub-parts (or patches) of the input image.

[0077] The highlighted image may include a superposition of the original input image (a portion thereof) and a heatmap / weight map, the superimposed heatmap / weight map indicating, for example, how much each pixel contributes to the model output (i.e., feature activation). For illustrative purposes, the combination of the original image portion and the heatmap / weight map can itself be represented as an example image, the same size as or smaller than the original input image.

[0078] As you can see, example images can be generated in several ways:

[0079] For segmentation tasks, the model output is itself an image, and each layer in the network also corresponds to an image (typically lower resolution and with more features per pixel than the input image). Therefore, each activated feature node is associated with a specific pixel in the output or intermediate image. The essence of convolutional segmentation networks is that each output (or intermediate) pixel is only affected by a subset (or patches within) of the original image, the maximum size of which (or "field of view") is determined by the network architecture. Then, the example image for the activated feature node can simply be a patch in the input image associated with its field of view.

[0080] For classification tasks, the field of view of the final model output (typically the penultimate layer) is the entire input image. Here, smaller feature node activation examples can be generated, for example, by a moving window method over the input image, where a patch of the desired size can be selected if the pixels it contains are assigned more than the minimum average weight in the superimposed heatmap / weight map of the input image. Therefore, regions that are not highlighted in the heatmap are not selected; instead, regions that are clearly "highlighted" (i.e., those with significant heat or weights contributing to them) are chosen.

[0081] Other methods can be applied, but these are the most direct. Of course, the window method can also be applied to segmentation models. This approach may be preferable if the aforementioned field of view in such a model covers most of the input image.

[0082] Step 5 (S5). All the highlighted images generated by the visualization method generally described in Step 4 can be stored by feature, thereby generating an image database for each feature.

[0083] Subsequent steps will be described for a single feature, but these steps can be performed separately for each feature, such as... Figure 2b As shown schematically in the diagram.

[0084] Step 6 (S6). The GAN generator is trained on the highlighted image for each feature. The GAN generator is configured to generate representative images associated with the selected features.

[0085] Step 7 (S7). The trained feature GAN generator can be embedded in a graphical user interface that allows expert users to explore the feature image example space.

[0086] It should be understood that, regarding Figure 2b Steps 1 (S1) to 7 (S7) described correspond to the discussion of... Figure 2a The general steps S1 to S7 are described in the overview of the method.

[0087] Figure 3This section illustrates a possible illustrative example of a feature exploration graphical user interface (GUI) that allows domain experts to explore the output of a trained GAN regarding features. In other words, it provides the user with a visual format the mathematical understanding of the elements in an image that trigger the deep neural network to recognize features. As described below, the user can explore a space of feature image examples.

[0088] a. The GAN generator works by transforming an N-dimensional space of random numbers into a representative image. The GUI begins with a specific selection of random input values ​​(random selection, center point, etc.) corresponding to an output example.

[0089] b.GUI has methods for navigating in a random input space. One way to implement this navigation is to provide N sliders, each associated with a dimension of the random input space, which slide from the minimum value of a dimension to its maximum allowed value. Other methods can be used to explore multidimensional spaces.

[0090] c. As the user moves through the random input space, the images generated by the GAN corresponding to the selected random input are continuously displayed to the user, causing the example features to transform from one to another.

[0091] d. Instead of user-led exploration of random space / feature space, random input can also be generated via random walks in the random input space, thereby showing the user a video of smoothly and continuously changing feature examples.

[0092] It should be understood that the described illustrative examples can provide a visualization tool that offers expert users a smooth / continuous representation of visual features constructed or identified by a deep neural network. During the development and improvement phases of a deep neural network assigned to a specific task, the visualization tool can be used to assist collaboration between AI and domain experts. The visualization tool can be used, for example, as a support tool for interpretable artificial intelligence in relation to medical applications during regulatory approval and clinical application phases. The output of the system and method according to the invention can allow a mechanism to identify methods for improving the operation of the deep neural network, for example, by highlighting gaps in the training library, anomalous images in the training library (including hospital / image capture features irrelevant to the expected results), and by identifying features relevant to the results but previously unknown to the domain expert, etc.

[0093] Although illustrative embodiments of the invention have been disclosed in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to the precise embodiments, and that various changes and modifications can be made therein by those skilled in the art without departing from the scope of the invention as defined by the appended claims and their equivalents.

[0094] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and make other variations to the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude multiple. A single processor or other unit can perform the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to achieve dominance. Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference marks in the claims should not be construed as limiting the scope.

Claims

1. A method of providing a representation of a feature linked to a node of a deep neural network, the feature being related to an outcome of the deep neural network, the method comprising: providing a training corpus for the deep neural network, the training corpus comprising: a plurality of samples associated with the outcome, wherein the plurality of samples comprises images, sounds, or signal traces; training the deep neural network using the plurality of samples from the training corpus; selecting a layer of the deep neural network to visualize at least one feature in a feature of the selected layer that is associated with a node in the selected layer, wherein the at least one feature is associated with the outcome; providing an input corpus to the deep neural network; identifying, in each sample of a plurality of samples in the input corpus, a set of elements that triggers identification of the feature linked to the node in the selected layer of the deep neural network; creating a feature identification corpus from the identified elements; using the feature identification corpus to synthesize a plurality of sets of elements of samples that have characteristics that trigger identification of the feature linked to the node in the selected layer of the deep neural network; and using the synthesized plurality of sets of elements to provide a representation of the feature linked to the node in the selected layer of the deep neural network in the plurality of samples associated with the outcome.

2. The method of claim 1, wherein the representation of the feature comprises: synthesizing a corpus of examples that have characteristics that trigger the deep neural network to identify the feature.

3. The method of claim 1 or claim 2, wherein identifying, in each sample of the plurality of samples in the input bank, a set of elements that trigger the deep neural network to identify the feature comprises: using a gradient-based method to identify, in each sample of the plurality of samples, a set of elements that triggers identification of the feature.

4. The method of claim 3, wherein the gradient-based method comprises one or more of: Grad-CAM, CAM, deconvolution, or guided backpropagation.

5. The method of claim 3, wherein an entry in the feature identification corpus comprises a combination of: a set of elements of the sample in the input corpus, and an indication from the gradient-based method about the set of elements of the sample in the input corpus that triggers the deep neural network to identify a relative importance in the feature.

6. The method of any one of claims 1 or 2, wherein synthesizing a set of multiple elements of the sample having properties that trigger the deep neural network to identify the feature comprises: using a generative modeling method guided by the feature identification corpus.

7. The method of claim 6, wherein the generative modeling method comprises one or more of: a generative adversarial network or a variational autoencoder.

8. The method of any one of claims 1 or 2, wherein the method comprises: selecting a feature related to an outcome to be represented.

9. The method of any one of claims 1 or 2, further comprising: using the deep neural network to identify an alternative feature in the plurality of samples associated with the outcome; creating an alternative feature identification corpus from the input corpus by identifying, in each sample of the plurality of samples in the input corpus, a set of elements that triggers the deep neural network to identify the alternative feature; using the alternative feature identification corpus to synthesize a plurality of sets of elements of samples that have characteristics that trigger the deep neural network to identify the alternative feature; and using the synthesized plurality of sets of elements to provide a representation of the alternative feature in the plurality of samples associated with the outcome. to provide a representation of the feature linked to a node of the deep neural network in the plurality of samples associated with the result using the synthesized plurality of element sets.

10. The method of claim 9, wherein the alternative features identified by the deep neural network are selected so that a difference between the element sets that trigger the deep neural network to identify the feature and the alternative features in each of the plurality of samples is maximized.

11. A computer program product operable to perform the method of any one of claims 1 to 10 when executed on a computer.

12. An apparatus to provide a representation of a feature linked to a node of a deep neural network, the feature being related to a result of the deep neural network, the apparatus comprising: a deep neural network trained with a training library, the training library comprising a plurality of samples associated with the result and configured by training the deep neural network using the plurality of samples from the training library, wherein the plurality of samples comprises images, sounds, or signal traces; a data processor configured to select a layer of the deep neural network to visualize at least one of the features in the selected layer that is associated with a node in the selected layer, wherein the at least one feature is associated with the result; a library creation logic configured to create a feature identification library from an input library by identifying in each of the plurality of samples forming the input library an element set that triggers identification of the feature linked to the node in the selected layer of the deep neural network; a synthesis logic configured to synthesize a plurality of element sets of a sample having a characteristic that triggers identification of the feature linked to the node in the selected layer of the deep neural network using the feature identification library; and a feature visualization logic configured to provide a representation of the feature linked to the node in the selected layer of the deep neural network in the plurality of samples associated with the result using the synthesized plurality of element sets.

Images