Explainable visual attention for deep learning

Abstract

To provide systems or techniques that facilitate explainable visual attention for deep learning.SOLUTION: A system can access a medical image generated by a medical imaging scanner, and perform, via execution of a deep learning neural network, an inferencing task on the medical image. In various embodiments, the deep learning neural network can receive as input the medical image and can produce as output both an inferencing task result and an attention map indicating on which pixels or voxels of the medical image the deep learning neural network focused in generating the inferencing task result.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates generally to deep learning, and more specifically, to explainable visual attention for deep learning. [Background technology]

[0002] Deep learning neural networks can be trained to perform inference tasks on medical images. In performing inference tasks, it may be desirable for the deep learning neural network to pay more attention to certain portions of a medical image than to other portions of the medical image. Unfortunately, existing techniques for encouraging such attention are poorly interpreted and explained.

[0003] Therefore, systems and techniques that can facilitate deep learning attention to medical images with enhanced interpretability or explainability are desirable. Summary of the Invention

[0004] The following presents a summary to provide a basic understanding of one or more embodiments. This summary is not intended to identify key or critical elements or to delineate the scope of particular embodiments or the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description presented later. In one or more embodiments described herein, a device, system, computer-implemented method, apparatus, or computer program product for facilitating explainable visual attention for deep learning is described.

[0005] According to one or more embodiments, a system is provided. The system may include a non-transitory computer readable memory capable of storing computer executable components. The system may further include a processor operatively coupled to the non-transitory computer readable memory and capable of executing the computer executable components stored in the non-transitory computer readable memory. In various embodiments, the computer executable components may include an access component capable of accessing medical images generated by a medical imaging scanner. In various aspects, the computer executable components may constitute an execution component capable of performing inference tasks on the medical images via execution of a deep learning neural network. In various implementations, the deep learning neural network may receive the medical image as an input and may generate as an output both an inference task result and an attention map indicating which pixels or voxels of the medical image the deep learning neural network attended to in generating the inference task result.

[0006] According to one or more embodiments, a computer-implemented method is provided. In various embodiments, the computer-implemented method may include accessing, by a device operatively coupled to a processor, a medical image generated by a medical imaging scanner. In various aspects, the computer-implemented method may comprise performing, by the device, an inference task on the medical image via execution of a deep learning neural network. In various implementations, the deep learning neural network may receive the medical image as an input and may generate, as an output, both an inference task result and an attention map indicating which pixels or voxels of the medical image the deep learning neural network attended to in generating the inference task result.

[0007] According to one or more embodiments, a computer program product for facilitating explainable visual attention for deep learning is provided. In various embodiments, the computer program product may comprise a non-transitory computer-readable memory having program instructions embodied therein. In various aspects, the program instructions may be executable by the processor to cause the processor to access a deep learning neural network including a primary processing channel, the primary processing channel may be trained to perform an inference task using a set of training images and a set of ground truth inference task results each corresponding to the set of training images. In various implementations, the program instructions may be further executable to cause the processor to insert an auxiliary processing channel into the deep learning neural network such that the auxiliary processing channel branches off from the primary processing channel and is in parallel with at least a portion of the primary processing channel. In various aspects, the program instructions may be further executable to cause the processor to train the auxiliary processing channel using a set of ground truth attention maps each corresponding to a set of training images to generate a pixel-wise or voxel-wise attention map that indicates where the deep learning neural network will focus in an input image when performing an inference task. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates a block diagram of an example, non-limiting system for facilitating explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Diagram 2] FIG. 1 is a block diagram of an example, non-limiting system including a deep learning neural network, inference task results, and an attention map that facilitates explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Diagram 3]FIG. 1 is an exemplary, non-limiting block diagram illustrating how a deep learning neural network can generate inference task results and attention maps in accordance with one or more embodiments described herein. [Figure 4] 1 illustrates an illustrative, non-limiting example of a medical image and corresponding attention map in accordance with one or more embodiments described herein. [Diagram 5] 1 illustrates an illustrative, non-limiting example of a medical image and corresponding attention map in accordance with one or more embodiments described herein. [Figure 6] FIG. 1 illustrates a block diagram of an example, non-limiting system including a training component and an annotated training dataset that facilitates explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Figure 7] FIG. 2 is an exemplary, non-limiting block diagram of an annotated training dataset according to one or more embodiments described herein. [Figure 8] FIG. 1 is an exemplary, non-limiting block diagram illustrating how a deep learning neural network can be trained in accordance with one or more embodiments described herein. [Figure 9] 13 shows exemplary, non-limiting experimental results on explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Figure 10] 13 shows exemplary, non-limiting experimental results on explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Figure 11] 13 shows exemplary, non-limiting experimental results on explainable visual attention for deep learning in accordance with one or more embodiments described herein. [Figure 12] 1 illustrates a flow diagram of an example, non-limiting, computer-implemented method for facilitating explainable visual attention for deep learning, in accordance with one or more embodiments described herein. [Figure 13]FIG. 1 illustrates a block diagram of an exemplary, non-limiting operating environment in which one or more embodiments described herein may be facilitated. [Figure 14] 1 illustrates an example of a networking environment operable to execute various implementations described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following detailed description is merely illustrative and is not intended to limit the embodiments or the application / uses of the embodiments, and further, is not intended to be bound by any express or implied information presented in the preceding "Background" or "Summary" sections or in the "Detailed Description" section.

[0010] One or more embodiments will now be described with reference to the drawings, in which like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent that in various instances, one or more embodiments may be practiced without these specific details.

[0011] Deep learning neural networks may be trained (e.g., in a supervised manner, an unsupervised manner, or a reinforcement learning manner) to perform inference tasks (e.g., classification, segmentation, regression) on medical images (e.g., on computed tomography (CT) scan images, on magnetic resonance imaging (MRI) scan images, on x-ray scan images, on ultrasound scan images, on positron emission tomography (PET) scan images). To facilitate the inference task, it may be desirable for the deep learning neural network to pay more attention to certain parts of the medical image than other parts of the medical image. After all, it may not be the case that all depicted regions of the medical image are critical or otherwise important with respect to the inference task. Thus, the inference task may be performed accurately or precisely by focusing primarily on any regions of the medical image that are favorable to the task and downplaying or ignoring any other regions of the medical image that are unfavorable to the task.

[0012] Existing techniques attempt to achieve such attention or focus by equipping deep learning neural networks with learnable attention blocks. A learnable attention block is one or more hidden layers following an upstream portion of a deep learning neural network and preceding a downstream portion of the deep learning neural network, such that the learnable attention block receives various feature / activation maps calculated by the upstream portion, the learnable attention block calculates a respective attention score (also called an alignment score) for each of those various feature / activation maps, and the downstream portion calculates its own feature / activation map, weighting each of those various feature / activation maps by the calculated attention score. Thus, the learnable attention block can be thought of as determining the relative importance of the various feature / activation maps received from the upstream portion (e.g., features / activation maps for which a numerically low attention score is calculated can be thought of as less important, and features / activation maps for which a numerically high attention score is calculated can be thought of as more important). In other words, the upstream portion can calculate various feature / activation maps, the learnable attention block can determine which of those various feature / activation maps are more or less important, and the downstream portion can focus on which of those various feature / activation maps are determined to be more important.

[0013] Unfortunately, such existing techniques suffer from insufficient interpretability and explainability. This lack of interpretability and explainability is due, at least in part, to a variety of reasons.

[0014] First, the attention scores of the existing techniques are completely hidden within the deep learning neural network. In other words, when the existing techniques are implemented, the attention scores calculated by the trainable attention block are not provided or otherwise treated as an independent output of the deep learning neural network. Thus, a user or technician of the deep learning neural network is not made aware of such attention scores at all, i.e., the user or technician is not informed of where the deep learning neural network has focused its attention in performing an inference task on a medical image.

[0015] Second, even if the attention scores of the existing techniques are extracted and provided as independent outputs of the deep learning neural network, such attention scores do not have easily comprehensible semantic meanings with respect to medical images. In fact, as described above, the learnable attention block is sandwiched between the upstream and downstream parts of the deep learning neural network, and the attention scores calculated by the learnable attention block serve as weights for the respective feature / activation maps generated by the upstream part. Thus, the attention scores can be considered to indicate which of those feature / activation maps should receive more or less focus from the deep learning neural network. However, unfortunately, it is not at all clear how such feature / activation maps relate to medical images. After all, such feature / activation maps are hidden, abstract, numerical representations (e.g., latent vectors) and do not have easily identifiable or interpretable physical meanings with respect to medical images. Thus, even if a user or technician of the deep learning neural network is made aware of the attention scores calculated by the learnable attention block, such attention scores would not convey any physically relevant information about the medical image to the user or technician. Instead, such attention scores merely allow the user or technician to identify those various features / activation maps to which the deep learning neural network is paying more or less attention. Because those features / activation maps bear no easily understandable relationship to whatever visual content is depicted in the medical image, such information can be considered virtually useless.

[0016] Third, even if the attention scores of existing techniques have some physically related meaning with respect to medical images, such physically related meaning will be layer-dependent or relative, rather than layer-transcendent or absolute. Again, the learnable attention block receives multiple feature / activation maps from an upstream portion of the deep learning neural network and assigns respective attention scores to those received feature / activation maps. It should be noted that different layers of the deep learning neural network can generate different numbers, different sizes, different formats, or different types of feature / activation maps. Thus, the depth within the network at which the learnable attention block is located (e.g., it can be located more upstream of the deep learning neural network, or it can be located more downstream of the deep learning neural network) determines the number and type of feature / activation maps to which attention scores are assigned by the learnable attention block, which correspondingly determines the meaning (if any) of the calculated attention score. This may therefore be considered confusing to a user or technician, as the meaning of the attention score may differ between neural networks with learnable attention blocks in different positions (e.g., assume that two separate neural networks with learnable attention blocks in different positions have been trained to perform an inference task).Suppose that two separate neural networks with differently-located learnable attention blocks are trained to perform the inferencing task; in such case, attention scores produced by one neural network will not have the same substantive meaning as attention scores produced by the other neural network).

[0017] Fourth, the learnable attention block of existing techniques is unguided and therefore prone to learning strange or undesirable attention patterns. In particular, when existing techniques are implemented, ground truth attention scores are not used to directly train the learnable attention block. Instead, the learnable attention block is updated via backpropagation with the rest of the deep learning neural network (e.g., to maximize the final reinforcement learning reward, to minimize the final error relative to the ground truth inference task result). This lack of ground truth attention scores can be at least partially attributed to the above-mentioned fact that the attention scores of existing techniques do not have easily interpretable physical meanings (e.g., if a given feature / activation map does not have a clear physical meaning, it is not possible to know in advance what ground truth attention scores should correspond to a given feature / activation map). Such unguided training ultimately leads the learnable attention block to overfit or learn to assign attention scores in strange, unexpected, or puzzling ways. This again undermines the interpretability and explainability of existing techniques.

[0018] Therefore, systems and techniques that can bring deep learning attention to medical images with enhanced interpretability and explainability are desirable.

[0019] Various embodiments described herein may address one or more of these technical problems. One or more embodiments described herein may include a system, computer-implemented method, apparatus, or computer program product that may facilitate explainable visual attention for deep learning. In particular, various embodiments described herein may include configuring or training a deep learning neural network to perform an inference task on an input medical image, where the deep learning neural network may comprise a primary processing channel and an auxiliary processing channel branching off from the primary processing channel. As described herein, the primary processing channel may be responsible for generating an inference task result on the input medical image as an output, while the auxiliary processing channel may instead be responsible for generating an attention map on the input medical image as an output. In various aspects, the attention map may indicate or represent a pixel-wise (or voxel-wise) attention score for the input medical image. In other words, the attention map may be considered to indicate how much the deep learning neural network considers or infers that each individual pixel (or each individual voxel) of the input medical image is important or unimportant to the performance of the inference task. In still other words, the attention map can be considered to identify or indicate which pixels (or voxels) of the input medical image the deep learning neural network pays attention to. In various embodiments, as described herein, the auxiliary processing channel can be trained in a supervised manner using training medical images and their corresponding ground truth attention maps, and the primary processing channel can be trained using a loss function that is weighted pixel-wise (or voxel-wise) by such ground truth attention maps. In various cases, such ground truth attention maps can be obtained based on running any suitable edge or boundary detector on the training medical images.In various other cases (e.g., for image-to-image regression tasks), such ground truth attention maps can instead be obtained based on the differences between the training medical images and the respective corresponding ground truth inference task results. In still other cases, such ground truth attention maps can instead be obtained based on the uncertainties or errors that the deep learning neural network exhibited with respect to the training medical images during previous training epochs.

[0020] Various embodiments described herein can be considered to facilitate visual attention with increased explainability or interpretability as compared to existing techniques.

[0021] First, as described above, the attention scores of existing techniques are completely hidden. That is, such attention scores are not provided or extracted as an independent output viewable by a user or technician. In contrast, the attention map described herein is provided as an independent output viewable by a user or technician. Thus, when various embodiments described herein are implemented, a user or technician can become informed of where a deep learning neural network focuses its attention when performing an inference task on a medical image.

[0022] Second, as mentioned above, the attention scores of the existing techniques do not have a readily-interpretable physical meaning with respect to medical images. After all, such attention scores correspond to (e.g., are used internally to weight) various feature / activation maps that are internally calculated by hidden neural network layers, and such various feature / activation maps are abstract latent vectors that have no readily understandable physical or semantic relationship to the visual content depicted in the medical image. In contrast, the attention maps described herein have a readily-interpretable physical meaning with respect to medical images. Specifically, the attention maps described herein indicate a respective attention score for each individual pixel (or individual voxel) of the medical image. Thus, the attention map can be thought of as indicating which particular pixels (or voxels) of the medical image are determined by the deep learning neural network to be more or less important for the performance of the inference task. Thus, the attention map can be visually rendered (e.g., like a heat map) to visually highlight the pixels (or voxels) of the medical image that are receiving more attention or focus from the deep learning neural network. This direct relationship to individual pixels (or voxels) of a medical image gives the attention maps described herein a clear, physically relevant meaning, making them readily understandable, interpretable, or explainable.

[0023] Third, as mentioned above, the attention scores of existing techniques are layer dependent. After all, even if such attention scores provide at least some practical meaning, such practical meaning depends on the particular characteristics of the feature / activation maps they weight, and different neural network layers (e.g., layers located at different network depths) will generate feature / activation maps with different characteristics. Thus, different neural networks configured to perform inference tasks but with learnable attention blocks located at different positions will generate attention scores with inconsistent and therefore incomparable meanings, confusing users or engineers (e.g., a different network might compute attention scores for feature / activation maps produced by a b-th hidden layer, for any suitable positive integers a>b>1. Thus, such attention scores cannot be compared to each other). In contrast, the attention maps described herein indicate pixel-wise (or voxel-wise) attention scores regardless of how deep into the network the auxiliary processing channel is located. Indeed, the specific structure of one or more layers of auxiliary processing channels may need to be adjusted depending on how deep into the network they branch off from the primary processing channel, but as described herein, the output of an auxiliary processing channel is always a pixel-wise (or voxel-wise) map of attention scores.Thus, different neural networks configured to perform inference tasks but with auxiliary processing channels in different locations will nevertheless generate attention maps with consistent and therefore comparable meanings, reducing user or technician confusion (e.g., each network will generate a respective pixel-wise (or voxel-wise) map of attention scores, and such maps may therefore be properly compared with each other).

[0024] Fourth, as mentioned above, the learnable attention blocks of existing techniques are not guided by ground truth attention scores. Such lack of guidance can cause such learnable attention blocks to learn to allocate attention in strange or puzzling ways. In contrast, the auxiliary processing channels described herein can be trained based on training medical images and their corresponding ground truth attention maps. Such ground truth attention maps (which can be used to dynamically weight the training loss function of the primary processing channel) can be obtained via edge detection, uncertainty quantification, or image-annotation difference computation, as described herein. In any case, the use of such ground truth attention maps can reduce the likelihood that the auxiliary processing channels learn to allocate attention in unusual, strange, or puzzling ways.

[0025] For at least these reasons, the various embodiments described herein can be considered to bring attention to visual deep learning with improved explainability.

[0026] Various embodiments described herein can be considered as computerized tools (e.g., any suitable combination of computer-executable hardware or computer-executable software) that can facilitate explainable visual attention for deep learning. In various aspects, such computerized tools can constitute an access component, an execution component, or a display component.

[0027] In various embodiments, the particular medical image may be present. In various aspects, the particular medical image may exhibit any suitable format, size, or dimensionality (e.g., the particular medical image may be a two-dimensional pixel array, or the particular medical image may be a three-dimensional voxel array). In various implementations, the particular medical image may be generated or imaged by any suitable medical imaging modality or device (e.g., generated or imaged by a CT scanner, by an MRI scanner, by an X-ray scanner, by an ultrasound scanner, or by a PET scanner). In various cases, the particular medical image may visually depict any suitable anatomical structure of any suitable medical patient.

[0028] In various aspects, an inference task may be executable for a particular medical image. In various implementations, the inference task may be any suitable predictive computation or functionality applicable to an image. As some non-limiting examples, the inference task may be image classification, image segmentation, or image-to-image regression. In various aspects, the inference task may be considered to have some diagnostic or prognostic relevance with respect to an anatomical structure depicted in a particular medical image.

[0029] In any event, it may be desirable to perform an inference task on a particular medical image. Moreover, during such execution, it may be desirable to discover which pixels (or voxels) of a particular medical image are receiving more or less attention or focus (e.g., discover which pixels (or voxels) of a particular medical image are receiving more or less disposition with respect to an inference task). In various aspects, the computerized tools described herein can facilitate such objectives.

[0030] In various embodiments, an access component of a computerized tool can receive or otherwise access the particular medical image electronically. In some aspects, the access component can electronically retrieve the particular medical image from any suitable centralized or distributed data structure (e.g., graph data structures, relational data structures, hybrid data structures), whether remote or local to the access component. In any case, the access component can retrieve or access the particular medical image electronically such that other components of the computerized tool can electronically interact with (e.g., read, write, edit, copy, manipulate) the particular medical image.

[0031] In various embodiments, the execution component of the computerized tool can electronically store, maintain, control, or otherwise access the deep learning neural network. In various aspects, the deep learning neural network can include a primary processing channel and an auxiliary processing channel.

[0032] In various embodiments, the primary processing channel can exhibit any suitable deep learning internal architecture. For example, the primary processing channel can include any suitable number of layers of any suitable type (e.g., an input layer, one or more hidden layers, an output layer, any of which can be convolutional layers, dense layers, non-linearity layers, pooling layers, batch normalization layers, or padding layers). As another example, the primary processing channel can include any suitable number of neurons in various layers (e.g., different layers can have the same or different number of neurons from each other). As yet another example, the primary processing channel can include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, rectified linear unit) in various neurons (e.g., different neurons can have the same or different activation functions from each other). As yet another example, the primary processing channels may include any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, recurrent connections).

[0033] In various cases, the auxiliary processing channel may similarly exhibit any suitable deep learning internal architecture. For example, the auxiliary processing channel may include any suitable number of layers of any suitable type (e.g., an input layer, one or more hidden layers, an output layer, any of which may be a convolutional layer, a dense layer, a nonlinearity layer, a pooling layer, a batch normalization layer, or a padding layer). As another example, the auxiliary processing channel may include any suitable number of neurons in various layers (e.g., different layers may have the same or different number of neurons from each other). As yet another example, the auxiliary processing channel may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, rectified linear unit) for various neurons (e.g., different neurons may have the same or different activation functions from each other). As yet another example, the auxiliary processing channel may include any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, recurrent connections).

[0034] In various embodiments, the auxiliary processing channel can branch off from the primary processing channel such that the auxiliary processing channel is in parallel with at least a portion of the primary processing channel. For example, the primary processing channel can have a given hidden layer between its input layer and its output layer, and the input layer of the auxiliary processing channel can be configured to receive as input any feature / activation maps generated by that given hidden layer such that the auxiliary processing channel is in parallel with all layers of the primary processing channel that are downstream of that given hidden layer.

[0035] In any event, the primary processing channel may be configured to perform an inference task on an input medical image, while the auxiliary processing channel may instead be configured to generate a pixel-wise (or voxel-wise) attention map on such input medical image. Thus, the execution component may electronically execute a deep learning neural network on a particular medical image, and such execution may result in an inference task result and an attention map. More specifically, the execution component may feed a particular medical image to an input layer of the primary processing channel, which may complete a forward pass through one or more hidden layers of the primary processing channel, such forward pass may cause an output layer of the primary processing channel to compute an inference task result. Furthermore, the input layer of the auxiliary processing channel may receive one or more activations from any hidden layer of the primary processing channel from which the auxiliary processing channel branches, and such one or more activations may complete a forward pass through one or more hidden layers of the auxiliary processing channel, such forward pass may cause an output layer of the auxiliary processing channel to compute an attention map.

[0036] In various aspects, the inference task result can be any computational result that the primary processing channel determines or predicts for a particular medical image. Thus, the inference task result can be any suitable electronic data having a format, size, or dimensionality that depends on the inference task that the primary processing channel is configured to perform. As a non-limiting example, if the inference task is image classification, the inference task result can be a classification label that the primary processing channel predicts for a particular medical image. As another non-limiting example, if the inference task is image segmentation, the inference task result can be a segmentation mask that the primary processing channel predicts for a particular medical image. As yet another non-limiting example, if the inference task is image-to-image regression (e.g., denoising, resolution enhancement, modality transformation), the inference task result can be a regression version (e.g., a denoised version, a resolution-enhanced version, a modality-transformed version) of the particular medical image that the primary processing channel predicted.

[0037] In various embodiments, the attention map can be a pixel-by-pixel (or voxel-by-voxel) array of attention scores corresponding to a particular medical image, where each attention score can be a scalar weight with a magnitude indicating the level of attention, focus, importance, or significance that a respective pixel (or voxel) of a particular medical image has with respect to the inference task (e.g., the attention score can be a decimal that varies continuously or discretely from 0 to 1, where 0 indicates no importance with respect to the inference task and 1 indicates high importance with respect to the inference task). For example, if the attention map indicates that a given pixel (or voxel) of a particular medical image has a low attention score (e.g., close to 0), this can be interpreted to mean that the given pixel (or voxel) does not significantly affect or contribute to the disposition of the inference task. On the other hand, if the attention map instead indicates that a given pixel (or voxel) of a particular medical image has a high attention score (e.g., close to 1), this can be interpreted as meaning that the given pixel (or voxel) significantly influences or contributes to the disposition of the inference task. In other words, the attention map can be thought of as a prediction that identifies how much individual pixels (or voxels) of a particular medical image will or will not be attended to by the primary processing channel. In any case, the attention map can be thought of as having or indicating a clear and easily understandable physical, semantic, or practical meaning with respect to the particular medical image (e.g., a particular medical image depicts some visual content, and the attention map indicates, on a pixel-by-pixel (or voxel-by-voxel) basis, which parts or regions of such visual content are more or less task-dependent).

[0038] In this manner, the primary processing channel can be thought of as performing inference tasks on a particular medical image, and the auxiliary processing channel can be thought of as predicting or inferring which particular pixels (or voxels) of a particular medical image will be more or less noteworthy by the primary processing channel.

[0039] In various embodiments, a display component of the computerized tool can electronically render the inference task results or attention map on any suitable electronic display, where the inference task results or attention map can be viewed by a user or technician of the computerized tool. In this manner, the user or technician can see not only the inference task results, but also which individual pixels (or voxels) of a particular medical image contributed most to the inference task results.

[0040] It should be noted that in various aspects, the attention map may be utilized by a user or technician for failure analysis of inference task results. As a non-limiting example, a user or technician may determine that an inference task result is inaccurate. That is, the user or technician may conclude that the primary processing channel performed the inference task on a particular medical image inaccurately. In such a case, the attention map may help determine how or why the primary processing channel performed the inference task inaccurately. In fact, the attention map may indicate that some unexpected pixels (or voxels) of a particular medical image received disproportionately high attention or focus. This may indicate that the primary processing channel was distracted or otherwise confused by the visual content presented by those unexpected pixels (or voxels). In other words, the attention map is not agnostic to the primary processing channel, but it can help reveal insights about visual properties that the primary processing channel should be agnostic to. A user or technician can then use such insight to help retrain or fine-tune the primary processing channel to avoid similar failures in the future.

[0041] To help the inference task results and attention maps be accurate (e.g., to help the primary processing channel correctly perform the inference task on the input medical images and to help the auxiliary processing channel correctly predict which pixels (or voxels) of such input medical images are more or less important with respect to the inference task), the deep learning neural network can first be trained. In various aspects, the access component can receive, obtain, or otherwise access the annotated training dataset, and the computerized tool can include a training component that can train the deep learning neural network on the annotated training dataset.

[0042] In various embodiments, the annotated training dataset may include a set of training medical images. In various embodiments, the training medical images may be any suitable medical images having the same size, format, or dimensions (e.g., the same number or arrangement of pixels (or voxels)) as the particular medical image. In various embodiments, the annotated training dataset may also include a set of ground-truth inferencing task results each corresponding to the set of training medical images. In various embodiments, the ground-truth inferencing task results may be correct or accurate inferencing task results (e.g., correct or accurate classification labels, correct or accurate segmentation masks, correct or accurate regression outputs) that are known to correspond to or are deemed to correspond to the respective training medical images. In various embodiments, the annotated training dataset may further include a set of ground-truth attention maps each corresponding to the set of training medical images. In various implementations, the ground truth attention map can be a correct or accurate arrangement of pixel-wise or voxel-wise attention scores that are known or assumed to correspond to the respective training medical images.

[0043] In some embodiments, the ground truth attention map for a given training medical image can be obtained by running an edge / boundary detector on the given training medical image. Indeed, visually perceptible edges, boundaries, or other contours (e.g., of anatomical structures) shown in a given training medical image may be known or considered to be highly relevant or otherwise critical for the inference task. Thus, whatever pixels (or voxels) of the given training medical image that constitute or otherwise belong to those visually perceptible edges, boundaries, or other contours may be assigned a higher ground truth attention score. To achieve this, any suitable edge / boundary detector can be implemented. In particular, the edge / boundary detector can be any suitable machine learning model that has been pre-trained, via any suitable training paradigm (e.g., supervised training, unsupervised training, reinforcement learning), to segment pixels (or voxels) in an input image that are suspected or determined to belong to an edge, boundary, or contour of the input image. Thus, an edge / boundary detector can be run on a given training medical image to obtain an inferred segmentation mask that indicates which pixels (or voxels) of the given training medical image belong to edges, boundaries, or contours that are visually depicted in the given training medical image, and such an inferred segmentation mask can be treated or otherwise captured as a ground truth attention map for the given training medical image.

[0044] In other embodiments, the ground truth attention map for a given training medical image can instead be obtained by comparing the given training medical image with the ground truth inference task result corresponding to the given training medical image. In fact, if the inference task is inter-image regression, the ground truth inference task result can be an image of the same size, format, or dimension as the given training medical image. In fact, the ground truth inference task result can be a regressed version (e.g., a denoised version, a resolution-enhanced version, a modality-transformed version) of the given training medical image. In such a situation, the intensity values ​​(e.g., Hounsfield units) of some pixels (or voxels) of the given training medical image may undergo little change when transformed into the ground truth inference task result, while the intensity values ​​of other pixels (or voxels) of the given training medical image may undergo large changes when transformed into the ground truth inference task result. Thus, the pixels (or voxels) of the given training medical image that undergo large changes in generating the ground truth inference task result can be assigned a higher ground truth attention score. To achieve this, a difference array can be computed by applying matrix subtraction between the given training medical image and the ground-truth inferencing task result, the difference array can be normalized, and the absolute value of the normalized difference array can be treated or otherwise taken as the ground-truth attention map for the given training medical image.

[0045] In yet another embodiment, the ground truth attention map for a given training medical image can instead be obtained by quantifying the uncertainty or error indicated by the primary processing channel with respect to the given training medical image. In particular, it may be the case that the primary processing channel has been previously executed (e.g., during a prior training epoch) for the given training medical image. If the inference task is image segmentation or inter-image regression, such a previous execution may result in an output array. This output array may be considered as a previously predicted segmentation mask or a previously predicted regressed image. In contrast, the ground truth inference task result may be considered as a correct or accurate segmentation mask or regressed image that is known or considered to correspond to the given training medical image. Here, the intensity values ​​of some pixels (or voxels) of the output array may be close to what the ground truth inference task result indicates that it should be, and the intensity values ​​of other pixels (or voxels) of the output array may be far from what the ground truth inference task result indicates that it should be. Thus, pixels (or voxels) in the output array that differ significantly from what the ground truth inference task results indicate can be assigned a higher ground truth attention score. To achieve this, a difference array can be calculated by applying a matrix subtraction between the output array and the ground truth inference task results, the difference array can be normalized, and the absolute value of the normalized difference array can be treated or otherwise taken as the ground truth attention map for a given training medical image.

[0046] In any case, the training component can train a deep learning neural network on the annotated training dataset, as follows: Prior to the start of such training, trainable internal parameters (e.g., weight matrices, bias vectors, convolutional kernels) of the deep learning neural network (e.g., of the primary processing channel, of the auxiliary processing channel) can be randomly initialized.

[0047] In various embodiments, the training component can select any suitable training medical images, any suitable corresponding ground truth inference task results, and any suitable corresponding ground truth attention maps from the annotated training dataset. In various embodiments, the training component can run a deep learning neural network on the selected training medical images, thereby causing the primary processing channel to generate a first output and the auxiliary processing channel to generate a second output.

[0048] More specifically, the training component can provide selected training medical images to an input layer of the primary processing channel, where the selected training medical images can complete a forward pass through one or more hidden layers of the primary processing channel, where such forward pass can cause an output layer of the primary processing channel to compute a first output. Additionally, the input layer of the auxiliary processing channel can receive one or more activations from any hidden layer of the primary processing channel from which the auxiliary processing channel branches, where such one or more activations can complete a forward pass through one or more hidden layers of the auxiliary processing channel, where such forward pass can cause an output layer of the auxiliary processing channel to compute a second output.

[0049] Note that the format, size, or dimensionality of the first output can be controlled or otherwise dictated by the number, arrangement, or sizes of the neurons or of other internal parameters (e.g., convolutional kernels) that are contained in or that otherwise make up the output layer of the primary processing channel. Thus, the first output can be forced to have a format, size, or dimensionality appropriate for the inference task by adding, removing, or otherwise adjusting neurons or other internal parameters from or within the output layer of the primary processing channel. Thus, the first output can be thought of as a predicted inference task result (e.g., predicted classification label, predicted segmentation mask, predicted regression output) that the primary processing channel believes should correspond to the selected training medical image. In contrast, the selected ground truth inference task result may be a correct or accurate inference task result (e.g., a correct or accurate classification label, a correct or accurate segmentation mask, a correct or accurate regression output) that is known or deemed to correspond to the selected training medical image. In various cases, if the primary processing channel has never or very little prior training, the first output may be highly inaccurate (e.g., may differ significantly from the selected ground truth inference task result).

[0050] Similarly, it should be noted that the format, size, or dimensionality of the second output may be controlled or otherwise dictated by the number, arrangement, or size of neurons or other internal parameters (e.g., convolution kernels) included in or constituting the output layer of the auxiliary processing channel. Thus, the second output may be forced to have any desired format, size, or dimensionality by adding, removing, or otherwise adjusting neurons or other internal parameters to, from, or within the output layer of the auxiliary processing channel. Thus, the second output may be considered a predicted attention map (e.g., a predicted array of attention scores per pixel (or per voxel)) that the auxiliary processing channel believes should correspond to the selected training medical image. In contrast, the selected ground truth attention map may be a correct or accurate attention map (e.g., a correct or accurate array of attention scores per pixel (or per voxel)) that is known or deemed to correspond to the selected training medical image. In various cases, if the auxiliary processing channel has had little or no training so far, the second output may be highly inaccurate (e.g., may be very different from the selected ground truth attention map).

[0051] In any case, the training component can calculate a first error or loss (e.g., mean absolute error (MAE), mean squared error (MSE), cross-entropy error) between the first output and a selected ground truth inference task result. In various embodiments, the training component can weight the first error or loss on a pixel-by-pixel (or voxel-by-voxel) basis according to the selected ground truth attention map (e.g., achieved via point-wise multiplication and summation), and the training component can update internal parameters of the primary processing channel by performing backpropagation (e.g., stochastic gradient descent) driven by the first error or loss weighted by the selected ground truth attention map.

[0052] Similarly, the training component can calculate a second error or loss (e.g., MAE, MSE, cross-entropy error) between the second output and the selected ground truth attention map, and the training component can update internal parameters of the auxiliary processing channel by performing backpropagation (e.g., stochastic gradient descent) driven by the second error or loss.

[0053] In various embodiments, such training procedures can be repeated for any suitable number of training medical images, such that the trainable internal parameters of the primary processing channel are iteratively optimized to accurately perform an inference task on input medical images, and such training ultimately results in the trainable internal parameters of the auxiliary processing channels being iteratively optimized to accurately predict how important individual pixels (or voxels) of such input medical images are with respect to the inference task.

[0054] Various embodiments described herein may be employed to use hardware or software to solve problems that are highly technical in nature (e.g., to facilitate explainable visual attention for deep learning), are not abstract, and cannot be performed as a series of mental acts by humans. Furthermore, some of the processing performed may be performed by specialized computers (e.g., deep learning neural networks with internal parameters such as convolution kernels) to perform defined acts related to deep learning. For example, such defined acts may include accessing medical images generated by a medical imaging scanner by a device operably coupled to a processor, and performing an inference task on the medical image by the device and through execution of a deep learning neural network, which receives the medical image as an input and generates as an output both an inference task result and an attention map indicating which pixels or voxels of the medical image the deep learning neural network attended to in generating the inference task result.

[0055] Such defined acts are not something a human would perform manually. In fact, neither the human mind nor a human with pen and paper can electronically access scanned medical images (e.g., x-rays, MRIs, CT images) and perform an inference task electronically by running a neural network on the scanned medical images, which computes as output both the inference task result (e.g., classification labels, segmentation masks, regression images) and an attention map indicating which particular pixels (or voxels) of the scanned medical images the neural network attended to (e.g., which particular pixels (or voxels) contributed most significantly to the inference task result). In fact, neural networks are essentially computerized components and cannot be meaningfully performed or in any way trained by a human mind without a computer. Computerized tools that can run neural networks on scanned medical images are likewise essentially computerized and cannot be implemented in any sensible, practical, or rational way without a computer.

[0056] Moreover, various embodiments described herein can integrate various teachings on explainable visual attention for deep learning into practical applications. As mentioned above, existing techniques incorporate learnable attention blocks into neural networks, where such learnable attention blocks calculate attention scores that are used to weight each hidden activation map. Unfortunately, such learnable attention blocks suffer from a lack of explainability and interpretability. Indeed, the attention scores provided by such learnable attention blocks are completely hidden (e.g., not extracted as the output of an independent neural network) and do not have a readily understandable physical meaning. Moreover, even if such attention scores could yield some physical meaning, such meaning would depend on the depth in the network where the learnable attention block is located. Moreover, such learnable attention blocks are trained in an unguided manner that may lead to learning strange or unexplainable attention patterns.

[0057] In contrast, various embodiments described herein may address one or more of these technical problems. In particular, various embodiments described herein include training or configuring a neural network to receive a medical image as an input and generate two different outputs: an inference task result for the input medical image and an attention map indicating a relative level of importance or non-importance for the inference task result for individual pixels (or voxels) of the input medical image. In various aspects, the inference task result may be generated by a primary processing channel of the neural network, and the attention map may be generated by an auxiliary processing channel that branches off from the primary processing channel and is in parallel with some downstream portion of the primary processing channel.

[0058] Unlike the attention scores of existing techniques, the attention maps described herein are provided as a separate output of the neural network or otherwise handled (e.g., a user or technician is not made aware of the attention scores of existing techniques, but a user or technician is made aware of the attention maps described herein).

[0059] Also, unlike the attention scores of existing techniques, the attention maps described herein have easily understandable physical or semantic meanings. For example, each weight of the attention map indicates how important or unimportant each pixel (or voxel) of the input medical image is with respect to the inference task result, and such importance or unimportance is meaningful because it is clear how the pixel (or voxel) is related to the visual content of the input medical image. In contrast, each attention score of existing techniques only indicates how important or unimportant each hidden activation map is, and such importance or unimportance is meaningless because it is unclear how the hidden activation map is related to the visual content of the input medical image (e.g., the hidden activation map is an abstract representation or latent vector that has no obvious or interpretable meaning).

[0060] Moreover, unlike the attention scores of existing techniques, the meaning of the attention map described herein is layer-independent. In fact, what meaning (if any) the attention score calculated by the learnable attention block has depends on where the learnable attention block is located in the neural network (e.g., the learnable attention block generates attention scores whatever hidden activation maps it receives as input. Different hidden layers generate different amounts, types, or formats of hidden activation maps. Accordingly, learnable attention blocks in different locations generate different amounts of attention scores with different meanings). In contrast, the attention map described herein indicates the importance, attention, or focus of each pixel (or voxel) of the input medical image, regardless of the depth in the network where the auxiliary processing channel is located (e.g., the auxiliary processing channel can be appropriately configured to generate a pixel-by-pixel (or voxel-by-voxel) array of attention scores, regardless of the depth where the auxiliary processing channel is located).

[0061] Moreover, unlike the learnable attention blocks of existing techniques, the auxiliary processing channels described herein can be trained in a guided fashion using ground truth attention maps, since such attention scores do not have clear physical meaning (e.g., if one does not know a priori what a training hidden activation map means or represents, one should not know the ground truth attention scores corresponding to that training hidden activation map). In contrast, one can know or estimate a priori how important or unimportant each pixel (or voxel) of a training medical image is with respect to the inference task, so that a ground truth attention map can be obtained to train the auxiliary processing channel (e.g., as described herein, such ground truth attention maps can be obtained via edge / boundary detection, via image annotation comparison, or by past uncertainty / error quantification).

[0062] Moreover, the attention map described herein can be leveraged for neural network failure analysis in a way that the attention scores of existing techniques cannot. Indeed, suppose the primary processing channel produces an inference task result inaccurately or incorrectly. In such cases, potential reasons or explanations for such inaccuracies can be obtained from the attention map generated by the auxiliary processing channel (e.g., the attention map may indicate that certain pixels (or voxels) of the input medical image were unexpectedly or erroneously emphasized or given importance by the primary processing channel). In this way, the attention map can be leveraged to identify insights, such as visual features or artifacts that distracted or threw-off the primary processing channel, which are then useful for retraining or fine-tuning the primary processing channel.

[0063] For at least these reasons, the various embodiments described herein facilitate visual attention of medical images for deep learning with enhanced or increased explainability compared to existing techniques. Thus, the various embodiments described herein certainly constitute concrete and tangible technical improvements or technical advantages in the field of deep learning. Thus, such embodiments clearly qualify as useful and practical applications of computers.

[0064] Further, various embodiments described herein can control real-world tangible devices based on the disclosed teachings. For example, various embodiments described herein can electronically train or run real-world deep learning neural networks on real-world images (e.g., x-ray scan images, CT scan images) and electronically render real-world results (e.g., classification labels, segmentation masks, regression images, attention maps) on a real-world computer screen.

[0065] It should be understood that the illustrations and descriptions herein provide non-limiting examples of various embodiments and are not necessarily drawn to scale.

[0066] 1 illustrates a block diagram of an exemplary, non-limiting system 100 that can facilitate explainable visual attention for deep learning in accordance with one or more embodiments described herein. As shown, the attention system 102 can be electronically integrated with a medical image 104 via any suitable wired or wireless electronic connection.

[0067] In various embodiments, the medical image 104 may be any suitable image exhibiting any suitable format, size, or dimension. As a non-limiting example, the medical image 104 may be an x ​​x y array of pixels for any suitable positive integers x and y. As another non-limiting example, the medical image 104 may be an x ​​x y x z array of voxels for any suitable positive integers x, y, and z. As a non-limiting example, the medical image 104 may be captured or generated by a CT scanner, in which case the medical image 104 may be considered a CT scan image. As another non-limiting example, the medical image 104 may be captured or generated by an MRI scanner, in which case the medical image 104 may be considered an MRI scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by an X-ray scanner, in which case the medical image 104 may be considered an X-ray scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by an ultrasound scanner, in which case the medical image 104 may be considered an ultrasound scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by a PET scanner, in which case the medical image 104 may be considered a PET scan image. In various embodiments, the medical image 104 may be subjected to any suitable image reconstruction technique, such as filtered backprojection.

[0068] In various embodiments, the medical image 104 may visually depict or illustrate any suitable anatomical structure of any suitable medical patient (e.g., human, animal, or otherwise). In various embodiments, the anatomical structure may be any suitable body organ of a medical patient, any suitable body tissue of a medical patient, any suitable body part of a medical patient, any suitable body fluid of a medical patient, any suitable body cavity of a medical patient, any suitable surgical implant (e.g., medical tubing, medical stitches, medical stents, pacemakers, medical rods, medical plates, medical screws), any suitable pathology or injury thereof, or any suitable portion thereof.

[0069] In various embodiments, an inference task may be executable on the medical image 104 with respect to anatomical structures depicted in the medical image 104. In various embodiments, the inference task may be any suitable predictive calculation. As a non-limiting example, the inference task may be image classification, which involves assigning the medical image 104 to one of two or more possible categories or classes (e.g., determining whether the anatomical structures of the medical image 104 belong to a disease class or, alternatively, to a healthy class). As another non-limiting example, the inference task may be image segmentation, which involves assigning each pixel (or voxel) of the medical image 104 to a respective one of two or more possible categories or classes (e.g., determining which pixels (or voxels) of the medical image 104 belong to a tumor class and which pixels (or voxels) instead belong to a background class). As yet another non-limiting example, the inference task may be image-to-image regression, which involves generating a new image based on the medical image 104 (e.g., converting the medical image 104 from one modality (X-ray) to another (MRI), generating a denoised version of the medical image 104, generating an enhanced version of the medical image 104).

[0070] In various cases, it may be desirable to perform inference tasks on medical images 104. It may also be desirable to ascertain which particular pixels (or voxels) of medical images 104 contribute most significantly to such performance. As described herein, attention system 102 can facilitate such objectives.

[0071] In various embodiments, the attention system 102 can comprise a processor 106 (e.g., a computer processing unit, microprocessor) and a non-transient computer-readable memory 108 operatively or operatively or communicatively connected or coupled to the processor 106. The non-transient computer-readable memory 108 can store computer-executable instructions that, when executed by the processor 106, can cause the processor 106 or other components of the attention system 102 (e.g., the access component 110, the execution component 112, the display component 114) to perform one or more actions. In various embodiments, the non-transient computer-readable memory 108 can store and the processor 106 can execute the computer-executable components (e.g., the access component 110, the execution component 112, the display component 114).

[0072] In various embodiments, the attention system 102 can include an access component 110. In various aspects, the access component 110 can electronically receive or otherwise access the medical images 104. In various aspects, the access component 110 can electronically obtain the medical images 104 from any suitable centralized or distributed data structure (not shown) or from any suitable centralized or distributed computing device (not shown). As a non-limiting example, the access component 110 can electronically obtain the medical images 104 from whatever medical imaging modality (e.g., CT scanner, X-ray scanner, MRI scanner, ultrasound scanner, PET scanner) that generated or captured the medical images 104. In any case, the access component 110 can electronically obtain or access the medical images 104 such that other components of the attention system 102 can electronically interact with the medical images 104.

[0073] In various embodiments, the attention system 102 can include an executive component 112. In various aspects, the executive component 112 can execute a deep learning neural network on the medical images 104 to obtain both an inference task result and an attention map that indicates which particular pixels (or voxels) of the medical images 104 the deep learning neural network attended to to generate the inference task result.

[0074] In various embodiments, the attention system 102 can include a display component 114. In various implementations, the display component 114 can visually render the inference task results or the attention map on any suitable electronic display.

[0075] 2 illustrates a block diagram of an exemplary, non-limiting system 200 including a deep learning neural network, an inference task result, and an attention map that can facilitate explainable visual attention for deep learning in accordance with one or more embodiments described herein. As shown, system 200 can include the same components as system 100 in some cases, and can further include a deep learning neural network 202, an inference task result 208, and an attention map 210.

[0076] In various embodiments, the executive component 112 can electronically store, maintain, control, or otherwise access the deep learning neural network 202. In various aspects, the deep learning neural network 202 can include a primary processing channel 204 and an auxiliary processing channel 206.

[0077] In various embodiments, the primary processing channel 204 may have or otherwise exhibit any suitable deep learning internal architecture. For example, the primary processing channel 204 may have an input layer, one or more hidden layers, and an output layer. In various embodiments, any of such layers may be coupled by any suitable inter-neuron or inter-layer connections, such as forward connections, skip connections, or recurrent connections. Furthermore, in various embodiments, any of such layers may be any suitable type of neural network layer having any suitable learnable or trainable internal parameters. For example, any of such input layers, one or more hidden layers, or output layers may be convolutional layers, and the learnable or trainable parameters may be convolution kernels. As another example, any of such input layers, one or more hidden layers, or output layers may be dense layers, and the learnable or trainable parameters may be weight matrices or bias values. As yet another example, any of such input layers, one or more hidden layers, or output layers may be batch normalization layers, and the learnable or trainable parameters may be shift or scale factors. Moreover, in various cases, any such layers may be any suitable type of neural network layer having any suitable fixed or non-trainable internal parameters. For example, any such input layer, one or more hidden layers, or output layer may be non-linearity layers, padding layers, pooling layers, or concatenation layers.

[0078] In various cases, the auxiliary processing channel 206 may have or otherwise exhibit any suitable deep learning internal architecture. For example, the auxiliary processing channel 206 may have an input layer, one or more hidden layers, and an output layer. In various embodiments, any of such layers may be coupled by any suitable inter-neuron or inter-layer connections, such as forward connections, skip connections, or recurrent connections. Furthermore, in various embodiments, any of such layers may be any suitable type of neural network layer having any suitable learnable or trainable internal parameters. For example, any of such input layers, one or more hidden layers, or output layers may be convolutional layers, and the learnable or trainable parameters may be convolution kernels. As another example, any of such input layers, one or more hidden layers, or output layers may be dense layers, and the learnable or trainable parameters may be weight matrices or bias values. As yet another example, any such input layer, one or more hidden layers, or output layer may be a batch normalization layer, whose learnable or trainable parameters may be shift factors or scale factors. Furthermore, in various cases, any such layer may be any suitable type of neural network layer having any suitable fixed or non-learnable internal parameters. For example, any such input layer, one or more hidden layers, or output layer may be a nonlinearity layer, a padding layer, a pooling layer, or a concatenation layer.

[0079] In various embodiments, the primary processing channel 204 can be considered the trunk of the deep learning neural network 202, and the auxiliary processing channel 206 can branch off from the primary processing channel 204. Thus, the auxiliary processing channel 206 can be considered to be in parallel with at least a portion (e.g., a downstream end) of the primary processing channel 204.

[0080] In various implementations, the primary processing channel 204 can be configured or trained to perform an inference task on the input medical image, as described herein. Additionally, the auxiliary processing channel 206 can be configured or trained to predict how important, significant, useful, or dispositional each particular pixel (or voxel) of the input medical image is with respect to performing an inference task, as described herein. Thus, in various cases, the execution component 112 can electronically execute the deep learning neural network 202 on the medical image 104, such execution can cause the deep learning neural network 202 to generate both an inference task result 208 and an attention map 210 as output. More specifically, such execution can cause the primary processing channel 204 to generate an inference task result 208 as output, and such execution can cause the auxiliary processing channel 206 to generate an attention map 210 as output. Various non-limiting embodiments are described with respect to FIG. 3.

[0081] FIG. 3 shows an example, non-limiting block diagram 300 illustrating how the deep learning neural network 202 can generate the inference task result 208 and the attention map 210 in accordance with one or more embodiments described herein.

[0082] In various embodiments, the primary processing channel 204 can include n neural network layers (layers 204(1) through 204(n)) for any suitable positive integer n. In various aspects, layer 204(1) can be regarded as the input layer of the primary processing channel 204, layer 204(n) can be regarded as the output layer of the primary processing channel 204, and all of the remaining layers (e.g., 204(2) through 204(n - 1)) can be regarded as the hidden layers of the primary processing channel 204. As described above, any of such n layers can be any suitable type of neural network layer having any suitable trainable or non - trainable internal parameters (e.g., any of such layers can be a dense layer, a convolutional layer, a batch normalization layer, a non - linearity layer, an upsampling layer, a downsampling layer, a pooling layer, a padding layer, or a concatenation layer). In various embodiments, as shown, the layers of the primary processing channel 204 can be coupled together via forward connections such that the activations generated by layer 204(c) are received by layer 204(c + 1). Here, c is any suitable positive integer such that 1 ≤ c < n. However, this is merely a non - limiting example for ease of explanation. In various cases, any other suitable type of inter - layer connection (e.g., skip connection, recurrent connection) can be implemented in the primary processing channel 204.

[0083] In various embodiments, the auxiliary processing channel 206 can include m neural network layers: layer 206(1) to layer 206(m) for any suitable positive integer m. In various aspects, layer 206(1) can be regarded as the input layer of the auxiliary processing channel 206, layer 206(m) can be regarded as the output layer of the auxiliary processing channel 206, and all the remaining layers (e.g., 206(2) to 206(m - 1)) can be regarded as hidden layers. As described above, any of such m layers can be any suitable type of neural network layer having any suitable trainable or non-trainable internal parameters (e.g., any of such layers can be dense layers, convolutional layers, batch normalization layers, non-linearity layers, upsampling layers, downsampling layers, pooling layers, padding layers, or concatenation layers). In various embodiments, as shown, the layers of the auxiliary processing channel 206 can be coupled together via forward connections such that the activation generated by layer 206(d) is received by layer 206(d + 1). Here, d is any suitable positive integer where 1 ≤ d < m. However, this is merely a non-limiting example for ease of explanation. In various cases, any other suitable type of inter-layer connection (e.g., skip connections, recurrent connections) can be implemented in the auxiliary processing channel 206.

[0084] In various aspects, the primary processing channel 204 can have layers 204(j) for any suitable positive integer j (1 < j < n). That is, the layer 204(j) can be a hidden layer of the primary processing channel 204. In various embodiments, as shown, the layer 206(1) of the auxiliary processing channel 206 can be configured to receive as input whatever activation is generated by the layer 204(j). In other words, the auxiliary processing channel 206 can be considered to branch from the primary processing channel 204 at the layer 204(j). Thereby, the auxiliary processing channel 206 can be parallel to any portion of the primary processing channel 204 that is downstream of the layer 204(j) (for example, made parallel from the layer 204(j + 1) to the layer 204(n)). With this structure, the primary processing channel 204 can be considered as the trunk (hence the term "primary") that forms the basis of the deep learning neural network 202, and the auxiliary processing channel 206 can be considered as a secondary branch (hence the term "auxiliary") that branches from that underlying trunk.

[0085] As described above, the execution component 112 can electronically execute the deep learning neural network 202 on the medical image 104, and such execution can result in the inference task result 208 and the attention map 210. More specifically, the execution component 112 can feed the medical image 104 to layer 204(1) of the primary processing channel 204. In various aspects, the medical image 104 can complete a forward pass through hidden layers of the primary processing channel 204 (e.g., layer 204(2) through layer 204(n-1)). In various implementations, the layer 204(n) of the primary processing channel 204 can calculate the inference task result 208 based on whatever activations the layer 204(n) receives from layer 204(n-1) (or from any other layer of the primary processing channel 204 if skip or recurrent connections are implemented). During the forward pass through the primary processing channel 204, layer 204(j) may generate one or more activations at some point. In various aspects, the one or more activations of layer 204(j) may be fed to layer 206(1) of the auxiliary processing channel 206 or may be routed in other ways. In various implementations, those one or more activations may complete a forward pass through hidden layers of the auxiliary processing channel 206 (e.g., from layer 206(2) to layer 206(m-1)). In various cases, layer 206(m) of the auxiliary processing channel 206 may compute an attention map 210 based on whatever activations layer 206(m) receives from layer 206(m-1) (or receives from any other layers of the auxiliary processing channel 206 if skip connections or recurrent connections are implemented).

[0086] In various embodiments, the inference task result 208 may be any suitable electronic data that represents or otherwise indicates the results of performing an inference task on the medical image 104. Thus, the format, size, or dimensions of the inference task result 208 may depend on the inference task. As a non-limiting example, consider the inference task to be image classification. In that case, the inference task result 208 may be a classification label that the primary processing channel 204 predicted for the medical image 104. As another non-limiting example, consider the inference task to be image segmentation. In that case, the inference task result 208 may be a pixel-wise (or voxel-wise) segmentation mask that the primary processing channel 204 predicted for the medical image 104. As yet another non-limiting example, consider the inference task to be image-to-image regression. In that case, the inferencing task result 208 can be a regression image that the primary processing channel 204 predicts for the medical image 104 (e.g., the inferencing task result 208 can be a predicted denoised version of the medical image 104; the inferencing task result 208 can be a predicted modality-transformed version of the medical image 104).

[0087] In various embodiments, the attention map 210 can be an array of attention scores, where each attention score is a scalar weight that represents or indicates how important or unimportant each pixel (or voxel) of the medical image 104 is with respect to the inference task result 208. Thus, the attention map 210 can have the same format, size, or dimensionality as the medical image 104. As a non-limiting example, assume that the medical image 104 is an x×y pixel array. In such a case, the attention map 210 can be an x×y array of attention scores, and the attention score (i, j) of such an array can be a scalar real value whose magnitude indicates how much or how little the pixel (i, j) of the medical image 104 contributes to the generation of the attention score (where i, j are any suitable positive integers such that 1≦i<x, 1≦j<y). As another non-limiting example, the medical image 104 can be an x×y×z voxel array. In this case, the attention map 210 can be an x×y×z array of attention scores, and the attention score (i, j, k) of such an array can be a scalar real value whose magnitude indicates how much the voxel (i, j, k) of the medical image 104 contributes to the generation of the inference task result 208 for any suitable positive integers 1≦i<x, 1≦j<y, and 1≦k<z.

[0088] In various cases, the magnitude of any attention score indicated by the attention map 210 can vary continuously or discretely between any suitable minimum value (e.g., 0) that indicates the minimum contribution to the inference task and any suitable maximum value (e.g., 1) that indicates the maximum contribution to the inference task. However, this is merely a non-limiting example for ease of explanation. In various other cases, the minimum value can instead represent the maximum contribution, and the maximum value can instead represent the minimum contribution.

[0089] In any case, the attention map 210 can be thought of as indicating which particular pixels (or voxels) of the medical image 104 are more favorable / influential or less favorable / influential with respect to the inference task. In other words, the primary processing channel 204 can perform an inference task on the medical image 104, and the auxiliary processing channel 206 can predict which pixels (or voxels) of the medical image 104 have been most (or least) attention or least attention by the primary processing channel 204. In still other words, for any given pixel (or voxel) of the medical image 104, the auxiliary processing channel 206 can be thought of as inferring how much or how little the primary processing channel 204 will focus on that given pixel (or voxel). Non-limiting embodiments are described with respect to Figures 4-5.

[0090] 4-5 show illustrative, non-limiting examples of medical images and corresponding attention maps in accordance with one or more embodiments described herein.

[0091] Consider first Fig. 4. Fig. 4 illustrates a non-limiting example in which the medical image 104 is a two-dimensional sinogram. In this non-limiting example, the attention map 210 is a two-dimensional array having the same format, size, or dimensions as the sinogram, and indicating how important or unimportant individual pixels of the sinogram are with respect to the performance of the inference task. In particular, dark regions of the attention map 210 can be considered to indicate, represent, or otherwise identify pixels of the sinogram that do not strongly influence or impact the inference task result 208. In other words, such dark regions of the attention map 210 can be considered to indicate which pixels of the sinogram are predicted or determined by the auxiliary processing channel 206 to receive less attention or focus from the primary processing channel 204. Conversely, light regions of the attention map 210 can be considered to indicate, represent, or otherwise identify pixels of the sinogram that strongly influence or impact the inference task result 208. That is, such bright areas of the attention map 210 can be thought of as indicating which pixels of the sinogram are predicted or determined by the auxiliary processing channel 206 to receive more attention or focus from the primary processing channel 204.

[0092] Consider now Fig. 5, which illustrates a non-limiting example in which the medical image 104 is a two-dimensional CT scan image. In this non-limiting example, the attention map 210 is a two-dimensional array having the same format, size, or dimensions as the CT scan image, and indicating how important or unimportant individual pixels of the CT scan image are with respect to the performance of the inference task. In particular, dark regions of the attention map 210 can be considered to indicate, represent, or otherwise identify pixels of the CT scan image that do not strongly influence or have no impact on the inference task result 208 (e.g., indicating which pixels of the CT scan image are predicted or determined by the auxiliary processing channel 206 to receive less attention or focus from the primary processing channel 204). Conversely, light regions of the attention map 210 can be considered to indicate, represent, or otherwise identify pixels of the CT scan image that strongly influence or have an impact on the inference task result 208 (e.g., indicating which pixels of the CT scan image are predicted or determined by the auxiliary processing channel 206 to receive more attention or focus from the primary processing channel 204).

[0093] In any case, the execution component 112 can execute the deep learning neural network 202 on the medical images 104, and such execution can provide both inference task results 208 and attention maps 210 as outputs.

[0094] In various embodiments, the display component 114 can electronically render the inference task result 208 or the attention map 210 on any suitable electronic display (e.g., computer screen, computer monitor). Thus, a user or technician associated with the attention system 102 can visually inspect the inference task result 208 or the attention map 210. Such visual inspection allows the user or technician to recognize not only the inference task result 208 but also which particular pixels (or voxels) of the medical image 104 were determined to have contributed most to the inference task result 208. This can be considered to provide the user or technician with clear and easily understandable meaning or insight as to what particular visual content exemplified by the medical image 104 most or least influenced the inference task result 208. In some cases, such meaning or insight can assist the user or technician in troubleshooting the deep learning neural network 202.

[0095] As a non-limiting example, suppose the user or technician concludes that the inference task result 208 is at least partially inaccurate or incorrect. In such a case, the user or technician can inspect the attention map 210. The attention map 210 can indicate that a significant amount of attention was paid to an unexpected subset of pixels (or voxels) of the medical image 104. Thus, the user or technician can conclude that whatever visual content is indicated by such an unexpected subset of pixels (or voxels) has distracted or otherwise thrown off the primary processing channel 204. In other words, the attention map 210 can help the user or technician identify visual content that is not agnostic to the primary processing channel 204, but that should be agnostic to the primary processing channel 204. In still other words, the attention map 210 can aid a user or technician in identifying why the primary processing channel 204 failed to correctly analyze the medical image 104, and such identification can be used to help retrain or fine-tune the primary processing channel 204 to avoid similar failures in the future.

[0096] In various aspects, the display component 114 can electronically transmit the inference task results 208 or the attention map 210 to another suitable computing device (not shown).

[0097] For the inference task results 208 or attention map 210 to be correct, the deep learning neural network 202 may first be trained as described with respect to FIGS. 6-8.

[0098] 6 illustrates a block diagram of an exemplary, non-limiting system 600 including a training component and an annotated training dataset that can facilitate explainable visual attention for deep learning in accordance with one or more embodiments described herein. As shown, system 600 can be comprised of the same components as system 200 in some cases and can further include a training component 602 and an annotated training dataset 604.

[0099] In various embodiments, the access component 110 can electronically receive, acquire, or otherwise access the annotated training data set 604 from any suitable source, and the training component 602 can electronically train the deep learning neural network 202 with the annotated training data set 604. Various non-limiting aspects are described with respect to FIGS.

[0100] FIG. 7 illustrates an example, non-limiting block diagram 700 of an annotated training dataset 604, in accordance with one or more embodiments described herein.

[0101] In various embodiments, the annotated training data set 604 may comprise a set of training medical images 702. In various aspects, the set of training medical images 702 may include q images, training medical image 702(1) through training medical image 702(q), for any suitable positive integer q. In various implementations, each of the set of training medical images may exhibit the same format, size, or dimensions as the medical image 104. As a non-limiting example, consider the medical image 104 as an x×y pixel array. In such a case, each of the set of training medical images 702 may similarly be an x×y pixel array. As another non-limiting example, consider the medical image 104 as an x×y×z voxel array. In such a case, each of the set of training medical images 702 may similarly be an x×y×z voxel array.

[0102] In various embodiments, the annotated training data set 604 can include a set of ground truth inference task results 704. In various embodiments, each of the set of ground truth inference task results 704 can correspond (e.g., one-to-one) to a set of training medical images 702. Thus, since the set of training medical images 702 can include q images, the set of ground truth inference task results 704 can include q inference task results: ground truth inference task result 704(1) through ground truth inference task result 704(q). In various cases, each of the set of ground truth inference task results 704 can have the same format, size, or dimensions as the inference task results 208. Indeed, in various embodiments, each of the set of ground truth inference task results 704 can be, indicate, or otherwise represent a correct or accurate inference task result that is known or deemed to correspond to a respective one of the set of training medical images 702. As a non-limiting example, ground truth inference task result 704(1) may correspond to training medical image 702(1). Thus, ground truth inference task result 704(1) may be considered a correct or accurate classification label, segmentation mask, or regression output that is known or deemed to correspond to training medical image 702(1). As another non-limiting example, ground truth inference task result 704(q) may correspond to training medical image 702(q). Thus, ground truth inference task result 704(q) may be considered a correct or accurate classification label, segmentation mask, or regression output that is known or deemed to correspond to training medical image 702(q).

[0103] In various embodiments, the annotated training dataset 604 may include a set of ground truth attention maps 706(1). In various embodiments, each of the set of ground truth attention maps 706 may correspond (e.g., one-to-one) to a set of training medical images 702. Thus, since the set of training medical images 702 may include q images, the set of ground truth attention maps 706 may include q maps: ground truth attention map 706(1) through ground truth attention map 706(q). In various embodiments, each of the set of ground truth attention maps 706 may have the same format, size, or dimensions as the attention map 210. Indeed, in various embodiments, each of the set of ground truth attention maps 706 may be, indicate, or otherwise represent any correct or accurate attention map that is known or deemed to correspond to a respective one of the training medical images 702. As a non-limiting example, ground truth attention map 706(1) may correspond to training medical image 702(1). Thus, ground truth attention map 706(1) may be considered to indicate correct or accurate respective attention scores for each individual pixel (or voxel) of training medical image 702(1). As another non-limiting example, ground truth attention map 706(q) may correspond to training medical image 702(q). Thus, ground truth attention map 706(q) may be considered to indicate correct or accurate respective attention scores for each individual pixel (or voxel) of training medical image 702(q).

[0104] In various embodiments, the set of ground truth attention maps 706 may be obtained or otherwise generated by performing edge, boundary, or contour localization on the set of training medical images 702. After all, given a medical image depicting an anatomical structure, one or more visible edges, boundaries, or contours of that anatomical structure may be particularly relevant for performing an inference task on the given medical image. Thus, there may be an edge detector (not shown). In various embodiments, the edge detector may be any suitable machine learning model that may be pre-trained in any suitable manner (e.g., supervised, unsupervised, reinforcement learning fashion) to segment visually perceptible edges, boundaries, or contours in the input image. Thus, the edge detector may be performed on the set of training medical images 702, respectively, and the resulting segmentation masks generated by the edge detector may be treated as a set of ground truth attention maps 706, respectively. As a non-limiting example, an edge detector may be run on the training medical image 702(1), such that the edge detector may generate a segmentation mask indicating which pixels (or voxels) of the training medical image 702(1) belong to or otherwise constitute a visible edge, boundary, or contour of whatever anatomical structure is depicted in the training medical image 702(1), and such segmentation mask may be treated as or taken as the ground truth attention map 706(1).As another non-limiting example, an edge detector may be run on the training medical image 702(q) such that the edge detector generates a segmentation mask indicating which pixels (or voxels) of the training medical image 702(q) belong to or otherwise constitute a visible edge, boundary, or contour of an anatomical structure depicted in the training medical image 702(q), and such segmentation mask may be treated or taken as the ground truth attention map 706(q).

[0105] It should be noted that the use of an edge detector described above is merely a non-limiting example of how a set of ground truth attention maps 706 may be obtained. In other embodiments, the set of ground truth attention maps 706 may be obtained by running any other suitable type of feature extractor on the set of training medical images 702, respectively. In such cases, the feature extractor may be trained in any suitable manner to receive an image and generate a segmentation mask indicating which pixels (or voxels) of that image belong to any suitable visually perceptible feature depicted in the image. In other words, the various embodiments described herein are not limited to only edge, boundary, or contour detection.

[0106] In various embodiments, the set of ground truth attention maps 706 may be obtained or otherwise generated by comparing the set of training medical images 702 with the set of ground truth inference task results 704. In fact, when the inference task is image-to-image regression, each of the set of ground truth inference task results 704 may be considered as one transformed version of each of the set of training medical images 702. As such, pixels (or voxels) whose intensity values ​​change the most between the set of training medical images 702 and the set of ground truth inference task results 704 are particularly relevant for performing the inference task. Thus, the set of ground truth attention maps 706 may be obtained based on respective differences calculated between the set of training medical images 702 and the set of ground truth inference task results 704. As a non-limiting example, the ground truth inference task result 704(1) may be subtracted from the training medical image 702(1), thereby obtaining a first difference array. In various cases, the ground truth attention map 706(1) can be equal to or otherwise based on a normalized, absolute-value-version of that first difference array. As another non-limiting example, the ground truth inference task result 704(q) can be subtracted from the training medical image 702(q), resulting in the q-th difference array. In various cases, the ground truth attention map 706(q) can be equal to or otherwise based on a normalized, absolute-value-version of that q-th difference array.

[0107] It should be noted that the above-described use of matrix subtraction between the set of ground truth inference task results 704 and the set of training medical images 702 is merely a non-limiting example of how the set of ground truth attention maps 706 may be obtained. In other embodiments, the set of ground truth attention maps 706 may be obtained by computing any other suitable type of relationship between the set of ground truth inference task results 704 and the set of training medical images 702, respectively. In other words, the various embodiments described herein are not limited to only computing subtractive differences between the set of ground truth inference task results 704 and the set of training medical images 702, respectively.

[0108] In various embodiments, the set of ground truth attention maps 706 may be obtained or otherwise generated by measuring pixel-wise (or voxel-wise) uncertainties previously exhibited by the primary processing channel 204 on the set of training medical images 702. Indeed, if the inference task is image segmentation or image-to-image regression, execution of the primary processing channel 204 on a given medical image may cause the primary processing channel 204 to generate an output array (e.g., generate a segmentation mask or a regression image). If a ground truth inference task result is available for the given medical image, the ground truth inference task result may be compared to the output array, and pixels (or voxels) with significantly different intensity values ​​between the ground truth inference task result and the output array may be particularly relevant for performing the inference task. Thus, a set of ground truth attention maps 706 can be obtained by respectively comparing the set of ground truth inference task results 704 with sequences previously output by the primary processing channel 204 based on the set of training medical images 702. As a non-limiting example, it may be the case that the primary processing channel 204 was previously run on the training medical image 702(1) (e.g., during a previous training epoch) and such run resulted in a first prior output sequence (e.g., a first prior segmentation mask or regressed image that the primary processing channel 204 predicted for the training medical image 702(1)).In various embodiments, the ground truth inference task result 704(1) may be subtracted from the first pre-output array, resulting in a first difference array. In various embodiments, the ground truth attention map 706(1) may be equal to or otherwise based on a normalized absolute version of the first difference array. As another non-limiting example, the primary processing channel 204 may have previously been run (e.g., during a previous training epoch) on the training medical image 702(q), and such run may have resulted in a q-th pre-output array (e.g., the q-th pre-segmentation mask or regression image that the primary processing channel 204 predicted for the training medical image 702(q)). In various embodiments, the ground truth inference task result 704(q) may be subtracted from the q-th pre-output array, resulting in a q-th difference array. In various embodiments, the ground truth attention map 706(q) may be equal to or otherwise based on a normalized absolute version of the q-th difference array.

[0109] It should be noted that the above-mentioned techniques for obtaining the set of ground truth attention maps 706 are merely non-limiting examples. In various other embodiments, any other suitable techniques or any other suitable information can be utilized to generate or obtain the set of ground truth attention maps 706 (e.g., a segmentation mask generated from or otherwise associated with the set of training medical images 702 or the set of ground truth inference task results 704 can be utilized; brightness or contrast gradients exhibited within the set of training medical images 702 or the set of ground truth inference task results 704 can be utilized; metadata such as imaging energy levels (kilovolt peak setting, milliamp setting) associated with the set of training medical images 702 or the set of ground truth inference task results 704 can be utilized; any other information that can reveal or influence how important or unimportant which regions of a given training medical image are can be utilized).

[0110] Consider now Figure 8. Figure 8 shows an example, non-limiting block diagram 800 illustrating how the deep learning neural network 202 may be trained in accordance with one or more embodiments described herein.

[0111] In various embodiments, the training component 602 can train the deep learning neural network 202 on the annotated training dataset 604 as follows.

[0112] Prior to such training, the training component 602 may initialize the trainable internal parameters (e.g., weight matrices, bias vectors, convolutional kernels) of the primary processing channel 204 and the auxiliary processing channel 206 in any suitable manner (e.g., random initialization).

[0113] In various aspects, the training component 602 can select any suitable training medical images, any suitable corresponding ground truth inferencing task results, and any suitable corresponding ground truth attention maps from the annotated training dataset 604. These can be referred to as training medical images 802, ground truth inferencing task results 804, and ground truth attention maps 806, respectively.

[0114] In various embodiments, the training component 602 can electronically execute the deep learning neural network 202 on the training medical images 802, and such execution can result in both the output 808 and the output 810. More specifically, the training component 602 can provide the training medical images 802 to layer 204(1) of the primary processing channel 204, which can complete a forward pass through the hidden layers of the primary processing channel 204, and layer 204(n) of the primary processing channel 204 can compute the output 808. Furthermore, during such forward pass, layer 204(j) of the primary processing channel 204 can compute one or more activations, and such one or more activations can be provided to layer 206(1) of the auxiliary processing channel 206, which can complete a forward pass through the hidden layers of the auxiliary processing channel 206, and layer 206(m) of the auxiliary processing channel 206 can compute the output 810.

[0115] It should be noted that the format, size, or dimensionality of the output 808 may be determined by the number, arrangement, size, or other characteristics of the neurons, convolution kernels, or other internal parameters of the layer 204(n) (or any other layer of the primary processing channel 204). Thus, the output 808 can be forced to have any format, size, or dimensionality that is desired or suitable for the inference task by adding, removing, or otherwise adjusting characteristics of the layer 204(n) (or any other layer of the primary processing channel 204). Thus, the output 808 can be thought of as a predicted inference task result (e.g., predicted classification label, predicted segmentation mask, predicted regression output) that the primary processing channel 204 believes should correspond to the training medical images 802. On the other hand, the ground truth inference task result 804 can be thought of as a correct or accurate inference task result (e.g., correct or accurate classification label, correct or accurate segmentation mask, correct or accurate regression output). In various cases, if the primary processing channel 204 has had no or little prior training, the output 808 may be very inaccurate (e.g., very different from the ground truth inference task result 804).

[0116] Similarly, it should be noted that the format, size, or dimensionality of the output 810 may be determined by the number, arrangement, size, or other characteristics of the neurons, convolution kernels, or other internal parameters of the layer 206(m) (or any other layer of the auxiliary processing channel 206). Thus, the output 810 may be forced to have the same format, size, or dimensionality as the training medical image 802 by adding, removing, or otherwise adjusting characteristics of the layer 206(m) (or any other layer of the auxiliary processing channel 206). Thus, the output 810 may be considered as a predicted attention map (e.g., a predicted array of attention scores per pixel (or per voxel)) that the auxiliary processing channel 206 believes should correspond to the training medical image 802. Similarly to the above, if the auxiliary processing channel 206 has not received any or very little training so far, the output 810 may be very inaccurate (e.g., may differ significantly from the ground truth attention map 806).

[0117] In various embodiments, the training component 602 can calculate a loss 812 between the output 808 and the ground truth inference task result 804 via any suitable error or objective function (e.g., MAE, MSE, cross-entropy). In various embodiments, the training component 602 can generate a weighted loss 814 by applying a point-wise multiplication and sum to the loss 812 and the ground truth attention map 806. This can be thought of as weighting the loss 812 according to the pixel-wise (or voxel-wise) attention score indicated by the ground truth attention map 806. In various cases, the training component 602 can update trainable internal parameters of the primary processing channel 204 by performing backpropagation (e.g., stochastic gradient descent) driven by the weighted loss 814. It should be noted that the weighted loss 814 can be implemented regardless of the internal architecture of the primary processing channel 204. In other words, the pixel-wise (or voxel-wise) weighting of the loss function according to the ground truth attention map can be thought of as architecture-agnostic or otherwise plug-and-play.

[0118] Additionally, in various embodiments, the training component 602 can compute a loss 816 between the output 810 and the ground truth attention map 806 via any suitable error or objective function (e.g., MAE, MSE, cross-entropy). In various embodiments, the training component 602 can update trainable internal parameters of the auxiliary processing channel 206 by performing backpropagation (e.g., stochastic gradient descent) driven by the loss 816.

[0119] In various embodiments, the above-described training procedure can be repeated for any suitable number of training medical images (e.g., for each training medical image in the annotated training data set 604). Such training can ultimately cause the trainable internal parameters of the primary processing channel 204 to be iteratively optimized to accurately or correctly perform an inference task on an input medical image. Such training can also ultimately cause the trainable internal parameters of the auxiliary processing channel 206 to be iteratively optimized to accurately or correctly predict how important individual pixels (or voxels) of such input medical images are with respect to an inference task (e.g., to accurately or correctly predict which pixels (or voxels) the primary processing channel 204 will focus on).

[0120] In various cases, the training component 602 can implement any suitable training termination criterion or any suitable training batch sizes when training the deep learning neural network 202.

[0121] In the discussion related to FIG. 8, the primary processing channel 204 and the auxiliary processing channel 206 are primarily described as being trained from scratch (e.g., from random internal parameter initialization) simultaneously with each other. However, this is merely a non-limiting example for ease of illustration and explanation. In various other embodiments, the deep learning neural network 202 may originally include the primary processing channel 204 and lack the auxiliary processing channel 206. In such a case, the training component 602 may train the primary processing channel 204 from scratch (e.g., from random internal parameter initialization) using a set of training medical images 702 and a set of ground truth inference task results 704. It should be noted that such training of the primary processing channel 204 may, in some implementations, include leveraging a set of ground truth attention maps 706, as described with respect to FIG. 8 (e.g., the primary processing channel 204 may be updated via backpropagation driven by a weighted loss 814). However, such training of only the primary processing channel 204 may not involve a set of ground truth attention maps 706 in other embodiments (e.g., the training component 602 may refrain from computing the weighted loss 814, and the primary processing channel 204 may thus be updated via backpropagation driven by the loss 812). Now, after the primary processing channel 204 has been trained in such a manner, the training component 602 may insert the auxiliary processing channel 206 into the deep learning neural network 202. After such insertion, the training component 602 may randomly initialize the trainable internal parameters of the auxiliary processing channel 206. However, since the primary processing channel 204 in such a scenario may have already undergone some training, the training component 602 may refrain from randomly initializing the trainable internal parameters of the primary processing channel 204.The training component 602 can then train both the auxiliary processing channel 206 and the primary processing channel 204 as described with respect to Figure 8. In various embodiments, this can be thought of as training the auxiliary processing channel 206 from scratch while fine-tuning the primary processing channel 204.

[0122] It should be noted that in various aspects, weighting the training outputs (e.g., 808) generated by the primary processing channel 204 according to a set of ground truth attention maps 706 can significantly improve the performance of the primary processing channel 204. Indeed, the inventors of various embodiments described herein have experimentally verified such a performance boost, as described with respect to FIGS.

[0123] 9-11 show exemplary non-limiting experimental results regarding explainable visual attention for deep learning in accordance with one or more embodiments described herein.

[0124] In particular, various embodiments described herein were reduced to practice during various experiments. In such experiments, the inference task was image-to-image regression. More specifically, the inference task was the translation or transformation of low-energy CT scan images to high-energy CT scan images. In such experiments, a first version of the primary processing channel 204 was reduced for practice, and such first version was trained without leveraging a set of ground truth attention maps 706. Also, in such experiments, a second version of the primary processing channel 204 was reduced to practice, and such second version was trained with the aid of a set of ground truth attention maps 706. Such experiments included running both the first and second versions of the primary processing channel 204 on low-energy CT scan images for which high-energy ground truth was available. During such experiments, the second version of the primary processing channel 204 outperformed the first version of the primary processing channel 204. In particular, the second version of the primary processing channel 204 was able to predict the high-energy CT scan image more accurately (e.g., with a higher Dice score) than the first version of the primary processing channel 204. Indeed, FIG. 9 shows a ground truth high-energy image 900 corresponding to a particular low-energy CT scan image. During such an experiment, the first version of the primary processing channel 204 was run on that particular low-energy CT scan image, thereby obtaining a predicted high-energy image 1000, as shown in FIG. 10. Also during such an experiment, the second version of the primary processing channel 204 was run on that particular low-energy CT scan image, thereby obtaining a predicted high-energy image 1100, as shown in FIG. 11. As can be seen, the predicted high-energy image 1100 is much more similar to the ground truth high-energy image 900 than the predicted high-energy image 1000. In other words, weighting the training loss function of the primary processing channel 204 according to a set of ground truth attention maps 706 significantly improved the performance of the primary processing channel 204.

[0125] 12 illustrates a flow diagram of an example, non-limiting, computer-implemented method 1200 that can facilitate explainable visual attention for deep learning in accordance with one or more embodiments described herein. In various cases, the attention system 102 can facilitate the computer-implemented method 1200.

[0126] In various embodiments, act 1202 may include accessing a medical image (e.g., 104) generated by a medical imaging scanner by a device (e.g., via 110) operably coupled to a processor (e.g., 106).

[0127] In various aspects, act 1204 may include performing an inference task on the medical image by the device (e.g., via 112) and through execution of a deep learning neural network (e.g., 202), which may receive the medical image as an input and generate as output both an inference task result (e.g., 208) and an attention map (e.g., 210) indicating which pixels or voxels of the medical image the deep learning neural network attended to in generating the inference task result.

[0128] In various implementations, act 1206 may include visually rendering, by the device (e.g., via 114), the inference task results and the attention map on an electronic display.

[0129] 12, the computer-implemented method 1200 may include training, by the device (e.g., via 602), a deep learning neural network on the annotated training dataset (e.g., 604). In various embodiments, the deep learning neural network may include a first processing channel (e.g., 204) and a second processing channel (e.g., 206). In various embodiments, the annotated training dataset may comprise training medical images (e.g., 802), ground truth inference task results (e.g., 804) corresponding to the training medical images, and ground truth attention maps (e.g., 806) corresponding to the training medical images.In various aspects, the training may include randomly initializing, by the device (e.g., via 602), trainable internal parameters of a first processing channel and trainable internal parameters of a second processing channel; running, by the device (e.g., via 602), a deep learning neural network on the training medical images, thereby causing the first processing channel to generate a first output (e.g., 808) and the second processing channel to generate a second output (e.g., 810); calculating, by the device (e.g., via 602), a first loss (e.g., 812) between a ground truth inference task result and the first output; weighting, by the device (e.g., via 602), the first loss; and weighting, by the device (e.g., via 602), the first loss. , weighting, by the device (e.g., via 602), the first loss via point-wise multiplication with the ground truth attention map (e.g., thereby obtaining 814); weighting, by the device (e.g., via 602), the first loss via point-wise multiplication with the ground truth attention map (e.g., thereby obtaining 814); updating, by the device (e.g., via 602), trainable internal parameters of the first processing channel via backpropagation driven by the weighted first loss (e.g., driven by 814); calculating, by the device (e.g., via 602), a second loss (e.g., 816) between the ground truth attention map and the second output; and updating, by the device (e.g., via 602), the trainable internal parameters of the second processing channel via backpropagation driven by the second loss.

[0130] Although not explicitly shown in FIG. 12, the inference task can be image-to-image regression, and the ground truth attention map can be based on the relationship between the training medical images and the ground truth inference task results (e.g., as described with respect to FIG. 7).

[0131] Although not explicitly shown in FIG. 12, the inference task can be image segmentation or inter-image regression, and the ground truth attention map can be generated via running a feature extractor on training medical images (e.g., as described with respect to FIG. 7).

[0132] Although not explicitly shown in FIG. 12, the inference task can be image segmentation or image-to-image regression, and the ground truth attention map can indicate the pixel-wise or voxel-wise uncertainty or error exhibited by the deep learning neural network with respect to the training medical images during previous training epochs (e.g., as described with respect to FIG. 7).

[0133] Although not explicitly shown in FIG. 12 , the inference task result may be generated by a first processing channel (e.g., 204) of the deep learning neural network, and the attention map may be generated by a second processing channel (e.g., 206) of the deep learning neural network, which may branch off from and be in parallel with at least a portion of the first processing channel (e.g., 204(j+ TIFF2025069057000002.tif44) through 204(n), a device (e.g., via 602) can train a first processing channel from scratch before the second processing channel is inserted into the deep learning neural network, and a device (e.g., via 602) can train a second processing channel from scratch while fine-tuning the first processing channel.

[0134] Although the disclosure herein primarily describes various embodiments as applied to medical images, this is merely a non-limiting example for ease of illustration and explanation, in various cases the various embodiments described herein may be applied to promote visual attention to any suitable image, even non-medical images.

[0135] Various embodiments described herein may include a computer program product for facilitating explainable visual attention for deep learning. In various aspects, the computer program product may include a non-transitory computer-readable memory (e.g., 108) having program instructions embodied therein. In various aspects, the program instructions may be executable by a processor (e.g., 106) to cause the processor to access a deep learning neural network (e.g., 202) including a primary processing channel (e.g., 204), which may be trained to perform an inference task using a set of training images (e.g., 702) and a set of ground truth inference task results (e.g., 704) each corresponding to the set of training images. In various aspects, the program instructions may be further executable to cause the processor to insert an auxiliary processing channel (e.g., 206) into the deep learning neural network such that the auxiliary processing channel branches off from the primary processing channel and is in parallel with at least a portion of the primary processing channel. In various aspects, the program instructions may be further executable to cause the processor to train the auxiliary processing channel using a set of ground truth attention maps (e.g., 706) each corresponding to a set of training images to generate pixel-wise or voxel-wise attention maps that indicate where the deep learning neural network will focus in the input images when performing an inference task. In various aspects, the set of ground truth attention maps may be based on a relationship between the set of training images and the set of ground truth inference task results. In various other aspects, the set of ground truth attention maps may be generated via execution of a feature extractor on the set of training images. In still other aspects, the set of ground truth attention maps may indicate which regions of the set of training images the primary processing channel analyzed unsurely or incorrectly in one or more previous training epochs, respectively.

[0136] As described herein and shown at least in FIG. 4, a sinogram can be considered a type of image.

[0137] In various embodiments, the machine learning algorithms or models may be implemented in any suitable manner to facilitate any suitable aspects described herein. To facilitate some of the above machine learning aspects of the various embodiments, consider the following discussion of artificial intelligence (AI). Various embodiments described herein may employ artificial intelligence to facilitate automation of one or more features or functionality. Components may employ various AI-based schemes to implement various embodiments / examples disclosed herein. To provide or assist in many of the decisions described herein (e.g., determine, ascertain, infer, calculate, predict, prognose, estimate, derive, forecast, detect, compute), components described herein may examine the entirety or a subset of the data to which they are granted access and may provide or determine an inference about the state of a system or environment from a set of observations captured via events or data. Decisions may be employed to identify a particular context or action, or may generate a probability distribution over states, for example. Decisions may be probabilistic; that is, to calculate a probability distribution over states of interest based on a consideration of data and events. Decision can also refer to the techniques employed to compose higher-level events from a set of events or data.

[0138] Such determinations may result in constructing new events or actions from a set of observed events or stored event data, whether the events are closely correlated in time, and whether the events and data come from one or more event and data sources. The components disclosed herein may employ a variety of classification (explicitly trained (e.g., via training data) as well as implicitly trained (e.g., via observing behavior, preferences, historical information, receiving extrinsic information, etc.)) schemes or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in connection with performing automated or determined actions in connection with the claimed subject matter. Thus, the classification schemes or systems may be used to automatically learn and perform numerous functions, actions, or decisions.

[0139] A classifier is an input attribute vector z = (z1, z2, z3, z4, z n) can be mapped to a confidence that the input belongs to the class, such as f(z)=confidence(class). Such classification can employ probabilistic or statistical analysis (e.g., considering the utility and cost of the analysis) to determine what action to take automatically. Support vector machines (SVMs) are one example of a classifier that can be employed. SVMs work by finding a hyper-surface in the space of possible inputs, which attempts to split the triggering criteria and non-triggering events. Intuitively, this makes the classification correct for test data that is close but not identical to the training data. Other directed and undirected model classification approaches include, for example, naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, or probabilistic classification models providing different patterns of independence, any of which can be employed. Classification as used herein also is inclusive of statistical regression that is utilized to develop models of priority.

[0140] To provide additional context for the various embodiments described herein, Figure 13 and the following discussion are intended to provide a brief, general description of a suitable computing environment 1300 in which various embodiments of the embodiments described herein may be implemented. Although the embodiments have been described above in the general context of computer-executable instructions executable on one or more computers, those skilled in the art will recognize that the embodiments may also be implemented in combination with other program modules, or as a combination of hardware and software.

[0141] Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the methods of the present invention can be practiced on other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

[0142] The illustrated embodiments may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0143] A computing device typically includes various media, which may include computer-readable storage media, machine-readable storage media, or communication media, and these two terms are used differently in this specification as follows. A computer-readable storage medium or machine-readable storage medium can be any available storage medium that can be accessed by a computer, including both volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or machine-readable storage medium can be implemented in relation to any method or technology for storing information, such as computer-readable or machine-readable instructions, program modules, structured or unstructured data, etc.

[0144] A computer readable storage medium may include, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CDROM), digital versatile disk (DVD), Blu-ray disk (BD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, solid state drive or other solid state storage device, or other tangible or non-transitory medium that can be used to store the desired information. In this regard, the terms "tangible" or "non-transitory" as applied to storage, memory, or computer readable medium are understood herein to be qualifiers excluding only ephemeral signals that propagate themselves, and are not intended to waive all claims to all standard storage, memory, or computer readable media that are not only ephemeral signals that propagate themselves.

[0145] The computer-readable storage medium can be accessed by one or more local or remote computing devices for various operations on the information stored by the medium, for example, via access requests, queries or other data retrieval protocols.

[0146] Communication media typically embodies computer-readable instructions, data structures, program modules, or other structured or unstructured data in a data signal, such as a modulated data signal, e.g., carrier wave or other transport mechanism, and includes any information delivery or transport media. The term "modulated data signal" or signal refers to a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal or signals. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0147] 13, an exemplary environment 1300 for implementing various embodiments of the aspects described herein includes a computer 1302 including a processing unit 1304, a system memory 1306, and a system bus 1308. The system bus 1308 couples system components including, but not limited to, the system memory 1306 to the processing unit 1304. The processing unit 1304 may be any of a variety of commercially available processors. Dual microprocessors and other multi-processor architectures may also be employed as the processing unit 1304.

[0148] The system bus 1308 can be any of several types of bus structures that can be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of bus architectures available commercially. The system memory 1306 includes a ROM 1310 and a RAM 1312. A basic input / output system (BIOS) can be stored in a non-volatile memory such as a ROM, an erasable programmable read-only memory (EPROM), or an EEPROM, and the BIOS contains the basic routines that help transfer information between elements within the computer 1302, such as during start-up. The RAM 1312 can also include a high-speed RAM, such as a static RAM, for caching data.

[0149] The computer 1302 further includes an internal hard disk drive (HDD) 1314 (e.g., EIDE, SATA), one or more external storage devices 1316 (e.g., a magnetic floppy disk drive (FDD) 1316, a memory stick or flash drive reader, a memory card reader, etc.), and a drive 1320, e.g., a solid state drive, optical disk drive, etc., that can read from and write to a disk 1322, such as a CD-ROM disk, DVD, BD, etc. Alternatively, where a solid state drive is involved, the disk 1322 is not included unless separate. Although the internal HDD 1314 is illustrated as being disposed within the computer 1302, the internal HDD 1314 may also be configured for external use within a suitable enclosure (not shown). Additionally, although not shown in the environment 1300, a solid state drive (SSD) may also be used in addition to or in place of the HDD 1314. HDD 1314, external storage device(s) 1316, and drives 1320 may be connected to system bus 1308 by HDD interface 1324, external storage interface 1326, and drive interface 1328, respectively. Interface 1324 for external drive implementations may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are within the contemplation of the embodiments described herein.

[0150] The drives and associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. To the computer 1302, the drives and storage media accommodate the storage of any data in a suitable digital format. Although the above description of computer-readable storage media refers to each type of storage device, those skilled in the art should understand that other types of computer-readable storage media, whether currently existing or developed in the future, may also be used in the exemplary operating environment, and further, any such storage media may include computer-executable instructions for performing the methods described herein.

[0151] A number of program modules can be stored in the drives and RAM 1312, including an operating system 1330, one or more application programs 1332, other program modules 1334, and program data 1336. All or portions of the operating system, applications, modules, or data may also be cached in RAM 1312. The systems and methods described herein can be implemented using various commercially available operating systems or combinations of operating systems.

[0152] The computer 1302 may optionally configure emulation techniques. For example, a hypervisor (not shown) or other intermediary may emulate a hardware environment for the operating system 1330, and the emulated hardware may optionally be different from the hardware illustrated in FIG. 13. In such an embodiment, the operating system 1330 may configure one virtual machine (VM) of multiple VMs hosted on the computer 1302. Additionally, the operating system 1330 may provide a runtime environment, such as the Java Runtime Environment or the .NET Framework, for the application 1332. The runtime environment is a consistent execution environment that allows the application 1332 to run on any operating system that includes the runtime environment. Similarly, the operating system 1330 may support containers, and the application 1332 may be in the form of a container. A container is a lightweight, standalone, executable package of software that includes, for example, code, runtime, system tools, system libraries, and settings for an application.

[0153] Additionally, computer 1302 can have enabled security modules, such as a Trusted Processing Module (TPM). For example, with a TPM, a boot component will hash the next boot component in time and wait for the result to match a secure value before loading the next boot component. This process can be applied at any layer of the computer 1302's code execution stack, for example, at the application execution level or at the operating system (OS) kernel level, thereby enabling security at any level of code execution.

[0154] A user can enter commands and information into the computer 1302 through one or more wired / wireless input devices, such as a keyboard 1338, a touch screen 1340, and a pointing device such as a mouse 1342. Other input devices (not shown) can include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller or headset, a game pad, a stylus pen, an image input device such as a camera(s), a gesture sensor input device, a visual motion sensor input device, an emotion or face detection device, a biometric input device such as a fingerprint or iris scanner, and the like. These and other input devices are often connected to the processing unit 1304 through an input device interface 1344 that may be coupled to the system bus 1308, but may be connected by other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, a BLUETOOTH interface, and the like.

[0155] A monitor 1346 or other type of display device can also be connected to the system bus 1308 via an interface, such as a video adapter 1348. In addition to the monitor 1346, computers typically include other peripheral output devices (not shown), such as speakers, printers, etc.

[0156] The computer 1302 can operate in a networked environment using logical connections via wired or wireless communications to one or more remote computers, such as a remote computer(s) 1350. The remote computer 1350 can be a workstation, a server computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment appliance, a peer device, or other common network node, and typically includes many or all of the elements described in connection with the computer 1302, although for purposes of brevity, only a memory / storage device 1352 is illustrated. The logical connections illustrated include wired / wireless connections to a local area network (LAN) 1354 or a larger network, e.g., a wide area network (WAN) 1356. Such LAN and WAN networking environments are commonplace in offices and companies, facilitating enterprise-wide computer networks, such as an intranet, all of which may be connected to a global communications network, e.g., the Internet.

[0157] When used in a LAN networking environment, the computer 1302 can be connected to the local network 1354 via a wired or wireless communication network interface or adapter 1358. The adapter 1358 can facilitate wired or wireless communication to the LAN 1354, and the LAN 1354 can also include a wireless access point (AP) disposed thereon for communicating with the adapter 1358 in a wireless mode.

[0158] When used in a WAN networking environment, the computer 1302 may include a modem 1360 or may be connected to a communications server on the WAN 1356 via other means for establishing communications over the WAN 1356, such as via the Internet. The modem 1360 may be internal or external, a wired or wireless device, and may be connected to the system bus 1308 via the input device interface 1344. In a networked environment, program modules depicted relative to the computer 1302, or portions thereof, may be stored in the remote memory / storage device 1352. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0159] When used in either a LAN or WAN networking environment, the computer 1302 may access a cloud storage system or other network-based storage systems in addition to or instead of the external storage device 1316 described above, such as, but not limited to, a networked virtual machine that provides one or more aspects of information storage or processing. In general, a connection between the computer 1302 and the cloud storage system may be established via the LAN 1354 or WAN 1356, such as by an adapter 1358 or modem 1360, respectively. Upon connecting the computer 1302 to the associated cloud storage system, the external storage interface 1326, with the aid of the adapter 1358 or modem 1360, may manage the storage provided by the cloud storage system as well as other types of external storage. For example, the external storage interface 1326 may be configured to provide access to cloud storage sources as if those sources were physically connected to the computer 1302.

[0160] The computer 1302 may be operable to communicate with any wireless device or entity operatively arranged for wireless communication, such as a printer, a scanner, a desktop or portable computer, a portable data assistant, a communications satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, a newsstand, a store shelf, etc.), and a telephone. This includes Wireless Fidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, communication can be of a predefined structure, such as a traditional network, or simply ad-hoc communication between at least two devices.

[0161] FIG. 14 is a schematic block diagram of a sample computing environment 1400 with which the disclosed subject matter can interact. The exemplary computing environment 1400 includes one or more client(s) 1410. The client(s) 1410 can be hardware or software (e.g., threads, processes, computing devices). The sample computing environment 1400 also includes one or more server(s) 1430. The server(s) 1430 can also be hardware or software (e.g., threads, processes, computing devices). The server(s) 1430 can house threads for performing transformations, for example, by employing one or more embodiments as described herein. One possible communication between the client(s) 1410 and the server(s) 1430 can be in the form of a data packet adapted to be transmitted between two or more computer processes. The sample computing environment 1400 includes a communication framework 1450 that can be employed to facilitate communication between the client(s) 1410 and the server(s) 1430. The client(s) 1410 are operably connected to one or more client data store(s) 1420 that can be employed to store information local to the client(s) 1410. Similarly, the server(s) 1430 are operably connected to one or more server data store(s) 1440 that can be employed to store information local to the servers 1430.

[0162] Various embodiments may be systems, methods, devices, or computer program products at any possible level of technical detail of integration. A computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to execute aspects of the various embodiments. A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media may also include: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media is not to be construed as being ephemeral signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0163] The computer readable program instructions described herein can be downloaded to each computing / processing device from a computer readable storage medium or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network. The network can be composed of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing / processing device. The computer readable program instructions for performing the operations of the various embodiments can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language, or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can utilize state information of the computer readable program instructions to execute computer readable program instructions and personalize the electronic circuitry to perform various aspects.

[0164] Various aspects are described herein with reference to flowchart or block diagrams of methods, apparatus (systems), and computer program products according to various embodiments. It will be understood that each block of the flowchart or block diagrams, and combinations of blocks in the flowchart or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the function / act specified in the flowchart or block diagram block or blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device to function in a particular manner, and in such a way that the computer-readable storage medium having instructions stored therein constitutes an article of manufacture including instructions implementing aspects of the function / act specified in the flowchart or block diagram block or blocks. The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and executed on the computer, other programmable apparatus, or other device to generate a sequence of operational acts on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the function / acts specified in the flowchart or block diagram block or blocks.

[0165] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions constituting one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams or flowchart diagrams, and combinations of blocks in the block diagrams or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs the specified functions or acts, or executes a combination of special purpose hardware and computer instructions.

[0166] Although the subject matter has been described above in the general context of computer-executable instructions for a computer or computer program product executing on a computer, those skilled in the art will recognize that the present disclosure can also be implemented or can be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that various aspects can be implemented with other computer system configurations, including single-processor or multi-processor computer systems, mini-computing devices, mainframe computers, as well as computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronics, and the like. The illustrated aspects can also be implemented in distributed computing environments where tasks are performed by remote processing devices linked through a communications network. However, some, if not all aspects of the present disclosure can be implemented on stand-alone computers. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0167] As used herein, terms such as "component," "system," "platform," "interface," and the like, may refer to or include computer-related entities or entities related to operating machines having one or more specific functionalities. The entities disclosed herein may be either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, or a computer. By way of illustration, both an application running on a server and the server may be components. One or more components may reside within a process or thread of execution, and a component may be localized on one computer or distributed among two or more multiple computers. In another example, each component may execute from various computer-readable media having various data structures stored thereon. The components may communicate via local or remote processes, such as following a signal having one or more data packets (e.g., data from one component interacting with another system over a network such as the Internet via a signal, whether local or distributed). As another example, a component may be a device having a particular function provided by mechanical parts operated by electrical or electronic circuitry operated by a software or firmware application executed by a processor. In such a case, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component may be a device that provides a particular functionality through electronic parts without mechanical parts, where the electronic parts may include a processor or other means for executing software or firmware that provides at least a portion of the functionality of the electronic parts.In one aspect, the components may emulate electronic components, for example, via a virtual machine in a cloud computing system.

[0168] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A, if X employs B, or if X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. As used herein, the term "and / or" is intended to have the same meaning as "or." Furthermore, the articles "a" and "an," as used in this specification and the accompanying drawings, should generally be construed to mean "one or more," unless otherwise specified or clear from the context to be directed to the singular form. As used herein, the term "example" or "exemplary" is used to mean serving as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited by such examples. Additionally, any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, and is not meant to exclude equivalent exemplary structures and techniques known to those skilled in the art.

[0169] In the present disclosure, non-limiting examples are described. For ease of explanation or description, various parts of the present disclosure utilize the terms "each", "every", or "all" when discussing various examples. Such usage of the terms "each", "every", or "all" is non-limiting. In other words, when the present disclosure provides an explanation that applies to "each", "every", or "all" of a particular object or component, this should be understood as a non-limiting example, and it should be further understood that in various other examples, such an explanation may apply to less than "each", "every", or "all" of the particular object or component.

[0170] As used herein, the term "processor" may refer to substantially any computing processing unit or device, including, but not limited to, a single core processor, a single processor with software multithreaded execution, a multicore processor, a multicore processor with software multithreaded execution, a multicore processor with hardware multithreading technology, a parallel platform, and a parallel platform with distributed shared memory. Additionally, a processor may refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Additionally, a processor may utilize nanoscale architectures, including, but not limited to, molecular or quantum dot based transistors, switches, and gates to optimize space usage or improve performance of user equipment. A processor may also be implemented as a combination of computing processing units. In this disclosure, terms such as "memory," "storage," "data storage," "database," and substantially any other information storage component associated with the operation and functionality of a component are utilized to refer to a "memory component," an entity embodied in a memory, or a component that comprises a memory. It should be understood that the memory or memory components described herein can be either volatile memory or non-volatile memory, or can include both volatile and non-volatile memory.By way of example and not limitation, non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). Volatile memory may include, for example, a RAM acting as an external cache memory. By way of example and not limitation, RAM is available in many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Additionally, it is intended that the memory components of the systems or computer-implemented methods disclosed herein include, without being limited to, these and any other suitable types of memory.

[0171] What has been described above includes merely examples of systems and computer-implemented methods. Of course, for purposes of describing this disclosure, it is not possible to describe every conceivable combination of components or computer-implemented methods, but many further combinations and permutations of the present disclosure are possible. Moreover, to the extent that terms such as "comprising," "containing," "having," and the like are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a similar manner as the term "comprising" is interpreted when employed as a transitional word in the claims.

[0172] The description of various embodiments has been presented for illustrative purposes, but is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will become apparent without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0173] Further aspects of the invention are provided by the subject matter of the following clauses. [Embodiment 1] 1. A system comprising: The present invention includes a processor that executes computer-executable components stored in a non-transitory computer-readable memory, the computer-executable components including: an access component that accesses a medical image generated by a medical imaging scanner; an execution component that performs, via execution of a deep learning neural network, an inferencing task on the medical image; The system includes a deep learning neural network that receives as input the medical image and produces as output both an inferencing task result and an attention map indicating on which pixels or voxels of the medical image the deep learning neural network focused in generating the inferencing task result. [Embodiment 2] The computer executable components further include: The system of any preceding embodiment, further comprising a display component that visually renders the inferencing task result and the attention map on an electronic display. [Embodiment 3] The computer executable components further include:

[0023] The system of any preceding embodiment, further comprising a training component that trains the deep learning neural network on an annotated training dataset. [Embodiment 4] the deep learning neural network comprises a first processing channel and a second processing channel, the annotated training dataset comprises a training medical image, a ground-truth inferencing task result corresponding to the training medical image, and a ground-truth attention map corresponding to the training medical image, and the training component comprises: randomly initializes trainable internal parameters of the first processing channel and of the second processing channel; executing the deep learning neural network on the training medical image, thereby causing the first processing channel to produce a first output and causing the second processing channel to produce a second output; computes a first loss between the ground-truth inferencing task result and the first output; weights the first loss via point-wise multiplication with the ground-truth attention map; updates the trainable internal parameters of the first processing channel, via backpropagation driven by the weighted first loss; computes a second loss between the ground-truth attention map and the second output; updates the trainable internal parameters of the second processing channel, via backpropagation driven by the second loss; The system of any preceding embodiment, [Embodiment 5] The system of any preceding embodiment, wherein the inferencing task is image-to-image regression, and the ground-truth attention map is based on a relation between the training medical image and the ground-truth inferencing task result. [Embodiment 6] The system of any preceding embodiment, wherein the inferencing task is image segmentation or image-to-image regression, and the ground-truth attention map is generated via execution of a feature extractor on the training medical image. [Embodiment 7] The system of any preceding embodiment, wherein the inferencing task is image segmentation or image-to-image regression, and the ground-truth attention map indicates pixel-wise or voxel-wise uncertainties exhibited by the deep learning neural network with respect to the training medical image during a previous training epoch. [Embodiment 8] The inferencing task result is produced by a first processing channel of the deep learning neural network, the attention map is produced by a second processing channel of the deep learning neural network, the second processing channel branches off from and is parallel to at least a portion of the first processing channel, the training component trains, from a first random internal parameter initialization, the first processing channel prior to the second processing channel being inserted into the deep learning neural network, and the training component trains, from a second random internal parameter initialization while fine-tuning the first processing channel. initialization,the second processing channel while fine-tuning the first processing channel. [Embodiment 9] 1. A computer-implemented method comprising: accessing, by a device operatively coupled to a processor, a medical image generated by a medical imaging scanner; performing, by the device and via execution of a deep learning neural network, an inferencing task on the medical image; the deep learning neural network receives as input the medical image and produces as output both an inferencing task result and an attention map indicating on which pixels or voxels of the medical image the deep learning neural network focused in generating the inferencing task result. [Embodiment 10] 2. The computer-implemented method of any preceding embodiment, comprising visually rendering, by the device, the inferencing task result and the attention map on an electronic display. [Embodiment 11] 11. The computer-implemented method of any preceding embodiment, comprising training, by the device, the deep learning neural network on an annotated training dataset. [Embodiment 12] the deep learning neural network comprises a first processing channel and a second processing channel, the annotated training dataset comprises a training medical image, a ground-truth inferencing task result corresponding to the training medical image, and a ground-truth attention map corresponding to the training medical image, and the training comprises: randomly initializing, by the device, trainable internal parameters of the first processing channel and of the second processing channel; executing, by the device, the deep learning neural network on the training medical image, thereby causing the first processing channel to produce a first output and causing the second processing channel to produce a second output; computing, by the device, a first loss between the ground-truth inferencing task result and the first output; The device weights the first loss via point-wise multiplication with the ground-truth attention map; and updating, by the device, the trainable internal parameters of the first processing channel, via backpropagation driven by the weighted first loss; computing, by the device, a second loss between the ground-truth attention map and the second output; updating, by the device, the trainable internal parameters of the second processing channel, via backpropagation driven by the second loss. [Embodiment 13] 2. The computer-implemented method of claim 1, wherein the inferencing task is image-to-image regression and the ground-truth attention map is based on a relation between the training medical image and the ground-truth inferencing task result. [Embodiment 14] 2. The computer-implemented method of claim 1, wherein the inferencing task is image segmentation or image-to-image regression, and the ground-truth attention map is generated via execution of a feature extractor on the training medical image. [Embodiment 15] 11. The computer-implemented method of any preceding embodiment, wherein the inferencing task is image segmentation or image-to-image regression, and the ground-truth attention map indicates pixel-wise or voxel-wise uncertainties exhibited by the deep learning neural network with respect to the training medical image during a previous training epoch. [Embodiment 16] The inferencing task result is produced by a first processing channel of the deep learning neural network, the attention map is produced by a second processing channel of the deep learning neural network, the second processing channel branches off from and is parallel to at least a portion of the first processing channel, the device trains, from a first random internal parameter initialization, the first processing channel prior to the second processing channel being inserted into the deep learning neural network, and the device trains, from a second random internal parameter initialization,the second processing channel while fine-tuning the first processing channel. [Embodiment 17] A computer program product for facilitating explainable visual attention for deep learning, the computer program product comprising a non-transitory computer-readable memory having program instructions embodied therewith, the program instructions comprising: accessing a deep learning neural network comprising a primary processing channel, the primary processing channel is trained, using a set of training images and a set of ground-truth inferencing task results respectively corresponding to the set of training images, to perform an inferencing task; inserting an auxiliary processing channel into the deep learning neural network, such that the auxiliary processing channel branches off from, and is parallel to at least a portion of, the primary processing channel; train, using a set of ground-truth attention maps respectively corresponding to the set of training images, the auxiliary processing channel to produce pixel-wise or voxel-wise attention maps indicating where within inputted images the deep learning neural network focuses when performing the inferencing task; A computer program product executable by a processor to cause the processor to execute the [Embodiment 18] The computer program product of any preceding embodiment, wherein the set of ground-truth attention maps are based on relations between the set of training images and the set of ground-truth inferencing task results. [Embodiment 19] A computer program product as described in any preceding embodiment, wherein the set of ground-truth attention maps are generated via execution of a feature extractor on the set of training images. [Embodiment 20] The computer program product of any preceding embodiment, wherein the set of ground-truth attention maps respectively indicate which regions of the set of training images the primary processing channel incorrectly analyzed in one or more previous training epochs. [Explanation of symbols]

[0174] 100, 200: System 102: Attention system 104: Medical image 106: Processor 108: Non-transient computer readable memory 110: Access component 112: Execution component 114: Display component 202: Deep learning neural network 204: Primary processing channel 206: Auxiliary processing channel 208: Inference task result 210: Attention map 602: Training component 604: Annotated training dataset 702: Training medical images 704: Ground truth inference task result 706: Ground truth attention map 802: Training medical images 804: Ground truth inference task result 806: Ground truth attention map 808: First output 810: Second output 812: First loss 816: Second loss 814: Weighted loss 900: Ground truth high energy image 1000:Predicted high energy image 1100:Predicted high energy image 1300:Computing environment 1302:Computer 1304:Processing unit 1306:System memory 1308:System bus 1310:ROM 1312:RAM 1314:Hard disk drive (HDD) 1316:External storage device 1320:Drive 1322:Disk 1324, 1326, 1328:Interface 1330:Operating system 1332:Application program 1334:Program module 1336:Program data 1338:Keyboard 1340:Touch screen 1342:Mouse 1344:Input device interface 1346:Monitor 1348:Video adapter 1350:Remote computer 1352:Remote memory device / remote storage device 1354:Local network / LAN 1356:WAN 1358: network adapter 1360: modem 1400: computing environment 1410: client 1420: client data store 1430: server 1440: server data store 1450: communication framework

Claims

1. 1. A system comprising: A processor executing computer-executable components stored in a non-transitory computer-readable memory, the computer-executable components comprising: The executable components are an access component for accessing medical images generated by the medical imaging scanner; an execution component that performs inference tasks on the medical images through execution of a deep learning neural network; The system, wherein the deep learning neural network receives the medical image as an input and generates as output both an inference task result and an attention map indicating which pixels or voxels of the medical image the deep learning neural network focused on in generating the inference task result.

2. The system of claim 1 , wherein the computer-executable components further comprise a display component that visually displays the inference task results and the attention map on an electronic display.

3. 2. The system of claim 1, wherein the computer-executable components further comprise a training component that trains the deep learning neural network with an annotated training data set.

4. The deep learning neural network includes a first processing channel and a second processing channel, the annotated training dataset includes training medical images, ground truth inference task results corresponding to the training medical images, and ground truth attention maps corresponding to the training medical images, and the training component includes: Randomly initializing trainable internal parameters of a first processing channel and a second processing channel; running the deep learning neural network on the training medical images to cause the first processing channel to generate a first output and the second processing channel to generate a second output; calculating a first loss between the ground truth inference task result and the first output; weighting the first loss by point-wise multiplication with the ground truth attention map; updating the trainable internal parameters of the first processing channel by backpropagation driven by the weighted first loss; computing a second loss between the ground truth attention map and the second output; updating the trainable internal parameters of the second processing channel by backpropagation driven by the second loss; The system of claim 3 , further comprising:

5. The system of claim 4 , wherein the inference task is image-to-image regression, and the ground truth attention map is based on relationships between the training medical images and the ground truth inference task results.

6. The system of claim 4 , wherein the inference task is image segmentation or inter-image regression, and the ground truth attention map is generated by running a feature extractor on the training medical images.

7. 5. The system of claim 4, wherein the inference task is image segmentation or inter-image regression, and the ground truth attention map indicates pixel-wise or voxel-wise uncertainty exhibited by the deep learning neural network with respect to the training medical images during previous training epochs.

8. 4. The system of claim 3, wherein the inference task result is generated by a first processing channel of the deep learning neural network and the attention map is generated by a second processing channel of the deep learning neural network, the second processing channel branching off from at least a portion of the first processing channel and being in parallel with the first processing channel, the training component training the first processing channel from a first random internal parameter initialization before the second processing channel is inserted into the deep learning neural network, and the training component training the second processing channel from a second random internal parameter initialization while fine-tuning the first processing channel.

9. 1. A computer-implemented method comprising: accessing, by a device operatively coupled to the processor, a medical image generated by the medical imaging scanner; and performing, by said device and through execution of a deep learning neural network, an inference task on said medical image; The deep learning neural network receives the medical image as an input and generates as output both an inference task result and an attention map indicating pixels or voxels of the medical image on which the deep learning neural network is focused to generate the inference task result.

10. 10. The computer-implemented method of claim 9, comprising visually rendering the inference task results and the attention map by the device on an electronic display.

11. 10. The computer-implemented method of claim 9, comprising training the deep learning neural network at the device with an annotated training data set.

12. The deep learning neural network includes a first processing channel and a second processing channel, the annotated training dataset includes training medical images, ground truth inference task results corresponding to the training medical images, and ground truth attention maps corresponding to the training medical images, and the training includes: randomly initializing, by the device, trainable internal parameters of the first processing channel and the second processing channel; running, with the device, the deep learning neural network on the training medical images to cause the first processing channel to generate a first output and the second processing channel to generate a second output; calculating, by the device, a first loss between the ground truth inference task result and the first output; The device weights the first loss by point-wise multiplication with the ground truth attention map; updating, by the device, the trainable internal parameters of the first processing channel via backpropagation driven by the weighted first loss; calculating, by the device, a second loss between the ground truth attention map and the second output; and updating, by the device, the trainable internal parameters of the second processing channel via backpropagation driven by the second loss.

13. 13. The computer-implemented method of claim 12, wherein the inference task is image-to-image regression, and the ground truth attention map is based on relationships between the training medical images and the ground truth inference task results.

14. 13. The computer-implemented method of claim 12, wherein the inference task is image segmentation or inter-image regression and the ground truth attention map is generated via running a feature extractor on the training medical images.

15. 13. The computer-implemented method of claim 12, wherein the inference task is image segmentation or inter-image regression, and the ground truth attention map indicates pixel-wise or voxel-wise uncertainty exhibited by the deep learning neural network with respect to the training medical images during previous training epochs.

16. 12. The computer-implemented method of claim 11, wherein the inference task result is generated by a first processing channel of the deep learning neural network, the attention map is generated by a second processing channel of the deep learning neural network, the second processing channel branching off from at least a portion of the first processing channel and in parallel with the first processing channel, the device training the first processing channel from a first random internal parameter initialization before the second processing channel is inserted into the deep learning neural network, and the device training the second processing channel from a second random internal parameter initialization while fine-tuning the first processing channel.

17. 1. A computer program product for facilitating explainable visual attention for deep learning, the computer program product including: a non-transitory computer readable memory having program instructions embodied therein, the program instructions comprising: accessing a deep learning neural network including a primary processing channel, the primary processing channel being trained to perform an inference task using a set of training images and a set of ground truth inference task results each corresponding to the set of training images; inserting the auxiliary processing channel into the deep learning neural network such that the auxiliary processing channel branches off from and is in parallel with at least a portion of the primary processing channel; training the auxiliary processing channel to generate pixel-wise or voxel-wise attention maps that indicate where the deep learning neural network should focus in input images when performing an inference task, using a set of evidence attention maps that respectively correspond to the set of training images; A computer program product executable by a processor to cause the processor to execute the

18. 20. The computer program product of claim 17, wherein the set of ground truth attention maps is based on relationships between the set of training images and the set of ground truth inference task results.

19. 20. The computer program product of claim 17, wherein the set of ground truth attention maps is generated via running a feature extractor on the set of training images.

20. 20. The computer program product of claim 17, wherein each of the set of ground truth attention maps indicates which regions of a set of training images the primary processing channel incorrectly analyzed in one or more previous training epochs.

Images