Interpretable visual attention for deep learning

Abstract

Systems or techniques are provided that facilitate interpretable visual attention for deep learning. In various embodiments, a system may access medical images (e.g., 104) generated by a medical imaging scanner. In various aspects, the system may perform an inference task on the medical image via execution of a deep learning neural network (e.g., 202). In various examples, the deep learning neural network may receive the medical image as input and may generate, as output, both an inference task result (e.g., 208) and an attention map (e.g., 210) indicating which pixels or voxels of the medical image the deep learning neural network focuses on when the inference task result is generated.

Description

Technical Field

[0001] The subject disclosure relates generally to deep learning, and more particularly to explainable visual attention for deep learning. Background Art

[0002] Deep learning neural networks can be trained to perform inference tasks on medical images. When performing inference tasks, the deep learning neural network can be expected to pay closer attention to certain parts of the medical image than to other parts of the medical image. Unfortunately, existing techniques for facilitating such attention are not adequately described or explained.

[0003] Therefore, systems or techniques that can facilitate deep learning attention on medical images with enhanced explainability or interpretability may be desirable. Summary of the invention

[0004] The following is a summary of the invention to provide a basic understanding of one or more embodiments. The summary is not intended to identify key or important elements, nor is it intended to delineate any scope of a specific embodiment or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to a more detailed description presented later. In one or more embodiments described herein, an apparatus, system, computer-implemented method, device, or computer program product that facilitates interpretable visual attention for deep learning is described.

[0005] According to one or more embodiments, a system is provided. The system may include a non-transitory computer readable memory that may store computer executable components. The system may also include a processor that may be operably coupled to the non-transitory computer readable memory and may execute computer executable components stored in the non-transitory computer readable memory. In various embodiments, the computer executable component may include an access component that may access a medical image generated by a medical imaging scanner. In various aspects, the computer executable component may include an execution component that may perform an inference task on the medical image via execution of a deep learning neural network. In various instances, the deep learning neural network may receive the medical image as input and may generate both an inference task result and an attention map as output, the attention map indicating which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

[0006] According to one or more embodiments, a computer-implemented method is provided. In various embodiments, the computer-implemented method may access a medical image generated by a medical imaging scanner by a device operatively coupled to a processor. In various aspects, the computer-implemented method may include performing an inference task on the medical image by the device and via execution of a deep learning neural network. In various instances, the deep learning neural network may receive the medical image as input and may generate both an inference task result and an attention map as output, the attention map indicating which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

[0007] According to one or more embodiments, a computer program product for facilitating interpretable visual attention for deep learning is provided. In various embodiments, the computer program product may include a non-transitory computer-readable memory having program instructions embodied therein. In various aspects, the program instructions may be executable by a processor so that the processor accesses a deep learning neural network including a main processing channel to perform an inference task, wherein the main processing channel can be trained using a set of training images and a set of true value inference task results corresponding to the set of training images. In various instances, the program instructions may be further executable so that the processor inserts an auxiliary processing channel into the deep learning neural network so that the auxiliary processing channel branches out from the main processing channel and is parallel to at least a portion of the main processing channel. In various cases, the program instructions may be further executable so that the processor uses a set of true value attention maps corresponding to the set of training images to train the auxiliary processing channel to generate a pixel-level or voxel-level attention map, which indicates where the deep learning neural network focuses on within the input image when performing the inference task. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A block diagram of an example non-limiting system that facilitates explainable visual attention for deep learning in accordance with one or more embodiments described herein is illustrated.

[0009] Figure 2 A block diagram of an example non-limiting system that facilitates explainable visual attention for deep learning according to one or more embodiments described herein is illustrated, the system comprising a deep learning neural network, inference task results, and an attention map.

[0010] Figure 3 An example non-limiting block diagram illustrating how a deep learning neural network may generate inference task results and attention maps according to one or more embodiments described herein is illustrated.

[0011] Figures 4 to 5 Illustrated are example non-limiting examples of medical images and corresponding attention maps in accordance with one or more embodiments described herein.

[0012] Figure 6 A block diagram of an example non-limiting system for facilitating explainable visual attention for deep learning according to one or more embodiments described herein is illustrated, the system including a training component and an annotated training dataset.

[0013] Figure 7 An example non-limiting block diagram of an annotated training dataset according to one or more embodiments described herein is illustrated.

[0014] Figure 8 An example non-limiting block diagram showing how a deep learning neural network may be trained is illustrated in accordance with one or more embodiments described herein.

[0015] Figures 9 to 11 Example non-limiting experimental results on explainable visual attention for deep learning according to one or more embodiments described herein are illustrated.

[0016] Fig.12 A flow chart illustrating an example non-limiting computer-implemented method that facilitates explainable visual attention for deep learning according to one or more embodiments described herein.

[0017] Fig.13 A block diagram illustrating an example non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0018] Fig.14 An example networking environment is illustrated that is operable to perform various implementations described herein. DETAILED DESCRIPTION

[0019] The following detailed description is merely illustrative and is not intended to limit the embodiments or the application / use of the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" or "invention content" section or "detailed description" section.

[0020] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0021] A deep learning neural network may be trained (e.g., in a supervised manner, in an unsupervised manner, in a reinforcement learning manner) to perform an inference task (e.g., classification, segmentation, regression) on a medical image (e.g., a computed tomography (CT) scan image, a magnetic resonance imaging (MRI) scan image, an X-ray scan image, an ultrasound scan image, a positron emission tomography (PET) scan image). During or otherwise to facilitate the performance of the inference task, the deep learning neural network may be expected to pay closer attention to certain portions of the medical image than to other portions of the medical image. After all, it may be the case that not every delineated region of the medical image is decisive or otherwise important with respect to the inference task. Thus, by focusing primarily on any regions of the medical image that are decisive for the task and by downplaying or ignoring any other regions of the medical image that are not decisive for the task, the inference task may be performed accurately or precisely.

[0022] The prior art attempts to achieve such attention or focus by configuring a deep learning neural network with a learnable attention block. The learnable attention block is one or more hidden layers after an upstream portion of the deep learning neural network and before a downstream portion of the deep learning neural network, such that the learnable attention block receives various features / activation maps calculated by the upstream portion, such that the learnable attention block calculates a corresponding attention score (also referred to as an alignment score) for each of those various features / activation maps, and such that the downstream portion weights those various features / activation maps by the calculated attention scores, respectively, in order to calculate its own feature / activation map. Thus, the learnable attention block may be viewed as determining the relative importance levels of the various features / activation maps received from the upstream portion (e.g., features / activation maps for which numerically low attention scores are calculated may be viewed as less important; features / activation maps for which numerically high attention scores are calculated may be viewed as more important). In other words, the upstream portion may compute various features / activation maps, the learnable attention block may determine which of those various features / activation maps are more or less salient, and the downstream portion may focus on any of those various features / activation maps that are determined to be more salient.

[0023] Unfortunately, such prior art exhibits insufficient accountability or explainability. Such lack of accountability or explainability is due, at least in part, to a variety of reasons.

[0024] First, the attention scores of the prior art are completely hidden within the deep learning neural network. In other words, when the prior art is implemented, the attention scores calculated by the learnable attention block are not provided or otherwise treated as independent outputs of the deep learning neural network. Therefore, the user or technician of the deep learning neural network is completely unaware of such attention scores, which means that when performing inference tasks on medical images, the user or technician is not informed of where the deep learning neural network is focusing its attention.

[0025] Second, even if the attention scores of the prior art are extracted and provided as independent outputs of the deep learning neural network, such attention scores do not carry easily understood semantic meanings with respect to the medical image. In fact, as mentioned above, the learnable attention block is sandwiched between the upstream and downstream parts of the deep learning neural network, and the attention scores calculated by the learnable attention block are used as weights for the corresponding features / activation maps generated by the upstream part. Therefore, the attention scores can be regarded as indicating which of those features / activation maps will receive more or less focus from the deep learning neural network. Unfortunately, however, it is not clear at all how such features / activation maps are related to medical images. After all, such features / activation maps are hidden, abstract numerical representations (e.g., latent vectors) that do not have easily recognizable or interpretable physical meanings with respect to the medical image. Therefore, even if the user or technician of the deep learning neural network is aware of the attention scores calculated by the learnable attention block, such attention scores do not convey any physically relevant information about the medical image to the user or technician. Instead, such attention scores would merely identify to the user or technician those various features / activation maps to which the deep learning neural network is paying more or less attention; because those features / activation maps have no easily understandable relationship to any visual content depicted in the medical image, such information may be deemed to be equivalent to being useless.

[0026] Third, even if the attention scores of the prior art are used to carry some physical-related meaning with respect to the medical image, such physical-related meaning will be layer-related or relative, rather than layer-transcendent or absolute. Thirdly, the learnable attention block receives multiple features / activation maps from the upstream part of the deep learning neural network and assigns corresponding attention scores to those received features / activation maps. It should be noted that different layers of the deep learning neural network may produce different numbers, sizes, formats, or different types of features / activation maps. Therefore, the depth within the network at which the learnable attention block is located (for example, it may be located closer to the upstream side of the deep learning neural network, or it may be located closer to the downstream side of the deep learning neural network) indicates how many numbers and types of features / activation maps the learnable attention block assigns attention scores to, which correspondingly indicates the meaning of the calculated attention scores (if any). This may be viewed as confusing to a user or technician because the meaning of the attention score may therefore vary across a neural network having learnable attention blocks located at different positions (e.g., suppose two separate neural networks with learnable attention blocks located at different positions are trained to perform an inference task; in such a case, the attention score produced by one neural network would not have the same substantive meaning as the attention score produced by the other neural network).

[0027] Fourth, the learnable attention blocks of the prior art are unguided and are therefore susceptible to learning unfamiliar or unexpected attention patterns. In particular, when implementing the prior art, the true attention scores are not used to directly train the learnable attention blocks. Instead, the learnable attention blocks are updated with the rest of the deep learning neural network via backpropagation (e.g., to minimize the final error relative to the result of the true inference task, so as to maximize the final reinforcement learning reward). Such a lack of true attention scores can be attributed at least in part to the above-mentioned fact that the attention scores of the prior art do not carry easily interpretable physical meanings (e.g., when a given feature / activation map does not have a clear physical meaning, it is not possible to know in advance what true attention scores should correspond to the given feature / activation map). Such unguided training may eventually cause the learnable attention blocks to become too fit, or otherwise learn to assign attention scores in a trivial, unexpected or unexplainable way. Again, this undermines the interpretability or explainability of the prior art.

[0028] Therefore, systems or techniques that can facilitate deep learning attention on medical images with enhanced explainability or interpretability may be desirable.

[0029] Various embodiments described herein may solve one or more of these technical problems. One or more embodiments described herein may include a system, a computer-implemented method, an apparatus, or a computer program product that can facilitate interpretable visual attention for deep learning. In particular, various embodiments described herein may involve configuring or training a deep learning neural network to perform an inference task on an input medical image, wherein the deep learning neural network may include a main processing channel and an auxiliary processing channel branching from the main processing channel. As described herein, the main processing channel may be responsible for generating the inference task result of the input medical image as an output, while the auxiliary processing channel may alternatively be responsible for generating an attention map of the input medical image as an output. In various aspects, the attention map may indicate or represent an attention score at the pixel level (or voxel level) of the input medical image. In other words, the attention map may be considered to indicate the importance or unimportance of each individual pixel (or each individual voxel) of the input medical image that the deep learning neural network considers or infers for the execution of the inference task. In other words, the attention map may be considered to identify or indicate which pixels (or voxels) of the input medical image the deep learning neural network focuses its attention on. In various instances, as described herein, the auxiliary processing channels may be trained in a supervised manner using training medical images and respectively corresponding ground truth attention maps, and the primary processing channels may be trained using a loss function weighted by such ground truth attention maps in a pixel-level (or voxel-level) manner. In various cases, such ground truth attention maps may be obtained based on executing any suitable edge detector or boundary detector on the training medical images. In various other cases (e.g., for image-to-image regression tasks), such ground truth attention maps may be obtained instead based on the difference between the training medical images and the respectively corresponding ground truth inference task results. In other cases, such ground truth attention maps may be obtained instead based on the uncertainty or error exhibited by the deep learning neural network relative to the training medical images during a previous training period.

[0030] The various embodiments described herein may be viewed as facilitating visual attention with enhanced interpretability or explainability compared to the prior art.

[0031] First, as mentioned above, the attention scores of the prior art are completely hidden. That is, such attention scores are not provided or extracted as independent outputs that can be viewed by a user or technician. In sharp contrast, the attention maps described herein are provided as independent outputs that can be viewed by a user or technician. Therefore, when implementing the various embodiments described herein, the user or technician can be informed of where the deep learning neural network focuses its attention when performing inference tasks on medical images.

[0032] Second, as mentioned above, the attention scores of the prior art do not have an easily interpretable physical meaning with respect to medical images. After all, such attention scores correspond to (e.g., used for internal weighting) various features / activation maps calculated internally by the hidden neural network layer, and such various features / activation maps are abstract latent vectors that do not have an easily understandable physical or semantic relationship with the visual content depicted in the medical image. In sharp contrast, the attention maps described herein have an easily interpretable physical meaning with respect to medical images. Specifically, the attention maps described herein indicate the corresponding attention scores for each individual pixel (or each individual voxel) of the medical image. Therefore, the attention map can be viewed as indicating which specific pixels (or voxels) of the medical image the deep learning neural network determines to be more or less important for the performance of the inference task. Therefore, the attention map can be visually rendered (e.g., as a heat map) to visually emphasize the pixels (or voxels) of the medical image that receive more attention or focus from the deep learning neural network. Such direct relationship with individual pixels (or voxels) of a medical image enables the attention maps described herein to describe clear physically relevant meanings, which makes the attention maps easily understandable, interpretable, or explainable.

[0033] Third, as mentioned above, the attention scores of the prior art are layer-dependent. After all, even if at least some substantive meaning is gleaned from such attention scores, such substantive meaning will depend on the specific characteristics of the features / activation maps weighted by those attention scores, and different neural network layers (e.g., layers located at different depths within the network) produce features / activation maps with different characteristics. Therefore, different neural networks configured to perform inference tasks but with learnable attention blocks located at different locations will produce attention scores with inconsistent and therefore incomparable meanings, which is confusing to users or technicians (e.g., for any suitable positive integer a>b>1, one network may calculate the attention score of the feature / activation map produced by the ath hidden layer, while a different network may calculate the attention score of the feature / activation map produced by the bth hidden layer; therefore, such attention scores cannot be properly compared with each other). In sharp contrast, the attention maps described herein indicate pixel-level (or voxel-level) attention scores regardless of the depth within the network where the auxiliary processing channel is located. In fact, although the specific construction of one or more layers of the auxiliary processing channel may have to be adjusted depending on the depth within the network at which the auxiliary processing channel branches from the main processing channel, as described herein, the output of the auxiliary processing channel is always a pixel-level (or voxel-level) map of attention scores. Therefore, different neural networks configured to perform inference tasks but with auxiliary processing channels located in different positions will still produce attention maps with consistent and therefore comparable meanings, which reduces confusion for users or technicians (e.g., each network will produce a corresponding pixel-level (or voxel-level) attention score map, and therefore such maps can be appropriately compared with each other).

[0034] Fourth, as mentioned above, the learnable attention blocks of the prior art are not guided by the true attention scores. Such lack of guidance may cause such learnable attention blocks to learn to allocate attention in strange or unexplained ways. In sharp contrast, the auxiliary processing channels described herein can be trained based on training medical images and respectively corresponding true attention maps. Such true attention maps (which can be used to dynamically weight the training loss function of the main processing channel) can be obtained via edge detection, uncertainty quantification, or image annotation difference calculation, as described herein. In any case, the use of such true attention maps can reduce the possibility that the auxiliary processing channels learn to allocate attention in unusual, strange, or unexplained ways.

[0035] For at least these reasons, the various implementations described herein may be viewed as facilitating visual deep learning attention with improved interpretability.

[0036] The various embodiments described herein may be viewed as computerized tools (e.g., any suitable combination of computer-executable hardware or computer-executable software) that can facilitate interpretable visual attention for deep learning. In various aspects, such computerized tools may include access components, execution components, or display components.

[0037] In various embodiments, there may be a specific medical image. In various aspects, the specific medical image may be in any suitable format, size, or dimension (e.g., the specific medical image may be a two-dimensional array of pixels, or the specific medical image may be a three-dimensional array of voxels). In various instances, the specific medical image may be generated or captured by any suitable medical imaging modality or equipment (e.g., generated or captured by a CT scanner, by an MRI scanner, by an X-ray scanner, by an ultrasound scanner, or by a PET scanner). In various cases, the specific medical image may visually depict any suitable anatomical structure of any suitable medical patient.

[0038] In various aspects, it may be possible to perform an inference task on a particular medical image. In various instances, the inference task may be any suitable predictive computation or functionality applicable to the image. As some non-limiting examples, the inference task may be image classification, image segmentation, or image-to-image regression. In various aspects, the inference task may be viewed as having some diagnostic or prognostic relevance with respect to the anatomical structures depicted in the particular medical image.

[0039] In any case, it may be desirable to perform an inference task on a particular medical image. Furthermore, it may be desirable to discover which pixels (or voxels) of a particular medical image receive more or less attention or focus during such performance (e.g., to discover which pixels (or voxels) of a particular medical image are more or less decisive with respect to the inference task). In various aspects, the computerized tools described herein may facilitate such purposes.

[0040] In various embodiments, an access component of a computerized tool may electronically receive or otherwise electronically access a particular medical image. In some aspects, the access component may electronically retrieve the particular medical image from any suitable centralized and / or decentralized data structure (e.g., a graph data structure, a relational data structure, a hybrid data structure), whether remote from the access component or local to the access component. In any case, the access component may electronically obtain or access the particular medical image so that other components of the computerized tool may electronically interact with (e.g., read, write, edit, copy, manipulate) the particular medical image.

[0041] In various embodiments, the execution components of the computerized tool may electronically store, maintain, control, or otherwise access the deep learning neural network. In various aspects, the deep learning neural network may include a primary processing channel and an auxiliary processing channel.

[0042] In various instances, the primary processing channel may exhibit any suitable deep learning internal architecture. For example, the primary processing channel may include any suitable number of layers of any suitable type (e.g., an input layer, one or more hidden layers, an output layer, any of which may be a convolutional layer, a dense layer, a nonlinear layer, a pooling layer, a batch normalization layer, or a padding layer). As another example, the primary processing channel may include any suitable number of neurons in the various layers (e.g., different layers may have the same or different numbers of neurons as each other). As yet another example, the primary processing channel may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, rectified linear unit) in the various neurons (e.g., different neurons may have the same or different activation functions as each other). As yet another example, the primary processing channel may include any suitable inter-neuronal or inter-layer connections (e.g., forward connections, skip connections, recursive connections).

[0043] In various cases, the auxiliary processing channel may also exhibit any suitable deep learning internal architecture. For example, the auxiliary processing channel may include any suitable number of layers of any suitable type (e.g., an input layer, one or more hidden layers, an output layer, any of which may be a convolutional layer, a dense layer, a nonlinear layer, a pooling layer, a batch normalization layer, or a padding layer). As another example, the auxiliary processing channel may include any suitable number of neurons in various layers (e.g., different layers may have the same or different numbers of neurons as each other). As yet another example, the auxiliary processing channel may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, rectified linear unit) in various neurons (e.g., different neurons may have the same or different activation functions as each other). As yet another example, the auxiliary processing channel may include any suitable inter-neuron connection or inter-layer connection (e.g., forward connection, skip connection, recursive connection).

[0044] In various aspects, the auxiliary processing channel can branch off from the main processing channel such that the auxiliary processing channel is parallel to at least some portions of the main processing channel. For example, the main processing channel can have a given hidden layer between its input layer and its output layer, and the input layer of the auxiliary processing channel can be configured to receive as input any feature / activation map produced by the given hidden layer such that the auxiliary processing channel is parallel to all layers of the main processing channel downstream of the given hidden layer.

[0045] In any case, the main processing channel may be configured to perform an inference task on an input medical image, while the auxiliary processing channel may alternatively be configured to generate a pixel-level (or voxel-level) attention map for such input medical image. Thus, the execution component may electronically execute a deep learning neural network on a specific medical image, and such execution may produce an inference task result and an attention map. More specifically, the execution component may feed a specific medical image to the input layer of the main processing channel, the specific medical image may complete a forward pass through one or more hidden layers of the main processing channel, and such forward pass may enable the output layer of the main processing channel to calculate the inference task result. In addition, the input layer of the auxiliary processing channel may receive one or more activations from any hidden layer of the main processing channel at the auxiliary processing channel branch, such one or more activations may complete a forward pass through one or more hidden layers of the auxiliary processing channel, and such forward pass may enable the output layer of the auxiliary processing channel to calculate an attention map.

[0046] In various aspects, the inference task result can be any calculation result determined or predicted by the main processing channel for a specific medical image. Therefore, the inference task result can be any suitable electronic data, the format, size or dimension of the electronic data depends on the inference task that the main processing channel is configured to perform. As a non-limiting example, if the inference task is image classification, the inference task result can be the classification label that the main processing channel has predicted for the specific medical image. As another non-limiting example, if the inference task is image segmentation, the inference task result can be the segmentation mask that the main processing channel has predicted for the specific medical image. As another non-limiting example, if the inference task is image to image regression (e.g., denoising, resolution enhancement, modality transformation), the inference task result can be a regression version (e.g., denoising version, resolution enhancement version, modality transformation version) that the main processing channel has predicted for the specific medical image.

[0047] In various instances, an attention map may be a pixel-level (or voxel-level) array of attention scores corresponding to a particular medical image, wherein each attention score may be a scalar weight whose magnitude indicates a level of attention, focus, importance, or significance that a corresponding pixel (or voxel) of the particular medical image has with respect to an inference task (e.g., an attention score may be a decimal number that varies continuously or discretely from 0 to 1, where 0 indicates no importance with respect to the inference task and where 1 indicates high importance with respect to the inference task). For example, if an attention map indicates that a given pixel (or voxel) of a particular medical image has a low magnitude attention score (e.g., close to 0), this may be interpreted to mean that the given pixel (or voxel) will not significantly affect or contribute to the handling of the inference task. On the other hand, if the attention map instead indicates that a given pixel (or voxel) of a particular medical image has a high magnitude attention score (e.g., close to 1), this may be interpreted to mean that the given pixel (or voxel) will significantly affect or contribute to the handling of the inference task. In other words, the attention map can be viewed as a prediction of the degree to which each individual pixel (or voxel) of a particular medical image is attended to by the primary processing channel. In any case, the attention map can be viewed as having or exhibiting a clear and easily understood physical, semantic, or substantive meaning with respect to a particular medical image (e.g., a particular medical image depicts some visual content, and the attention map shows in a pixel-level (or voxel-level) manner which parts or regions of such visual content are more or less decisive with respect to the task).

[0048] In this way, the primary processing channel can be viewed as performing an inference task on a particular medical image, and the auxiliary processing channel can be viewed as predicting or inferring which specific pixels (or voxels) of the particular medical image will be given more or less attention by the primary processing channel.

[0049] In various embodiments, the display component of the computerized tool can electronically render the inference task results or attention maps on any suitable electronic display so that the inference task results or attention maps can be viewed by a user or technician of the computerized tool. In this way, the user or technician can be informed not only of the inference task results, but also which individual pixels (or voxels) of a particular medical image contribute most to the inference task results.

[0050] It should be noted that in various aspects, the attention map can be used by a user or technician for fault analysis of inference task results. As a non-limiting example, assume that a user or technician determines that the inference task result is incorrect. That is, the user or technician may conclude that the primary processing channel inaccurately performed the inference task for a particular medical image. In such cases, the attention map can help determine how or why the primary processing channel inaccurately performed the inference task. In fact, the attention map may show that some unexpected pixels (or voxels) of a particular medical image receive disproportionately high attention or focus. This may indicate that the primary processing channel is distracted or otherwise interrupted by any visual content illustrated by those unexpected pixels (or voxels). In other words, the attention map can help reveal insights about visual characteristics that are unknown to the primary processing channel but should be unknown to the primary processing channel. Therefore, the user or technician can use such insights to help retrain or fine-tune the primary processing channel to avoid similar failures in the future.

[0051] In order to help make the inference task results and attention maps accurate (e.g., to help the primary processing channel correctly perform the inference task on the input medical image, and to help the auxiliary processing channel correctly predict which pixels (or voxels) of such input medical image are more or less important with respect to the inference task), the deep learning neural network may first undergo training. In various aspects, the access component may receive, retrieve, or otherwise access the annotated training data set, and the computerized tool may include a training component that can train the deep learning neural network on the annotated training data set.

[0052] In various aspects, the annotated training data set may include a set of training medical images. In various instances, the training medical images may be any suitable medical images having the same size, format, or dimensionality (e.g., the same number or arrangement of pixels (or voxels)) as the specific medical images. In various cases, the annotated training data set may also include a set of true value inference task results corresponding to the set of training medical images, respectively. In various aspects, the true value inference task results may be correct or accurate inference task results (e.g., correct or accurate classification labels, correct or accurate segmentation masks, correct or accurate regression outputs) that are known or considered to correspond to the corresponding training medical images. In various instances, the annotated training data set may also include a set of true value attention maps corresponding to the set of training medical images, respectively. In various cases, the true value attention map may be a correct or accurate array of pixel-level or voxel-level attention scores that are known or considered to correspond to the corresponding training medical images.

[0053] In some embodiments, a true attention map for a given training medical image may be obtained by executing an edge detector / boundary detector on the given training medical image. In practice, it may be the case that the visually perceptible edges, boundaries, or other contours (e.g., contours of anatomical structures) illustrated in a given training medical image are known or considered to be highly relevant or otherwise deterministic with respect to the inference task. Therefore, any pixels (or voxels) of the given training medical image that constitute or otherwise belong to those visually perceptible edges, boundaries, or other contours may be assigned a higher true attention score. To achieve this, any suitable edge detector / boundary detector may be implemented. In particular, the edge detector / boundary detector may be any suitable machine learning model pre-trained via any suitable training paradigm (e.g., supervised training, unsupervised training, reinforcement learning) to segment within an input image any pixels (or voxels) that it infers or determines to belong to an edge, boundary, or contour of the input image. Thus, an edge detector / boundary detector may be performed on a given training medical image; such execution may produce an inferred segmentation mask indicating which pixels (or voxels) of the given training medical image belong to edges, boundaries, or contours visually illustrated in the given training medical image; and such inferred segmentation mask may be viewed or otherwise treated as a ground-truth attention map for the given training medical image.

[0054] In other embodiments, the truth attention map for a given training medical image may alternatively be obtained by comparing the given training medical image with the truth inference task result corresponding to the given training medical image. In fact, if the inference task is an image-to-image regression, the truth inference task result may be an image of the same size, format, or dimension as the given training medical image; in fact, the truth inference task result may be a regression version (e.g., a denoised version, a resolution enhanced version, a modality transformed version) of the given training medical image. In such cases, it may be the case that the intensity values ​​(e.g., Hounsfield units) of some pixels (or voxels) of the given training medical image undergo little or no change when converted to the truth inference task result, and it may be the case that the intensity values ​​of other pixels (or voxels) of the given training medical image undergo significant changes when converted to the truth inference task result. Therefore, any pixel (or voxel) of the given training medical image that undergoes significant changes in the generation of the truth inference task result may be assigned a higher truth attention score. To achieve this, a difference array can be calculated by applying matrix subtraction between a given training medical image and the ground-truth inference task result, the difference array can be normalized, and the absolute value of the normalized difference array can be considered or otherwise treated as a ground-truth attention map for the given training medical image.

[0055] In yet other embodiments, the true value attention map for a given training medical image may alternatively be obtained by quantifying the uncertainty or error exhibited by the main processing channel relative to the given training medical image. In particular, it may be the case that the main processing channel was previously (e.g., during a previous training period) performed on a given training medical image. If the inference task is image segmentation or image-to-image regression, such previous execution may produce an output array. The output array may be regarded as a previously predicted segmentation mask or a previously predicted regression image, which the main processing channel believes to correspond to the given training medical image. In contrast, the true value inference task result may be regarded as a correct or accurate segmentation mask or regression image that is known or considered to correspond to the given training medical image. Now, it may be the case that the intensity values ​​of some pixels (or voxels) of the output array are close to the values ​​that the true value inference task result indicates that these intensity values ​​should have, and it may be the case that the intensity values ​​of other pixels (or voxels) of the output array are far from the values ​​that the true value inference task result indicates that these intensity values ​​should have. Therefore, any pixel (or voxel) of the output array (and thus the given training medical image by the agent) that is significantly different from the pixel (or voxel) indicated by the truth inference task result can be assigned a higher truth attention score. To achieve this, a difference array can be calculated by applying matrix subtraction between the output array and the truth inference task result, the difference array can be normalized, and the absolute value of the normalized difference array can be regarded or otherwise used as a truth attention map for the given training medical image.

[0056] In any case, the training component can train the deep learning neural network on the annotated training data set as follows. Before starting such training, the trainable internal parameters (e.g., weight matrix, bias vector, convolution kernel) of the deep learning neural network (e.g., main processing channel, auxiliary processing channel) can be randomly initialized.

[0057] In various aspects, the training component can select any suitable training medical images, any suitable respectively corresponding truth inference task results, and any suitable respectively corresponding truth attention maps from the annotated training dataset. In various instances, the training component can execute a deep learning neural network on the selected training medical images, so that the primary processing channel generates a first output and the auxiliary processing channel generates a second output.

[0058] More specifically, the training component may feed the selected training medical image to the input layer of the primary processing channel, the selected training medical image may complete a forward pass through one or more hidden layers of the primary processing channel, and such forward pass may cause the output layer of the primary processing channel to calculate a first output. In addition, the input layer of the auxiliary processing channel may receive one or more activations from any hidden layer of the primary processing channel at the location where the auxiliary processing channel branches out, such one or more activations may complete a forward pass through one or more hidden layers of the auxiliary processing channel, and such forward pass may cause the output layer of the auxiliary processing channel to calculate a second output.

[0059] It should be noted that the format, size or dimensionality of the first output may be controlled or otherwise indicated by the number, arrangement or size of neurons or other internal parameters (e.g., convolution kernels) contained in or otherwise constituting the output layer of the primary processing channel. Therefore, the first output may be forced to have any format, size or dimensionality suitable for the inference task by adding neurons or other internal parameters to the output layer of the primary processing channel, removing neurons or other internal parameters from the output layer of the primary processing channel, or otherwise adjusting neurons or other internal parameters within the output layer of the primary processing channel. Therefore, the first output may be considered as the predicted inference task result (e.g., predicted classification label, predicted segmentation mask, predicted regression output) that the primary processing channel believes should correspond to the selected training medical image. In contrast, the selected true value inference task result may be a correct or accurate inference task result (e.g., correct or accurate classification label, correct or accurate segmentation mask, correct or accurate regression output) that is known or believed to correspond to the selected training medical image. In various cases, if the primary processing channel has undergone no or little training to date, the first output may be highly inaccurate (eg, may differ significantly from the results of the selected truth inference task).

[0060] Likewise, it should be noted that the format, size or dimensionality of the second output may be controlled or otherwise indicated by the number, arrangement or size of neurons or other internal parameters (e.g., convolution kernels) contained in or otherwise constituting the output layer of the auxiliary processing channel. Thus, the second output may be forced to have any desired format, size or dimensionality by adding neurons or other internal parameters to the output layer of the auxiliary processing channel, removing neurons or other internal parameters from the output layer of the auxiliary processing channel, or otherwise adjusting neurons or other internal parameters within the output layer of the auxiliary processing channel. Thus, the second output may be considered as a predicted attention map (e.g., a predicted array of pixel-level (or voxel-level) attention scores) that the auxiliary processing channel believes should correspond to the selected training medical image. In contrast, the selected true attention map may be a correct or accurate attention map (e.g., a correct or accurate array of pixel-level (or voxel-level) attention scores) that is known or believed to correspond to the selected training medical image. In various cases, if the auxiliary processing channel has not undergone training or has undergone little training so far, the second output may be extremely inaccurate (e.g., may differ greatly from the selected true attention map).

[0061] In any case, the training component may calculate a first error or loss (e.g., mean absolute error (MAE), mean square error (MSE), cross entropy) between the first output and the selected truth inference task result. In various aspects, the training component may weight the first error or loss in a pixel-level (or voxel-level) manner according to the selected truth attention map (e.g., which may be implemented via point-by-point multiplication and summation), and the training component may update the internal parameters of the main processing channel by performing backpropagation (e.g., stochastic gradient descent) driven by the first error or loss as weighted by the selected truth attention map.

[0062] Similarly, the training component may calculate a second error or loss (e.g., MAE, MSE, cross entropy error) between the second output and the selected true attention map, and the training component may update the internal parameters of the auxiliary processing channel by performing backpropagation (e.g., stochastic gradient descent) driven by the second error or loss.

[0063] In various aspects, such training process may be repeated for any suitable number of training medical images. Such training may ultimately result in the trainable internal parameters of the primary processing channel being iteratively optimized for accurately performing the inference task on the input medical image, and such training may also ultimately result in the trainable internal parameters of the auxiliary processing channel being iteratively optimized for accurately predicting the importance of individual pixels (or voxels) of such input medical images with respect to the inference task.

[0064] Various embodiments described herein may be used to solve problems that are highly technical in nature (e.g., to facilitate interpretable visual attention for deep learning) using hardware or software that are not abstract and cannot be performed as a set of psychological behaviors of a human. In addition, some of the processes performed may be performed by a dedicated computer (e.g., a deep learning neural network with internal parameters such as convolution kernels) to implement defined actions related to deep learning. For example, such defined actions may include: accessing, by a device operatively coupled to a processor, a medical image generated by a medical imaging scanner; and performing, by the device and via execution of a deep learning neural network, an inference task on the medical image, wherein the deep learning neural network receives the medical image as input and produces as output both an inference task result and an attention map that indicates which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

[0065] Such defined actions are not performed manually by a human. In fact, neither a human mind nor a person with a pen and paper can electronically access a scanned medical image (e.g., an X-ray image, an MRI image, a CT image) and electronically perform an inference task by executing a neural network on the scanned medical image, where the neural network computes as output both an inference task result (e.g., a classification label, a segmentation mask, a regression image) and an attention map that shows which specific pixels (or voxels) of the scanned medical image the neural network focused on (e.g., which specific pixels (or voxels) contributed most to the inference task result). In fact, neural networks are also inherently computerized constructs that cannot be meaningfully executed or trained by the human mind in any way at all without a computer. Computerized tools that can execute neural networks on scanned medical images are likewise inherently computerized and cannot be implemented in any sensible, practical, or reasonable way without a computer.

[0066] In addition, various embodiments described herein can integrate various teachings related to promoting interpretable visual attention for deep learning into practical applications. As described above, the prior art incorporates learnable attention blocks into neural networks, wherein such learnable attention blocks calculate attention scores for weighting corresponding hidden activation maps. Unfortunately, such learnable attention blocks lack interpretability or explanability. In fact, the attention scores provided by such learnable attention blocks are completely hidden (e.g., not extracted as independent neural network outputs) and do not have easily understood physical meanings. In addition, even if some physical meanings are collected from such attention scores, such meanings will depend on the depth within the network where the learnable attention blocks are located. In addition, such learnable attention blocks are trained in an unguided manner, which may result in learning strange or unexplainable attention patterns.

[0067] In sharp contrast, various embodiments described herein can solve one or more of these technical problems. In particular, various embodiments described herein relate to training or configuring a neural network to receive a medical image as input and produce two distinct outputs: an inference task result for the input medical image; and an attention map indicating the relative importance or unimportance level of each individual pixel (or voxel) of the input medical image relative to the inference task result. In various aspects, the inference task result can be generated by a main processing channel of the neural network, and the attention map can be generated by an auxiliary processing channel that branches out from the main processing channel and is parallel to a downstream portion of the main processing channel.

[0068] Unlike prior art attention scores, the attention maps described herein are provided or otherwise treated as a separate distinct output of the neural network (e.g., a user or technician is not informed of prior art attention scores, whereas a user or technician is informed of the attention maps described herein).

[0069] Also unlike the attention scores of the prior art, the attention maps described herein carry physical or semantic meanings that are easily discernible. For example, each weight in the attention map shows how important or unimportant the corresponding pixel (or voxel) of the input medical image is relative to the result of the inference task, and such importance or unimportance is meaningful because it is clear how the pixel (or voxel) is related to the visual content of the input medical image. In contrast, each attention score of the prior art only shows how important or unimportant the corresponding hidden activation map is, and such importance or unimportance is meaningless because it is unclear how the hidden activation map is related to the visual content of the input medical image (e.g., the hidden activation map is an abstract representation or latent vector that does not have an obvious or interpretable meaning).

[0070] In addition, unlike the attention scores of the prior art, the meaning of the attention maps described herein is not layer-dependent. In fact, any meaning (if any) of the attention scores calculated by the learnable attention block depends on where the learnable attention block is located within the neural network (e.g., the learnable attention block generates attention scores for any hidden activation maps it receives as input; different hidden layers generate different numbers, types, or formats of hidden activation maps; therefore, learnable attention blocks located in different locations will generate different numbers of attention scores with different meanings). In contrast, regardless of the depth within the network at which the auxiliary processing channel is located, the attention maps described herein indicate the importance, attention, or focus of each pixel (or voxel) of the input medical image (e.g., regardless of the depth at which the auxiliary processing channel is located, the auxiliary processing channel can be appropriately configured to generate a pixel-level (or voxel-level) array of attention scores).

[0071] Furthermore, unlike the learnable attention blocks of the prior art, the auxiliary processing channels described herein can be trained in a guided manner using true attention maps. In fact, true attention scores may be unavailable or unobtainable for the prior art, precisely because such attention scores lack a clear physical meaning (e.g., if one does not know in advance what the training hidden activation map means or represents, then one certainly cannot know which true attention scores should correspond to the training hidden activation map). In contrast, true attention maps can be obtained to train the auxiliary processing channels because the importance or unimportance of each pixel (or voxel) of the training medical image with respect to the inference task can be known or estimated in advance (e.g., such true attention maps can be obtained via edge detection / boundary detection, via image annotation comparison, or via past uncertainty / error quantification, as described herein).

[0072] In addition, the attention maps described herein can be used for fault analysis of neural networks in a way that the attention scores of the prior art cannot. In fact, suppose that the primary processing channel generates the inference task results incorrectly or inaccurately. In such cases, potential causes or explanations for such inaccuracies can be gleaned from the attention maps generated by the auxiliary processing channels (e.g., the attention maps may show that certain pixels (or voxels) of the input medical image are accidentally or incorrectly emphasized or considered important by the primary processing channel). In this way, the attention maps can be utilized to identify insights such as visual features or artifacts that distract or interrupt the primary processing channel, and such insights can be used to subsequently retrain or fine-tune the primary processing channel.

[0073] For at least these reasons, the various embodiments described herein facilitate visual attention of medical images for deep learning with enhanced or increased interpretability compared to the prior art. Therefore, the various embodiments described herein certainly constitute tangible and concrete technical improvements or technical advantages in the field of deep learning. Therefore, such embodiments clearly qualify as useful and practical applications of computers.

[0074] Furthermore, various embodiments described herein can control real-world tangible devices based on the disclosed teachings. For example, various embodiments described herein can electronically train or execute real-world deep learning neural networks on real-world images (e.g., X-ray scan images, CT scan images), and can electronically present real-world results (e.g., classification labels, segmentation masks, regression images, attention maps) on a real-world computer screen.

[0075] It should be understood that the drawings and description herein provide non-limiting examples of various embodiments and are not necessarily drawn to scale.

[0076] Figure 1 A block diagram of an example non-limiting system 100 that can facilitate explainable visual attention for deep learning according to one or more embodiments described herein is illustrated. As shown, the attention system 102 can be electronically integrated with the medical image 104 via any suitable wired or wireless electronic connection.

[0077] In various embodiments, the medical image 104 may be any suitable image exhibiting any suitable format, size, or dimension. As a non-limiting example, for any suitable positive integers x and y, the medical image 104 may be an x×y pixel array. As another non-limiting example, for any suitable positive integers x, y, and z, the medical image 104 may be an x×y×z voxel array. In various aspects, the medical image 104 may be captured or otherwise generated by any suitable medical imaging modality. As a non-limiting example, the medical image 104 may be captured or generated by a CT scanner, in which case the medical image 104 may be considered a CT scan image. As another non-limiting example, the medical image 104 may be captured or generated by an MRI scanner, in which case the medical image 104 may be considered an MRI scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by an X-ray scanner, in which case the medical image 104 may be considered an X-ray scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by an ultrasound scanner, in which case the medical image 104 may be considered an ultrasound scan image. As yet another non-limiting example, the medical image 104 may be captured or generated by a PET scanner, in which case the medical image 104 may be considered a PET scan image. In various instances, the medical image 104 may have been subjected to any suitable image reconstruction technique, such as filtered back projection.

[0078] In various aspects, medical image 104 may visually depict or illustrate any suitable anatomical structure of any suitable medical patient (e.g., human, animal, or other). In various examples, the anatomical structure may be any suitable body organ of a medical patient, any suitable body tissue of a medical patient, any suitable body part of a medical patient, any suitable body fluid of a medical patient, any suitable body cavity of a medical patient, any suitable surgical implant (e.g., a medical catheter, a medical suture, a medical stent, a pacemaker, a medical pole, a medical plate, a medical screw), any suitable lesion or injury thereof, or any suitable portion thereof.

[0079] In various aspects, the inference task may be capable of being performed on the medical image 104 or otherwise relative to the anatomical structure depicted in the medical image 104. In various instances, the inference task may be any suitable predictive computation. As a non-limiting example, the inference task may be image classification, which may involve assigning the medical image 104 to one of two or more possible categories or classes (e.g., determining whether the anatomical structure of the medical image 104 belongs to a diseased class or a healthy class). As another non-limiting example, the inference task may be image segmentation, which may involve assigning each pixel (or voxel) of the medical image 104 to a corresponding one of two or more possible categories or classes (e.g., determining which pixels (or voxels) of the medical image 104 belong to a tumor class and which pixels (or voxels) alternatively belong to a background class). As yet another non-limiting example, the inference task may be image-to-image regression, which may involve generating a new image based on the medical image 104 (e.g., converting the medical image 104 from one modality (X-ray) to another modality (MRI); generating a denoised version of the medical image 104; generating a quality enhanced version of the medical image 104).

[0080] In various situations, it may be desirable to perform inference tasks on a medical image 104. It may also be desirable to uncover which specific pixels (or voxels) of the medical image 104 contribute most to such performance. As described herein, the attention system 102 may facilitate such purposes.

[0081] In various embodiments, the attention system 102 may include a processor 106 (e.g., a computer processing unit, a microprocessor) and a non-transitory computer-readable memory 108 that is operably or operatively or communicatively connected or coupled to the processor 106. The non-transitory computer-readable memory 108 may store computer-executable instructions that, when executed by the processor 106, may cause the processor 106 or other components of the attention system 102 (e.g., an access component 110, an execution component 112, a display component 114) to perform one or more actions. In various embodiments, the non-transitory computer-readable memory 108 may store computer-executable components (e.g., an access component 110, an execution component 112, a display component 114), and the processor 106 may execute the computer-executable components.

[0082] In various embodiments, the attention system 102 may include an access component 110. In various aspects, the access component 110 may electronically receive or otherwise electronically access the medical image 104. In various instances, the access component 110 may electronically retrieve the medical image 104 from any suitable centralized or decentralized data structure (not shown) or from any suitable centralized or decentralized computing device (not shown). As a non-limiting example, the access component 110 may electronically retrieve the medical image 104 from any medical imaging modality (e.g., CT scanner, X-ray scanner, MRI scanner, ultrasound scanner, PET scanner) that generated / captured the medical image 104. In any case, the access component 110 may electronically obtain or access the medical image 104 so that other components of the attention system 102 may electronically interact with the medical image 104.

[0083] In various embodiments, the attention system 102 may include an execution component 112. In various aspects, the execution component 112 may execute a deep learning neural network on the medical image 104, which may produce both an inference task result and an attention map indicating which specific pixels (or voxels) of the medical image 104 the deep learning neural network focused on to generate the inference task result.

[0084] In various embodiments, the attention system 102 can include a display component 114. In various examples, the display component 114 can visually render the inference task results or attention map on any suitable electronic display.

[0085] Figure 2 A block diagram of an example non-limiting system 200 that can facilitate interpretable visual attention for deep learning according to one or more embodiments described herein is illustrated, the system including a deep learning neural network, an inference task result, and an attention map. As shown, in some cases, the system 200 may include the same components as the system 100, and may also include a deep learning neural network 202, an inference task result 208, and an attention map 210.

[0086] In various embodiments, execution component 112 may electronically store, electronically maintain, electronically control, or otherwise electronically access deep learning neural network 202. In various aspects, deep learning neural network 202 may include primary processing channel 204 and auxiliary processing channel 206.

[0087] In various instances, the main processing channel 204 may have or otherwise exhibit any suitable deep learning internal architecture. For example, the main processing channel 204 may have an input layer, one or more hidden layers, and an output layer. In various instances, any of such layers may be coupled together by any suitable inter-neuron connection or inter-layer connection (such as a forward connection, a skip connection, or a recursive connection). In addition, in various cases, any of such layers may be a neural network layer of any suitable type with any suitable learnable or trainable internal parameters. For example, any of such input layer, one or more hidden layers, or output layer may be a convolutional layer, and the learnable or trainable parameters of the convolutional layer may be a convolution kernel. As another example, any of such input layer, one or more hidden layers, or output layer may be a dense layer, and the learnable or trainable parameters of the dense layer may be a weight matrix or a bias value. As yet another example, any of such input layer, one or more hidden layers, or output layer may be a batch normalization layer, and the learnable or trainable parameters of the batch normalization layer may be a shift factor or a scale factor. Further, in various cases, any of such layers may be any suitable type of neural network layer with any suitable fixed or non-trainable internal parameters. For example, any of such input layer, one or more hidden layers, or output layer may be a non-linear layer, a padding layer, a pooling layer, or a cascade layer.

[0088] In various cases, the auxiliary processing channel 206 may have or otherwise exhibit any suitable deep learning internal architecture. For example, the auxiliary processing channel 206 may have an input layer, one or more hidden layers, and an output layer. In various instances, any of such layers may be coupled together by any suitable inter-neuron connection or inter-layer connection (such as a forward connection, a skip connection, or a recursive connection). In addition, in various cases, any of such layers may be a neural network layer of any suitable type with any suitable learnable or trainable internal parameters. For example, any of such input layer, one or more hidden layers, or output layer may be a convolutional layer, and the learnable or trainable parameters of the convolutional layer may be a convolution kernel. As another example, any of such input layer, one or more hidden layers, or output layer may be a dense layer, and the learnable or trainable parameters of the dense layer may be a weight matrix or a bias value. As yet another example, any of such input layer, one or more hidden layers, or output layer may be a batch normalization layer, and the learnable or trainable parameters of the batch normalization layer may be a shift factor or a scale factor. Further, in various cases, any of such layers may be any suitable type of neural network layer with any suitable fixed or non-trainable internal parameters. For example, any of such input layer, one or more hidden layers, or output layer may be a non-linear layer, a padding layer, a pooling layer, or a cascade layer.

[0089] In various aspects, the primary processing channel 204 can be regarded as the backbone of the deep learning neural network 202, and the secondary processing channel 206 can branch out from the primary processing channel 204. Thus, the secondary processing channel 206 can be regarded as being parallel to at least some portions of the primary processing channel 204 (e.g., the downstream end).

[0090] In various instances, as described herein, the primary processing channel 204 can be configured or trained to perform an inference task on an input medical image. Additionally, as described herein, the secondary processing channel 206 can be configured or trained to predict the degree of importance, significance, usefulness, or determinacy of each particular pixel (or voxel) of such an input medical image with respect to the execution of the inference task. Thus, in various cases, the execution component 112 can electronically execute the deep learning neural network 202 on the medical image 104, and such execution can cause the deep learning neural network 202 to produce both an inference task result 208 and an attention map 210 as outputs. More specifically, such execution can cause the primary processing channel 204 to produce the inference task result 208 as an output, and such execution can cause the secondary processing channel 206 to produce the attention map 210 as an output. With respect to Figure 3 Various non-limiting aspects are described.

[0091] Figure 3 Illustrates an example non-limiting block diagram 300 showing how the deep learning neural network 202 can produce an inference task result 208 and an attention map 210 in accordance with one or more embodiments described herein.

[0092] In various embodiments, the primary processing channel 204 can include n neural network layers, for any suitable positive integer n: layers 204(1) through 204(n). In various aspects, layer 204(1) can be regarded as the input layer of the primary processing channel 204, layer 204(n) can be regarded as the output layer of the primary processing channel 204, and all remaining layers (e.g., 204(2) through 204(n - 1)) can be regarded as the hidden layers of the primary processing channel 204. As mentioned above, any one of such n layers can be any suitable type of neural network layer with any suitable trainable or non-trainable internal parameters (e.g., any one of such layers can be a dense layer, a convolutional layer, a batch normalization layer, a non-linear layer, an upsampling layer, a downsampling layer, a pooling layer, a padding layer, or a concatenation layer). In various instances, as shown, the layers of the primary processing channel 204 can be coupled together via forward connections such that, for any suitable positive integer 1 ≤ c < n, the activations produced by layer 204(c) are received by layer 204(c + 1). However, for purposes of illustration, this is only a non-limiting example. In various cases, any other suitable type of inter-layer connection (e.g., skip connections, recurrent connections) can be implemented in the primary processing channel 204.

[0093] In various embodiments, the auxiliary processing channel 206 may include m neural network layers, for any suitable positive integer m: layers 206(1) to 206(m). In various aspects, layer 206(1) may be regarded as the input layer of the auxiliary processing channel 206, layer 206(m) may be regarded as the output layer of the auxiliary processing channel 206, and all the remaining layers (e.g., 206(2) to 206(m - 1)) may be regarded as the hidden layers of the auxiliary processing channel 206. As mentioned above, any one of such m layers may be any suitable type of neural network layer with any suitable trainable or non-trainable internal parameters (e.g., any one of such layers may be a dense layer, a convolutional layer, a batch normalization layer, a non-linear layer, an upsampling layer, a downsampling layer, a pooling layer, a padding layer, or a concatenation layer). In various instances, as shown in the figure, the layers of the auxiliary processing channel 206 may be coupled together via forward connections such that for any suitable positive integer 1 ≤ d < m, the activations generated by layer 206(d) are received by layer 206(d + 1). However, for the sake of illustration, this is only a non-limiting example. In various cases, any other suitable type of inter-layer connection (e.g., skip connections, recurrent connections) may be implemented in the auxiliary processing channel 206.

[0094] Now, in various aspects, the main processing channel 204 may have layers 204(j) for any suitable positive integer 1 < j < n. That is to say, layer 204(j) may be the hidden layer of the main processing channel 204. In various instances, as shown in the figure, layer 206(1) of the auxiliary processing channel 206 may be configured to receive any activations generated by layer 204(j) as input. In other words, the auxiliary processing channel 206 may be regarded as branching out from the main processing channel 204 at layer 204(j). This may make the auxiliary processing channel 206 parallel to any part of the main processing channel 204 downstream of layer 204(j) (e.g., parallel to layers 204(j + 1) to 204(n)). With this configuration, the main processing channel 204 may be regarded as the basic backbone of the deep learning neural network 202 (hence called "main"), and the auxiliary processing channel 206 may be regarded as a secondary branch branching out from this basic backbone (hence called "auxiliary").

[0095] Now, as mentioned above, the execution component 112 may electronically execute the deep learning neural network 202 on the medical image 104, and such execution may produce the inference task result 208 and the attention map 210. More specifically, the execution component 112 may feed the medical image 104 to the layer 204(1) of the primary processing channel 204. In various aspects, the medical image 104 may complete a forward pass through the hidden layers of the primary processing channel 204 (e.g., from layer 204(2) to layer 204(n-1)). In various examples, the layer 204(n) of the primary processing channel 204 may compute the inference task result 208 based on any activations received by the layer 204(n) from the layer 204(n-1) (or any activations received from any other layer of the primary processing channel 204 if skip connections or recursive connections are implemented). During the forward pass through the primary processing channel 204, the layer 204(j) may produce one or more activations at a time. In various aspects, one or more activations of layer 204(j) may be fed or otherwise routed to layer 206(1) of auxiliary processing channel 206. In various examples, those one or more activations may complete a forward pass through the hidden layers of auxiliary processing channel 206 (e.g., from layer 206(2) to layer 206(m-1)). In various cases, layer 206(m) of auxiliary processing channel 206 may compute attention map 210 based on any activations that layer 206(m) receives from layer 206(m-1) (or any activations received from any other layer of auxiliary processing channel 206 if skip connections or recursive connections are implemented).

[0096] In various aspects, the inference task result 208 may be any suitable electronic data representing or otherwise indicating the result of performing the inference task on the medical image 104. Therefore, the format, size, or dimensionality of the inference task result 208 may depend on the inference task. As a non-limiting example, assume that the inference task is image classification. In such a case, the inference task result 208 may be a classification label that the primary processing channel 204 has predicted for the medical image 104. As another non-limiting example, assume that the inference task is image segmentation. In such a case, the inference task result 208 may be a pixel-level (or voxel-level) segmentation mask that the primary processing channel 204 has predicted for the medical image 104. As yet another non-limiting example, assume that the inference task is image-to-image regression. In such a case, the inference task result 208 may be a regression image that the primary processing channel 204 has predicted for the medical image 104 (e.g., the inference task result 208 may be a predicted denoised version of the medical image 104; the inference task result 208 may be a predicted modality transformed version of the medical image 104).

[0097] In various instances, the attention map 210 may be an array of attention scores, where each attention score is a scalar weight representing or indicating how important or unimportant a corresponding pixel (or voxel) of the medical image 104 is with respect to the inference task result 208. Thus, the attention map 210 may exhibit the same format, size, or dimensionality as the medical image 104. As a non-limiting example, assume that the medical image 104 is an x×y array of pixels. In such a case, the attention map 210 may be an x×y array of attention scores, and the attention score (i, j) of such an array may be a real-valued scalar whose magnitude indicates, for any suitable positive integer 1≤i≤x and 1≤j≤y, the contribution of the pixel (i, j) of the medical image 104 in generating the inference task result 208. As another non-limiting example, assume that the medical image 104 is an x×y×z array of voxels. In such a case, the attention map 210 can be an x×y×z array of attention scores, and the attention score (i,j,i) of such an array can be a real-valued scalar whose magnitude indicates the degree to which the voxel (i,j,k) of the medical image 104 contributes in generating the inference task result 208 for any suitable positive integer 1≤i≤x, 1≤j≤y, and 1≤k≤z.

[0098] In various cases, the magnitude of any attention score indicated by the attention map 210 may vary continuously or discretely between any suitable minimum value (e.g., 0) indicating the minimum contribution to the inference task and any suitable maximum value (e.g., 1) indicating the maximum contribution to the inference task. However, for ease of explanation, this is merely a non-limiting example. In various other cases, the minimum value may alternatively represent the maximum contribution, and the maximum value may alternatively represent the minimum contribution.

[0099] In any case, the attention map 210 may be viewed as showing which particular pixels (or voxels) of the medical image 104 are more or less decisive / influential with respect to the inference task. In other words, the primary processing channel 204 may perform the inference task on the medical image 104, and the auxiliary processing channel 206 may predict which pixels (or voxels) of the medical image 104 are or have been focused on most (or least) by the primary processing channel 204. In yet other words, for any given pixel (or voxel) in the medical image 104, the auxiliary processing channel 206 may be viewed as guessing how focused the primary processing channel 204 is on that given pixel (or voxel). Figures 4 to 5 Non-limiting aspects are described.

[0100] Figures 4 to 5 Illustrated are example non-limiting examples of medical images and corresponding attention maps in accordance with one or more embodiments described herein.

[0101] First, consider Figure 4 . Figure 4 A non-limiting example is illustrated in which the medical image 104 is a two-dimensional sinogram. In this non-limiting example, the attention map 210 is a two-dimensional array having the same format, size, or dimension as the sinogram, and showing how important or unimportant each individual pixel of the sinogram is relative to the performance of the inference task. In particular, the darker areas of the attention map 210 may be considered to indicate, represent, or otherwise identify pixels of the sinogram that do not strongly affect or act on the inference task result 208. In other words, such darker areas of the attention map 210 may be considered to show which pixels of the sinogram predicted or determined by the auxiliary processing channel 206 do not receive much attention or focus from the main processing channel 204. Conversely, the brighter areas of the attention map 210 may be considered to indicate, represent, or otherwise identify pixels of the sinogram that do strongly affect or act on the inference task result 208. That is, such brighter areas of the attention map 210 may be viewed as showing which pixels of the sinusoidal pattern predicted or determined by the auxiliary processing channel 206 receive much attention or focus from the primary processing channel 204 .

[0102] Next, consider Figure 5 . Figure 5 A non-limiting example is illustrated in which the medical image 104 is a two-dimensional CT scan image. In this non-limiting example, the attention map 210 is a two-dimensional array having the same format, size, or dimension as the CT scan image and showing how important or unimportant each individual pixel of the CT scan image is with respect to the performance of the inference task. In particular, the darker areas of the attention map 210 may be considered to indicate, represent, or otherwise identify pixels of the CT scan image that do not strongly influence or contribute to the inference task result 208 (e.g., showing which pixels of the CT scan image are predicted or determined by the auxiliary processing channel 206 to not receive much attention or focus from the main processing channel 204). Conversely, the lighter areas of the attention map 210 may be considered to indicate, represent, or otherwise identify pixels of the CT scan image that do strongly influence or contribute to the inference task result 208 (e.g., showing which pixels of the CT scan image are predicted or determined by the auxiliary processing channel 206 to receive much attention or focus from the main processing channel 204).

[0103] In any case, the execution component 112 can execute the deep learning neural network 202 on the medical image 104, and such execution can produce both the inference task results 208 and the attention map 210 as output.

[0104] In various embodiments, the display component 114 may electronically render the inference task result 208 or the attention map 210 on any suitable electronic display (e.g., a computer screen, a computer monitor). Thus, a user or technician associated with the attention system 102 may visually inspect the inference task result 208 or the attention map 210. Such visual inspection may allow the user or technician to be aware of not only the inference task result 208, but also which specific pixels (or voxels) of the medical image 104 were determined to contribute most to the inference task result 208. This may be viewed as providing the user or technician with clear, easily understood meaning or insight as to which specific visual content illustrated by the medical image 104 had the greatest or least impact on the inference task result 208. In some cases, such meaning or insight may assist the user or technician in troubleshooting the deep learning neural network 202.

[0105] As a non-limiting example, assume that the user or technician determines that the inference task result 208 is at least partially inaccurate or incorrect. In such a case, the user or technician may examine the attention map 210. It is possible that the attention map 210 indicates that a large amount of attention is given to an unexpected subset of pixels (or voxels) of the medical image 104. Therefore, the user or technician may determine that any visual content is exemplified by such an unexpected subset of pixels (or voxels) that distracts the primary processing channel 204 or otherwise interrupts the primary processing channel. In other words, the attention map 210 can help the user or technician identify visual content that is not unknown to the primary processing channel 204 but should be unknown to the primary processing channel 204. In other words, the attention map 210 can help the user or technician identify the reason why the primary processing channel 204 failed to correctly analyze the medical image 104, and such identification can be used to help retrain or fine-tune the primary processing channel 204 to avoid similar failures in the future.

[0106] In various aspects, the display component 114 can electronically transmit the inference task results 208 and / or the attention map 210 to any other suitable computing device (not shown).

[0107] In order to make the inference task result 208 or the attention map 210 correct, the deep learning neural network 202 may first undergo training, such as with respect to Figures 6 to 8 as described.

[0108] Figure 6A block diagram of an example non-limiting system 600 that can facilitate interpretable visual attention for deep learning according to one or more embodiments described herein is illustrated, the system including a training component and an annotated training dataset. As shown, in some cases, the system 600 can include the same components as the system 200, and can also include a training component 602 and an annotated training dataset 604.

[0109] In various embodiments, access component 110 can electronically receive, retrieve, or otherwise access annotated training dataset 604 from any suitable source, and training component 602 can electronically train deep learning neural network 202 based on annotated training dataset 604. Figures 7 and 8 Various non-limiting aspects have been described.

[0110] Figure 7 An exemplary non-limiting block diagram 700 of an annotated training dataset 604 is shown according to one or more embodiments described herein.

[0111] In various embodiments, the annotated training dataset 604 may include a set of training medical images 702. In various aspects, for any suitable positive integer q, the set of training medical images 702 may include q images: training medical image 702(1) to training medical image 702(q). In various instances, each training medical image in the set of training medical images may exhibit the same format, size, or dimension as the medical image 104. As a non-limiting example, assume that the medical image 104 is an x×y pixel array. In such a case, each training medical image in the set of training medical images 702 may also be an x×y pixel array. As another non-limiting example, assume that the medical image 104 is an x×y×z voxel array. In such a case, each training medical image in the set of training medical images 702 may also be an x×y×z voxel array.

[0112] In various cases, the annotated training data set 604 may include a set of truth inference task results 704. In various aspects, the set of truth inference task results 704 may correspond to (e.g., in a one-to-one manner) a set of training medical images 702, respectively. Thus, since the set of training medical images 702 may include q images, the set of truth inference task results 704 may include q inference task results: truth inference task result 704(1) to truth inference task result 704(q). In various cases, each truth inference task result in the set of truth inference task results 704 may have the same format, size, or dimensionality as the inference task result 208. In fact, in various instances, each truth inference task result in the set of truth inference task results 704 may be, indicate, or otherwise represent any correct or accurate inference task result that is known or believed to correspond to a corresponding one of the set of training medical images 702. As a non-limiting example, the truth inference task result 704(1) may correspond to the training medical image 702(1). Thus, the truth inference task result 704(1) may be considered as a known or believed correct or accurate classification label, segmentation mask, or regression output corresponding to the training medical image 702(1). As another non-limiting example, the truth inference task result 704(q) may correspond to the training medical image 702(q). Thus, the truth inference task result 704(q) may be considered as a known or believed correct or accurate classification label, segmentation mask, or regression output corresponding to the training medical image 702(q).

[0113] In various aspects, the annotated training dataset 604 may include a set of ground-truth attention maps 706(1). In various instances, the set of ground-truth attention maps 706 may correspond (e.g., in a one-to-one manner) to the set of training medical images 702, respectively. Thus, since the set of training medical images 702 may include q images, the set of ground-truth attention maps 706 may include q maps: ground-truth attention map 706(1) to ground-truth attention map 706(q). In various cases, each ground-truth attention map in the set of ground-truth attention maps 706 may have the same format, size, or dimensionality as the attention map 210. In fact, in various instances, each ground-truth attention map in the set of ground-truth attention maps 706 may be, indicate, or otherwise represent any correct or accurate attention map that is known or believed to correspond to a corresponding one of the set of training medical images 702. As a non-limiting example, the ground-truth attention map 706(1) may correspond to the training medical image 702(1). Thus, the ground-truth attention map 706(1) may be considered to indicate the correct or accurate corresponding attention score for each individual pixel (or voxel) of the training medical image 702(1). As another non-limiting example, the ground-truth attention map 706(q) may correspond to the training medical image 702(q). Thus, the ground-truth attention map 706(q) may be considered to indicate the correct or accurate corresponding attention score for each individual pixel (or voxel) of the training medical image 702(q).

[0114] In various aspects, a set of ground-truth attention maps 706 may be obtained or otherwise generated by performing edge, boundary, or contour localization on a set of training medical images 702. After all, when given a medical image depicting an anatomical structure, it may be the case that one or more visible edges, boundaries, or contours of the anatomical structure are particularly relevant for performing an inference task on the given medical image. Therefore, an edge detector (not shown) may be present. In various instances, the edge detector may be any suitable machine learning model that may be pre-trained in any suitable manner (e.g., supervised, unsupervised, reinforcement learning) to segment edges, boundaries, or contours that are visually perceptible in an input image. Therefore, the edge detector may be performed on a set of training medical images 702, respectively, and the resulting segmentation masks generated by the edge detector may be respectively treated as a set of ground-truth attention maps 706. As a non-limiting example, an edge detector may be performed on the training medical image 702(1), which may cause the edge detector to generate a segmentation mask indicating which pixels (or voxels) of the training medical image 702(1) belong to or otherwise constitute the visible edges, boundaries, or contours of any anatomical structure depicted in the training medical image 702(1), and such segmentation mask may be considered or taken as the ground-truth attention map 706(1). As another non-limiting example, an edge detector may be performed on the training medical image 702(q), which may cause the edge detector to generate a segmentation mask indicating which pixels (or voxels) of the training medical image 702(q) belong to or otherwise constitute the visible edges, boundaries, or contours of any anatomical structure depicted in the training medical image 702(q), and such segmentation mask may be considered or taken as the ground-truth attention map 706(q).

[0115] Note that the above use of an edge detector is merely a non-limiting example of how a set of ground-truth attention maps 706 may be obtained. In other embodiments, a set of ground-truth attention maps 706 may be obtained by executing any other suitable type of feature extractor on a set of training medical images 702, respectively. In such cases, the feature extractor may be trained in any suitable manner to receive an image and produce a segmentation mask that indicates which pixels (or voxels) of the image belong to any suitable visually perceptible features depicted in the image. In other words, the various embodiments described herein are not limited to only edge, boundary, or contour detection.

[0116] In various aspects, a set of ground-truth attention maps 706 may be obtained or otherwise generated by comparing a set of training medical images 702 to a set of ground-truth inference task results 704. In practice, when the inference task is image-to-image regression, then each ground-truth inference task result in the set of ground-truth inference task results 704 may be viewed as a transformed version of a corresponding one of the set of training medical images 702. Thus, it may be the case that any pixel (or voxel) whose intensity value undergoes the largest change between the set of training medical images 702 and the set of ground-truth inference task results 704 is particularly relevant for performing the inference task. Thus, a set of ground-truth attention maps 706 may be obtained based on corresponding differences calculated between the set of training medical images 702 and the set of ground-truth inference task results 704. As a non-limiting example, the ground-truth inference task results 704(1) may be subtracted from the training medical images 702(1), thereby producing a first difference array. In various cases, the ground-truth attention map 706(1) may be equal to or otherwise based on a normalized absolute value version of the first difference array. As another non-limiting example, the ground truth inference task result 704(q) may be subtracted from the training medical image 702(q), thereby producing a qth difference array. In various cases, the ground truth attention map 706(q) may be equal to or otherwise based on a normalized absolute value version of the qth difference array.

[0117] It should be noted that the above-described use of matrix subtraction between a set of truth inference task results 704 and a set of training medical images 702 is merely a non-limiting example of how a set of truth attention maps 706 may be obtained. In other embodiments, a set of truth attention maps 706 may be obtained by respectively calculating any other suitable type of relationship between a set of truth inference task results 704 and a set of training medical images 702. In other words, the various embodiments described herein are not limited to only calculating the subtraction difference between a set of truth inference task results 704 and a set of training medical images 702, respectively.

[0118] In various examples, a set of ground-truth attention maps 706 may be obtained or otherwise generated by measuring the pixel-level (or voxel-level) uncertainty that the primary processing channel 204 has previously exhibited relative to a set of training medical images 702. In practice, when the inference task is image segmentation or image-to-image regression, executing the primary processing channel 204 on a given medical image may cause the primary processing channel 204 to produce an output array (e.g., to produce a segmentation mask or a regression image). If a ground-truth inference task result is available for the given medical image, the ground-truth inference task result may be compared to the output array, and it may be the case that pixels (or voxels) whose intensity values ​​differ significantly between the ground-truth inference task result and the output array are particularly relevant for performing the inference task. Thus, a set of ground-truth attention maps 706 may be obtained by respectively comparing a set of ground-truth inference task results 704 with arrays previously output by the primary processing channel 204 based on a set of training medical images 702. As a non-limiting example, it may be the case that the primary processing channel 204 was previously (e.g., during a previous training period) executed on the training medical image 702(1), and such execution may have produced a first previous output array (e.g., a first previous segmentation mask or regression image predicted by the primary processing channel 204 for the training medical image 702(1)). In various aspects, the true value inference task result 704(1) may be subtracted from the first previous output array to produce a first difference array. In various instances, the true value attention map 706(1) may be equal to or otherwise based on a normalized absolute value version of the first difference array. As another non-limiting example, it may be the case that the primary processing channel 204 was previously (e.g., during a previous training period) executed on the training medical image 702(q), and such execution may have produced a qth previous output array (e.g., a qth previous segmentation mask or regression image predicted by the primary processing channel 204 for the training medical image 702(q)). In various aspects, the truth inference task result 704(q) can be subtracted from the qth previous output array, thereby producing a qth difference array. In various examples, the truth attention map 706(q) can be equal to or otherwise based on a normalized absolute value version of the qth difference array.

[0119] It should be noted that the above-mentioned techniques for obtaining a set of ground-truth attention maps 706 are merely non-limiting examples. In various other embodiments, any other suitable techniques or any other suitable information may be used to generate or obtain a set of ground-truth attention maps 706 (e.g., a segmentation mask generated from or otherwise associated with a set of training medical images 702 or a set of ground-truth inference task results 704 may be used; a brightness or contrast gradient exhibited within a set of training medical images 702 or a set of ground-truth inference task results 704 may be used; metadata such as imaging energy levels (kV peak settings, mA settings) associated with a set of training medical images 702 or a set of ground-truth inference task results 704 may be used; any other information that can reveal or influence the importance or unimportance of which regions of a given training medical image may be used).

[0120] Now, consider Figure 8 . Figure 8 An example non-limiting block diagram 800 showing how a deep learning neural network 202 may be trained is illustrated in accordance with one or more embodiments described herein.

[0121] In various embodiments, the training component 602 can execute the deep learning neural network 202 on the annotated training data set 604 as follows.

[0122] Prior to such training, the training component 602 may initialize the trainable internal parameters (eg, weight matrices, bias vectors, convolution kernels) of the primary processing channel 204 and the auxiliary processing channel 206 in any suitable manner (eg, random initialization).

[0123] In various aspects, the training component 602 can select any suitable training medical image, any suitable corresponding ground truth inference task result, and any suitable corresponding ground truth attention map from the annotated training dataset 604. These can be referred to as training medical image 802, ground truth inference task result 804, and ground truth attention map 806, respectively.

[0124] In various examples, the training component 602 may electronically execute the deep learning neural network 202 on the training medical image 802, and such execution may produce both an output 808 and an output 810. More specifically, the training component 602 may feed the training medical image 802 to the layer 204(1) of the primary processing channel 204, the training medical image 802 may complete a forward pass through the hidden layers of the primary processing channel 204, and the layer 204(n) of the primary processing channel 204 may compute the output 808. Furthermore, during such forward pass, the layer 204(j) of the primary processing channel 204 may compute one or more activations, such one or more activations may be fed to the layer 206(1) of the auxiliary processing channel 206, such one or more activations may complete a forward pass through the hidden layers of the auxiliary processing channel 206, and the layer 206(m) of the auxiliary processing channel 206 may compute the output 810.

[0125] Note that the format, size, or dimensionality of the output 808 may be dictated by the number, arrangement, size, or other characteristics of the neurons, convolution kernels, or other internal parameters of the layer 204 (n) (or any other layer of the primary processing channel 204). Thus, the output 808 may be forced to have any format, size, or dimensionality desired or otherwise suitable for the inference task by adding, removing, or otherwise adjusting the characteristics of the layer 204 (n) (or any other layer of the primary processing channel 204). Thus, the output 808 may be viewed as a predicted inference task result (e.g., a predicted classification label, a predicted segmentation mask, a predicted regression output) that the primary processing channel 204 believes should correspond to the training medical image 802. On the other hand, the true inference task result 804 may be a correct or accurate inference task result (e.g., a correct or accurate classification label, a correct or accurate segmentation mask, a correct or accurate regression output) that is known or believed to correspond to the training medical image 802. In various cases, if the primary processing channel 204 has undergone no or little training to date, the output 808 may be highly inaccurate (eg, may differ significantly from the truth inference task result 804).

[0126] Similarly, it is noted that the format, size, or dimensionality of the output 810 may be dictated by the number, arrangement, size, or other characteristics of the neurons, convolution kernels, or other internal parameters of the layer 206 (m) (or any other layer of the auxiliary processing channel 206). Thus, the output 810 may be forced to have the same format, size, or dimensionality as the training medical image 802 by adding, removing, or otherwise adjusting the characteristics of the layer 206 (m) (or any other layer of the auxiliary processing channel 206). Thus, the output 810 may be viewed as a predicted attention map (e.g., a predicted array of pixel-level (or voxel-level) attention scores) that the auxiliary processing channel 206 believes should correspond to the training medical image 802. On the other hand, the ground-truth attention map 806 may be a correct or accurate attention map (e.g., a correct or accurate array of pixel-level (or voxel-level) attention scores) that is known or believed to correspond to the training medical image 802. As described above, if the auxiliary processing channel 206 has not undergone training or has undergone little training to date, the output 810 may be extremely inaccurate (e.g., may differ greatly from the ground-truth attention map 806).

[0127] In various aspects, the training component 602 may calculate the loss 812 between the output 808 and the ground truth inference task result 804 via any suitable error or objective function (e.g., MAE, MSE, cross entropy). In various examples, the training component 602 may generate a weighted loss 812 by applying point-wise multiplication and summation to the loss 814 and the ground truth attention map 806. This may be viewed as weighting the loss 812 according to the pixel-level (or voxel-level) attention score indicated by the ground truth attention map 806. In various cases, the training component 602 may update the trainable internal parameters of the primary processing channel 204 by performing backpropagation (e.g., stochastic gradient descent) driven by the weighted loss 814. It is noted that the weighted loss 814 may be implemented regardless of the internal architecture of the primary processing channel 204. In other words, the pixel-level (or voxel-level) weighting according to the loss function of the ground truth attention map may be viewed as architecture agnostic or otherwise as plug-and-play.

[0128] Furthermore, in various aspects, the training component 602 can calculate a loss 816 between the output 810 and the ground-truth attention map 806 via any suitable error or objective function (e.g., MAE, MSE, cross entropy). In various examples, the training component 602 can update the trainable internal parameters of the auxiliary processing channel 206 by performing back propagation (e.g., stochastic gradient descent) driven by the loss 816.

[0129] In various aspects, the above training process may be repeated for any suitable number of training medical images (e.g., for each training medical image in the annotated training dataset 604). Such training may ultimately result in the trainable internal parameters of the primary processing channel 204 being iteratively optimized to accurately or correctly perform the inference task on the input medical image. Such training may also ultimately result in the trainable internal parameters of the auxiliary processing channel 206 being iteratively optimized to accurately or correctly predict the importance of individual pixels (or voxels) of such input medical images with respect to the inference task (e.g., to accurately or correctly predict which pixels (or voxels) the primary processing channel 204 will focus on).

[0130] In various cases, when training deep learning neural network 202, training component 602 can implement any suitable training termination criteria or any suitable training batch size.

[0131] and Figure 8 The associated discussion primarily describes the primary processing channel 204 and the auxiliary processing channel 206 as being trained simultaneously from scratch (e.g., initialized from random internal parameters) with each other. However, for ease of illustration and explanation, this is merely a non-limiting example. In various other embodiments, it may be the case that the deep learning neural network 202 initially includes the primary processing channel 204 without the auxiliary processing channel 206. In such cases, the training component 602 may train the primary processing channel 204 from scratch (e.g., initialized from random internal parameters) using a set of training medical images 702 and a set of true inference task results 704. Note that in some instances, such training of the primary processing channel 204 alone may involve utilizing a set of true attention maps 706, such as with respect to Figure 8 814). However, in other instances, such training of the primary processing channel 204 alone may not involve a set of ground-truth attention maps 706 (e.g., the training component 602 may refrain from calculating the weighted loss 814, and the primary processing channel 204 may therefore be updated via back-propagation driven by the loss 812). Now, after the primary processing channel 204 has been trained in such a manner, the training component 602 may insert the auxiliary processing channel 206 into the deep learning neural network 202. After such insertion, the training component 602 may randomly initialize the trainable internal parameters of the auxiliary processing channel 206. However, because the primary processing channel 204 may have already undergone some training in such a scenario, the training component 602 may refrain from randomly initializing the trainable internal parameters of the primary processing channel 204. The training component 602 may then train both the auxiliary processing channel 206 and the primary processing channel 204, such as with respect to the training component 602. Figure 8In various aspects, this can be viewed as training the auxiliary processing channel 206 from scratch while the primary processing channel 204 is being fine-tuned.

[0132] In various aspects, it is noted that weighting the training outputs (e.g., 808) produced by the primary processing channel 204 according to a set of ground truth attention maps 706 can significantly improve the performance of the primary processing channel 204. In fact, the inventors of various embodiments described herein have experimentally verified such performance improvements, such as relative to Figures 9 to 11 as described.

[0133] Figures 9 to 11 Example non-limiting experimental results on explainable visual attention for deep learning according to one or more embodiments described herein are illustrated.

[0134] In particular, various embodiments described herein were put into practice during various experiments. In such experiments, the inference task was image-to-image regression. More specifically, the inference task was to convert or transform a low-energy CT scan image into a high-energy CT scan image. In such experiments, a first version of the main processing channel 204 was put into practice, wherein such first version was trained without utilizing a set of true value attention maps 706. Also in such experiments, a second version of the main processing channel 204 was put into practice, wherein such second version was trained by utilizing a set of true value attention maps 706. Such experiments involved performing both the first version and the second version of the main processing channel 204 on a low-energy CT scan image for which high-energy truth was available. During such experiments, the performance of the second version of the main processing channel 204 was superior to that of the first version of the main processing channel 204. In particular, the second version of the main processing channel 204 was able to more accurately predict high-energy CT scan images (e.g., with a higher Dice score) than the first version of the main processing channel 204. In fact, Fig. 9 A ground-truth high energy image 900 corresponding to a particular low energy CT scan image is illustrated. During such experiments, a first version of the primary processing channel 204 was performed on the particular low energy CT scan image, thereby producing Fig.10 1000. Also during such experiments, a second version of the primary processing channel 204 was performed on this particular low energy CT scan image, thereby producing Fig.11 The predicted high energy image 1100 is shown. It can be seen that the predicted high energy image 1100 is more similar to the ground truth high energy image 900 than the predicted high energy image 1000. In other words, weighting the training loss function of the main processing channel 204 according to a set of ground truth attention maps 706 significantly improves the performance of the main processing channel 204.

[0135] Fig.12 A flowchart illustrating an example non-limiting computer-implemented method 1200 that can facilitate interpretable visual attention for deep learning according to one or more embodiments described herein. In various cases, the attention system 102 can facilitate the computer-implemented method 1200.

[0136] In various embodiments, act 1202 may include accessing, by a device (eg, via 110 ) operatively coupled to a processor (eg, 106 ) a medical image (eg, 104 ) generated by a medical imaging scanner.

[0137] In various aspects, action 1204 may include performing an inference task on the medical image by a device (e.g., via 112) and via execution of a deep learning neural network (e.g., 202), wherein the deep learning neural network may receive the medical image as an input and may produce as output both an inference task result (e.g., 208) and an attention map (e.g., 210) that indicates which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

[0138] In various examples, act 1206 can include visually rendering, by the device (eg, via 114 ), the inference task results and the attention map on an electronic display.

[0139] Although not in Fig.12, but the computer-implemented method 1200 may include: training a deep learning neural network on an annotated training data set (e.g., 604) by a device (e.g., via 602). In various aspects, the deep learning neural network may include a first processing channel (e.g., 204) and a second processing channel (e.g., 206). In various instances, the annotated training data set may include a training medical image (e.g., 802), a truth inference task result corresponding to the training medical image (e.g., 804), and a truth attention map corresponding to the training medical image (e.g., 806). In various cases, the training may include: randomly initializing trainable internal parameters of the first processing channel and the second processing channel by a device (e.g., via 602); executing the deep learning neural network on the training medical image by the device (e.g., via 602) so that the first processing channel produces a first output (e.g., 808) and the second processing channel produces a second output (e.g., 810); calculating, by the device (e.g., via 602), a first loss between the truth inference task result and the first output (e.g., 812); calculating, by the device (e.g., via 60 2) weighting the first loss via point-wise multiplication with the true-valued attention map (e.g., thereby producing 814); updating, by the device (e.g., via 602), trainable internal parameters of the first processing channel via back-propagation driven by the weighted first loss (e.g., driven by 814); calculating, by the device (e.g., via 602), a second loss (e.g., 816) between the true-valued attention map and the second output; and updating, by the device (e.g., via 602), trainable internal parameters of the second processing channel via back-propagation driven by the second loss.

[0140] Although in Fig.12 It is not explicitly shown in , but the inference task can be image-to-image regression, and the ground-truth attention map can be based on the relationship between the training medical images and the ground-truth inference task results (e.g., as relative to Figure 7 described).

[0141] Although in Fig.12 It is not explicitly shown in , but the inference task can be image segmentation or image-to-image regression, and the ground-truth attention map can be generated by executing a feature extractor on a training medical image (e.g., as relative to Figure 7 described).

[0142] Although in Fig.12 , but the inference task may be image segmentation or image-to-image regression, and the ground-truth attention map may indicate the pixel-level or voxel-level uncertainty or error exhibited by the deep learning neural network during previous training epochs relative to the training medical images (e.g., as relative to Figure 7 described).

[0143] Although in Fig.12 It is not explicitly shown in , but the inference task result may be generated by a first processing channel (e.g., 204) of the deep learning neural network, the attention map may be generated by a second processing channel (e.g., 206) of the deep learning neural network, the second processing channel may branch out from at least a portion of the first processing channel and be parallel to at least a portion of the first processing channel (e.g., parallel to 204(j+1) to 204(n)), the device (e.g., via 602) may train the first processing channel from scratch before the second processing channel is inserted into the deep learning neural network, and the device (e.g., via 602) may train the second processing channel from scratch while fine-tuning the first processing channel.

[0144] For ease of illustration and explanation, although the disclosure herein mainly describes various embodiments as being applied to medical images, this is only a non-limiting example. In various cases, the various embodiments described herein may be applied to promote visual attention to any suitable image (e.g., even to non-medical images).

[0145] Various embodiments described herein may include a computer program product for facilitating interpretable visual attention for deep learning. In various aspects, the computer program product may include a non-transitory computer readable memory (e.g., 108) having program instructions embodied therein. In various instances, the program instructions may be executable by a processor (e.g., 106) to cause the processor to access a deep learning neural network (e.g., 202) including a main processing channel (e.g., 204), wherein the main processing channel may be trained using a set of training images (e.g., 702) and a set of true value inference task results (e.g., 704) corresponding to the set of training images, respectively, to perform inference tasks. In various cases, the program instructions may be further executable to cause the processor to insert an auxiliary processing channel (e.g., 206) into the deep learning neural network, so that the auxiliary processing channel branches out from the main processing channel and is parallel to at least a portion of the main processing channel. In various aspects, the program instructions may further be executable to cause the processor to train the auxiliary processing channel using a set of ground-truth attention maps (e.g., 706) corresponding to a set of training images, respectively, to generate pixel-level or voxel-level attention maps that indicate where within the input image the deep learning neural network focuses when performing an inference task. In various aspects, a set of ground-truth attention maps may be based on a relationship between a set of training images and a set of ground-truth inference task results. In various other aspects, a set of ground-truth attention maps may be generated via executing a feature extractor on a set of training images. In yet other aspects, a set of ground-truth attention maps may respectively indicate which areas of a set of training images were analyzed uncertainly or incorrectly by the primary processing channel in one or more previous training periods.

[0146] As described herein, and at least Figure 4 As shown, a sinusoid can be considered as a type of image.

[0147] In various instances, the machine learning algorithm or model may be implemented in any suitable manner to facilitate any suitable aspect described herein. In order to facilitate some of the above machine learning aspects of various embodiments, consider the following discussion of artificial intelligence (AI). Various embodiments described herein may employ artificial intelligence to facilitate the automation of one or more features or functionalities. These components may employ various AI-based schemes to perform various embodiments / examples disclosed herein. In order to provide or contribute to the numerous determinations described herein (e.g., determination, detection, inference, calculation, prediction, prognosis, estimation, derivation, forecast, detection, estimation), the components described herein may examine the entirety or subset of the data to which they are granted access rights, and may provide reasoning about or determine the state of a system or environment from a set of observations captured via events and / or data. For example, determination may be used to identify a specific context or action, or a probability distribution of a state may be generated. These determinations may be probabilistic; that is, the calculation of the probability distribution of the state of interest is based on consideration of data and events. Determination may also refer to a technique for composing a higher-level event from a set of events or data.

[0148] Such determinations may result in the construction of new events or actions from a set of observed events or stored event data, whether or not the events are closely related in time, and whether or not the events and data come from one or more event and data sources. The components disclosed herein may employ various classification (explicitly trained (e.g., via training data) and implicitly trained (e.g., via observed behavior, preferences, historical information, receiving external information, etc.)) schemes or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in conjunction with performing automatic or determined actions related to the claimed subject matter. Thus, the classification schemes or systems may be used to automatically learn and perform a variety of functions, actions, or determinations.

[0149] The classifier can take the input attribute vector z=(z1,z2,z3,z4,z n ) is mapped to the confidence that the input belongs to a certain category, such as according to f(z) = confidence(category). Such classification can use probability-based or statistical analysis (e.g., analyzing utility and cost considerations) to determine the actions to be automatically performed. Support vector machines (SVMs) may be examples of classifiers that can be used. SVMs operate by finding a hypersurface in the space of possible inputs, where the hypersurface attempts to separate triggering criteria from non-triggering events. Intuitively, this makes the classification correct for testing data that is close to but different from the training data. Other directed and undirected model classification methods include, for example, naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, or any of the probabilistic classification models that provide different independent patterns. Classification as used herein also includes statistical regression for developing priority models.

[0150] In order to provide additional context for the various embodiments described herein, Fig.13 The following discussion is intended to provide a brief general description of a suitable computing environment 1300 in which various embodiments of the embodiments described herein may be implemented. Although the embodiments have been described above in the general context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will recognize that the embodiments may also be implemented in conjunction with other program modules or as a combination of hardware and software.

[0151] Generally, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In addition, those skilled in the art will appreciate that the methods of the present invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc., each of which is operatively coupled to one or more associated devices.

[0152] The illustrated embodiments of the embodiments herein may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0153] Computing devices typically include various media, which may include computer-readable storage media, machine-readable storage media, or communication media, where the two terms are used differently in this article, as described below. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by a computer, and include volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable storage media or machine-readable storage media can be implemented in conjunction with any method or technology for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0154] Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disk (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible or non-transitory media that can be used to store the desired information. In this regard, the terms "tangible" or "non-transitory" as applied to storage, memory, or computer-readable media herein should be understood to exclude only propagating transient signals themselves as a modifier, and not to disclaim all standard storage, memory, or computer-readable media that are not merely propagating transient signals themselves.

[0155] Computer-readable storage media may be accessed by one or more local or remote computing devices, eg, via access requests, queries, or other data retrieval protocols, to perform various operations relative to the information stored by the media.

[0156] Communication media typically embodies computer readable instructions, data structures, program modules, or other structured or unstructured data in a data signal, which may be, for example, a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery or transmission media. The term "modulated data signal" or "signal" refers to a signal that has one or more of its characteristics set or changed to encode information in one or more signals. By way of example, and not limitation, communication media include wired media (such as a wired network or direct-wired connection) and wireless media (such as acoustic, RF, infrared and other wireless media).

[0157] Reference again Fig.13 , an example environment 1300 for implementing various embodiments of various aspects described herein includes a computer 1302, which includes a processing unit 1304, a system memory 1306, and a system bus 1308. The system bus 1308 couples system components including, but not limited to, the system memory 1306 to the processing unit 1304. The processing unit 1304 can be any of a variety of commercially available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1304.

[0158] The system bus 1308 may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 1306 includes ROM 1310 and RAM 1312. A basic input / output system (BIOS) may be stored in a nonvolatile memory such as ROM, erasable programmable read-only memory (EPROM), EEPROM, where the BIOS contains basic routines that help transfer information between elements within the computer 1302, such as during startup. The RAM 1312 may also include high-speed RAM, such as static RAM for caching data.

[0159] The computer 1302 also includes an internal hard disk drive (HDD) 1314 (e.g., EIDE, SATA), one or more external storage devices 1316 (e.g., a magnetic floppy disk drive (FDD) 1316, a memory stick or flash drive reader, a memory card reader, etc.), and a drive 1320, such as a solid-state drive, an optical drive, which can read or write from a disk 1322 (such as a CD-ROM disk, a DVD, a BD, etc.). Alternatively, in the case of a solid-state drive, the disk 1322 will not be included unless it is separated. Although the internal HDD 1314 is illustrated as being located within the computer 1302, the internal HDD 1314 can also be configured to be used externally in a suitable infrastructure (not shown). In addition, although not shown in the environment 1300, a solid-state drive (SSD) can be used in addition to or instead of the HDD 1314. The HDD 1314, external storage device 1316, and drive 1320 may be connected to the system bus 1308 via a HDD interface 1324, an external storage interface 1326, and a drive interface 1328, respectively. The interface 1324 for an external drive implementation may include at least one or both of a Universal Serial Bus (USB) and an Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technology. Other external drive connection technologies are within the contemplation of the embodiments described herein.

[0160] The drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For the computer 1302, the drives and storage media accommodate the storage of any data in a suitable digital format. Although the above description of computer-readable storage media refers to corresponding types of storage devices, it should be understood by those skilled in the art that other types of computer-readable storage media (whether currently existing or developed in the future) can also be used in the exemplary operating environment, and further, any such storage media can contain computer-executable instructions for performing the methods described herein.

[0161] A number of program modules may be stored in the drives and RAM 1312, including an operating system 1330, one or more application programs 1332, other program modules 1334, and program data 1336. All or portions of the operating system, applications, modules, or data may also be cached in RAM 1312. The systems and methods described herein may be implemented using various commercially available operating systems or combinations of operating systems.

[0162] Computer 1302 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment for operating system 1330, and the emulated hardware may optionally be different from the hardware of the operating system 1330. Fig.13 1302. In such embodiments, operating system 1330 may include one of a plurality of virtual machines (VMs) hosted at computer 1302. In addition, operating system 1330 may provide an operating environment, such as a Java operating environment or a .NET framework, to application 1332. The operating environment is a consistent execution environment that allows application 1332 to run on any operating system that includes the operating environment. Similarly, operating system 1330 may support containers, and application 1332 may be in the form of a container that is a lightweight, standalone, executable software package that includes, for example, the application's code, runtime, system tools, system libraries, and settings.

[0163] In addition, the computer 1302 may be enabled with a security module, such as a trusted processing module (TPM). For example, in the case of a TPM, the boot component is hashed in the next boot component, and the result is waited for to match the security value before loading the next boot component. This process may occur at any layer in the code execution stack of the computer 1302, such as applied to the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0164] A user may enter commands and information into the computer 1302 through one or more wired / wireless input devices, such as a keyboard 1338, a touch screen 1340, and a pointing device such as a mouse 1342. Other input devices (not shown) may include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control or other remote control, a joystick, a virtual reality controller or virtual reality headset, a game pad, a stylus, an image input device (e.g., a camera), a gesture sensor input device, a visual movement sensor input device, an emotion or facial detection device, a biometric input device (e.g., a fingerprint or iris scanner), and the like. These and other input devices are often connected to the processing unit 1304 through an input device interface 1344, which may be coupled to the system bus 1308, but these and other input devices may be connected through other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR port, a fingerprint scanner, a fingerprint scanner, or the like. interface, etc.) connection.

[0165] A monitor 1346 or other type of display device may also be connected to the system bus 1308 via an interface, such as a video adapter 1348. In addition to the monitor 1346, computers typically include other peripheral output devices (not shown), such as speakers, printers, and the like.

[0166] Computer 1302 can work in a networked environment using logical connections to one or more remote computers, such as remote computer 1350, via wired or wireless communications. Remote computer 1350 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment appliance, peer device, or other public network node, and typically includes many or all of the elements described with respect to computer 1302, although only memory / storage device 1352 is illustrated for simplicity. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1354 or a larger network, such as a wide area network (WAN) 1356. Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global communication network, such as the Internet.

[0167] When used in a LAN networking environment, the computer 1302 can be connected to the local network 1354 through a wired or wireless communication network interface or adapter 1358. The adapter 1358 can facilitate wired or wireless communication with the LAN 1354, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter 1358 in a wireless mode.

[0168] When used in a WAN networking environment, the computer 1302 may include a modem 1360, or may be connected to a communications server on the WAN 1356 via other means for establishing communications over the WAN 1356, such as through the Internet. The modem 1360, which may be an internal or external device and a wired or wireless device, may be connected to the system bus 1308 via the input device interface 1344. In a networked environment, program modules depicted relative to the computer 1302, or portions thereof, may be stored in the remote memory / storage device 1352. It should be appreciated that the network connections shown are examples and other means of establishing a communications link between the computers may be used.

[0169] When used in a LAN or WAN networking environment, in addition to or as an alternative to the external storage devices 1316 described above, the computer 1302 may access a cloud storage system or other network-based storage system, such as, but not limited to, a network virtual machine that provides one or more aspects of information storage or processing. Generally, the connection between the computer 1302 and the cloud storage system may be established through the LAN 1354 or WAN 1356, for example, by an adapter 1358 or a modem 1360, respectively. When the computer 1302 is connected to the associated cloud storage system, the external storage interface 1326 may manage the storage provided by the cloud storage system with the aid of the adapter 1358 or the modem 1360, as other types of external storage devices. For example, the external storage interface 1326 may be configured to provide access to cloud storage sources as if those sources were physically connected to the computer 1302.

[0170] The computer 1302 may be operable to communicate with any wireless device or entity that is operatively arranged to wirelessly communicate, such as a printer, scanner, desktop or portable computer, portable data assistant, communication satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a self-service machine, a newsstand, a store shelf, etc.), and a telephone. This may include Wireless Fidelity (Wi-Fi) and Wireless technology. Therefore, the communication can be a predefined structure like a conventional network, or just an ad hoc communication between at least two devices.

[0171] Fig.1414 is a schematic block diagram of a sample computing environment 1400 with which the disclosed subject matter can interact. The sample computing environment 1400 includes one or more clients 1410. The client 1410 can be hardware or software (e.g., a thread, a process, a computing device). The sample computing environment 1400 also includes one or more servers 1430. The server 1430 can also be hardware or software (e.g., a thread, a process, a computing device). For example, the server 1430 can accommodate threads to perform transformations by adopting one or more embodiments as described herein. One possible communication between the client 1410 and the server 1430 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1400 includes a communication framework 1450 that can be used to facilitate communication between the client 1410 and the server 1430. The client 1410 is operably connected to one or more client data repositories 1420, which can be used to store information local to the client 1410. Similarly, server 1430 may be operably connected to one or more server data repositories 1440 , which may be used to store information local to server 1430 .

[0172] Various embodiments can be systems, methods, devices or computer program products at any possible level of technical detail of integration. A computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for various aspects of various embodiments implemented by a processor. A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination of the foregoing. An incomplete list of more specific examples of computer-readable storage media may also include the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the above items. As used herein, computer-readable storage media should not be understood as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0173] Computer-readable program instructions as described herein can be downloaded to corresponding computing / processing equipment from computer-readable storage media, or downloaded to external computers or external storage devices via networks (e.g., the Internet, local area networks, wide area networks, or wireless networks). Networks can include copper transmission cables, optical transmission optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers. Network adapter cards or network interfaces in each computing / processing equipment receive computer-readable program instructions from the network, and forward computer-readable program instructions for storage in computer-readable storage media in corresponding computing / processing equipment. Computer-readable program instructions for the operation of various embodiments for implementation can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming languages ​​or similar programming languages). Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform various aspects.

[0174] Flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to various embodiments are used to describe various aspects described herein. It should be understood that each frame of a flowchart illustration or a block diagram, and a combination of frames in a flowchart illustration or a block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a component for implementing the function / action specified in one or more frames of a flowchart or a block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device or other equipment to act in a particular manner, so that a computer-readable storage medium with instructions stored therein includes an article, which includes instructions for implementing various aspects of the function / action specified in one or more frames of a flowchart or a block diagram. Computer-readable program instructions can also be loaded onto a computer, other programmable data processing devices or other devices, so that a series of operating actions are performed on a computer, other programmable devices or other devices to produce a computer-implemented process, so that the instructions executed on a computer, other programmable devices or other devices implement the function / action specified in one or more frames of a flowchart or a block diagram.

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible specific implementation architecture, functionality and operation of the system, method and computer program product according to various embodiments. In this regard, each frame in the flow chart or block diagram can represent a module, fragment or part of an instruction, which includes one or more executable instructions for realizing a specified logical function. In some alternative specific implementations, the function indicated in the frame may not occur in the order indicated in the figure. For example, in fact, the two frames shown in succession can be executed substantially at the same time, or sometimes these frames may be executed in reverse order, depending on the functionality involved. It will also be noted that the combination of each frame of the block diagram or flow chart illustration and the frame in the block diagram or flow chart illustration can be realized by a system based on dedicated hardware that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0176] Although the present subject matter has been described above in the general context of computer executable instructions of a computer program product running on one or more computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. Typically, a program module includes routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that various aspects may be practiced with other computer system configurations, including single-processor computer systems or multi-processor computer systems, small computing devices, large computers, and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronics, etc. The illustrated aspects may also be practiced in a distributed computing environment in which tasks are performed by a remote processing device linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0177] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to or include computer-related entities or entities related to an operating machine having one or more specific functionalities. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program, a processor, an object, an executable file, an execution thread, a program, or a computer running on a processor. By way of illustration, both an application running on a server and a server may be components. One or more components may reside within a process or a thread of execution, and a component may be located on a computer or distributed between two or more computers. As another example, the corresponding component may be executed according to various computer-readable media having various data structures stored thereon. The components may communicate via a local or remote process, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, or a network (such as the Internet with other systems) via a signal). As another example, a component may be a device having a specific functionality provided by a mechanical part operated by an electrical or electronic circuit, which is operated by a software or firmware application executed by a processor. In such cases, the processor may be internal or external to the device and may execute at least a portion of a software or firmware application. As yet another example, a component may be a device that provides specific functionality through electronic components rather than mechanical parts, where the electronic components may include a processor or other means for executing software or firmware that at least partially imparts the functionality to the electronic components. In one aspect, the component may emulate an electronic component, such as via a virtual machine within a cloud computing system.

[0178] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive replacement. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. As used herein, the term "and / or" is intended to have the same meaning as "or". In addition, unless otherwise specified or clear from the context that it is for a singular form, the articles "a" and "an" used in this specification and the drawings should generally be understood to mean "one or more". As used herein, the term "example" or "exemplary" is used to indicate use as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as an "example" or "exemplary" is not necessarily to be understood as being preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0179] The disclosure herein describes non-limiting examples. For ease of description or explanation, when discussing various examples, the various parts of the disclosure herein utilize the terms "each," "each," or "all." Such usage of the terms "each," "each," or "all" is non-limiting. In other words, when the disclosure herein provides a description of "each," "each," or "all" specific objects or parts that are applied to some specific objects or parts, it should be understood that this is a non-limiting example, and it should also be understood that in various examples, it may be that such description is applicable to less than "each," "each," or "all" specific objects or parts in the specific object or part.

[0180] As used in this specification, the term "processor" may refer to substantially any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize nanoscale architectures (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of user equipment. The processor may also be implemented as a combination of computing processing units. In the present disclosure, terms such as "repository", "storage device", "data repository", "data storage device", "database", and substantially any other information storage component related to the operation and functionality of the component are used to refer to a "memory component", an entity embodied in a "memory", or a component including a memory. It should be understood that the memory or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that can act as an external cache memory. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the system or computer-implemented method herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0181] What has been described above includes only examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing the present disclosure, but many other combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including," "having," "having," and the like are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "comprising," as interpreted when "including" is used as a transitional word in a claim.

[0182] Descriptions of various embodiments have been given for purposes of illustration, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best illustrate the principles of the embodiments, practical applications or technical improvements over technologies found on the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system, comprising: A processor (e.g., 106) that executes computer-executable components stored in a non-transitory computer-readable memory (e.g., 108), wherein the computer-executable components include: an access component (e.g., 110) that accesses a medical image (e.g., 104) generated by a medical imaging scanner; and An execution component (e.g., 112) that performs an inference task on the medical image via execution of a deep learning neural network (e.g., 202), wherein the deep learning neural network receives the medical image as an input and produces as output both an inference task result (e.g., 208) and an attention map (e.g., 210), the attention map indicating which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

2. The system of claim 1, wherein the computer executable component further comprises: A display component (e.g., 114) that visually renders the inference task results and the attention map on an electronic display.

3. The system of claim 1 , wherein the computer executable component further comprises: A training component (e.g., 602) is provided to train the deep learning neural network on an annotated training dataset (e.g., 604).

4. The system of claim 3, wherein the deep learning neural network comprises a first processing channel (e.g., 204) and a second processing channel (e.g., 206); wherein the annotated training dataset comprises training medical images (e.g., 802), truth inference task results corresponding to the training medical images (e.g., 804), and truth attention maps corresponding to the training medical images (e.g., 806); and wherein the training component: randomly initializing trainable internal parameters of the first processing channel and the second processing channel; executing the deep learning neural network on the training medical image such that the first processing channel produces a first output (e.g., 808) and the second processing channel produces a second output (e.g., 810); Calculating a first loss between the truth inference task result and the first output (e.g., 812); weighting the first loss via point-wise multiplication with the ground-truth attention map; updating the trainable internal parameters of the first processing channel via back-propagation driven by the weighted first loss (e.g., 814); Calculating a second loss between the ground-truth attention map and the second output (e.g., 816); and The trainable internal parameters of the second processing channel are updated via back-propagation driven by the second loss.

5. The system of claim 4, wherein the inference task is image-to-image regression, and wherein the ground-truth attention map is based on the relationship between the training medical images and the ground-truth inference task results.

6. The system of claim 4, wherein the inference task is image segmentation or image-to-image regression, and wherein the ground-truth attention map is generated via executing a feature extractor on the training medical image.

7. The system of claim 4, wherein the inference task is image segmentation or image-to-image regression, and wherein the ground-truth attention map indicates pixel-level or voxel-level uncertainty exhibited by the deep learning neural network during a prior training epoch with respect to the training medical image.

8. The system of claim 3, wherein the inference task result is generated by a first processing channel (e.g., 204) of the deep learning neural network, wherein the attention map is generated by a second processing channel (e.g., 206) of the deep learning neural network, wherein the second processing channel branches off from the first processing channel and is parallel to at least a portion of the first processing channel, wherein before the second processing channel is inserted into the deep learning neural network, the training component initializes the training of the first processing channel according to first random internal parameters, and wherein the training component initializes the training of the second processing channel according to second random internal parameters while fine-tuning the first processing channel.

9. A computer-implemented method, the computer-implemented method comprising: accessing, by a device (eg, via 110 ) operatively coupled to a processor (eg, 106 ), a medical image (eg, 104 ) generated by a medical imaging scanner; and An inference task is performed on the medical image by the device (e.g., via 112) and via execution of a deep learning neural network (e.g., 202), wherein the deep learning neural network receives the medical image as input and produces as output both an inference task result (e.g., 208) and an attention map (e.g., 210), the attention map indicating which pixels or voxels of the medical image the deep learning neural network focused on when generating the inference task result.

10. The computer-implemented method of claim 9, further comprising: The inference task results and the attention map are visually rendered by the device (eg, via 114 ) on an electronic display.

11. The computer-implemented method of claim 9, further comprising: The deep learning neural network is trained by the device (e.g., via 602) based on the annotated training dataset (e.g., 604).

12. The computer-implemented method of claim 11, wherein the deep learning neural network comprises a first processing channel (e.g., 204) and a second processing channel (e.g., 206); wherein the annotated training dataset includes training medical images (e.g., 802), a truth inference task result corresponding to the training medical image (e.g., 804) and a truth attention map corresponding to the training medical image (e.g., 806); and The training includes: Initializing, by the device, trainable internal parameters of the first processing channel and the second processing channel randomly; executing, by the device, the deep learning neural network on the training medical image, thereby causing the first processing channel to produce a first output (e.g., 808) and causing the second processing channel to produce a second output (e.g., 810); Calculating, by the device, a first loss between the truth inference task result and the first output (e.g., 812); weighting, by the device, the first loss via point-wise multiplication with the ground-truth attention map; updating, by the device via back-propagation driven by the weighted first loss (e.g., 814), the trainable internal parameters of the first processing channel; calculating, by the device, a second loss between the true attention map and the second output (e.g., 816); and The trainable internal parameters of the second processing channel are updated, by the device, via back-propagation driven by the second loss.

13. The computer-implemented method of claim 12, wherein the inference task is image-to-image regression, and wherein the ground-truth attention map is based on a relationship between the training medical images and the ground-truth inference task results.

14. The computer-implemented method of claim 12, wherein the inference task is image segmentation or image-to-image regression, and wherein the ground-truth attention map is generated via executing a feature extractor on the training medical image.

15. The computer-implemented method of claim 12, wherein the inference task is image segmentation or image-to-image regression, and wherein the ground-truth attention map indicates pixel-level or voxel-level uncertainty exhibited by the deep learning neural network during a prior training epoch with respect to the training medical images.

Images