Neural network interpretation method based on depth prior expectation gradient

Abstract

The invention belongs to the technical field of artificial intelligence and convolutional neural networks, provides a convolutional neural network feature attribution interpretation method which is more efficient, stable and high in adaptability, can clearly reveal an internal decision-making mechanism, and provides a neural network interpretation method based on a depth priori expectation gradient. Firstly, the feature extraction capability of a deep convolutional neural network CNN is utilized to process an input image to obtain feature representation, after a flattened feature vector is obtained, a probability distribution model is utilized to capture the distribution characteristic of the feature so as to obtain prior probability distribution, bilinear up-sampling is performed on the prior probability distribution, and the prior probability distribution is obtained. The method comprises the following steps: matching a spatial dimension of an original input image, integrating prior probability distribution matched with the spatial dimension of the original input image into an expected gradient frame, and calculating feature importance based on an expected gradient value to obtain a feature attribution graph capable of visually displaying a region of interest in the input image. The method is mainly applied to artificial intelligence detection and automatic control occasions.

Description

Technical Field

[0001] The present invention relates to the field of computer science, specifically to the fields of deep learning and explainable artificial intelligence, and particularly to the research and application of convolutional neural networks. This invention proposes a modeling method for post-feature attribution interpretation based on deep prior information, suitable for convolutional neural networks. Specifically, it relates to a neural network prediction method based on expected gradient interpretability based on deep prior information. Background Art

[0002] With the widespread application of deep neural networks in key areas such as healthcare, finance, and autonomous driving, the importance of explainable artificial intelligence (XAI) has become increasingly prominent. In these fields, understanding the model's decision-making process is crucial to building trust and ensuring safety. Traditional deep neural networks often behave like "black boxes" and, while capable of generating highly accurate predictions, lack a clear explanation of the basis for their decisions. Among the many XAI methods, feature attribution methods, particularly gradient-based techniques, are widely used because they can explain the behavior of deep neural networks by evaluating the impact of input features on model outputs.

[0003] Interpretability methods for convolutional neural networks (CNNs) have become particularly important. From a theoretical perspective, interpretation methods help reveal the semantic representation mechanisms of CNNs, particularly the characteristics and decision-making processes of the network's intermediate layers. These methods can identify potential representation flaws, such as erroneous logic or model overfitting caused by data bias, thereby promoting model optimization and reliability. From a practical perspective, interpretability research can promote the application of human-computer interaction in deep learning models. Clear semantics in high-level convolutional layers can help people trust the network's predictions. By providing interpretable prediction evidence, users can better trust and understand the model's predictions.

[0004] In the field of image processing, feature attribution maps are one of the most commonly used post-hoc visual explanation methods, highlighting the key features that model decisions rely on. Researchers at home and abroad have conducted extensive work on developing visual explanation methods for convolutional neural networks. However, existing explanation methods have several problems:

[0005] (1) Explanation methods lack sensitivity. Sensitivity requires that the explanation method can correctly reflect the importance of features. For example, if an input feature has a significant impact on the model's prediction, but its gradient value is exactly zero, the traditional gradient method may not capture the importance of this feature.

[0006] (2) Interpretation methods lack implementation invariance. Traditional gradient methods may exhibit discontinuities when dealing with piecewise nonlinear models, making it difficult for interpretation results to fully reflect the decision-making basis of the model. In terms of implementation invariance, traditional gradient methods are sensitive to implementation details of neural networks (such as parameter initialization or weight sharing), resulting in significant differences in interpretation results for functionally equivalent networks, thus affecting the consistency of interpretation.

[0007] (3) The baseline in the interpretation method does not meet the definition of missing. In the gradient attribution method, the choice of baseline is crucial to the accuracy of the attribution results. The definition of "missing" information is a core issue in the gradient method, because the baseline is usually regarded as the "missing" part or "default" state in the model input, representing the minimum or "no information" state of the input feature in a certain sense. Zero baseline is a common choice, but it is not always reasonable in many scenarios. In medical data sets, if the blood glucose value is zero as the baseline, this choice may lead to incorrect interpretation. Whether the existing baseline calculation method can truly represent the definition of "missing" information is still controversial.

[0008] In summary, existing model interpretation methods need more research to address challenges in sensitivity, implementation invariance, and missing definitions to improve their usability and credibility in practical applications. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, inspired by deep priors and the Shapley value, also known as the Shapley contribution value, a mathematical method for fairly distributing cooperative benefits in cooperative game theory, a more efficient, stable, and adaptable convolutional neural network feature attribution interpretation method is proposed. By conducting in-depth analysis and visualization of the feature attribution representation of the convolutional neural network prediction results, the present invention can clearly reveal its inherent decision-making mechanism. This method has significant effects in improving the interpretability of the model, enhancing user trust, and identifying potential defects in the model. It has important academic value and practical significance for promoting research and application in the fields of deep learning and explainable artificial intelligence. To this end, the technical solution adopted by the present invention is a neural network interpretation method based on deep prior expected gradient. First, the feature extraction capability of the deep convolutional neural network (CNN) is used to process the input image, and the feature representation is obtained from the final convolutional layer of the deep convolutional neural network (CNN). After obtaining the flattened feature vector, the probability distribution model is used to capture the distribution characteristics of the features, thereby obtaining the prior probability distribution. The prior probability distribution is bilinearly upsampled to match the spatial dimension of the original input image, and the prior probability distribution that matches the spatial dimension of the original input image is integrated into the expected gradient framework. By calculating the feature importance based on the expected gradient value, a feature attribution map that can intuitively display the focus area in the input image is obtained.

[0010] The specific steps are as follows:

[0011] (1) Deep feature extraction: The input image is processed using the feature extraction capability of the convolutional neural network (CNN) to obtain feature representations from the final convolutional layer. The output is a four-dimensional tensor of shape (B, C, H, W), where C represents the number of feature maps and H and W correspond to the spatial dimensions. The tensor is flattened along the spatial dimensions (H, W) to obtain a two-dimensional matrix of shape (N, C), where each row corresponds to a feature vector at a specific spatial position in the image.

[0012] (2) Calculation of prior probability distribution: After obtaining the flattened feature vector, the probability distribution model used is the Multivariate Gaussian Model (MGM) or the Bayesian Gaussian Mixture Model (BGMM) to capture the distribution characteristics of the features and thus obtain the prior probability distribution;

[0013] (3) Prior baseline calculation: The prior probability distribution obtained in the previous step is upsampled to match the spatial dimension of the original input image to obtain the prior distribution, and then the prior distribution is integrated into the expected gradient framework, that is, the prior baseline is obtained by integrating the inverse of the prior distribution into the original baseline of the expected gradient;

[0014] (4) Expected gradient calculation: The obtained prior baseline is used as the starting point of the path integral in the expected gradient method. Interpolation is performed between the prior baseline and the input image through uniform sampling. Each interpolation is input into the convolutional neural network (CNN). The gradient of a specific category is obtained through backpropagation. The gradient is then averaged. The gradient mean is multiplied by the input image and the prior baseline difference map through Riemann sum integral to obtain the feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map consistent with the resolution of the input image.

[0015] In the bird image classification task, a sparrow picture with a resolution of 224×224 is input to the pre-trained convolutional neural network ResNet-50 network. First, the network extracts features step by step through the convolution layer, and the last convolution layer outputs a 7×7×2048 feature tensor. After flattening the tensor into a 49×2048 feature matrix along the spatial dimension, the Bayesian Gaussian mixture model BGMM is used for multimodal distribution modeling. Through variational inference, three mixture components are automatically identified, corresponding to the feather texture, beak features and background area, that is, the prior probability distribution; then, the feature distribution mean output by BGMM is bilinearly upsampled to restore it to the original image. The image size is calculated to generate a spatially aligned prior probability mask, i.e., the prior distribution. By subtracting the prior distribution from the mean baseline sampled from the training dataset, i.e., the original baseline, we obtain a prior baseline that suppresses the characteristic response of birds. 80 interpolation points are uniformly sampled between the prior baseline and the input image. The gradient of the "sparrow" category is input into a CNN at each interpolation point. The mean gradient is multiplied by the difference map between the input image and the prior baseline through Riemann sum integral to obtain a feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map with a resolution of 224×224, showing that the model decision mainly depends on the sparrow's head outline, wing texture and tail shape, and the contribution of branches in the background.

[0016] 50-100 interpolation points are uniformly sampled on the path between the prior baseline and the input image to generate an intermediate input sequence. The convolutional neural network (CNN) outputs the gradient for each interpolation point, and the gradient mean is multiplied pixel by pixel by the difference between the prior baseline and the input image to obtain a feature importance score matrix. In the autonomous driving scenario, the input image is the image of the vehicle's brake light area. The importance score matrix is reorganized to the original image size, denoised by Gaussian filtering, and normalized to generate a high-resolution attribution heat map that intuitively displays the area of interest, including road signs or pedestrian outlines.

[0017] The characteristics and beneficial effects of the present invention are:

[0018] 1) The proposed interpretation method is sensitive. By modeling a baseline that better matches the missingness definition and incorporating it into the expected gradient framework, the proposed method integrates and averages the gradients by uniformly sampling multiple interpolation points along the path from the baseline to the input. This path integration approach covers the entire range of the model's nonlinear response and can capture the comprehensive contribution of segmented activation features.

[0019] 2) The technical solution proposed in the present invention has implementation invariance. The present invention is based on the theoretical framework of Shapley values, which satisfies the axioms of symmetry, linearity and invalidity, and achieves implementation invariance through the expected gradient approximation.

[0020] 3) The modeling approach proposed in this paper complies with the definition of missingness. By modeling the feature distribution using a deep prior, the inverse of the prior distribution is incorporated into the baseline. This baseline effectively characterizes missing information, thereby enhancing the credibility of the attribution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the overall framework flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the invention is an interpretability method based on enhanced gradient and class activation mapping: selecting a suitable baseline, performing feature extraction through forward propagation, using enhanced gradient to approximately calculate the Shapley value weight coefficient of each channel of the activation map, linearly weightedly combining each channel of the activation map to obtain a class activation map, and performing post-processing operations to obtain the final saliency map.

[0023] A neural network interpretation method based on deep prior expected gradient first uses the feature extraction capability of the deep convolutional neural network (CNN) to process the input image, obtains feature representation from the final convolutional layer of the deep convolutional neural network (CNN), and after obtaining the flattened feature vector, uses the probability distribution model to capture the distribution characteristics of the features to obtain the prior probability distribution. The prior probability distribution is bilinearly upsampled to match the spatial dimension of the original input image, and the prior probability distribution that matches the spatial dimension of the original input image is integrated into the expected gradient framework. By calculating the feature importance based on the expected gradient value, a feature attribution map that can intuitively display the focus area in the input image is obtained.

[0024] The specific steps are as follows:

[0025] (1) Deep feature extraction: The input image is processed using the feature extraction capability of the convolutional neural network (CNN) to obtain feature representations from the final convolutional layer. The output is a four-dimensional tensor of shape (B, C, H, W), where C represents the number of feature maps and H and W correspond to the spatial dimensions. The tensor is flattened along the spatial dimensions (H, W) to obtain a two-dimensional matrix of shape (N, C), where each row corresponds to a feature vector at a specific spatial position in the image.

[0026] (2) Calculation of prior probability distribution: After obtaining the flattened feature vector, the probability distribution model used is the Multivariate Gaussian Model (MGM) or the Bayesian Gaussian Mixture Model (BGMM) to capture the distribution characteristics of the features and thus obtain the prior probability distribution;

[0027] (3) Prior baseline calculation: The prior probability distribution obtained in the previous step is upsampled to match the spatial dimension of the original input image to obtain the prior distribution, and then the prior distribution is integrated into the expected gradient framework, that is, the prior baseline is obtained by integrating the inverse of the prior distribution into the original baseline of the expected gradient;

[0028] (4) Expected gradient calculation: The obtained prior baseline is used as the starting point of the path integral in the expected gradient method. Interpolation is performed between the prior baseline and the input image through uniform sampling. Each interpolation is input into the convolutional neural network (CNN). The gradient of a specific category is obtained through backpropagation. The gradient is then averaged. The gradient mean is multiplied by the input image and the prior baseline difference map through Riemann sum integral to obtain the feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map consistent with the resolution of the input image.

[0029] In the bird image classification task, a sparrow picture with a resolution of 224×224 is input to the pre-trained convolutional neural network ResNet-50 network. First, the network extracts features step by step through the convolution layer, and the last convolution layer outputs a 7×7×2048 feature tensor. After flattening the tensor into a 49×2048 feature matrix along the spatial dimension, the Bayesian Gaussian mixture model BGMM is used for multimodal distribution modeling. Through variational inference, three mixture components are automatically identified, corresponding to the feather texture, beak features and background area, that is, the prior probability distribution; then, the feature distribution mean output by BGMM is bilinearly upsampled to restore it to the original image. The image size is calculated to generate a spatially aligned prior probability mask, i.e., the prior distribution. By subtracting the prior distribution from the mean baseline sampled from the training dataset, i.e., the original baseline, we obtain a prior baseline that suppresses the characteristic response of birds. 80 interpolation points are uniformly sampled between the prior baseline and the input image. The gradient of the "sparrow" category is input into a CNN at each interpolation point. The mean gradient is multiplied by the difference map between the input image and the prior baseline through Riemann sum integral to obtain a feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map with a resolution of 224×224, showing that the model decision mainly depends on the sparrow's head outline, wing texture and tail shape, and the contribution of branches in the background.

[0030] 50-100 interpolation points are uniformly sampled on the path between the prior baseline and the input image to generate an intermediate input sequence. The convolutional neural network (CNN) outputs the gradient for each interpolation point, and the gradient mean is multiplied pixel by pixel by the difference between the prior baseline and the input image to obtain a feature importance score matrix. In the autonomous driving scenario, the input image is the image of the vehicle's brake light area. The importance score matrix is reorganized to the original image size, denoised by Gaussian filtering, and normalized to generate a high-resolution attribution heat map that intuitively displays the area of interest, including road signs or pedestrian outlines.

[0031] The technical solution of the present invention further comprises the following specific steps:

[0032] 1) Deep Feature Extraction: A convolutional neural network (CNN) is used to extract features from the input image, obtaining feature representations from the final convolutional layer. The output is a four-dimensional tensor of shape (B, C, H, W), where C represents the number of feature maps, and H and W correspond to the spatial dimensions. This tensor is flattened along the spatial dimensions (H, W) to produce a two-dimensional matrix of shape (N, C).

[0033] 2) Prior probability distribution calculation: After obtaining the flattened feature vector, the prior probability distribution is calculated to model the uncertainty inherent in feature attribution.

[0034] 3) Prior Baseline Calculation: The prior probability distribution calculated in the previous step is upsampled to match the spatial dimensions of the original input image. After the prior distribution is calculated, it is integrated into the expected gradient framework. The prior baseline is calculated by integrating the inverse of the prior distribution into the original expected gradient baseline. This optimization ensures that the baseline effectively models missing information, thus meeting the framework's interpretability goal.

[0035] 4) Expected Gradient Calculation: The calculated prior baseline is used as the starting point for the path integral in the expected gradient method. Feature importance is efficiently approximated based on the expected gradient, and the integral is computed using a Riemann sum. This method provides a method for approximating Shapley-valued feature importance. The expected gradient averages the gradients calculated at multiple reference points, providing a more robust feature attribution map.

[0036] As a further solution of the present invention, deep feature extraction specifically includes: selecting deep convolutional neural networks of different structures suitable for the target task (such as residual connection type ResNet, lightweight MobileNet or dense connection type DenseNet), performing standardization preprocessing on the input image, and then passing through the convolution layer, pooling layer and nonlinear activation layer in sequence through forward propagation, and finally extracting features from the last convolution layer. The output is a four-dimensional feature tensor with a shape of (B, C, H, W), where B is the batch size, C is the number of channels, and H and W are the spatial dimensions of the feature map. To facilitate subsequent probabilistic modeling, the feature tensor is flattened along the spatial dimension (H, W) and converted into a two-dimensional matrix (N=H×W) with a shape of (N, C), where each row corresponds to a feature vector at a certain spatial position in the original image. This step adapts to the feature extraction requirements of different network architectures to ensure the versatility and scalability of feature representation.

[0037] As a further solution of the present invention, the calculation of the prior probability distribution specifically includes: based on the flattened feature matrix, a probability model is used to model the distribution characteristics of the feature vector. According to the feature complexity, a multivariate Gaussian model (MGM) or a Bayesian Gaussian mixture model (BGMM) is selected: MGM calculates the mean and covariance through maximum likelihood estimation, which is applicable to a single modal distribution; BGMM determines the number and parameters of the mixed components through variational inference, which can capture multimodal distribution characteristics. The uncertainty of feature attribution is modeled by quantifying the correlation between features through the covariance matrix, or quantifying the degree of overlap of the subspace distribution through the mixed weights. For example, the feature subspace corresponding to the high-probability mixed component in the BGMM can represent a significant semantic area. This process provides a structured probability prior for baseline optimization and enhances the ability to characterize the state of missing features.

[0038] After obtaining the flattened feature vectors, a prior probability distribution is further calculated to model the inherent uncertainty in feature attribution. Subsequently, the prior distribution obtained from the probabilistic model is sampled to match the spatial dimensions of the original input image. After calculating the prior distribution, it is integrated into the expected gradient framework. By calculating the feature importance based on the expected gradient values, a feature attribution map is generated that intuitively demonstrates the model's attention areas in the input image. The prior distribution can be calculated using methods such as multivariate Gaussian models, Bayesian Gaussian mixture models, histograms, Dirichlet process mixture models, and maximum likelihood estimation. After obtaining the prior probability distribution, the expected gradient constructs a baseline based on prior information by randomly selecting multiple baseline images from the background dataset as the original baseline. By integrating the inverse of the prior distribution into the original expected gradient baseline, a baseline based on prior information is constructed, which better conforms to the definition of "missing." This optimization ensures that the baseline effectively models missing information. After the baseline is modeled, for the model input, the expected gradient is randomly determined on the straight path between the baseline image and the input image to calculate multiple intermediate interpolations. The obtained interpolations are then input into the convolutional neural network, and the gradient of the specific category is obtained through backpropagation. Finally, the gradient is averaged and multiplied by the difference between the pixel values of the input image and the baseline image as the approximate Shapley value.

[0039] As a further solution of the present invention, prior baseline modeling specifically includes: combining the prior distribution with the expected gradient framework to construct a reasonable baseline. First, the prior distribution output by the probability model is bilinearly upsampled and restored to the spatial size of the input image to generate a probability mask map that is aligned with the input image space. Subsequently, the original baseline (such as the zero vector or the mean of the background data set) is linearly combined with the opposite of the prior distribution to generate an optimized baseline vector that better fits the definition of "missing information". For example, the contribution of the prior mean is adjusted by a scaling factor to suppress the eigenvalues of known semantic areas. Finally, the rationality of the baseline is verified by calculating the difference map between the input and the baseline to ensure that the target object area in the difference map is effectively suppressed, thereby meeting the requirements of the interpretability framework for missing information simulation.

[0040] As a further solution of the present invention, the expected gradient calculation specifically includes: uniformly sampling multiple interpolation points along the path between the baseline vector and the input image to generate an intermediate input sequence. Gradients for a specific category are obtained through backpropagation, and then the gradients are averaged. The gradient mean is multiplied by the input-baseline difference value through Riemann sum approximate integral operations to approximate the Shapley value feature importance and obtain a feature importance score. This process reduces noise interference through random sampling and path averaging, ensuring the stability of the attribution results. Finally, a high-resolution feature attribution map is generated to intuitively display the key features on which the model decision depends.

[0041] Shapley value

[0042] The Shapley value originates from cooperative game theory and is used to fairly distribute the benefits of cooperation. In machine learning, it measures the contribution of each feature to model predictions. Calculating the Shapley value requires evaluating all possible feature combinations, which is computationally complex.

[0043] Expected Gradients

[0044] Expected Gradients, used to explain deep learning models, evaluates the impact of features on predictions by calculating the integral of the gradient of input features relative to a baseline.

[0045] Expected gradient is a method for calculating the contribution of input features to deep learning models based on the Shapley value concept. It can be considered an efficient approximation of the Shapley value, suitable for feature attribution analysis in deep learning models, avoiding the high computational cost of the Shapley value.

[0046] The Shapley value originates from game theory and assigns importance to features by calculating their marginal contributions across all possible subsets. It emphasizes fairness and theoretical completeness, but its computational complexity grows exponentially with the number of features, making it difficult to scale to high-dimensional scenarios. The expected gradient is based on the gradient integral. By calculating the gradient integral of the input feature relative to the baseline, it directly utilizes model gradient information without traversing feature subsets, resulting in significantly higher computational efficiency. The core difference between the two lies in their methodology: the Shapley value relies on combinatorial games, while the expected gradient relies on differential analysis.

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments and drawings in the present invention, all other embodiments and drawings obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Those skilled in the art should understand that the content described specifically below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0048] A specific embodiment of the present invention is as follows: In the bird image classification task, a sparrow picture with a resolution of 224×224 is input to the pre-trained ResNet-50 network. First, the network extracts features step by step through the convolutional layer, and outputs a 7×7×2048 feature tensor (B=1, C=2048, H=7, W=7) at the last convolutional layer. After the tensor is flattened into a 49×2048 feature matrix along the spatial dimension, the Bayesian Gaussian mixture model (BGMM) is used for multimodal distribution modeling, and three mixture components are automatically identified through variational inference, corresponding to feather texture, beak features and background area respectively. Subsequently, the mean of the feature distribution output by the BGMM is bilinearly upsampled to restore it to the original image size, and a spatially aligned prior probability mask is generated. By calculating the mean baseline (original baseline) sampled from the training dataset and the prior probability distribution, a prior baseline that suppresses the bird feature response is obtained (for example, the leaf texture in the background area is enhanced and the bird body outline is weakened). 80 interpolation points were uniformly sampled between the prior baseline (a green background with leaf texture) and the input image (containing a sparrow). The gradient of the network output, "sparrow," was calculated for each intermediate input. The mean gradient was multiplied by the input-baseline difference map using a Riemann sum integral to approximate the Shapley value feature importance, resulting in a feature importance score matrix. The score matrix was finally reconstructed into a 224×224 resolution feature attribution map, clearly showing that the model's decision primarily relies on the sparrow's head outline (contributing 35.2%), wing texture (28.7%), and tail morphology (19.5%), while background branches only contribute 2.1%. This effectively validates the model's decision logic based on bird morphology rather than background noise.

[0049] An embodiment of the present invention provides an expected gradient interpretability method based on deep priors. The specific implementation process includes: performing deep feature extraction on the input image through a convolutional neural network (CNN), calculating the prior probability distribution based on the prior features, sampling the prior distribution obtained from the probability model, calculating the prior distribution, integrating it into the expected gradient framework, and constructing a prior-based baseline by integrating the opposite of the prior distribution into the original baseline of the expected gradient. The feature importance is efficiently approximated based on the expected gradient, and the integral is calculated using the Riemann sum. The expected gradient is averaged by the gradients calculated at multiple reference points, thereby providing a more robust feature attribution map. Figure 1 A schematic diagram depicting the overall process of the method.

[0050] In some optional implementations of this embodiment, the specific implementation of the deep feature further includes: selecting a pre-trained deep convolutional neural network (such as ResNet, MobileNet or DenseNet) and performing standardized preprocessing on the input image (such as normalizing to the range of [0,1]). The feature tensor of the last convolutional layer is extracted by forward propagation, and its shape is (B, C, H, W), where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map. The feature tensor is flattened into a two-dimensional matrix (N=H×W) of shape (N, C) along the spatial dimension, and each row corresponds to a feature vector at a certain spatial position in the image. For example, a 224×224 RGB image is input, and the last convolutional layer of ResNet-50 outputs a 7×7×2048 feature tensor. After flattening, a 49-row 2048-dimensional feature matrix is obtained.

[0051] In some optional implementations of this embodiment, the specific implementation method of the prior probability distribution calculation further includes: based on the flattened feature matrix, selecting a probability model according to the complexity of the feature distribution. For unimodal distribution features (such as natural images with a single background), a multivariate Gaussian model (MGM) is used to calculate the mean vector and covariance matrix through maximum likelihood estimation; for multimodal distribution features (such as the coexistence of lesions and normal tissues in medical images), a Bayesian Gaussian mixture model (BGMM) is used to automatically determine the number of mixed components and their parameters (weights, mean, covariance) through variational inference. For example, in lung CT image analysis, BGMM can separate the feature subspaces of healthy tissue and lesion areas, and quantify the significance of different components through mixing weights.

[0052] In some optional implementations of this embodiment, the specific implementation method of the prior baseline calculation further includes: upsampling the prior probability distribution output by the probability model to the resolution of the input image through bilinear interpolation to generate a spatially aligned probability mask. The original baseline (such as the pixel mean of the background data set) is proportionally fused with the inverse of the prior mean vector to construct an optimized baseline. For example, in the analysis of diabetic retinopathy, the baseline is composed of the mean of the normal fundus image minus the prior mean of the lesion feature, so that the baseline can effectively suppress the characteristic response of the lesion area. The rationality of the baseline is verified by calculating the difference map between the input image and the baseline to ensure that the lesion area is significantly highlighted in the difference map, while the normal area is suppressed.

[0053] In some optional implementations of this embodiment, the specific implementation method of the expected gradient calculation further includes: uniformly sampling 50-100 interpolation points on the path between the baseline vector and the input image to generate an intermediate input sequence. The gradient of the model output is calculated for each interpolation point, and the gradient mean is multiplied pixel by pixel by the input-baseline difference value through the Riemann sum approximate integral operation to obtain the feature importance score. For example, in an autonomous driving scenario, path integration of the gradient of the vehicle detection model can stably identify the key contribution of the brake light area. Finally, the importance score is reorganized into the original image size, denoised and normalized by Gaussian filtering to generate a high-resolution attribution heat map, which intuitively displays the model's focus area (such as road signs or pedestrian outlines).

[0054] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A neural network interpretation method based on deep prior expected gradient, characterized by: First, the feature extraction capability of the deep convolutional neural network (CNN) is used to process the input image, and the feature representation is obtained from the final convolutional layer of the deep convolutional neural network (CNN). After obtaining the flattened feature vector, the probability distribution model is used to capture the distribution characteristics of the features, thereby obtaining the prior probability distribution. The prior probability distribution is bilinearly upsampled to match the spatial dimensions of the original input image, and the prior probability distribution that matches the spatial dimensions of the original input image is integrated into the expected gradient framework. By calculating the feature importance based on the expected gradient value, a feature attribution map that can intuitively display the focus area in the input image is obtained.

2. The neural network interpretation method based on deep prior expected gradient according to claim 1, characterized in that: The specific steps are as follows: (1) Deep feature extraction: The input image is processed using the feature extraction capability of the convolutional neural network (CNN) to obtain feature representations from the final convolutional layer. The output is a four-dimensional tensor of shape (B, C, H, W), where C represents the number of feature maps and H and W correspond to the spatial dimensions. The tensor is flattened along the spatial dimensions (H, W) to obtain a two-dimensional matrix of shape (N, C), where each row corresponds to a feature vector at a specific spatial position in the image. (2) Calculation of prior probability distribution: After obtaining the flattened feature vector, the probability distribution model used is the Multivariate Gaussian Model (MGM) or the Bayesian Gaussian Mixture Model (BGMM) to capture the distribution characteristics of the features and thus obtain the prior probability distribution; (3) Prior baseline calculation: The prior probability distribution obtained in the previous step is upsampled to match the spatial dimension of the original input image to obtain the prior distribution, and then the prior distribution is integrated into the expected gradient framework, that is, the prior baseline is obtained by integrating the inverse of the prior distribution into the original baseline of the expected gradient; (4) Expected gradient calculation: The obtained prior baseline is used as the starting point of the path integral in the expected gradient method. Interpolation is performed between the prior baseline and the input image through uniform sampling. Each interpolation is input into the convolutional neural network (CNN). The gradient of a specific category is obtained through backpropagation. The gradient is then averaged. The gradient mean is multiplied by the difference map between the input image and the prior baseline through Riemann sum integral to obtain the feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map consistent with the input image resolution.

3. The neural network interpretation method based on deep prior expected gradient according to claim 2, characterized in that: In the bird image classification task, a sparrow picture with a resolution of 224×224 is input to the pre-trained convolutional neural network ResNet-50 network. First, the network extracts features step by step through the convolution layer, and the last convolution layer outputs a 7×7×2048 feature tensor. After flattening the tensor into a 49×2048 feature matrix along the spatial dimension, the Bayesian Gaussian mixture model BGMM is used for multimodal distribution modeling. Through variational inference, three mixture components are automatically identified, corresponding to the feather texture, beak features and background area, that is, the prior probability distribution; then, the feature distribution mean output by BGMM is bilinearly upsampled to restore it to the original image. The image size is calculated to generate a spatially aligned prior probability mask, i.e., the prior distribution. By subtracting the prior distribution from the mean baseline sampled from the training dataset, i.e., the original baseline, we obtain a prior baseline that suppresses the characteristic response of birds. 80 interpolation points are uniformly sampled between the prior baseline and the input image. The gradient of the "sparrow" category is input into a CNN at each interpolation point. The mean gradient is multiplied by the difference map between the input image and the prior baseline through Riemann sum integral to obtain a feature importance score matrix. Finally, the score matrix is reconstructed into a feature attribution map with a resolution of 224×224, showing that the model decision mainly depends on the sparrow's head outline, wing texture and tail shape, and the contribution of branches in the background.

4. The neural network interpretation method based on deep prior expected gradient according to claim 2, characterized in that: 50-100 interpolation points are uniformly sampled on the path between the prior baseline and the input image to generate an intermediate input sequence. The convolutional neural network (CNN) outputs the gradient for each interpolation point, and the gradient mean is multiplied pixel by pixel by the difference between the prior baseline and the input image to obtain a feature importance score matrix. In the autonomous driving scenario, the input image is the image of the vehicle's brake light area. The importance score matrix is reorganized to the original image size, denoised by Gaussian filtering, and normalized to generate a high-resolution attribution heat map that intuitively displays the area of interest, including road signs or pedestrian outlines.

Images