Image super-resolution network interpretation method based on class activation graph

Abstract

The invention discloses an image super-resolution network interpretation method based on a class activation graph, and belongs to the technical field of pattern recognition and machine learning, and the method comprises the steps: inputting a low-resolution image into a super-resolution network, and generating a corresponding super-resolution image; calculating the edge loss between the to-be-explained local areas in the super-resolution image and the target high-resolution image, and calculating the gradient of the output layer of the super-resolution network; distributing a pixel-level weight for the feature map of the output layer according to the gradient, and generating an activation map of the output layer; based on the relationship between the adjacent feature maps, the activation maps are reversely propagated layer by layer until the input layer, so that the activation maps of all layers of the network are generated; and explaining the contribution of each pixel in the input image to the generation of the to-be-explained local region through the activation graph of the input layer. According to the method, different layers in the super-resolution network are subjected to visual analysis to generate an activation graph to reflect a region of interest in the network, so that the potential of the CAM in explaining the super-resolution network is released.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition and machine learning, and in particular to image super-resolution based on deep neural networks, and more specifically to an image super-resolution network interpretation method based on class activation maps. Background Art

[0002] Image super-resolution (SR) is the process of reconstructing a high-resolution (HR) image from a low-resolution (LR) image. In recent years, the development of deep learning has accelerated its widespread application in image super-resolution tasks. Although deep learning has brought image super-resolution to a new level, the mechanism of super-resolution networks remains a mystery, leading to the widespread belief that these networks are black-box models. Therefore, studying the working mechanism of super-resolution networks and elucidating the regions of interest at different layers is crucial to enhance our understanding of how these models perform super-resolution, and is of great research significance.

[0003] Although neural networks are often viewed as black boxes, some studies have attempted to analyze their behavior and reveal their operating mechanisms. The integrated gradient method provides a human-interpretable attribution representation by evaluating the contribution of input features to the output, which helps to understand how the model makes predictions. The integrated gradient method is currently widely used in many fields. The local attribution map (LAM) method aims to analyze super-resolution networks based on the integrated gradient method. It can identify the contributing pixels involved in the network's recovery of local blocks, thereby achieving attribution analysis of super-resolution networks.

[0004] Among the related technologies of computer vision, the class activation map (CAM) based method is a commonly used interpretation method. CAM-based methods generate interpretation results through a linear weighted combination of feature maps of a specific layer. Various variants of CAM have been widely used in computer vision tasks such as image classification, semantic segmentation, and object detection. The original CAM method is designed for classification networks and requires a specific model architecture, that is, a global average pooling (GAP) layer is set before the classification layer. Grad-CAM uses gradients to measure the contribution of feature maps and performs weighted summation, providing high flexibility without changing the model architecture. Layer-CAM and HiResCAM are similar to Grad-CAM, but they use the pixel values ​​of the gradient map as weights, which improves the effectiveness of shallow CAM and produces refined activation maps. In addition, some studies have tried to improve the interpretation results by increasing the resolution of activation maps. FG-CAM enhances the application of CAM in the shallow layers of the network by exploiting the relationship between adjacent feature maps and adopting progressive layer-by-layer calculation. The activation maps obtained by FG-CAM in the shallow layers have high resolution and less noise.

[0005] LAM has laid a solid foundation for the study of the interpretability of super-resolution networks and has achieved remarkable results in interpretation and visualization. However, the interpretation results obtained by LAM only reflect the contribution of the input pixels to the interpreted area, but cannot evaluate the role of each layer within the network. In addition, LAM requires manual determination of hyperparameters and baseline inputs during implementation, which introduces a certain degree of subjectivity.

[0006] Therefore, the existing methods have the following problems:

[0007] In the interpretation of super-resolution networks, usually only the contribution of input pixels to specific regions can be identified, and there is a lack of in-depth analysis of the role of each layer of the network, which is limited.

[0008] Among them, some methods require manual setting of hyperparameters and baseline inputs, which may bring a certain degree of subjectivity and affect the accuracy of the interpretation.

[0009] In addition, it is impossible to deeply explore the hierarchical relationship. For example, the LAM (local attribution map) method can provide certain visualization results, but it mainly focuses on the contribution of input pixels and cannot deeply reveal the role of different network layers in super-resolution tasks. Summary of the invention

[0010] In view of this, the present invention provides an image super-resolution network interpretation method based on class activation maps, which can at least partially solve the limitations, subjectivity, and inability to deeply explore hierarchical relationships in the interpretation of super-resolution networks in the above-mentioned prior art.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] The embodiment of the present invention provides an image super-resolution network interpretation method based on a class activation map, comprising the following steps:

[0013] S10, inputting the low-resolution image into a super-resolution network to generate a corresponding super-resolution image;

[0014] S20, calculating the edge loss between the super-resolution image and the local area to be interpreted in the target high-resolution image, and calculating the gradient of the output layer of the super-resolution network based on the edge loss;

[0015] S30, assigning pixel-level weights to the feature map of the output layer based on the gradient of the output layer, and generating an activation map of the output layer;

[0016] S40, based on the relationship between adjacent feature maps, back-propagating the activation map layer by layer until the input layer, thereby generating activation maps of each layer of the network;

[0017] S50, explaining the contribution of each pixel in the input image to the generation of a target patch in the super-resolution image through the activation map of the input layer; the target patch is a local area to be explained.

[0018] Furthermore, it also includes:

[0019] S60. Based on the interpretation results, determine the quantitative evaluation index of the interpretation method for the image super-resolution task for horizontal comparison among the interpretation methods.

[0020] Furthermore, the step S10 includes:

[0021] The low-resolution image I LR Input into the super-resolution network f to obtain the corresponding super-resolution image I SR ;

[0022] I SR =f(I LR ,θ) (1)

[0023] Where θ represents the network parameters of the super-resolution network f.

[0024] Furthermore, the step S20 includes:

[0025] Calculate the edge loss between the super-resolution image and the local area to be explained in the target high-resolution image edge :

[0026]

[0027] Where [w, W] and [h, H] are the pixel coordinate ranges of the area to be interpreted; E is the edge map obtained from the high-resolution image; Represents the pixel value of the super-resolution image at the (x, y) position; Represents the pixel value of the high-resolution image at position (x,y);

[0028] According to the edge loss edge Compute the gradient of the output layer of the super-resolution network

[0029]

[0030] In the formula, Represents the pixel value at the (x,y) position of the feature map of the network output layer l.

[0031] Furthermore, the step S30 includes:

[0032] set up is the weight, which is multiplied pixel by pixel with the feature map to obtain L' l ; along the channel dimension L' l Sum and apply the ReLU activation function to eliminate negative responses, resulting in the activation map L of the lth layer l ; The lth layer is the last convolutional layer of the network;

[0033] L' l =g l ⊙A l (3)

[0034]

[0035] Among them, g l represents the gradient pixel of the output layer l, A l represents the feature map of the output layer l, Indicates L' l The sum of the kth channel of .

[0036] Furthermore, the step S40 includes:

[0037] For the Pixel Shuffle layer in the super-resolution network, the relationship between adjacent feature maps is expressed as:

[0038] L' n = Downsample(L' n+1 ) (5)

[0039] Where n represents the nth layer of the network, and n is between 1 and (l-1). n+1Downsample to the corresponding scale, that is, get the previous layer L' n ;

[0040] For the convolutional layer in the super-resolution network, the relationship between adjacent feature maps is constructed as follows:

[0041]

[0042] Among them, i represents the i-th pixel in the n-th layer, j represents the j-th pixel in the n+1-th layer, represents the connection weight between i and j, and the importance of i and j are and Separating the positive and negative weights yields: Represents the pixel value of i in layer n, interval Represents pixel value The domain of

[0043] Substitute the calculation results of formula (5) and formula (7) into formula (4) to calculate the activation map of each layer in the network until the input layer of the network.

[0044] Furthermore, in step S50, the activation map of the input layer is used to represent the contribution of each pixel in the input image to the super-resolution result of the image region to be interpreted, as shown below:

[0045]

[0046] Among them, the pixel value in L1 represents the contribution of the pixel at the corresponding position in the input image to the area to be interpreted in the image.

[0047] Furthermore, the calculation process of the quantitative evaluation index in step S60 includes:

[0048] The activation map of the input layer is binarized according to the threshold thr, and the binarized image is used as the mask I mask , and the original input low-resolution image I LR Multiply to obtain the saliency map S;

[0049]

[0050] S=I mask ⊙I LR (11)

[0051] Represents the pixel value at position (x, y) in the input layer activation map; m xy I mask The pixel value at position (x,y) in the image;

[0052] Determine the average edge loss increase AELI as:

[0053]

[0054] Among them, o and s represent the original input image I LR and the edge loss of the area to be explained in the saliency map S; the smaller the value of AELI, the better the effect of the explanation result.

[0055] It can be seen from the above technical solution that compared with the prior art, it has the following advantages:

[0056] The image super-resolution network interpretation method based on class activation maps proposed in the present invention aims to unleash the potential of the CAM method in SR networks. By applying the CAM-based method to the classic super-resolution network and making targeted improvements, the present invention can generate activation maps that reveal the attention areas of each layer of the network, which helps to deeply understand the super-resolution network, and the interpretation results obtained by accessing the feature map of the network are more faithful. By using the present invention, not only can the attention areas at different levels be visualized, but also how the super-resolution network generates high-resolution images can be more clearly explained, making these "black box" networks more transparent and easy to understand, and providing a more solid foundation for future research. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A flow chart of the image super-resolution network interpretation method based on class activation maps provided by the present invention.

[0059] Figure 2 Schematic diagram of the image super-resolution network interpretation method based on class activation map provided by the present invention.

[0060] Figure 3 A schematic diagram of the visualization of the interpretation results of different remote sensing scenes provided by the present invention.

[0061] Figure 4 A schematic diagram for intuitive comparison of LAM and SRCAM provided by the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In computer vision related technologies, the class activation map (CAM) based method is a commonly used interpretation method. CAM based methods can effectively reflect the focus areas in different network layers and have been widely used in image classification, target detection, semantic segmentation and other fields. However, CAM has not yet been developed and applied in super-resolution networks.

[0064] Existing super-resolution network interpretation methods only reflect the contribution of input pixels to the output results, and it is difficult to reflect the focus areas of different layers within the network. To address this problem, the present invention proposes an interpretation method for super-resolution networks based on CAM, called SRCAM, which aims to visualize and analyze different layers in the super-resolution network and generate activation maps to reflect the focus areas within the network, thereby releasing the potential of CAM in interpreting super-resolution networks.

[0065] In addition, the present invention also proposes an evaluation index for super-resolution network interpretation methods, average edge loss increase (AELI). Experiments on complex remote sensing images show that SRCAM generates visually reasonable interpretation result maps and achieves excellent performance in quantitative evaluation indicators.

[0066] Specifically, refer to Figure 1 As shown, the embodiment of the present invention discloses an image super-resolution network interpretation method based on a class activation map, comprising the following steps:

[0067] S10, inputting the low-resolution image into a super-resolution network to generate a corresponding super-resolution image;

[0068] S20, calculating the edge loss between the super-resolution image and the local area to be interpreted in the target high-resolution image, and calculating the gradient of the output layer of the super-resolution network based on the edge loss;

[0069] S30, assigning pixel-level weights to the feature map of the output layer based on the gradient of the output layer, and generating an activation map of the output layer;

[0070] S40, based on the relationship between adjacent feature maps, back-propagating the activation map layer by layer until the input layer, thereby generating activation maps of each layer of the network;

[0071] S50, explaining the contribution of each pixel in the input image to the generation of a target patch in the super-resolution image through the activation map of the input layer; the target patch is a local area to be explained;

[0072] S60. Based on the interpretation results, determine the quantitative evaluation index of the interpretation method for the image super-resolution task for horizontal comparison among the interpretation methods.

[0073] The present invention designs an interpretation method SRCAM for image super-resolution networks based on the CAM method. First, a low-resolution image is first processed by a super-resolution network to obtain a corresponding super-resolution image. Subsequently, the edge loss between the local area to be interpreted in the super-resolution image and the high-resolution image is calculated. The gradient is obtained from the output layer of the network using the edge loss, and pixel-level weights are assigned to the feature map based on this gradient, thereby generating an activation map of the output layer. Finally, the activation maps of the first few layers are calculated step by step using the relationship between adjacent feature maps, and this process continues until the input layer is reached. The activation map of the input layer can be interpreted as the contribution of each pixel in the input image to the target patch. The present invention applies the CAM-based method to the classic super-resolution network and makes targeted improvements. SRCAM can generate an activation map that reveals the attention area of ​​each layer of the network, which helps to deeply understand the super-resolution network, and the interpretation results obtained by accessing the feature map of the network are more faithful.

[0074] The technical solution of the present invention is described in detail below:

[0075] 1. Overall framework of the method

[0076] The image super-resolution interpretation method based on class activation map consists of three steps: edge loss calculation, output layer activation map generation, and activation map generation of each layer of the network. Figure 2 As shown in Figure 1, SRCAM first calculates the edge loss between the SR and HR patches to be interpreted. Subsequently, the gradient obtained from the edge loss is used to generate the activation map of the output layer. Finally, this activation map is propagated back to the input layer layer by layer to obtain the activation maps of each layer in the network.

[0077] 2. Margin loss calculation

[0078] Assume that f represents the super-resolution network and θ represents the corresponding network parameters. For a given low-resolution image I LR , will I LR Input into the super-resolution network f, we can obtain the corresponding super-resolution image I SR .

[0079] I SR =f(I LR,θ) (1)

[0080] Unlike the classification network, we use the texture restoration effect in the local area of ​​the image to backpropagate and calculate the network gradient. Specifically, we calculate the edge loss between the local area of ​​the super-resolution image and the high-resolution image, and use this loss to replace the predicted score in the classification network. The definition of edge loss is as follows:

[0081]

[0082] Where [w,W] and [h,H] are the pixel coordinate ranges of the region to be interpreted. E is the edge map obtained from the high-resolution image. Compared with the feature detector in LAM, the edge loss considers both the existence of edge features and the pixel intensity at the edge position.

[0083] 3. Activation map generation of the output layer

[0084] The gradient obtained by the edge loss can be expressed as: set up is the weight, which is multiplied pixel by pixel with the feature map to obtain L' l Then, L' is l Sum and apply the ReLU activation function to eliminate negative responses, resulting in the activation map L of the lth layer l . Usually, the lth layer is the last layer.

[0085] L' l =g l ⊙A l (3)

[0086]

[0087] Among them, g l represents the gradient pixel of the output layer l, A l Represents the feature map of the output layer l, and represents L' l The sum of the kth channel of .

[0088] 4. Generation of activation maps for each layer in the network

[0089] After we get the activation map from the output layer, we gradually calculate the activation maps of the previous layers until we reach the input layer. The activation map of the input layer represents the contribution of each pixel in the input image to the region to be interpreted. In this step, we consider the convolutional layers and Pixel Shuffle layers commonly used in super-resolution networks.

[0090] For the Pixel Shuffle layer, we only need to replace L' n+1 Downsample to the corresponding scale, that is, get the previous layer L' n, as shown below:

[0091] L' n = Downsample(L' n+1 ) (5)

[0092] Among them, n represents the nth layer of the network, and n takes a value between 1 and (l-1). For the convolution layer, the above method is used to construct the relationship between adjacent feature maps, as shown below:

[0093]

[0094] Among them, i represents the i-th pixel in the n-th layer, j represents the j-th pixel in the n+1-th layer, represents the connection weight between i and j, and the importance of i and j are and Separating the positive and negative weights yields: Represents the pixel value of i in layer n, interval Represents pixel value Finally, we combine the calculation results with formula (4) to calculate the activation map of each layer in the network.

[0095] 5. Calculation of Average Margin Loss Growth (AELI)

[0096] In image classification tasks, Average Drop (AD) is often used as a quantitative evaluation metric to explain the results. And the input image I input Pixel-by-pixel multiplication is performed between c AD is obtained by measuring the original image I input The quality of the obtained activation map is evaluated by the difference in predicted scores between the activation map and the saliency map. As shown in the following equation:

[0097]

[0098] Among them, c is the prediction score of the original image being classified as category c, s c is the prediction score of the saliency map.

[0099] However, directly multiplying the activation map with the original image will cause the saliency map to darken overall, change the value of each pixel and destroy the correlation between pixels. This calculation method is not suitable for end-to-end image super-resolution tasks. Therefore, we modify the method of obtaining saliency maps and propose a new evaluation metric for super-resolution network interpretation methods based on edge loss, called Average Edge Loss Increase (AELI). Specifically, we binarize the activation map using a threshold thr to identify contributing pixels. Then, the binarized image is used as a mask and compared with the original input I LR Multiply them to get the saliency map.

[0100]

[0101] S=I mask ⊙I LR (11)

[0102] Represents the pixel value at position (x, y) in the input layer activation map; m xy I mask The pixel value at position (x,y) in the image;

[0103] Therefore, AELI can be defined as:

[0104]

[0105] Among them, o and s represent the original input image I LR and the edge loss of the area to be explained in the saliency map S; the smaller the value of AELI, the better the effect of the explanation result.

[0106] The present invention designs an image super-resolution network interpretation method based on class activation maps, which includes three parts: edge loss calculation, output layer activation map generation, and activation map generation of each layer of the network. Remote sensing images are widely used in various fields such as earth science, environmental monitoring, and land cover change detection. Compared with natural images, this puts higher requirements on the reliability of remote sensing images, and a deep understanding of the super-resolution process can help improve this reliability.

[0107] Therefore, in the experimental part, the present invention uses the publicly available Aerial Image Dataset (AID). AID is a large remote sensing image dataset constructed by collecting images from Google Earth, containing 10,000 images, covering 30 different remote sensing scene types including airports, bare land, baseball fields, beaches, bridges, etc. In the experiment, the AID dataset is divided into a training set and a test set at a ratio of 8:2, and the training set is used to train the super-resolution network. The images in the test set are used to interpret the super-resolution network and evaluate the interpretation results. Similar to the previous interpretation method, we also set the location of the local area to be interpreted at the center of the test image with a size of 100×100. The experiment focuses on the classic super-resolution network EDSR with a scale factor of 8, and visualizes the activation map to identify the attention area in different network layers. We use the activation map of the input layer to represent the contribution of each pixel in the input image to the area to be interpreted. The qualitative evaluation results show that the method of the present invention can obtain visually reasonable interpretation results, and the quantitative evaluation results further prove the superiority of the method of the present invention.

[0108] In the experiments, we compare the proposed method with other CAM-based interpretation methods, including Grad-CAM, FG-Grad-CAM, HiResCAM, and Layer-CAM. Grad-CAM and FG-Grad-CAM assign global weights to feature maps, while HiResCAM and Layer-CAM use pixel-level weights. In addition, we also compare the proposed method with LAM, an interpretation method designed for super-resolution networks.

[0109] Qualitative Evaluation: For qualitative evaluation, we visualize the activation maps generated by the explanation methods and display them on Figure 3 The upper part of the remote sensing image shows a bridge, and the lower part of the remote sensing image shows a farmland, showing the visualization of the interpretation results of different remote sensing scenes. The image below shows the AELI calculated from the activation map of the input layer when the threshold is set to 0.05.

[0110] The red box area in the input image corresponds to the patch being interpreted. For each input image, the visualization shows the activation maps obtained from the output layer, the hidden layer (taking the 10th residual block as an example), and the input layer in turn. LAM only focuses on capturing the contribution of the input image pixels to the target patch and does not reflect the information of the internal layers of the network. Therefore, we compare the results of LAM with the activation map of the input layer. Figure 3It can be intuitively observed that Grad-CAM and FG-Grad-CAM, which assign global weights to feature maps, focus on the overall features of the image, but cannot accurately locate the area to be explained. It is obvious that methods using global weights are not suitable for super-resolution networks. Methods using pixel-level weights (such as HiResCAM and Layer-CAM) can accurately locate the block being explained and highlight the main features within the block. However, both methods directly apply activation map calculations at each layer, which introduces significant noise when applied to shallow layers, resulting in fragmented explanation maps that cannot capture enough information to effectively reconstruct the target area. As a method specifically designed for super-resolution networks, the explanation map generated by LAM covers most of the key features within the target block. However, LAM only reflects the contribution of the input image to the target block, and hyperparameters and baseline inputs need to be manually determined during the calculation process. In addition, the calculation process needs to process all images on the path from the baseline input to the original image, which is inefficient.

[0111] In contrast, the SRCAM method proposed in this paper can effectively generate activation maps and reveal the focus areas of each layer within the SR network, and is significantly better than other CAM-based methods. The activation map of the SRCAM input layer effectively and comprehensively highlights the key features related to the area to be interpreted. Comparing the three activation maps of SRCAM, it can be found that by using edge loss to calculate the gradient, the output layer activation map of the network emphasizes the texture of the image. As the network layer becomes shallower, the attention of the activation map gradually expands and includes pixels near the edge. In addition, we also Figure 3 The AELI obtained by different methods is shown in Figure 2, with the threshold set to 0.05. From the AELI of different methods, it can be seen that the SRCAM method has a clear advantage, which indicates that the pixels highlighted by SRCAM contribute more to the recovery of the area to be interpreted.

[0112] To further compare the visual effects of LAM and SRCAM, we superimpose the activation map of the input layer onto the original image, which can more intuitively represent the contribution area emphasized by the explanation map. The results are shown in Figure 4 For the bridge image, the SRCAM activation map shows that EDSR not only focuses on the left and middle edges of the bridge, but also on the edge on the right, even though this edge is not part of the region to be interpreted. In contrast, LAM's interpretation completely ignores the right edge. For the farmland image, the SRCAM result additionally highlights the edges below the target block, while LAM does not take this into account, and these additional edges are considered beneficial in the image SR process. In addition, the AELI obtained by SRCAM is better than that of LAM, further confirming that the additionally highlighted edges in the SRCAM activation map are beneficial to recovering the target block.

[0113] Quantitative evaluation: Fidelity is an important feature for evaluating interpretation methods, which measures whether the interpretation results accurately highlight the areas related to the network output. It can be seen from the calculation process of AELI that it effectively measures the degree of correlation between the interpretation results and the area to be interpreted, thereby reflecting the fidelity of the interpretation method. Therefore, the present invention uses the proposed AELI for evaluation, and the quantitative results are shown in Table 1, which only compares the four interpretation methods that can locate the area to be interpreted. As shown in Table 1, we set the threshold to 0.05, 0.10, 0.15, and 0.20. As the threshold increases, the number of valid input pixels decreases, resulting in a corresponding decrease in the quality of the output image. By comparing different interpretation methods horizontally, it is clear that our SRCAM always achieves the lowest loss and AELI at each threshold. This shows that the contribution pixels obtained by SRCAM are more accurate and can better assist in restoring the target patches, thereby obtaining a more faithful interpretation result.

[0114] Table 1

[0115]

[0116] Margin loss and AELI obtained by different methods at different thresholds. The smaller the value, the better the explanation effect.

[0117] The present invention can generate visually plausible explanation result maps, clearly reveal the regions of interest at different layers within the network, and use the average edge loss increase (AELI) metric to evaluate the fidelity of the explanation results. Experimental results show that the SRCAM method outperforms other CAM-based explanation methods and can more accurately highlight the pixels that contribute to the restoration of the target area.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image super-resolution network interpretation based on class activation maps, characterized in that: The following steps are involved: S10, inputting the low-resolution image into a super-resolution network to generate a corresponding super-resolution image; S20, calculating the edge loss between the super-resolution image and the local area to be interpreted in the target high-resolution image, and calculating the gradient of the output layer of the super-resolution network based on the edge loss; S30, assigning pixel-level weights to the feature map of the output layer based on the gradient of the output layer, and generating an activation map of the output layer; S40, based on the relationship between adjacent feature maps, back-propagating the activation map layer by layer until the input layer, thereby generating activation maps of each layer of the network; S50, explaining the contribution of each pixel in the input image to the generation of a target patch in the super-resolution image through the activation map of the input layer; the target patch is a local area to be explained.

2. The image super-resolution network interpretation method based on class activation map according to claim 1, characterized in that: Also includes: S60. Based on the interpretation results, determine the quantitative evaluation index of the interpretation method for the image super-resolution task for horizontal comparison among the interpretation methods.

3. The image super-resolution network interpretation method based on class activation map according to claim 1, characterized in that: The step S10 comprises: The low-resolution image I LR Input into the super-resolution network f to obtain the corresponding super-resolution image I SR ; I SR =f(I LR ,θ) (1) Where θ represents the network parameters of the super-resolution network f.

4. The image super-resolution network interpretation method based on class activation map according to claim 3 is characterized in that: The step S20 comprises: Calculate the edge loss between the super-resolution image and the local area to be explained in the target high-resolution image edge : Where [w, W] and [h, H] are the pixel coordinate ranges of the area to be interpreted; E is the edge map obtained from the high-resolution image; Represents the pixel value of the super-resolution image at the (x, y) position; Represents the pixel value of the high-resolution image at position (x,y); According to the edge loss edge Calculate the gradient of the output layer of the super-resolution network In the formula, Represents the pixel value at the (x,y) position of the feature map of the network output layer l.

5. The image super-resolution network interpretation method based on class activation map according to claim 4 is characterized in that: The step S30 includes: set up is the weight, which is multiplied pixel by pixel with the feature map to obtain L' l ; along the channel dimension L' l Sum and apply the ReLU activation function to eliminate negative responses, resulting in the activation map L of the lth layer l ; The lth layer is the last convolutional layer of the network; L' l =g l ⊙A l (3) Among them, g l represents the gradient pixel of the output layer l, A l represents the feature map of the output layer l, Indicates L' l The sum of the kth channel of .

6. The image super-resolution network interpretation method based on class activation map according to claim 5, characterized in that: The step S40 includes: For the Pixel Shuffle layer in the super-resolution network, the relationship between adjacent feature maps is expressed as: THE' n =Downsample(The n+1 ) (5) Where n represents the nth layer of the network, and n is between 1 and (l-1). n+1 Downsample to the corresponding scale, that is, get the previous layer L' n ; For the convolutional layer in the super-resolution network, the relationship between adjacent feature maps is constructed as follows: Among them, i represents the i-th pixel in the n-th layer, j represents the j-th pixel in the n+1-th layer, represents the connection weight between i and j, and the importance of i and j are and Separating the positive and negative weights yields: Represents the pixel value of i in layer n, interval Represents pixel value The domain of Substitute the calculation results of formula (5) and formula (7) into formula (4) to calculate the activation map of each layer in the network until the input layer of the network.

7. The image super-resolution network interpretation method based on class activation map according to claim 6, characterized in that: In step S50, the activation map of the input layer is used to represent the contribution of each pixel in the input image to the super-resolution result of the image region to be interpreted, as shown below: Among them, the pixel value in L1 represents the contribution of the pixel at the corresponding position in the input image to the area to be interpreted in the image.

8. The image super-resolution network interpretation method based on class activation map according to claim 2, characterized in that: The calculation process of the quantitative evaluation index in step S60 includes: The activation map of the input layer is binarized according to the threshold thr, and the binarized image is used as the mask I mask , and the original input low-resolution image I LR Multiply to obtain the saliency map S; S=I mask ⊙I LR (11) Represents the pixel value at position (x, y) in the input layer activation map; m xy Indicates I mask The pixel value at position (x,y) in the image; Determine the average edge loss increase AELI as: Among them, o and s represent the original input image I LR and the edge loss of the area to be explained in the saliency map S; the smaller the value of AELI, the better the effect of the explanation result.

Images