Deep learning network visualization method for remote sensing image road extraction

Abstract

The invention discloses a deep learning network visualization method for remote sensing image road extraction, and relates to the field of artificial intelligence deep learning visualization. A remote sensing image road extraction model pre-trained based on a PyTorch deep learning framework can be used for single-sample area-level interpretation result analysis and node-by-node reverse contribution degree propagation to realize contribution degree interpretation of a road segmentation result, and finally, the contribution degree of each node to an extraction result is displayed through a visualization algorithm. A method is provided for optimization analysis of a road extraction network by utilizing a final visual contribution degree thermodynamic diagram, and attribution explanation is carried out on a road extraction process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence deep learning visualization, and specifically, to a deep learning network visualization method for remote sensing image road extraction. Background Art

[0002] Road information, as a typical application target of remote sensing image interpretation, has important application values in fields such as urban planning, intelligent transportation, and emergency disaster relief.

[0003] In recent years, artificial intelligence technology based on deep learning has become the mainstream technical solution in the field of road extraction due to its powerful feature learning ability. However, there are still technical bottlenecks in the process of its industrial application. On the one hand, existing deep learning models generally have the "black box" characteristic, and their decision-making process lacks transparency. Especially in high-precision road extraction tasks, the model is difficult to provide credible feature bases, resulting in difficulty in passing technical verification in application scenarios with high requirements for result credibility. On the other hand, due to the lack of effective model interpretation methods, the correlation between the feature extraction mechanism and the morphological characteristics of roads is not clear, the error propagation path is difficult to trace, and the adjustment of the network structure mainly relies on empirical attempts, thus leading to a long R & D cycle and large resource consumption. Currently, in the field of remote sensing image road extraction, there is an urgent need to establish a visualization interpretation method that matches the deep learning model, and this technology should be able to reveal the internal relationship between network feature learning and road spatial distribution, and quantitatively evaluate the contribution degree of each network layer to the extraction result. Summary of the Invention

[0004] Aiming at the problems of the existing technology, the present invention provides a deep learning network visualization method for remote sensing image road extraction. By graphically attributing and explaining the process of the network extracting roads, it helps to cope with the "black box" characteristic of the deep learning model and improve the optimization efficiency of the deep learning model.

[0005] The specific solution proposed by the present invention is as follows:

[0006] Pre-train a remote sensing image road extraction model using the PyTorch deep learning framework. The remote sensing image road extraction model is a deep learning semantic segmentation network for realizing end-to-end pixel-level road segmentation; the network inputs a remote sensing image map with three RGB channels, and the network outputs a binary road extraction result.

[0007] Select the remote sensing image road samples to be explained and the rectangular areas to be explained. The process of selecting the image area: intercept a rectangular area of N×M pixels in the road sample image, where N and M are required to be no greater than the height and width of the input remote sensing road sample image respectively.

[0008] Use the contribution degree interpretation algorithm based on network topology to calculate the contribution degrees of each node. The implementation process of the contribution degree interpretation algorithm based on network topology includes:

[0009] Construct a directed acyclic topology graph corresponding to the road extraction network structure, where nodes represent operation units in the network, edges represent the flow paths of feature data, and the topology graph has a single input node (corresponding to the input of the original remote sensing image) and a single output node (corresponding to the output of the semantic segmentation result);

[0010] For the selected road sample images, use network topology for the forward propagation of the network model to obtain the semantic segmentation result of the network model, and save the parameters required for contribution degree calculation during the forward propagation process;

[0011] Convert the segmentation result into the contribution degree matrix of the output node according to the selected image area;

[0012] According to the constructed network topology, decompose the contribution degree to the output node node by node in reverse order;

[0013] Enforce two major principles during the propagation process. One is that the distribution of the contribution degree should be closely based on the activation value of the current node, and the other is to ensure that the total amount of the contribution degree remains constant during its propagation process. Different types of nodes have different contribution degree propagation methods.

[0014] Use a visualization algorithm to visualize the contribution degrees of each node to the final extraction result. The visualization algorithm process includes:

[0015] Divide the visualization algorithm into two modes: the contribution degree visualization of each channel of a single node and the contribution degree visualization of channel fusion;

[0016] For the contribution degree visualization of each channel of a single node, input the contribution degree matrix of each channel of the target network node, perform normalization processing on the contribution degree matrix of each channel, convert the normalized matrix into an 8-bit grayscale heat map, and display the heat maps of each channel in the channel order;

[0017] For the contribution degree visualization of channel fusion, input the set of contribution degree matrices of all channels of the target network node, and fuse the contribution degree heat maps of each channel of a single node through the principal component analysis (PCA) algorithm.

[0018] The beneficial effects of the present invention are:

[0019] The present invention provides a visualization method for a road extraction network of remote sensing images, which improves the interpretability of the model without affecting the accuracy of the original training model. In terms of visualization granularity, in addition to various network layers, visualizations of different data processing nodes are also provided. Network layers include but are not limited to, for example, convolutional layers, pooling layers, batch processing layers, activation function layers, etc.; data processing nodes such as addition nodes, splicing nodes, etc. In terms of operation, the interpretation area is determined according to the extraction results of road samples, and a contribution explanation algorithm and a visualization algorithm are encapsulated, providing an intuitive and effective visualization method. In terms of the basis for interpretation, the finally visualized contribution heat map directly reflects the correlation degree of each node to the output extraction result, providing an effective analysis tool for both network structure and model performance evaluation and optimization decisions. Description of the Drawings

[0020] Figure 1 is a schematic flowchart of the method of the present invention;

[0021] Figure 2 is a schematic flowchart of the contribution explanation algorithm of the method of the present invention;

[0022] Figure 3 is a specific example diagram of the fusion visualization algorithm of the method of the present invention. (a) is a single-channel contribution heat map, and (b) is a channel fusion contribution heat map;

[0023] Figure 4 is a specific sample and region example diagram of the method of the present invention. (a) is the original sample image, (b) is the road extraction result image of the remote sensing image road extraction model, and (c) is the selected region mask;

[0024] Figure 5 is a single-channel visualization result diagram of a single node of a specific sample of the method of the present invention;

[0025] Figure 6 is a channel fusion visualization result diagram of different nodes of a specific sample of the method of the present invention. Detailed Embodiments

[0026] PyTorch is an open-source Python machine learning library based on Torch and is used for applications such as natural language processing. PyTorch can be regarded as numpy with GPU support, and at the same time, it can also be regarded as a powerful deep neural network with automatic differentiation capabilities.

[0027] The basic idea of remote sensing image road extraction is to identify and extract the image parts that are consistent with the road attributes based on the feature information of various objects in the remote sensing image. The remote sensing image road extraction network is a deep learning network that implements end-to-end road extraction. Its task belongs to semantic segmentation in deep learning image processing. Common network modules include convolutional layers, pooling layers, spatial pyramid pooling, attention modules, etc.

[0028] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0029] The present invention provides a deep learning network visualization method for remote sensing image road extraction, and the method flow chart is as follows: Figure 1 As shown:

[0030] Use PyTorch to train the remote sensing image road extraction model, which saves the trained model weights.

[0031] Load the weight file into the corresponding remote sensing image road extraction network and turn on the inference mode to obtain an inferable deep learning model.

[0032] The present invention requires the selection of specific samples and regions, such as Figure 4 As shown, the boxed area contains the area that the model recognizes as a road. Figure 4 (a) is the original image of the selected sample. Figure 4 (b) is the selected road area.

[0033] Based on the selected samples and areas, Figure 2 The contribution explanation flowchart in is used to explain the pre-trained model.

[0034] To obtain a reasonable deep learning model, it is necessary to build the network topology of the corresponding model through PyTorch based on the pre-trained remote sensing image road extraction model, where the nodes of the network topology represent the operation units in the network, the edges represent the flow path of the feature data, and the topology graph has a single input node and a single output node.

[0035] After obtaining the network topology of the model, the forward reasoning of the model is performed according to the selected samples to be explained, and the calculation is performed node by node from the image input node to the output node, and finally the extraction result of the model is obtained.

[0036] During the reasoning process, specific parameter information needs to be saved according to the different node types. For example, the convolution layer needs to save the weight, step size and other information of the convolution kernel, the maximum pooling layer needs to save the index taken by the pooling, and other nodes also need to save similar reasoning information.

[0037] After obtaining the extraction results of the model for the selected samples, according to the selected interpretation region, calculate the binary mask for the corresponding region, and then shield the contribution interpretation propagation of the non-selected regions.

[0038] Based on the obtained binary mask and the model extraction results, calculate the contribution matrix of the output nodes. The calculation of the contribution matrix of the output nodes is as follows:

[0039] R y = Y ⊙ M y

[0040] where R y is the contribution matrix of the output nodes, Y is the extraction result of the network model, and M y is the binary mask, and ⊙ is element-wise multiplication.

[0041] After obtaining the contribution matrix of the output nodes, it is necessary to perform backward contribution propagation for each node to obtain the contribution matrix of each node. For the contribution propagation of each node, the contribution will be distributed according to the activation contribution value of each node, and at the same time, the conservation in the contribution propagation needs to be ensured. First, it is necessary to calculate the activation value of each successor node, which is calculated according to the following formula:

[0042]

[0043] where z K is the activation value of the successor node K, a J-＞K is the input activation value from node J to node K, and ρ(a J-＞K ; w JK ) represents the weighted function mapping from node J to node K, and M is the number of K's predecessor nodes.

[0044] Next, calculate the normalized score of node K, and the calculation formula is as follows:

[0045]

[0046] where s K is the normalized score of node K, R K is the contribution matrix of node K, represents element-wise division.

[0047] Then, use the weight mapping of node K to node J to calculate the contribution score of node K to node J backward, and the calculation formula is as follows:

[0048] R K-＞J = a J-＞K ⊙ ρ(s K ; w KJ )

[0049] Finally, calculate the contribution score R of node JJ , the contribution input of each node is the sum of the contributions of its successor nodes, and the calculation formula is as follows:

[0050]

[0051] where R J is the contribution matrix of node J, and N is the number of successor nodes of J.

[0052] By calculating the network topology between nodes in the graph, the relevance can be assigned node by node, so as to decompose the decision-making process of the final output. For different types of nodes, the weight mapping rules of node K to node J in backpropagation are also different. The following gives the relevance propagation mappings of several typical nodes:

[0053] For nodes in linear layers such as convolution and deconvolution, their backward weights are the same as the forward weights, but the propagation method uses the opposite mapping method. For example, the backward relevance propagation of the convolution layer is a deconvolution mapping with the same convolution kernel weights, the same stride, and the same padding size. Similarly, deconvolution is a convolution mapping with the same parameters.

[0054] For nodes in non-linear layers such as the pooling layer, their backward mapping will propagate according to the forward mapping method. For example, max pooling will accumulate the contributions backward according to the index position where the maximum value is obtained forward, and average pooling will distribute the contributions evenly according to the range of the window taken.

[0055] For functional nodes such as activation functions and normalization layers, the backward contribution is distributed according to the purpose of the function. For example, since the forward processes of ReLU nodes and BatchNorm nodes do not change the distribution of activation values, and the output parts are all provided by the same original positions, such nodes will directly propagate the contributions to the previous node.

[0056] For nodes for data processing, such as addition nodes, concatenation nodes, padding nodes, etc., the backward contribution will be distributed according to the data processing method. For example, the addition node distributes the contribution of the node to each precursor node according to the square of the activation value of the precursor node, and the concatenation node distributes according to the dimension position occupied by each precursor node in the concatenated feature map.

[0057] The above classification includes all types of nodes in network propagation. Starting from the contribution matrix of the output node, the contributions are assigned node by node according to the reverse topological order of the computational graph, and the contribution matrices of all nodes are gradually obtained, thus decomposing the contribution degree to the output.

[0058] After obtaining the contribution matrix of each node, the contribution matrix of each node is converted into a displayable heat map through a visualization algorithm.

[0059] One of the visualization methods of the present invention is the single-channel heat map display. As shown in Figure 3 (a), it focuses on capturing the relevant features of each channel, and its conversion calculation is as follows:

[0060]

[0061] where S c ' is the normalized pixel value for heat map display, min(S c ) and max(S c ) respectively represent the minimum and maximum values in S c .

[0062] In addition to the single-channel display method, the present invention also provides a display method of channel fusion on this basis. As shown in Figure 3 (b), the fusion calculation formula is as follows:

[0063]

[0064] where S i (x, y) is the pixel value at the corresponding position after fusion. PCA is the principal component analysis algorithm. By selecting the pixel value of the most significant channel among the pixel positions corresponding to all channels, it represents the main features in the image.

[0065] Finally, a single-channel heat map and a channel-fused heat map are formed, providing an analysis tool for network optimization and model decision-making for users. The specific visualization results are as shown in Figure 5 and Figure 6 .

[0066] The above-described embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A visualization method for a deep learning network for road extraction from remote sensing images, characterized in that, Comprising: A pre-trained remote sensing image road extraction network; Selecting a road sample image and an image region to be explained; A contribution explanation algorithm based on network topology to explain the pre-trained road extraction model, and then obtaining a contribution matrix of each node in the model; Visualizing the contribution matrix of each node as a contribution heatmap through a visualization algorithm.

2. The remote sensing image road extraction network according to claim 1, characterized in that: The remote sensing image road extraction network is a deep learning semantic segmentation network for realizing end-to-end pixel-level road segmentation; Its model pre-training adopts the PyTorch deep learning framework; The network input is a remote sensing image map with three channels of red, green, and blue (RGB), and the network output is a binary road extraction result.

3. The image region according to claim 1, characterized in that, The image region is a rectangular region of N×M pixels intercepted from the road sample image, and N and M are required to be not greater than the height and width of the input remote sensing road sample image respectively.

4. The contribution explanation algorithm based on network topology according to claim 1, characterized in that: Constructing a network topology corresponding to the structure of the remote sensing image road extraction network, which is a directed acyclic graph, whose nodes represent operation operations, and the edges represent the paths of data flow. The starting point is a single input image node, and the ending point is a single semantic segmentation output node; According to the selected road sample image, using the network topology to perform forward inference of the network model, obtaining the semantic segmentation result of the network model, and saving the parameters required for contribution calculation during the forward inference process; Converting the segmentation result into a contribution matrix of the output node according to the selected image region; According to the constructed network topology, decomposing the contribution to the output node node by node in reverse order; The contribution explanation algorithm follows two major principles. One is that the distribution of contributions should be closely based on the activation value of the current node, and the other is to ensure that the total amount of contributions remains constant during its propagation. Different types of nodes have different contribution propagation methods.

5. The visualization algorithm according to claim 1, characterized in that: It can realize two modes of visualization, including: visualization of the contribution of each channel of a single node and visualization of the contribution of channel fusion; For the visualization of the contribution of each channel of a single node, by normalizing the contribution to the 8-bit gray scale interval, and then displaying the contribution heatmap of each channel; For the visualization of the contribution of channel fusion, through the principal component analysis (PCA) algorithm, fusing the contribution heatmaps of each channel of a single node, and then displaying the contribution heatmap at the node level.

Images