A road extraction method for remote sensing images based on graph convolution

Abstract

The present invention discloses a method for extracting roads from remote sensing images based on graph convolution. Graph convolution can aggregate feature information between adjacent nodes, extract features within a larger neighborhood of nodes, and effectively solve the problem of local position information loss. The designed method can be regarded as multi-task learning. First, CNN is used to realize feature extraction of remote sensing images. Secondly, a graph structure model is constructed based on the road features extracted by CNN, which is mainly composed of nodes and corresponding edge relationships. The road feature information extracted by the CNN branch is regarded as a node, and the difference between nodes is regarded as an edge. The local position information is obtained by obtaining the relationship between nodes. The present invention solves the problem of local road position information loss caused by the generalization effect of convolutional neural networks by using graph convolution, thereby effectively improving the accuracy of road segmentation.

Description

Technical Field

[0001] The present invention relates to the field of road extraction based on remote sensing images, and specifically to a method for automatically extracting roads from remote sensing images based on a graph convolution model. Background Art

[0002] As a kind of infrastructure, roads play an important role in geography, economy, military and other fields. Using high-resolution remote sensing images to extract key roads and obtain valuable geographic information that is difficult for people to obtain can provide a basis for subsequent vehicle navigation, urban planning, road network updates or support for disaster relief tasks, which is of great significance to promoting urban development.

[0003] Research on road extraction based on remote sensing images has a history of several decades. However, due to the special structure of the road itself, which is different from ordinary segmentation targets, and the complex background of remote sensing images, it is easy to be affected by other background factors during road extraction. In particular, with the improvement of aerial image resolution in recent years, some buildings have spectral values ​​similar to roads, which are easily mistaken for roads during road extraction. It is still difficult to extract roads based on high-resolution remote sensing images. At present, the methods for road extraction are mainly divided into traditional methods and deep learning methods. Traditional road extraction methods mainly include spectral analysis, edge detection, threshold segmentation, etc. For example, Shi et al. used adaptive neighborhood to classify spectral feature space to distinguish between roads and non-roads, but this method requires training SVM for each input image and is not suitable for complex prototype intersections. Huilin et al. proposed to segment the image into binary images of only the target and the background based on grayscale features, but the speed is not fast enough and it is difficult to distinguish irrelevant features. Gaetano et al. first proposed to use the Canny operator for edge extraction, and then post-processed it through graph cut theory to obtain the final road map. These traditional methods basically extract road features in an unsupervised manner, have a strong dependence on model parameters, most of them require a certain amount of manual interaction, have a low degree of automation, and cannot achieve ideal generalization effects for different remote sensing image data sets.

[0004] With the vigorous development of artificial intelligence, deep learning methods have become a trend and have made breakthrough progress in the field of image analysis. Convolutional neural networks can not only be used to deal with target detection and image classification problems, but also can be used for fine reasoning such as semantic segmentation. More and more studies have begun to use computer vision-related algorithms to achieve automatic road extraction. Wei et al. added road structure information to the loss function based on the FCN network architecture to improve the road segmentation results. Kestur et al. proposed the UFCN (U-type FCN) model to achieve road extraction from low-altitude remote sensing images of drones. The model consists of a set of convolution stacks and corresponding mirror deconvolution stacks, and uses jump links to save local feature information. Zhang et al. proposed a road extraction network that combines the advantages of ResNet and U-Net. The network consists of residual units, which simplifies the training of deep networks. Filin et al. proposed to use deep neural networks for road extraction, obtain predicted mask images and vectors of individual roads, and use the vectors for post-processing. MniCoswtea et al. proposed a three-step method to extract roads from remote sensing images. The first step is to use a combination of multiple UNet networks to extract the initial road results. Secondly, based on the fusion of the road map in the first stage, the optimization algorithm generates road vectors of corresponding thickness. Finally, the missing connections are added based on the inference map to improve the accuracy of road extraction. The above road extraction methods are generally based on simple fully convolutional neural networks or a combination of multiple networks. To a certain extent, the pixel-level prediction accuracy is guaranteed and the accuracy of road extraction is improved. However, the iteration of ordinary convolutional layers and pooling layers is prone to local information loss, which ultimately affects the road segmentation results.

[0005] Therefore, based on the shortcomings of the current road segmentation network, the present invention introduces the GCN model to capture local location information. First, the CNN branch is used to extract road feature information, and the adjacency matrix is ​​initialized by the Gaussian kernel function according to the feature information extracted by CNN. Through two layers of graph convolution, the management between nodes is finally obtained to obtain local location information, thereby effectively improving the accuracy of road segmentation. Summary of the invention

[0006] The purpose of the present invention is to provide a high-accuracy road extraction method based on a graph convolution model to achieve automatic extraction of roads in remote sensing images, so as to solve the problems mentioned in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a road extraction method based on a graph convolution model, comprising the following steps:

[0008] Step S1: Design a feature extraction branch based on the CNN model to realize feature extraction of remote sensing images;

[0009] Step S2: Design a local information capture branch based on the GCN model, and generate corresponding graph nodes according to the feature information extracted by the CNN model;

[0010] Step S3: Use the designed road extraction network based on the graph convolution model to perform overall design, thereby realizing road extraction from high-resolution remote sensing images and obtaining results.

[0011] Figure 1 This is the overall architecture diagram of the present invention. According to the characteristics of roads in remote sensing images, most of the current road extraction methods are based on the stacking of convolutional layers and pooling layers to obtain feature information, thereby increasing the receptive field and achieving high-resolution remote sensing image extraction. However, this method will cause a large amount of local position information to be lost, resulting in problems such as road discontinuity and discontinuity. Therefore, the present invention proposes to use a GCN model to realize remote sensing image road extraction, which can effectively obtain local position information. The overall framework of the present invention adopts an encoder-decoder structure, in which the encoder stage is divided into a road extraction branch based on the CNN model and a feature extraction branch based on GCN. First, a CNN-based feature extraction branch is designed according to the road characteristics, and the ResNet34 pre-trained on the ImageNet dataset is used as the encoder. The last fully connected layer is removed, and the improved ResNet34 is used to extract the features of relevant road information. Then, the obtained relevant road feature information is used as the input of the GCN model. First, a relevant graph structure is constructed according to the features, and the feature information is regarded as a node, and the difference between the nodes is regarded as an edge. The designed graph convolution model is used to obtain the relationship between the nodes in the graph structure to obtain the local position information. The entire network adopts a multi-task joint learning method to realize the road information extraction in high-resolution remote sensing images, thereby effectively improving the road segmentation accuracy.

[0012] As a further embodiment of the present invention:

[0013] The step S1 is to implement the design based on the CNN feature extraction branch, which mainly adopts the ResNet34 network structure as the encoder of the road extraction network. The main improvement is to remove the fully connected layer. Since ResNet34 is mainly implemented by the bottleneck layer, the main network structure is as follows Figure 2 As shown in the figure, two convolutional layers are used to reduce and increase the feature dimension respectively. The main purpose is to reduce the number of parameters and increase the number of network layers. The module adopts a skip connection method to achieve the fusion of shallow information and deep features, thereby effectively solving the gradient disappearance problem in the training process.

[0014] The improved ResNet34 is used as the feature extraction branch of the designed network, and the original image is used as the input of this branch to extract the relevant road feature information in the remote sensing image. The main expression of this branch can be shown as formula (1):

[0015] Y=g(x) (1)

[0016] Where X is the input original image, function ɡ represents a series of convolutional pooling operations, and after the CNN feature extraction function, the road feature information Y is finally obtained.

[0017] As a further embodiment of the present invention:

[0018] Due to the special road structure, which is usually long and thin and occupies the entire image information, it is particularly important to obtain the global context information of the image. However, many methods currently obtain context information by adding attention mechanisms or repeatedly stacking convolutional layers. However, this method is prone to local information loss problems, resulting in road information loss. Step S2 of the present invention implements the design of the GCN local feature capture branch.

[0019] As an emerging network architecture, GCN can operate on any non-Euclidean structured data compared to traditional CNN, and can effectively process graph structured data by modeling the relationship between samples. The working principle of GCN is to aggregate the features between adjacent nodes based on the adjacency matrix, thereby expanding the receptive field and solving the problem of local information loss. It can more accurately capture road-related information in remote sensing images. Since the use of GCN avoids a large amount of local information loss, more road information can be obtained in the road extraction task, thereby significantly improving the road extraction results. This topic proposes to use GCN to capture local information and regard the road extraction problem as a graph node classification problem.

[0020] In order to capture the relationship between the features extracted by CNN, first, we establish a graph structure based on the road feature information extracted by the CNN branch. The graph structure consists of nodes and edges. The defined graph structure is represented by G = (V, E), where V represents the set of nodes, E represents the set of edges of the graph, the node V represents the feature information extracted by CNN, and the edge E represents the difference between the nodes. The adjacency matrix A of the graph is used to represent the difference between each node in the graph structure and other nodes.

[0021] We use a Gaussian kernel function to initialize the adjacency matrix of the graph structure, and then use message propagation to exchange information between nodes through several layers of graph convolution, so that features can be extracted in a larger neighborhood and the loss of local information can be avoided.

[0022] Next, the specific graph convolution process is introduced in detail. The remote sensing image is input into the CNN feature extraction branch. After several layers of convolution operations, the important feature information in the image is obtained and used as the node feature input of GCN. In the GCN-based branch, the adjacency matrix A and feature X generated according to the features obtained by CNN are used as input. The specific graph convolution operation is as follows.

[0023]

[0024] Where L is the Laplacian matrix of the graph, A is the relationship matrix pre-generated based on feature information, D is the degree matrix of the vertex, and the normalized Laplacian matrix is The final transformation yields equation (2).

[0025] Transforming the two parameters θ0θ1 into a single parameter θ, we obtain the following convolution formula (3).

[0026]

[0027] because The eigenvalue range is [0, 2], which will lead to a series of problems such as numerical instability and gradient explosion or disappearance. Therefore, the normalization technique shown in (4) is used, where A′ = A + I N , D i ' i =∑A i ' j .

[0028]

[0029] To sum up, the specific operation of the graph convolution we use is shown as follows

[0030]

[0031] Among them, H (l) is the feature information output by the lth layer of graph convolution, and is also the input information of the l+1th layer of graph convolution. When l=0, H (l) =x, θ is the learnable weight. δ is the activation function. In this paper, ReLU is selected as the activation function to achieve nonlinear operation. During the GCN training process, the information exchange between all nodes is realized through the association matrix A. The surrounding feature information is gathered and its state is changed through the linear transformation W.

[0032] Through the above-mentioned graph convolution operation, which is equivalent to the convolution layer and pooling layer in the convolution operation, local perception of feature information in remote sensing images is achieved, and comprehensive operation of local information is performed to achieve information exchange between different nodes, thereby obtaining global information. The maximum range of information that the current node can receive from other nodes is the receptive field of the graph, but because the graph convolution does not have sampling like the pooling layer, it avoids the loss of local information. As various experiments have shown, the GCN layer should not be too deep, so the GCN designed in this paper has only two layers.

[0033] As a further embodiment of the present invention:

[0034] In step S3, the CNN branch is used to extract features, and then the GCN branch is used to capture the relationship between features. The relationship flow between nodes in the graph structure is upsampled after passing through two layers of graph convolution, and finally the decoder outputs the final road segmentation image.

[0035] Since the GCN model is used to replace the convolution layer and the pooling layer, not only the receptive field is expanded, but also the local position information is not lost in the process of graph convolution. Therefore, the road segmentation network based on graph convolution proposed in the present invention can obtain more dense context information, thereby better realizing high-resolution remote sensing image road segmentation, and finally obtaining a more realistic and more accurate road extraction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall architecture diagram of the present invention.

[0037] Figure 2 For the bottleNeck network architecture. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] The real-time scheme of this experiment is as follows:

[0040] Step S1: Construct the experimental dataset. The experiment is mainly trained on the Massachusetts dataset, which includes 1108 training sets, 14 validation sets, and 49 test sets. The pixel size of each image is 1500*1500. Secondly, the dataset is an aerial remote sensing image, covering an area of ​​more than 2634 square kilometers. The shooting areas include cities, suburbs, and rural areas.

[0041] Step S2: Construct a network, using the improved ResNet34 as the CNN branch, which is mainly used for high-resolution remote sensing image feature extraction. It is mainly composed of bottleNeck modules, which reduces the amount of network parameters and can deepen the network level, so as to obtain deeper feature information. The feature information extracted by the CNN branch is used as the node of the graph structure, and the Gaussian kernel function is used to initialize the adjacency matrix of the graph structure. Then, through two layers of graph convolution, the message propagation between nodes is used to exchange information between nodes, so that features can be extracted in a larger neighborhood and the loss of local information can be avoided. The constructed network adopts a multi-task learning method. After feature extraction and obtaining the relationship between features, the acquired feature information is continuously upsampled, and finally the road segmentation result is output through the decoder.

[0042] Step S3: Dataset enhancement. Since the Massachusetts dataset has a small test set, in order to improve the robustness of the prediction, we also performed data enhancement on the test dataset, including horizontal flipping, vertical flipping, diagonal flipping, etc. Through the above series of operations on the dataset, the number and variability of the dataset are enhanced. The network trained using this data will have a better generalization effect and can also show good results for new remote sensing image datasets.

[0043] Step S4: Network training. To avoid overfitting, data enhancement techniques such as image translation, horizontal flipping, vertical flipping, and multi-scale transformation are performed on the training images during the training process. The network is trained using the pytorch framework. Considering the imbalance of positive and negative samples in road segmentation, that is, the pixel ratio of the road is much smaller than that of the background, it is easy to lose important information during iteration. Therefore, dice loss is used to train the network, and the parameters are updated adaptively using the Adam optimizer. At the same time, the training batch size is set to 8, and training is performed simultaneously on 4 2080ti GPUs. The learning rate is initially set to 3e-4. When the loss function does not decrease for more than 3 times, the learning rate is divided by 5 until the model converges.

[0044] Step S5: Network test. This paper defines the road segmentation problem as a binary classification problem of pixels, that is, road pixels are positive and background pixels are negative. Accuracy, precision, recall, and F1-score are used as evaluation criteria for prediction results. Accuracy indicates the percentage of correctly predicted results in the total samples. Precision is also called precision, which is the probability that a sample is actually a road sample among all samples predicted to be roads. Recall indicates the probability that a pixel is predicted to be a road pixel among pixels that are actually roads. Since accuracy and recall are contradictory, F1-score considers both accuracy and recall at the same time, so that both can reach the highest value at the same time and achieve a balance.

[0045]

[0046] Among them, TP, TN, FN, and FP represent the true positive, false positive, true negative, and false negative pixels of the predicted image pixels, respectively.

[0047] The main innovation of the present invention is to use graph convolution to obtain the relationship between feature information and effectively capture local location information. The performance on the Massachusetts dataset proves that the road extraction network architecture designed by the present invention has better performance.

[0048] In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements.

[0050] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, and therefore it is intended to include all changes within the meaning and scope of the equivalent elements of the claims within the present invention; any figure mark in the claims should not be regarded as limiting the claims involved.

[0051] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for extracting roads from remote sensing images based on a graph convolutional model, characterized in that: The following steps are included: Step S1: Design a feature extraction branch based on the CNN model to realize feature extraction of remote sensing images; Step S2: Design a local information capture branch based on the GCN model, and generate corresponding graph nodes according to the feature information extracted by the CNN model; Step S3: using the designed road extraction network based on the graph convolution model to perform overall design, thereby realizing road extraction from high-resolution remote sensing images and obtaining results; According to the road characteristics in remote sensing images, the GCN model is used to realize road extraction in remote sensing images; the overall framework adopts an encoder-decoder structure, which is divided into a feature extraction branch based on the CNN model and a local information capture branch based on the GCN in the encoder stage. First, the feature extraction branch based on the CNN model is designed according to the road characteristics, and the ResNet34 pre-trained on the ImageNet dataset is used as the encoder. The last fully connected layer is removed, and the improved ResNet34 is used to extract the features of relevant road information. Then, the obtained relevant road feature information is used as the input of the GCN model. First, the relevant graph structure is constructed according to the features, and the feature information is regarded as a node, and the difference between the nodes is regarded as an edge. The designed graph convolution model is used to obtain the relationship between the nodes in the graph structure to obtain the local position information. The entire network adopts a multi-task joint learning method to realize the extraction of road information in high-resolution remote sensing images, thereby improving the accuracy of road segmentation; S1 implements the design based on CNN feature extraction branch, designs the encoder of the road extraction network based on the ResNet34 network structure, removes the fully connected layer, and uses two convolutional layers to reduce and increase the feature dimension respectively because ResNet34 is implemented using the bottleneck layer; The skip connection method is used to achieve the fusion of shallow information and deep features, thereby solving the gradient disappearance problem during training; The improved ResNet34 is used as the feature extraction branch of the designed network, and the original image is used as the input of this branch to extract the relevant road feature information in the remote sensing image. The expression of this branch is shown in formula (1); Y=g(x) (1) Where x is the input original image, and the function ɡ represents a series of convolutional pooling operations. After the CNN feature extraction function, the road feature information Y is finally obtained; S2 implements the design of the GCN local information capture branch; GCN, as an emerging network architecture, uses GCN to capture local information and transforms the road extraction problem into a graph node classification problem; In order to capture the relationship between the features extracted by CNN, a graph structure is established based on the road feature information extracted by the CNN branch. The graph structure consists of nodes and edges. The defined graph structure is represented by G = (V, E), where V represents the set of nodes, E represents the set of edges of the graph, the node V represents the feature information extracted by CNN, and the edge E represents the difference between the nodes. The adjacency matrix A of the graph is used to represent the difference between each node in the graph structure and other nodes. A Gaussian kernel function is used to initialize the adjacency matrix of the graph structure. After that, several layers of graph convolution are performed and message propagation is used to exchange information between nodes, thereby extracting features within the neighborhood and avoiding local information loss. The remote sensing image is input into the CNN feature extraction branch. After several layers of convolution operations, the feature information in the image is obtained as the node feature input of GCN. In the GCN-based branch, the adjacency matrix A and feature x generated based on the features obtained by CNN are used as input. The specific graph convolution operation is shown in formula (2): θ Represented as convolution kernel, θ0θ1 is the convolution parameter, I N is the identity matrix; Where L is the Laplacian matrix of the graph, A is the relationship matrix pre-generated based on feature information, D is the degree matrix of the vertex, and the normalized Laplacian matrix is The final transformation yields equation (2); The two parameters θ0θ1 are transformed into a single parameter θ, and the convolution formula (3) is obtained as follows: because The eigenvalue range is [0, 2], which will lead to numerical instability and gradient explosion or vanishing problems. Therefore, (4) is used for normalization, where A′=A+I N , D i ' i =∑A i ' j ; To sum up, the specific operation of the graph convolution used is shown as follows Among them, H (l) is the feature information output by the lth layer of graph convolution, and is also the input information of the l+1th layer of graph convolution. When l=0, H (l) =x, θ is the learnable weight; δ is the activation function, and ReLU is selected as the activation function to achieve nonlinear operation. During the GCN training process, the information exchange between all nodes is realized through the association matrix A; the surrounding feature information is gathered and its state is changed through the linear transformation W; Through the above graph convolution operation, the convolution layer and pooling layer in the convolution operation can realize the local perception of feature information in the remote sensing image, and perform comprehensive operations on local information to realize information exchange between different nodes, so as to obtain global information; the maximum range of information that the current node can receive from other nodes is the receptive field of the graph, and the designed GCN has two layers; In S3, the CNN branch is used for feature extraction, and then the GCN branch is used to capture the relationship between features. The relationship flow between nodes in the graph structure passes through two layers of graph convolution, and the acquired deep feature information is upsampled, and finally the decoder outputs the final road segmentation image.

Images