Computer system and storage medium for extracting a model of a road from a remote sensing image

Abstract

The application provides a model for extracting a road from a remote sensing image, a computer system and a storage medium. The model comprises an encoder, a feature extraction module for performing snake convolution on an input image, and a center module configured to perform snake convolution on a feature map output by the feature extraction module to extract low-level features, perform DW convolution and PW convolution on the feature map output by the feature extraction module, respectively, perform detection on the feature maps output by the DW convolution and the PW convolution, respectively, to generate a first graph structure and a second graph structure, extract spatial features and channel features from the first graph structure and the second graph structure by using a graph convolution network, respectively, fuse the spatial features, the channel features and the low-level features to obtain a multi-scale feature map, a decoder for restoring the multi-scale feature map output by the encoder to an output with the same size as the input image, and a classifier for classifying road targets and backgrounds in the feature map output by the decoder.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to road extraction in a complex background, in particular to a model for extracting road information from remote sensing images, a computer system and a storage medium. BACKGROUND

[0002] Road information in high-resolution remote sensing images is widely used in various fields. With the acceleration of urbanization and the continuous development of intelligent transportation systems, the accuracy and real-time performance of road information are of great significance to urban planning, traffic management, disaster response and other fields. Traditional road extraction methods, such as pixel-based classification and object-oriented extraction, have achieved certain results, but still face many challenges in dealing with complex scenes, improving extraction accuracy and efficiency, etc.

[0003] In recent years, deep learning technology, which has a significant advantage in processing large-scale remote sensing images, has brought new ideas and methods to road extraction. Various deep learning networks have been used for road extraction, such as U-Net, SegNet, FCN, etc. These networks can extract and restore features through an encoder-decoder pair, which can handle global information while preserving edge details. In addition, in order to further improve the accuracy and efficiency of road extraction, researchers have also proposed many improvement methods, such as introducing attention mechanisms, using transfer learning, optimizing loss functions, etc. Overall, the application of deep learning in road extraction mainly relies on convolutional neural networks (CNN) to extract the features of the target. CNN can automatically learn and extract deep features in images by simulating the connection mode of human brain neurons, and has strong recognition ability for road structures in remote sensing images.

[0004] For detecting roads covered by asphalt pavement or trees in high-resolution remote sensing images, some challenges still limit the accuracy of extraction: 1) the intra-class difference between roads is large, while the inter-class difference between urban objects (especially roads and buildings) is not obvious; 2) roads covered by trees, shadows and buildings are difficult to extract. These CNN networks based on convolution kernel feature extraction are limited by their limited receptive field, which cannot accurately perceive the slender road information, so the convolution network needs to increase the perception ability of the slender road. SUMMARY

[0005] The application proposes a model for extracting roads from remote sensing images based on dynamic snake convolution combined with graph convolution, a computer system and a storage medium, which at least solves the problem of extracting roads from high-resolution remote sensing images in a complex background, to improve the accuracy of extracting roads from high-resolution remote sensing images.

[0006] In a first aspect, a model for extracting a road from a remote sensing image includes an encoder comprising a feature extraction module configured to convolve an input image using a multi-level dynamic snake convolution, and a center module comprising a dynamic snake convolution block configured to snake convolve a feature map output by the feature extraction module to extract low-level features, a depthwise separable convolution network configured to convolve the feature map output by the feature extraction module using DW and PW convolutions, respectively, a graph structure establishment module configured to detect the feature map output by the DW convolution using a Sobel operator and generate a first graph structure, detect the feature map output by the PW convolution using the Sobel operator and generate a second graph structure, a graph convolution network configured to extract spatial features and channel features from the first and second graph structures, respectively, and a fusion module configured to fuse the spatial features, the channel features, and the low-level features to obtain a multi-scale feature map, a decoder configured to restore the multi-scale feature map output by the encoder to an output having the same size as the input image, and a classifier configured to classify road targets and backgrounds in the feature map output by the decoder.

[0007] In some examples, the feature extraction module comprises an initial block and a second block connected in sequence, a pooling layer, a first residual network, and a second residual network, the initial block convolves the input image, and a feature map generated thereby is input to the second block, the second block continues to convolve, and a feature map generated thereby is input to the pooling layer, the first residual network, and the second residual network in sequence, and a feature map generated by the second residual network is divided into three paths and input to the dynamic snake convolution block of the center module and the DW and PW convolutions of the depthwise separable convolution network.

[0008] In some examples, the initial block has a plurality of convolution layers, each of which comprises a dynamic snake convolution used to extract road information, and each of which is followed by a batch normalization and a rectified linear unit activation function.

[0009] In some examples, the second block has a plurality of convolution layers, each of which comprises a dynamic snake convolution used to extract road information, and each of which is followed by a batch normalization and a rectified linear unit activation function.

[0010] In some examples, the convolution layers inside the first residual network use dynamic snake convolutions.

[0011] In some examples, the convolution layers inside the second residual network use dynamic snake convolutions.

[0012] In some examples, the feature maps of the encoder are passed to the decoder through a skip link, and the decoder uses these feature maps to generate the output through a transpose convolution operation.

[0013] In some examples, the decoder comprises a first dynamic snake convolution block, a second dynamic snake convolution block, a third dynamic snake convolution block and a fourth dynamic snake convolution block, the first dynamic snake convolution block is used to perform snake convolution on the multi-scale feature maps output by the center module first, then the transposed convolution is used to perform transposition operation on the feature maps after the snake convolution, to complete the first group of up-sampling; the feature maps obtained by the first group of up-sampling are superimposed with the feature maps output by the first residual network, and then input to the second dynamic snake convolution block, the snake convolution is performed first, and then the transposed convolution is used to perform transposition operation on the feature maps after the snake convolution, to complete the second group of up-sampling; the feature maps obtained by the second group of up-sampling are superimposed with the feature maps output by the second block, and then input to the third dynamic snake convolution block, the snake convolution is performed first, and then the transposed convolution is used to perform transposition operation on the feature maps after the snake convolution, to complete the third group of up-sampling; the feature maps obtained by the third group of up-sampling are superimposed with the feature maps output by the initial block, and then input to the fourth dynamic snake convolution block, the snake convolution is performed, and the output of the decoder is obtained.

[0014] In a second aspect, a computer system is provided, comprising a processor and a memory, wherein the model for extracting roads from remote sensing images is stored in the memory and configured to be executed by the processor.

[0015] In a third aspect, a computer readable storage medium is provided, configured to store non-transitory computer readable instructions, which can execute the model for extracting roads from remote sensing images when executed by a computer.

[0016] The method constructs a road extraction model, wherein the dynamic snake convolution can well perceive the elongated road information, improve the feature extraction capability of the network, and the graph convolution is used to identify the context information, so that the road and the background can be well distinguished, in a complex environment, the modules work coordinately, the proposed model has superior performance, especially in extracting small road targets and complex roads. For extracting roads from high-resolution remote sensing images, a larger receptive field cannot produce better results, and too large a receptive field often brings too much noise, so it is not easy to expand the receptive field of the module, but the receptive field should be more concentrated, and the model has high application value; the extracted road information can be used to correct the road data set. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a road extraction network structure diagram provided by an embodiment of the present application, which combines dynamic snake convolution and graph convolution.

[0018] Figure 2 FIG. 5 is a snake convolution schematic diagram.

[0019] Figure 3 FIG. 7 is a center module structure diagram.

[0020] Figure 4 This is a road remote sensing image and corresponding annotation data provided in an embodiment of the present invention.

[0021] Figure 5 This is data provided by an embodiment of the present invention to verify the model of the present invention. Detailed Implementation

[0022] Figure 1 A model (referred to as "network" or "model") for extracting road information from remote sensing imagery is shown. For example... Figure 1 As shown, the model combines dynamic serpentine convolution with graph convolution. A continuous bias is added to the kernel of the dynamic serpentine convolution, causing each subsequent kernel to oscillate with the previous one, thus dynamically adapting to the target. Graph convolution has attracted widespread attention due to its ability to handle topological structures. When applied to image processing, it can effectively distinguish between different features; for example, graph convolution can extract common roads from images because common road information is readily apparent. However, in areas with complex terrain, graph convolution struggles to differentiate roads from background information. Therefore, this invention combines dynamic serpentine convolution with graph convolution, enabling effective road extraction in complex environments.

[0023] The model comprises an encoder and a decoder. The encoder includes a feature extraction module and a central module. The feature extraction module includes an initial block, a second block, pooling layers, a first residual network, and a second residual network. The model uses an RGB remote sensing image as input to extract image information. The encoder starts with an initial block, which has three convolutional layers, each containing a dynamic serpentine convolution to extract road information. Each convolutional layer is followed by a batch normalization (BN) and rectified linear unit (RELU) activation function. The feature map generated by convolving the remote sensing image in the initial block is input to the second block, which continues to convolve. The second block has the same composition as the initial block. The pooling layers perform pooling operations on the feature map generated by the convolution in the second block. The pooled feature map is then fed into the first residual network for feature extraction. In one embodiment, the first residual network includes several first residual blocks, for example, three. The feature map generated by the first residual network is then fed into the second residual network for feature extraction. The second residual network comprises multiple residual blocks connected in series, such as four second residual blocks, six third residual blocks, and three fourth residual blocks. The first, second, third, and fourth residual blocks have the same composition but different parameters. Those skilled in the art can determine the specific number of residual blocks and their parameters based on the complexity of the remote sensing image background. In a preferred embodiment, dynamic serpentine convolution is also used within the residual blocks.

[0024] The center module includes a depth separable convolution network, a graph structure establishing module, a graph convolution network (GCN), and a dynamic snake convolution block. The features extracted by the second residual network are sent to the dynamic snake convolution block, the DW convolution, and the PW convolution of the depth separable convolution network in three paths respectively. The dynamic snake convolution block performs snake convolution on the feature maps output by the feature extraction module to extract low-level features. The DW convolution and the PW convolution perform convolution on the feature maps output by the feature extraction module respectively. The graph structure establishing module detects the feature maps output by the DW convolution using a Sobel operator and generates a first graph structure, and detects the feature maps output by the PW convolution using a Sobel operator and generates a second graph structure. Spatial features are extracted from the first graph structure using multi-layer graph convolution (for example, 3 layers). Channel features are extracted from the graph structure using multi-layer graph convolution (for example, 3 layers). The spatial features and the channel features extracted by the graph convolution network are called high-level features. In order to restore more segmentation details such as texture, boundary, and spatial information, the final fusion module fuses the low-level features extracted by the dynamic snake convolution block and the high-level features extracted by the graph convolution network to obtain a multi-scale feature map.

[0025] The decoder restores the multi-scale feature map output by the encoder to an output with the same size as the input image. The decoder uses multiple sets of up-sampling operations to restore the multi-scale feature map to an output with the same size as the input image. The feature maps of the encoder are passed to the decoder through a skip link, and the decoder uses these feature maps to generate the output through operations such as transposed convolution. Figure 1 In the illustrated embodiment, first, the first dynamic snake convolution block performs 3 times of snake convolution on the multi-scale feature map output by the center module, and then the transposed convolution performs a transposition operation on the feature map after 3 times of snake convolution to complete the first set of up-sampling. After the feature map obtained by the first set of up-sampling is superimposed with the feature map output by the first residual network, it is input to the second dynamic snake convolution block, which first performs 3 times of snake convolution, and then uses the transposed convolution to perform a transposition operation on the feature map after 3 times of snake convolution to complete the second set of up-sampling. After the feature map obtained by the second set of up-sampling is superimposed with the feature map output by the second block, it is input to the third dynamic snake convolution block, which first performs 3 times of snake convolution, and then uses the transposed convolution to perform a transposition operation on the feature map after 3 times of snake convolution to complete the third set of up-sampling. After the feature map obtained by the third set of up-sampling is superimposed with the feature map output by the initial block, the obtained feature map is input to the fourth dynamic snake convolution block, which performs 3 times of snake convolution to obtain the output of the decoder. The size of the dynamic snake convolution in the decoder can be 3x3.

[0026] The classifier classifies the feature map generated by the decoder into road targets and background targets. The classifier uses a sigmoid activation function for classification.

[0027] It can be seen that the model of the application uses dynamic snake convolution globally, so that the extracted features are easier to pass in the network. The following will be described in combination with Figure 2 The principle of dynamic snake convolution is introduced.

[0028] The calculation of snake convolution is as follows: assuming that the standard two-dimensional convolution coordinates are K, the center coordinates are K i =(x i, ,y i ). The 3*3 kernel K expanded by 1 is represented as

[0029] K={(x-1,y-1),(x-1,y),…,(x+1,y+1)}(1)

[0030] The deformation offset Δ is introduced to make the convolution kernel pay more attention to the complex geometric features of the target. However, if the model is allowed to learn the deformation offset freely, the perception field will deviate from the target, especially in the case of tubular structure.

[0031] Therefore, an iterative strategy is adopted to select the subsequent position to be observed for each target to be processed in turn, so as to ensure the continuity of attention and avoid the perception range being too scattered due to too large deformation offset.

[0032] In snake convolution, we straighten the standard convolution kernel in the x-axis and y-axis directions. Taking a convolution kernel with a size of 9 as an example, the specific position of each grid in K is represented as K i±c =(x i±c ,y i±c ), where c={0,1,2,3,4} represents the horizontal distance from the center grid. Selecting each grid position K i±c in the convolution kernel K is a cumulative process.

[0033] Starting from the center position K i , the position away from the center grid depends on the previous position: compared with K i , K i+1 increases the offset Δ={δ∣δ∈[-1,1]}. Therefore, the offset needs to be added Σ to ensure that the convolution kernel meets the linear morphological structure.

[0034] Where the x-axis direction becomes

[0035]

[0036] Equation (2) becomes

[0037]

[0038] Since the offset Δ is usually a decimal number, the implementation of bilinear interpolation is

[0039] K =∑ K′ B(K',K)·K'

[0040] where K represents the fractional position of equation (2) and equation (3), K' represents all integral space positions, and B is a bilinear interpolation kernel, which is divided into two one-dimensional kernels, respectively

[0041] B(K,K')=b(K x ,K' x )·b(K y ,K' y )

[0042] Dynamic snake convolution covers the change of 9*9 range (X axis, Y axis) in the deformation process. The design of dynamic snake convolution is to better adapt to the elongated tubular structure based, so as to better perceive the key features.

[0043] Referring Figure 3 , the application uses depth separable convolution in the center module, which decomposes the traditional convolution into two parts of depth (DW) and pointwise convolution (PW) to reduce the operation cost. In the DW step, only one 1D convolution kernel is convolved in one channel, and the convolution result is consistent with the size of the input. In the PW step, it uses multiple 1*1*N (N is the total number of channels in the DW output) to construct the graph structure using Sobel operator on the result of DW and PW. The graph convolution network captures the global road context information in channel and spatial features.

[0044] The Sobel gradient operator is used to construct the adjacency matrix of the feature map, G x Convolved in x direction, G y Convolved in y direction, the calculation formula of the adjacent matrix is as follows:

[0045]

[0046] Wherein, the operation δ changes the position of the channel in the grid coordinates and down-samples the channel features. Thus, the Sobel operator can reasonably process the channel size while reducing the number of parameters. D And I N is the identity matrix. X s is the input spatial feature, is the output spatial feature; X c is the input channel feature, is the output channel feature.

[0047] After generating the adjacency matrix, a fully connected graph is established. Define the undirected graph G=(V,E), where there are N nodes v i ∈V and edges (v i ,vj )∈E, the formula definition of graph convolution is as follows:

[0048]

[0049] Wherein, sigma is a nonlinear activation function, T is the transpose of the matrix, l is the number of layers of the current graph convolution, is the order matrix of G, W is the trainable weight matrix of a specific layer. The model adopts three layers of graphs to calculate the representation of spatial and channel features.

[0050] When training the model, a road semantic segmentation dataset is constructed using satellite remote sensing images, if the model is mainly used to extract mountain road information, then these images mainly come from mountainous areas are the best. And the image should contain the effective area of the road. The national road network data is used to label the road in the image, and then the road is calibrated and manually corrected. Figure 4 The first row in the table is a satellite remote sensing image, and the second row is the corresponding road labeling data. The road labeling data is open source, and the present application can use these satellite remote sensing images with road labeling data to train the model.

[0051] The sliding window is used to cut the dataset of the remote sensing image. For example, it is divided into 256*256 size images, and the training set, validation set and test set are divided, so that they contain 204, 11 and 40 images respectively. The resolution of all images can be set to 2 meters. When training the model, the training set is shuffled to fully explore the ability of the model to extract context information.

[0052] In this pixel-based road extraction task, recall, precision, F1 score and intersection over union (IOU) are used to evaluate the model, and the calculation formula is as follows:

[0053]

[0054] Wherein, TP represents the number of pixels correctly extracted as roads, FP represents the number of pixels extracted as other objects, TN represents the number of pixels correctly extracted as background, and FN represents the number of pixels extracted as background.

[0055] Figure 5 The true color image in the first row is input into the model of the present application and the control model (without using graph convolution), and the segmentation results of the two are compared with the road labeling data. Figure 5 The second row in the table is the road labeling data, the third row is the segmentation result of the control model, and the fourth row is the segmentation result of the model of the present application. The Figure 1The two graph convolution branches of the center module of the model shown are deleted, only the third branch, the snake convolution, is reserved to constitute the control model. The evaluation results are shown in Table 1. The IOU values of the model of the application are higher than those of the control model without using graph convolution, and the recall rate of the application is higher. When we extract the road, we use the mask of the label to extract the result, and the higher recall rate means that the model of the application extracts more road information. In addition, the model of the application achieves an mIOU of 83.72% in the public data set (Massachusetts road data set), which shows that the model of the application not only can exhibit good results in the face of complex backgrounds, but also has strong generalization.

[0056] Table 1 Evaluation results of the model

[0057]

[0058] With reference to Figure 5 For the extraction of two-row road information, as can be seen from the red rectangular mark in the figure, due to the interference and shielding of the surrounding buildings on the road in the mark frame, the road information in this part is not extracted when graph convolution is not used, and the road information of this road section can be better extracted using the application. For the extraction of multi-section road information, as can be seen from the yellow rounded rectangular mark in the figure, this road section is a crossroads, and when graph convolution is not used, the road and the background cannot be well distinguished and part of the extraction is wrong. After using graph convolution, errors can be well avoided, and even part of the road information not marked is extracted. The road information on the upper part of the yellow rectangular frame is not obvious in true color, and both of them cannot completely extract the road section. For a single road section, as shown in the green elliptical mark in the figure, the application can extract complete and smooth results, while when graph convolution is not used, the road extraction appears to be in layers.

[0059] The above analysis verifies the effectiveness of the road extraction model of the application based on dynamic snake convolution combined with a graph convolution network. The dynamic snake convolution is used to fully extract road information, and the graph structure is constructed by using a Sobel operator, and better results can be obtained after combining with graph convolution. In a complex background, the interference of background information is effectively avoided, the road and background information are distinguished as accurately as possible, the results are more persuasive and objective, and the evaluation indexes of the model are effectively improved.

[0060] The present application also provides an embodiment of a computer system. The model for extracting road information from remote sensing images is deployed on the computer system. The computer system comprises a processor and a memory. The memory is configured to store non-transitory computer-readable instructions (e.g., instructions for implementing the model for extracting road information from remote sensing images). The processor is configured to execute the model for extracting road information from remote sensing images, which, when executed by the processor, can segment roads from true color images. The memory and the processor can be interconnected by a bus system and / or other forms of connection mechanism.

[0061] For example, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capability and / or program execution capability. For example, the central processing unit (CPU) can be of X86 or ARM architecture, etc. The processor can be a general-purpose processor or a special-purpose processor, and can control other components in the computer to perform desired functions.

[0062] For example, the memory can comprise various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, comprise random access memory (RAM) and / or cache memory, etc. The non-volatile memory may, for example, comprise read-only memory (ROM), a hard disk, erasable programmable read-only memory (EPROM), compact disc read-only memory (CD-ROM), a USB memory, flash memory, etc. One or more computer program modules can be stored on the computer-readable storage medium, and the processor can execute the one or more computer program modules to implement various functions of the computer.

[0063] The present application also provides an embodiment of a computer-readable storage medium for storing non-transitory computer-readable instructions, which, when executed by a computer, can implement the model for extracting road information from remote sensing images. That is, when the model for extracting road information from remote sensing images provided by the embodiments of the present application is implemented in the form of software and sold or used as an independent product, it can be stored in a computer-readable storage medium. For related descriptions of the storage medium, reference can be made to the corresponding description of the memory in the computer above, which will not be described here again.

Claims

1. A model for extracting roads from remote sensing imagery, characterized in that, include: An encoder, comprising: The feature extraction module is configured to convolve the input image using multi-level dynamic serpentine convolution; and The central module includes: Dynamic serpentine convolution blocks are configured to perform serpentine convolutions on the feature maps output by the feature extraction module to extract low-level features; A depthwise separable convolutional network, in which the DW convolution and PW convolution are respectively performed on the feature map output by the feature extraction module; The graph structure building module is configured to use the Sobel operator to detect the feature map output by the DW convolution and generate the first graph structure, and use the Sobel operator to detect the feature map output by the PW convolution and generate the second graph structure. Graph convolutional networks are configured to extract spatial and channel features from a first graph structure and a second graph structure, respectively. and The fusion module is configured to fuse spatial features, channel features, and low-level features to obtain a multi-scale feature map; The decoder is configured to recover the multi-scale feature map output by the encoder into an output of the same size as the input image. as well as A classifier is configured to classify road targets and background in the feature map output by the decoder.

2. The model for extracting roads from remote sensing imagery according to claim 1, characterized in that, The feature extraction module includes an initial block and a second block, a pooling layer, a first residual network, and a second residual network connected in sequence. The initial block convolves the input image, and the generated feature map is fed into the second block. The second block continues to convolve, and the feature map generated by the second block is fed into the pooling layer, the first residual network, and the second residual network in sequence. The feature map generated by the second residual network is fed into the dynamic serpentine convolution block of the central module and the DW and PW convolutions of the depth-separable convolutional network in three separate paths.

3. The model for extracting roads from remote sensing images according to claim 2, characterized in that, The initial block has several convolutional layers, each containing a dynamic serpentine convolution used to extract road information, and each convolutional layer is followed by a batch normalization and rectified linear unit activation function.

4. The model for extracting roads from remote sensing images according to claim 2, characterized in that, The second block has several convolutional layers, each containing a dynamic serpentine convolution used to extract road information, followed by a batch normalization and rectified linear unit activation function.

5. The model for extracting roads from remote sensing images according to claim 2, characterized in that, The convolutional layers inside the first residual network employ dynamic serpentine convolution.

6. The model for extracting roads from remote sensing images according to claim 2, characterized in that, The convolutional layers inside the second residual network employ dynamic serpentine convolution.

7. The model for extracting roads from remote sensing imagery according to claim 2, characterized in that, The encoder's feature maps are passed to the decoder via skip links, and the decoder uses these feature maps to generate output through a transposed convolution operation.

8. The model for extracting roads from remote sensing imagery according to claim 7, characterized in that, The decoder comprises a first dynamic serpentine convolutional block, a second dynamic serpentine convolutional block, a third dynamic serpentine convolutional block, and a fourth dynamic serpentine convolutional block. First, the first dynamic serpentine convolutional block performs serpentine convolution on the multi-scale feature map output from the central module, and then transposes the feature map after the serpentine convolution, completing the first set of upsampling. The feature map obtained from the first set of upsampling is superimposed with the feature map output from the first residual network and input into the second dynamic serpentine convolutional block. Serpentine convolution is performed first, and then transposes the feature map after the serpentine convolution, completing the second set of upsampling. The feature map obtained from the second set of upsampling is superimposed with the feature map output from the second block and input into the third dynamic serpentine convolutional block. Serpentine convolution is performed first, and then transposes the feature map after the serpentine convolution, completing the third set of upsampling. The feature map obtained from the third set of upsampling is superimposed with the feature map output from the initial block and input into the fourth dynamic serpentine convolutional block for serpentine convolution, yielding the decoder output.

9. A computer system, characterized in that, Includes a processor and a memory, wherein the model for extracting roads from remote sensing imagery as described in any one of claims 1-8 is stored in the memory and configured to be executed by the processor.

10. A computer-readable storage medium for storing non-transitory computer-readable instructions, characterized in that, When the non-transitory computer-readable instructions are executed by a computer, the model for extracting roads from remotely sensed images as described in any one of claims 1-8 can be run.

Images