Remote sensing image road extraction method and system based on multi-scale difference feature fusion mechanism

Abstract

The invention relates to the technical field of image processing, in particular to a road extraction method and system based on a multi-scale difference feature fusion mechanism, and the method comprises the following steps: carrying out the preprocessing of remote sensing image data, and obtaining a training and testing data set; constructing a road extraction network based on a multi-scale difference feature fusion mechanism; and training the road extraction network by using a training data set, and inputting a test set remote sensing image into the trained extraction network to obtain a road extraction result. The method emphatically solves the problem that relevance information loss and redundant information stacking are easily caused when intermediate feature transfer is performed in a coding-decoding structure, the multi-scale feature difference fusion mechanism is introduced, the long-distance dependency relationship modeling capability is enhanced, the difference features are extracted through the feature subtraction mechanism, cross-layer transfer fusion is performed, redundant information interference is reduced, and the reliability of the coding-decoding structure is improved. And the road extraction accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a remote sensing image road extraction method and system based on a multi-scale difference feature fusion mechanism. Background Art

[0002] High-resolution remote sensing images contain many key information, such as urban natural environment data, urban human environment data, economic data, etc., as well as rich ground feature information, such as river distribution, road distribution, building distribution, etc. As one of the basic ground feature elements, road information plays an important role in urban planning, public transportation management, smart city construction, disaster monitoring, unmanned driving and other research fields. At the same time, as the core of the national transportation system, roads have important economic significance.

[0003] Extracting road information from high-resolution remote sensing images belongs to the interpretation and classification of remote sensing data. The accuracy of the results is affected by many factors, such as the imaging process and the complexity of the actual objects in the selected area at the image level, and the design and construction of the road extraction method at the technical level. In the object space, the road is a three-dimensional model containing a variety of color information. Roads of the same type have similar colors and geometric features, and their differences are feature information that can be captured; in the image space, high-resolution remote sensing images have irregular feature distribution, inconsistent edges, and complex spectral information due to problems such as object occlusion and road radiation differences. These features and characteristics bring new challenges to the construction of remote sensing image road extraction methods.

[0004] The current remote sensing image road extraction method using the encoding-decoding structure shows high accuracy and automation. It mostly uses the DCNN module as the main body of the road extraction network. Affected by the inductive bias characteristics of the convolutional neural network, the feature extraction module is relatively efficient in capturing local correlation information, but performs poorly in extracting global feature information. Roads have complex structures, large spans, narrow and tortuous features. It is difficult to further enhance the coherence of the extraction results by simply stacking convolutional neural network layers, and it is easy to have partial breaks in long-distance extraction results. At the same time, the jump connection structure in the remote sensing image road extraction method with the encoding-decoding structure mostly uses a single element-by-element addition or splicing operation, which is easy to cause problems such as redundant information stacking and loss of key feature information. There is a lack of explicit modeling of the relationship between channel dimensions of intermediate features. At the same time, feature transfer is limited to the same level, resulting in partial loss of spatial correlation information. When affected by trees and buildings, local mis-extraction and omission are prone to occur.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a remote sensing image road extraction method and system based on a multi-scale difference feature fusion mechanism, to reduce the incomplete and discontinuous extraction of long-distance roads, as well as the phenomenon of wrong extraction, erroneous extraction and missing extraction of local areas under occlusion, and to improve the accuracy of road extraction.

[0007] In a first aspect, an embodiment of the present invention provides a remote sensing image road extraction method based on a multi-scale difference feature fusion mechanism, the method comprising the following steps:

[0008] Step S1: Perform cropping preprocessing on the acquired remote sensing image data to obtain a training data set and a test data set;

[0009] Step S2: constructing a remote sensing image road extraction network based on a multi-scale difference feature fusion mechanism, wherein the road extraction network adopts an encoding-decoding structure, including a decoder, an encoder, and a multi-scale difference feature fusion module;

[0010] The encoder has five layers in total, the first layer is a convolutional downsampling module based on a convolutional neural network, the second to fifth layers all use multi-scale residual modules for feature extraction, and the second to fifth layers are stacked with 3, 4, 6 and 3 groups of multi-scale residual modules respectively; the decoder has five layers in total, each layer includes an upsampling module based on transposed convolution;

[0011] The multi-scale difference feature fusion module is used to connect the encoder and the decoder, and is composed of three difference embedding modules and two fusion modules;

[0012] Step S3: Use the training data set to train the remote sensing image road extraction network based on the multi-scale difference feature fusion mechanism, calculate the loss of the road segmentation prediction feature map and the real road label, perform back propagation through the obtained loss, optimize the model parameters, and complete the model training.

[0013] Step S4: Use the trained road extraction model to extract road information in the remote sensing image to obtain a road extraction result.

[0014] In a second aspect, an embodiment of the present invention provides a road extraction system based on a multi-scale difference feature fusion mechanism, the system comprising:

[0015] at least one processor;

[0016] at least one memory for storing at least one program;

[0017] When the at least one program is executed by the at least one processor, the at least one processor implements the road extraction method based on the multi-scale difference feature fusion mechanism as described in any one of the above.

[0018] Furthermore, the multi-scale difference feature fusion module inputs four groups of encoding features with different depths, outputs three groups of intermediate transfer features with different difference fusion degrees, and performs feature addition operation on the output feature map and the decoding features at the same level;

[0019] Furthermore, the input of the difference embedding module is divided into two parts, the first part is the intermediate transfer feature map of the current layer, i.e., the shallow feature map, and the second part is the intermediate transfer feature map of the adjacent deep layer, i.e., the deep feature map; the input of the fusion module is divided into two parts, the first part is the difference feature map of the current layer, i.e., the shallow difference feature map, and the second part is the difference feature map of the adjacent deep layer, i.e., the deep difference feature map

[0020] Furthermore, the difference embedding module includes a shallow feature extraction module, a deep feature extraction module and an extraction embedding module. The calculation formulas of the shallow feature extraction module, the deep feature extraction module and the extraction embedding module based on the channel attention mechanism are as follows:

[0021] X1′=δ(W 12 σ(W 11 A(C5(X1))))

[0022] X2′=δ(W 22 σ(W 21 A(X2))))

[0023] P=|(X1′!X2′)|

[0024]

[0025] Among them, X1 and X2 are the shallow feature map and the deep feature map after upsampling, respectively, X1′ and X2′ are the channel descriptors of the corresponding feature maps, δ represents the Sigmoid function, σ represents the ReLU activation function, C5(·) is a 5×5 convolutional layer, A(·) is an average pooling layer, and W 11 and W 12 is the fully connected layer parameter of the shallow feature extraction module, W 21 and W 22 is the fully connected layer parameter of the deep feature extraction module, ! is the element-by-element subtraction operation, is the element-by-element multiplication operation, |·| is the absolute value calculation, P represents the difference weight between the shallow and deep feature maps, and the redundant overlapping information of the intermediate features is reduced by feature subtraction operation. S is the difference feature map that integrates the difference information;

[0026] Furthermore, the fusion module is constructed based on spatial attention. By fusing the difference feature map across layers, the model is promoted to extract detailed road information in remote sensing images, extract spatial difference information, and model the dependency relationship between long-distance features. The calculation formula is as follows:

[0027] S F =c(M(σ(C3(S U )))+S D ),A(σ(C3(S U )))+S D ))

[0028]

[0029] Among them, c(·) is the feature concatenation operation, C3(·) is the 3×3 convolution layer, A(·) is the average pooling layer, M(·) is the maximum pooling layer, S D is the deep difference feature after upsampling, S U is the shallow difference feature, S F is the spatial difference feature descriptor, K is the intermediate transfer feature that fuses the difference information;

[0030] Furthermore, the multi-scale residual module in the encoder is constructed based on a convolutional neural network, by splitting the input feature map into 4 groups along the channel dimension, each group corresponds to a branch, the first branch is an identity mapping structure, the second branch extracts features through a convolutional layer, the third branch input and the second branch output are added and calculated, and then the features are extracted through a convolutional layer, the fourth branch input and the third branch output are added and calculated, and then the features are extracted through a convolutional layer, and the output feature maps of each branch are spliced ​​to obtain the output of the multi-scale residual module;

[0031] Furthermore, the decoder has five layers, each of which is a group of upsampling modules, and the upsampling modules include a convolution layer, a transposed convolution layer with a step size of 2, an activation function ReLU, and a batch normalization BN layer;

[0032] Furthermore, AdamW is used to train and optimize the remote sensing road extraction network based on the multi-scale difference feature fusion mechanism, and the binary cross entropy loss function and the Dice coefficient loss function are combined to calculate the training loss.

[0033] The road extraction method based on the multi-scale difference feature fusion mechanism proposed in this application aims to solve the problem of insufficient long-distance dependency modeling capability and more redundancy and erroneous information in intermediate feature transmission caused by the single remote sensing road extraction network. By introducing a multi-scale difference feature fusion module, the model is promoted to extract detailed road information in remote sensing images, enhance the coherence and integrity of the model's long-distance road extraction, and reduce the phenomenon of wrong extraction and omission. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0035] In the attached picture:

[0036] Figure 1 This is a flow chart of a road extraction method based on a multi-scale difference feature fusion mechanism according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a remote sensing road extraction network structure based on a multi-scale difference feature fusion mechanism according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the structure of a multi-scale residual module according to an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the structure of a multi-scale difference feature fusion module in an embodiment of the present invention; the multi-scale difference feature fusion module in an embodiment of the present invention is composed of three difference embedding modules and two fusion modules;

[0040] Figure 5 Schematic diagram of the difference embedding module structure of an embodiment of the present invention; the difference embedding module of the embodiment of the present invention is one of the sub-modules of the multi-scale difference feature fusion module, is constructed based on the channel attention mechanism and the feature subtraction mechanism, and is composed of a shallow feature extraction module, a deep feature extraction module and an extraction embedding module;

[0041] Figure 6 Schematic diagram of the fusion module structure of an embodiment of the present invention; the fusion module of the embodiment of the present invention is one of the sub-modules of the multi-scale difference feature fusion module, and is constructed based on the spatial attention mechanism;

[0042] Figure 7 Schematic diagram of road extraction results of an embodiment of the present invention; the figure includes two visual comparison groups, each comparison group includes an original remote sensing image, a labeled image, an extraction result of the method of an embodiment of the present invention, and three extraction results of existing road extraction comparison methods;

[0043] Figure 8 Schematic diagram of the structure of a remote sensing road extraction system based on a multi-scale difference feature fusion mechanism according to an embodiment of the present invention; the remote sensing road extraction system according to the method of the embodiment of the present invention comprises at least one processor and at least one memory and stores at least one program; DETAILED DESCRIPTION

[0044] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0045] like Figure 1As shown, the road extraction method based on the multi-scale difference feature fusion mechanism in this embodiment includes the following steps:

[0046] Step S1: randomly divide the obtained public remote sensing image dataset DeepGlobe to obtain a training dataset and a test dataset, and perform cropping preprocessing on the training set and test set images;

[0047] Step S2: construct a remote sensing image road extraction network based on a multi-scale difference feature fusion mechanism, wherein the road extraction network adopts an encoding-decoding structure, including an encoder, a multi-scale difference feature fusion module, and a decoder. The remote sensing image road extraction network structure based on the multi-scale difference feature fusion mechanism is as follows: Figure 2 As shown;

[0048] The encoder has five layers. The first layer is a convolutional downsampling module based on a convolutional neural network. The second to fifth layers all use multi-scale residual modules for feature extraction. The second to fifth layers are stacked with 3, 4, 6 and 3 groups of multi-scale residual modules respectively. The sizes of the encoded feature maps after feature extraction in each layer are

[0049] The multi-scale residual module in the encoder is constructed based on a convolutional neural network, and its structure is as follows: Figure 3 As shown in the figure, the input feature map is split into 4 groups along the channel dimension, each group corresponds to a branch, the first branch is an identity mapping structure, the second branch extracts features through a convolutional layer, the third branch input and the second branch output are added and calculated, and then the convolutional layer is used to extract features, the fourth branch input and the third branch output are added and calculated, and then the convolutional layer is used to extract features, and the output feature maps of each branch are spliced ​​to obtain the multi-scale residual module output. The calculation formula is as follows:

[0050]

[0051] Among them, x i Input feature map for each branch, y i Output feature maps for each branch, T i is the convolutional layer of each branch.

[0052] The decoder has five layers, each of which includes an upsampling module based on transposed convolution, and the upsampling module includes a convolution layer, a transposed convolution layer with a step size of 2, an activation function ReLU and a batch normalization BN layer;

[0053] The multi-scale difference feature fusion module is used to connect the encoder and the decoder, and is composed of three difference embedding modules and two fusion modules. Its structure is as follows: Figure 4As shown, the input is four sets of encoding feature maps with different depths, and the output is three sets of transfer features with different difference fusion degrees. The transfer features with different difference fusion degrees are summed with the decoding features at the same level.

[0054] The input of the difference embedding module is divided into two parts. The first part is the intermediate transfer feature map of the current layer, and the second part is the intermediate transfer feature map of the adjacent deep layer. The difference embedding module includes a shallow feature extraction module, a deep feature extraction module and an extraction embedding module. Its structure is as follows: Figure 5 As shown in the figure, the shallow extraction module and the deep feature extraction module are both constructed based on the channel attention mechanism, and the extraction embedding module is constructed based on the feature subtraction mechanism. Through the feature subtraction mechanism, the redundant overlapping information of the intermediate transmission features is reduced, and the difference feature information is embedded in the intermediate features. The calculation formulas of the three submodules are as follows:

[0055] X1′=δ(W 12 σ(W 11 A(C5(X1))))

[0056] X2′=δ(W 22 σ(W 21 A(X2))))

[0057] P=|(X1′!X2′)|

[0058]

[0059] Among them, X1 and X2 are the shallow feature map and the deep feature map after transposed convolution upsampling, respectively, X1′ and X2′ are the channel descriptors of the corresponding feature maps, δ represents the Sigmoid function, σ represents the ReLU activation function, C5(·) is a 5×5 convolutional layer, A(·) is an average pooling layer, and W 11 and W 12 is the fully connected layer parameter of the shallow feature extraction module, W 21 and W 22 is the fully connected layer parameter of the deep feature extraction module, ! is the element-by-element subtraction operation, is the element-by-element multiplication operation, |·| is the absolute value calculation, P represents the difference weight between the shallow and deep feature maps, and the redundant overlapping information of the intermediate features is reduced by feature subtraction operation. S is the difference feature map that integrates the difference information;

[0060] The fusion module is constructed based on the spatial attention mechanism, and its structure is as follows Figure 6As shown in the figure, the input is divided into two parts. The first part is the current layer difference feature map, and the second part is the adjacent deep layer difference feature map. The difference features output by the extraction and embedding module are further fused across layers to promote the network to transform shallow single features into deep multi-features. The fusion module is built based on spatial attention. By fusing the difference feature maps across layers, the model is promoted to extract detailed road information in remote sensing images, extract spatial difference information, and model the dependency relationship between long-distance features. The calculation formula is as follows:

[0061] S F =c(M(σ(C3(S U )))+S D ),A(σ(C3(S U )))+S D ))

[0062]

[0063] Among them, c(·) is the feature concatenation operation, C3(·) is the 3×3 convolution layer, M(·) is the maximum pooling layer, S D is the deep difference feature after upsampling, S U is the shallow difference feature, S F is the spatial difference feature descriptor, K is the intermediate transfer feature that fuses the difference information;

[0064] Step S3: Use the training data set to train the remote sensing image road extraction network based on the multi-scale difference feature fusion mechanism.

[0065] The model training loss is calculated using a combined loss function, which consists of a binary cross entropy loss function and a Dice coefficient loss function; the cross entropy function is a common loss function in the field of semantic segmentation, which is calculated based on the normalized exponential function. Since the road extraction in remote sensing images involves road information as a positive sample and non-road background information as a negative sample, it belongs to a pixel-level binary classification problem, so a binary cross entropy loss function is used; at the same time, the remote sensing image dataset belongs to a high-resolution remote sensing image, which contains complex ground object information, of which the road area accounts for a small proportion as a whole. In order to solve the problem of imbalance between positive and negative samples, the Dice coefficient loss function is introduced to more accurately evaluate the model segmentation effect. The formulas of the binary cross entropy loss function and the Dice coefficient loss function are as follows:

[0066]

[0067]

[0068] Where N is the total number of pixels, y i Represents the true label value of the corresponding pixel, p iRepresents the predicted value of the corresponding pixel.

[0069] Use AdamW optimizer to update model parameters for network training and accelerate model convergence.

[0070] Step S4: In the embodiment of the present invention, the trained road extraction model and the test set obtained in step S1 are used to perform a road extraction test, and the test results and the real labels are compared for accuracy evaluation. The evaluation index formula is as follows:

[0071]

[0072] Among them, TP represents pixels correctly predicted as roads, FP represents pixels that are background pixels incorrectly predicted as roads, FN represents pixels that are road pixels incorrectly predicted as non-road pixels, Precision represents the ratio of pixels correctly predicted as roads to all pixels predicted as roads, Recall represents the ratio of pixels correctly predicted as roads to the total number of road pixels, F1 score represents the harmonic mean of precision and recall; IoU represents the ratio of the intersection and union of the two sets of predicted values ​​and true values.

[0073] Table 1 Statistics of extraction results of three existing methods and the method of the embodiment of the present invention

[0074]

[0075]

[0076] The model road extraction accuracy results in Table 1 show that the method described in the embodiment of the present invention achieves the highest accuracy except for the recall rate index. Figure 7 As shown, the method described in the embodiment of the present invention can achieve accurate and coherent extraction of road information under complex backgrounds, and can effectively reduce the phenomenon of wrong extraction and omission.

[0077] and Figure 1 Corresponding to the method, refer to Figure 8 The embodiment of the present invention provides a remote sensing image road extraction system based on a multi-scale difference feature fusion mechanism, comprising:

[0078] At least one processor:

[0079] at least one memory for storing at least one program;

[0080] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image road extraction method based on a multi-scale difference feature fusion mechanism, characterized in that: The method comprises the steps of: Step S1: Perform cropping preprocessing on the acquired remote sensing image data to obtain a training data set and a test data set; Step S2: constructing a remote sensing image road extraction network based on a multi-scale difference feature fusion mechanism, wherein the road extraction network adopts an encoding-decoding structure, including a decoder, an encoder, and a multi-scale difference feature fusion module; The encoder has five layers in total, the first layer is a convolutional downsampling module based on a convolutional neural network, the second to fifth layers all use multi-scale residual modules for feature extraction, and the second to fifth layers are stacked with 3, 4, 6 and 3 groups of multi-scale residual modules respectively; the decoder has five layers in total, each layer includes an upsampling module based on transposed convolution; The multi-scale difference feature fusion module is used to connect the encoder and the decoder, and is composed of three difference embedding modules and two fusion modules; Step S3: Use the training data set to train the remote sensing image road extraction network based on the multi-scale difference feature fusion mechanism, calculate the loss of the road segmentation prediction feature map and the real road label, perform back propagation through the obtained loss, optimize the model parameters, and complete the model training. Step S4: Use the trained road extraction model to extract road information in the remote sensing image to obtain a road extraction result.

2. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The multi-scale difference feature fusion module inputs four groups of encoding features with different depths, outputs three groups of intermediate transfer features with different difference fusion degrees, and performs feature addition operation on the output feature map and the decoding features at the same level.

3. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The input of the difference embedding module is divided into two parts, the first part is the intermediate transfer feature map of the current layer, i.e., the shallow feature map, and the second part is the intermediate transfer feature map of the adjacent deep layer, i.e., the deep feature map; the input of the fusion module is divided into two parts, the first part is the difference feature map of the current layer, i.e., the shallow difference feature map, and the second part is the difference feature map of the adjacent deep layer, i.e., the deep difference feature map.

4. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The difference embedding module includes a shallow feature extraction module, a deep feature extraction module and an extraction embedding module. The calculation formulas of the shallow feature extraction module, the deep feature extraction module and the extraction embedding module based on the channel attention mechanism are as follows: X1′=δ(W 12 σ(W 11 A(C5(X1)))) X2′=δ(W 22 σ(W 21 A(X2)))) P=|(X1′!X2′)| Among them, X1 and X2 are the shallow feature map and the deep feature map after upsampling, respectively, X1′ and X2′ are the channel descriptors of the corresponding feature maps, δ represents the Sigmoid function, σ represents the ReLU activation function, C5(·) is a 5×5 convolutional layer, A(·) is an average pooling layer, and W 11 and W 12 is the fully connected layer parameter of the shallow feature extraction module, W 21 and W 22 is the fully connected layer parameter of the deep feature extraction module, ! is the element-by-element subtraction operation, is the element-by-element multiplication operation, |·| is the absolute value calculation, P represents the difference weight between the shallow and deep feature maps, and the redundant overlapping information of the intermediate features is reduced by the feature subtraction operation. S is the difference feature map that integrates the difference information.

5. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The fusion module is built based on spatial attention. By fusing the difference feature maps across layers, the model is promoted to extract detailed road information in remote sensing images, extract spatial difference information, and model the dependency relationship between long-distance features. The calculation formula is as follows: S F =c(M(σ(C3(S U )))+S D ),A(σ(C3(S U )))+S D )) Among them, c(·) is the feature concatenation operation, C3(·) is the 3×3 convolution layer, M(·) is the maximum pooling layer, S D is the deep difference feature map after upsampling, S U is the shallow difference feature map, S F is the spatial difference feature descriptor, and K is the intermediate transfer feature that fuses the difference information.

6. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The multi-scale residual module in the encoder is constructed based on a convolutional neural network. The input feature map is split into 4 groups along the channel dimension, each group corresponds to a branch, the first branch is an identity mapping structure, the second branch extracts features through a convolutional layer, the third branch input and the second branch output are added and calculated, and then the features are extracted through a convolutional layer, the fourth branch input and the third branch output are added and calculated, and then the features are extracted through a convolutional layer, and the output feature maps of each branch are spliced ​​to obtain the output of the multi-scale residual module.

7. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: The decoder has five layers in total, each layer is a group of upsampling modules, and the upsampling modules include a convolution layer, a transposed convolution layer with a step size of 2, an activation function ReLU and a batch normalization BN layer.

8. The road extraction method based on multi-scale difference feature fusion mechanism according to claim 1 is characterized in that: AdamW is used to train and optimize the remote sensing road extraction network based on the multi-scale difference feature fusion mechanism, and the binary cross entropy loss function and the Dice coefficient loss function are combined to calculate the training loss.

9. A remote sensing road extraction system based on a multi-scale difference feature fusion mechanism, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.

Images