A method, system and storage medium for extracting roads from a remote sensing image

Abstract

The application provides a method and system for extracting a road from a remote sensing image and a storage medium, the method comprising: inputting original remote sensing image data to an encoder to extract feature information of the remote sensing image by the encoder; inputting an output of a second residual module to an inflation convolution module to perform inflation convolution processing, and inputting an output of the inflation convolution module to a decoder; wherein the encoder comprises a convolution pooling module, a plurality of residual modules and a down-sampling module; processing the output of the inflation convolution module through an up-sampling module, then through a plurality of up-sampling modules corresponding to the plurality of first residual modules, adding the output of each first residual module to the input of the corresponding up-sampling module in a manner of jump connection, and obtaining a binary graph of the road extracted from the original remote sensing image through a first deconvolution layer and a second convolution layer respectively from the output of the last up-sampling module. The application can more accurately extract road content from the original remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method and system for extracting a road from a remote sensing image and a storage medium. BACKGROUND

[0002] Remote Sensing Images (RSIs) refer to films or photos recording the sizes of electromagnetic waves of various ground objects, and are mainly divided into aerial photos and satellite photos. Extracting a road from Remote Sensing Images (RSIs) is one of the basic tasks of computer vision, and the application range of road extraction from Remote Sensing Images is very wide, including road map updating, digital mapping, geographic information updating, environmental assessment, disaster monitoring and assessment and response, etc. However, two characteristics of Remote Sensing Images make this task extremely challenging. First, the roads with long-distance strip structure in high-resolution Remote Sensing Images are often blocked or truncated by other objects, and the directions of the roads in the images are often discretely distributed. Second, the span of the length and width of the roads in Remote Sensing Images is usually very different - the width may cover only 2-3 pixels, while the length may span the entire image. In this case, it is very important for a model to capture the multi-scale spatial context of the image.

[0003] In recent years, many deep learning methods based on Convolutional Neural Networks (CNN) have been successfully applied to semantic segmentation. Among them, LinkNet and D-LinkNet are both convolutional neural networks that can be used for road extraction. However, the convolutional neural networks designed by both LinkNet and D-LinkNet have the following problems: on the one hand, the range of the receptive field is limited, and on the other hand, the ability to capture anisotropic long-range information is not flexible enough, and in addition, it is inevitable to include interference information from irrelevant areas, thus leading to poor accuracy of extracting roads from Remote Sensing Images. Due to the above two characteristics of Remote Sensing Images, the shortcomings of the existing convolutional neural networks for road extraction from Remote Sensing Images will be particularly evident.

[0004] Therefore, how to provide a more accurate method for extracting a road from a Remote Sensing Image is a technical problem to be solved. SUMMARY

[0005] In view of this, the embodiments of the present application provide a method and system for extracting a road from a Remote Sensing Image and a storage medium to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a method for extracting a road from a Remote Sensing Image, the method comprising the following steps:

[0007] The raw remote sensing image data is input into an encoder, which extracts feature information from the remote sensing image. The encoder includes a convolutional pooling module, multiple residual modules, and multiple downsampling modules arranged alternately between and connected to the residual modules. The convolutional pooling module includes a first convolutional layer and a max-pooling layer. The multiple residual modules include multiple first residual modules and one second residual module. After initial feature extraction and initial downsampling by the convolutional pooling module, the raw remote sensing image data sequentially passes through the alternately connected first residual modules and downsampling modules, and is output via the second residual module.

[0008] The output of the second residual module is input to the dilated convolution module for dilated convolution processing, and the output of the dilated convolution module is input to the decoder; wherein, the dilated convolution module includes multiple dilated convolution layers with gradually increasing dilation rates; the decoder includes multiple upsampling modules corresponding to the multiple residual modules, a first deconvolution layer and a second convolution layer;

[0009] The output of the dilated convolution module is processed by an upsampling module, and then sequentially passed through multiple upsampling modules corresponding to multiple first residual modules. The outputs of each first residual module in the encoder are added to the input of the corresponding upsampling module in the decoder through a skip connection. The output of the last upsampling module is then passed through the first deconvolution layer and the second convolution layer to obtain a binary map of the road extracted from the original remote sensing image.

[0010] In some embodiments of the present invention, each first residual module includes a different number of basic residual blocks and a multi-scale strip pooling squeeze excitation residual block, and the second residual module includes multiple multi-scale strip pooling squeeze excitation residual blocks. The multi-scale strip pooling squeeze excitation residual block adds a multi-scale strip pooling module and a squeeze excitation module to the basic residual block structure. The multi-scale strip pooling module includes multiple strip pooling blocks of different widths.

[0011] The input of the multi-scale bar pooling module first passes through multiple bar pooling blocks of different widths to extract multi-scale feature information. Then, it is processed by the average pooling, convolution operation, normalization processing, dimension expansion and interpolation operation of the multi-scale bar pooling module. The output of the multi-scale bar pooling module is then input into the squeezing excitation module.

[0012] The input to the extrusion excitation module is processed through extrusion operation, excitation processing, and pixel-level multiplication operation.

[0013] In some embodiments of the present invention, the input of the multi-scale strip pooling module first passes through multiple strip pooling blocks of different widths to extract multi-scale feature information, and then is processed by the average pooling, convolution operation, normalization processing, dimension expansion and interpolation operation of the multi-scale strip pooling module. The process includes: performing average pooling operation on the input feature information through multiple strip pooling blocks of different widths contained in the multi-scale strip pooling module to extract multi-scale feature information; the feature information after average pooling operation of each width of strip pooling block is processed by convolution operation, normalization processing, dimension expansion, nearest neighbor interpolation operation and pixel-level addition processing; and then processing the pixel-level addition processing feature information by the activation function ReLU, a convolutional layer with a kernel size of 1×1 and the activation function Sigmoid to obtain a feature map; performing pixel-level multiplication processing on the feature map and the feature information input to the multi-scale strip pooling module; and inputting the pixel-level multiplied feature information into the squeezing excitation module.

[0014] In some embodiments of the present invention, the input of the squeeze excitation module undergoes a squeezing operation, excitation processing, and pixel-level multiplication operation, including: the squeeze excitation module squeezes the size of each channel of the feature information of the original remote sensing image extracted by the multi-scale strip pooling module to a length of 1 and a width of 1 while keeping the number of channels unchanged through a global average pooling step; the channel is then excited by passing through a first fully connected layer, an activation function ReLU, a second fully connected layer, and an activation function Sigmoid; and the excitation result is multiplied with the feature information input to the squeeze excitation module at the pixel level to obtain the output of the squeeze excitation module.

[0015] In some embodiments of the present invention, the basic residual block comprises two convolutional layers connected end to end, and the multi-scale strip pooling squeeze excitation residual block comprises two convolutional layers and a multi-scale strip pooling module and a squeeze excitation module connected sequentially between the two convolutional layers.

[0016] In some embodiments of the present invention, the dilated convolution module includes multiple dilated convolution layers connected end-to-end with increasing dilation rates; the step of inputting the output of the second residual module into the dilated convolution module for dilated convolution processing includes: passing the feature information after residual learning by the second residual module through the dilated convolution layers in an end-to-end order, and adding the feature information processed by all the dilated convolution layers in the dilated convolution module consecutively with the feature information processed by each dilated convolution layer in the dilated convolution module to obtain the feature information output by the dilated convolution module.

[0017] In some embodiments of the present invention, the pooling window size of the max pooling layer is 3×3 and the stride is 2; the downsampling module includes a convolutional layer with a kernel size of 1×1 and a stride of 2; the upsampling module includes two convolutional layers with a kernel size of 1×1 and a deconvolutional layer with a kernel size of 3×3 and a stride of 2 located in the middle.

[0018] In some embodiments of the present invention, the binary image of the road extracted from the original remote sensing image is an image in which each pixel has only two possible color values ​​or two possible gray levels; the kernel size of the first convolutional layer is 7×7 and the stride is 2; the kernel size of the first deconvolutional layer is 4×4 and the stride is 2; the kernel size of the second convolutional layer is 3×3 and the stride is 3.

[0019] Another aspect of the present invention provides a system for extracting roads from remote sensing images, including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system performing the steps of the method as described in any of the above embodiments.

[0020] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0021] The method, system, and storage medium for extracting roads from remote sensing images provided by this invention design an encoder comprising alternating residual modules and downsampling modules, and employs a first residual module and a second residual module for collaborative processing to extract feature information from the original remote sensing image. The encoder output passes through a dilated convolutional layer with an increasing dilation rate in the dilated convolution module to increase the receptive field of the convolution operation. The feature information passed through the dilated convolution module is converted into a binary map of the road extracted from the original remote sensing image by the decoder. The design of the first residual module, the second residual module, and the dilated convolution module improves the feature extraction capability of the original remote sensing image, and enhances the ability to extract roads from the original remote sensing image without increasing computational complexity.

[0022] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0023] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0025] Figure 1 This is a flowchart of a method for extracting roads from remote sensing images according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of a method for extracting roads from remote sensing images in another embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the downsampling module in one embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the upsampling module in one embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the basic residual block ResBlock in one embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of a multi-scale strip-shaped pooling extrusion excitation residual block in one embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of a multi-scale strip pooling module and a squeezing excitation module in one embodiment of the present invention.

[0032] Figure 8 This is a schematic diagram of a deformable strip-shaped pooling block in one embodiment of the present invention.

[0033] Figure 9 This is a schematic diagram of strip-shaped pooling blocks of different sizes.

[0034] Figure 10 This is a schematic diagram of a dilated convolution module in one embodiment of the present invention.

[0035] Figure 11 This is a comparison chart showing the results of extracting roads from remote sensing images using the present invention and the existing technology D-LinkNet. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0037] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0038] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0039] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0040] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0041] As mentioned in the background section, some existing road extraction methods are mainly based on encoder-decoder structures. They construct convolutional neural networks by stacking standard square convolutional kernels and pooling kernels to extract and model local information from images, thereby achieving road extraction from remote sensing images. However, due to the characteristics of roads being connected, long and narrow, and having discrete orientations, simply using standard square convolutional kernels and pooling kernels is far from sufficient for extracting and modeling local image information, which makes the methods for road extraction from remote sensing images impractical.

[0042] To address the shortcomings of existing neural networks for extracting roads from remote sensing images, this invention provides a method, system, and storage medium for extracting roads from remote sensing images. This method improves the encoder-decoder structure by using a first convolutional layer and a max-pooling layer for initial processing in the encoder, followed by processing through alternately connected residual blocks and downsampling modules. Combined with the processing of multiple upsampling modules in the decoder structure, roads are gradually extracted from the original remote sensing image.

[0043] Figure 1 This is a flowchart of a method for extracting roads from remote sensing images according to an embodiment of the present invention. The method includes the following steps:

[0044] Step S110: Input the original remote sensing image data into the encoder, and the encoder extracts the feature information of the remote sensing image; wherein, the encoder includes a convolutional pooling module, multiple residual modules, and multiple downsampling modules placed between the multiple residual modules and alternately connected with the residual modules. The convolutional pooling module includes a first convolutional layer and a max pooling layer. The multiple residual modules include multiple first residual modules and a second residual module. After the original remote sensing image data is initially extracted and initially downsampled by the convolutional pooling module, it is sequentially passed through the alternately connected first residual module and downsampling module, and then output through the second residual module.

[0045] Max pooling (layer) is a downsampling method. Pooling layers can effectively reduce the size of the parameter matrix, thereby reducing the number of parameters in the final connected layers. Adding pooling layers can speed up computation and prevent overfitting. The downsampling module uses a 1×1 convolutional kernel with a stride of 2 for downsampling. Each first residual module contains a different number of basic residual blocks and a multi-scale strip pooling squeezed excitation residual block. The second residual module contains multiple multi-scale strip pooling squeezed excitation residual blocks. The multi-scale strip pooling squeezed excitation residual block adds a multi-scale strip pooling module and a squeezed excitation module to the basic residual block structure. The multi-scale strip pooling module contains multiple strip pooling blocks of different widths. The input of the multi-scale strip pooling module first passes through the multiple strip pooling blocks of different widths to extract multi-scale feature information. Then, it is processed by the multi-scale strip pooling module's average pooling, convolution operation, normalization, dimension expansion, and interpolation operations. The output of the multi-scale strip pooling module is then input to the squeezed excitation module. The input to the extrusion excitation module is processed through extrusion operation, excitation processing, and pixel-level multiplication operation.

[0046] The input to the multi-scale bar pooling module first passes through multiple bar pooling blocks of different widths to extract multi-scale feature information. Then, it undergoes average pooling, convolution, normalization, dimensionality expansion, and interpolation operations within the multi-scale bar pooling module. Specifically, the input feature information is average-pooled through multiple bar pooling blocks of different widths to extract multi-scale feature information. The feature information after average pooling of each width's bar pooling blocks undergoes convolution, normalization, dimensionality expansion, nearest neighbor interpolation, and pixel-level addition. The pixel-level addition is then processed sequentially using the ReLU activation function, a 1×1 convolutional layer, and the Sigmoid activation function to obtain a feature map. This feature map is then multiplied at the pixel level with the feature information input to the multi-scale bar pooling module. Finally, the pixel-level multiplied feature information is input to the squeezing excitation module.

[0047] The input to the squeeze excitation module undergoes squeezing, excitation processing, and pixel-level multiplication. The squeezing excitation module uses a global average pooling step to squeeze the size of each channel of the feature information of the original remote sensing image extracted by the multi-scale strip pooling module to a length of 1 and a width of 1 while keeping the number of channels unchanged. The excitation is then performed through a first fully connected layer, a ReLU activation function, a second fully connected layer, and a gating mechanism with sigmoid activation (i.e., an excitation module containing a sigmoid activation function). The excitation result is then multiplied pixel-level with the feature information input to the squeezing excitation module to obtain the output of the squeezing excitation module.

[0048] The basic residual block ResBlock contains two convolutional layers connected end-to-end, with a kernel size of 3×3. The multi-scale strip pooling squeeze excitation residual block contains two convolutional layers and a multi-scale strip pooling module and a squeeze excitation module connected sequentially between the two convolutional layers.

[0049] Step S120: Input the output of the second residual module into the dilated convolution module for dilated convolution processing, and input the output of the dilated convolution module into the decoder; wherein, the dilated convolution module includes multiple dilated convolution layers with gradually increasing dilation rates; the decoder includes multiple upsampling modules corresponding to the multiple residual modules, a first deconvolution layer and a second convolution layer.

[0050] The dilated convolution module contains multiple concatenated dilated convolutional layers with increasing dilation rates. Dilated / atrous convolution is a convolutional operation widely used in tasks such as semantic segmentation and object detection. It can expand the receptive field without reducing the feature map resolution; it is also called dilated convolution or atrous convolution. Dilated convolution increases the receptive field of the convolutional operation. The D-block module further processes image features, thereby improving the model's ability to extract features from the input image. The output of the D-block is input to the decoder.

[0051] The output of the second residual module is input into the dilated convolution module for dilated convolution processing, including: the feature information after residual learning by the second residual module is sequentially passed through dilated convolution layers with a kernel size of 3×3 and an increasing dilation rate, which are connected end to end in the dilated convolution layer; the feature information processed by all dilated convolution layers in the dilated convolution module is added to the feature information processed by each dilated convolution layer in the dilated convolution module to obtain the feature information output by the dilated convolution module.

[0052] Step S130: The output of the dilated convolution module is processed by an upsampling module, and then sequentially processed by multiple upsampling modules corresponding to multiple first residual modules. The outputs of each first residual module in the encoder are added to the input of the corresponding upsampling module in the decoder through a skip connection. The output of the last upsampling module is processed by the first deconvolution layer and the second convolution layer to obtain the binary map of the road extracted from the original remote sensing image.

[0053] The role of skip connections is to fuse feature information of the corresponding size during the encoder downsampling process during the decoder's upsampling process, so as to obtain more accurate semantic segmentation results.

[0054] Among them, the binary map of the road extracted from the original remote sensing image is an image in which each pixel has only two possible color values ​​or two possible gray levels.

[0055] In one embodiment of the present invention, the max pooling layer has a pooling window size of 3×3 and a stride of 2; the downsampling module includes a convolutional layer with a kernel size of 1×1 and a stride of 2, and the upsampling module includes two convolutional layers with a kernel size of 1×1 and a deconvolutional layer with a kernel size of 3×3 and a stride of 2 in the middle. The downsampling module differs from the max pooling layer in that it uses a convolutional layer with a kernel size of 1×1 and a stride of 2 for downsampling, progressively compressing the feature map size. This allows for flexible downsampling of image pixels and the compression and expansion of channel dimensions. In some other embodiments of the present invention, a padding operation is also included after the max pooling layer.

[0056] In one embodiment of the present invention, the kernel size of the first convolutional layer is 7×7 and the stride is 2, the kernel size of the first deconvolutional layer is 4×4 and the stride is 2, the kernel size of the second convolutional layer is 3×3 and the stride is 3, and the padding is 1.

[0057] In step S110 of one embodiment of the present invention, feature information is initially extracted from the original remote sensing image through the first convolutional layer and max pooling layer included in the encoder. The initially extracted feature information is processed by a predetermined number of alternately stacked first residual modules and downsampling modules included in the encoder. The feature information processed by each first residual module is output to the upsampling module in the decoder corresponding to each first residual module through a skip connection. The second residual module included in the encoder performs residual learning on the feature information processed by the multiple alternately stacked first residual modules and multiple downsampling modules, and the feature information after residual learning by the second residual module is output to the encoder. The codec and decoder are connected by a dilated convolution module. The first residual module contains multiple basic residual blocks and a multi-scale strip pooling squeeze-excitation residual block. The second residual module contains a preset number of multi-scale strip pooling squeeze-excitation residual blocks. These multi-scale strip pooling squeeze-excitation residual blocks add a multi-scale strip pooling module and a squeeze-excitation module to the basic residual block structure. The multi-scale strip pooling module contains multiple strip pooling blocks of different widths used for dimensional expansion and interpolation operations on the input feature information. The squeeze-excitation module is used for squeezing, excitation, and pixel-level multiplication operations on the feature information processed by the multi-scale strip pooling module.

[0058] The remote sensing image road extraction method proposed in this invention effectively captures anisotropic remote contextual information, addressing the characteristics of roads in remote sensing images, such as long-distance strip structures, susceptibility to occlusion and truncation by other objects, and discrete distribution of road directions within the image. This makes it better suited for remote sensing image road extraction tasks. The SE-MSPDLinkNet neural network involved in this method includes an encoder, a D-block, and a decoder. The original remote sensing image is first input into the encoder. The encoder primarily extracts image features through multiple residual modules and multiple downsampling operations, and its output contains the feature information of the input image. The encoder output is then input into the D-block module. The D-block module contains multiple dilated convolutional layers with gradually increasing dilation rates. Dilated convolution increases the receptive field of the convolution operation, and the D-block module further processes the image features, thereby improving the model's feature extraction capability. The D-block output is then input into the decoder. The decoder primarily restores the input features through multiple upsampling operations; simultaneously, the outputs of the residual modules in the encoder corresponding to the decoder positions are added to the input of the corresponding upsampling operations of the decoder through skip connections. After multiple upsampling operations, a deconvolutional layer and a convolutional layer are passed through to obtain the road extraction binary map of the original input image.

[0059] Specifically, this invention proposes a neural network model for road extraction, SE-MSPDLinkNet, whose encoder possesses stronger representational capabilities to capture both long-range and local contextual information. The SE-MSPDLinkNet neural network model includes a basic residual block (Res-block) and a multi-scale strip pooling squeezed excitation residual block (Res-MSPSE-block). This multi-scale strip pooling squeezed excitation residual block employs a multi-scale strip pooling module (MSPM or MSP), which uses deformable multi-scale strip pooling kernels to obtain anisotropic long-range contextual information while still capturing local context. Furthermore, an attention mechanism based on a squeezed excitation module (SE) is introduced, which adaptively recalibrates channel feature responses by explicitly modeling the interdependencies between channels. The squeezed excitation module significantly improves the performance of the neural network CNN with a slight increase in computational cost.

[0060] Figure 2 The following is a flowchart of a method for extracting roads from remote sensing images in another embodiment of the present invention. The input original remote sensing image passes through an encoder, a dilation block (D-block), and a decoder in sequence, as follows:

[0061] (1) First, the original remote sensing image is input into the encoder. The original remote sensing image first passes through a convolutional layer (with a kernel size of 7×7 and a stride of 2) and a max pooling layer (with a kernel size of 3×3 and a stride of 2). The original remote sensing image has the dimensions H×W×C, where H is the height along the vertical direction, W is the width along the horizontal direction, and C is the number of channels.

[0062] (2) The output of the max pooling layer passes through four alternately connected residual modules and three downsampling modules. After passing through multiple residual modules and multiple downsampling operations in the encoder, the image's feature information is gradually extracted. The encoder's output contains the feature information of the input image. The downsampling module's structure includes a convolutional layer (kernel size 1×1, stride 2), as shown below. Figure 3 As shown, Figure 3 This is a schematic diagram of the downsampling module in one embodiment of the present invention.

[0063] The four residual modules include three first residual modules and one second residual module. The first three residual modules are the first residual modules. In this embodiment, the four residual modules are composed of different numbers (2, 3, 5, and 0) of basic residual blocks (ResBlock) and multi-scale strip pooling squeezed excitation residual blocks. The last residual block in the first residual module is a multi-scale strip pooling squeezed excitation residual block, and the remaining residual blocks are basic residual blocks (ResBlock). The second residual module contains only multi-scale strip pooling squeezed excitation residual blocks. Figure 5 and Figure 6 As shown, where, Figure 5 This is a schematic diagram of a basic residual block ResBlock in one embodiment of the present invention. Figure 6 This is a schematic diagram of a multi-scale strip pooling squeeze excitation residual block in one embodiment of the present invention. x is the input of the residual block, and the function F1(x)+x represents the output of the residual block. The function F1(x) represents the process of the input x of the residual block passing through a convolutional layer Conv (3×3), a multi-scale strip pooling module, a squeeze excitation module, and another convolutional layer Conv (3×3). The symbol Conv represents convolution.

[0064] In one embodiment of the present invention, the structure of the multi-scale strip pooling extrusion excitation residual block is based on the structure of the basic residual block ResBlock, with the addition of a multi-scale strip pooling module and an extrusion excitation module, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of a multi-scale strip pooling module and a squeezing excitation module in one embodiment of the present invention. The structure of the multi-scale strip pooling module includes: deformable strip pooling blocks of different widths. Figure 7 The K value in the equation represents the width of the strip pooling block. The feature maps extracted from strip pooling blocks of different scales undergo pixel-level addition. Then, a weight map is generated by passing the weight map through a 1×1 convolutional layer with ReLU activation, sigmoid activation (also known as a S-type activation gate mechanism). The weight map is then multiplied pixel-level with the features input to the multi-scale strip pooling module and output to the squeezing activation module. The specific process is shown in the following equation, where z is the output of the multi-scale strip pooling module, S is pixel-level multiplication, σ is the sigmoid activation function, φ is a 1×1 convolution, ReLU is the activation function, and y1, y2, and y3 are the feature maps obtained after the input feature map is extracted by strip pooling blocks of different scales. The formula is expressed as follows:

[0065] z=S(x,σ(φ(ReLU(y1+y2+y3)));

[0066] ReLU(x) = max(0,x);

[0067] The purpose of using pixel-level multiplication is to maintain the consistency of feature information or feature maps.

[0068] The structure of the squeeze excitation module includes: compressing the feature map of size H×W for each channel to the average value of all pixels in the feature map through global average pooling to obtain a 1×1×C feature vector (C is the number of channels), and then performing excitation processing through a fully connected layer, the ReLU activation function, a fully connected layer and the Sigmoid activation function (the two fully connected layers have the same structure but are not the same), and then performing pixel-level multiplication with the feature input to the squeeze excitation module to obtain the output of the squeeze excitation module.

[0069] Figure 8 This is a schematic diagram of a deformable strip pooling block in one embodiment of the present invention. Its structure includes: a pair of strip pooling blocks, comprising a horizontally oriented strip pooling block of length W and width K (K×W) and a vertically oriented strip pooling block of length H and width K (H×K), where K can be 1, 2, or 3 (here K is set to 2). This module facilitates road feature extraction. Road features are extracted using average pooling and "convolution + normalization (Batch Normalization, BN)," and then the result is expanded to H×W through dimensionality expansion and interpolation, so that the expanded feature map has the same size as the feature map input to the deformable strip pooling block. Figure 8 As shown, the deformable strip pooling block utilizes nearest-neighbor interpolation and pixel-level addition of the results from two H×W feature maps. Existing pooling blocks used for average pooling are square (i.e., equal in length and width), while this invention employs strip pooling blocks with varying lengths and widths, performing average pooling separately in the horizontal and vertical directions. These are then processed through convolution, normalization, dimension expansion, and interpolation. Finally, the results in the horizontal and vertical directions are added pixel-level to obtain the output of the multi-scale strip pooling module. In some embodiments of this invention, M = K (K = 1, 2, 3…), where M represents the “width” of the deformable strip pooling block used.

[0070] Figure 9 The diagram illustrates strip pooling blocks of different scales. K=1 indicates average pooling using 1×W and H×1. After pooling, H=7 becomes H=6, therefore nearest neighbor interpolation is needed to restore the feature map size at the input. The cases for K=2 and K=3 follow the same logic. Figure 9As shown, after processing by the convolutional layer, the data undergoes normalization, dimensionality expansion, and nearest neighbor interpolation. Specifically, 2×W is equivalent to sequentially merging rows 1-2, 2-3, 3-4…6-7 together, and then… Figure 7 The step shown involves performing pixel-level addition on the results of processing strip pooling blocks of different scales, which better preserves the contextual relationships, thereby enabling the second residual module based on strip pooling blocks of different scales to extract better contextual relationships.

[0071] (3) Input the output of the second residual module in the encoder into the dilated convolution module (D-block), and then input the output of the dilated convolution module into the decoder; wherein, the dilated convolution module contains multiple dilated convolution layers with gradually increasing dilation rates. Dilated convolution can increase the receptive field of the convolution operation. The D-block module further processes the image features, thereby improving the model's ability to extract features from the input image.

[0072] In this embodiment of the invention, there are four dilated convolutional layers with 3×3 kernels connected end to end and increasing dilation rates. The output of the last residual module of the encoder is the input of the D-block.

[0073] like Figure 10 As shown, Figure 10 This is a schematic diagram of a dilated convolution module in one embodiment of the present invention. The dilation rates of the four dilated convolution layers in this embodiment are 1, 2, 4, and 8, respectively, exhibiting an increasing order, while the size remains constant.

[0074] (4) In the decoder, multiple upsampling operations are required to restore the input features. On the one hand, the output of the dilated convolution module is input into the first upsampling module. On the other hand, the output of the first residual module in the encoder, which corresponds to the upsampling module in the decoder, is input into the upsampling module through a skip connection. The features that have passed through four consecutive dilated convolution layers are added to the features after each dilated convolution layer to obtain the output of the D-block.

[0075] The decoder structure includes four interconnected upsampling modules, one deconvolutional layer, and one convolutional layer. The output of the D-block serves as the input to the decoder, which is fed into the first upsampling module.

[0076] Each upsampling module consists of two 1×1 convolutional layers and an intermediate deconvolutional layer with a kernel size of 3×3 and a stride of 2, as shown below. Figure 4 As shown, Figure 4This is a schematic diagram of an upsampling module in one embodiment of the present invention. At each upsampling module position, the output of the residual module at each corresponding position in the encoder is added to the output of each upsampling module via a skip connection and then input into the next upsampling module.

[0077] (5) After multiple upsampling operations in the decoder stage, the original input image's road extraction binary map is obtained through the first deconvolution layer and the second convolution layer. The first deconvolution layer has a 4×4 kernel size and a stride of 2, while the second convolution layer has a 3×3 kernel size. A binary map refers to an image where each pixel has only two possible color values ​​or two possible grayscale levels. Taking a black and white binary map as an example, for a road binary map, pixels belonging to the road are white, and the remaining pixels are black as the background. Figure 11 As shown.

[0078] Figure 11 This is a comparison of the results of extracting roads from remote sensing images using the present invention and the existing technology D-LinkNet. The leftmost image shows the original remote sensing images of four examples, the middle image shows the binary image extracted using the existing technology D-LinkNet, the last image shows the binary image extracted using the method proposed in this invention, and the remaining images are the results extracted by similar models designed in the middle. The area circled by the square in the original remote sensing image was only identified by the preferred embodiment of this invention, SE-MSPDLinkNet.

[0079] Table 1. Comparison of computational costs between an embodiment of the present invention and the prior art D-LinkNet.

[0080]

[0081] Table 1 is a comparison chart of the computational costs of an embodiment of the present invention and the prior art D-LinkNet. Based on the prior art D-LinkNet and the present invention, the total number of parameters, computational load and inference time are very small when the same task is completed under the same conditions. In layman's terms, the additional computational cost added by the present invention and the prior art is not high.

[0082] Combination Figure 11 As shown in Table 1, the road binary map extracted by the method proposed in this invention has more road details than the existing D-LinkNet. The method proposed in this invention can achieve higher accuracy than the existing D-LinkNet. The SE-MSPDLinkNet neural network proposed in this invention only adds a small amount of additional computational cost compared to the existing D-LinkNet, but can significantly improve the ability to extract roads from the original remote sensing images.

[0083] The method, system, and storage medium for extracting roads from remote sensing images provided by this invention design an encoder comprising alternating residual modules and downsampling modules, and employs a first residual module and a second residual module for collaborative processing to extract feature information from the original remote sensing image. The encoder output passes through a dilated convolutional layer with an increasing dilation rate in the dilated convolution module to increase the receptive field of the convolution operation. The feature information passed through the dilated convolution module is converted into a binary map of the road extracted from the original remote sensing image by the decoder. The design of the first residual module and the second residual module improves the feature extraction capability of the original remote sensing image, and improves the ability to extract roads from the original remote sensing image without increasing computational complexity.

[0084] Furthermore, this invention includes a designed multi-scale strip pooling squeeze excitation residual block. This block employs a multi-scale strip pooling module, which uses deformable multi-scale strip pooling kernels to capture anisotropic remote contextual information. This better adapts to the discrete distribution of road directions in remote sensing images during road extraction, while still capturing local context. Semantic segmentation of the remote sensing image is performed based on the extracted local and remote information. Additionally, an attention mechanism based on the squeeze excitation module is introduced. By explicitly modeling the interdependencies between channels, the channel feature responses are adaptively recalibrated, enhancing channel feature representation. The squeeze excitation module significantly improves the performance of the neural network (CNN) with a slight increase in computational cost, thereby extracting road binary maps more accurately from the original remote sensing image.

[0085] Corresponding to the above method, the present invention also provides a system for extracting roads from remote sensing images. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system performs the steps of the method described above.

[0086] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the method as described above. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0087] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0088] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0089] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting roads from remote sensing images, characterized in that, The method includes the following steps: The raw remote sensing image data is input into an encoder, which extracts feature information from the remote sensing image. The encoder includes a convolutional pooling module, multiple residual modules, and multiple downsampling modules arranged alternately between and connected to the residual modules. The convolutional pooling module includes a first convolutional layer and a max-pooling layer. The multiple residual modules include multiple first residual modules and one second residual module. After initial feature extraction and initial downsampling by the convolutional pooling module, the raw remote sensing image data sequentially passes through the alternately connected first residual modules and downsampling modules, and is output via the second residual module. The output of the second residual module is input to the dilated convolution module for dilated convolution processing, and the output of the dilated convolution module is input to the decoder; wherein, the dilated convolution module includes multiple dilated convolution layers with gradually increasing dilation rates; the decoder includes multiple upsampling modules corresponding to the multiple residual modules, a first deconvolution layer and a second convolution layer; The output of the dilated convolution module is processed by an upsampling module, and then sequentially processed by multiple upsampling modules corresponding to multiple first residual modules. The outputs of each first residual module in the encoder are added to the input of the corresponding upsampling module in the decoder through a skip connection. The output of the last upsampling module is then processed by the first deconvolution layer and the second convolution layer to obtain the binary map of the road extracted from the original remote sensing image. Each first residual module contains a different number of basic residual blocks and a multi-scale strip pooling squeeze excitation residual block. The second residual module contains multiple multi-scale strip pooling squeeze excitation residual blocks. The multi-scale strip pooling squeeze excitation residual block adds a multi-scale strip pooling module and a squeeze excitation module to the basic residual block structure. The multi-scale strip pooling module contains multiple strip pooling blocks of different widths. The input of the multi-scale bar pooling module first passes through multiple bar pooling blocks of different widths to extract multi-scale feature information. Then, it is processed by the average pooling, convolution operation, normalization processing, dimension expansion and interpolation operation of the multi-scale bar pooling module. The output of the multi-scale bar pooling module is then input into the squeezing excitation module. The input to the extrusion excitation module is processed through extrusion operation, excitation processing, and pixel-level multiplication operation.

2. The method according to claim 1, characterized in that, The input to the multi-scale bar pooling module first passes through multiple bar pooling blocks of different widths to extract multi-scale feature information, and then undergoes processing including average pooling, convolution, normalization, dimensionality expansion, and interpolation operations. The multi-scale bar pooling module uses multiple bar pooling blocks of varying widths to perform average pooling on the input feature information to extract multi-scale feature information. The feature information after average pooling of each bar pooling block undergoes convolution, normalization, dimensionality expansion, nearest neighbor interpolation, and pixel-level addition. Finally, it is processed by the ReLU activation function and a convolution kernel size of [missing value]. The convolutional layer and the sigmoid activation function process the pixel-level additive feature information to obtain a feature map. The feature map is then multiplied at the pixel level with the feature information input to the multi-scale strip pooling module. The feature information after pixel-level multiplication is then input to the squeezing excitation module.

3. The method according to claim 1, characterized in that, The input to the extrusion module undergoes extrusion operation, extrusion processing, and pixel-level multiplication operation, including: The squeezing excitation module squeezes the size of each channel of the feature information of the original remote sensing image extracted by the multi-scale strip pooling module to a length of 1 and a width of 1 while keeping the number of channels unchanged. The excitation is performed by passing the first fully connected layer, the ReLU activation function, the second fully connected layer, and the Sigmoid activation function. The excitation result is then multiplied at the pixel level with the feature information input to the squeezing excitation module to obtain the output of the squeezing excitation module.

4. The method according to claim 1, characterized in that, The basic residual block contains two convolutional layers connected end-to-end, and the multi-scale strip pooling squeeze excitation residual block contains two convolutional layers and a multi-scale strip pooling module and a squeeze excitation module connected sequentially between the two convolutional layers.

5. The method according to claim 1, characterized in that, The dilated convolution module comprises multiple dilated convolution layers connected end-to-end with increasing dilation rates. The step of inputting the output of the second residual module into the dilated convolution module for dilated convolution processing includes: The feature information after residual learning by the second residual module is passed through the dilated convolutional layer in a head-to-tail order. The feature information processed by all the dilated convolutional layers in the dilated convolutional module is added to the feature information processed by each dilated convolutional layer in the dilated convolutional module to obtain the feature information output by the dilated convolutional module.

6. The method according to claim 1, characterized in that, The pooling window size of the maximum pooling layer is: And the step size is 2; The downsampling module includes a convolutional kernel with a size of [missing information]. And a convolutional layer with a stride of 2; The upsampling module includes two convolutional kernels with a size of [missing information]. The size of the convolutional layer and the kernel located in the middle is... And a deconvolutional layer with a stride of 2.

7. The method according to claim 1, characterized in that, The binary image of the road extracted from the original remote sensing image is an image in which each pixel has only two possible color values ​​or two possible gray levels; The kernel size of the first convolutional layer is And the step size is 2; The kernel size of the first deconvolution layer is And the step size is 2; The kernel size of the second convolutional layer is And the step size is 3.

8. A system for extracting roads from remotely sensed images, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images