Building extraction method from remote sensing images based on position attention and multi-scale residual features

Abstract

The present invention discloses a remote sensing image building extraction method based on position attention and multi-scale residual features, and relates to the technical field of building extraction. The present invention comprises the following steps: Step 1: preprocessing a remote sensing image dataset, obtaining a remote sensing image dataset, making a remote sensing image sample label according to the obtained remote sensing image dataset, and performing a data enhancement operation of vertically flipping the RGB remote sensing image dataset and the corresponding remote sensing image single-channel digital label respectively. The present invention uses skip connections and deep supervision design in a feature decoding module constructed by a remote sensing image building extraction model FCM‑Net. The skip connection combines high-level semantics and low-level semantics from feature maps of different scales to reduce the semantic gap between the encoder and the decoder; the deep supervision design is to use each output block (X 0,1 , X 0,2 , X 0,3 He He X 0,4 ) is added after the 1×1 convolution and FPReLu activation function to alleviate the gradient vanishing problem and speed up the convergence.

Description

Technical Field

[0001] The present invention belongs to the field of building extraction methods, and in particular, relates to a remote sensing image building extraction method based on position attention and multi-scale residual features. Background Art

[0002] With the rapid development of remote sensing technology and its wide application in fields such as earth observation and resource management, remote sensing images are becoming increasingly important in the field of geographic information science. Segmentation methods using artificial neural network algorithms have become a research hotspot in the field of satellite remote sensing image segmentation. As the main ground object in remote sensing images, accurate segmentation of buildings is of great significance in urban planning, environmental monitoring, and disaster assessment.

[0003] At present, the methods for extracting buildings from remote sensing images are mainly divided into traditional image processing methods and extraction methods based on deep learning. Traditional image processing methods for extracting remote sensing buildings mainly rely on the basic characteristics of the image, such as color, texture, etc. This type of method extracts buildings from remote sensing images by designing a suitable classifier as the core idea. Common methods include extraction methods based on texture and shape features and the use of an enhanced morphological building index and geometric shape constraints of the substrate to complete the automatic extraction of buildings from remote sensing images. However, due to the small amount of calculation, this type of method only has good results on simple and regular remote sensing images. For complex and irregular images, it has prominent limitations and is difficult to become the mainstream of remote sensing image building extraction methods.

[0004] In recent years, deep learning algorithms have been widely used in the field of imaging. The remote sensing image building extraction method based on deep learning can learn the characteristics of buildings in remote sensing images to achieve fast and effective pixel-level segmentation and extraction of buildings in remote sensing images. Compared with traditional image processing methods, the accuracy is greatly improved, and high-precision segmentation of remote sensing image buildings is achieved, overcoming the problem that traditional image processing methods cannot completely extract irregular buildings in complex remote sensing images.

[0005] Chinese patent document number CN202310811719.2, published on November 14, 2024, records a high-resolution remote sensing image building semantic segmentation method and device, the method comprising: preprocessing the remote sensing image to obtain the target image; inputting the target image into the segmentation model to obtain the building segmentation result output by the segmentation model. The high-resolution remote sensing image building semantic segmentation method and device provided by the present invention extracts a refined feature map and a middle-level feature map that fuses multiple scale features through a first feature extraction layer with a pyramid structure, performs convolution operations of different scales in parallel through the size-reduced refined feature map, fuses the features unified into the same scale into a deep feature map, and then restores the building segmentation result by fusing the feature maps of the three semantic levels. It can increase the receptive field of the convolution layer by layer without losing information, so that the output of each convolution contains a larger range of semantic information, improve the fineness of image semantic segmentation, and accurately identify buildings.

[0006] Chinese patent document number CN202010782544.3, published on 2024-02-09, records a building segmentation method based on dense boundary blocks and attention mechanism, including: obtaining remote sensing images of marked buildings to generate a training set; constructing a loss function, and optimizing the image segmentation model using the training set; using the optimized image segmentation model to process the remote sensing image to be segmented, outputting the building map after marking the building, and completing the building segmentation. The image segmentation model of the present invention includes an encoder, a skip connection unit, and a decoder, and uses a deep convolutional network for segmentation, so that the segmentation result can be obtained by inputting the image to be segmented, avoiding additional calculations, and making the entire process more automated.

[0007] However, there are the following problems in the above prior art:

[0008] (1) The activation functions in most current remote sensing image segmentation algorithms involve exponential functions, so the spatial dependence between pixels is not strong, and the ability to analyze irregular objects is insufficient.

[0009] (2) The current remote sensing image segmentation algorithm cannot quickly notice the coordinate information of the building and cannot accurately focus on the target area. Repeated pooling operations lead to the loss of image spatial information, which is not conducive to feature extraction of different targets and objects of different sizes, and gradient explosion may occur.

[0010] (3) The current remote sensing image segmentation algorithm has problems such as insufficient segmentation accuracy and slow convergence speed.

[0011] In view of this, the present invention is proposed. Summary of the invention

[0012] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a remote sensing image building extraction method based on position attention and multi-scale residual features, thereby solving the problems raised in the above-mentioned background technology.

[0013] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0014] A remote sensing image building extraction method based on position attention and multi-scale residual features comprises the following steps:

[0015] Step 1: Preprocess the remote sensing image dataset, obtain the remote sensing image dataset, make remote sensing image sample labels based on the obtained remote sensing image dataset, and perform data enhancement operations on vertically flipping the RGB remote sensing image dataset and the corresponding remote sensing image single channel digital labels;

[0016] Step 2: Establish a remote sensing image building extraction model FCM-Net. First, extract the original remote sensing image features. Second, enhance the building coordinate information and extract feature information of different scales based on the obtained image features. Finally, restore the spatial structure of the original remote sensing image.

[0017] Step 3: Division of remote sensing image data set: divide the remote sensing image data set preprocessed in Step 1 into training set, validation set and test set according to the proportion;

[0018] Step 4: Model training and verification. Based on the remote sensing image building extraction model FCM-Net constructed in Step 2, the parameters of each layer are trained and updated. All neural network parameters are initialized first, and model-related hyperparameters are set.

[0019] Step 5: Apply the trained FCM-Net algorithm model to extract buildings from remote sensing images

[0020] Optionally, perform the following steps when making remote sensing image sample labels in Step 1:

[0021] Step 1.1: Use Labelme to manually annotate images and generate a remote sensing image segmentation dataset. After the annotation is completed, use the labelme2voc script to batch convert the JSON files generated after the annotation into a semantic segmentation dataset, and convert the RGB color images in the remote sensing image segmentation dataset into the corresponding single-channel digital labels of remote sensing images.

[0022] Step 1.2: When preprocessing the remote sensing image flipping, the RGB remote sensing image dataset and the corresponding single-channel digital label of the remote sensing image produced in Step 1.1 are respectively subjected to the data enhancement operation of vertical flipping.

[0023] Optionally, the FCM-Net network model includes a feature encoding module, coordinate information enhancement and a multi-scale residual block;

[0024] The feature encoding module consists of X 0,0 , X 1,0 , X 2,0 , X 3,0 and X 4,0 composition;

[0025] The coordinate information enhancement and multi-scale residual module consists of a coordinate attention CA and three dilated spatial pyramid pooling modules MRASP with multi-residual connection mechanisms;

[0026] The feature extraction of the original remote sensing image in Step 2 is completed by the feature encoding module of FCM-Net, and the restoration of the spatial structure of the original remote sensing image is completed by the feature decoding module of FCM-Net; the enhancement of building coordinate information and the extraction of feature information of different scales in Step 2 are completed by the coordinate information enhancement and multi-scale residual module of the FCM-Net algorithm.

[0027] Optionally, the operation and construction process of the feature encoding module when extracting features from the original remote sensing image is as follows:

[0028] Step 2.1.1: Input the remote sensing image dataset after preprocessing in Step 1 into X 0,0 In the conv+BN+FPReLu activation function combination operation, the number of channels becomes 64, plus a downsampling operation. The size is obtained to obtain a remote sensing image feature map X0 with shallow features, and X0 is input into X 1,0 middle.

[0029] The expression of the FPReLu activation function in Step 2.1.1 is:

[0030]

[0031] The FPReLu activation function is characterized by the introduction of the two-dimensional spatial condition T(x) and a random parameter that can change with the calculation;

[0032] Step 2.1.2 Input X 1,0 The feature map of remote sensing images with shallow features in the repetition X 0,0 The operation flow in the output is compared with X 0,0 The output result X1 of the deeper feature is obtained. After the output result X1 is obtained, the former output is used as the latter input in a linear order, and then passes through X 2,0 , X 3,0 and X 4,0 Each time, there is a remote sensing image feature map with deeper features than the previous node output. 4,0The output result is the remote sensing image deep feature map X4.

[0033] Step 2.1.3: From X 1,0 Output shallow remote sensing image feature map X1 to X 4,0 Each layer of the output deep remote sensing image feature map X4 will be copied and input into the coordinate information enhancement and multi-scale residual module of the FCM-Net network, where X 0,0 The output result will be copied four times and enter the feature decoding module, that is, the original remote sensing image features are extracted.

[0034] Optionally, the operation and construction process of the coordinate information enhancement and multi-scale residual module is as follows;

[0035] Step 2.2.1: By X 1,0 The shallow remote sensing image feature map X1 copied at the time of output enters the CA in the coordinate information enhancement and multi-scale residual module, and is processed to obtain a remote sensing image coordinate information map, and the obtained remote sensing image coordinate information map is input into the feature decoding module for processing to obtain a remote sensing image coordinate information map C1.

[0036] By X 2,0 The output shallow remote sensing image feature map X2, by X 3,0 The output shallow remote sensing image feature maps X3 and X 4,0 The output deep remote sensing image feature map X4 will enter the same MRASP after entering the coordinate information enhancement and multi-scale residual module; the remote sensing image feature map input from the feature encoding module to the dilated space pyramid pooling module of the multi-residual connection mechanism will enter four parallel convolutional layers with different dilation rates, and the feature information extracted by each individual sampling rate will be fused to obtain the final remote sensing image multi-scale feature map;

[0037] Step 2.2.3: After the feature map is input into MRASP, five operations will be performed in parallel.

[0038] Step 2.2.4: For the L2, L3, L4 and L5 processing directions, the first step is to perform a 1×1 convolution operation in parallel to extract features through dilated convolution. However, in the second step, in order to collect the multi-scale features of the original remote sensing feature map, a 3×3 convolution operation is performed, and the sampling rates of the 3×3 convolutions in the four directions are set to 6, 12, 18 and 24 respectively, and a 1×1 convolution operation is performed at the end.

[0039] Step 2.2.5: Set up a residual mechanism in the L2, L3, L4 and L5 processing flow directions, combine the original feature map with the multi-sampling feature processing results in the L2, L3, L4 and L5 processing flow directions, and perform a fusion operation on the multi-sampling feature processing results to obtain the final processing results in the L2, L3, L4 and L5 processing flow directions respectively.

[0040] Step 2.2.6: There is only a residual mechanism in the L1 processing flow direction, which fuses the original feature map in the input MRASP with the results of the L2, L3, L4 and L5 processing flow directions, that is, the multi-sampling features are fused with the original features.

[0041] Step 2.2.7 From feature encoding module X 1,0 The output of is processed by coordinate information enhancement and multi-scale residual module to become remote sensing image coordinate information map C1; and X 2,0 , X 3,0 and X 4,0 The output of is processed by the coordinate information enhancement and multi-scale residual module to become the remote sensing image multi-scale feature map M1, the remote sensing image multi-scale feature map M2 and the remote sensing image multi-scale feature map M3 respectively. Finally, the remote sensing image coordinate information map and the remote sensing image multi-scale feature map output by the coordinate information enhancement and multi-scale residual module are input into the feature decoding module, which completes the enhancement of building coordinate information and the extraction of feature information of different scales.

[0042] Optionally, the operation and construction process of the FCM-Net feature decoding module when restoring the spatial structure of the original remote sensing image is:

[0043] Step 2.3.1: In the 4th layer of the FCM-Net feature decoding module, the multi-scale feature map M3 of the remote sensing image output by MRASP No. 3 is upsampled and sent to the X 3,1 In the middle, it is fused with the multi-scale feature map M2 of the remote sensing image output by the third layer No. 2 MRASP;

[0044] Step 2.3.2: In the third layer of the FCM-Net feature decoding module, the multi-scale feature map M2 of the remote sensing image output by the second MRASP is first upsampled to restore the size and dimension of the feature map to be consistent with the second layer. 2,1 Then, it is fused with the multi-scale feature map M1 of the remote sensing image output by the second layer 1 MRASP, and finally output to X 3,1 The fusion operation in

[0045] Step 2.3.3: In the second layer of the FCM-Net feature decoding module, the multi-scale feature map M1 of the remote sensing image output by MRASP No. 1 is sent to X after upsampling. 1,1, and connected to X by a jump 2,1 and X 2,2 ;X 2,1 The result is upsampled to X 1,2 , and jump to X 2,2 ;X 2,2 The output result is upsampled, and the obtained multi-scale feature map M1 of the remote sensing image is sent to X 1,3 .

[0046] Step 2.3.4 In the first layer of the FCM-Net feature decoding module, the remote sensing image coordinate information map C1 output by CA is upsampled and sent to X 0,1 and connected to X via a jump 1,1 , X 1,2 and X 1,3 Medium; X 1,1 The output result is upsampled and then sent to X 0,2 and connected to X via a jump 1,2 and X 1,3 Medium; X 1,2 The output result is upsampled and then sent to X 0,3 and connected by jump X 1,3 Medium; X 1,2 The output result is upsampled and sent to X 0,4 Perform fusion operation.

[0047] Step 2.3.5: In the 0th layer of the FCM-Net feature decoding module: X from the feature encoding module 0,0 The input remote sensing image shallow feature map X0 is input through the skip connection X 0,1 In the above example, the upsampling result of the remote sensing image coordinate information map C1 output by CA is fused; and ... 0,2 , X 0,3 and X 0,4 , participate in the fusion operation of these three nodes. 0,2 and X 0,3 All nodes need to further process and participate in the fusion operation of the node through skip connections with the nodes behind them in layer 0.

[0048] Step 2.3.6: After all the processes of the feature encoding module are completed, deep supervision comes into play, adding 1×1 convolution and FPReLu activation function after each output block of the 0th layer; that is, the output of these four nodes calculates the Loss (loss function) of the corresponding remote sensing image sample labels in Step 1, that is, the binary cross entropy loss function;

[0049] Step 2.3.7: By X 0,4 The node finally outputs the remote sensing image building extraction results.

[0050] Optionally, the training set, validation set, and test set in Step 3 are mutually exclusive data.

[0051] Optionally, set model-related hyperparameters including but not limited to training rounds, batch size, optimizer selection, learning rate, and maximum value of gradient clipping strategy.

[0052] Optionally, the input in Step 5 is remote sensing image data collected by satellites. The remote sensing image is preprocessed and input into the remote sensing image building extraction model FCM-Net, and the extraction result of the building in the remote sensing image is output.

[0053] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0054] (1) A new activation function FPReLu is designed in the feature encoding module constructed in the remote sensing image building extraction model FCM-Net disclosed in the present invention. The FPReLu activation function is characterized by introducing the T(x) two-dimensional spatial condition and a random parameter that can be changed with the calculation to achieve the capture of pixel-level visual information.

[0055] (2) The remote sensing image building extraction model FCM-Net disclosed in the present invention constructs a coordinate information enhancement and multi-scale residual module that is used in combination with coordinate attention and a newly designed MRASP (multi-residual connection mechanism hollow space pyramid pooling module), wherein the MRASP designed in the present invention sets four parallel convolutional layers with different hollow rates, and fuses the feature information extracted by each individual sampling rate to obtain the final multi-scale feature map of the remote sensing image, and adds a residual connection mechanism to MRASP to fuse the multi-sampled features with the original features.

[0056] (3) The feature decoding module constructed by the remote sensing image building extraction model FCM-Net disclosed in the present invention uses skip connections and deep supervision design. Skip connections combine high-level semantics and low-level semantics from feature maps of different scales to reduce the semantic gap between the encoder and decoder; deep supervision design is to use each output block (X) of the 0th layer of the feature decoding module. 0,1 , X 0,2 , X 0,3 and X 0,4 ) is added after the 1×1 convolution and FPReLu activation function to alleviate the gradient vanishing problem and speed up the convergence.

[0057] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0059] Figure 1 It is a schematic diagram of the FCM-Net network model;

[0060] Figure 2 It is the network flow chart of FCM-Net;

[0061] Figure 3 This is a flow chart of the building extraction method for remote sensing images;

[0062] Figure 4 Schematic diagram of FPReLu principle;

[0063] Figure 5 It is a schematic diagram of feature encoding module;

[0064] Figure 6 Schematic diagram of coordinate information enhancement and multi-scale residual module;

[0065] Figure 7 Schematic diagram of the MRASP module with multiple residual connection mechanism;

[0066] Figure 8 Schematic diagram of the location of the feature decoding module in FCM-Net.

[0067] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] The present invention will now be described in further detail with reference to the accompanying drawings.

[0069] See also Figure 1-8 As shown, in this embodiment, a remote sensing image building extraction method based on position attention and multi-scale residual features is provided, comprising the following steps:

[0070] Step 1: Preprocess the remote sensing image dataset, obtain the remote sensing image dataset, make remote sensing image sample labels based on the obtained remote sensing image dataset, and perform data enhancement operations on vertically flipping the RGB remote sensing image dataset and the corresponding remote sensing image single channel digital labels;

[0071] The data set used in the present invention is the WHU-RS data set collected from Google Earth (Google Inc.), which comes from the WHU data set provided by Professor Ji Shunping of Wuhan University and his team. It is a new publicly available data set, and the data set includes 8189 512×512 pixel RGB remote sensing images, and 6000 images are randomly selected as the data set to be used in the present invention.

[0072] In Step 1, perform the following steps when making remote sensing image sample labels:

[0073] Step 1.1: Use Labelme to manually annotate images and generate a remote sensing image segmentation dataset. After the annotation is completed, use the labelme2voc script to batch convert the JSON files generated after the annotation into a semantic segmentation dataset, and convert the RGB color images in the remote sensing image segmentation dataset into the corresponding single-channel digital labels of remote sensing images.

[0074] Note: Labelme annotation tool is used as the annotation tool. This tool was designed by others and is not implemented in the patent.

[0075] Step1.2: When preprocessing the flip of the remote sensing image, the RGB remote sensing image dataset and the corresponding single-channel digital label of the remote sensing image produced in Step1.1 are subjected to data enhancement operations of vertical flipping. 12,000 RGB remote sensing images are obtained, of which 6,000 are original RGB remote sensing image data, and the other 6,000 are RGB remote sensing image data after vertical flipping data enhancement operations. 12,000 remote sensing image single-channel digital labels, of which 6,000 are remote sensing image single-channel digital labels, and the other 6,000 are remote sensing image single-channel digital labels after vertical flipping data enhancement operations.

[0076] After step 1.2, the image dataset includes 12,000 RGB remote sensing images and 12,000 remote sensing image single-channel digital labels corresponding to the RGB remote sensing images, totaling 24,000 images and 12,000 sets of remote sensing image data (one RGB remote sensing image and its corresponding remote sensing image single-channel digital label are one set).

[0077] Step 2: Establish a remote sensing image building extraction model FCM-Net. First, extract the original remote sensing image features. Second, enhance the building coordinate information and extract feature information of different scales based on the obtained image features. Finally, restore the spatial structure of the original remote sensing image.

[0078] The FCM-Net network model includes feature encoding module, coordinate information enhancement and multi-scale residual block;

[0079] The feature encoding module consists of X 0,0 , X 1,0 , X 2,0 , X 3,0 and X 4,0 composition;

[0080] Note: X-type nodes (X in the network 0,0 , X 2,0 , X 3,1 The convolutional unit is composed of two layers of conv (3×3 convolution, filled with 1) + BN (normalization layer) + FPReLu activation function, which are combined into a front-to-back linear arrangement of combined blocks.

[0081] The coordinate information enhancement and multi-scale residual module consists of a coordinate attention CA and three dilated spatial pyramid pooling modules MRASP with multi-residual connection mechanisms, such as Figure 6 shown.

[0082] The feature extraction of the original remote sensing image in Step 2 is completed by the feature encoding module of FCM-Net, and the restoration of the spatial structure of the original remote sensing image is completed by the feature decoding module of FCM-Net; the location of the spatial structure of the original remote sensing image is restored in FCM-Net. Figure 8 shown.

[0083] The operation and construction process of the feature encoding module when extracting features from the original remote sensing image is as follows: Step 2.1.1 The size of the remote sensing image data set after preprocessing in Step 1 is 512×512 pixels, and the single-channel remote sensing image data set is input to X 0,0 In the example, conv (3×3 convolution, filled with 1) + BN (normalization layer) + FPReLu activation function combination operation (hereinafter collectively referred to as cBF combination operation) is performed twice, the number of channels becomes 64, and a downsampling operation is added (the downsampling operation is to reduce the size of the remote sensing image to half of the original size). The size is 256×256 pixels, 64 channels of remote sensing image feature map X0 with shallow features, input X 1,0 middle.

[0084] The expression of the FPReLu activation function is:

[0085]

[0086] The FPReLu activation function is characterized by the introduction of the two-dimensional spatial condition T(x) and a random parameter that can be changed with the calculation. It can enhance the spatial dependency between pixels and achieve the capture of pixel-level visual information. The principle of the FPReLu activation function is as follows Figure 4 As shown;

[0087] Step 2.1.2 Input X 1,0 The feature map of remote sensing images with shallow features in the repetition X 0,0 The operation flow in (two cBF combination operations + one downsampling operation) is as follows: 0,0 The output result X1 of the deeper feature is 128×128 pixels and 128 channels. Then the former output is used as the latter input in a linear order, and passes through X 2,0 , X 3,0 and X 4,0 Each time, there is a remote sensing image feature map with deeper features than the previous node output. 4,0 The output result is a remote sensing image deep feature map X4 with a size of 16×16 elements and 1024 channels.

[0088] Step 2.1.3 The linear combination operation of the five X-type nodes constitutes the feature encoding module of the FCM-Net network. Figure 5 As shown. From X 1,0 Output shallow remote sensing image feature map X1 to X 4,0 Each layer of the output deep remote sensing image feature map X4 will be copied and input into the coordinate information enhancement and multi-scale residual module of the FCM-Net network. 0 ,0 The output result will be copied four times and enter the feature decoding module.

[0089] The operation and construction process of the FCM-Net feature decoding module when restoring the spatial structure of the original remote sensing image is as follows:

[0090] Step 2.3.1 In the fourth layer: the multi-scale feature map M3 of the remote sensing image output by MRASP No. 3 is 16×16 pixels and 1024 channels. After upsampling (upsampling, the size and dimension of the feature map are restored to be consistent with the third layer), it becomes 32×32 pixels and 512 channels and is sent to the X 3,1 In the process, it is fused with the multi-scale feature map M2 of the remote sensing image output by the third layer MRASP No. 2.

[0091] Among them, the X-type nodes of the FCM-Net feature decoding module need to first perform a fusion operation on all the input images. Then use two layers of linear arrangement consisting of conv (3×3 convolution) + BN (normalization layer) + FPReLu activation function to form a combined block operation.

[0092] Note: Upsampling is a technique for upsampling an image from a low resolution to a higher resolution. In deep learning, upsampling is often used for image segmentation, super-resolution, and other tasks to restore the original image size, and the low-resolution feature map is mapped to a high resolution to achieve image reconstruction.

[0093] Step 2.3.2 In the third layer: the multi-scale feature map M2 of the remote sensing image output by MRASP No. 2 has a size of 32×32 pixels and 512 channels. It not only needs upsampling operation (restoring the size and dimension of the feature map to be consistent with the second layer) but also changes its size to 64×64 pixels and 256 channels to the X 2,1 In addition, it needs to be fused with the multi-scale feature map M1 of the remote sensing image output by the second layer 1 MRASP. The size of M1 is 64×64 pixels, 256 channels, and it also needs to be output to X 3,1 The fusion operation in the above example is then upsampled to a size of 128×128, with 128 channels to X 2,2 middle.

[0094] Step 2.3.3 In the second layer: the multi-scale feature map M1 of the remote sensing image output by MRASP No. 1 is 64×64 pixels and 256 channels. After upsampling, the size becomes 128×128 pixels and 128 channels, and then sent to X 1,1 , and connected to X by a jump 2,1 and X 2,2 ;X 2,1 The result is upsampled to X 1,2 , and jump to X 2,2 ;X 2,2 The output result is upsampled to 128×128 pixels, 128 channels, and sent to X 1,3 .

[0095] Step 2.3.4 In the first layer: the remote sensing image coordinate information map C1 output by CA has a size of 128×128 pixels and 128 channels. After upsampling, the size becomes 256×256 pixels and 64 channels, and then sent to X 0,1 and connected to X via a jump 1 ,1 , X 1,2 and X 1,3 Medium; X 1,1 The output is upsampled to 256×256 pixels, 64 channels, and then sent to X 0,2 and connected to X via a jump 1,2 and X 1,3 Medium; X 1,2 The output is upsampled to 256×256 pixels, 64 channels, and then sent to X 0,3and connected by jump X 1,3 Medium; X 1,2 The output of the upsampled size is changed to 256 × 256 pixels, 64 channels, and then sent to X 0,4 Perform fusion operation.

[0096] Step 2.3.5 In layer 0: X from the feature encoding module 0,0 The input remote sensing image shallow feature map X0 is 256×256 pixels in size and has 64 channels. It is input through a skip connection. 0,1 In the above example, the upsampling result (restore the size and dimension of the feature map) of the remote sensing image coordinate information map C1 output by CA is fused. And connected to X 0,2 , X 0,3 and X 0,4 , participate in the fusion operation of these three nodes. 0,2 and X 0,3 All nodes need to be further processed (cBF combination operation) with the nodes behind them in layer 0 through skip connections and participate in the fusion operation of the node.

[0097] Step 2.3.6 After all the processes of the feature encoding module are completed, deep supervision (additional Loss is newly added in the middle part) comes into play. 0,1 , X 0,2 , X 0,3 and X 0,4 ) and then add 1×1 convolution and FPReLu activation function. That is, the outputs of these four nodes are used to calculate the Loss (loss function) of the corresponding remote sensing image sample labels in Step 1, that is, the binary cross entropy loss function.

[0098] Step 2.3.7 By X 0,4 The node finally outputs the remote sensing image building extraction result with a size of 256×256 pixels and 64 channels.

[0099] Note: The binary cross entropy loss function is an existing technology and there is no innovation here. The formula of binary cross entropy is:

[0100]

[0101] y is a binary label 0 or 1, p(y) is the probability that the output belongs to the y label, N is the number of images involved in the loss calculation, and N in the present invention is 4

[0102] In Step 2, the coordinate information of buildings is enhanced and feature information of different scales is extracted by the coordinate information enhancement and multi-scale residual module of the FCM-Net algorithm.

[0103] The operation and construction process of coordinate information enhancement and multi-scale residual module is as follows;

[0104] Step 2.2.1 By X 1,0 The shallow remote sensing image feature map X1 copied at the output enters the CA in the coordinate information enhancement and multi-scale residual module, and is processed to obtain a remote sensing image coordinate information map with a size of 128×128 pixels and 128 channels. The result is input into the feature decoding module. The remote sensing image coordinate information map C1 with a size of 128×128 pixels and 128 channels is processed.

[0105] Note: CA is an existing model, and there is no innovation here. CA is an attention mechanism. First, define the input feature map Input as X, perform one-dimensional average pooling on each channel of X from the horizontal direction (H, 1) and the vertical direction (1, W), and obtain two feature maps (t1 and t2) after pooling. The two generated feature maps are spliced ​​(cat splices the horizontal and vertical pooling results in the spatial dimension), and then a 1×1 convolution F is performed. 1×1 The operation generates an intermediate feature map M:

[0106] M=δ(F 1×1 (cat(t1, t2))) splits the intermediate feature map M along the spatial dimension to obtain two separate tensors Fh and Fw. Then use two 1×1 convolutions F 1×1 Unify the number of channels to the same as the input feature map X, and then send it to the Sigmoid function to get a pair of attention weights g h and g w , δ is the nonlinear activation function Sigmoid. The calculation process is:

[0107] g h =δ(F 1×1 (Fh))

[0108] g w =δ(F 1×1 (Fw))

[0109] Finally, the weights are multiplied by the input feature map.

[0110] Step 2.2.2 By X 2,0 The output shallow remote sensing image feature map X2, by X 3,0 The output shallow remote sensing image feature map X3, by X 3,0 The output shallow remote sensing image feature maps X3 and X 4,0 After the output deep remote sensing image feature map X4 enters the coordinate information enhancement and multi-scale residual module, it will enter the same MRASP (multi-residual connection mechanism hollow spatial pyramid pooling module).

[0111] The structure of MRASP is as follows Figure 7 As shown:

[0112] The remote sensing image feature map input from the feature encoding module to the dilated spatial pyramid pooling module of the multi-residual connection mechanism will enter four parallel convolutional layers with different dilation rates, and the feature information extracted by each individual sampling rate will be fused to obtain the final multi-scale feature map of the remote sensing image. Figure 7 In order to classify the central pixel, MRASP uses four different sampling rates of 6, 12, 18, and 24 dilated convolutions to extract features in parallel. Five residual mechanisms are added to MRASP to fuse the multi-sampled features with the original features. The multi-scale feature maps of the remote sensing images corresponding to the different depth features of the input are obtained respectively.

[0113] Step 2.2.3 After the feature map is input into MRASP, five operations will be performed in parallel.

[0114] Step 2.2.4 performs dilated convolution and parallel feature extraction in the L2, L3, L4 and L5 processing flow directions. The first step is a 1×1 convolution operation. However, in the second step, in order to collect the multi-scale features of the original remote sensing feature map, a 3×3 convolution operation is performed. The sampling rates of the 3×3 convolutions in the four directions are set to 6, 12, 18 and 24 respectively, and a 1×1 convolution operation is performed at the end.

[0115] Step 2.2.5 The residual mechanism is set in the L2, L3, L4 and L5 processing flow directions, and the original feature map is fused with the multi-sampling feature processing results in the L2, L3, L4 and L5 processing flow directions (respectively, the feature information map with a sampling rate of 6, the feature information map with a sampling rate of 12, the feature information map with a sampling rate of 18 and the feature information map with a sampling rate of 24). Figure 7 The concat+1×1Conv operation in the L2, L3, L4 and L5 processing flow directions are respectively obtained.

[0116] Step 2.2.6 In the L1 processing flow direction, there is only a residual mechanism, which fuses the original feature map in the input MRASP with the results of the L2, L3, L4 and L5 processing flow directions, that is, the multi-sampled features are fused with the original features ( Figure 7 concat+1×1Conv operation in .

[0117] Step 2.2.7 Finally, from the feature encoding module X 1,0 The output of is processed by coordinate information enhancement and multi-scale residual module to become remote sensing image coordinate information map C1; and X 2,0 , X 3,0 and X4,0 The output of is processed by the coordinate information enhancement and multi-scale residual module to become the remote sensing image multi-scale feature map M1, the remote sensing image multi-scale feature map M2 and the remote sensing image multi-scale feature map M3. The remote sensing image coordinate information map and the remote sensing image multi-scale feature map output by the coordinate information enhancement and multi-scale residual module are input into the feature decoding module.

[0118] Note: Dilated convolution is an existing technology and is not innovative. Dilated convolution (also called expanded convolution or dilated convolution) is a convolution operation commonly used in deep learning. The purpose of dilated convolution is to increase the receptive field without reducing the resolution of the feature map. It is achieved by padding zeros in the middle of the convolution kernel. There are two ways to implement it: one is to fill zeros inside the convolution kernel, and the other is to sample at equal intervals on the input feature map. The receptive field of dilated convolution can also be expanded arbitrarily without introducing additional parameters.

[0119] Embodiment: The shallow remote sensing image feature map X1 with a size of 128×128 pixels and 128 channels copied from the output of X1,0 enters the CA in the coordinate information enhancement and multi-scale residual module, and performs one-dimensional average pooling on each channel of the shallow remote sensing image feature map from the horizontal direction (H, 1) and the vertical direction (1, W). After pooling, the two feature maps (t1 and t2) obtained are t1 with a size of 128×1 and 128 channels, and t2 with a size of 1×128 and 128 channels. The processed remote sensing image coordinate information map is 128×128 pixels and 128 channels, and the result is input into the feature decoding module; and then a 1×1 convolution operation is performed to generate an intermediate feature map M, which is 128×128 pixels and 128 channels. The intermediate feature map M is split along the spatial dimension to obtain two separate tensors Fh (128×1) and Fw (1×128). Then two 1×1 convolutions are used to obtain the weight g h and g w , and finally the weight g h and g w It is multiplied by the input shallow remote sensing image feature map with a size of 128×128 pixels and 128 channels to obtain a remote sensing image coordinate information map C1 with a size of 128×128 pixels and 128 channels, and is input into the feature decoding module.

[0120] By X 2,0The output size is 64×64 pixels, and the shallow remote sensing image feature map X2 with 256 channels enters MRASP (multi-residual connection mechanism of hollow space pyramid pooling module), and five operations are performed in parallel to enter the shallow remote sensing image feature map M2 in the L2 direction. The first step is a 1×1 convolution operation to further extract features, but in the second step, in order to collect the multi-scale features of the original remote sensing feature map, a 3×3 convolution operation with a sampling rate of 6 is performed, and then a 1×1 convolution operation is performed. The size remains unchanged, still 64×64 pixels, 256 channels, and finally a concat+1×1Conv operation is performed with M2. After concat, the number of channels becomes twice the original, that is, 512 channels. In the 1×1Conv operation, there are 256 1×1 convolutions, and the number of channels is changed back to 256 channels, so that the shallow remote sensing image feature map M2 in the L2 direction becomes a feature information map with a sampling rate of 6. In the same process, the L3 direction, L4 direction and L5 direction respectively obtain feature information maps with a sampling rate of 12, a feature information map with a sampling rate of 18 and a feature information map with a sampling rate of 24, all of which are 64×64 pixels and 256 channels. Finally, the L1 processing flow direction is not processed at all and is fused with the results of the L2, L3, L4 and L5 processing flow directions, that is, the multi-sampled features are fused with the original features ( Figure 7 The concat+1×1Conv operation in , where the number of channels after concat becomes 1280, the size remains unchanged, still 64×64 pixels, and the final 1×1Conv operation is 256 1×1 convolution kernels, and the number of channels changes back to 256. The final output is a 64×64 pixel, 256-channel multi-scale feature map of remote sensing images that is input into the feature decoding module.

[0121] And by X 3,0 With X 4,0 The output sizes are 32×32 pixels, 512 channels of shallow remote sensing image feature map X3 and 16×16 pixels, 1024 channels of remote sensing image deep feature map X4, which enter MRASP (multi-residual connection mechanism of hollow spatial pyramid pooling module), and the remote sensing image multi-scale feature map M2 of 32×32 pixels, 512 channels and the remote sensing image multi-scale feature map M3 of 16×16 pixels, 1024 channels are obtained and input into the feature decoding module.

[0122] Step 3: Division of remote sensing image data set: divide the remote sensing image data set preprocessed in Step 1 into training set, validation set and test set according to the proportion; the training set in Step 3 accounts for 70% of the total data set, that is, 8400 remote sensing image pictures. The validation set and test set each account for 15% of the total data set, that is, 1800 remote sensing image pictures respectively.

[0123] In the actual experiment of the present invention, the data set division ratio (training set: validation set: test set) is 70%:15%:15%.

[0124] Step 4: Model training and verification. Based on the remote sensing image building extraction model FCM-Net constructed in Step 2, train and update the parameters of each layer. Initialize all neural network parameters first, set model-related hyperparameters, and set model-related hyperparameters including but not limited to training rounds, batch size, optimizer selection, learning rate, and maximum value of gradient clipping strategy. Example: Train the FCM-Net remote sensing image building extraction model, and the entire training process is 100 epochs. The weight decay coefficient is set to 0.05, the learning rate is set to 0.0000001, the batch size is set to 4, the number of iterations for the first 50 epochs is 217, the number of iterations for the next 50 epochs is 292, and the total number of iterations is 25450. Mosaic data enhancement is enabled in each round, and the resolution of the input image is uniformly adjusted to 512×512 format. After initializing the parameters, the remote sensing image training set and remote sensing image verification set data processed by Step 1 are divided into multiple batches. Each time, a batch of remote sensing image training set data is input into FCM-Net for training to obtain the training loss value loss of the batch. After one round of training for all batches of data in the entire remote sensing image training set (in the actual training process, there will likely be multiple rounds), the remote sensing image verification set is input into FCM-Net in batches to obtain the corresponding batch loss value batch_loss. During training and verification, FCM-Net will automatically learn and adjust parameters based on each loss and batch_loss situation. When the training process has been carried out for one or more rounds until the batch_loss value converges, the FCM-Net model training is completed;

[0125] Step 5: Apply the trained FCM-Net algorithm model to extract buildings from remote sensing images

[0126] The input in Step 5 is remote sensing image data collected by satellite. The remote sensing image is preprocessed and input into the remote sensing image building extraction model FCM-Net, and the extraction result of the building in the remote sensing image is output.

[0127] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A remote sensing image building extraction method based on position attention and multi-scale residual features, characterized in that: The following steps are involved: Step 1: Preprocess the remote sensing image dataset, obtain the remote sensing image dataset, make remote sensing image sample labels according to the obtained remote sensing image dataset, and perform data enhancement operations of vertically flipping the RGB remote sensing image dataset and the corresponding remote sensing image single channel digital labels; Step 2: Build the FCM-Net remote sensing image building extraction model. First, extract the original remote sensing image features. Second, enhance the building coordinate information and extract feature information of different scales based on the obtained image features. Finally, restore the spatial structure of the original remote sensing image. The operation and construction process of the feature encoding module when extracting features from original remote sensing images is as follows: Step 2.1.1: Input the remote sensing image dataset after preprocessing in Step 1 into X 0,0 In the conv+BN+FPReLu activation function combination operation, the number of channels becomes 64, plus a downsampling operation; the size is 2 to obtain the remote sensing image feature map X0 with shallow features, and X0 is input into X 1,0 middle; The expression of the FPReLu activation function in Step 2.1.1 is: The FPReLu activation function is characterized by the introduction of the two-dimensional spatial condition T(x) and a random parameter that can change with the calculation; Step 2.1.2 Input X 1,0 The feature map of remote sensing images with shallow features in the repetition X 0,0 The operation flow in the output is compared with X 0,0 The output result X1 of the deeper feature is obtained. After the output result X1 is obtained, the former output is used as the latter input in a linear order, and then passes through X 2,0 , X 3,0 and X 4,0 ; Each time, there is a remote sensing image feature map with deeper features than the previous node output, X 4,0 The output result is the remote sensing image deep feature map X4; Step 2.1.3: From X 1,0 Output shallow remote sensing image feature map X1 to X 4,0 Each layer of the output deep remote sensing image feature map X4 will be copied and input into the coordinate information enhancement and multi-scale residual module of the FCM-Net network, where X 0,0 The output result will be copied four times and enter the feature decoding module, that is, the original remote sensing image features are extracted; The operation and construction process of coordinate information enhancement and multi-scale residual module is as follows; Step 2.2.1: By X 1,0 The shallow remote sensing image feature map X1 copied at the time of output enters the CA in the coordinate information enhancement and multi-scale residual module, and is processed to obtain a remote sensing image coordinate information map, and the obtained remote sensing image coordinate information map is input into the feature decoding module for processing to obtain a remote sensing image coordinate information map C1; Shallow remote sensing image feature map X2, by X 3,0 The output shallow remote sensing image feature maps X3 and X 4,0 After the output deep remote sensing image feature map X4 enters the coordinate information enhancement and multi-scale residual module, it will enter the same MRASP; The remote sensing image feature map input from the feature encoding module to the dilated spatial pyramid pooling module of the multi-residual connection mechanism will enter four parallel convolutional layers with different dilation rates, and the feature information extracted by each individual sampling rate will be fused to obtain the final multi-scale feature map of the remote sensing image; Step 2.2.3: After the feature map is input into MRASP, five operations will be performed in parallel; Step 2.2.4: The L2, L3, L4 and L5 processing flow directions of performing hole convolution and parallel feature extraction processing all perform 1×1 convolution operations in the first step, but in the second step, in order to collect the multi-scale features of the original remote sensing feature map, a 3×3 convolution operation is performed, and the sampling rates of the 3×3 convolutions in the four directions are set to 6, 12, 18 and 24 respectively, and the end is a 1×1 convolution operation; Step 2.2.5: Set the residual mechanism in the L2, L3, L4 and L5 processing flow directions, combine the original feature map with the multi-sampling feature processing results in the L2, L3, L4 and L5 processing flow directions, and perform a fusion operation on the multi-sampling feature processing results to obtain the final processing results in the L2, L3, L4 and L5 processing flow directions respectively; Step 2.2.6: The L1 processing flow direction only has a residual mechanism, which fuses the original feature map in the input MRASP with the results of the L2, L3, L4 and L5 processing flow directions, that is, the multi-sampled features are fused with the original features; Step 2.2.7 From feature encoding module X 1,0 The output of is processed by coordinate information enhancement and multi-scale residual module to become remote sensing image coordinate information map C1; and X 2,0 , X 3,0 and X 4,0 The output of is processed by the coordinate information enhancement and multi-scale residual module to become the remote sensing image multi-scale feature map M1, the remote sensing image multi-scale feature map M2 and the remote sensing image multi-scale feature map M3 respectively. Finally, the remote sensing image coordinate information map and the remote sensing image multi-scale feature map output by the coordinate information enhancement and multi-scale residual module are input into the feature decoding module, that is, the building coordinate information is enhanced and the feature information of different scales is extracted; The operation and construction process of the FCM-Net feature decoding module when restoring the spatial structure of the original remote sensing image is as follows: Step 2.3.1: In the 4th layer of the FCM-Net feature decoding module, the multi-scale feature map M3 of the remote sensing image output by MRASP No. 3 is upsampled and sent to the X 3,1 In the middle, it is fused with the multi-scale feature map M2 of the remote sensing image output by the third layer No. 2 MRASP; Step 2.3.2: In the third layer of the FCM-Net feature decoding module, the multi-scale feature map M2 of the remote sensing image output by the second MRASP is first upsampled to restore the size and dimension of the feature map to be consistent with the second layer. 2,1 Then, it is fused with the multi-scale feature map M1 of the remote sensing image output by the second layer 1 MRASP, and finally output to X 3,1 The fusion operation in Step 2.3.3: In the second layer of the FCM-Net feature decoding module, the multi-scale feature map M1 of the remote sensing image output by MRASP No. 1 is sampled and sent to X 1,1 , and connected to X by a jump 2,1 and X 2,2 ;X 2,1 The result is upsampled to X 1,2 , and jump to X 2,2 ;X 2,2 The output result is upsampled, and the obtained multi-scale feature map M1 of the remote sensing image is sent to X 1,3 ; Step 2.3.4 In the first layer of the FCM-Net feature decoding module, the remote sensing image coordinate information map C1 output by CA is upsampled and sent to X 0,1 and connected to X via a jump 1,1 , X 1,2 and X 1,3 Medium; X 1,1 The output result is upsampled and then sent to X 0 ,2 and connected to X via a jump 1,2 and X 1,3 Medium; X 1,2 The output result is upsampled and then sent to X 0,3 and connected by jump X 1,3 Medium; X 1,2 The output result is upsampled and sent to X 0,4 Perform fusion operation in Step 2.3.5: In the 0th layer of the FCM-Net feature decoding module: X from the feature encoding module 0,0 The input remote sensing image shallow feature map X0 is input through the skip connection X 0,1 In the above example, the upsampling result of the remote sensing image coordinate information map C1 output by CA is fused; and ... 0,2 , X 0,3 and X 0,4 , participate in the fusion operation of these three nodes; X 0,2 and X 0,3 All nodes need to be further processed and participate in the fusion operation of the node they are in through skip connections with the nodes behind them in layer 0; Step 2.3.6: After all the processes of the feature encoding module are completed, deep supervision comes into play, adding 1×1 convolution and FPReLu activation function after each output block of the 0th layer; that is, the output of these four nodes is used to calculate the Loss function, i.e., the binary cross entropy loss function, with the corresponding remote sensing image sample labels in Step 1; Step 2.3.7: By X 0,4 The node finally outputs the remote sensing image building extraction results; Step 3: Division of remote sensing image data set: divide the remote sensing image data set preprocessed in Step 1 into training set, validation set and test set according to the proportion; Step 4: Model training and verification. Based on the FCM-Net remote sensing image building extraction model constructed in Step 2, the parameters of each layer are trained and updated. All neural network parameters are initialized first, and model-related hyperparameters are set. Step 5: Apply the trained FCM-Net algorithm model to extract buildings from remote sensing images.

2. The remote sensing image building extraction method based on position attention and multi-scale residual features according to claim 1 is characterized in that: In Step 1, perform the following steps when making remote sensing image sample labels: Step 1.1: Use Labelme to manually annotate images and generate a remote sensing image segmentation dataset. After the annotation is completed, use the labelme2voc script to batch convert the JSON files generated after the annotation into a semantic segmentation dataset, and convert the RGB color labels in the remote sensing image segmentation dataset into the corresponding single-channel digital labels of the remote sensing images. Step 1.2: When preprocessing the remote sensing image flipping, the RGB remote sensing image dataset and the corresponding single-channel digital label of the remote sensing image produced in Step 1.1 are respectively subjected to the data enhancement operation of vertical flipping.

3. The remote sensing image building extraction method based on position attention and multi-scale residual features according to claim 1 is characterized in that: The FCM-Net network model includes feature encoding module, coordinate information enhancement and multi-scale residual block; The feature encoding module consists of X 0,0 , X 1,0 , X 2,0 , X 3,0 and X 4,0 composition; The coordinate information enhancement and multi-scale residual module consists of a coordinate attention CA and three dilated spatial pyramid pooling modules MRASP with multi-residual connection mechanisms; The feature extraction of the original remote sensing image and the restoration of the spatial structure of the original remote sensing image in Step 2 are both completed by the feature encoding module of FCM-Net; In Step 2, the coordinate information of buildings is enhanced and feature information of different scales is extracted by the coordinate information enhancement and multi-scale residual module of the FCM-Net algorithm.

4. The remote sensing image building extraction method based on position attention and multi-scale residual features according to claim 1 is characterized in that: The training set, validation set, and test set in Step 3 are mutually exclusive data.

5. The remote sensing image building extraction method based on position attention and multi-scale residual features according to claim 1 is characterized in that: Setting model-related hyperparameters includes but is not limited to training rounds, batch size, optimizer selection, learning rate, and maximum value of gradient clipping strategy.

6. The remote sensing image building extraction method based on position attention and multi-scale residual features according to claim 1 is characterized in that: The input in Step 5 is remote sensing image data collected by satellite. The remote sensing image is preprocessed and input into the remote sensing image building extraction model FCM-Net, and the extraction result of the building in the remote sensing image is output.

Images