Panchromatic-multispectral image building detection method and attention feature fusion network

Abstract

The present application relates to a kind of panchromatic-multispectral image building detection method and attention feature fusion network, belong to image processing technical field, solve the problem of poor detection performance in prior art feature fusion method, lower utilization efficiency of satellite image and lower detection precision.The attention feature fusion network of the present application adopts heuristic attention feature fusion module, adopts the structure of double flow, can handle panchromatic-multispectral image pair;Using multiple attention-based feature fusion modules, can more effectively carry out feature fusion, to obtain higher detection performance;Can directly extract features from panchromatic image and multispectral image, can more effectively utilize the information of panchromatic image and multispectral image, the utilization efficiency of satellite image is high;High detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method for building detection in panchromatic-multispectral images and an attention feature fusion network. Background Technology

[0002] Building detection involves processing remote sensing images to detect buildings in each image and marking them with bounding boxes. As a popular problem in computer vision, building detection is a crucial research foundation for intelligent interpretation of remote sensing images and is widely used in automated systems for urban planning, disaster assessment, and map creation.

[0003] Existing building detection methods typically employ deep learning-based target detection techniques, combining remote sensing data with detector training to extract buildings from remote sensing images. Due to equipment limitations, existing Earth observation satellites struggle to simultaneously acquire high spatial and hyperspectral resolution images; therefore, satellites usually provide both panchromatic and multispectral remote sensing images. To facilitate downstream applications and effectively utilize information from both types of remote sensing images, researchers have proposed panchromatic sharpening methods to fuse images, subsequently using the fused image for building detection. However, while image fusion achieves better visual results and simultaneously gains spectral information while improving spatial resolution, image fusion is a suboptimal strategy for deep learning-based target detection methods, resulting in the loss of crucial visual information for the detector and reduced detection performance. In contrast, feature-level fusion can be jointly trained with the detector, reducing potential information loss and representing a superior strategy for panchromatic-multispectral remote sensing image target detection. However, existing feature fusion methods often employ symmetrical structures for feature fusion, neglecting the unequal importance of panchromatic and multispectral images in building detection, leading to poor detection performance.

[0004] In summary, existing technologies suffer from problems such as poor detection performance of feature fusion methods, low efficiency in utilizing satellite imagery, and low detection accuracy. Summary of the Invention

[0005] In view of the above problems, the present invention provides a method for building detection in panchromatic-multispectral images and an attention feature fusion network, which solves the problems of poor detection performance, low utilization efficiency of satellite images and low detection accuracy of existing feature fusion methods.

[0006] This invention provides an attention feature fusion network for performing a four-layer feature fusion process on panchromatic and multispectral images to obtain four fused features. The attention feature fusion network includes a panchromatic image feature extraction network, a multispectral image feature extraction network, and four attention feature fusion modules.

[0007] The panchromatic image feature extraction network consists of four panchromatic image feature extraction residual blocks; the multispectral image feature extraction network consists of four multispectral image feature extraction residual blocks.

[0008] The first panchromatic image feature extraction residual block is used to extract the first panchromatic image feature from the panchromatic image; the first multispectral image feature extraction residual block is used to extract the first multispectral image feature from the multispectral image;

[0009] The i-th panchromatic image feature extraction residual block is used to extract the i-th panchromatic image feature from the (i-1)-th fused feature, i = 2, 3, 4; the i-th multispectral image feature extraction residual block is used to extract the i-th multispectral image feature from the (i-1)-th multispectral image feature, i = 2, 3, 4; the i-th attention feature fusion module is used to fuse the i-th panchromatic image feature and the i-th multispectral image feature to obtain the i-th fused feature, i = 1, 2, 3, 4.

[0010] Furthermore, the i-th attention feature fusion module includes the i-th feature fusion submodule, which is used to fuse the i-th panchromatic image feature and the processed i-th multispectral image feature to obtain the i-th fused feature;

[0011] The i-th feature fusion submodule is divided into two parallel paths:

[0012] The first input of the i-th feature fusion submodule is connected to the second input of the i-th feature fusion submodule; the first input of the i-th feature fusion submodule includes, in sequence, a first element-wise multiplication, feature concatenation and a first 3×3 convolutional layer; the output of the first input of the i-th feature fusion submodule is connected to the output of the second input of the i-th feature fusion submodule through element-wise addition.

[0013] Furthermore, the i-th attention feature fusion module also includes an i-th channel attention submodule, which is used to process the i-th multispectral image feature to obtain the i-th channel attention feature;

[0014] The i-th channel attention submodule is divided into two parallel paths:

[0015] The input of the first path of the i-th channel attention submodule is connected to the input of the second path of the i-th channel attention submodule; the first path of the i-th channel attention submodule includes global average pooling, a first 1×1 convolutional layer, a ReLU activation function, a second 1×1 convolutional layer, and a first Sigmoid function in sequence; the output of the first path of the i-th channel attention submodule is connected to the output of the second path of the i-th channel attention submodule through a second element-wise multiplication.

[0016] Furthermore, the i-th attention feature fusion module also includes the i-th spatial attention submodule, which is used to process the i-th channel attention feature to obtain the i-th spatial attention weight and the i-th spatial attention feature;

[0017] The i-th spatial attention submodule is divided into two parallel paths:

[0018] The input of the first path of the i-th spatial attention submodule is connected to the input of the second path of the i-th spatial attention submodule. The first path of the i-th spatial attention submodule includes channel max pooling, a second 3×3 convolutional layer, and a second Sigmoid function. Then, the first path of the i-th spatial attention submodule is divided into two paths: the output of the first path of the i-th spatial attention submodule is connected to the output of the second path of the i-th spatial attention submodule through a third element-wise multiplication. The output of the second path of the first path of the i-th spatial attention submodule is connected to the first element-wise multiplication of the first path of the i-th feature fusion submodule.

[0019] Furthermore, the i-th attention feature fusion module also includes the i-th feature compression submodule, which is used to process the i-th spatial attention feature to obtain the i-th compressed feature;

[0020] The i-th feature compression submodule includes a third 3×3 convolutional layer, and the output of the i-th feature compression submodule is connected to the first path of feature splicing of the i-th feature fusion submodule.

[0021] Furthermore, the attention feature fusion network also includes an upsampling module; this upsampling module is located before the spectral image feature extraction network and is used to upsample the multispectral image input to the attention feature fusion network to the same image size as the panchromatic image input to the attention feature fusion network, and then input it into the multispectral image feature extraction network.

[0022] This invention also provides a method for building detection in panchromatic-multispectral images based on the above-mentioned attention feature fusion network, comprising the following steps:

[0023] Step S1. Acquire multiple sets of remote sensing images from multiple regions. Each set of remote sensing images includes panchromatic and multispectral images of multiple regions within the same region.

[0024] The multispectral image of each region is bilinearly interpolated using an upsampling module to obtain a multispectral image of the same size as the panchromatic image of that region. Then, the panchromatic and multispectral images are preprocessed together to form multiple preprocessed panchromatic-multispectral image pairs, which are used to construct the training set.

[0025] Step S2. Construct an attention feature fusion network; based on the attention feature fusion network, construct the network model;

[0026] Step S3. Train the network model using the training set. After training, the building detection model is obtained.

[0027] Step S4. Input the panchromatic-multispectral image pair to be detected into the building detection model for detection, and obtain the building detection results.

[0028] Furthermore, the preprocessing in step S1 specifically includes dividing the panchromatic image and the multispectral image with the same size as the panchromatic image into n×n pixel image blocks, where n is a positive integer not less than 256.

[0029] Furthermore, step S4 specifically includes:

[0030] Step S4-1. Input the panchromatic-multispectral image pair to be detected into the building detection model, and perform 4-layer feature fusion processing on the panchromatic-multispectral image pair to be detected through the attention feature fusion network to obtain 4 fused features;

[0031] Step S4-2. Perform feature enhancement on the four fused features to obtain the building detection results.

[0032] Furthermore, the network model in step S2 also includes a feature pyramid network for feature enhancement of the four fused features.

[0033] Compared with the prior art, the present invention has at least the following beneficial effects:

[0034] (1) The attention feature fusion network proposed in this invention adopts a heuristic attention feature fusion module and a dual-stream structure, which can process panchromatic-multispectral image pairs; by using multiple attention-based feature fusion modules, feature fusion can be performed more effectively, thereby obtaining higher detection performance.

[0035] (2) The attention feature fusion network proposed in this invention can directly extract features from panchromatic and multispectral images, and can make more effective use of information from panchromatic and multispectral images, resulting in high utilization efficiency of satellite images.

[0036] (3) The attention feature fusion network proposed in this invention can directly extract features from panchromatic and multispectral images, and can make more effective use of the information of panchromatic and multispectral images, resulting in high detection accuracy. Attached Figure Description

[0037] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0038] Figure 1 This is a schematic diagram of the attention feature fusion network disclosed in this invention;

[0039] Figure 2 This is a schematic diagram of the attention feature fusion module disclosed in this invention;

[0040] Figure 3 This is a panchromatic-multispectral image and a diagram showing the annotations and comparisons.

[0041] Figure 4 This is a visual comparison of the detection results of the panchromatic-multispectral image building detection method disclosed in this invention with existing detection methods. Detailed Implementation

[0042] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0043] This invention discloses an attention feature fusion network, which is a heuristic attention feature fusion network used to perform 4-layer feature fusion processing on panchromatic images and multispectral images to obtain 4 fused features.

[0044] This attention feature fusion network includes a panchromatic image feature extraction network, a multispectral image feature extraction network, and four attention feature fusion modules, such as... Figure 1 As shown.

[0045] The panchromatic image feature extraction network consists of four panchromatic image feature extraction residual blocks, namely PAN-Layer1, PAN-Layer2, PAN-Layer3 and PAN-Layer4, which are used to output panchromatic image features with strides of 4, 8, 16 and 32, respectively.

[0046] The panchromatic image is processed by a panchromatic image feature extraction network to extract features layer by layer, and then feature fusion is performed on the panchromatic image features of each layer.

[0047] The multispectral image feature extraction network consists of four multispectral image feature extraction residual blocks, namely MS-Layer1, MS-Layer2, MS-Layer3 and MS-Layer4, which are used to output multispectral image features with strides of 4, 8, 16 and 32, respectively.

[0048] Multispectral images are processed layer by layer by a multispectral image feature extraction network to obtain multispectral image features at each layer, and feature fusion is performed on the multispectral image features at each layer.

[0049] The i-th panchromatic image feature PAN i and the i-th multispectral image feature MS i The features are fused through the i-th feature fusion module to produce the i-th fused feature Fusion. i The i-th fused feature replaces the i-th panchromatic image feature PAN. i The residual block for the next panchromatic image feature extraction is fed into the next image for further feature extraction; while the i-th multispectral image feature MS... i Without any modification, it is directly fed into the next multispectral image feature extraction residual block.

[0050] Specifically, the first panchromatic image feature extraction residual block is used to extract the first panchromatic image feature from the panchromatic image; the first multispectral image feature extraction residual block is used to extract the first multispectral image feature from the multispectral image.

[0051] The i-th panchromatic image feature extraction residual block is used to extract the i-th panchromatic image feature from the (i-1)-th fused feature, i = 2, 3, 4; the i-th multispectral image feature extraction residual block is used to extract the i-th multispectral image feature from the (i-1)-th multispectral image feature, i = 2, 3, 4; the i-th attention feature fusion module is used to fuse the i-th panchromatic image feature and the i-th multispectral image feature to obtain the i-th fused feature, i = 1, 2, 3, 4.

[0052] Specifically, such as Figure 2 As shown, the i-th attention feature fusion module includes the i-th feature fusion submodule, which is used to fuse the i-th panchromatic image feature and the processed i-th multispectral image feature to obtain the i-th fused feature.

[0053] From an architectural perspective, the i-th feature fusion submodule is divided into two parallel paths:

[0054] The first input of the i-th feature fusion submodule is connected to the second input of the i-th feature fusion submodule; the first input of the i-th feature fusion submodule includes, in sequence, a first element-wise multiplication, feature concatenation and a first 3×3 convolutional layer; the output of the first input of the i-th feature fusion submodule is connected to the output of the second input of the i-th feature fusion submodule through element-wise addition.

[0055] From the perspective of input and output, the i-th panchromatic image feature is divided into two paths after entering the i-th feature fusion module.

[0056] It's easy to understand that, since multispectral image features possess more semantic information, their generated spatial attention weights can more effectively help panchromatic image features focus on salient regions. Therefore, the feature fusion module uses the second path of the i-th spatial attention weight as a mask, and performs element-wise multiplication with the first path of the i-th panchromatic image feature along the spatial dimension to enhance the effective information in the panchromatic image feature, resulting in the i-th panchromatic image feature after spatial attention. The i-th compressed feature and the i-th panchromatic image feature after spatial attention are concatenated along the channel dimension and passed through the first 3×3 convolutional layer to obtain the i-th fused feature for supplementing panchromatic image information.

[0057] It should be noted that, since panchromatic features possess richer spatial resolution and texture features, making them more advantageous for building detection, this invention also employs skip connections to retain more panchromatic image information. Specifically, the second path of the i-th panchromatic image feature is added element-wise to the i-th fusion feature of the panchromatic image information to be supplemented, resulting in the final i-th fusion feature, which serves as the output of the i-th feature fusion module.

[0058] Specifically, the i-th attention feature fusion module also includes an i-th channel attention submodule, which is used to process the i-th multispectral image feature to obtain the i-th channel attention feature.

[0059] From an architectural perspective, the i-th channel attention submodule is divided into two parallel paths:

[0060] The input of the first path of the i-th channel attention submodule is connected to the input of the second path of the i-th channel attention submodule; the first path of the i-th channel attention submodule includes global average pooling, a first 1×1 convolutional layer, a ReLU activation function, a second 1×1 convolutional layer, and a first Sigmoid function in sequence; the output of the first path of the i-th channel attention submodule is connected to the output of the second path of the i-th channel attention submodule through a second element-wise multiplication.

[0061] From the perspective of input and output, the i-th multispectral image feature serves as the input to the i-th attention feature fusion module. It first passes through the i-th channel attention submodule to filter out the i-th channel attention feature with high information content. Specifically, after entering the i-th channel attention submodule, the i-th multispectral image feature is divided into two paths: the first path of the i-th multispectral image feature undergoes global average pooling to obtain the i-th global feature; the i-th global feature sequentially passes through a first 1×1 convolutional layer, a ReLU activation function, and a second 1×1 convolutional layer for inter-channel information interaction, and is then normalized to [0,1] by the first Sigmoid function to obtain the i-th channel attention weight; the second path of the i-th multispectral image feature and the i-th channel attention weight undergo a second element-wise multiplication process along the channel dimension to retain the channel features with high information content, thus obtaining the i-th channel attention feature.

[0062] Specifically, the i-th attention feature fusion module also includes the i-th spatial attention sub-module, which is used to process the i-th channel attention feature to obtain the i-th spatial attention weight and the i-th spatial attention feature.

[0063] From an architectural perspective, the i-th spatial attention submodule is divided into two parallel paths:

[0064] The input of the first path of the i-th spatial attention submodule is connected to the input of the second path of the i-th spatial attention submodule. The first path of the i-th spatial attention submodule includes channel max pooling, a second 3×3 convolutional layer, and a second Sigmoid function. Then, the first path of the i-th spatial attention submodule is divided into two paths: the output of the first path of the i-th spatial attention submodule is connected to the output of the second path of the i-th spatial attention submodule through a third element-wise multiplication. The output of the second path of the first path of the i-th spatial attention submodule is connected to the first element-wise multiplication of the first path of the i-th feature fusion submodule.

[0065] From the perspective of input and output, the i-th spatial attention submodule is used to select the i-th spatial attention feature that is beneficial to feature fusion from the i-th channel attention feature in the spatial dimension. Specifically, after the i-th channel attention feature enters the i-th spatial attention submodule, it is divided into two paths: the first path of the i-th channel attention feature first goes through channel max pooling to obtain the i-th spatial response map, and then goes through the second 3×3 convolutional layer and the second Sigmoid function to obtain the i-th spatial attention weight normalized to [0,1]; the i-th spatial attention weight is divided into two paths: the first path of the i-th spatial attention weight and the second path of the i-th channel attention feature are multiplied element-wise in the spatial dimension by the third element-wise multiplication to retain features with high spatial information content, thus obtaining the i-th spatial attention feature; the second path of the i-th spatial attention weight is used as a mask and multiplied element-wise in the spatial dimension by the i-th panchromatic image feature by the first element-wise multiplication to improve the effective information in the panchromatic image feature, thus obtaining the i-th panchromatic image feature after spatial attention.

[0066] Specifically, the i-th attention feature fusion module also includes the i-th feature compression submodule, which is used to process the i-th spatial attention feature to obtain the i-th compressed feature.

[0067] From an architectural perspective, the i-th feature compression submodule includes a third 3×3 convolutional layer, and the output of the i-th feature compression submodule is connected to the first path of feature concatenation of the i-th feature fusion submodule.

[0068] From the perspective of input and output, the number of channels of the i-th spatial attention feature is reduced from C to C / r2 by the third 3×3 convolutional layer in the i-th feature compression submodule, where r2 is the compression coefficient, preferably set to 4, to obtain the i-th compressed feature.

[0069] For the i-th channel attention submodule, the i-th spatial attention submodule, and the i-th feature compression submodule, this series of processes obtains the second path and the i-th compressed feature with the i-th spatial attention weight from the i-th multispectral image feature.

[0070] Specifically, the attention feature fusion network also includes an upsampling module; this upsampling module is located before the spectral image feature extraction network and is used to upsample the multispectral image input to the attention feature fusion network to the same image size as the panchromatic image input to the attention feature fusion network, and then input it into the multispectral image feature extraction network.

[0071] This invention also provides a method for building detection in panchromatic-multispectral images based on the above-mentioned attention feature fusion network, comprising the following steps:

[0072] Step S1. Acquire multiple sets of remote sensing images from multiple regions. Each set of remote sensing images includes panchromatic and multispectral images of multiple areas within the same region.

[0073] The multispectral image of each region is bilinearly interpolated using an upsampling module to obtain a multispectral image of the same size as the panchromatic image of that region. Then, the panchromatic and multispectral images are preprocessed together to form multiple preprocessed panchromatic-multispectral image pairs, which are used to construct the training set and the test set.

[0074] It should be noted that the preprocessing in step S1 specifically includes dividing the panchromatic image and the multispectral image with the same size as the panchromatic image into n×n pixel image blocks, where n is a positive integer not less than 256.

[0075] Step S2. Construct an attention feature fusion network; based on the attention feature fusion network, construct the network model.

[0076] It should be noted that the network model in step S2 also includes a feature pyramid network, which is used to enhance the four fused features output by the attention feature fusion network.

[0077] Step S3. Train the network model using the training set. After training, the building detection model is obtained.

[0078] Step S4. Input the panchromatic-multispectral image pair to be detected into the building detection model for detection, and obtain the building detection results.

[0079] Specifically, step S4 includes:

[0080] Step S4-1. Input the panchromatic-multispectral image pair to be detected into the building detection model, and perform 4-layer feature fusion processing on the panchromatic-multispectral image pair to be detected through the attention feature fusion network to obtain 4 fused features.

[0081] Step S4-2. Perform feature enhancement on the four fused features to obtain the building detection results.

[0082] Compared with existing technologies, the attention feature fusion network proposed in this invention adopts a heuristic attention feature fusion module and a dual-stream structure, which can process panchromatic-multispectral image pairs; by utilizing multiple attention-based feature fusion modules, feature fusion can be performed more effectively, thereby obtaining higher detection performance; it can directly extract features from panchromatic and multispectral images, and can more effectively utilize the information of panchromatic and multispectral images, resulting in high utilization efficiency of satellite images; and it has high detection accuracy.

[0083] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution of this invention is provided through a specific embodiment.

[0084] Example 1

[0085] In terms of data sources, this embodiment uses the 5M-Building dataset to evaluate the network model. This dataset consists of remote sensing images taken by the Gaofen-2 satellite over multiple regions. Taking Shandong Province, China, as an example, the dataset for Shandong Province includes 109 panchromatic images and corresponding multispectral images of various regions within Shandong. The spatial resolutions of the panchromatic and multispectral images are 0.8 meters and 3.2 meters, respectively. The multispectral images contain four bands: red, green, blue, and near-infrared, with a pixel resolution ranging from 2000 to 5000 pixels. The buildings in the dataset exhibit diverse shapes and densities. To facilitate subsequent feature fusion, the multispectral images are first upsampled to the same size as the panchromatic images using bilinear interpolation. Then, both the multispectral images and the panchromatic images are cropped into 512×512 pixel blocks, with a 64-pixel overlap between blocks to ensure the integrity of building instances as much as possible, forming the training and testing samples. Ultimately, the training set for 5M-Building contained 3750 images and 62487 building instances, while the test set contained 1233 images and 15550 building instances. Since the images taken by the Gaofen-2 satellite have a bit depth of 10 and cannot be used directly, this invention uses pixel histograms to normalize the images, ensuring that each spectral value is within the range [0, 255].

[0086] Panchromatic-multispectral image of a region in the 5M-Building dataset and its annotations are as follows: Figure 3 As shown, it should be noted that the remote sensing images in the 5M-Building dataset come with built-in annotations.

[0087] Regarding the training and application details of the network model, this invention first constructs an attention feature fusion network, then builds a network model based on the attention feature fusion network, and implements the detection method using the MMDetection framework based on PyTorch. This invention selects six detection methods to verify the effectiveness of the attention feature fusion network constructed in this invention. The panchromatic image feature extraction network and the multispectral image feature extraction network are initialized using ImageNet pre-trained weights, while the remaining networks are randomly initialized using a Gaussian distribution with a mean of 0 and a variance of 0.01. The training process uses the SGD optimizer, with an initial learning rate set to 0.01, a momentum parameter set to 0.9, and a weight decay parameter set to 0.0005. The network model uses a slow-start strategy in the first 500 iterations to ensure the stability of the training process, with a start factor of 0.33. The training batch size is set to 4, and the network is trained for 12 iterations on a single NVIDIA 2080TI graphics card, with the learning rate decayed by a factor of 10 in the 8th and 11th iterations, respectively. Gradient clipping was applied during training; gradients with an L2 magnitude greater than 35 were clipped to prevent gradient explosion. During testing, a non-maximum suppression algorithm with an intersection-over-union (IoU) threshold of 0.3 was used to remove redundant detection boxes, and a score threshold of 0.05 was used to filter the boxes, retaining a maximum of 300 boxes. Before feeding images into the network model, they were scaled to 800×800 using bilinear interpolation and augmented with random horizontal flipping. Images were normalized using the mean and variance of natural images. The same configuration was used during testing, but the flipping augmentation was removed.

[0088] To verify the effectiveness of the model, benchmark experiments were conducted on the 5M-building dataset. The benchmark experiments selected six detection methods to verify the detection performance of various input image-detection model combinations: Faster R-CNN (FRCNN), RetinaNet (RtnNet), FoveaNet (FvNet), FSAF, Grid Anchor Faster R-CNN (GFRCNN), and Reppoints, as shown in Table 1.

[0089] Table 1 Comparison of detection effects of the present invention with other methods

[0090]

[0091] The detection model consists of two parts: a backbone network for feature extraction and a detection module for object detection. We changed the input of the detection model to verify the impact of different input images on the detection performance of the proposed attention feature fusion network and the building detection model built based on the attention feature fusion network, corresponding to the first three rows in Table 1; and replaced the backbone network of the detection model with existing feature fusion networks of different types to verify the advantages of the proposed attention feature fusion network compared with existing feature fusion networks.

[0092] Specifically, firstly, panchromatic images, multispectral images, and fused images are used as inputs to the detection model to verify the impact of different input images on the detection performance of the attention feature fusion network proposed in this invention and the building detection model built based on the attention feature fusion network.

[0093] Secondly, the attention feature fusion network proposed in this invention is retained as the backbone network, and then the backbone network of the detection model is replaced with existing different feature fusion networks. Each network uses panchromatic-multispectral image pairs as input to verify the effectiveness of the attention feature fusion network proposed in this invention.

[0094] Table 1 shows the building detection models trained using panchromatic images, multispectral images, and Brovey fused images as input. Channel Weighted Fusion (CWF), Cross Gates (CRGs), Crossreference Module (CRM), Gated Information Fusion (GIF), and Adaptive Feature Fusion Module (AFFM) are five comparative feature fusion methods. As can be seen from Table 1, the detection accuracy of this invention is higher than the other methods. Compared to the "fused image" baseline, this method achieves significant performance improvements across all six detection methods.

[0095] To verify the effectiveness of each sub-module of the model, ablation experiments were conducted on key sub-modules.

[0096] Table 2 shows the performance gains from adding and removing each submodule. The feature fusion method is based on feature concatenation. FRCNN was selected as the detection method.

[0097] Table 2 Performance Comparison of Each Submodule

[0098]

[0099] It can be seen that each submodule has a positive effect on feature fusion, and the best detection effect can be obtained by using all submodules. Therefore, the present invention preferably includes a feature compression submodule, a channel attention submodule, a spatial attention submodule, and a feature compression submodule.

[0100] Table 3 shows the performance impact of different orders of the channel attention submodule, spatial attention submodule, and feature compression submodule.

[0101] Table 3 Performance Changes Due to Submodule Order Table 3 Performance changes resulting from submodule order

[0102]

[0103]

[0104] It can be observed that arranging the submodules in the order of channel attention, spatial attention, and feature compression yields the best detection results. Therefore, the present invention preferably includes a scheme that sequentially comprises the channel attention submodule, the spatial attention submodule, and the feature compression submodule.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An attention feature fusion network, characterized in that, The 4-layer feature fusion processing is performed on the panchromatic image and the multispectral image to obtain four fusion features. The panchromatic image feature extraction network includes four panchromatic image feature extraction residual blocks. The first panchromatic image feature extraction residual block is used for extracting the first panchromatic image feature from the panchromatic image. The i-th panchromatic image feature extraction residual block is used for extracting the i-th panchromatic image feature from the (i-1)-th fusion feature, i=2, 3, 4.

2. The attention feature fusion network according to claim 1, characterized in that, The i-th attention feature fusion module is used for performing feature fusion on the i-th panchromatic image feature and the i-th multispectral image feature to obtain the i-th fusion feature, i=1, 2, 3, 4. The i-th attention feature fusion module includes an i-th feature fusion sub-module, which is used for performing feature fusion on the i-th panchromatic image feature and the processed i-th multispectral image feature to obtain the i-th fusion feature. The i-th feature fusion sub-module is divided into two parallel paths:

3. The attention feature fusion network according to claim 2, characterized in that, The input end of the first path of the i-th feature fusion sub-module is connected with the input end of the second path of the i-th feature fusion sub-module; the first path of the i-th feature fusion sub-module includes a first element-wise multiplication, feature splicing and a first 3x3 convolution layer in sequence; and the output end of the first path of the i-th feature fusion sub-module is connected with the output end of the second path of the i-th feature fusion sub-module through element-wise addition. The i-th attention feature fusion module further includes an i-th channel attention sub-module, which is used for processing the i-th multispectral image feature to obtain an i-th channel attention feature. The i-th channel attention sub-module is divided into two parallel paths:

4. The attention feature fusion network according to claim 3, characterized in that, The input end of the first path of the i-th channel attention sub-module is connected with the input end of the second path of the i-th channel attention sub-module; the first path of the i-th channel attention sub-module includes a global average pooling, a first 1x1 convolution layer, a ReLU activation function, a second 1x1 convolution layer and a first Sigmoid function in sequence; and the output end of the first path of the i-th channel attention sub-module is connected with the output end of the second path of the i-th channel attention sub-module through a second element-wise multiplication. The i-th attention feature fusion module further includes an i-th spatial attention sub-module, which is used for processing the i-th channel attention feature to obtain an i-th spatial attention weight and an i-th spatial attention feature. The i-th spatial attention sub-module is divided into two parallel paths: The input end of the first path of the i-th spatial attention sub-module is connected with the input end of the second path of the i-th spatial attention sub-module; the first path of the i-th spatial attention sub-module includes a global average pooling, a first 1x1 convolution layer, a ReLU activation function, a second 1x1 convolution layer and a first Sigmoid function in sequence; and the output end of the first path of the i-th spatial attention sub-module is connected with the output end of the second path of the i-th spatial attention sub-module through a second element-wise multiplication. An input end of a first path of the i-th spatial attention sub-module is connected with an input end of a second path of the i-th spatial attention sub-module; the first path of the i-th spatial attention sub-module sequentially comprises a channel max-pooling, a second 3x3 convolutional layer and a second Sigmoid function, and then the first path of the i-th spatial attention sub-module is divided into two paths: an output end of a first path of the first path of the i-th spatial attention sub-module is connected with an output end of the second path of the i-th spatial attention sub-module through a third element-wise multiplication; and an output end of a second path of the first path of the i-th spatial attention sub-module is connected to a first element-wise multiplication of a first path of the i-th feature fusion sub-module.

5. The attention feature fusion network according to claim 4, characterized in that, The i-th attention feature fusion module further comprises an i-th feature compression sub-module for processing the i-th spatial attention feature to obtain an i-th compressed feature. The i-th feature compression sub-module comprises a third 3x3 convolutional layer, and an output end of the i-th feature compression sub-module is connected to a feature concatenation of the first path of the i-th feature fusion sub-module.

6. The attention feature fusion network according to claim 5, characterized in that, The attention feature fusion network further comprises an upsampling module; the upsampling module is located before the spectral image feature extraction network and is used for upsampling a multispectral image input into the attention feature fusion network to the same image size as a panchromatic image input into the attention feature fusion network, and then inputting the multispectral image into the multispectral image feature extraction network.

7. A building detection method for panchromatic-multispectral images based on the attention feature fusion network according to any one of claims 1-6, characterized in that, The method comprises the following steps: Step S1. Obtain a plurality of groups of remote sensing images of a plurality of regions, each group of remote sensing images comprising a panchromatic image and a multispectral image of a plurality of areas in the same region; The multispectral image of each area is bilinearly interpolated by the upsampling module to obtain a multispectral image having the same size as the panchromatic image of the area, and then the multispectral image and the panchromatic image are preprocessed together to form a plurality of preprocessed panchromatic-multispectral image pairs for constructing a training set; Step S2. Construct an attention feature fusion network; Based on the attention feature fusion network, a network model is constructed; Step S3. Train the network model using the training set, and obtain a building detection model after the training is completed; Step S4. Input a panchromatic-multispectral image pair to be detected into the building detection model for detection to obtain a building detection result.

8. The panchromatic-multispectral image building detection method according to claim 7, characterized in that, The preprocessing in step S1 specifically comprises dividing the panchromatic image and the multispectral image having the same size as the panchromatic image into n*n pixel image blocks, n being a positive integer not less than 256.

9. The panchromatic-multispectral image building detection method according to claim 8, characterized in that, Step S4 specifically comprises: Step S4-1. Input the panchromatic-multispectral image pair to be detected into the building detection model, and obtain four fusion features by performing four-layer feature fusion processing on the panchromatic-multispectral image pair to be detected through the attention feature fusion network; Step S4-2. Perform feature enhancement on the four fusion features to obtain a building detection result.

10. The panchromatic-multispectral image building detection method according to claim 9, characterized in that, The network model in step S2 further comprises a feature pyramid network for performing feature enhancement on the four fusion features.

Images