Dual-stream Remote Sensing Image Fusion Method Based on Residual Channel Attention Mechanism

Abstract

The present invention provides a dual-stream remote sensing image fusion method based on a residual channel attention mechanism. A convolutional neural network extracts features from a panchromatic image and a low-resolution multispectral remote sensing image respectively, and then fuses them to form a compact feature map. Then, a residual attention network is constructed. The residual attention network uses the attention mechanism to model the interdependence between feature channels, adaptively adjusts the features of each channel, so as to be able to focus on more useful channels and improve the recognition and learning ability. The residual attention network adopts multiple residual connections, among which the long residual connection allows shallow residual learning, and the long and short residual connections allow a large amount of shallow information to pass through these identity-based skip connections, simplifying the flow of information. Finally, after reconstruction by a transposed convolutional layer and a convolutional layer, a high-quality remote sensing image can be generated, which is of great significance to the field of remote sensing image fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image fusion, and particularly to a dual-stream remote sensing image fusion method based on a residual channel attention mechanism. Background Art

[0002] Remote sensing image fusion is an algorithm that fuses high-resolution panchromatic remote sensing images (PAN images) and low-resolution multispectral remote sensing images (LMS images) into high-resolution multispectral remote sensing images. High-resolution multispectral remote sensing images can calculate the reflection spectra of each pixel on the Earth's surface to obtain various information, providing assistance for subsequent remote sensing scene segmentation, classification, and feature extraction, such as forest resource surveys, land cover classification, precision agriculture, and weather forecasting. However, due to the limitations of current hardware, it is difficult to obtain remote sensing images with high resolution through a single sensor. Only the panchromatic image of a single band on the ground surface and the multispectral remote sensing image of multiple bands can be obtained separately. These two types of images carry different information but are complementary in information. In order to obtain high-resolution multispectral remote sensing images, panchromatic sharpening has been developed as a key technology in remote sensing image fusion. As remote sensing images become increasingly important, remote sensing image fusion algorithms are constantly being improved. How to fuse the spatial information and spectral information of panchromatic images and multispectral remote sensing images as much as possible to improve the fusion effect is a key issue in remote sensing image fusion.

[0003] In recent years, many advanced deep learning-based methods have been proposed, which have great potential. Deep learning models are constructed by multiple transformation layers. In each layer, the input data is linearly filtered to generate output data. Multiple layers are stacked to form a total transformation with high nonlinearity. Deep learning, especially convolutional neural networks (CNNs), provides better transformation modeling, which helps to fit complex transformations. During the training process, the parameters are updated under the supervision of training samples to improve the fitting accuracy. Inspired by the powerful capabilities of deep learning in the field of computer vision, in remote sensing image fusion, the features extracted by deep learning methods have stronger representation capabilities than those extracted by traditional methods. For example, in the aspect of remote sensing image fusion based on a three-layer CNN structure of convolutional neural network, the remote sensing image fusion algorithm PNN (Pansharpening by CNN) based on CNN was first proposed, which significantly improved the performance of the remote sensing image fusion algorithm and generated high-resolution multispectral remote sensing images; the deep residual network for remote sensing image fusion DRPNN (Pansharpening by DeepResidual CNN), this algorithm fuses the residual connection learning features. With the support of the residual connection architecture, a very deep convolutional network framework can be formed, ensuring that the network is not easily degraded while improving the fusion accuracy and also enhancing the network performance; (Two-stream fusion network) TFnet extracts the features of PAN and MS images respectively by using a two-channel CNN, and uses residual connections to learn shallow features and then strengthens the deep features based on the identity jump process, strengthening the feature learning and improving the performance of the fusion network. However, the above methods cannot focus on extracting the spatial information and spectral information in the image, resulting in the fusion effect of the final pan-sharpened image not reaching the best effect. Summary of the Invention

[0004] In view of this, the present invention proposes a two-stream remote sensing image fusion method based on a residual channel attention mechanism, constructs a residual attention network, and enables the network to focus on more useful channels and enhance the learning ability through the channel attention mechanism in the residual attention network, ensuring that a final high-resolution multispectral remote sensing image can be obtained.

[0005] The technical solution of the present invention is implemented as follows:

[0006] A two-stream remote sensing image fusion method based on a residual channel attention mechanism includes the following steps:

[0007] Step S1, use a convolutional neural network to extract features from a panchromatic image and a low-resolution multispectral remote sensing image, and splice the two to obtain a spliced feature;

[0008] Step S2: Construct a residual attention network, which includes a residual attention module containing a channel attention mechanism;

[0009] Step S3: Input the concatenated features into the residual attention network for convolution processing to obtain initial features. The residual attention module performs weighted allocation processing on the initial features according to the channel attention mechanism to obtain new features, and obtains enhanced features based on the new features;

[0010] Step S4: Enlarge the size of the enhanced features through a deconvolution layer, and then reconstruct the enlarged features through a convolution layer to obtain a high-resolution multispectral remote sensing image.

[0011] Preferably, before extracting features from the panchromatic image in Step S1, downsample the panchromatic image to adapt to the size of the low-resolution multispectral remote sensing image.

[0012] Preferably, the residual attention network in Step S2 further includes a residual attention group, which includes several blocks, a long residual connection, and a short residual connection. The short residual connection stacks the remaining blocks, and the long residual connection and the short residual connection allow shallow information to directly propagate backward through identity mapping.

[0013] Preferably, the specific steps of Step S3 include:

[0014] Step S31: Input the concatenated feature F b-1 into the residual attention network, and obtain the initial feature X after two convolutions;

[0015] Step S32: Input the initial feature X into the residual attention module, and obtain the new feature from the channel attention mechanism Obtain the enhanced feature F b-1 based on the new feature b .

[0016] Preferably, the specific expression of Step S31 is:

[0017]

[0018] where X b-1 is the output obtained after the first convolution of the concatenated feature F b-1 , X b is the output obtained after the second convolution, the initial feature X is one of X b , W 1 and W 2 are the weights of the first and second convolutional layers respectively, b 1 and b 2Denote the biases of the first and second convolutional layers. 3×3 represents the size of the convolutional kernel, and δ(·) represents the ReLU activation function.

[0019] Preferably, the specific expression of step S32 is:

[0020] F b = CA(X b ) + F b-1 ;

[0021] where CA(·) represents the channel attention mechanism function, and the new feature

[0022] Preferably, the specific steps for the residual attention module in step S3 to perform weighted distribution processing on the initial feature according to the channel attention mechanism and obtain the enhanced feature are as follows:

[0023] Step S33: Obtain the number of channels C, perform global average pooling on the input initial feature X, and obtain the channel description z;

[0024] Step S34: After passing the channel description z through the downsampling layer and the upsampling layer in sequence, obtain the channel statistic w, and the channel statistic w contains the weight coefficient w for each channel c ;

[0025] Step S35: Multiply the weight coefficient w c and the initial feature X to obtain the new feature

[0026] Preferably, in step S33, the number of channels C = (1, 2,..., c), the initial feature X = [X1, X2,... X c , and the specific expression of the channel description z c for the c-th channel is:

[0027]

[0028] where f GP (·) is the global average pooling function, H and W are the sizes of the feature map, and x c (i, j) is the value of the c-th layer feature x c at (i, j).

[0029] Preferably, the specific expression of the channel statistic w in step S34 is:

[0030] w = S(W U δ(W D z));

[0031] where S(·) represents the sigmoid activation function, δ(·) represents the ReLU activation function, and WD is the weight set of the dimensionality reduction convolutional layer, W U is the weight set of the dimensionality increase convolutional layer.

[0032] Preferably, the new features in step S35 The specific expression is:

[0033]

[0034] where w c and x c are the weight coefficient and the initial feature of the c-th layer channel respectively.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The present invention provides a dual-stream remote sensing image fusion method based on a residual channel attention mechanism, which uses a convolutional neural network as a feature extractor to represent a panchromatic image and a low-resolution multispectral remote sensing image, and fuses and splices the two after feature extraction into a spliced feature, and then constructs a residual attention network. A residual attention module containing a channel attention mechanism is set in the residual attention network. The channel attention mechanism can learn the mutual dependence between channels to recalibrate channel features, so as to focus on extracting the spatial information and spectral information in the image to comprehensively reconstruct the panchromatic sharpened image and ensure that a high-resolution multispectral remote sensing image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only the preferred embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 is the flowchart of the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention;

[0039] Figure 2 is the flow schematic diagram of the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention;

[0040] Figure 3 is the flowchart of the residual attention network of the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention;

[0041] Figure 4 is the flowchart of the channel attention mechanism of the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention;

[0042] Figure 5 This is a comparison chart of other remote sensing image fusion algorithms for the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention. Specific Embodiment

[0043] To better understand the technical content of the present invention, a specific embodiment is provided below, and the present invention will be further described in conjunction with the accompanying drawings.

[0044] See Figures 1 to 4 , the dual-stream remote sensing image fusion method based on the residual channel attention mechanism provided by the present invention includes the following steps:

[0045] Step S1: Use a convolutional neural network to extract features from the panchromatic image and the low-resolution multispectral remote sensing image, and splice the two to obtain a spliced feature;

[0046] Step S2: Construct a residual attention network, the residual attention network includes a residual attention module, and the residual attention module contains a channel attention mechanism;

[0047] Step S3: Input the spliced feature into the residual attention network for convolutional processing to obtain an initial feature, and the residual attention module performs weighted allocation processing on the initial feature according to the channel attention mechanism to obtain a new feature, and obtain a strengthened feature according to the new feature;

[0048] Step S4: Enlarge the size of the strengthened feature through a deconvolution layer, and then reconstruct the enlarged feature through a convolutional layer to obtain a high-resolution multispectral remote sensing image.

[0049] For the dual-stream remote sensing image fusion method based on the residual channel attention mechanism of the present invention, first, a convolutional neural network is used as a feature extractor to represent the panchromatic image and the low-resolution multispectral remote sensing image, and then the two are spliced on the channel to form a compact feature representation, that is, a spliced feature. Since the panchromatic image has rich spatial information and the low-resolution multispectral remote sensing image is a carrier of spectral information, when using a convolutional neural network for feature extraction, the feature image can be well reconstructed. After obtaining the spliced feature, the spliced feature is input into the constructed residual attention network. The residual attention network contains multiple residual attention modules, and the residual attention module contains a channel attention mechanism. The channel attention mechanism can adaptively assign different weights to each channel by modeling the mutual dependence between feature channels. This mechanism allows the network to focus on more useful channels and enhance the learning ability. The strengthened feature processed by the residual attention network can obtain the final high-resolution multispectral remote sensing image after two convolutions.

[0050] Preferably, before performing feature extraction on the panchromatic image, the panchromatic image is downsampled to adapt to the size of the low-resolution multispectral remote sensing image.

[0051] The size of the panchromatic image is generally 256*256, and the general size of the multispectral remote sensing image is 64*64. Therefore, before performing feature extraction, the panchromatic image needs to be downsampled so that the size of the panchromatic image can be applied to the low-resolution multispectral remote sensing image. To retain features and prevent information loss, the convolutional neural network as a feature extractor does not use pooling layers, batch normalization, and ReLU. Instead, they are simply connected together to implement the fusion strategy, and the concatenated feature F after fusion and concatenation b-1 The specific expression is:

[0052]

[0053] Let the inputs of the panchromatic image and the low-resolution multispectral remote sensing image be represented by X p and X m respectively, and the features extracted by the convolutional neural network be represented by and respectively. The superscript 1 indicates extracting features from the first layer. Then the expression of is:

[0054]

[0055]

[0056]

[0057]

[0058] where W 1 and W 2 represent the weights of the first and second convolutional layers respectively, and b 1 and b 2 represent the biases of the first and second convolutional layers respectively. 3×3 represents the size of the convolutional kernel, and represents the convolutional output of the second layer.

[0059] Preferably, the residual attention network in the step S2 further includes a residual attention group. The residual attention group includes several blocks, a long residual connection, and a short residual connection. The short residual connection stacks the remaining blocks, and the long residual connection and the short residual connection allow shallow information to directly propagate backward through the identity mapping.

[0060] Multiple residual connections can learn shallow features to strengthen deep features. Since the long residual connection and the short residual connection allow shallow information to directly propagate backward through the identity mapping, it is beneficial to the flow of information.

[0061] Preferably, the specific steps of step S3 include:

[0062] Step S31: Input the splicing feature F b-1 into the residual attention network, and obtain the initial feature X after two convolutions;

[0063] The specific expression of step S31 is:

[0064]

[0065] where X b-1 is the output obtained after the first convolution of the splicing feature F b-1 , X b is the output obtained after the second convolution, the initial feature X is one of X b , W 1 and W 2 are the weights of the first and second convolutional layers respectively, b 1 and b 2 represent the biases of the first and second convolutional layers, 3×3 represents the size of the convolutional kernel, and δ(·) represents the ReLU activation function.

[0066] Step S32: Input the initial feature X into the residual attention module, and obtain a new feature from the channel attention mechanism According to the new feature and the splicing feature F b-1 obtain the enhanced feature F b .

[0067] Preferably,

[0068] The specific expression of step S32 is:

[0069] F b = CA(X b ) + F b-1 ;

[0070] where CA(·) represents the channel attention mechanism function, and the new feature

[0071] After the splicing feature is input into the residual attention network, under the action of multi-residual connection and channel attention mechanism, an enhanced feature can be obtained. Due to the existence of the channel attention mechanism, different weights can be adaptively assigned to each channel by modeling the mutual dependence between feature channels. This mechanism allows the network to focus on more useful channels and enhance the learning ability.

[0072] Preferably, the specific steps of the residual attention module in step S3 for weighted distribution processing of the initial features according to the channel attention mechanism and obtaining the enhanced features are as follows:

[0073] Step S33: Obtain the number of channels C, perform global average pooling on the input initial feature X, convert the global spatial information into a channel descriptor, and obtain the channel descriptor z;

[0074] The number of channels C = (1, 2,..., c), the initial feature X = [X1, X2,... X c , and the specific expression of the channel descriptor z c of the c-th channel is:

[0075]

[0076] where f GP (·) is the global average pooling function, H and W are the sizes of the feature map, and x c (i, j) is the value of the feature x c at (i, j) in the c-th layer.

[0077] In addition to global pooling, an aggregation technique is introduced. The channel dependencies are fully captured from the aggregated information through global average pooling, and then a sigmoid activation function is introduced to learn the non-linear interactions between channels.

[0078] Step S34: After passing the channel descriptor z through the downsampling layer and the upsampling layer in sequence, obtain the channel statistic w, and the channel statistic w contains the weight coefficient w c ;

[0079] The specific expression of the channel statistic w in step S34 is:

[0080] w = S(W U δ(W D z));

[0081] where S(·) represents the sigmoid activation function, δ(·) represents the ReLU activation function, W D is the weight set of the downsampling convolutional layer, and W U is the weight set of the upsampling convolutional layer.

[0082] The role of the downsampling convolutional layer is to reduce the number of channels. Set the downsampling ratio to r. After the downsampled signal is activated by the ReLU activation function, the number of channels is increased r times through the upsampling convolutional layer, and the weight coefficient w c of each channel can be obtained, and the final channel statistic w is obtained and used to re-adjust the initial feature X.

[0083] Step S35: Multiply the weight coefficient wc Multiply with the initial feature X to obtain a new feature

[0084] The new feature in step S35 The specific expression is:

[0085]

[0086] where w c and x c are the weight coefficient and the initial feature of the c-th layer channel respectively.

[0087] The entire adjustment process of the channel attention mechanism is actually a re-weighted distribution of the features of different channels.

[0088] The present invention proposes a dual-stream fusion architecture compared with the existing methods. The convolutional neural network extracts features from the panchromatic image and the multispectral remote sensing image respectively, and then fuses them to form a compact feature map, which can represent the spatial and spectral information of both the panchromatic and multispectral images simultaneously. During the convolution process, each convolution operator has only a local receptive field and cannot fully utilize the context information. Therefore, the obtained features also lack context information. When traditional algorithms fuse, they treat each channel of the feature map equally and ignore the mutual dependence relationship between the channels of each feature map. For this reason, the present invention uses an attention mechanism to model the mutual dependence relationship between the feature channels and adaptively adjust the features of each channel. Such an attention mechanism enables the network proposed by us to focus on more useful channels and improve the recognition and learning ability. At the same time, multi-residual connections are adopted, aiming to enable the network to adapt to deeper structures. Among them, the long residual connection allows shallow residual learning. In each residual module, several residual blocks are stacked using short residual connections. The long residual connection, the short residual connection, and the residual group allow a large amount of shallow information to pass through these identity-based skip connections, simplifying the flow of information.

[0089] To demonstrate the effectiveness of the present invention, a comparison of the fusion effects of the present invention and other remote sensing image fusion methods (PCA, MTF_GLP, PNN, etc.) was carried out. The experimental data sets are as follows:

[0090]

[0091] After experiments were carried out according to the above experimental data sets, the parameters of each image fusion method and evaluation indexes are shown in Table 1 (RCAMTFNet is the present invention). The fusion results are compared as Figure 5 shown. According to Table 1 and Figure 5 shown, it can be seen from the comparison of various evaluation indexes that the method proposed by the present invention is significantly better than the existing remote sensing image methods.

[0092] Table 1

[0093]

[0094] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A dual-stream remote sensing image fusion method based on a residual channel attention mechanism, characterized in that Including the following steps: Step S1: Use a convolutional neural network to extract features from the panchromatic image and the low-resolution multispectral remote sensing image, and splice the two to obtain a spliced feature; Step S2: Construct a residual attention network, the residual attention network includes a residual attention module, and the residual attention module contains a channel attention mechanism; Step S3: Input the spliced feature into the residual attention network for convolutional processing to obtain an initial feature. The residual attention module performs weighted allocation processing on the initial feature according to the channel attention mechanism to obtain a new feature, and obtains a strengthened feature according to the new feature. The specific steps include: Step S31: Input the splicing feature F b-1 into the residual attention network, and obtain the initial feature X after two convolutions. The specific expression is as follows: Among them, X b-1 is the output obtained after the first convolution of the splicing feature F b-1 After the first convolution, the output obtained is X b is the output obtained after the second convolution. The initial feature X is one of X b W 1 and W 2 are the weights of the first and second convolutional layers respectively. b 1 and b 2 represent the biases of the first and second convolutional layers. 3×3 represents the size of the convolutional kernel, and δ(·) represents the ReLU activation function; Step S32: Input the initial feature X into the residual attention module, and obtain a new feature by the channel attention mechanism According to the new feature and the concatenated feature F b-1 obtain the enhanced feature F b , and the specific expression is: F b = CA(X b ) + F b-1 ; where CA(·) represents the channel attention mechanism function, and the new feature The specific steps for the residual attention module to perform weighted allocation processing on the initial feature according to the channel attention mechanism and obtain a strengthened feature are: Step S33: Obtain the number of channels C, perform global average pooling on the input initial feature X, and obtain the channel description z. The number of channels C = (1, 2,..., c), the initial feature X = [X1, X2,... X c , and the channel description z of the c-th channel c is specifically expressed as: where f GP (·) is the global average pooling function, H and W are the dimensions of the feature map, and x c (i, j) is the value of the feature x c at (i, j) in the c-th layer; Step S34: After passing the channel description z through the downsampling layer and the upsampling layer in sequence, a channel statistic w is obtained, and the channel statistic w contains the weight coefficient w of each channel c , and the specific expression of the channel statistic w is as follows: w = S(W U δ(W D z)); where S(·) represents the sigmoid activation function, δ(·) represents the ReLU activation function, and W D is the weight set of the dimensionality reduction convolutional layer, and W U is the weight set of the dimensionality increase convolutional layer; Step S35: Multiply the weight coefficient w c by the initial feature X to obtain a new feature The new feature has the specific expression as follows: where w c and x c are the weight coefficient and the initial feature of the c-th layer channel, respectively. Step S4: Enlarge the size of the strengthened feature through a deconvolution layer, and then reconstruct the enlarged feature through a convolutional layer to obtain a high-resolution multispectral remote sensing image.

2. The dual-stream remote sensing image fusion method based on the residual channel attention mechanism according to claim 1, wherein, Before extracting features from the panchromatic image in Step S1, downsample the panchromatic image to adapt to the size of the low-resolution multispectral remote sensing image.

3. The dual-stream remote sensing image fusion method based on the residual channel attention mechanism according to claim 1, wherein The residual attention network in Step S2 further includes a residual attention group. The residual attention group includes several blocks, a long residual connection, and a short residual connection. The short residual connection stacks the remaining blocks, and the long residual connection and the short residual connection allow shallow information to directly propagate backward through the identity mapping.

Images