A dual-branch network pan-sharpening method based on detail injection

Abstract

The present invention discloses a method for panchromatic sharpening based on detail injection using a dual-branch network. First, the original data is divided into training, verification and test data, and the corresponding data sets are generated and preprocessed; then a dual-branch panchromatic sharpening network model based on detail injection is constructed; the dual-branch panchromatic sharpening network model based on detail injection is trained; the test data is fused using the trained network model; and finally the quality of the fused image is evaluated. The present invention uses different independent subnetworks for feature extraction for source images of different categories, and continuously utilizes the extracted feature maps during feature fusion to make up for the information lost during the fusion process while promoting information flow; and solves the problem that the existing method does not fully utilize the unique information in the two types of source images and the model is not interpretable, resulting in low performance of the fusion result.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image fusion in image processing, and in particular relates to a double-branch network full color sharpening method based on detail injection. Background Art

[0002] Remote sensing technology is an important way to obtain information about land objects. In recent years, remote sensing technology has developed rapidly, and the scale of remote sensing image data has continued to grow. It has played an increasingly important role in land object exploration, environmental monitoring, national defense and military fields. For the application of remote sensing data, remote sensing images with high resolution in both spectrum and space often play a leading role in analysis, processing and decision-making, and thus become an urgent need for users. However, due to the limitations of sensor imaging conditions and processes, a single sensor cannot obtain remote sensing images with both high spatial and high spectral resolution. Generally, remote sensing satellites obtain panchromatic images (PAN) and multispectral (MS) images through two types of sensors. Multispectral images decompose the incident light into different spectral bands such as red, green, blue, and near-infrared before imaging, so the spectral resolution is high, but the spatial information they contain is insufficient; the spectral range of panchromatic images is wider than that of multispectral images, so the spectral resolution is lower, but it has rich spatial information. The purpose of panchromatic sharpening is to fuse the two types of images and combine the complementary advantages of the two types of images to obtain remote sensing images with high resolution in both spectrum and space. The fused image retains the spectral information in the multispectral image, which is conducive to distinguishing different ground objects. At the same time, it has a high spatial resolution, which is convenient for observing the shape and structure of the target. The panchromatic sharpening technology reduces the uncertainty and incompleteness of analyzing the target using a single remote sensing data, and improves the application value of remote sensing data.

[0003] In recent years, panchromatic sharpening has attracted the attention of more and more researchers and has made considerable progress. Traditional panchromatic sharpening methods usually require the use of certain tools to extract features such as contours, edges, and textures from images, then perform feature selection, and then inject these features through appropriate fusion rules to obtain fused images. Most of these steps need to be manually designed and implemented, which is highly complex, and the obtained fused images usually cannot achieve a good balance between spectral and spatial quality. With the development of deep learning technology, convolutional neural networks have been widely used in the field of computer vision and have achieved remarkable development, and have gradually been applied to the field of panchromatic sharpening. The panchromatic sharpening method based on convolutional neural networks avoids the problem of manually designing fusion schemes in traditional algorithms, but instead jointly optimizes processes such as feature extraction and selection, and fused image reconstruction to reduce the impact of intermediate processes on the fusion results. In addition, the performance of the generated fused image is better than that of traditional algorithms.

[0004] In the methods currently proposed to solve the pan-sharpening problem by using convolutional neural networks, pan-sharpening is usually regarded as super-resolution reconstruction of multispectral images, that is, the upsampled low-resolution multispectral image and the panchromatic image are used as the input of the network model, and the mapping of the input to the ideal high-resolution multispectral (HRMS) image is directly learned. However, this ignores the difference between the two types of source images and does not fully utilize the spatial information in the panchromatic image and the spectral information in the multispectral image. On the other hand, many networks regard the pan-sharpening process as a black box, and the model structure does not have a clear physical interpretability. At the same time, no prior knowledge in the field of pan-sharpening is introduced, so the learning ability and performance of the model are limited. Summary of the invention

[0005] The purpose of the present invention is to provide a dual-branch network panchromatic sharpening method based on detail injection. For source images of different categories, different independent sub-networks are used for feature extraction, and the extracted feature maps are continuously used in the process of feature fusion to make up for the information lost in the fusion process and promote information flow. In addition, the strategy of detail injection is used to directly learn the details that need to be injected into the upsampled multispectral image, which simplifies the model learning process and improves its physical interpretability, solving the problem that the existing method does not fully utilize the unique information in the two types of source images and the model interpretability is not strong, resulting in low performance of the fusion result.

[0006] The technical solution adopted by the present invention is a double-branch network full color sharpening method based on detail injection, which is specifically implemented according to the following steps:

[0007] Step 1: Divide the original data into training, verification and test data, generate corresponding data sets and perform preprocessing;

[0008] Step 2: Construct a double-branch pan-sharpening network model based on detail injection;

[0009] Step 3, training a dual-branch full-color sharpening network model based on detail injection;

[0010] Step 4: Use the trained network model to fuse the test data;

[0011] Step 5: Evaluate the quality of the fused image.

[0012] The present invention is also characterized in that:

[0013] Step 1 is implemented according to the following steps:

[0014] Step 1.1: Given multiple sets of original multispectral MS images and panchromatic PAN images, each set of multispectral images contains a three-dimensional matrix The full-color image is a two-dimensional matrix in is a real number domain symbol, H and W are the height and width of the panchromatic image, B is the number of bands of the multispectral image, r represents the ratio of the spatial resolution of the multispectral image to the panchromatic image, and the original multispectral image and the panchromatic image are randomly divided into training data, verification data, and test data in a ratio of 8:1:1;

[0015] Step 1.2: For each set of images in the training data and validation data, filter the original multispectral MS image using a filter that matches the modulation transfer function (MTF) of the sensor that acquires the multispectral image, then downsample the filtered original multispectral MS image using the nearest neighbor interpolation method to obtain a spatially degraded original multispectral MS image, and downsample the panchromatic PAN image using the bicubic interpolation method to obtain a spatially degraded panchromatic PAN image. Upsample the spatially degraded original multispectral MS image to make the spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image consistent in size. Use the spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image as the simulated input data, and use the original multispectral MS image as the reference image, and crop the simulated input data and the reference image into corresponding small-sized image blocks according to the set step size to generate a training set and validation set with a large amount of data.

[0016] Step 1.3: For the generated training set and validation set, the simulated input data and the reference image are normalized by dividing by the maximum pixel grayscale value, and then the normalized full-color PAN image is low-pass filtered to obtain a low-pass version of the normalized full-color PAN image, and the high-frequency component of the normalized full-color PAN image is obtained by subtracting the low-pass version of the normalized full-color PAN image from the full-color PAN image (this process is called high-pass filtering in the network structure diagram).

[0017] Step 1.4: For the test data, two types of simulated test data and real test data are generated respectively. The method of generating simulated test data is: perform spatial degradation operations on the original MS and PAN images respectively, then upsample the degraded MS image, then normalize the upsampled degraded MS image and the degraded PAN image, and perform high-pass filtering on the obtained PAN image to obtain the high-frequency components of the degraded PAN image, and use the normalized upsampled degraded MS image and the high-frequency components of the degraded PAN image as simulated test data. The method of generating real test data is: upsample the original MS image, then normalize the upsampled MS image and the original PAN image, and then perform high-pass filtering on the obtained PAN image to obtain the high-frequency components of the original PAN image. The high-frequency components of the normalized upsampled MS image and PAN image are used as real test data.

[0018] Step 2 is implemented as follows:

[0019] Step 2.1, construct two parallel feature extraction subnetworks a and b with similar structures. Each subnetwork consists of the first convolution layer, the first activation function, the second convolution layer, the second activation function, the third convolution layer and the third activation function. The convolution kernel size of the first convolution layer is set to 1×1, and the convolution kernel size of the second and third convolution layers is set to 3×3. The number of convolution kernels in each convolution layer is 32, the sliding stride is 1, and the activation function uses the rectified linear unit ReLU.

[0020] Step 2.2, splicing the output feature maps of the two parallel feature extraction subnetworks in step 2.1 along the channel direction and sending them to the feature fusion subnetwork, the feature fusion subnetwork is composed of three consecutive residual units, and the structure of each residual unit is composed of the first convolutional layer, the first activation function, the second convolutional layer, the second activation function and the third convolutional layer, and the input of the first convolutional layer and the output of the third convolutional layer are element-wise added. After obtaining the outputs of the first and second residual units, the outputs of the two residual units are respectively spliced ​​with the output feature maps of the first convolutional layer of the two-branch feature extraction subnetwork, and the spliced ​​data is used as the input of the next residual unit. The convolution kernel size of the first convolutional layer is set to 1×1, and the number of convolution kernels is 64. The convolution kernel size of the second and third convolutional layers is 3×3, and the number is 64 and 32 respectively. The sliding stride of the convolution kernel of each layer is 1, and the activation function adopts the rectified linear unit ReLU;

[0021] Step 2.3, send the output of the feature fusion subnetwork to the image reconstruction subnetwork. The structure of the image reconstruction subnetwork consists of the first convolutional layer, the first activation function and the second convolutional layer. Set the convolution kernel size of the first convolutional layer to 1×1, the number of convolution kernels to 32, the convolution kernel size of the second convolutional layer to 3×3, the number to 4 or 8, the sliding stride of the convolution kernel is 1, and the activation function uses the rectified linear unit ReLU;

[0022] Step 2.4: Add the upsampled multispectral image to the output of the image reconstruction subnetwork to obtain the final fused image.

[0023] Step 3 is implemented as follows:

[0024] Step 3.1, the feature extraction subnetwork a, feature extraction subnetwork b, feature fusion subnetwork and image reconstruction subnetwork in step 2 are combined to obtain the overall network structure model. The specific combination method is to first parallelize the feature extraction subnetwork a and the feature extraction subnetwork b, then serialize the feature fusion subnetwork, and finally serialize the image reconstruction subnetwork. Then, the training set and test set generated in step 1 are used to conduct supervised training on the constructed overall network structure model. The loss function used in the training process is

[0025] L=L M +α·L s +β·L λ (1)

[0026] Where L M , L S , L λ are the mean square error term between the network output and the reference image, the spatial loss term between the network output and the input full-color PAN image, and the spectral loss term between the network output and the input original multispectral MS image.

[0027] Step 3.2, the three loss terms are defined as follows:

[0028]

[0029]

[0030]

[0031] In the formula, M is the output image and reference image of the network, m and p are the degraded original multispectral MS and panchromatic PAN images, N is the batch size of the training samples, and repmat is the process of expanding the p image along the band direction to the same Operations with the same number of bands, f degrade Indicates the use of spatial degradation operation with MTF filtering on the image;

[0032] Step 3.3, use the Adam method to update the model parameters, set the learning rate during training to 0.001, and stop training until the loss value converges or reaches the set maximum number of iterations to obtain a trained detail injection-based dual-branch panchromatic sharpening network model.

[0033] Step 4 is implemented according to the following steps:

[0034] The simulated test data and real test data generated in step 1 are respectively input into the detail injection-based dual-branch panchromatic sharpening network model trained in step 3 to obtain the corresponding simulated fusion results and real fusion results. The pixel values ​​of the obtained fusion results are clipped to the [0,1] interval, and then denormalized by multiplying the maximum pixel grayscale value to obtain the final fusion result.

[0035] Step 5 is implemented according to the following steps:

[0036] For the simulated fused image, the original multispectral MS image is used as a reference, and the quality of the fused image obtained in step 4 is evaluated by the evaluation index; for the real fused image, the fused image obtained in step 4 is compared with the original MS or full-color PAN image, and the corresponding evaluation index is calculated to evaluate the quality of the fused image.

[0037] The beneficial effects of the present invention are as follows: first, the model uses two parallel branch sub-networks to extract features from two source images respectively, and continuously applies the acquired feature information to the subsequent fusion process, so that the model can make full use of the spatial and spectral information contained in the input image, improve information reusability and promote information flow, and overcome the problems of insufficient information utilization and spatial information loss in the fusion process caused by using MS and PAN images as network input in the prior art. Second, the model learns the detail image that needs to be compensated to the upsampled LRMS image, and introduces the prior knowledge of full color sharpening while simplifying the network learning process, and improves the physical interpretability of the model, overcoming the problems of high complexity of the learning process and weak model interpretability caused by directly learning the ideal fused image. Third, the model uses a large number of 1×1 convolution kernels, which reduces the number of model parameters while maintaining the spatial information of the image or feature map. Fourth, a new loss function was designed when training the network model. On the basis of the commonly used mean square error loss function, constraints on the spectral and spatial information of the fused image were added, which strengthened the spectral and spatial consistency between the fused image and the input image, and effectively alleviated the problems of high-frequency detail loss and large spectral distortion of the fused image that exist in training the network using the mean square error loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart for implementing the present invention.

[0039] Figure 2 This is a structural diagram of the network model used in the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] The double-branch network full color sharpening method based on detail injection is shown in the flowchart Figure 1 As shown, the specific implementation steps are as follows:

[0042] Step 1: Divide the original data into training, verification and test data, generate corresponding data sets and perform preprocessing;

[0043] Step 1 is implemented according to the following steps:

[0044] Step 1.1: Given multiple sets of original multispectral MS images and panchromatic PAN images, each set of multispectral images contains a three-dimensional matrix The full-color image is a two-dimensional matrix in is a real number domain symbol, H and W are the height and width of the panchromatic image, B is the number of bands of the multispectral image, r represents the ratio of the spatial resolution of the multispectral image to the panchromatic image, and the original multispectral image and the panchromatic image are randomly divided into training data, verification data, and test data in a ratio of 8:1:1;

[0045] Step 1.2: For each set of images in the training data and validation data, filter the original multispectral MS image using a filter that matches the modulation transfer function (MTF) of the sensor that acquires the multispectral image, then downsample the filtered original multispectral MS image using the nearest neighbor interpolation method to obtain a spatially degraded original multispectral MS image, and downsample the panchromatic PAN image using the bicubic interpolation method to obtain a spatially degraded panchromatic PAN image. Upsample the spatially degraded original multispectral MS image to make the spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image have the same size, and use the upsampled degraded original multispectral MS image as the input of the feature extraction subnetwork model a in step 2, and use the original multispectral MS image as the reference image. Use the spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image as the simulated input data. Because a large amount of data sets are required when training the overall network structure, the simulated input data and the reference image are cropped into corresponding small-sized image blocks according to the set stride to generate a training set and validation set with a large amount of data.

[0046] Step 1.3: For the generated training set and validation set, the simulated input data and the reference image are normalized by dividing by the maximum pixel grayscale value, and then the normalized full-color PAN image is low-pass filtered to obtain a low-pass version of the normalized full-color PAN image, and the high-frequency component of the normalized full-color PAN image is obtained by subtracting the low-pass version of the normalized full-color PAN image from the full-color PAN image. The high-frequency component of the degraded full-color PAN image is used as the input of the feature extraction sub-network model b in step 2;

[0047] Step 1.4: For the test data, two types of simulated test data and real test data are generated respectively. The method of generating simulated test data is: perform spatial degradation operations on the original MS and PAN images respectively, then upsample the degraded MS image, then normalize the upsampled degraded MS image and the degraded PAN image, and perform high-pass filtering on the obtained PAN image to obtain the high-frequency components of the degraded PAN image, and use the normalized upsampled degraded MS image and the high-frequency components of the degraded PAN image as simulated test data. The method of generating real test data is: upsample the original MS image, then normalize the upsampled MS image and the original PAN image, and then perform high-pass filtering on the obtained PAN image to obtain the high-frequency components of the original PAN image. The high-frequency components of the normalized upsampled MS image and PAN image are used as real test data.

[0048] Step 2: Refer to the attached Figure 2 Construct a dual-branch pan-sharpening network model based on detail injection;

[0049] Step 2 is implemented as follows:

[0050] Step 2.1, construct two parallel feature extraction subnetworks a and b with similar structures. Each subnetwork consists of the first convolution layer, the first activation function, the second convolution layer, the second activation function, the third convolution layer and the third activation function. The convolution kernel size of the first convolution layer is set to 1×1, and the convolution kernel size of the second and third convolution layers is set to 3×3. The number of convolution kernels in each convolution layer is 32, the sliding stride is 1, and the activation function uses the rectified linear unit ReLU.

[0051] Step 2.2, splicing the output feature maps of the two parallel feature extraction subnetworks in step 2.1 along the channel direction and sending them to the feature fusion subnetwork, the feature fusion subnetwork is composed of three consecutive residual units, and the structure of each residual unit is composed of the first convolutional layer, the first activation function, the second convolutional layer, the second activation function and the third convolutional layer, and the input of the first convolutional layer and the output of the third convolutional layer are element-wise added. After obtaining the outputs of the first and second residual units, the outputs of the two residual units are respectively spliced ​​with the output feature maps of the first convolutional layer of the two-branch feature extraction subnetwork, and the spliced ​​data is used as the input of the next residual unit. The convolution kernel size of the first convolutional layer is set to 1×1, and the number of convolution kernels is 64. The convolution kernel size of the second and third convolutional layers is 3×3, and the number is 64 and 32 respectively. The sliding stride of the convolution kernel of each layer is 1, and the activation function adopts the rectified linear unit ReLU;

[0052] Step 2.3, send the output of the feature fusion subnetwork to the image reconstruction subnetwork. The structure of the image reconstruction subnetwork consists of the first convolutional layer, the first activation function and the second convolutional layer. Set the convolution kernel size of the first convolutional layer to 1×1, the number of convolution kernels to 32, the convolution kernel size of the second convolutional layer to 3×3, the number to 4 or 8 (determined by the number of bands of the multispectral image), the sliding stride of the convolution kernel is 1, and the activation function uses the rectified linear unit ReLU;

[0053] Step 2.4: Add the upsampled multispectral image to the output of the image reconstruction subnetwork to obtain the final fused image.

[0054] Step 3, training a dual-branch full-color sharpening network model based on detail injection;

[0055] Step 3 is implemented as follows:

[0056] Step 3.1, the feature extraction subnetwork a, feature extraction subnetwork b, feature fusion subnetwork and image reconstruction subnetwork in step 2 are combined to obtain the overall network structure model. The specific combination method is to first parallelize the feature extraction subnetwork a and the feature extraction subnetwork b, then serialize the feature fusion subnetwork, and finally serialize the image reconstruction subnetwork. Then, the training set and validation set generated in step 1 are used to conduct supervised training on the constructed overall network structure model. The loss function used in the training process is

[0057] L=L M +α·L s +β·L λ (1)

[0058] Where L M , L S , L λ are the mean square error term between the network output and the reference image, the spatial loss term between the network output and the input full-color PAN image, and the spectral loss term between the network output and the input original multispectral MS image.

[0059] Step 3.2, the three loss terms are defined as follows:

[0060]

[0061]

[0062]

[0063] In the formula, M is the output image and reference image of the network, m and p are the degraded original multispectral MS and panchromatic PAN images, N is the batch size of the training samples, and repmat is the process of expanding the p image along the band direction to the same Operations with the same number of bands, f degrade Indicates the use of spatial degradation operation with MTF filtering on the image;

[0064] Step 3.3, use the Adam method to update the model parameters, set the learning rate during training to 0.001, and stop training until the loss value converges or reaches the set maximum number of iterations to obtain a trained detail injection-based dual-branch panchromatic sharpening network model.

[0065] Step 4: Use the trained network model to fuse the test data;

[0066] Step 4 is implemented according to the following steps:

[0067] The simulated test data and real test data generated in step 1 are respectively input into the detail injection-based dual-branch panchromatic sharpening network model trained in step 3 to obtain the corresponding simulated fusion results and real fusion results. The pixel values ​​of the obtained fusion results are clipped to the [0,1] interval, and then denormalized by multiplying the maximum pixel grayscale value to obtain the final fusion result.

[0068] Step 5: Evaluate the quality of the fused image.

[0069] Step 5 is implemented according to the following steps:

[0070] For the simulated fused image, the original multispectral MS image is used as a reference, and the quality of the fused image obtained in step 4 is evaluated by the evaluation index; for the real fused image, the fused image obtained in step 4 is compared with the original MS or full-color PAN image, and the corresponding evaluation index is calculated to evaluate the quality of the fused image.

[0071] The technical effects of the present invention are further described below in conjunction with simulation experiments.

[0072] 1. The hardware platform used in the simulation experiment of the present invention is: the CPU model is Intel (R) Core (TM) i7-10700, the main frequency is 2.90GHz, and the memory is 32GB; the software platform is: the system is Windows 10 Home Chinese version, the programming language is Python3.6, and the deep learning framework is Tensorflow 1.15.0. The data sets used in the simulation experiment of the present invention are the Ikonos data set obtained by the Ikonos satellite at Mount Wellington in Tasmania, Australia, and the WorldView3 data set obtained by the WorldView3 satellite in Bangkok, Thailand. The spatial resolutions of the multispectral images and panchromatic images provided by the Ikonos dataset are 4m and 1m, respectively, and the sizes are 3287*3031 and 13148*12124, respectively. The multispectral images have 4 spectral bands; the spatial resolutions of the multispectral images and panchromatic images provided by the WorldView3 dataset are 0.31m and 1.24m, respectively, and the sizes are 3159*3571 and 12636*14284, respectively. The multispectral images have 8 spectral bands, and the PAN images of the two datasets are all single spectral bands. The original data of the two datasets are cropped into 200*200 MS image blocks and 800*800 PAN image blocks with a step size of 200, and the image blocks do not overlap with each other. The training set, validation set, and test set are generated for the cropped original data in the manner described in step 1. The data set divisions of the two satellite sensors are shown in Table 4-2.

[0073] Table 1. Division of the dataset (the sizes of MS and PAN images are shown in brackets)

[0074]

[0075] 2. Simulation experiment and result analysis

[0076] Simulation experiment 1:

[0077] The proposed method introduces a loss function that constrains the fused image according to the spectral or spatial information contained in the input image. The main idea is to alleviate the problem of high-frequency detail loss and large spectral distortion of the fused image caused by training the network with the mean square error loss function. Therefore, the simulation experiment 1 of the present invention fuses the simulated test data on the Ikonos data set, and compares the fused image quality when the network is trained with the proposed loss function and the mean square error loss function under the same network model, so as to verify the effectiveness of the proposed loss function.

[0078] To evaluate the quality of the fused image, the following six indicators are used to quantify the quality of the fused image.

[0079] (1) Spectral Angle Mapper (SAM): measures the spectral similarity between the fused image and the corresponding reference image. Its value is a positive number greater than or equal to 0. The smaller the value, the higher the spectral similarity between the two images.

[0080] (2) Relative global comprehensive error ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse): It measures the global spectral quality of the fused image. Its value is a positive number greater than or equal to 0. The smaller the value, the smaller the spectral difference between the two images.

[0081] (3) Relative Average Spectral Error (RASE): It reflects the global spectral quality of the fused image. Its value is a positive number greater than or equal to 0. The smaller the value, the lower the degree of spectral distortion of the fused image.

[0082] (4) Spatial Correlation Coefficient (SCC): It measures the similarity of spatial details between the fused image and the reference image. Its value is between 0 and 1. The closer the value is to 1, the stronger the spatial correlation between the two images and the higher the spatial quality of the fused image.

[0083] (5) General image quality metric Q: It measures the correlation, average brightness similarity, and contrast similarity between the fused image and the reference image. Its value is between 0 and 1. The closer the value of Q is to 1, the higher the quality of the fused image.

[0084] (6) Structural Similarity (SSIM): It can globally evaluate the similarity between two images from three aspects: brightness, contrast, and structure. Its value is between 0 and 1. The closer the value is to 1, the higher the similarity between the two images.

[0085] The above six indicators are used to evaluate the quality of the fusion results obtained by training the network using the above two loss functions. The indicator values ​​are shown in Table 2.

[0086] Table 2 Comparison of the proposed loss function and the mean square error loss function in terms of fusion result quality

[0087]

[0088] As can be seen from Table 2, the spatial quality and spectral quality of the fused image obtained by training the network using the proposed loss function have been improved compared with the MSE loss function, indicating the effectiveness of the proposed loss function. This is because the introduction of spatial constraints and spectral consistency in the proposed loss function enables the model to obtain additional spatial and spectral information clues from the input PAN and MS images during the optimization process. These clues are conducive to the model to mine effective information or features from the input image, thereby enhancing the spatial consistency and spectral consistency of the fused image and the input image.

[0089] Simulation experiment 2:

[0090] To verify the effectiveness of the method proposed in the present invention, it is compared with three existing panchromatic sharpening methods based on convolutional neural networks under the same hardware and software conditions and data sets. The three methods are:

[0091] 1) Yang et al. proposed PanNet, a fusion method for high-frequency areas of the image to be fused, in the paper "PanNet: a deep network architecture for pan-sharpening".

[0092] 2) DiCNN1, a shallow network fusion method based on detail injection, proposed by He et al. in the paper “Pansharpening via Detail Injection Based Convolutional Neural Networks”.

[0093] 3) Deng et al. proposed a deep network fusion method FusionNet based on detail injection in the paper "Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening".

[0094] The experiment types are divided into simulation experiments and real experiments. The data to be fused for the two types of experiments are generated as described in step 1. In the simulation experiment, the above six indicators are used to evaluate the quality of the fused image; in the real experiment, the following three indicators are used to evaluate the quality of the fused image:

[0095] 1) Spectral distortion index D λ : Determines the spectral similarity between the fused image and the observed MS image. Its value is a positive number greater than 0. The smaller the value, the more similar the spectral information of the fused image is to the original MS image.

[0096] 2) Spatial distortion index D S: Measures the spatial similarity between the fused image and the original PAN image. Its value is a positive number greater than 0. The smaller the value, the smaller the spatial distortion of the fused image.

[0097] 3) Quality no-reference index (QNR): It jointly measures the spectral distortion and spatial distortion of the fused image. λ and D S The two indicators are calculated together, and their values ​​are in the interval [0, 1]. The larger the value, the higher the spectral and spatial quality of the fused image.

[0098] The proposed method and the above three existing methods were respectively subjected to simulation experiments and real experiments on the Ikonos and WorldView3 datasets. The index values ​​of the corresponding fusion results are shown in Tables 3, 4, and 5.

[0099] Table 3 Comparison of simulation experiment results on Ikonos dataset

[0100]

[0101] Table 4 Comparison of simulation experiment results on WorldView3 dataset

[0102]

[0103] Table 5 Comparison of real experimental results on Ikonos and WorldView3 datasets

[0104]

[0105] For the simulation experiment, it can be seen from Tables 3 and 4 that on the Ikonos and WorldView3 datasets, the fusion results produced by the method proposed in the present invention have achieved a more obvious quality improvement compared with the three existing technologies; in the real experiment, it can be seen from Table 5 that the fusion results produced by the method proposed in the present invention on the Ikonos dataset have achieved a more ideal quality, and also achieved competitive results on the WorldView3 dataset.

[0106] The method proposed in the present invention uses two parallel branch networks to extract features from two types of source images respectively, and continuously reuses the extracted features in the process of feature fusion to continuously inject spatial and spectral information from the source images. This method can effectively and fully utilize the spatial and spectral information contained in the two images, improve the reusability of information and promote information flow. At the same time, the method introduces a loss function with spatial constraints and spectral constraints, which can enhance the spectral consistency and spatial consistency between the fused image and the two types of input images. Therefore, the proposed method has a superior performance.

[0107] Based on the results of simulation experiments 1 and 2, the method proposed in the present invention effectively solves the problem that the prior art does not fully utilize the unique information contained in the two types of source images, resulting in low quality of fusion results. At the same time, it effectively alleviates the problems of high-frequency detail loss and severe spectral distortion of the fused image that exist when the network is trained using the commonly used mean square error loss function.

Claims

1. A dual-branch network full-color sharpening method based on detail injection, characterized in that: Follow the steps below to implement it: Step 1: Divide the original data into training, verification and test data, generate corresponding data sets and perform preprocessing; The step 1 is specifically implemented according to the following steps: Step 1.1: Given multiple sets of original multispectral MS images and panchromatic PAN images, each set of multispectral images contains a three-dimensional matrix The full-color image is a two-dimensional matrix in is the real number field symbol, H , W are the height and width of the full-color image, respectively. B is the number of bands of the multispectral image, r Represents the ratio of spatial resolution between multispectral and panchromatic images. The original multispectral image and panchromatic image are randomly divided into training data, validation data, and test data in a ratio of 8:1:1; Step 1.2: For each group of images in the training data and the validation data, the original multispectral MS image is filtered by a filter that matches the modulation transfer function (MTF) of the sensor that acquires the multispectral image, and then the filtered original multispectral MS image is downsampled by the nearest neighbor interpolation method to obtain a spatially degraded original multispectral MS image. At the same time, the panchromatic PAN image is downsampled by the bicubic interpolation method to obtain a spatially degraded panchromatic PAN image. The spatially degraded original multispectral MS image is upsampled to make the spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image have the same size. The spatially degraded original multispectral MS image and the spatially degraded panchromatic PAN image are used as simulation input data, and the original multispectral MS image is used as a reference image. The simulation input data and the reference image are cropped into corresponding small-sized image blocks according to the set step size to generate a training set and a validation set. Step 1.3, for the generated training set and validation set, normalize the simulated input data and the reference image by dividing by the maximum pixel grayscale value, and then low-pass filter the normalized full-color PAN image to obtain a low-pass version of the normalized full-color PAN image, and obtain the high-frequency component of the normalized full-color PAN image by subtracting the low-pass version of the normalized full-color PAN image from the full-color PAN image; Step 1.4, for the test data, two types of simulated test data and real test data are generated respectively; the method of generating the simulated test data is: performing spatial degradation operations on the original MS and PAN images respectively, and then upsampling the degraded MS image, and then normalizing the upsampled degraded MS image and the degraded PAN image, and performing high-pass filtering on the obtained PAN image to obtain the high-frequency components of the degraded PAN image, and the high-frequency components of the normalized upsampled degraded MS image and the degraded PAN image are used as simulated test data, and the method of generating the real test data is: upsampling the original MS image, and then normalizing the upsampled MS image and the original PAN image, and then performing high-pass filtering on the obtained PAN image to obtain the high-frequency components of the original PAN image, and the high-frequency components of the normalized upsampled MS image and the PAN image are used as real test data; Step 2: Construct a double-branch pan-sharpening network model based on detail injection; The step 2 is specifically implemented according to the following steps: Step 2.1, construct two parallel feature extraction subnetworks a and b with similar structures. Each subnetwork consists of the first convolution layer, the first activation function, the second convolution layer, the second activation function, the third convolution layer and the third activation function. The convolution kernel size of the first convolution layer is set to 1×1, and the convolution kernel size of the second and third convolution layers is set to 3×3. The number of convolution kernels in each convolution layer is 32, the sliding stride is 1, and the activation function uses the rectified linear unit ReLU. Step 2.2, splicing the output feature maps of the two parallel feature extraction subnetworks in step 2.1 along the channel direction and sending them to the feature fusion subnetwork, the feature fusion subnetwork is composed of three consecutive residual units, and the structure of each residual unit is composed of the first convolutional layer, the first activation function, the second convolutional layer, the second activation function and the third convolutional layer, and the input of the first convolutional layer and the output of the third convolutional layer are element-wise added. After obtaining the outputs of the first and second residual units, the outputs of the two residual units are respectively spliced ​​with the output feature maps of the first convolutional layer of the two-branch feature extraction subnetwork, and the spliced ​​data is used as the input of the next residual unit. The convolution kernel size of the first convolutional layer is set to 1×1, and the number of convolution kernels is 64. The convolution kernel size of the second and third convolutional layers is 3×3, and the number is 64 and 32 respectively. The sliding stride of the convolution kernel of each layer is 1, and the activation function adopts the rectified linear unit ReLU; Step 2.3, send the output of the feature fusion subnetwork to the image reconstruction subnetwork. The structure of the image reconstruction subnetwork consists of the first convolutional layer, the first activation function and the second convolutional layer. Set the convolution kernel size of the first convolutional layer to 1×1, the number of convolution kernels to 32, the convolution kernel size of the second convolutional layer to 3×3, the number to 4 or 8, the sliding stride of the convolution kernel is 1, and the activation function uses the rectified linear unit ReLU; Step 2.4, add the upsampled multispectral image to the output of the image reconstruction subnetwork to obtain the final fused image; Step 3, training a dual-branch full-color sharpening network model based on detail injection; Step 4: Use the trained network model to fuse the test data; Step 5: Evaluate the quality of the fused image.

2. The method for panchromatic sharpening based on double-branch network with detail injection according to claim 1, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1, the feature extraction subnetwork a, feature extraction subnetwork b, feature fusion subnetwork and image reconstruction subnetwork in step 2 are combined to obtain the overall network structure model. The specific combination method is to first parallelize the feature extraction subnetwork a and the feature extraction subnetwork b, then serialize the feature fusion subnetwork, and finally serialize the image reconstruction subnetwork. Then, the training set and validation set generated in step 1 are used to conduct supervised training on the constructed overall network structure model. The loss function used in the training process is L=L M +α·L S +β·L λ (1) Where L M , L S , L λ are the mean square error term between the network output and the reference image, the spatial loss term between the network output and the input full-color PAN image, and the spectral loss term between the network output and the input original multispectral MS image. Step 3.2, the three loss terms are defined as follows: In the formula, M is the output image and reference image of the network, m and p are the degraded original multispectral MS and panchromatic PAN images, N is the batch size of the training samples, and repmat is the process of expanding the p image along the band direction to the same Operations with the same number of bands, f degrade Indicates the use of spatial degradation operation with MTF filtering on the image; Step 3.3, use the Adam method to update the model parameters, set the learning rate during training to 0.001, and stop training until the loss value converges or reaches the set maximum number of iterations to obtain a trained detail injection-based dual-branch panchromatic sharpening network model.

3. The method for panchromatic sharpening based on double-branch network with detail injection according to claim 2, characterized in that: The step 4 is specifically implemented according to the following steps: The simulated test data and real test data generated in step 1 are respectively input into the detail injection-based dual-branch panchromatic sharpening network model trained in step 3 to obtain the corresponding simulated fusion results and real fusion results. The pixel values ​​of the obtained fusion results are clipped to the [0,1] interval, and then denormalized by multiplying the maximum pixel grayscale value to obtain the final fusion result.

4. The method for panchromatic sharpening based on double-branch network with detail injection according to claim 3, characterized in that: The step 5 is specifically implemented according to the following steps: For the simulated fused image, the original multispectral MS image is used as a reference, and the quality of the fused image obtained in step 4 is evaluated by the evaluation index; for the real fused image, the fused image obtained in step 4 is compared with the original MS or full-color PAN image, and the corresponding evaluation index is calculated to evaluate the quality of the fused image.

Images