CNN-DI mixed fusion method and system for multispectral pan-sharpening
By designing details injected into hybrid networks and adaptive correlation residual networks in multispectral image pan-sharpening, the problems of sampling loss and low model interpretability are solved, high-quality image fusion is achieved, and the details of remote sensing images are significantly improved. fidelity and spatial spectral consistency.
Patent Information
- Application Number
- CN202410738636.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-06-07
AI Technical Summary
The existing multispectral image pan-sharpening methods have problems such as sampling loss, low model interpretability, ignoring adjacent spatial information and spectral correlation, and insufficient processing context information, resulting in poor reliability and quality of fusion results.
A three-stage hybrid network based on the idea of detail injection is designed, including a hybrid expert network for detail injection, an adaptive correlation residual network and a global detail injection network. The network is built through the adaptive channel attention mechanism and upsampled spectral reconstruction network to establish space and spectrum correlation, and use multi-scale high-frequency information module to reduce detail blur and artifact phenomena.
Effectively eliminate the difference in the size ratio of the source image, reduce spectral information loss, improve the detail fidelity and spatial spectral consistency of the fusion results, and significantly improve the fusion effect of remote sensing images.
Smart Images

Figure CN119006296A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image fusion, and in particular to a CNN-DI hybrid fusion method and system for multi-spectral pan-sharpening. Background Art
[0002] High-resolution multispectral (HRMS) images are essential research data in the field of remote sensing because they can provide rich information about land objects. However, due to production costs and industrial technology limitations, HRMS images cannot be directly obtained. Currently, researchers mainly use pan-sharpening technology, that is, fusing panchromatic images (PAN) and low-resolution multispectral (LRMS) images to obtain HRMS images. Among them, the detailed information provided by PAN images can help detect and analyze small-scale land features, such as buildings, road networks, etc. At the same time, LRMS images can provide spectral characteristics of different bands, which can be used for tasks such as land object classification, vegetation monitoring, and environmental change analysis.
[0003] Deep learning-based and traditional fusion methods are widely used in existing pan-sharpening fusion methods. Whether it is traditional methods or deep learning-based methods, their common goal is to extract actual features from different single-source images and generate fused images through designed fusion strategies or network models. Traditional methods are mainly based on detail injection methods and degradation model-based methods. Degradation model-based methods usually require complex modeling of factors such as lighting and materials in the scene. However, the characteristics of these processes may vary depending on different sensors, different environmental conditions, and different data sets. The above factors will affect the reliability of pan-sharpening results. In contrast, the detail injection method effectively avoids the above shortcomings because it directly processes the pixels of the image. However, the detail injection method will lead to unsatisfactory fusion results due to the subjectivity of parameter selection and the complexity of calculation.
[0004] The pan-sharpening method based on deep learning can improve the shortcomings of traditional methods by introducing convolutional neural networks (CNNs) and utilizing their powerful feature learning and representation capabilities. However, compared with traditional methods, the interpretability and principle are weaker. In recent years, in order to give full play to the advantages of different methods, researchers have tried to combine traditional methods with deep learning to design hybrid network architectures. However, they still have certain limitations: (1) Directly upsampling low-resolution multispectral images and then training them with the same size as PAN images ignores the sampling loss caused by the ratio difference between the original low-resolution multispectral images and PAN images; 2) When processing remote sensing data of different resolutions from the same satellite, separate training is required, which reduces the interpretability of the model to a certain extent; 3) When injecting details into LRMS images, they are usually directly replaced with PAN image information, ignoring that adjacent spatial information and spectra are related; 4) For the collection of high-frequency information of PAN images, only simple convolution operations are used, which is relatively weak in processing contextual information. Summary of the invention
[0005] The purpose of the present invention is to propose a CNN-DI hybrid fusion method and system for multi-spectral pan-sharpening. In order to achieve the remote sensing image fusion task, a new three-stage hybrid network based on the detail injection idea is designed.
[0006] According to a first aspect of an embodiment of the present disclosure, a CNN-DI hybrid fusion method for multi-spectral pan-sharpening is provided, comprising the following steps:
[0007] Get low-resolution multispectral image MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0008] The low-resolution multispectral image MS and the image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0009] The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0010] The feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0011] The feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0012] The feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0013] Furthermore, the full-color image is downsampled to obtain P ↓2 ∈R m / 2×n / 2×b and P ↓4 ∈R m / 4×n / 4×b , specifically:
[0014] P ↓2 =Conv3(DP(P))
[0015] P ↓4 =Conv3(DP(P ↓2 ))
[0016] Where DP(·) and Conv3(·) represent the downsampling operation of bicubic interpolation and the 3×3 convolution operation, respectively. ↓2 and ↓4 represent downsampling to 1 / 2 and 1 / 4 of the original image size.
[0017] Furthermore, the gating network output of the detail injection hybrid expert network is:
[0018] G(x) i =τ(Conv1(AP(Concat(MS,P ↓4 ))))
[0019] Where G(x) i represents the output of the gating network, i = 0, 1; τ(·) represents the softmax function;
[0020] The outputs of the panchromatic image expert model and multispectral image expert model in the detail-injected hybrid expert network are:
[0021]
[0022]
[0023] where ρ(·) represents the PReLU activation function, represents the convolution multiplication; then extract and The high-frequency component of the feature represents the spatial information and is related to G(x) i The components are multiplied as follows:
[0024]
[0025] in Represents the high-frequency component output; finally, Output from expert model with multispectral image Add as follows:
[0026]
[0027] Among them G mde Representation details are injected into the hybrid expert network output.
[0028] Furthermore, the panchromatic image expert model includes a 1×1 convolution, a PReLU activation function, and a 3×3 convolution; and the multispectral image expert model includes a ReLU activation function, a 3×3 convolution, a PReLU activation function, and a 3×3 convolution.
[0029] Furthermore, the feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z, specifically: feature map G mde ∈R m×n×B First, upsampling is achieved through 3×3 convolution and pixel shuffling, as shown below:
[0030] f ↑2 =SP(Conv3(G mde ))
[0031] where f ↑2 ∈R 2m×2n×B represents the upsampled features, ↑2 means the upsampled image is twice the size of the original image; SP(·) represents the pixel shuffle operation; Conv3(·) represents the 3×3 convolution operation;
[0032] Use adaptive global pooling layer to learn upsampled features f ↑2 After getting The upsampled feature f ↑2 After the convolution layer and mean operation, the integer is make and Multiply them to get feature X, and this process is defined as adaptive convolution pooling ACP(·):
[0033] X=ACP(f ↑2 )
[0034] where X∈R B×1 As a channel descriptor; channels are mapped through convolution kernel activation operations:
[0035] Z=δ(Conv1(σ(Conv1(X))))⊙f ↑2
[0036] where Z∈R 2m×2n×Brepresents the output of the first spectral reconstruction network, Conv1(·) represents a 1×1 convolution operation, σ(·) and δ(·) represent activation functions ReLU and sigmoid, respectively, and ⊙ represents pixel multiplication.
[0037] Furthermore, the spatial-spectral adaptation module uses dynamic convolution to operate from both spatial and spectral levels as follows:
[0038] Branch spa =Cony3(σ(Conv3(Conv3(δ(S)))))
[0039] Branch Spe =FN(Conv1(σ(Conv1(AP(S)))))
[0040] Among them, Branch Spa ∈R 2m×2n×B and Branch Spe ∈R k×k×B denotes spatial and spectral dynamic convolution branches, respectively. k×k denotes the kernel size. ρ(·) denotes the PReLU activation function. Conv1(·) denotes a 1×1 convolution operation. Conv3(·) denotes a 3×3 convolution operation. σ(·) and δ(·) denote activation functions ReLU and sigmoid, respectively. FN(·) denotes a Filter Norm operation. Then Branch Spa 、Branch Spe And then process with feature group S:
[0041]
[0042] Where S ssa represents the feature group after mathematical transformation operation, represents convolution multiplication, ⊙ represents pixel multiplication, and finally a difference operation is performed as follows:
[0043] F ACR =Conv3(σ(S sa ))+S1
[0044] Among them, F ACR ∈R 2m×2n×B It represents the feature set output by the adaptive correlation residual network.
[0045] Furthermore, the feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′, specifically: feature map F ACR ∈R 2m×2n×B Upsampling is achieved through 3×3 convolution and pixel shuffling as follows:
[0046] f ↑2 ′=SP(Conv3(F ACR ))
[0047] where f ↑2 ′∈R 4m×4n×B represents the output feature group, ↑2 means upsampling to twice the original image; SP(·) represents the pixel shuffle operation; Conv3(·) represents the 3×3 convolution operation;
[0048] Use adaptive global pooling layer to learn upsampled features f ↑2 'After getting The upsampled feature f ↑2 ′After the convolution layer and mean operation, it is reshaped into Will and Multiply them to get X′, and this process is defined as adaptive convolution pooling ACP(·):
[0049] X′=ACP(f ↑2 )
[0050] where X′∈R B×1 As a channel descriptor; channels are mapped through convolution kernel activation operations:
[0051] Z′=v(Conv1(σ(Conv1(X′))))⊙f ↑2 '
[0052] where Z′∈R 4m×4n×B represents the output of the second spectral reconstruction network, Conv1(·) represents a 1×1 convolution operation, σ(·) and δ(·) represent activation functions ReLU and sigmoid, respectively, and ⊙ represents pixel multiplication.
[0053] Furthermore, the feature map Z′ is input into the global detail injection network to obtain the final image fusion result, specifically: feature map Z′∈R 4m×4n×B The high-frequency components are obtained through 1×1 convolution and average pooling operations. The high-frequency information of the full-color image containing texture details is obtained using a multi-scale high-frequency information module, which is combined with the high-frequency components of the feature map Z′ to reduce detail blur and artifacts. All the combined information is fused through 3×3 convolution and softmax activation function, and finally two 3×3 convolution blocks are added to the global detail injection network to obtain the final fusion result, and the global detail injection network is embedded in a residual network.
[0054] Furthermore, we design a loss function: first upsample the source multispectral image:
[0055] MS↑4 =UP ↑4 (MS)
[0056] Among them, MS ↑4 is the up-sampled feature, UP ↑4 It represents the upsampling operation of scale=4; then the spectral loss optimization is performed, and the expression is as follows:
[0057] L Spec =||HS-MS ↑4 ||1
[0058] Where ||·||1 represents the L1 loss function;
[0059] The upsampled features MS ↑4 The full-color image P is first processed by the Sobel operator, and the maximum value of the two is selected to obtain the optimal texture information for comparison with the fused image, as follows:
[0060] G Max =MAX(Sobel(MS ↑4 ), Sobel(P)
[0061] Among them G Max represents the most prominent texture information part in the source image, MAX(·) represents the maximum value selection, and Sobel(·) represents the Sobel operator; then G Max The fused image HS is sent to the gradient loss for calculation:
[0062] L Grad =||Sobel(HS)-G Max ||1
[0063] Where ||·||1 represents the L1 norm, and Sobel (HS) is the fused image after calculation by the Sobel operator;
[0064] Considering the spectral loss and gradient loss between the fused image and the source image, the total loss function is:
[0065] L total =α×L Spec +β×L Grad
[0066] Among them, α and β are weight coefficients, which are used to balance the contribution of spectral and gradient losses.
[0067] According to a second aspect of an embodiment of the present disclosure, a CNN-DI hybrid fusion system for multi-spectral pan-sharpening is provided, comprising:
[0068] Image acquisition module, used to acquire low-resolution multispectral images MS∈Rm×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0069] Feature map G mde Acquisition module, the low-resolution multispectral image MS and image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0070] The feature map Z acquisition module, the feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0071] Feature Group F ACR Acquisition module, the feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0072] The feature map Z′ acquisition module, the feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0073] Fusion module: the feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0074] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the CNN-DI hybrid fusion method for multi-spectral pan-sharpening is implemented.
[0075] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the CNN-DI hybrid fusion method for multi-spectral pan-sharpening is implemented.
[0076] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: 1. In order to realize the remote sensing image fusion task, the present invention designs a three-stage hybrid network based on the detail injection idea. In addition, a spectral reconstruction network with adaptive channel attention mechanism and upsampling is proposed to eliminate the size ratio difference between source images while reducing the loss of spectral information.
[0077] 2. In the hybrid expert network based on detail injection, independent sub-models handle different sub-tasks, and the fusion task is divided into two sub-tasks: full-resolution pan-sharpening and reduced-resolution pan-sharpening.
[0078] 3. Adaptive correlation residual network with residual structure, looking for the correlation between space and spectrum. The spatial information source based on different source images is the same, and once the correlation is established, it will not change due to changes in scale.
[0079] 4. In order to obtain an ideal fusion result, the present invention designs a global detail injection network; utilizes the acquired high-frequency information and combines it with the high-frequency information of the output result of the previous stage to reduce detail blur and artifacts in the fusion result. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.
[0081] Figure 1 It is the overall flow chart of the CNN-DI hybrid fusion method for multi-spectral pan-sharpening;
[0082] Figure 2 Infuse details into the hybrid expert network structure graph;
[0083] Figure 3 Reconstruct the network structure diagram for the first spectrum;
[0084] Figure 4 It is the structure diagram of the adaptive correlation residual network;
[0085] Figure 5 Reconstructing the network structure diagram for the second spectrum;
[0086] Figure 6 Infuse the network structure graph with global details;
[0087] Figure 7 The figure is a qualitative comparison between this method and other advanced fusion methods on the GF-2, IKONOS and WorldView-II datasets. Specific implementation methods
[0089] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.
[0090] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0091] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0092] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.
[0093] Embodiment 1:
[0094] like Figure 1 As shown, this embodiment provides a CNN-DI hybrid fusion method for multi-spectral pan-sharpening, including the following steps:
[0095] Step 1. Obtain the registered low-resolution multispectral image MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0096] Step 2: The low-resolution multispectral image MS and image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0097] Specifically, the proposal of the detail injection hybrid expert network stems from mixing paired full-resolution and reduced-resolution images and training them together to obtain the best weights. The key to solving the problem lies in how to flexibly select parameters, as the information between different resolutions is different. If mixed training is performed directly, the fusion result produced cannot reach the fusion result of the model trained with only a single resolution. To address this problem, each independent sub-model in the detail injection hybrid expert network processes different sub-tasks, and the fusion task is divided into two sub-tasks for full resolution and reduced resolution. Moreover, the hybrid expert network has obvious advantages in processing multimodal images. The spatial-spectral information in the multispectral image is separated by detail injection to obtain a high-quality fused image, so a detail injection hybrid expert network is designed.
[0098] like Figure 2 As shown in the figure, the input data of the detail injection hybrid expert network is the low-resolution multispectral image MS and the image P ↓4 ; For each MS and P ↓4 The designed expert models are also different. The panchromatic image expert model includes 1×1 convolution, PReLU activation function and 3×3 convolution; the multispectral image expert model includes ReLU activation function, 3×3 convolution, PReLU activation function and 3×3 convolution. Then extract the output features of the two models and represents the high-frequency component of spatial information and is related to G(x) i The components are multiplied; finally, Output from the multispectral image expert model Add them together to get G mde , injecting details into the hybrid expert network output.
[0099] Step 3. The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0100] like Figure 3 As shown, considering that the spectral information mainly comes from low-resolution multispectral images, and in subsequent operations, the pseudo-spectral feature maps used are different from the source input. Therefore, a spectral reconstruction network is designed to reduce the loss of spectral information, and a pixel shuffling operation is introduced to achieve upsampling. Spectral information is mainly related to channels, so the channel attention mechanism is usually the first choice. However, the ordinary channel attention mechanism uses fixed weight parameters to weight different channels in the input data, and cannot accurately capture the subtle changes and correlations in the input data. When processing, the spectral and spatial features come from different source images, so the present invention designs an adaptive channel attention mechanism.
[0101] Step 4. The feature map Z and image P ↓2After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0102] like Figure 4 As shown in the figure, the design of the adaptive correlation residual network is based on existing research that shows that adjacent spatial information and spectra are related. If the spatial features of low-resolution multispectral images are directly replaced, the correlation of the spatial spectrum is ignored. Therefore, a spatial-spectral adaptive module is designed to adaptively find the correlation between the two by designing separate spatial and spectral dynamic filters. Of course, this is based on the input data being a pair of well-aligned panchromatic images and low-resolution multispectral images with the same geographic objects. The spatial information source of the source image is the same, and once the correlation is established, it will not change due to changes in scale. At the same time, in order to make the network learn complex mapping relationships more effectively, the spatial-spectral adaptive module is embedded in the residual structure, so an adaptive correlation residual network is designed.
[0103] Step 5. The feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0104] like Figure 5 As shown in Figure 1, considering that the spectral information mainly comes from low-resolution multispectral images, and in subsequent operations, the pseudo-spectral feature maps used are different from the source input, a spectral reconstruction network is designed again to reduce the loss of spectral information and realize upsampling operations.
[0105] Step 6. The feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0106] like Figure 6 As shown in the figure, in order to enhance the global details to improve the final fusion effect, a global detail injection network is designed. In the final stage, the original full-color image is used directly to guide the information injection operation. By taking advantage of the dilated convolution and multi-scale convolution, a multi-scale high-frequency information module is designed to extract the high-frequency information of the full-color image, which mainly includes a multi-scale convolution block that combines dilated convolutions with dilation rates of 1, 3, and 5 with the ReLU activation function and some 3×3 convolutions. In order to accelerate the training convergence of the network, the global detail injection network is embedded in a residual network.
[0107] In order to ensure the quality of the fused image by retaining more meaningful feature information, the present invention mainly considers the spectral loss and gradient loss between the fused image and the source image;
[0108] Spectral loss: The source image is usually regarded as the spectral degradation of the fused image HS, so it can be considered that the smaller the loss between the two, the more consistent the fusion result is with the visual effect. Since the ratio between the two is 4, the source image is first upsampled to make it consistent in size.
[0109] Gradient loss function: Uses the texture information of different source images to ensure that the fusion result retains the spatial information of the source image to the greatest extent.
[0110] The present invention selects test image sequences on the GF-2, IKONOS and WorldView-III datasets to compare with eleven state-of-the-art remote sensing image fusion methods. Figure 7 It shows the overall effect and local feature details. Figure 7 The left side of each group shows the experimental results of the down-resolution image, and the right side shows the experimental results of the full-resolution image. For the down-resolution image, due to the existence of the ground truth image, the spatial and spectral residuals of different fusion methods are also shown above (from left to right). Figure 7 In the first group, the present invention compared the results on the GF-2 dataset. Compared with other methods, the present invention more effectively reduced the interference of noise, while the texture details were clearer, achieved the best spatial and spectral fidelity, and had a smaller error value. The results of the present invention were in the optimal state. The second and third groups are the comparison results between the IKONOS and WorldView-II datasets, respectively. It can be seen from the results that when conducting these two types of comparative experiments, the present invention can retain more significant features in detail processing while avoiding serious spectral distortion problems. This plays a vital role in the practical application of remote sensing images.
[0111] Table 1 is a quantitative comparison between the present invention and other advanced fusion methods on the GaoFen-2 dataset.
[0112]
[0113] In addition to subjective qualitative analysis, the present invention also uses objective quantitative indicators to evaluate the performance of the fusion results. The present invention evaluates the fusion results of down-resolution by using seven currently popular reference metric methods, including CC, FSIM, ERGAS, RASE, SAM, PSNR, and SSIM. The full-resolution experiment is implemented on the original image, and there is no available reference image. Therefore, the most commonly used non-reference image evaluation indicators are used to evaluate the performance of the fused image, including D λ , D SAnd QNR. The present invention uses 150 pairs of untrained full-resolution and reduced-resolution images as test sets to complete different remote sensing image fusion tasks, and the quantitative results are shown in Table 1. Obviously, except for CC and SAM, the present invention obtains suboptimal results on the low-resolution data set, and the other indicators are the best. All evaluation indicators of the present invention in the full-resolution quantitative experiment have reached a high level. It should be noted that when evaluating CC, FSIM, SSIM, and PSNR, the larger the calculated results are, the better the quality of the fused image is. For ERGAS, SAM, and RASE, the smaller the value is, the better the quality of the fused image is. D λ It is used to measure the spectral correlation between the original LRMS image and the HRMS result. S It is used to evaluate the spatial correlation between the original PAN image and the HRMS result. QNR is dependent on D λ and D S A relatively comprehensive measure of the two. QNR can roughly reflect the panchromatic sharpening performance at full resolution. A large QNR indicates better quality of the fused image, while a small QNR indicates better quality of the fused image. λ and D S This means less distortion.
[0114] In the present invention, a CNN-DI hybrid fusion method for multi-spectral pan-sharpening is proposed for performing remote sensing image fusion tasks. In order to achieve the mixed training of paired full-resolution images and reduced-resolution images, a hybrid expert network based on the idea of detail injection is proposed. Among them, each expert model is used to process different subtasks, and the fusion task is divided into two subtasks for full-resolution pan-sharpening and reduced-resolution pan-sharpening. A spectral reconstruction network with an adaptive channel attention module and upsampling is also proposed to eliminate the size ratio difference between source images while reducing the loss of spectral information. Secondly, it is believed that adjacent spatial information and spectra are related and the spatial information source of the source image is the same. Therefore, an adaptive correlation residual network is designed to reduce the information loss problem during detail injection from the essence of the image by finding the correlation between space and spectrum. A global detail injection network is designed to obtain an ideal fusion result. Multi-scale dilated convolution is used to obtain high-frequency information, which is combined with the high-frequency information of the output result of the previous stage to reduce the detail blur and artifact phenomenon in the fusion result. Sufficient experiments have proved that the present invention can generate satisfactory fusion results in subsequent visual tasks and also ensure quantitative indicators. Therefore, the method proposed in the present invention is helpful to the development of remote sensing image fusion.
[0115] Embodiment 2:
[0116] This embodiment provides a CNN-DI hybrid fusion system for multi-spectral pan-sharpening, including:
[0117] Image acquisition module, used to acquire low-resolution multispectral images MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0118] Feature map G mde Acquisition module, the low-resolution multispectral image MS and image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0119] The feature map Z acquisition module, the feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0120] Feature Group F ACR Acquisition module, the feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0121] The feature map Z′ acquisition module, the feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0122] Fusion module: the feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0123] Embodiment three:
[0124] An electronic device includes a memory, a processor, and a computer program stored and running on the memory, wherein the processor implements the above-mentioned CNN-DI hybrid fusion method for multi-spectral pan-sharpening when executing the program, including:
[0125] Get low-resolution multispectral image MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0126] The low-resolution multispectral image MS and the image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0127] The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0128] The feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0129] The feature set F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0130] The feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0131] Embodiment 4:
[0132] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned CNN-DI hybrid fusion method for multi-spectral pan-sharpening, including:
[0133] Get low-resolution multispectral image MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; downsampling operation is performed on the full-color image to obtain image P ↓2 and image P ↓4 ;
[0134] The low-resolution multispectral image MS and the image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ;
[0135] The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z;
[0136] The feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ;
[0137] The feature set F ACR Input to the second spectral reconstruction network to obtain the feature map Z′;
[0138] The feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
[0139] Those skilled in the art should understand that the modules or steps of the present disclosure can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present disclosure is not limited to any specific combination of hardware and software.
[0140] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0141] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.
Claims
1. A CNN-DI hybrid fusion method for multi-spectral pan-sharpening, characterized in that: The following steps are involved: Get low-resolution multispectral image MS∈R m×n×B and the full-color image P∈R 4m×4n×b Where m and n represent the size of the image, B and b represent the channel information; Downsample the full-color image to obtain image P ↓2 and image P ↓4 ; The low-resolution multispectral image MS and the image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ; The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z; The feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ; The feature group F ACR Input into the second spectral reconstruction network to obtain the feature map Z′; The feature map Z′ is input into the global detail injection network to obtain the final image fusion result.
2. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: Downsample the full-color image to obtain P ↓2 ∈R m / 2×n / 2×b and P ↓4 ∈R m / 4×n / 4×b Specifically: P ↓2 =Conv3(DP(P)) P ↓4 =Conv3(DP(P ↓2 )) Where DP(·) and Conv3(·) represent the downsampling operation of bicubic interpolation and the 3×3 convolution operation, respectively. ↓2 and ↓4 represent downsampling to 1 / 2 and 1 / 4 of the original image size.
3. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: The gating network output of the detail injection hybrid expert network is: G(x) i =τ(Conv1(AP(Concat(MS,P ↓4 )))) Where G(x) i represents the output of the gating network, i=0,1; τ(·) represents the softmax function; The outputs of the panchromatic image expert model and multispectral image expert model in the detail-injected hybrid expert network are: where ρ(·) represents the PReLU activation function, represents the convolution multiplication; then extract and The high-frequency component of the feature represents the spatial information and is related to G(x) i The components are multiplied as follows: in Represents the high-frequency component output; finally, Output from expert model with multispectral image Add as follows: Among them G mde Representation details are injected into the hybrid expert network output.
4. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 3, characterized in that: The panchromatic image expert model includes a 1×1 convolution, a PReLU activation function, and a 3×3 convolution; the multispectral image expert model includes a ReLU activation function, a 3×3 convolution, a PReLU activation function, and a 3×3 convolution.
5. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: The feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z, specifically: feature map G mde ∈R m×n×B First, upsampling is achieved through 3×3 convolution and pixel shuffling, as shown below: f ↑2 =SP(Conv3(G mde )) where f ↑2 ∈R 2m×2n×B represents the upsampled features, ↑2 means the upsampled image is twice the size of the original image; SP(·) represents the pixel shuffle operation; Conv3(·) represents the 3×3 convolution operation; the upsampled features f are learned using the adaptive global pooling layer ↑2 After getting The upsampled feature f ↑2 After the convolution layer and mean operation, the integer is make and Multiply them to get feature X, and this process is defined as adaptive convolution pooling ACP(·): X=AACP(f ↑2 ) where X∈R B×1 As a channel descriptor; channels are mapped through convolution kernel activation operations: Z=δ(Conv1(σ(Conv1(X))))⊙f ↑2 where Z∈R 2m×2n×B represents the output of the first spectral reconstruction network, Conv1(·) represents a 1×1 convolution operation, σ(·) and δ(·) represent activation functions ReLU and sigmoid, respectively, and ⊙ represents pixel multiplication.
6. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: The spatial-spectral adaptation module uses dynamic convolution to operate from both spatial and spectral levels as follows: Branch Spa =Conv3(σ(Conv3(Conv3(δ(S)))) Branch Spe =FN((Conv1(o(Conv1(APS)))) Among them, Branch Spa ∈R 2m×2n×B and Branch Spe ∈R k×k×B denotes spatial and spectral dynamic convolution branches, respectively. k×k denotes the kernel size. ρ(·) denotes the PReLU activation function. Conv1(·) denotes a 1×1 convolution operation. Conv3(·) denotes a 3×3 convolution operation. σ(·) and δ(·) denote activation functions ReLU and sigmoid, respectively. FN(·) denotes a FilterNorm operation. Then Branch Spa 、Branch Spe And then process with feature group S: Where S ssa represents the feature group after mathematical transformation operation, represents convolution multiplication, ⊙ represents pixel multiplication, and finally a difference operation is performed as follows: F ACR =Conv3(σ(S ssa ))+S1 where F ACR ∈R 2m×2n×B It represents the feature set output by the adaptive correlation residual network.
7. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: The feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z′, specifically: feature map F ACR ∈R 2m×2n×B Upsampling is achieved through 3×3 convolution and pixel shuffling as follows: f ↑2 ’=SSP(Conv3((F ACR )) where f ↑2 '∈R 4m×4n×B represents the output feature group, ↑2 means upsampling to twice the original image; SP(·) represents the pixel shuffle operation; Conv3(·) represents the 3×3 convolution operation; Use adaptive global pooling layer to learn upsampled features f ↑2 'After getting The upsampled feature f ↑2 'After convolution layer and mean operation, the integer is Will and Multiply them to get X', and this process is defined as adaptive convolution pooling ACP(·): X’=ACP(f ↑2 ) where X'∈R B×1 As a channel descriptor; channels are mapped through convolution kernel activation operations: Z’=δ(Conv1(σ(Conv1(X’))))⊙f ↑2 ’ where Z'∈R 4m×4n×B represents the output of the second spectral reconstruction network, Conv1(·) represents a 1×1 convolution operation, σ(·) and δ(·) represent activation functions ReLU and sigmoid, respectively, and ⊙ represents pixel multiplication.
8. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: The feature map Z′ is input into the global detail injection network to obtain the final image fusion result, specifically: feature map Z′∈R 4m×4n×B The high-frequency components are obtained through 1×1 convolution and average pooling operations; the high-frequency information of the full-color image containing texture details is obtained using a multi-scale high-frequency information module, and it is combined with the high-frequency components of the feature map Z′; all the combined information is fused through 3×3 convolution and softmax activation function, and finally two 3×3 convolution blocks are added to the global detail injection network to obtain the final fusion result, and the global detail injection network is embedded in a residual network.
9. The CNN-DI hybrid fusion method for multi-spectral pan-sharpening according to claim 1, characterized in that: Design loss function: First upsample the source multispectral image: MS ↑4 =UP ↑4 (MS) Among them, MS ↑4 is the up-sampled feature, UP ↑4 It represents the upsampling operation of scale=4; then the spectral loss optimization is performed, and the expression is as follows: L Spec =|||HS-MS ↑4 |||1 Where ||·||1 represents the L1 loss function; The upsampled features MS ↑4 The full-color image P is first processed by the Sobel operator, and the maximum value of the two is selected to obtain the optimal texture information for comparison with the fused image, as follows: G Max =MAX(Sobel(MS ↑4 ),Sobel(P)) Among them G Max represents the most prominent texture information part in the source image, MAX(·) represents the maximum value selection, and Sobel(·) represents the Sobel operator; then G Max The fused image HS is sent to the gradient loss for calculation: L Grad =||Sobel(HS)-G Max |||1 Where ||·||1 represents the L1 norm, and Sobel (HS) is the fused image after calculation by the Sobel operator; Considering the spectral loss and gradient loss between the fused image and the source image, the total loss function is: L total =α×L Spec +β×L Grad Among them, α and β are weight coefficients, which are used to balance the contribution of spectral and gradient losses.
10. CNN-DI hybrid fusion system for multi-spectral pan-sharpening, characterized in that: include: Image acquisition module, used to acquire low-resolution multispectral images MS∈R m×n×B and the full-color image P∈R 4m×4n×b , where m and n represent the size of the image, B and b represent the channel information; Downsample the full-color image to obtain image P ↓2 and image P ↓4 ; Feature map G mde Acquisition module, the low-resolution multispectral image MS and image P ↓4 Input to the detail injection hybrid expert network to obtain the feature map G mde ; The feature map Z acquisition module, the feature map G mde Input into the first spectral reconstruction network to obtain the feature map Z; Feature Group F ACR Acquisition module, the feature map Z, image P ↓2 After the residual block is connected, the feature group S is obtained, which is input into the space-spectrum adaptive module of the adaptive correlation residual network to establish the correlation between space and spectrum. The adaptive correlation residual network outputs the feature group F ACR ; The feature map Z' acquisition module, the feature group F ACR Input to the second spectral reconstruction network to obtain the feature map Z'; Fusion module: the feature map Z' is input into the global detail injection network to obtain the final image fusion result.
Citation Information
Patent Citations
Hyper-spectral and panchromatic image fusion method for extracting multi-scale spatial-spectral characteristics based on AE
CN112634137A
Remote sensing image fusion sharpening method based on progressive cross-scale neural network
CN114140359A
Remote sensing panchromatic sharpening method and system based on cross spectrum-space fusion network
CN117274093A
Panchromatic Sharpening Method of Spectral Image Based on Fusion of Overall Structural Information and Spatial Detail Information
US20140301659A1
Systems and Methods for Multi-Spectral Image Fusion Using Unrolled Projected Gradient Descent and Convolutinoal Neural Network
US20200302249A1
Cited By
Panchromatic sharpening image fusion method and device based on double-domain flexible converter
CN120807318A
