A panchromatic sharpening method based on pix2pix shading concept

By introducing the concept of pix2pix shading and the absolute difference loss function of spectrum in the full color sharpening method, the problems of information loss and model complexity during feature extraction are solved, and higher quality image generation and robustness are achieved.

CN118396887BActive Publication Date: 2024-11-29SOUTHWEST FORESTRY UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410213529.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-27
Publication Date
2024-11-29
Estimated Expiration
2044-02-27

AI Technical Summary

Technical Problem

The existing full-color sharpening methods are prone to problems such as information loss, insufficient model generalization ability, overfitting and computational complexity during feature extraction, resulting in poor output image quality.

Method used

Using the full-color sharpening method based on the concept of pix2pix coloring, the histogram equalization preprocessing is achieved through the conversion of RGB color space and LAB color space and the limited contrast of the histogram, combined with the absolute difference loss function of spectrum, skip feature extraction, optimize model training, and make The model focuses more on the high and low frequency information of the image.

Benefits of technology

Improve the accuracy of image generation, make the output image closer to the original image, avoid spatial distortion, and enhance the robustness of the model and image processing effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118396887B_ABST
    Figure CN118396887B_ABST
Patent Text Reader

Abstract

The present invention relates to a panchromatic sharpening method based on the concept of pix2pix coloring, and belongs to the field of remote sensing image fusion. The present invention comprises S1: performing color space conversion and contrast-limited histogram equalization on an image; S2: inputting the original multispectral image in step S1 and the final pre-colored image as paired training data into a pix2pix model for training, and during the training process, using the spectrum absolute difference loss function as a conditional loss; S3: testing the pix2pix model after training, and outputting a panchromatic sharpened image. The present invention ensures that the model can learn an accurate color mapping relationship by performing color space conversion and contrast-limited histogram equalization on the image, so that the image output by the model has a higher color similarity with the original image; by defining the conditional loss from the frequency domain perspective, the model pays more attention to the structure and texture of the image, and has stronger robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image fusion and relates to a full color sharpening method based on the concept of pix2pix coloring. Background Art

[0002] Remote sensing image fusion (also known as pan-sharpening) is to extract spatial and spectral features from high-resolution panchromatic images (PAN) and low-resolution multispectral images (MS), and then fuse them to obtain high-resolution multispectral images. Traditional pan-sharpening methods can be divided into three categories: component substitution (CS), multiresolution analysis (MRA), and variational optimization (VO)-based methods. CS-based methods aim to separate the spatial and spectral information of low-resolution MS images in a suitable space and further fuse them with the corresponding PAN images. CS-based methods include IHS method, principal component analysis (PCA) method, and Gram-Schmidt (GS) method. MRA-based methods decompose PAN images into multi-scale spaces, extract spatial structures, and then inject them into MS to achieve fusion, such as DWT, MTF-GLP, etc. VO-based methods regard the pan-sharpening task as an ill-posed problem, and the goal is to minimize the loss.

[0003] With the successful application of deep learning technology in various computer vision tasks, a pan-sharpening algorithm based on convolutional neural network (CNN) was proposed. The PNN network was the first to use deep learning methods to fuse panchromatic and multispectral images. This method used CNN to extract effective information of panchromatic and multispectral images for the first time. However, the network structure is relatively simple with only three layers, and the nonlinear fitting ability is weak, which easily causes spectral distortion. Later, PanNet introduced ResNet to deepen the network depth, strengthen the nonlinear fitting ability, and alleviate the spectral distortion to a certain extent. As a result, more and more pan-sharpening based on deep learning have emerged, such as SRPPNN.

[0004] Although CNN has achieved good performance in visual tasks, the effectiveness of the CNN method is related to the depth of the network. The deeper the network, the better the effect. However, as the depth increases, training becomes very difficult, which seriously limits the performance of the full color sharpening network.

[0005] At the same time, the methods described above all deal with the panchromatic sharpening problem from the perspective of feature extraction. In the feature extraction process, there are problems such as loss of original information, insufficient model generalization ability, overfitting of the model and high computational complexity.

[0006] Therefore, it is necessary to provide a full color sharpening method based on the concept of pix2pix coloring, skipping feature extraction to avoid the impact of problems existing in the feature extraction process on the output image quality, while improving the accuracy of the pix2pix model learning of training data, making the model learning results more accurate, thereby outputting an image that is closer to the original image. Summary of the invention

[0007] In order to overcome the problems in the background technology, the present invention proposes a panchromatic sharpening method based on the concept of pix2pix coloring. By converting the multispectral image (MS) and the panchromatic image (PAN) into RGB color space and LAB color space and combining contrast-limited histogram equalization, the model can learn accurate color mapping relationships during training; by optimizing the spectrum absolute difference loss function, the model is constrained from the frequency domain perspective, so that the model pays more attention to the high and low frequency information of the image. Thereby improving the image processing effect of the model and outputting an image closer to the original image.

[0008] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0009] The panchromatic sharpening method comprises the following steps:

[0010] S1: Obtain a multispectral image and a downsampled panchromatic image of the same object, perform color space conversion and contrast-limited histogram equalization preprocessing on the multispectral image and the panchromatic image to obtain a pre-colored image. MS and PAN can be obtained through satellite photography.

[0011] S2: The multispectral image and the pre-colored image in step S1 are input as paired training data into the pix2pix model for training. During the training process, the spectrum absolute difference loss function is used as the conditional loss.

[0012] S3: The pix2pix model is tested after training and the pan-sharpened image is output.

[0013] Preferably, the spectrum absolute difference loss function is:

[0014]

[0015] Among them, G(x,z) is the output image generated by the generator network receiving input x and noise z, represents the Fourier transform of the target image y in the frequency domain, represents the Fourier transform of the generated image G(x,z) in the frequency domain, ||·||1 represents the L1 norm, that is, the sum of absolute values, Represents the expectation for all combinations of input x, target image y and noise z. The generator network G receives input x and noise z, generates image G(x,z), and the Fourier transform of target image y in the frequency domain is F(y). The Fourier transform of generated image G(x,z) in the frequency domain is F(G(x,z)), and the absolute spectral difference is calculated, that is, ||F(y)-F(G(x,z)||1, and the expectation is taken for all combinations of input x, target image y and noise z. The goal of this spectral absolute difference loss function is to minimize the Manhattan distance between the Fourier transforms of the generated image and the target image in the frequency domain, so that the loss function defines the conditional loss from the frequency domain perspective.

[0016] Preferably, in step S1, the specific preprocessing method is: converting the multispectral image and the panchromatic image from the RGB color space to the LAB color space, and matching the brightness channels of the bands of the multispectral image and the panchromatic image in the LAB color space, and then redistributing the gray levels of the image pixels under the restriction of the restriction function. After completion, converting the image from the LAB color space back to the RGB color space to obtain a pre-colored image.

[0017] Since MS has three bands and PAN is a single band, it is necessary to copy the PAN band three times and pre-process it with MS. The conversion of RGB color space to LAB color space can be achieved through gamma correction. The limit function limits the redistribution of pixel gray levels, thereby avoiding over-enhancement and maintaining the naturalness of the image and the comfort of visual perception. The gray levels of image pixels are redistributed so that the histogram of the image is more evenly distributed in gray levels, thereby enhancing the contrast of the image.

[0018] By preprocessing MS and PAN, a pre-colored image can be obtained. Compared with grayscale images, this image is colored and the color is relatively accurate. Pairing this pre-colored image with the original MS for training can provide color guidance for the model, so that the model can learn a more accurate color mapping relationship. The trained model can realize image colorization to realize the application of the colorization concept in the process of image full color sharpening.

[0019] Preferably, the restriction function is:

[0020]

[0021] Among them, g(v) is the output pixel value, v is the input pixel value, m is the parameter that controls the starting point of contrast enhancement, α is the parameter that controls the rate of contrast enhancement, and c is the normalization coefficient to ensure that the output pixel value is within a reasonable range.

[0022] Beneficial effects of the present invention:

[0023] 1. The present invention introduces the concept of coloring into the full-color sharpening process by converting the training data into the RGB color space and the LAB color space and performing contrast-limited histogram equalization processing, thereby ensuring that the model can learn the accurate color mapping relationship, so that the model can assign the correct color to the full-color image during the image generation process, and the image output by the model has a higher similarity with the original image.

[0024] 2. In the process of performing full-color sharpening processing on the image, the present invention skips the extraction of spatial information of the full-color image, effectively avoiding the occurrence of spatial distortion and affecting the quality of the model output image.

[0025] 3. The present invention defines conditional loss from the frequency domain perspective, so that the model pays more attention to the structure and texture of the image and has stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process of full-color sharpening processing and a test flow chart of the present invention, wherein the upper frame is the overall process schematic diagram, and the lower frame is the test flow chart.

[0027] Figure 2 It is a schematic diagram of the training data preprocessing process according to an embodiment of the present invention.

[0028] Figure 3 It is a schematic diagram of model training according to an embodiment of the present invention.

[0029] Figure 4 It is a comparison diagram of fused images obtained by the method of the present invention and other methods. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with specific embodiments.

[0031] Example 1

[0032] The MS and downsampled PAN obtained in this embodiment are as follows: Figure 2 As shown, the MS and PAN are converted from RGB color space to LAB color space, and then the MS and PAN bands are matched for brightness channels. After that, the grayscale level of the image pixel is limited by the function The image is then converted from the LAB color space back to the RGB color space to obtain the pre-colored image used as training data. Figure 2 shown.

[0033] like Figure 3 As shown, the above MS and the pre-colored image obtained after processing are input into the pix2pix model for training. During the training process, the spectrum absolute difference loss function is:

[0034] After the pix2pix model training is completed, the output image is as follows Figure 3 shown.

[0035] Example 2

[0036] In this embodiment, another image is selected and the method of the present invention is used to perform full color sharpening on the image. The processing effect is as follows: Figure 4 shown.

[0037] In this comparative example, the MS and PAN in Example 2 are used to perform full color sharpening processing on the image by six methods: Gram-Schmidt transform (GS), band-dependent spatial detail method (BDSD), principal component analysis (PCA), intensity, hue, saturation transform (IHS), generalized Laplace pyramid method based on modulation transfer function (MTF-GLP), and adaptive threshold wavelet transform (ATWT). The processing effect is as follows: Figure 4 shown.

[0038] pass Figure 4 It can be seen that, from the perspective of spectrum, the images processed by GS, BDSD, IHS and PCA methods, especially the images processed by BDSD, GS and PCA methods, have serious spectrum distortion, and their colors are brighter or darker than the MS images in the QB test set. From the perspective of spatial structure, the images processed by the method of the present invention are most similar to the PAN image, because the method of the present invention skips the part of spatial feature extraction and can better preserve the spatial information of the original PAN image.

[0039] The images processed in Example 2 and the comparative example were quantitatively tested, and the results are shown in Table 1.

[0040] Table 1

[0041]

[0042] It can be seen from Table 1 that the image processed by the method of the present invention has achieved the best results in both D_s and QNR, because the method of the present invention skips the extraction of the spatial information of the full-color image, thereby effectively avoiding spatial distortion. At the same time, the image processed by the method of the present invention also won the third place in SAM. And D_lambda is only listed as one of the parameters for calculating QNR. The data in Table 1 can show that the image processed and output by the method of the present invention has excellent effect.

[0043] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A full color sharpening method based on the concept of pix2pix coloring, Features: The panchromatic sharpening method comprises the following steps: S1: Obtain a multispectral image and a downsampled panchromatic image of the same object, perform color space conversion and contrast-limited histogram equalization preprocessing on the multispectral image and the panchromatic image to obtain a pre-colored image; The specific method of the preprocessing is: converting the multispectral image and the panchromatic image from the RGB color space to the LAB color space, and matching the brightness channels of the bands of the multispectral image and the panchromatic image in the LAB color space, and then redistributing the gray levels of the image pixels under the restriction of the restriction function, and after completion, converting the image from the LAB color space back to the RGB color space to obtain a pre-colored image; S2: The multispectral image and the pre-colored image in step S1 are input as paired training data into the pix2pix model for training. During the training process, the spectrum absolute difference loss function is used as the conditional loss. S3: The pix2pix model is tested after training and the pan-sharpened image is output.

2. A panchromatic sharpening method based on the pix2pix shading concept according to claim 1, Features: The spectrum absolute difference loss function is: Where: G(x,z) is the output image generated by the generator network receiving input x and noise z, represents the Fourier transform of the target image y in the frequency domain, Represents the Fourier transform of the generated image G(x,z) in the frequency domain, ∥·∥ 1 represents the Manhattan distance, that is, the sum of absolute values, represents the expectation for all combinations of input x, target image y, and noise z.

3. A panchromatic sharpening method based on the pix2pix shading concept according to claim 1, Features: The restriction function is: Among them, g(v) is the output pixel value, v is the input pixel value, m is a parameter that controls the starting point of contrast enhancement, α is a parameter that controls the rate of contrast enhancement, and c is a normalization coefficient.

Citation Information

Patent Citations

  • Hyperspectral and panchromatic image fusion method for extracting spatial features based on AAE

    CN110533620A

  • Panchromatic sharpening method and system based on multi-scale delay channel attention network

    CN114549366A