A remote sensing image fusion method combining ratio transformation and distribution conversion

Abstract

This invention discloses a remote sensing image fusion method combining ratio transformation and distribution transformation, comprising: performing mean filtering on a panchromatic image and an upsampled multispectral image to obtain high-frequency components of the panchromatic image and the upsampled multispectral image; based on this, obtaining the missing high-frequency details in the multispectral image, denoted as the first high-frequency details; performing standard normalization on the first high-frequency details to obtain the second high-frequency details; calculating the mean and standard deviation of each pixel in each channel of the upsampled multispectral image; concatenating the upsampled multispectral image and the first high-frequency details and inputting them into a convolutional network to generate two affine transformation parameters; injecting the obtained mean and standard deviation into the second high-frequency details to generate high-frequency details with the same distribution as the upsampled multispectral image; and combining this with the upsampled multispectral image to obtain the final fused image. This method solves the problems of spectral distortion and detail distortion in existing remote sensing image fusion algorithms.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, and in particular to a remote sensing image fusion method that combines ratio transformation and distribution transformation. Background Technology

[0002] With the rapid development of my country's aerospace technology, more and more satellites carrying various sensors are being launched into space, and remote sensing images are playing an increasingly important role in Earth observation. Limited by satellite payload capacity, for panchromatic and multispectral images acquired by the same satellite, panchromatic images have higher spatial resolution but lower spectral resolution (usually only one band), while multispectral images have lower spatial resolution but higher spectral resolution (usually more than four bands). In practical applications of remote sensing images, multispectral and panchromatic images are not used simultaneously. A fusion strategy is typically adopted to combine them into a single multispectral image with high spatial resolution. High spatial resolution multispectral images are characterized by possessing both the spectral resolution of multispectral images and the spatial resolution of panchromatic images, allowing for clear identification of small ground features and facilitating environmental monitoring and disaster prevention. Remote sensing image fusion technology is an important research direction in multi-source remote sensing data processing, involving interdisciplinary fields such as sensor technology, signal processing, computer applications, and image processing. It is widely used in urban planning, geographic exploration, vegetation and agricultural assessment, military defense, and environmental pollution control, and has significant practical implications for the development of my country's remote sensing industry.

[0003] Existing remote sensing image fusion algorithms are numerous and, while meeting fusion requirements to some extent, each has its own shortcomings. The most significant problem is that the fused image still suffers from distortion. Image distortion is categorized into spectral distortion and detail distortion. Methods based on component substitution and multi-resolution analysis suffer from spectral distortion, while deep learning-based methods suffer from detail distortion. The presence of distortion means that the fused image cannot be used directly like the original image; its inherent biases must be considered, thus limiting its application value.

[0004] Therefore, how to solve the problems of spectral distortion and detail distortion in existing remote sensing image fusion algorithms has become a key research issue. Summary of the Invention

[0005] In view of the above problems, the present invention provides a remote sensing image fusion method that combines ratio transformation and distribution transformation to at least solve some of the above technical problems. The method generates a low-resolution panchromatic image through a deep neural network, and then performs a ratio operation with the panchromatic image to generate high-frequency details missing in the multispectral image, thereby solving the problem of detail distortion caused by excessive grayscale difference between the low-resolution image and the panchromatic image. The high-frequency details are transformed into spectral gain factors that conform to the distribution of the multispectral image (i.e., high-frequency details that are distributed in the same way as the upsampled multispectral image) through a gain algorithm, and then injected into the multispectral image to generate a fused image, thereby greatly preserving the spectral information of the multispectral image; finally, a better high-resolution multispectral fused image is obtained.

[0006] This invention provides a remote sensing image fusion method combining ratio transformation and distribution transformation, comprising:

[0007] 1. A remote sensing image fusion method combining ratio transformation and distribution transformation, characterized in that it includes:

[0008] S1. Perform mean filtering on the panchromatic image and the upsampled multispectral image respectively to obtain the corresponding high-frequency components of the panchromatic image and the upsampled multispectral image.

[0009] S2. Based on the panchromatic image, the upsampled multispectral image, the high-frequency component of the panchromatic image, and the high-frequency component of the upsampled multispectral image, obtain the missing high-frequency details in the multispectral image, denoted as the first high-frequency details;

[0010] S3. Perform standard normalization on the first high-frequency detail to obtain the standard normalized high-frequency detail, which is denoted as the second high-frequency detail; and calculate the mean and standard deviation of each pixel in each channel of the upsampled multispectral image.

[0011] S4. The upsampled multispectral image and the first high-frequency detail are stitched together and then input into the convolutional network to generate the first affine transformation parameters and the second affine transformation parameters.

[0012] S5. Based on the first affine transformation parameters and the second affine transformation parameters, the mean and standard deviation of each pixel in each channel of the upsampled multispectral image are injected into the second high-frequency detail to generate a high-frequency detail that is distributed in the same way as the upsampled multispectral image, which is denoted as the third high-frequency detail.

[0013] S6. Based on the third high-frequency details and combined with the upsampled multispectral image, the final fused image is obtained.

[0014] Furthermore, S1 specifically includes:

[0015] S11. Acquire panchromatic and multispectral images;

[0016] S12. Upsample the multispectral image to obtain an upsampled multispectral image of the same scale as the panchromatic image;

[0017] S13. Perform mean filtering convolution calculations on the panchromatic image and the upsampled multispectral image respectively to obtain the corresponding low-frequency components of the panchromatic image and the upsampled multispectral image.

[0018] S14. Subtract the panchromatic image and the upsampled multispectral image from their respective low-frequency components to obtain the high-frequency components of the panchromatic image and the upsampled multispectral image.

[0019] Furthermore, S2 specifically includes:

[0020] S21. The high-frequency components of the upsampled multispectral image and the high-frequency components of the panchromatic image are concatenated and simultaneously input into a convolutional network to generate the high-frequency components of a low-resolution panchromatic image.

[0021] S22. The upsampled multispectral image and the panchromatic image are stitched together and simultaneously input into the network to generate a low-resolution panchromatic image;

[0022] S23. Add the high-frequency components of the low-resolution panchromatic image to the low-resolution panchromatic image to obtain the corrected low-resolution panchromatic image;

[0023] S24. Perform a ratio transformation on the panchromatic image and the low-resolution panchromatic image to obtain the missing details in the multispectral image, which are denoted as the first high-frequency details.

[0024] Furthermore, in step S3, the first high-frequency detail undergoes standard normalization processing, specifically including:

[0025] (1) Calculate the mean value of each pixel in the first high-frequency detail, expressed by the formula:

[0026]

[0027] Where, μ detail P represents the mean value of each pixel in the first high-frequency detail; detail Indicates the first high-frequency detail; P detail y,x This represents the value of each pixel in the first high-frequency detail; H represents the image height; W represents the width; x represents the horizontal coordinate of the pixel; y represents the vertical coordinate of the pixel;

[0028] (2) Calculate the standard deviation of each pixel in the first high-frequency detail; the formula is as follows:

[0029]

[0030] Where, σ detail ε represents the standard deviation of each pixel in the first high-frequency detail; ε is a constant.

[0031] (3) Based on the mean and standard deviation of each pixel in the first high-frequency detail, the first high-frequency detail is subjected to standard normalization to obtain the standard normalized high-frequency detail, which is denoted as the second high-frequency detail; the calculation formula is expressed as:

[0032]

[0033] in, This represents the high-frequency details after standard normalization, i.e., the second high-frequency details.

[0034] Further, in step S3, calculating the mean and standard deviation of each pixel in each channel of the upsampled multispectral image specifically includes:

[0035] (1) Calculate the mean value of each pixel in each channel of the upsampled multispectral image; the formula is expressed as:

[0036]

[0037] in, H represents the mean value of each pixel in each channel of the upsampled multispectral image; W represents the image height; x represents the x-coordinate of the pixel; y represents the y-coordinate of the pixel. This represents the pixel value at position (x, y) in channel c;

[0038] (2) Calculate the standard deviation of each pixel in each channel of the upsampled multispectral image based on the mean of each pixel in each channel; the formula is expressed as:

[0039]

[0040] Furthermore, in S4, the first affine transformation parameter is used to adjust the mean value of each pixel in each channel of the upsampled multispectral image;

[0041] The second affine transformation parameter is used to adjust the standard deviation of each pixel in each channel of the upsampled multispectral image.

[0042] Furthermore, the third high-frequency detail is represented as follows:

[0043]

[0044] in, This indicates the third high-frequency detail in channel c; This represents the standard deviation of pixels in the c-th channel of the upsampled multispectral image; γ represents the pixel mean of the c-th channel in the upsampled multispectral image; β represents the first affine transformation parameter; and γ represents the second affine transformation parameter.

[0045] Furthermore, the final fused image is represented as follows:

[0046]

[0047] in, This represents the fused image of the c-th channel; This indicates the third high-frequency detail in channel c; This represents the upsampled multispectral image of channel c.

[0048] Compared with existing technologies, the remote sensing image fusion method combining ratio transformation and distribution transformation described in this invention has the following beneficial effects:

[0049] 1. This invention improves the preservation of detail texture and spectral information, and solves the problem of detail distortion caused by excessive grayscale difference between low-resolution images and panchromatic images.

[0050] 2. Good fusion results can be achieved under different band ranges of panchromatic images and multispectral images.

[0051] 3. This invention achieves image fusion based on deep learning fusion method, and solves the problem of image distortion in traditional fusion method and traditional deep learning fusion method.

[0052] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a schematic diagram of the remote sensing image fusion method combining ratio transformation and distribution transformation provided in an embodiment of the present invention.

[0056] Figure 2This is a schematic diagram illustrating the generation process of a low-resolution panchromatic image provided in an embodiment of the present invention.

[0057] Figure 3 A diagram illustrating the distributed conversion gain process provided in an embodiment of the present invention. Detailed Implementation

[0058] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0059] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0060] See Figure 1 As shown, this embodiment of the invention provides a remote sensing image fusion method combining ratio transformation and distribution transformation, specifically including the following steps:

[0061] S1. Perform mean filtering on the panchromatic image and the upsampled multispectral image respectively to obtain the corresponding high-frequency components of the panchromatic image and the upsampled multispectral image.

[0062] S2. Based on the panchromatic image, the upsampled multispectral image, the high-frequency component of the panchromatic image, and the high-frequency component of the upsampled multispectral image, obtain the missing high-frequency details in the multispectral image, denoted as the first high-frequency details;

[0063] S3. Perform standard normalization on the first high-frequency detail to obtain the standard normalized high-frequency detail, which is denoted as the second high-frequency detail; and calculate the mean and standard deviation of each pixel in each channel of the upsampled multispectral image.

[0064] S4. The upsampled multispectral image and the first high-frequency detail are stitched together and then input into the convolutional network to generate the first affine transformation parameters and the second affine transformation parameters.

[0065] S5. Based on the first affine transformation parameters and the second affine transformation parameters, the mean and standard deviation of each pixel in each channel of the upsampled multispectral image are injected into the second high-frequency detail to generate a high-frequency detail that is distributed in the same way as the upsampled multispectral image, which is denoted as the third high-frequency detail.

[0066] S6. Based on the third high-frequency details and combined with the upsampled multispectral image, the final fused image is obtained.

[0067] The following sections will provide a detailed explanation of each of the above steps.

[0068] Step S1 above specifically includes:

[0069] S11. Acquire the panchromatic image P and multispectral image M collected by the target satellite;

[0070] S12. Upsample the multispectral image M to obtain an upsampled multispectral image of the same scale as the panchromatic image P. In this embodiment of the invention, the multispectral image M is specifically upsampled by 4 times;

[0071] S13, process the panchromatic image P and the upsampled multispectral image respectively. Perform mean filtering and convolution calculations to obtain the corresponding low-frequency component P of the panchromatic image. L and low-frequency components of upsampled multispectral images The convolution kernel K and the formula are as follows;

[0072]

[0073]

[0074] Where ⊙ represents the convolution operation;

[0075] S14. Use the panchromatic image P and the upsampled multispectral image respectively. Subtracting the corresponding low-frequency components from the corresponding high-frequency components yields the high-frequency component P of the panchromatic image. h and high-frequency components of upsampled multispectral images This can be expressed by the formula:

[0076]

[0077] Step S2 above specifically includes:

[0078] S21. Upsample the high-frequency components of the multispectral image. and panchromatic image high frequency component P h The high-frequency components (I) of a low-resolution panchromatic image are concatenated and simultaneously input into a convolutional network to generate the image. L h The formula is as follows:

[0079]

[0080] S22, Upsample the multispectral image The panchromatic image P is concatenated with the image P and simultaneously input into the network to generate a low-resolution panchromatic image I. L l ;

[0081]

[0082] S23, convert the high-frequency components of the low-resolution panchromatic image I L h Add to low-resolution panchromatic image I L l The process involves adding details and ultimately generating a corrected low-resolution panchromatic image I. L ;

[0083] The process is as follows Figure 2 As shown, the formula is as follows:

[0084] I L =I L l +I L h

[0085] S24. Combine the panchromatic image P and the low-resolution panchromatic image I. L Perform ratio transformation to obtain the missing high-frequency details P in the multispectral image. detail , recorded as the first high-frequency detail P detail The formula is as follows:

[0086] P detail =P / I L

[0087] In step S3 above, the first high-frequency detail P is... detail To remove the distribution characteristics, i.e., to perform standard normalization, specifically includes:

[0088] (1) Calculate the first high-frequency detail P detail The mean μ of each pixel detail The calculation formula is expressed as:

[0089]

[0090] Where, μ detail Indicates the first high-frequency detail P detail The mean value of each pixel in the image; P detail y,xThis represents the value of each pixel in the first high-frequency detail; H represents the image height; W represents the width; x represents the horizontal coordinate of the pixel; y represents the vertical coordinate of the pixel;

[0091] (2) Calculate the first high-frequency detail P detail The standard deviation σ of each pixel detail The calculation formula is expressed as follows:

[0092]

[0093] Where, σ detail Indicates the first high-frequency detail P detail The standard deviation of each pixel in the image; ε represents a constant;

[0094] (3) Based on the first high-frequency detail P detail The mean μ of each pixel detail and standard deviation σ detail For the first high-frequency detail P detail Perform standard normalization to obtain high-frequency details. Recorded as the second high-frequency detail The calculation formula is expressed as follows:

[0095]

[0096] Simultaneously, the same operation is performed on the multispectral image to obtain the mean value containing global characteristic information of the multispectral image. and standard deviation Calculate the mean value of each pixel in each channel of the upsampled multispectral image. and standard deviation Specifically, it includes:

[0097] (1) Calculate the mean value of each pixel in each channel of the upsampled multispectral image. The formula is expressed as:

[0098]

[0099] in, H represents the mean value of each pixel in each channel of the upsampled multispectral image; W represents the image height; x represents the x-coordinate of the pixel; y represents the y-coordinate of the pixel. This represents the pixel value at position (x, y) in channel c;

[0100] (2) Based on the mean value of each pixel in each channel of the upsampled multispectral image Calculate the standard deviation of each pixel in each channel of the upsampled multispectral image. The formula is expressed as:

[0101]

[0102] In step S4 above, the upsampled multispectral image is... and the first high-frequency detail P generated detail The data is concatenated together, then passed through a convolutional network, and then through two separate convolutional networks to generate values ​​for adjusting the mean. and standard deviation The first affine transformation parameter β and the second affine transformation parameter γ.

[0103] In step S5 above, based on the first affine transformation parameter β and the second affine transformation parameter γ, the mean value of each pixel in each channel of the upsampled multispectral image is calculated. and standard deviation Injected into the second high-frequency detail In the process, high-frequency details that are co-distributed with the upsampled multispectral image are generated. Recorded as the third high-frequency detail The process is as follows Figure 3 As shown, the formula is as follows:

[0104]

[0105] in, This indicates the third high-frequency detail in channel c; This represents the standard deviation of pixels in the c-th channel of the upsampled multispectral image; This represents the pixel mean of the c-th channel in the upsampled multispectral image;

[0106] In step S6 above, based on the third high-frequency detail Combined with the upsampled multispectral image Multiplication operations are used to inject details, resulting in the final fused image. The formula is as follows:

[0107]

[0108] in, This represents the fused image of the c-th channel; This indicates the third high-frequency detail in channel c; This represents the upsampled multispectral image of channel c.

[0109] This invention discloses a remote sensing image fusion method combining ratio transformation and distribution transformation. Based on the idea of ​​component substitution, this method leverages the advantages of deep learning and ratio transformation to acquire high-frequency details. These details are then subjected to distribution transformation and finally injected into a multispectral image to achieve high-fidelity image fusion. The main steps include: First, a high-frequency detail generation method is designed. This method utilizes the strong nonlinear expression capabilities of deep learning, concatenating panchromatic and multispectral images and simultaneously inputting them into a convolutional network to generate a low-resolution panchromatic image. Then, a ratio transformation is performed on the panchromatic and low-resolution panchromatic images to generate the high-frequency details missing in the multispectral image. Next, a spectral gain method based on Gaussian distribution transformation is designed. This method standardizes the high-frequency details, then injects the mean and standard deviation of the multispectral image to achieve distribution transformation, and finally injects this into the multispectral image to achieve image fusion. By combining these two main modules, a high-fidelity remote sensing image fusion method combining ratio transformation and distribution transformation is achieved. This method can simultaneously ensure that the fused image acquires better detail texture and more complete spectral information, reducing the problems of detail distortion and spectral distortion in the fused image.

[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A remote sensing image fusion method combining ratio transformation and distribution transformation, characterized in that, include: S1. Perform mean filtering on the panchromatic image and the upsampled multispectral image respectively to obtain the corresponding high-frequency components of the panchromatic image and the upsampled multispectral image. S2. Based on the panchromatic image, the upsampled multispectral image, the high-frequency component of the panchromatic image, and the high-frequency component of the upsampled multispectral image, obtain the missing high-frequency details in the multispectral image, which are denoted as the first high-frequency details. S3. Perform standard normalization on the first high-frequency detail to obtain the standard normalized high-frequency detail, which is denoted as the second high-frequency detail; and calculate the mean and standard deviation of each pixel in each channel of the upsampled multispectral image. S4. The upsampled multispectral image and the first high-frequency detail are stitched together and then input into the convolutional network to generate the first affine transformation parameters and the second affine transformation parameters. S5. Based on the first affine transformation parameters and the second affine transformation parameters, the mean and standard deviation of each pixel in each channel of the upsampled multispectral image are injected into the second high-frequency detail to generate a high-frequency detail that is distributed in the same way as the upsampled multispectral image, which is denoted as the third high-frequency detail. S6. Based on the third high-frequency details and combined with the upsampled multispectral image, the final fused image is obtained; S2 specifically includes: S21. The high-frequency components of the upsampled multispectral image and the high-frequency components of the panchromatic image are concatenated and simultaneously input into a convolutional network to generate the high-frequency components of a low-resolution panchromatic image. S22. The upsampled multispectral image and the panchromatic image are stitched together and simultaneously input into the network to generate a low-resolution panchromatic image; S23. Add the high-frequency components of the low-resolution panchromatic image to the low-resolution panchromatic image to obtain the corrected low-resolution panchromatic image; S24. Perform a ratio transformation on the panchromatic image and the low-resolution panchromatic image to obtain the missing details in the multispectral image, which are denoted as the first high-frequency details.

2. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 1, characterized in that, S1 specifically includes: S11. Acquire panchromatic and multispectral images; S12. Upsample the multispectral image to obtain an upsampled multispectral image of the same scale as the panchromatic image; S13. Perform mean filtering convolution calculations on the panchromatic image and the upsampled multispectral image respectively to obtain the corresponding low-frequency components of the panchromatic image and the upsampled multispectral image. S14. Subtract the panchromatic image and the upsampled multispectral image from their respective low-frequency components to obtain the high-frequency components of the panchromatic image and the upsampled multispectral image.

3. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 1, characterized in that, In step S3, the first high-frequency detail is subjected to standard normalization processing, specifically including: (1) Calculate the mean value of each pixel in the first high-frequency detail, expressed by the formula: Where, μ detail P represents the mean value of each pixel in the first high-frequency detail; detail Indicates the first high-frequency detail; P detail y,x This represents the value of each pixel in the first high-frequency detail; H represents the image height; W represents the width; x represents the horizontal coordinate of the pixel; y represents the vertical coordinate of the pixel; (2) Calculate the standard deviation of each pixel in the first high-frequency detail; the formula is as follows: Where, σ detail ε represents the standard deviation of each pixel in the first high-frequency detail; ε is a constant. (3) Based on the mean and standard deviation of each pixel in the first high-frequency detail, the first high-frequency detail is subjected to standard normalization to obtain the standard normalized high-frequency detail, which is denoted as the second high-frequency detail; the calculation formula is expressed as: in, This represents the high-frequency details after standard normalization, i.e., the second high-frequency details.

4. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 1, characterized in that, In step S3, the mean and standard deviation of each pixel in each channel of the upsampled multispectral image are calculated, specifically including: (1) Calculate the mean value of each pixel in each channel of the upsampled multispectral image; the formula is expressed as: in, H represents the mean value of each pixel in each channel of the upsampled multispectral image; W represents the image height; x represents the x-coordinate of the pixel; y represents the y-coordinate of the pixel. This represents the pixel value at position (x, y) in channel c; (2) Calculate the standard deviation of each pixel in each channel of the upsampled multispectral image based on the mean of each pixel in each channel; the formula is expressed as:

5. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 1, characterized in that, In step S4, the first affine transformation parameter is used to adjust the mean value of each pixel in each channel of the upsampled multispectral image; The second affine transformation parameter is used to adjust the standard deviation of each pixel in each channel of the upsampled multispectral image.

6. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 3, characterized in that, The third high-frequency detail is represented as follows: in, This indicates the third high-frequency detail in channel c; This represents the standard deviation of pixels in the c-th channel of the upsampled multispectral image; γ represents the pixel mean of the c-th channel in the upsampled multispectral image; β represents the first affine transformation parameter; and γ represents the second affine transformation parameter.

7. The remote sensing image fusion method combining ratio transformation and distribution transformation as described in claim 1, characterized in that, The final fused image is represented as follows: in, This represents the fused image of the c-th channel; This indicates the third high-frequency detail in channel c; This represents the upsampled multispectral image of channel c.

Images