A luminescent remote sensing image fusion method and device

Abstract

The present application relates to a kind of luminescent remote sensing image fusion methods, the method comprises: using multiple linear regression model, as the model of generating panchromatic image;I 40 Upsampling is I 10 ;By the model of generating panchromatic image, three wave bands of I 10 Fitting generates P n10 ;I 10 , P 10 And P n10 Generation F 10 ;F 10 Upsampling is I 5 ;By the model of generating panchromatic image, three wave bands of I 5 Fitting generates P n5 ;P 10 Input image super-resolution model, obtain P5;I 5 , P5 and P n5 Generation F 5 The present application also relates to a kind of luminescent remote sensing image fusion device, comprising: modeling module, first sampling module, the model of generating panchromatic image, first fusion module, image super-resolution model, second sampling module and second fusion module.The method and device of the present application, fusion has obtained 5 meter resolution color luminescent remote sensing image.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, and in particular to a method and apparatus for fusing nighttime light remote sensing images. Background Technology

[0002] The implementation faces an urgent need for data and methods. As a high-efficiency data acquisition means and research method, space observation has macroscopic, dynamic and objective monitoring capabilities. It can detect parameters of land, ocean, atmosphere and ground features related to human activities, and provide dynamic, multi-scale and periodic rich information for Sustainable Development Goals (SDGs) that are closely related to the Earth's surface environment and resources.

[0003] The SDGSAT-1 scientific satellite primarily utilizes thermal infrared, multispectral, and low-light sensors to detect surface parameters in areas of intense human activity, aiming to meticulously characterize traces of human activity and support the monitoring and assessment of relevant SDG indicators. The SDGSAT-1 satellite carries three payloads: a thermal infrared imager (TIS) for detecting the spatial distribution of surface thermal radiation; a glimmer imager (GI) for detecting nighttime light in cities and rural areas at different levels, and for scientifically exploring nighttime urban aerosols; and a multispectral imager (MI) for detecting coastal and near-shore environments. Through coordinated observation by these three payloads throughout the day, the satellite aims to achieve a detailed characterization of "traces of human activity," serving the realization of global SDGs and providing data support for research on indicators characterizing the interaction between humans and nature.

[0004] The SDGSAT-1 low-light payload boasts a 10-meter resolution panchromatic band and a 40-meter resolution red, green, and blue band detection capability. Compared to existing DMSP-OLS and SNPP-VIIRS night light sensors, this sensor offers significantly improved spectral and spatial resolution, enabling the extraction of detailed light data and the differentiation of lights, snow, clouds, and different colors of light, thus facilitating precise light information extraction and analysis.

[0005] Currently, by fusing 10-meter resolution panchromatic and 40-meter resolution color nighttime light remote sensing images captured by the SDGSAT-1, multi-band nighttime light remote sensing monitoring data with a 10-meter resolution is obtained. This enables the monitoring of light distribution and the differentiation of light types. The 10-meter spatial resolution allows visualization of commercial areas and main roads in small and medium-sized cities, as well as the identification of concentrated residential areas in towns and villages. Different light types can reveal the history of urban development. By simultaneously acquiring the characteristics of human nighttime activities and energy consumption in different spatial areas, the virtual scale of urban expansion can be revealed, thereby assessing the level of regional economic development. In addition, the 10-meter resolution multi-band nighttime light remote sensing data makes it relatively easy to find areas on the ground where the light brightness can remain relatively stable for a long time. At the same time, the SDGSAT-1 low-light payload also has the ability to detect moonlight close to a full moon (radiance of 10-4 to 10-5 W / m2 / sr). The 300km swath of stable light data and the near-full moon moonlight remote sensing data can be used for atmospheric aerosol climate effect research and weather numerical simulation, which helps to accurately invert the optical thickness of urban nighttime aerosols.

[0006] However, existing remote sensing image fusion algorithms are mainly used for fusion of optical remote sensing images. Compared with optical images, low-light images are characterized by a large number of low-value pixels and oversaturation of high-value pixels. Therefore, the fused images obtained using traditional optical remote sensing image fusion methods suffer from significant spectral distortion. Furthermore, SDGSAT-1 low-light images are characterized by high background noise and obvious banding; some panchromatic and multispectral images of Level 4 products exhibit registration errors, directly affecting the quality of the fused image. Moreover, existing image fusion methods used for SDGSAT-1 nighttime light remote sensing image fusion can only produce color nighttime light remote sensing images with a resolution of 10 meters, which cannot meet the needs of research projects requiring more detailed information. Summary of the Invention

[0007] In view of this, the main objective of the present invention is to provide a method and apparatus for fusing nighttime light remote sensing images. By combining a fusion method that reduces the impact of registration error (Reduced Misalignment Impact, RMI) and image super-resolution, a 5-meter resolution color nighttime light remote sensing image can be obtained by fusing 10-meter resolution panchromatic and 40-meter resolution color nighttime light remote sensing images.

[0008] To achieve the above objectives, this application provides a nighttime light remote sensing image fusion method, comprising:

[0009] A multiple linear regression model was adopted as the model for generating panchromatic images;

[0010] 40-meter resolution color night light remote sensing image I 40Upsampled to 10-meter resolution color nighttime light remote sensing image I 10 ; using the model for generating panchromatic images, I 10 A 10-meter resolution panchromatic image P was generated by fitting the three bands. n10 ;

[0011] Fusion I 10 10-meter resolution panchromatic nighttime remote sensing image P 10 and P n10 Generate a 10-meter resolution color image F 10 ;

[0012] F 10 Upsampled to 5-meter resolution color nighttime light remote sensing image I 5 ; using the model for generating panchromatic images, I 5 The three bands were fitted to generate a 5-meter resolution panchromatic image P. n5 ;

[0013] P 10 Input the image super-resolution model to obtain a 5-meter resolution panchromatic nighttime light remote sensing image P5;

[0014] Fusion I 5 P5 and P n5 Generate a 5-meter resolution color image F 5 .

[0015] In one possible implementation, it also includes: removing I 40 The strip noise is:

[0016] Determine I 40 Background value;

[0017] Search I 40 For pixels with exactly two bands whose values ​​are the background value, the values ​​of the bands in the searched pixels that are higher than the background value are modified to the background value.

[0018] In another possible implementation, it includes: the I 10 P is generated by fitting the three bands. n10 ,include:

[0019] P 10 Downsampled to 40-meter resolution panchromatic nighttime light remote sensing image P 40 ; will I 40 Pixels with DN values ​​higher than the threshold T are taken as independent variables, P 40 The pixels with DN values ​​higher than the threshold T are used as the dependent variable, and the regression coefficients of the model for generating the panchromatic image are obtained by using the least squares method.

[0020] The calculated regression coefficients, and I 10The three bands are substituted into the model for generating a panchromatic image, and P is fitted to generate P. n10 .

[0021] In another possible implementation, the threshold T ranges from 15 to 20.

[0022] In another possible implementation, the P 10 Before inputting the image super-resolution model, the following is also included:

[0023] P 10 Downsampled to a 20-meter resolution panchromatic nighttime light remote sensing image P 20 ; with P 10 P 20 The pre-trained deep learning-based image super-resolution model is fine-tuned to obtain the image super-resolution model.

[0024] In another possible implementation, the I 5 P is generated by fitting the three bands. n5 ,include:

[0025] Downsample P5 to a 10-meter resolution panchromatic nighttime light remote sensing image. L10 ; F 10 As the independent variable, P L10 As the dependent variable, the regression coefficients of the model for generating panchromatic images are obtained by solving the least squares method;

[0026] This will adopt F 10 P L10 The calculated regression coefficients, and I 5 The three bands are substituted into the model for generating a panchromatic image, and P is generated by fitting the model. n5 .

[0027] On the other hand, this application also provides a nighttime light remote sensing image fusion device, comprising:

[0028] The modeling module is used to employ a multivariate linear regression model as the model for generating panchromatic images;

[0029] The first sampling module is used to sample I 40 Upsampling is I 10 ;

[0030] A model for generating panchromatic images, used to convert I... 10 P is generated by fitting the three bands. n10 ; and used to transfer I 5 P is generated by fitting the three bands. n5 ;

[0031] The first fusion module is used for fusion I.10 P 10 and P n10 Generate F 10 ;

[0032] Image super-resolution model, used by P 10 Restore P5;

[0033] The second sampling module is used to sample F 10 Upsampling is I 5 ;

[0034] The second fusion module is used for fusion I. 5 P5 and P n5 Generate F 5 . Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a nighttime light remote sensing image fusion method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of a nighttime light remote sensing image fusion device according to an embodiment of the present invention. Detailed Implementation

[0037] Specifically, the process of a nighttime light remote sensing image fusion method according to an embodiment of the present invention is as follows: Figure 1 As shown, steps 1011-1014, 102, and 103 are included.

[0038] Step 1011: Use a multivariate linear regression model as the model for generating panchromatic images.

[0039] Step 1012: Place I 40 Upsampling is I 10 ; using the model for generating panchromatic images, I 10 P is generated by fitting the three bands. n10 .

[0040] Step 1013: Fusion I 10 P 10 and P n10 Generate F 10 .

[0041] Step 1014: F 10 Upsampling is I 5 ; using the model for generating panchromatic images, I 5 P is generated by fitting the three bands. n5 .

[0042] Step 102: Place P 10 Input the image super-resolution model to obtain P5.

[0043] Step 103: Fusion I 5 P5 and P n5 Generate F 5 .

[0044] Here, in step 1011, the model for generating the panchromatic image is expressed by the formula:

[0045]

[0046] Where N is the number of bands in the color nighttime light remote sensing image I, which is 3 here; I i For the i-th band of I; P n To fit the generated panchromatic image.

[0047] In one possible implementation, step 1012 also includes removing I. 40 To eliminate the influence of stripe noise on the fused image and improve its quality, the following steps are taken:

[0048] Determine I 40 Background value;

[0049] Search I 40 For pixels with exactly two bands whose values ​​are the background value, the values ​​of the bands in the searched pixels that are higher than the background value are modified to the background value.

[0050] Here, remove the I. 40 The basis for identifying stripe noise is as follows: Through statistical and comparative analysis of L4 level data products, it was found that in the R, G, and B bands, only two of the stripe noise pixels have values ​​that are background values. These background values ​​are typically the minimum values ​​in the image.

[0051] In another possible implementation, in step 1012, the step of I... 10 P is generated by fitting the three bands. n10 ,include:

[0052] P 10 Downsampling to P 40 ; will I 40 Pixels with DN values ​​higher than the threshold T are taken as independent variables, P 40 The pixels with DN values ​​higher than the threshold T are used as the dependent variable, and the regression coefficients of the model for generating the panchromatic image are obtained by using the least squares method.

[0053] The calculated regression coefficients, and I 10 The three bands are substituted into the model for generating a panchromatic image, and P is fitted to generate P. n10 .

[0054] The fitting generates P n10 The formula is expressed as:

[0055]

[0056] The threshold T ranges from 15 to 20.

[0057] Statistical analysis revealed that the DN value of background noise pixels was 1-5 higher than the background value, and the DN value of strip noise pixels was 5-7 higher than the background value. Therefore, the threshold T is 15-20.

[0058] Here, I 40 Pixels with DN values ​​higher than the threshold T are taken as independent variables, P 40 Using pixels with DN values ​​higher than the threshold T as the dependent variable reduces the impact of dark pixels and background noise in nighttime light remote sensing images, and also reduces spectral distortion in fused images, further improving the quality of fused images.

[0059] In step 1013, the fusion I 10 P 10 and P n10 Generate F 10 The following formula can be used to achieve this:

[0060]

[0061] in, For F 10 The i-th band.

[0062] In another possible implementation, in step 1014, the step of I... 5 P is generated by fitting the three bands. n5 ,include:

[0063] Downsample P5 to a 10-meter resolution panchromatic nighttime light remote sensing image. L10 ; F 10 As the independent variable, P L10 As the dependent variable, the regression coefficients of the model for generating panchromatic images are obtained by solving the least squares method;

[0064] This will adopt F 10 P L10 The calculated regression coefficients, and I 5 The three bands are substituted into the model for generating a panchromatic image, and P is generated by fitting the model. n5 .

[0065] The fitting generates P n5 The formula is expressed as:

[0066]

[0067] In another possible implementation, step 102 is preceded by:

[0068] P 10 Downsampled to a 20-meter resolution panchromatic nighttime light remote sensing image P 20 ; with P 10 P 20 The pre-trained deep learning-based image super-resolution model is fine-tuned to obtain the image super-resolution model.

[0069] Here, the pre-trained deep learning-based image super-resolution model is obtained by training multiple panchromatic nighttime light remote sensing images.

[0070] The deep learning-based image super-resolution model can adopt the SRCNN model proposed in the paper entitled "Image Super-Resolution Using Deep Convolutional Networks," which was published by Dong Chao, Chen Bianle, He Kaiming, and Tang Xiaoou on February 1, 2016, in IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 38, Issue 2, pp. 295-307.

[0071] Deep learning-based image super-resolution models can also employ very deep super-resolution networks (VDSR) or efficient sub-pixel convolutional neural networks (ESPCN).

[0072] In step 103, the fusion I 5 P5 and P n5 Generate F 5 The following formula can be used to achieve this:

[0073]

[0074] in, For F 5 The i-th band.

[0075] The structure of a nighttime light remote sensing image fusion device according to an embodiment of the present invention is as follows: Figure 2 As shown, it includes:

[0076] The modeling module is used to employ a multivariate linear regression model as the model for generating panchromatic images;

[0077] The first sampling module is used to sample I 40 Upsampling is I 10 ;

[0078] A model for generating panchromatic images, used to convert I... 10 P is generated by fitting the three bands. n10 ; and used to transfer I 5 P is generated by fitting the three bands. n5 ;

[0079] The first fusion module is used for fusion I. 10 P 10 and P n10 Generate F 10 ;

[0080] Image super-resolution model, used by P 10 Restore P5;

[0081] The second sampling module is used to sample F 10 Upsampling is I 5 ;

[0082] The second fusion module is used for fusion I. 5 P5 and P n5 Generate F 5 .

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A luminescent remote sensing image fusion method, characterized in that, Comprise: Adopting a multiple linear regression model as a model for generating a panchromatic image; 40-meter resolution color night light remote sensing image I 40 Upsampled to 10-meter resolution color nighttime light remote sensing image I 10 ; using the model for generating panchromatic images, I 10 A 10-meter resolution panchromatic image P was generated by fitting the three bands. n10 ; will I 10 P is generated by fitting the three bands. n10 This includes: P 10 Downsampled to 40-meter resolution panchromatic nighttime light remote sensing image P 40 ; will I 40 Pixels with DN values ​​higher than the threshold T are taken as independent variables, P 40 Using pixels with DN values ​​higher than a threshold T as the dependent variable, the regression coefficients of the model for generating the panchromatic image are obtained by solving the least squares method; the calculated regression coefficients are then used to... and I 10 The three bands are substituted into the model for generating a panchromatic image, and P is fitted to generate P. n10 ; Fusion I 10 , 10-meter resolution panchromatic night-time remote sensing image P 10 and P n10 generate a 10-meter resolution color image F 10 ; F 10 upsampling a 5-meter resolution color night light remote sensing image I 5 ; fitting three bands of I 5 through the model generating the panchromatic image to generate a 5-meter resolution panchromatic image P n5 ; P 10 inputting the image super-resolution model to obtain a 5-meter resolution panchromatic night light remote sensing image P5; Fusion I 5 , P5 and P n5 Generating 5-meter resolution color image F 5 .

2. The method of claim 1, wherein, Also included: removing I 40 strip noise for: Determination I 40 background value; Search I 40 The value of the band higher than the background value in the searched pixel is modified to the background value.

3. The method of claim 1, wherein, The threshold T is in the range of 15-20.

4. The method according to claim 1 or 2, characterized in that, The P 10 Before inputting the image super-resolution model, further comprising: P 10 down-sampling to 20-meter resolution panchromatic night light remote sensing image P 20 ; P 10 , P 20 Fine-tuning training of the pre-trained deep learning-based image super-resolution model to obtain an image super-resolution model.

5. The method according to claim 1 or 2, characterized in that, The I 5 The three wavebands are fitted to generate P n5 , comprising: P5 is down-sampled into a 10-meter resolution panchromatic night light remote sensing image P L10 F is down-sampled into a 10-meter resolution panchromatic night light remote sensing image P 10 P is taken as the independent variable L10 The regression coefficient of the model for generating the panchromatic image is obtained by using the least square method The F 10 , P L10 , the regression coefficient calculated, and I 5 of the three wave bands are substituted into the model for generating a panchromatic image, and P n5 is generated by fitting the model for generating a panchromatic image.

6. A luminescent remote sensing image fusion device, characterized in that, Comprise: The modeling module is configured to adopt a multiple linear regression model as a model for generating a panchromatic image; The first sampling module is configured to sample I 40 The up-sampling is I 10 ; A model for generating panchromatic images, used to convert I... 10 P is generated by fitting the three bands. n10 ; and used to transfer I 5 P is generated by fitting the three bands. n5 Among them, I 10 P is generated by fitting the three bands. n10 This includes: P 10 Downsampled to 40-meter resolution panchromatic nighttime light remote sensing image P 40 ; will I 40 Pixels with DN values ​​higher than the threshold T are taken as independent variables, P 40 Using pixels with DN values ​​higher than a threshold T as the dependent variable, the regression coefficients of the model for generating the panchromatic image are obtained by solving the least squares method; the calculated regression coefficients are then used to... and I 10 The three bands are substituted into the model for generating a panchromatic image, and P is fitted to generate P. n10 ; a first fusion module configured to fuse I 10 , P 10 , and P n10 to generate F 10 ; An image super-resolution model for generating P 10 P5 is recovered. The second sampling module is configured to sample F 10 The up-sampling is I 5 ; a second fusion module configured to fuse I 5 , P5 and P n5 generate F 5 .

Images