Multi-modal remote sensing image fusion method of improved star generative adversarial network (StarGAN)

By improving the network structure and loss function of StarGAN and combining it with super-resolution reconstruction technology, the problem of fusing nighttime light images with multispectral images was solved, generating high-quality, high-resolution nighttime light images and enhancing the application value of remote sensing images.

CN120852185APending Publication Date: 2025-10-28GUILIN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510896105.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-08-06
Publication Date
2025-10-28

AI Technical Summary

Technical Problem

Traditional multimodal remote sensing image fusion methods suffer from structural distortion, spectral bias, and information loss when processing nighttime light images and multispectral images. The original StarGAN model is unable to effectively handle the differences in the characteristics of remote sensing images, resulting in artifacts or information distortion in the fusion results.

Method used

By optimizing the network structure of StarGAN (Star Generative Adversarial Network), designing a targeted loss function, and introducing a detail enhancement module, combined with super-resolution reconstruction technology, efficient fusion of Luojia-1 night light imagery and Landsat 8 multispectral imagery is achieved, generating night light remote sensing images with high spatial resolution and rich spectral information.

Benefits of technology

It effectively solves the problems of structural distortion and spectral deviation in traditional fusion methods. The generated high-resolution nighttime light image is highly matched with the original image, which improves the spatial resolution and information expression of the nighttime light image and provides high-quality data support for urban research, environmental monitoring and disaster assessment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120852185A_ABST
    Figure CN120852185A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-modal remote sensing image fusion method of an improved star generative adversarial network (StarGAN), and aims to solve the problems of structural distortion and spectral deviation in a traditional fusion method and improve the resolution and information expression ability of a noctilucent image. According to the method, a Landsat 8 multispectral image and a Luoma No.1 noctilucent image are taken as data sources, a fusion data set is constructed through data preprocessing, and a super-resolution convolutional neural network (SRCNN) is utilized to perform up-sampling on the noctilucent image so as to reduce the scale difference of the multisource image. By optimizing a StarGAN network architecture and a loss function and introducing a detail enhancement fusion module, efficient fusion of a Landsat 8 multispectral image and an up-sampled noctilucent image is realized. Experimental results show that the fusion image generated by the method is highly consistent with an original noctilucent image in content consistency and structural fidelity, the resolution and detail information are remarkably improved, and higher-quality data support is provided for the fields of urban research, environment monitoring, disaster assessment and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of remote sensing image processing and deep learning, and specifically to a multimodal remote sensing image fusion method based on an improved StarGAN (Generative Adversarial Network). This method aims to effectively fuse nighttime light remote sensing images with multispectral remote sensing images to generate nighttime light remote sensing data products with high spatial resolution and rich spectral information. Background Art

[0002] Multimodal remote sensing image fusion is a research hotspot and challenge in the field of remote sensing science. Due to significant differences in spatial resolution, spectral characteristics, and imaging mechanisms among different modal remote sensing image data, traditional fusion methods often struggle to achieve efficient and high-quality fusion when processing multimodal data. Traditional interpolation algorithms, such as Principal Component Analysis (PCA), wavelet transform, and Gram-Schmidt transform, frequently suffer from structural distortion, spectral bias, and information loss when fusing nighttime light images and multispectral images, greatly limiting the application value of the fused images.

[0003] In recent years, deep learning technology has developed rapidly, and Generative Adversarial Networks (GANs) have shown great potential in the field of multimodal remote sensing image fusion. StarGAN, as an advanced GAN model, possesses powerful multi-domain image transformation capabilities, providing a new approach for multimodal remote sensing image fusion. However, directly applying the original StarGAN model to handle remote sensing image fusion tasks still has shortcomings. First, nighttime light images and multispectral images have significant differences in data characteristics, and the original StarGAN model struggles to effectively handle these differences, easily leading to artifacts or information distortion in the fusion results. Second, remote sensing images typically have high spatial resolution, rich texture details, and unique spectral information, and the design of the original StarGAN model did not fully consider these characteristics of remote sensing images.

[0004] To address the above problems, this invention proposes an improved StarGAN-based multimodal remote sensing image fusion method. By optimizing the network structure, designing a targeted loss function, introducing a detail enhancement module, and improving the training strategy, this invention's method can better adapt to the characteristics of remote sensing images, effectively fusing nighttime light remote sensing images and multispectral remote sensing images to generate high-quality, high-resolution nighttime light remote sensing images. This provides strong support for the refined application of nighttime light remote sensing images in urban research, environmental monitoring, disaster assessment, and other fields. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of structural distortion, spectral bias, and information loss in traditional multimodal remote sensing image fusion methods when processing nighttime light images and multispectral images. It provides a high-quality multimodal remote sensing image fusion method based on an improved StarGAN (Generative Adversarial Network). This method combines super-resolution reconstruction technology with optimized network structure, a targeted loss function, and the introduction of a detail enhancement mechanism to achieve efficient fusion of Luojia-1 nighttime light remote sensing images and Landsat 8 multispectral remote sensing images. This generates a 30-meter resolution nighttime light remote sensing image with high spatial resolution and rich spectral information, thereby enhancing the application value of nighttime light remote sensing images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a multimodal remote sensing image fusion method based on an improved star-shaped generative adversarial network (StarGAN), comprising the following steps:

[0008] Step S1: Acquire Landsat 8 multispectral images and Luojia-1 nighttime light images, and preprocess the Landsat 8 multispectral images and Luojia-1 nighttime light images.

[0009] Step S2: Use SRCNN super-resolution convolutional neural network to perform super-resolution reconstruction on the Luojia-1 night light image to obtain an upsampled night light image with the same spatial resolution as the Landsat 8 multispectral image.

[0010] Step S3: Input the Landsat 8 multispectral image and the upsampled night light image into the generator of the improved StarGAN to generate a fused image;

[0011] Step S4: Use the discriminator of the improved StarGAN to distinguish between the fused image and the real image generated by the generator, and feed the discrimination result back to the generator;

[0012] Step S5: Construct a multiple loss function to jointly optimize the parameters of the generator and the discriminator.

[0013] Preferably, in step S1, the preprocessing of the Landsat 8 multispectral image includes: seamless mosaicking, region of interest (ROI) extraction, radiometric calibration, and atmospheric correction; the preprocessing of the Luojia-1 nighttime light image includes: seamless mosaicking, reprojection, ROI extraction, and radiometric calibration.

[0014] Preferably, step S1 further includes converting the data type of the Luojia-1 night-light image from 32-bit floating-point to 16-bit integer.

[0015] Preferably, in step S2, the Luojia-1 nighttime image is upsampled by 4 times using an SRCNN super-resolution convolutional neural network based on PixelShuffle.

[0016] Preferably, in step S2, the SRCNN model includes three convolutional layers and one PixelShuffle layer.

[0017] Preferably, in step S3, the generator of the improved StarGAN includes: a night vision image encoder, a multispectral image encoder, a detail enhancement fusion module, and a decoder.

[0018] Preferably, in step S3, the detail enhancement fusion module in the generator of the improved StarGAN includes: an edge detection convolutional layer, an attention calculation module, and a residual enhancement convolutional layer.

[0019] Preferably, in step S4, the discriminator of the improved StarGAN adopts a multi-layer convolutional structure, combined with adaptive average pooling and multi-domain scoring mechanisms.

[0020] Preferably, in step S5, the multiple loss functions include: cross-entropy loss, structural similarity index (SSIM) loss, perceptual loss, and L1 loss.

[0021] The beneficial results of this invention are as follows:

[0022] This invention provides a multimodal remote sensing image fusion method based on an improved StarGAN (Generative Adversarial Network), aiming to effectively improve the fusion quality of nighttime light images and multispectral images. First, the nighttime light and multispectral images are preprocessed to eliminate intermodal differences and ensure data quality. Then, an SRCNN (Surveyor-Resolution Convolutional Neural Network) is used to upsample the low-resolution nighttime light image by 4 times, making its spatial resolution consistent with the multispectral image. The upsampled nighttime light and multispectral images are then input into an improved StarGAN generator for fusion to generate a high-resolution nighttime light image. The StarGAN discriminator evaluates the generated image in real time, continuously optimizing the generator parameters through adversarial learning and iteratively optimizing by integrating various loss functions to improve the quality of the fused image. This invention fully leverages the advantages of the improved StarGAN, effectively solving the structural distortion and spectral bias problems that may occur in traditional image fusion processes. It ensures that the generated high-resolution nighttime light image highly matches the original nighttime light image in terms of content consistency and structural fidelity, while simultaneously improving the spatial resolution and information expression of the nighttime light image, providing an efficient and reliable solution for remote sensing image fusion. Attached Figure Description

[0023] Figure 1. Overall architecture of the improved StarGAN multimodal remote sensing image fusion method

[0024] Figure 2 Overall architecture of SRCNN super-resolution convolutional neural network

[0025] Figure 3 Overall architecture of the generator

[0026] Figure 4 Detail Enhancement Blending Module

[0027] Figure 5 Overall architecture of the discriminator

[0028] Figure 6. Generator and discriminator optimization, (A) training process; (B) discriminator loss calculation;

[0029] (C) Input data; (D) Generator loss calculation

[0030] Figure 7. Raw image data

[0031] Figure 8 Image comparison using upsampling methods

[0032] Figure 9. Image comparison using the fusion method: (Top layer) Original Luojia-1 image; (Middle layer) GS fused image; (Bottom layer) Image fused using the method of this invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, preferred embodiments are described below, and the specific implementation of the present invention is further described in detail with reference to the accompanying drawings.

[0034] This invention discloses a multimodal remote sensing image fusion method based on an improved StarGAN, the overall architecture of which is shown in the attached figure. Figure 1 As shown, it includes the following steps:

[0035] Step S1: Acquire Landsat 8 multispectral images and Luojia-1 nighttime light images, and preprocess the Landsat 8 multispectral images and Luojia-1 nighttime light images.

[0036] Step S2: Use SRCNN super-resolution convolutional neural network to perform super-resolution reconstruction on the Luojia-1 night light image to obtain an upsampled night light image with the same spatial resolution as the Landsat 8 multispectral image.

[0037] Step S3: Input the Landsat 8 multispectral image and the upsampled night light image into the generator of the improved StarGAN to generate a fused image;

[0038] Step S4: Use the discriminator of the improved StarGAN to distinguish between the fused image and the real image generated by the generator, and feed the discrimination result back to the generator;

[0039] Step S5: Construct a multiple loss function to jointly optimize the parameters of the generator and the discriminator.

[0040] The specific data preprocessing scheme in step S1 includes the following:

[0041] Images of Luojia-1 nighttime light and Landsat-8 multispectral images of the same region were collected to ensure coverage of the same spatial area. Seamless mosaicking, region of interest (ROI) extraction, radiometric correction, and atmospheric correction were performed on the Landsat-8 OLI data; simultaneously, seamless mosaicking, reprojection, ROI extraction, and radiometric correction were performed on the Luojia-1 nighttime light images. To further optimize image quality, after removing outlier DN values ​​less than 1% and greater than 99%, a linear transformation (as shown in Equation 1) was applied to convert the 32-bit floating-point data of Luojia-1 to 16-bit depth, consistent with the Landsat-8 OLI band specifications. This helps preserve the unique information in the nighttime light data. Finally, all processed data were saved as single-band TIFF files.

[0042]

[0043] In the formula, Nighttime light image data is recorded in 32-bit floating-point format. It is the minimum value in the nighttime light image data. It is the maximum value. The nighttime light image data is recorded in a 16-bit depth format after conversion, making it consistent with the band specifications of the Landsat-8 OLI image. It is the maximum pixel value of the 16-bit depth image, ensuring that the converted data is distributed in the range of 0 to 65535.

[0044] In step S2, the method for super-resolution reconstruction of the Luojia-1 nighttime light image using the SRCNN super-resolution convolutional neural network is as follows:

[0045] This model is built based on PyTorch's nn.Module and includes three convolutional layers and one pixel rearrangement layer, as shown in the attached diagram. Figure 2 As shown. The first convolutional layer has 64 channels, uses a 9x9 kernel, and maintains the image size by padding the edges; the second convolutional layer has 32 channels, uses a 5x5 kernel; the third convolutional layer first increases the number of channels through convolution operations, and then uses a pixel shuffle layer to improve the image resolution. The mathematical expression for pixel shuffle is shown in Equation 2:

[0046]

[0047] In the formula, This indicates rounding down, while This represents the modulo operation, which, simply put, involves taking the input feature map. Reduce the number of channels in this feature map. times ( (This is an upsampling factor), while increasing both height and width. This multiplies the sampling rate, thus achieving the upsampling process.

[0048] During the forward propagation of the model, the input image passes through these layers sequentially, and a nonlinear transformation is performed after each layer using the LeakyReLU activation function, thereby outputting a high-resolution image. Model training utilizes preprocessed Luojia-1 images downsampled to 480m resolution via bicubic interpolation to simulate low-resolution nighttime illumination images. The target image is the original Luojia-1 image. Furthermore, the model employs mean squared error (MSE) and structural similarity index (SSIM) as loss functions, as shown in Equations 3 and 4.

[0049]

[0050] Mean squared error is used to measure the difference between the reconstructed image and the real image. It represents the number of samples. It is a real video. It is a predicted image.

[0051]

[0052] An SSIM value close to 1 indicates a high structural similarity between the reconstructed image and the original image. , It is the average of the reconstructed image and the real image. It is their standard deviation. It is their covariance. and It is a constant used for stable calculations.

[0053] The objective function is shown in Equation 5, which ensures that high-resolution details can be effectively reconstructed from low-resolution images.

[0054]

[0055] An improved StarGAN multimodal remote sensing image fusion method was developed. By using a trained SRCNN model, the spatial resolution of nighttime light images was enhanced, ensuring alignment between the Luojia-1 nighttime light images and the Landsat-8 OLI multispectral images. Compared to the traditional bicubic interpolation upsampling method, this method effectively reduces blurring and distortion in the Luojia-1 nighttime light images, laying the foundation for subsequent fusion.

[0056] The specific implementation of the improved StarGAN generator in step S3 is as follows:

[0057] The generator mainly consists of a night vision image encoder, a multispectral image encoder, a detail enhancement and fusion module, and a decoder, as shown in the attached diagram. Figure 3 As shown, the aim is to efficiently fuse different image features and enhance details. The generator first processes nighttime light images and multispectral images separately through two independent encoders. Each encoder is designed to contain two convolutional layers with 64 channels, using 9x9 convolutional kernels and the LeakyReLU activation function to maintain image size invariance and improve training stability, ensuring that the two modalities can be effectively represented in the same feature space. The encoded features are then fused through a detail enhancement fusion module, as shown in the attached diagram. Figure 4 As shown in the diagram, this module first uses learnable convolutional kernels to perform edge detection on the features of the nighttime light image, highlighting the structural information of the image. A 3x3 convolutional kernel with 64 channels is used, and the Xavier initialization method is employed to maintain consistency between the input and output variances. The mathematical expression for the convolutional operation of edge detection is shown in Equation 6:

[0058]

[0059] In the formula, It is the output feature map at the position The value, These are the convolution kernel weights. It is the value of the input feature map. It is the number of channels. Traverse the 3×3 convolution kernel, Iterate through all input channels.

[0060] Following edge detection, an attention mechanism is employed to further emphasize key features of the nightlight image. Nightlight edge features are extracted and stitched together with the original nightlight features. Attention weights are calculated using two 1x1 convolutional layers, and the attention map is normalized to the 0-1 range using a sigmoid function. This attention map is then multiplied by the original nightlight features to achieve adaptive weighting. During the fusion process, the enhanced nightlight features are stitched together with multispectral features, and then subjected to multi-layer convolutional fusion and residual enhancement. Finally, the fused features are reconstructed into the target image using a decoder, with skip connections set to preserve information from the original nightlight image, thus generating the final high-resolution nightlight remote sensing image.

[0061] In step S4, the discriminator of the improved StarGAN adopts a deep convolutional neural network structure, which is specifically implemented as follows:

[0062] The discriminator consists of five convolutional layers that progressively extract features from the input image, as shown in the attached diagram. Figure 5As shown, the input image is first processed through a convolutional layer, expanding the number of channels to 64 and using a convolution operation with a stride of 2 to reduce the spatial dimension of the feature map. The next three convolutional layers progressively increase the number of channels to 128, 256, and 512 respectively. Each layer is configured with batch normalization to stabilize the training process and uses the LeakyReLU activation function, further reducing the size of the feature map through convolution with a stride of 2. In the final feature extraction stage, adaptive average pooling is used to compress the feature map to a size of 1×1, and then a 1×1 convolutional layer is used to adjust the number of channels to the number of target domains. The final output shape is batch size × number of target domains, allowing each sample to correspond to a realism score for different domains. This design can more meticulously evaluate the realism and diversity of the fused image, improving its performance in multi-domain image conversion tasks.

[0063] In step S5, a multiple loss function is constructed to jointly optimize the generator and the discriminator, as shown in the appendix. Figure 6 As shown, the specific loss function is as follows:

[0064] Cross-entropy loss is used as the adversarial loss, as shown in Equation 7. By calculating the difference between the model output probability and the real label, the generator is guided to generate realistic fused images, and the discriminator distinguishes between real and fused images.

[0065]

[0066] In the formula, Indicates batch size; It is a sample For the real category The predicted score; and That is the sample The true category index.

[0067] The Structural Similarity Index (SSIM) is used as the content loss, as shown in Equation 4, to ensure that the fused image is structurally consistent with the nighttime light image and improve structural fidelity.

[0068] Perceptual loss is used as the detail loss, as shown in Equation 8. High-level features are extracted through a pre-trained convolutional neural network to enhance the detail representation of the fused image and improve visual quality.

[0069]

[0070] In the formula, It is an input image. It is the target image and They represent their respective positions in the pre-trained neural network. Layer feature representation, It refers to the number of feature layers.

[0071] By using L1 norm loss, as shown in Equation 9, the pixel difference between the fused image and the original night light image is further optimized, thereby improving the overall image quality and consistency.

[0072]

[0073] In the formula, For the purpose of influence, To merge images, It is the absolute difference between the fused image and the original nighttime light image.

[0074] The overall objective function expresses the adversarial game between the generator and the discriminator, as shown in Equation 7. The discriminator generates images that are as realistic as possible by minimizing the loss. It then improves its discrimination ability by maximizing the loss.

[0075]

[0076] Application examples:

[0077] We selected Luojia-1 nighttime light data from parts of the Pearl River Delta region in China in September 2020, along with Landsat-8OLI multispectral overlay images, as validation samples. The original images are attached. Figure 7 As shown.

[0078] Example data sources: Luojia-1 nighttime light data can be obtained from raw NTL images through the Luojia Science Data Center, with a spatial resolution of 130m, a field width of 260km, and using the WGS84 latitude and longitude coordinate system; Landsat-8 OLI data comes from the USGS EarthExplorer platform, containing 11 bands, with a multispectral resolution of 30m, and the coordinate system is UTM Zone 49N / WGS84.

[0079] (1) Data preprocessing

[0080] The processing flow follows step S1 of claim 1, performing seamless mosaicking, region of interest (ROI) extraction, radiometric correction, and atmospheric correction on the Landsat-8 OLI multispectral imagery; simultaneously, seamless mosaicking, reprojection, ROI extraction, and radiometric correction are performed on the Luojia-1 nighttime light imagery to improve data quality. To further optimize data quality, after removing outliers (DN) less than 1% and greater than 99%, a linear transformation (as shown in Equation 1) is applied to convert the 32-bit floating-point data of Luojia-1 to 16-bit depth, ensuring compliance with the Landsat-8 OLI band specifications. This helps preserve the unique information in the nighttime light data. Finally, all processed data is saved as single-band TIFF files.

[0081] (2) Model training and parameter configuration

[0082] 1) Implementation of SRCNN super-resolution reconstruction framework

[0083] The preprocessed 130m resolution Luojia-1 nighttime illumination image was downsampled to 480m using bicubic interpolation to simulate low-resolution nighttime illumination, and then cropped to a size of 32×32 pixels, generating 5000 images for subsequent processing. To ensure the representativeness and balance of the training and validation data, the dataset was divided into an 80% training set and a 20% validation set. The AdamW optimizer (β1=0.9, β2=0.999) was used to optimize the network, with an initial learning rate of 1×10⁻³, a batch size of 8, and 50 epochs.

[0084] 2) Improve the StarGAN fusion framework implementation

[0085] The pre-trained SRCNN super-resolution model was used to upsample the preprocessed Luojia-1 nighttime light imagery to align it with the Landsat-8 OLI multispectral imagery. The image input generator was 32×32 pixels, generating a total of 3000 images. The dataset was divided into 80% training set and 20% validation set. The AdamW optimizer (β1=0.9, β2=0.999) was used to optimize the network with an initial learning rate of 1×10⁻. 2 The batch size is 16, and the number of iterations is 100 epochs.

[0086] To systematically verify the effectiveness of this innovative method, this study selects the classic multimodal fusion method GS as the benchmark and adopts an analytical framework combining qualitative and quantitative methods to achieve refined analysis. For quantitative evaluation, a system is constructed that includes reference metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), and non-reference metrics such as Spectral Detail Loss Component (D_Spectral) and Spatial Detail Loss Component (D_Spatial).

[0087] (3) Results Analysis

[0088] The upsampling results are shown in the attached image. Figure 8 As shown, the resolution was first reduced to simulate the degradation of details in nighttime light remote sensing images. Then, bicubic interpolation upsampling and the SRCNN super-resolution algorithm were used for restoration. It can be seen that, compared to the blurriness before resolution reduction, while the bicubic interpolation result can basically maintain the outline boundaries, it is lacking in the preservation of detailed structures, and the light spot phenomenon is more obvious. In contrast, the image structure restored using the SRCNN super-resolution algorithm is closer to the actual high-resolution Luojia-1 data in terms of brightness values ​​and texture, especially in complex details with small angles and rich edge transformations, demonstrating the excellent performance of the SRCNN super-resolution algorithm in repairing highlight loss and maintaining feature continuity in nighttime light images.

[0089] Table 1: Quantitative Indicators

[0090]

[0091] As shown in Table 1, a quantitative comparison of the SRCNN super-resolution reconstruction algorithm and the traditional bicubic interpolation method reveals that SRCNN outperforms traditional methods in ERGAS, PSNR, and SSIM, achieving reductions of 26.76%, improvements of 9.42%, and increases of 38.87% respectively compared to bicubic interpolation. This demonstrates SRCNN's significant advantages in global feature preservation, noise suppression, detail restoration, and structural consistency during nighttime image super-resolution. However, SRCNN exhibits a relatively high SAM value, increasing by 61.16%, indicating some shortcomings in spectral consistency, although it remains within the spectral consistency confidence interval. Overall, SRCNN super-resolution reconstruction significantly outperforms traditional methods in key metrics such as global accuracy, denoising capability, and structural fidelity, with a comprehensive performance improvement exceeding 20%. This demonstrates the superiority of SRCNN super-resolution reconstruction over traditional bicubic interpolation's fixed kernel method in nighttime image detail reconstruction, providing higher-quality input for subsequent processing.

[0092] The image results obtained by the fusion method are attached. Figure 9As shown, the original Luojia-1 nighttime light image suffers from limitations in spatial resolution, resulting in blurred boundaries and a lack of detailed information. While traditional GS-based image fusion methods offer some improvement, issues such as blurred edges and missing textures still exist. In contrast, the improved starGAN-generated fusion image exhibits significantly enhanced visual quality. The spatial structural features of the multispectral image are effectively transferred to the nighttime light data, resulting in significantly improved edge sharpness for linear features such as roads and rivers, and clearer building outlines. Furthermore, the fusion image restores texture features such as road network branch structures missing from the original data; and nighttime light feature analysis transforms diffuse light spots into resolvable units with clear spatial distribution patterns.

[0093] Table 2: Quantitative Indicators

[0094]

[0095] like Figure 2 As shown, quantitative experimental analysis reveals that the traditional GS fusion method has significant performance defects in fusing nighttime light images and multispectral data. Its quantitative analysis indicators are low, confirming a systematic deficiency in cross-modal feature synergy, leading to prominent spectral distortion and spatial texture loss in the fused image. In contrast, the improved StarGAN model of this invention exhibits multidimensional performance advantages: its image reconstruction quality indicators PSNR and SSIM reach 42.75 dB and 0.983, respectively, verifying its accurate reconstruction capability of nighttime light radiation intensity distribution and spatial structure; its spectral fidelity indicators SAM are 0.342 and D_Spectral is 0.218, indicating effective suppression of spectral distortion; and its spatial quality indicator D_Spatial is 0.017, achieving near-lossless control of spatial detail loss.

[0096] At the technical implementation level, this invention innovatively constructs a cross-scale fusion architecture for Luojia-1 130-meter nighttime light imagery and Landsat 8 30-meter multispectral imagery, generating a 30-meter high-resolution nighttime light product with geometrically sharpened features. The fusion result significantly improves spatial detail representation. Compared to the diffuse light spots in traditional nighttime light data, the fused imagery can clearly resolve them into micro-scale road networks and building outlines, with a spatial resolution improvement rate of more than 4 times. Through both qualitative and quantitative evaluation, it can be proven that this invention successfully achieves lossless transfer of spatial information from multispectral images while maintaining the background characteristics of nighttime light radiation.

[0097] This invention precisely solves the fusion challenges of multi-source remote sensing data in terms of differences in radiometric response, heterogeneity of ground features, and scale mismatch by improving the multimodal remote sensing image fusion method of StarGAN. The constructed 30-meter-level high-resolution nighttime light dataset fills the gaps in existing technologies for observing mesoscale nighttime surface processes, and provides reliable data support for high-precision urban expansion monitoring and energy infrastructure assessment based on nighttime light remote sensing, which has important engineering application value.

[0098] The above embodiments are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all equivalent technical solutions also fall within the scope of this invention, and the patent protection scope of this invention should be defined by the claims.

[0099] All technical contents not described in detail in this invention are publicly known technologies.

Claims

1. An improved multimodal remote sensing image fusion method using StarGAN (Star-shaped Generative Adversarial Network), characterized in that, Includes the following steps: Step S1: Acquire Landsat 8 multispectral images and Luojia-1 nighttime light images, and preprocess the Landsat 8 multispectral images and Luojia-1 nighttime light images. Step S2: Use SRCNN super-resolution convolutional neural network to perform super-resolution reconstruction on the Luojia-1 night light image to obtain an upsampled night light image with the same spatial resolution as the Landsat 8 multispectral image. Step S3: Input the Landsat 8 multispectral image and the upsampled night light image into the generator of the improved StarGAN to generate a fused image; Step S4: Use the discriminator of the improved StarGAN to distinguish between the fused image and the real image generated by the generator, and feed the discrimination result back to the generator; Step S5: Construct a multiple loss function to jointly optimize the parameters of the generator and the discriminator.

2. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 1, characterized in that, In step S1, the preprocessing of the Landsat 8 multispectral image includes: seamless mosaicking, region of interest (ROI) extraction, radiometric calibration, and atmospheric correction; the preprocessing of the Luojia-1 nighttime light image includes: seamless mosaicking, reprojection, ROI extraction, and radiometric calibration.

3. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 1, characterized in that, Step S1 further includes converting the data type of the Luojia-1 night-light image from 32-bit floating-point to 16-bit integer.

4. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 1, characterized in that, In step S2, the Luojia-1 nighttime image is upsampled by 4 times using an SRCNN super-resolution convolutional neural network based on PixelShuffle.

5. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 4, characterized in that, The SRCNN model in step S2 includes three convolutional layers and one pixelShuffle layer.

6. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 1, characterized in that, The improved StarGAN generator includes: a night vision image encoder, a multispectral image encoder, a detail enhancement fusion module, and a decoder.

7. The multimodal remote sensing image fusion method of the improved StarGAN (StarGAN) according to claim 6, characterized in that, In step S3, the detail enhancement fusion module in the generator of the improved StarGAN includes: an edge detection convolutional layer, an attention calculation module, and a residual enhancement convolutional layer.

8. The method according to claim 1, characterized in that, In step S4, the discriminator of the improved StarGAN adopts a multi-layer convolutional structure, combined with adaptive average pooling and multi-domain scoring mechanism.

9. The method according to claim 1, characterized in that, In step S5, the multiple loss functions include: cross-entropy loss, structural similarity index (SSIM) loss, perceptual loss, and L1 loss.