Noctilucent image reconstruction method based on cross-modal attention network

Abstract

The invention belongs to the technical field of remote sensing data processing and information fusion, and particularly relates to a noctilucent image reconstruction method based on a cross-modal attention network, and the method comprises the steps: 1, screening and obtaining noctilucent image data and high-resolution panchromatic optical image data of the same region; step 2, preprocessing the noctilucent image and the high-resolution optical image respectively; step 3, constructing a cross-modal attention reconstruction network model; 4, dividing the preprocessed image data into a training set and a test set, inputting the training set and the test set into a cross-modal attention reconstruction network model, and carrying out network training and testing to obtain a final cross-modal attention reconstruction network model; and 5, cutting an image of a research area into a size required by the model, inputting the obtained final cross-modal attention reconstruction network model, and obtaining a reconstructed high-resolution noctilucent image. The method provided by the invention can generate high-resolution noctilucent data with spatial details.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing data processing and information fusion technology, specifically relating to a method for reconstructing nighttime light images based on a cross-modal attention network. Background Technology

[0002] Nighttime light remote sensing data effectively describes the distribution of artificial light on the Earth's surface and is highly correlated with human activities. In addition, nighttime light remote sensing images have advantages such as periodic observation, spatial continuity, and independent objectivity, and have become an important indicator of the intensity of nighttime social and economic activities in cities.

[0003] Existing nighttime light remote sensing data generally has low resolution; for example, NPP-VIIRS nighttime light data has a resolution of approximately 500 meters, and Luojia-1 nighttime light data has a resolution of 130 meters. Insufficient spatial resolution often limits the fine-scale modeling and application of nighttime light data. The higher the spatial resolution of remote sensing images, the richer the information they provide. The three-band nighttime light data provided by Jilin-1 satellite can capture richer spectral information, surpassing traditional DMSP / OLS or VIIRS single-band data. However, the lack of texture information in the data is still insufficient to support fine-scale applications.

[0004] Traditional image fusion methods, such as IHS transform, Brovey transform, and principal component analysis, are widely used to enhance the spatial detail features of remote sensing images. However, these methods have significant limitations when fusing nighttime light images with high-resolution optical images.

[0005] Unnatural injection of spatial details: Traditional methods typically overlay optical image details onto nighttime images globally and indiscriminately, failing to adaptively identify which textures are valid urban structures (such as roads and building outlines) and which are noise, resulting in artifacts or unnatural textures in the fused images.

[0006] Weak modal difference processing capability: Nighttime light imagery and optically resolved imagery differ significantly in imaging mechanisms and radiometric characteristics. Nighttime light imagery reflects the intensity of spontaneous radiation from the Earth's surface, while optically resolved imagery reflects the intensity of sunlight reflected from the surface. Traditional methods struggle to establish deep semantic relationships between these two heterogeneous modal data.

[0007] Deep learning-based reconstruction networks have shown great potential, but while there are many methods for image reconstruction based on optical satellite imagery and radar data, research on reconstruction based on nighttime light data is relatively limited. There is an urgent need for a new nighttime light image reconstruction technique that can fully leverage the complementary advantages of heterogeneous data sources while also achieving high spatial resolution and high spectral fidelity. Summary of the Invention

[0008] The purpose of this invention is to provide a night light image reconstruction method based on a cross-modal attention network. This method can effectively fuse low spatial resolution night light data with high resolution panchromatic data to generate high resolution night light data with spatial details. This overcomes the problems of existing night light data having relatively low spatial resolution, ignoring detailed information, and being difficult to monitor human activities with faint lights.

[0009] Technical solution to achieve the purpose of this invention:

[0010] A nighttime light image reconstruction method based on a cross-modal attention network includes:

[0011] Step 1: Filter and acquire nighttime light image data and high-resolution panchromatic optical image data of the same area;

[0012] Step 2: Preprocess the nighttime light images and high-resolution optical images respectively to generate a nighttime light image dataset and a high spatial resolution optical image dataset;

[0013] Step 3: Using data from the nighttime light image dataset and the high spatial resolution optical image dataset, construct a cross-modal attention reconstruction network model;

[0014] Step 4: Divide the night light image dataset and high spatial resolution optical image dataset generated in Step 2 into training set and test set, and input them into the cross-modal attention reconstruction network model constructed in Step 3 to carry out network training and testing to obtain the final cross-modal attention reconstruction network model.

[0015] Step 5: Crop the study area image to the size required by the model, input it into the final cross-modal attention reconstruction network model obtained in Step 4, and obtain the reconstructed high-resolution night light image.

[0016] Furthermore, the preprocessing in step two includes image reprojection, resampling, radiometric calibration, noise reduction, georegistration, and cropping.

[0017] Furthermore, step three includes:

[0018] Step 3.1: Construct a dual-branch network structure from the night light image dataset and the high spatial resolution optical image dataset. Extract shallow features from the night light image and the optical image respectively through the convolution module to form low-dimensional fusion features.

[0019] Step 3.2: Obtain deep features of the image through the channel attention module to obtain the fused feature map;

[0020] Step 3.3: Use convolutional and deconvolutional layers to perform feature map dimensionality reduction and upsampling, pass image feature information through skip connections, and obtain a high-resolution fused image through convolutional layers;

[0021] Step 3.4: Calculate the loss between the fused image and the label using the hybrid loss function, and update the network parameters using the Adam optimizer. When the loss function is minimized, the reconstruction model reaches its optimum. Save the optimal model parameters to obtain the cross-modal attention reconstruction network model.

[0022] Further, step 3.1 includes: inputting the night light image dataset and the high spatial resolution optical image dataset into the first branch and the second branch of the dual-branch network structure, respectively; the first branch includes two Conv3×3 convolutional layers for extracting shallow features from the night light image; the second branch includes two Conv3×3 convolutional layers for extracting shallow features from the high spatial resolution optical image; concatenating the shallow features of the night light image and the shallow features of the optical image along the channel dimension to form a multi-channel concatenated feature map; and then performing dimensionality reduction through a Conv1×1 convolutional layer to form a low-dimensional fused feature.

[0023] Further, step 3.2 includes: the first layer of the channel attention module includes two max pooling layers and two deconvolutional layers (Deconv3×3). The two max pooling layers extract deep features from the low-dimensional fused features, and the two deconvolutional layers (Deconv3×3) restore the feature map to the input size, so that the attention mechanism region corresponds to the input pixel, generating the depth feature map of the first layer; the second layer of the channel attention module includes a Conv1×1 convolutional layer, which extracts deep features from the low-dimensional fused features, generating the depth feature map of the second layer; the depth feature maps of the first and second layers are multiplied element-wise, and the weights of each channel feature map are adjusted to obtain the fused feature map.

[0024] Further, step 3.3 includes: the encoding part uses a convolutional layer Conv3×3 to reduce the dimensionality of the fused feature map; the decoding part upsamples the feature map through a deconvolutional layer Deconv3×3 to restore image detail information; the image feature information is passed to the decoding part through the network encoding part via skip connections; and finally, a high-resolution night light fused image is obtained through a convolutional layer with a Tanh activation function.

[0025] Furthermore, in step 3.4, the formula for calculating the loss between the fused image and the label is:

[0026] T Loss =α*L m +β*L n

[0027] Where α and β represent the coefficients balancing the two loss weights; L m L represents pixel-level loss. n Indicates structural loss; T LossIndicates the total loss;

[0028]

[0029] in, This represents L2 normal form, where w*h represents the data dimension, and I... out I represents the output fused image. real Indicates a label, This represents the gradient operator.

[0030] Furthermore, in step four, the ratio of the training set to the test set is 8:2.

[0031] The beneficial technical effects of this invention are as follows:

[0032] 1. This invention provides a night-light image reconstruction method based on a cross-modal attention network, achieving adaptive spatial detail injection. The core cross-modal attention module of this invention can dynamically calculate the correlation between low-resolution night-light features and high-resolution panchromatic features. This mechanism enables the network to "intelligently" find the spatial details most relevant to the night-light region from the panchromatic image and perform weighted fusion.

[0033] 2. This invention provides a nighttime light image reconstruction method based on a cross-modal attention network, which maximizes the preservation of the spectral characteristics of the nighttime light image. Through a dual-branch structure and skip connection design, this invention effectively separates and protects the spectral information of the original nighttime light image. Deep feature paths incorporate spatial details through an attention mechanism, while shallow feature paths directly transmit the original spectral information to the output layer through skip connections.

[0034] 3. This invention provides a nighttime light image reconstruction method based on a cross-modal attention network, generating images that combine high resolution and high usability. The final generated nighttime light image possesses both high spatial resolution and high spectral fidelity. Attached Figure Description

[0035] Figure 1 A flowchart of a night vision image reconstruction method based on a cross-modal attention network provided by the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the construction of the cross-modal attention reconstruction network model in the nighttime light image reconstruction method based on cross-modal attention network provided by the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0038] like Figure 1As shown, this invention provides a nighttime light image reconstruction method based on a cross-modal attention network, specifically including the following steps:

[0039] Step 1: Filter and acquire low spatial resolution night light image data and high spatial resolution panchromatic optical image data of the same area, ensuring that the spatial positions of the night light image data and the panchromatic optical image data are consistent.

[0040] Step 2: Preprocess the nighttime light images and high-resolution optical images respectively to generate a nighttime light image dataset and a high spatial resolution optical image dataset;

[0041] Preprocessing of nighttime light and optical images includes: image reprojection, resampling, radiometric calibration, and noise reduction. Georegistration and cropping are performed on the nighttime light data and optical images to generate the nighttime light image dataset LR_NTL and the high spatial resolution optical image dataset HR_O, respectively.

[0042] Step 3: Using data from the nighttime light image dataset and the high spatial resolution optical image dataset, construct a cross-modal attention reconstruction network model;

[0043] like Figure 2 As shown, this step specifically includes:

[0044] Step 3.1: Construct a dual-branch network structure and extract shallow features to form low-dimensional fused features.

[0045] The nighttime light image dataset and the high spatial resolution optical image dataset are respectively input into the first and second branches of the dual-branch network structure. The first branch includes two Conv3×3 convolutional layers for extracting shallow features from the nighttime light images. The second branch includes two Conv3×3 convolutional layers for extracting shallow features from the high spatial resolution optical images. The shallow features of the nighttime light images and the shallow features of the optical images are concatenated along the channel dimension to form a multi-channel concatenated feature map. Then, the dimensionality is reduced by a Conv1×1 convolutional layer to form a low-dimensional fused feature.

[0046] Step 3.2: Use the channel attention module to extract deep features from the image to obtain the fused feature map.

[0047] The first layer of the channel attention module consists of two max-pooling layers and two deconvolutional layers (Deconv3×3). The two max-pooling layers extract deep features from the low-dimensional fused features, and the two deconvolutional layers (Deconv3×3) restore the feature map to the input size, ensuring that the attention mechanism region corresponds to the input pixels, thus generating the first layer's deep feature map. The second layer of the channel attention module contains a Conv1×1 convolutional layer, which extracts deep features from the low-dimensional fused features, generating the second layer's deep feature map. The deep feature maps from the first and second layers are then element-wise multiplied, and the weights of each channel feature map are adjusted to obtain the fused feature map.

[0048] Step 3.3: Encoding and decoding to obtain a high-resolution nighttime light fusion image.

[0049] The encoding part uses a 3×3 convolutional layer (Conv3×3) to reduce the dimensionality of the fused feature map, while the decoding part uses a 3×3 deconvolutional layer (Deconv3×3) to upsample the feature map and recover image detail information. Skip connections are used to pass image feature information from the encoding part to the decoding part. Finally, a high-resolution night-light fused image is obtained through a convolutional layer with a Tanh activation function.

[0050] Step 3.4: Calculate the loss between the fused image and the label using the hybrid loss function, and update the network parameters using the Adam optimizer. When the loss function is minimized, the reconstruction model reaches its optimum. Save the optimal model parameters to obtain the cross-modal attention reconstruction network model.

[0051] The formula for calculating the loss between the fused image and the label is:

[0052] T Loss =α*L m +β*L n

[0053] Where α and β represent the coefficients balancing the two loss weights; L m L represents pixel-level loss. n Indicates structural loss; T Loss This indicates the total loss.

[0054]

[0055] in, This represents L2 normal form, where w*h represents the data dimension, and I... out I represents the output fused image. real Indicates a label, This represents the gradient operator.

[0056] Step 4: Divide the night light image dataset and high spatial resolution optical image dataset generated in Step 2 into training set and test set, and input them into the cross-modal attention reconstruction network model constructed in Step 3 to carry out network training and testing to obtain the final cross-modal attention reconstruction network model.

[0057] Step 5: Crop the study area image to the size required by the model, input it into the final cross-modal attention reconstruction network model obtained in Step 4, obtain the reconstructed high-resolution night light image, and stitch it together.

[0058] Example 1

[0059] Taking the nighttime light data from Jilin-1 and the panchromatic image data from Gaofen-1 in April 2023 as examples, this embodiment provides a nighttime light image reconstruction method based on a cross-modal attention network, specifically including the following steps:

[0060] Step 1: Acquire Jilin-1 nighttime light data and Gaofen-1 panchromatic image data for April 2023. Jilin-1 nighttime light data is in 3 bands with a spatial resolution of 0.92 meters. Gaofen-1 panchromatic image data has a spatial resolution of 0.5 meters.

[0061] Step 2: Preprocess the acquired Jilin-1 nighttime light image and Gaofen-1 panchromatic image by reprojection, resampling, radiometric calibration, and georegistration. Use the NNDiffuse method to fuse the images and generate pseudo-labels. Crop the Jilin-1 nighttime light image, Gaofen-1 image, and fused image separately.

[0062] Step 2.1: Reproject, resample, and radiometrically calibrate the panchromatic data from Jilin-1 and Gaojing-1 satellites, respectively.

[0063] Step 2.2: Based on the Gaofen-1 panchromatic band image, perform georegistration on the Jilin-1 image;

[0064] Step 2.3: Assign a value of 0 to any DN value less than 0 in the night-light image;

[0065] Step 2.4: Based on the NNDiffuse method, the nighttime light data of Jilin-1 and the panchromatic band of Gaofen-1 are fused to generate fused data with a resolution of 0.5 meters, which is used as pseudo-labels for network training.

[0066] Step 2.5: Crop the preprocessed Jilin-1 night vision data into an image set LR_NTL of size 128×128×3 pixels, and crop the Gaofen-1 panchromatic image into an image set HR_PAN of size 256×256×1 pixels.

[0067] Step 3: Construct a cross-modal attention network, which includes four main steps: learning shallow features of nighttime and optical images through a dual-branch Conv3×3 convolutional module and performing image stitching; acquiring deep image features through a channel attention module; performing feature map dimensionality reduction and upsampling using convolutional and deconvolutional layers to restore image detail information, and passing image feature information through skip connections; obtaining a high-resolution fused image through convolutional layers with a Tanh activation function; calculating the loss using a hybrid loss function, and updating the network parameters using the Adam optimizer.

[0068] T Loss =α*L m +β*L n

[0069] α and β are coefficients that balance the two loss weights. m L represents pixel-level loss. n T represents structural loss. Loss This indicates the total loss.

[0070]

[0071] This represents L2 normal form, where w*h represents the data dimension, and I... out For the output fused image, I real For tags, This represents the gradient operator.

[0072] The specific settings for training the cross-modal attention network are as follows:

[0073] Step 3.1: Upsample the LR_NTL image set to the same size as the HR_PAN image set. The two branch modules respectively use two Conv3×3 convolutional modules to perform shallow feature extraction on the LR_NTL and HR_PAN data, with a stride of 1, padding of SAME, and 128 channels. After feature concatenation, the image is reduced in dimensionality using a Conv1×1 convolutional layer.

[0074] Step 3.2, Channel Attention Module: The first layer extracts deep features through two max-pooling layers, and then restores the feature map to the input size through two deconvolutional layers (Deconv3×3). The second layer contains a convolutional layer (Conv1×1). The feature maps from the first and second layers are multiplied element-wise and fused for output, resulting in 128 channels.

[0075] Step 3.3: The encoding part uses a Conv3×3 convolutional layer to fuse the feature maps and reduce their dimensionality. The decoding part uses a Deconv3×3 deconvolutional layer to upsample the feature maps and restore image detail information. Both layers have 128 channels. The last convolutional layer is activated by the Tanh activation function, with 3 channels.

[0076] Step 3.4: During training, a hybrid loss function is used, expressed as follows:

[0077] T Loss =α*L m +β*L n

[0078] α and β are coefficients that balance the two loss weights. m L represents pixel-level loss. n T represents structural loss. Loss This indicates the total loss.

[0079]

[0080] This represents L2 normal form, where w*h represents the data dimension, and I... out For the output fused image, I real For tags, This represents the gradient operator.

[0081] The network parameters are updated by using a loss function and the Adam optimizer. When the loss function is minimized, the reconstructed model reaches its optimum. The optimal model parameters are saved to obtain the cross-modal attention reconstruction network model.

[0082] Step 4: Divide the cropped images from Step 2 into a training set and a test set in an 8:2 ratio. Input the training set into the network model from Step 3 to train the model, and use the test set to test the model.

[0083] Step 5: Crop the study area image to the size required by the model. The output of the model in Step 4 is the reconstructed high-resolution night light image. Then, stitch the images together.

[0084] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention. All contents not described in detail in the present invention can be derived from existing technologies.

Claims

1. A nighttime light image reconstruction method based on a cross-modal attention network, characterized in that, include: Step 1: Filter and acquire nighttime light image data and high-resolution panchromatic optical image data of the same area; Step 2: Preprocess the nighttime light images and high-resolution optical images respectively to generate a nighttime light image dataset and a high spatial resolution optical image dataset; Step 3: Using data from the nighttime light image dataset and the high spatial resolution optical image dataset, construct a cross-modal attention reconstruction network model; Step 4: Divide the night light image dataset and high spatial resolution optical image dataset generated in Step 2 into training set and test set, respectively, and input them into the cross-modal attention reconstruction network model constructed in Step 3. Conduct network training and testing to obtain the final cross-modal attention reconstruction network model. Step 5: Crop the study area image to the size required by the model, input it into the final cross-modal attention reconstruction network model obtained in Step 4, and obtain the reconstructed high-resolution night light image.

2. The nighttime image reconstruction method based on a cross-modal attention network according to claim 1, characterized in that, The preprocessing in step two includes image reprojection, resampling, radiometric calibration, noise reduction, georegistration, and cropping.

3. The nighttime image reconstruction method based on a cross-modal attention network according to claim 1, characterized in that, Step three includes: Step 3.1: Construct a dual-branch network structure from the night light image dataset and the high spatial resolution optical image dataset. Extract shallow features from the night light image and the optical image respectively through the convolution module to form low-dimensional fusion features. Step 3.2: Obtain deep features of the image through the channel attention module to obtain the fused feature map; Step 3.3: Use convolutional and deconvolutional layers to perform feature map dimensionality reduction and upsampling, pass image feature information through skip connections, and obtain a high-resolution fused image through convolutional layers; Step 3.4: Calculate the loss between the fused image and the label using the hybrid loss function, and update the network parameters using the Adam optimizer. When the loss function is minimized, the reconstruction model reaches its optimum. Save the optimal model parameters to obtain the cross-modal attention reconstruction network model.

4. The nighttime image reconstruction method based on a cross-modal attention network according to claim 3, characterized in that, Step 3.1 includes: inputting the night light image dataset and the high spatial resolution optical image dataset into the first branch and the second branch of the dual-branch network structure, respectively; the first branch includes two Conv3×3 convolutional layers for extracting shallow features from the night light image; the second branch includes two Conv3×3 convolutional layers for extracting shallow features from the high spatial resolution optical image; concatenating the shallow features of the night light image and the shallow features of the optical image along the channel dimension to form a multi-channel concatenated feature map; and then performing dimensionality reduction through a Conv1×1 convolutional layer to form a low-dimensional fused feature.

5. The nighttime image reconstruction method based on a cross-modal attention network according to claim 3, characterized in that, Step 3.2 includes: the first layer of the channel attention module includes two max pooling layers and two deconvolutional layers (Deconv3×3). The two max pooling layers extract deep features from the low-dimensional fused features, and the two deconvolutional layers (Deconv3×3) restore the feature map to the input size, so that the attention mechanism region corresponds to the input pixel, generating the depth feature map of the first layer; the second layer of the channel attention module includes a Conv1×1 convolutional layer, which extracts deep features from the low-dimensional fused features, generating the depth feature map of the second layer; the depth feature maps of the first and second layers are multiplied element-wise, and the weights of each channel feature map are adjusted to obtain the fused feature map.

6. The nighttime image reconstruction method based on a cross-modal attention network according to claim 3, characterized in that, Step 3.3 includes: the encoding part uses a Conv3×3 convolutional layer to reduce the dimensionality of the fused feature map; the decoding part upsamples the feature map through a Deconv3×3 deconvolutional layer to restore image detail information; the image feature information is passed to the decoding part through the network encoding part via skip connections; and finally, a high-resolution night light fused image is obtained through a convolutional layer with a Tanh activation function.

7. The nighttime image reconstruction method based on a cross-modal attention network according to claim 3, characterized in that, In step 3.4, the formula for calculating the loss between the fused image and the label is: T Loss =α*L m +β*L n Where α and β represent the coefficients balancing the two loss weights; L m L represents pixel-level loss. n Indicates structural loss; T Loss Indicates the total loss; in, This represents L2 normal form, where w*h represents the data dimension, and I... out I represents the output fused image. real Indicates a label, This represents the gradient operator.

8. The nighttime image reconstruction method based on a cross-modal attention network according to claim 1, characterized in that, In step four, the ratio of the training set to the test set is 8:2.

Images