Non-reference low-illumination remote sensing image enhancement method based on multi-branch feature fusion

Abstract

The invention relates to a non-reference low-illumination remote sensing image enhancement method based on multi-branch feature fusion. A model of the invention is totally divided into three modules: a denoising module, a multi-dimensional feature fusion enhancement module and a detail recovery module. The de-noising module is mainly responsible for de-noising the image; the multi-dimensional feature fusion enhancement module obtains an enhanced illumination image by designing a semantic information extraction branch, a spatial information extraction branch and a frequency information extraction branch and fully utilizing feature information under three different dimensions; and a detail recovery module designs a space detail recovery branch and a frequency detail recovery branch under the guidance of the enhanced illumination image in the previous stage to respectively extract space information and frequency information of the image, and a finally generated reflectivity image is used as an enhanced image. The model significantly improves the practicability and generalization ability of low-illumination remote sensing image enhancement.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and remote sensing image enhancement, and relates to a reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion, and specifically to a reference-free low-light remote sensing image enhancement algorithm that combines spatial information, frequency information and semantic information. Background Art

[0002] Low-light remote sensing image enhancement is an important research direction in the field of remote sensing image processing. Its purpose is to improve the quality of remote sensing images acquired under low-light conditions through a series of technical means, so as to better serve the needs of multiple fields where remote sensing images are applied. Low-light remote sensing image enhancement can be used for tasks such as night patrols and monitoring, providing important guarantees for regional security; it can also be used for many applications such as nighttime urban planning, disaster monitoring, and environmental monitoring, providing strong support for urban management and emergency response. Compared with low-light image enhancement tasks in natural scenes, low-light remote sensing image enhancement methods face many challenges: 1) Remote sensing images usually have higher spatial resolution and wider scales, and more object information needs to be retained at different scales during processing, while natural images tend to focus more on local areas. 2) Remote sensing images are greatly affected by factors such as the atmosphere and clouds. The uneven illumination and shadow effects in low light are more obvious, and the enhancement process needs to consider these complex factors. 3) The enhancement of remote sensing images not only needs to improve the visual effect, but also retain the ground object information to support subsequent remote sensing analysis tasks, such as land use classification, target detection, etc. These tasks have high requirements for image accuracy, while the enhancement of natural images relies more on visual effects and the evaluation criteria are relatively simple. Therefore, the study of low-light remote sensing image enhancement algorithms can not only improve the visual quality of images, but also provide key data support for specific application fields such as environmental monitoring, disaster assessment, autonomous driving and precision agriculture, and has important research and application value. At present, the mainstream low-light remote sensing image enhancement algorithms are mainly based on deep learning. According to the different supervision information, the related work can be divided into the following three categories:

[0003] The first is supervised learning, the most widely used method in the field of image enhancement. By training on a paired image dataset, the model can learn the mapping relationship between low-light images and enhanced images, and then use the trained mapping neural network to enhance low-light images. Among them, Lore et al. proposed a stacked sparse denoising autoencoder in the paper "K. Lore, A. Akintayo, and S. Sarkar, "LLNet: A Deep Autoencoder Approach to Natural Low-Light Image Enhancement," Pattern Recognition, vol. 61, pp. 650-662, 2017." They use this encoder to learn the latent information in the image and then enhance and denoise the image based on the obtained information. The introduction of this algorithm has opened up the application of deep learning in the field of low-light enhancement. In their paper "W.Wu, J.Weng, P.Zhang, X.Wang, W.Yang, and J.Jiang,"Uretinex-Net: Retinex-Based Deep Unfolding Network for Low-Light Image Enhancement," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 5901-5910," Wu et al. constructed the URetinex-Net model. This model transforms the optimization problem into a learnable network structure, adaptively fitting implicit priors in a data-driven manner to achieve noise suppression and detail preservation. Three carefully designed modules are responsible for data-dependent initialization, efficient unfolding optimization, and user-specified enhancement, iteratively learning to refine the illumination layer and the reflection layer. In their paper "Y. Wang, R. Wan, W. Yang, H. Li, L. Chau, and A. Kot, "Low-Light Image Enhancement with Normalizing Flow," in Proc. AAAI Conference on Artificial Intelligence, 2022, pp. 2604-2612," Wang et al. constructed the LLFlow model. This model uses a normalizing flow model to model the relationship between low-light and enhanced-light images and the conditional distribution of normally exposed images. The resulting model achieves both noise and artifact suppression while adjusting for lighting. Supervised learning generally produces good enhancement results, but it relies on image pairs for training, which poses challenges such as poor generalization, prone to overfitting, and cumbersome dataset preparation.

[0004] The second type is unsupervised learning, which does not require a paired dataset during training. The model can learn the statistical characteristics of the data based on the enhanced images and then infer how to enhance low-light images. Unsupervised models based on GANs are widely used for image enhancement. GAN models typically consist of two parts: a generator and a discriminator. GAN models require training the generator to generate realistic synthetic data from the data of interest, and training the discriminator to distinguish fake samples output by the generator from real data. Based on the GAN model, Jiang et al. proposed the first deep learning method for image enhancement using unsupervised learning in the paper "Y.Jiang, X.Gong, D.Liu, Y.Cheng, C.Fang, X.Shen, J.Yang, P.Zhou,and Z.Wang,"Enlightengan:Deep Light Enhancement without PairedSupervision,"IEEE Transactions on Image Processing,vol.30,pp.2340-2349,2021." The algorithm was named EnlightenGAN algorithm. It used unpaired images for training for the first time, and guided the enhancement of the generator by taking the inverted grayscale image of the input image as an attention mechanism, and used both global and local discriminators to improve the enhancement effect. In their paper "S. Yang, M. Ding, Y. Wu, Z. Li, and J. Zhang, "Implicit Neural Representation for Cooperative Low-Light Image Enhancement," in Proc. IEEE International Conference on Computer Vision, 2023, pp. 12918-12927," Yang et al. proposed the NeRCo algorithm, which leverages the similar brightness of images of varying brightness after implicit neural representation to normalize the degradation distribution of low-light images. To achieve better perceptual results, the algorithm first introduced the text-image pair prior of the CLIP model and proposed a collaborative training mechanism, enabling the model to be trained on unpaired data. Unsupervised learning avoids the tedious task of collecting paired training images and can automatically learn complex image features from the data, providing greater flexibility in model training and making the model more robust across a wider range of scenarios and lighting conditions. However, this method requires more prior knowledge or complex model design.

[0005] The third type is reference-free learning. This method does not require paired or unpaired image datasets during training. It dynamically adjusts the enhancement effect based on the given image, resulting in strong generalization capabilities. This type of algorithm uses a carefully designed loss function and image prior knowledge as a supervisory signal to guide network training. In their paper, "ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2024, pp. 3015-3024, Shi et al. proposed a ZERO-IG model that is independent of training data and noise distribution. This model fully considers the coupled relationship between denoising and enhancement, achieves pixel-level adaptive enhancement by adjusting each pixel in the low-light image, and effectively removes noise in the form of reflections, resulting in better overall network performance. Inspired by the theory of physical light transport, Wang et al. designed a physical quadruple invariant prior as a bridge between low-light images and enhanced light images in the paper "W.Wang, H.Yang, J.Fu, and J.Liu,"Zero-Reference Low-Light Enhancement Via Physical Quadruple Priors," in Proc.IEEE Conference on Computer Vision and Pattern Recognition, 2024, pp.26057-26066." The training process only requires enhanced light images, and the model can perform excellent performance in various low-light scenes.In the paper "X.Lv, S.Zhang, C.Wang, Y.Zheng, B.Zhong, and C.Li, "Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2024, pp. 25378-25388," Lv et al. embed Fourier priors into a pre-trained diffusion model to simultaneously address brightness and structure degradation. They decompose the sampling results of the back-diffusion process in the Fourier domain and use the amplitude of the generated prior to align the enhanced brightness with the distribution of natural images. They further design a spatial-frequency alternating optimization strategy to gradually refine the input phase. Because no-reference learning does not rely on paired or unpaired image data, the enhancement effect is generally not as fine-grained as supervised and unsupervised methods, especially in detail restoration and color correction. Summary of the Invention

[0006] Technical problems to be solved

[0007] To overcome the shortcomings of existing technologies, this paper proposes a no-reference low-light remote sensing image enhancement method based on multi-branch feature fusion. This algorithm integrates spatial, frequency, and semantic information. No paired data is required; the training process only requires low-light remote sensing images.

[0008] The purpose of the present invention is to improve the following aspects:

[0009] 1. Existing methods often focus on single-dimensional information processing during the enhancement process, lacking the coordinated use of spatial information, frequency domain information, and semantic information, resulting in limited enhancement effects;

[0010] 2. In complex scenes, low-light remote sensing images often suffer from uneven lighting distribution, diverse features, and rich details. Existing algorithms have difficulty balancing details and global brightness in complex scenes, resulting in unsatisfactory restoration results.

[0011] 3. The training process of existing methods mostly relies on paired data. However, in practical applications, obtaining high-quality paired data is often limited by many practical factors.

[0012] Technical Solution

[0013] A reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion, characterized by the following steps:

[0014] Perform semantic segmentation on low-light remote sensing images to obtain semantic feature map C;

[0015] Denoising the low-light remote sensing image to obtain a denoised low-light remote sensing image;

[0016] The denoised low-light remote sensing image is processed in parallel by the spatial information extraction branch to obtain the spatial information feature map A; and the frequency information feature map B is processed by the frequency information extraction branch;

[0017] The spatial information image feature map A and the semantic feature map C are processed by the input semantic information extraction branch to extract the related semantic information feature map D;

[0018] The semantic information feature map D, spatial information feature map A and frequency information feature map B are convolved together to output the enhanced illumination image F;

[0019] The denoised low-light remote sensing image and the enhanced light image F are processed according to the Retinex theory to obtain the reflectance map E;

[0020] The enhanced illumination image F and reflectance map E are spliced in the channel direction, and then the frequency detail recovery branch is performed to obtain the frequency detail recovery branch feature map G. At the same time, the spatial detail recovery branch is processed in parallel to obtain the spatial detail recovery branch feature map H. The frequency detail recovery branch feature map G and the spatial detail recovery branch feature map H are simultaneously subjected to convolution processing to obtain an enhanced image of the low-light remote sensing image.

[0021] The denoising process and the spatial detail restoration branch process both adopt multi-level stacked convolution sequential processing.

[0022] The spatial information extraction branch processing adopts multi-level convolution sequential processing, and the third level convolution is convolved with the result of the second level convolution and the result of the first level convolution, and the result of the second level convolution is convolved with the result of the third level convolution at the same time.

[0023] The semantic information extraction branch processing flow includes convolution and flattening processing; the semantic feature map C processing flow is through a first-level convolution and flattening process, and then through an activation function and two-level convolution processing; the spatial information image feature map A undergoes a first-level convolution and flattening process, and the processed information is added to the processing flow of the semantic feature map C before and after the activation function. The semantic feature map C is finally convolved with the spatial information image feature map A to output the semantic information feature map D.

[0024] The frequency information extraction branch processing flow includes dilated convolution, convolution, Fourier transform and inverse Fourier transform; the amplitude information of the denoised low-light remote sensing image after dilated convolution, convolution and Fourier transform is sequentially subjected to two-stage convolution processing, followed by inverse Fourier transform and then connected to the two-stage convolution processing to output the frequency information feature map B; the processing result of the dilated convolution is simultaneously input into the last stage of convolution processing; the phase information of the Fourier transform is input into the inverse Fourier transform.

[0025] The frequency detail recovery branch processing flow includes dilated convolution, convolution, Fourier transform, and inverse Fourier transform. The enhanced illumination image F and reflectance map E are sequentially subjected to dilated convolution, convolution, and Fourier transform of the image feature map spliced in the channel direction. The phase information after the convolution is subjected to two-stage convolution processing, followed by inverse Fourier transform. The two-stage convolution processing is then connected to output the frequency detail recovery branch feature map G. The processing result of the dilated convolution is simultaneously input into the last stage of convolution processing; the amplitude information of the Fourier transform is input into the inverse Fourier transform.

[0026] A neural network model system for implementing the multi-branch feature fusion-based reference-free low-light remote sensing image enhancement method is characterized by comprising a denoising module, a semantic segmentation model, a multi-dimensional feature fusion enhancement module and a detail recovery module; the multi-dimensional feature fusion enhancement module comprises a semantic information extraction branch, a spatial information extraction branch and a frequency information extraction branch module; the detail recovery module comprises a spatial detail recovery branch and a frequency recovery branch module and a Retinex theory processing module; the denoising module and the semantic segmentation model are respectively input ends of the low-light remote sensing image, and the low-light remote sensing image output by the denoising module is respectively input into the Retinex theory processing module, the spatial information extraction branch and the frequency information extraction branch module; the output of the spatial information extraction branch and the output of the semantic segmentation model are connected to the semantic information extraction branch; the output of the spatial information extraction branch, the output of the semantic information extraction branch and the output of the frequency information extraction branch module are connected to a convolution module to output an enhanced illumination map; the enhanced illumination map is input into the Retinex theory processing module, the output reflectance map of the Retinex theory processing module and the enhanced illumination map are respectively input into the spatial detail recovery branch and the frequency recovery branch module through a channel direction splicing module, and the outputs of the two modules are passed through a convolution module to output an enhanced image.

[0027] The training process of the neural network model system is as follows:

[0028] Low-light remote sensing images are used as training sample images;

[0029] Use the iSAID-dark dataset to train the semantic segmentation model WSCL, and input the trained model into the neural network model system as a low-light remote sensing image;

[0030] The activation function in the neural network model system is the ReLU activation function;

[0031] The overall loss function in the neural network model system is: The brightness coefficient Y H Indicates the average brightness of the enhanced image, set to 0.5, Y L Indicates the average brightness of the low-light image, L enhance represents the enhanced illumination map;

[0032] The pixel-by-pixel enhancement loss in the neural network model system is Among them S denoise is the denoised image output by the denoising module after the low-light image passes through, β = α -1 0.7 -α is a scaling factor that controls the degree of scaling and determines the level of contrast enhancement. Both α and β depend on S denoise brightness level.

[0033] A reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion and a method for using a trained neural network model system are characterized in that the low-light remote sensing image is used as the input image of the neural network model system, and an enhanced remote sensing image is obtained at the output end.

[0034] A computer program product, characterized in that it includes computer-executable instructions, which, when executed, are used to implement the reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion as described in claims 1 to 7.

[0035] Beneficial effects

[0036] This paper proposes a reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion. The model is divided into three modules: a denoising module, a multi-dimensional feature fusion enhancement module, and a detail recovery module. The denoising module is primarily responsible for denoising the image. The multi-dimensional feature fusion enhancement module fully utilizes feature information in three different dimensions by designing semantic information extraction branches, spatial information extraction branches, and frequency information extraction branches to obtain an enhanced illumination map. The detail recovery module, guided by the illumination map enhanced in the previous stage, designs spatial detail recovery branches and frequency detail recovery branches to extract the spatial and frequency information of the image, respectively. The resulting reflectance map serves as the enhanced image. This model significantly improves the practicality and generalization capabilities of low-light remote sensing image enhancement.

[0037] Compared with existing low-light image enhancement algorithms in remote sensing scenarios, this invention has the following advantages:

[0038] 1. Compared with the traditional method that only uses single-dimensional feature information, the present invention fully utilizes the multi-dimensional characteristics of the image by integrating spatial, frequency, and semantic information features, which can more comprehensively improve the image enhancement effect and make the enhanced image more outstanding in brightness, contrast and detail performance.

[0039] 2. By designing a semantic information extraction branch, the present invention can distinguish key objects in an image, such as buildings and roads, and avoid over-enhancement of non-important areas. This can significantly improve the accuracy of subsequent object detection and semantic segmentation tasks.

[0040] 3. The present invention is based on a reference-free enhancement framework and does not rely on any paired or unpaired training datasets. It breaks away from the traditional method’s reliance on labeled data and significantly reduces the cost of data acquisition and labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 : Specific flow chart of the present invention;

[0042] Figure 2 : The method of the present invention and the neural network model system structure diagram;

[0043] Figure 3 : Semantic information extraction branch method and structure diagram;

[0044] Figure 4 : Frequency information extraction branch method and structure diagram;

[0045] Figure 5 : Frequency detail recovery branch method and structure diagram;

[0046] Figure 6 : Schematic diagram of the low-light remote sensing image enhancement results of the present invention. DETAILED DESCRIPTION

[0047] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0048] The present invention proposes a reference-free low-light remote sensing image enhancement method that combines spatial information, frequency domain information, and semantic information, which mainly includes the following:

[0049] 1. First, download the training dataset iSAID-dark and obtain a total of 3755 synthetic low-light remote sensing training sample images with a pixel size of 512×512.

[0050] 2. Pre-train the semantic segmentation model weak-to-strong consistency learning (WSCL) using the iSAID-dark dataset. The trained model is then integrated into the low-light remote sensing image enhancement model to facilitate the subsequent extraction of semantic information.

[0051] 3. Design a denoising module, input the low-light remote sensing image, use several stacked convolution blocks to denoise the low-light remote sensing image, and output the low-light remote sensing image after noise removal.

[0052] 4. Design a multi-dimensional feature fusion enhancement module. This module fully extracts image features by designing semantic information extraction branches, spatial information extraction branches, and frequency information extraction branches. Then, the three branches are fused to output an enhanced illumination map.

[0053] 5. Design a detail restoration module. Based on Retinex theory, the reflectance map reflecting image details is generated by combining the de-noised low-light remote sensing image output by the denoising module and the enhanced illumination map output by the multi-dimensional feature fusion enhancement module. The enhanced illumination map and reflectance map are concatenated and input into this module in the channel direction. The spatial detail restoration branch and the frequency detail restoration branch are designed to output the enhanced reflectance map as the final enhanced image.

[0054] 6. Train the low-light remote sensing image enhancement network and obtain the trained model for low-light remote sensing image enhancement tasks in actual scenes.

[0055] To more clearly illustrate the technical solutions implemented in the present invention, the following briefly introduces the various modules required for the example description. Obviously, the figures described below are merely flowcharts of the present invention. Those skilled in the art can, without inventive effort, expand upon these figures and derive other figures.

[0056] Reference Figure 1 , the implementation steps of the present invention are as follows:

[0057] Step 1: Dataset Preparation

[0058] The training and test data used in the experiment are from the publicly available iSAID-dark dataset, which is built based on the test set of the iSAID dataset. It contains 3755 pairs of training images and 66 pairs of test images, with each image having a pixel size of 512×512.

[0059] Step 2: Algorithm Design

[0060] This paper designs a reference-free low-light remote sensing image enhancement algorithm based on multi-branch feature fusion. The entire model consists of three modules: a denoising module, a multi-dimensional feature fusion enhancement module, and a detail recovery module.

[0061] (1) Denoising module

[0062] Images captured in low-light environments suffer from insufficient photon capture, resulting in a low signal-to-noise ratio (SNR). This makes noise more noticeable in the image, and the presence of this noise can significantly negatively impact subsequent image processing tasks. Therefore, the denoising module plays a crucial role. Its primary function is to denoise input low-light remote sensing images to reduce the impact of noise on subsequent processing. Specifically, the module's network structure consists of three layers of 3×3 convolutions, each followed by a LeakyReLU activation function for nonlinear processing. This structure not only effectively removes noise from the image, but also preserves the image's essential structure and details.

[0063] (2) Multi-dimensional feature fusion enhancement module

[0064] This module is based on the Retinex theory, an algorithm proposed by Land et al. in the 1980s that aims to simulate how the human eye perceives image color and brightness. The theory assumes that an image S can be decomposed into a reflectance map R and an illumination map L using the following formula:

[0065] S=R·L

[0066] The illumination map represents the brightness information of the image, and the reflectance map represents the intrinsic properties of the image. This module mainly obtains an enhanced illumination map by utilizing spatial information, frequency information, and semantic information.

[0067] a. Spatial information extraction branch

[0068] The input of the spatial information extraction branch first passes through a 3×3 convolutional layer with 64 output channels. The data then enters a residual block, which consists of two 3×3 convolutional layers, each followed by a batch normalization (BN) layer and a ReLU activation function. The output of the residual block is added to the input via a residual connection to form the output of the residual block. This spatial information branch effectively captures and processes spatial features through three residual blocks with the same structure. Each residual block is designed to enhance feature extraction capabilities while maintaining the integrity of spatial information.

[0069] b. Frequency information extraction branch

[0070] Under the Fourier transform, the global information of the image can be extracted without introducing a large number of parameters, and the image can be converted from the spatial domain to the frequency domain. In the frequency domain, the larger the frequency, the faster the signal changes, which mainly represents the details (such as contours and edges) or noise in the image. The smaller the frequency, the slower the signal changes, which represents the background information in the image. After the Fourier transform, a matrix with the same size as the original image will be obtained, and each point is a complex number A+Bj. Specifically, considering the image Its Fourier transform function F is as follows:

[0071]

[0072] Where i and j represent the horizontal and vertical coordinates respectively, h and w represent the spatial coordinates, and the inverse process of F is expressed as F -1 Modulus describes the amplitude, and the direction arctanBA describes the phase angle. The brightness information mainly exists in the amplitude, while other information is preserved in the phase.

[0073] Based on this, the input of this module's network structure is the denoised low-light remote sensing image output by the denoising module. The input frequency information extraction branch first passes through a series of convolutional layers, including a 3×3 dilated convolution and a 3×3 standard convolution, each followed by a Reluctant Unit (ReLU) activation function. The feature map is then converted to the frequency domain via a Fourier transform, and the frequency representation is then decomposed into phase and amplitude components. The amplitude component is processed through three 3×3 convolutional layers to fully extract brightness-related feature information. It is then combined with the original phase component and converted back to the spatial domain via an inverse Fourier transform. Finally, the output is further processed through a 3×3 convolutional layer and a Reluctant Unit (ReLU) activation function to generate the final enhanced features.

[0074] c. Semantic information extraction branch

[0075] Semantic priors can provide a wealth of information to improve enhancement performance. To address this, a semantic segmentation network, Weak-to-Strong Consistency Learning (WSCL), is pre-trained using a large-scale dataset and introduced as a semantic information library for low-light remote sensing image enhancement algorithms. This semantic information library provides a semantic graph as a semantic prior, helping to improve the performance of low-light remote sensing image enhancement algorithms.

[0076] The semantic information extraction branch takes the output of the spatial information extraction branch as the image feature map and the semantic map extracted by the semantic segmentation network as the semantic feature map. They first undergo layer-normalization to stabilize the training process and accelerate convergence. The normalized feature map is flattened into a matrix of shape HW×C, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels. The flattened feature matrix is then multiplied with the semantic features via matrix multiplication to produce a C×C attention weight map. The features are then converted into a probability distribution via a Softmax layer. Finally, the output of the Softmax layer is element-wise multiplied with the original flattened feature map, combined with another layer-normalized and convolutional feature map, and reconstructed into the original H×W×C output feature map via element-wise addition. This process not only preserves spatial information but also enhances the expressiveness of features by introducing semantically-aware mapping and attention mechanisms, thereby improving performance in image processing tasks.

[0077] (3) Detail recovery module

[0078] According to the Retinex theoretical formula, the denoised low-light remote sensing image output by the denoising module and the enhanced illumination map output by the multi-dimensional feature fusion enhancement module are combined to generate a reflectance map reflecting image details. The enhanced illumination map and reflectance map are then concatenated channel-wise as input to this module. To effectively restore image details, a two-branch architecture is designed: a spatial detail restoration branch and a frequency detail restoration branch. These two branches work together to produce a high-quality enhanced image.

[0079] The spatial detail restoration branch is primarily responsible for removing noise and restoring image details in the spatial domain. This branch consists of three stacked 3×3 convolutional layers, each of which is followed by a nonlinear process using the LeakyReLU activation function. Each convolutional layer uses a 3×3 convolution kernel. This small kernel size allows for the extraction of detailed image features in local areas while reducing computational effort, resulting in better image detail restoration.

[0080] The frequency detail recovery branch is primarily responsible for extracting detail-related features in the frequency domain. Its structure is essentially the same as the frequency information extraction branch in the multi-dimensional feature fusion enhancement module, but its primary task is to perform convolution on the Fourier transformed phase component, while leaving the amplitude component unchanged. This design ensures that the frequency distribution of the image is not altered, thus preventing the introduction of new noise.

[0081] The spatial detail recovery branch and the frequency detail recovery branch extract detailed features in the spatial and frequency domains, respectively. These features are then fused in a subsequent fusion module to generate a more comprehensive image feature representation. This dual-branch structure not only removes noise but also retains rich detail information, achieving high-quality image enhancement.

[0082] Step 3: Loss Function

[0083] The main problems of low-light remote sensing images are low brightness and poor visibility. It is crucial to improve the overall brightness of low-light remote sensing images to match the normal level. Specifically, we calculate a brightness coefficient related to the original low-light image to help adjust the overall brightness level of the enhanced illumination map. The overall adjustment loss L over The following formula:

[0084]

[0085] The brightness coefficient Y H Indicates the average brightness of the enhanced image, set to 0.5, YL Indicates the average brightness of the low-light image, L enhance Represents the enhanced light map.

[0086] However, low-light remote sensing images usually have areas with different brightness, each of which requires different degrees of enhancement. Uniformly enhancing all pixels may lead to underexposure or overexposure. Therefore, based on the above overall adjustment loss, a pixel-by-pixel enhancement loss L is designed. pix As follows:

[0087]

[0088] Among them S denoise is the denoised image output by the denoising module after the low-light image passes through, β = α -1 0.7 -α is a scaling factor that controls the degree of scaling and determines the level of contrast enhancement. Both α and β depend on S denoise The brightness level of the low-light image can be adaptively enhanced at the pixel level through the above loss function, and the brightness and contrast of the low-light image can be further improved according to its brightness and pixel intensity.

[0089] The effects of the present invention can be further illustrated by the following experiments.

[0090] 1. Experimental Conditions

[0091] The model training and testing in this invention are completed on a host equipped with an Intel(R) Xeon(R) Silver 4214R CPU and an NVIDIA GeForce RTX 3090 GPU with 16GB RAM. The system environment is Windows 10, and the algorithm is implemented in Python.

[0092] The training and test data used in the experiment were both from the publicly available iSAID-dark dataset. The iSAID-dark dataset was constructed based on the test set of the iSAID dataset. The iSAID dataset covers a variety of scenes, including rural and urban areas, and contains a large amount of high-resolution aerial imagery. The iSAID-dark dataset selected 751 images from the iSAID test set as baseline data. These images were randomly manipulated to generate images of size 512×512, which were ultimately divided into 3755 training images and 66 test images. These enhanced-light remote sensing images were converted to low-light images using an inverse curve adjustment technique, thus constructing a paired low-light / enhanced-light remote sensing image dataset.

[0093] 2. Experimental content

[0094] According to the steps given in the specific implementation method, the low-light remote sensing image enhancement model is trained on the training set and the detection effect is evaluated on the test set.

[0095] Table 1 shows the quantitative results of low-light remote sensing image enhancement in the present invention.

[0096]

[0097] In order to verify the effectiveness of the present invention, the model is compared with six previously effective no-reference enhancement algorithms, including RetinexDIP, ZSPL-PSR, RRDNet, SGZ, QuadPrior and Zero-IG. Among them, the design of RetinexDIP, RRDNet and Zero-IG is based on Retinex theory, ZSPL-PSR is a method of learning to enhance images according to curve parameter adjustment, SGZ is based on the guidance of semantic information, and QuadPrior combines the mainstream generative diffusion model for enhancement. Figure 2 This is a schematic diagram of the low-light remote sensing image enhancement results, showing the enhancement results of two images. The left side is the low-light original image in the remote sensing scene, and the right side is the enhanced result. It can be seen that for remote sensing images taken under low light, the present invention can obtain relatively satisfactory enhancement results. In order to more clearly demonstrate the advantages of the present invention, the results of the present invention are quantitatively described in Table 1. The bold font represents the best results, where the Peak Signal-to-Noise Ratio (PSNR) is used to measure the ratio of image signal to noise, the Structural Similarity Index (SSIM) is used to measure the structural similarity of two images, and the Learned Perceptual Image PatchSimilarity (LPIPS) is used to measure the perceptual difference between the two images. It can be seen that the scores of the present invention on the PSNR and SSIM indicators are 18.561 and 0.662 respectively, which are better than other methods. The score on the LPIPS indicator is 0.486, which is only lower than the QuadPrior method, but far better than other methods. Based on the above qualitative and quantitative descriptions, we find that the present invention achieves significant performance improvements compared to other representative algorithms. Therefore, the present invention can be well applied to various low-light remote sensing image enhancement tasks.

Claims

1. A reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion, characterized by Here are the steps: Perform semantic segmentation on low-light remote sensing images to obtain semantic feature map C; Denoising the low-light remote sensing image to obtain a denoised low-light remote sensing image; The spatial information feature map A is obtained by parallel processing of the spatial information extraction branch on the denoised low-light remote sensing image; and the frequency information feature map B obtained by processing the frequency information extraction branch; The spatial information image feature map A and the semantic feature map C are processed by the input semantic information extraction branch to extract the related semantic information feature map D; The semantic information feature map D, spatial information feature map A and frequency information feature map B are convolved together to output the enhanced illumination image F; The denoised low-light remote sensing image and the enhanced light image F are processed according to the Retinex theory to obtain the reflectance map E; The enhanced illumination image F and reflectance map E are spliced in the channel direction, and then the frequency detail recovery branch is performed to obtain the frequency detail recovery branch feature map G. At the same time, the spatial detail recovery branch is processed in parallel to obtain the spatial detail recovery branch feature map H. The frequency detail recovery branch feature map G and the spatial detail recovery branch feature map H are simultaneously subjected to convolution processing to obtain an enhanced image of the low-light remote sensing image.

2. The method for enhancing low-light remote sensing images without reference based on multi-branch feature fusion according to claim 1, characterized in that: The denoising process and the spatial detail restoration branch process both adopt multi-level stacked convolution sequential processing.

3. The method for enhancing low-light remote sensing images without reference based on multi-branch feature fusion according to claim 1, characterized in that: The spatial information extraction branch processing adopts multi-level convolution sequential processing, and the third level convolution is convolved with the result of the second level convolution and the result of the first level convolution, and the result of the second level convolution is convolved with the result of the third level convolution at the same time.

4. The method for enhancing low-light remote sensing images without reference based on multi-branch feature fusion according to claim 1, characterized in that: The semantic information extraction branch processing flow includes convolution and flattening processing; the semantic feature map C processing flow is through a first-level convolution and flattening process, and then through an activation function and two-level convolution processing; the spatial information image feature map A undergoes a first-level convolution and flattening process, and the processed information is added to the processing flow of the semantic feature map C before and after the activation function. The semantic feature map C is finally convolved with the spatial information image feature map A to output the semantic information feature map D.

5. The method for enhancing low-light remote sensing images without reference based on multi-branch feature fusion according to claim 1, characterized in that: The frequency information extraction branch processing flow includes dilated convolution, convolution, Fourier transform and inverse Fourier transform; the amplitude information of the denoised low-light remote sensing image after dilated convolution, convolution and Fourier transform is sequentially subjected to two-stage convolution processing, followed by inverse Fourier transform and then connected to the two-stage convolution processing to output the frequency information feature map B; the processing result of the dilated convolution is simultaneously input into the last stage of convolution processing; the phase information of the Fourier transform is input into the inverse Fourier transform.

6. The method for enhancing low-light remote sensing images without reference based on multi-branch feature fusion according to claim 1, characterized in that: The frequency detail recovery branch processing flow includes dilated convolution, convolution, Fourier transform, and inverse Fourier transform. The enhanced illumination image F and reflectance map E are sequentially subjected to dilated convolution, convolution, and Fourier transform of the image feature map spliced in the channel direction. The phase information after the convolution is subjected to two-stage convolution processing, followed by inverse Fourier transform. The two-stage convolution processing is then connected to output the frequency detail recovery branch feature map G. The processing result of the dilated convolution is simultaneously input into the last stage of convolution processing; the amplitude information of the Fourier transform is input into the inverse Fourier transform.

7. A neural network model system for implementing the reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion as described in any one of claims 1 to 6, characterized in that: It includes a denoising module, a semantic segmentation model, a multi-dimensional feature fusion enhancement module and a detail recovery module; the multi-dimensional feature fusion enhancement module includes a semantic information extraction branch, a spatial information extraction branch and a frequency information extraction branch module; the detail recovery module includes a spatial detail recovery branch and a frequency recovery branch module and a Retinex theory processing module; The denoising module and semantic segmentation model are the input ends of the low-light remote sensing image respectively. The denoising module outputs the denoised low-light remote sensing image, which is then input into the Retinex theory processing module, the spatial information extraction branch, and the frequency information extraction branch module respectively. The output of the spatial information extraction branch and the output of the semantic segmentation model are connected to the semantic information extraction branch; The outputs of the spatial information extraction branch, the semantic information extraction branch, and the frequency information extraction branch modules are connected to a convolution module to output an enhanced illumination map. The enhanced illumination map is input into the Retinex theoretical processing module. The reflectance map output by the Retinex theoretical processing module and the enhanced illumination map are spliced in the channel direction and input into the spatial detail recovery branch and the frequency recovery branch modules respectively. The outputs of the two modules are passed through a convolution module to output the enhanced image.

8. The neural network model system according to claim 7, characterized in that: The training process of the neural network model system is as follows: Low-light remote sensing images are used as training sample images; Use the iSAID-dark dataset to train the semantic segmentation model WSCL, and input the trained model into the neural network model system as a low-light remote sensing image; The activation function in the neural network model system is the ReLU activation function; The overall loss function in the neural network model system is: The brightness coefficient Y H Indicates the average brightness of the enhanced image, set to 0.5, Y L Indicates the average brightness of the low-light image, L enhance represents the enhanced illumination map; The pixel-by-pixel enhancement loss in the neural network model system is Among them S denoise is the denoised image output by the denoising module after the low-light image passes through, β = α -1 0.7 -α is a scaling factor that controls the degree of scaling and determines the level of contrast enhancement. Both α and β depend on S denoise brightness level.

9. A reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion according to any one of claims 1 to 6, and a method for using a neural network model system trained according to claim 7 or 8, characterized in that: The low-light remote sensing image is used as the input image of the neural network model system, and an enhanced remote sensing image is obtained at the output.

10. A computer program product, characterized in that The method comprises computer executable instructions, which are used to implement the reference-free low-light remote sensing image enhancement method based on multi-branch feature fusion as described in claims 1 to 7 when executed.

Images