Low-light remote sensing image enhancement method, device, equipment and storage medium based on Retinex

Abstract

The present application discloses a low-light remote sensing image enhancement method based on Retinex. The method decomposes the low-light remote sensing image to be enhanced into a reflection map and an illumination map based on the Retinex theory, and then extracts features from the illumination map and the reflection map respectively to obtain a reflection feature map and an illumination feature map. The reflection feature map and the illumination feature map are then fused and reconstructed to obtain a reconstructed illumination map and a reconstructed reflection map. A neural network module is then designed to enhance and fuse the reconstructed illumination map and the reconstructed reflection map, and the enhanced reflection map and illumination map are fused into a normal light image. A local reconstruction neural network module is then designed to perform local fine-tuning to obtain the final enhanced image. This not only improves the interpretability of the algorithm and the image enhancement effect, but also solves the problems of poor universality and low robustness of the current low-light remote sensing image enhancement methods, as well as noise, blur, insufficient quality, color cast, and other problems in the enhanced images.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a Retinex-based low-light remote sensing image enhancement method, device, computer equipment, and computer-readable storage medium. Background Art

[0002] Remote sensing image enhancement technology can not only improve the quality of images and enhance specific information, but also improve the usability of images in terms of vision and analysis. Therefore, it is of great significance in remote sensing image analysis and interpretation. Directly acquired remote sensing images are easily affected by factors such as the environment, weather, and sensors, resulting in problems such as insufficient brightness and excessive noise. Therefore, they cannot be directly used for traditional remote sensing tasks such as object detection and classification. It is necessary to use the corresponding low-light remote sensing image enhancement method for enhancement before it can be applied. At present, low-light image enhancement methods include traditional low-light image enhancement methods and low-light image enhancement methods based on deep learning. Traditional low-light image enhancement methods require manual design of features or optimization functions to enhance low-light images, which is difficult to cope with different scenes, and the enhancement results often have problems such as noise, blur, insufficient quality, and color cast. Low-light image enhancement methods based on deep learning are constantly developing. Although they can solve the above problems and achieve more robust results in restoring color details, removing noise, and improving image quality, end-to-end image enhancement methods directly regress the enhanced image, lack theoretical support, and have poor interpretability and effect. Therefore, how to improve the interpretability of the enhancement algorithm and the image enhancement effect has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a low-light remote sensing image enhancement method, device, computer equipment and storage medium based on Retinex to improve the enhancement effect of low-light remote sensing images.

[0004] In a first aspect, the present application provides a low-light remote sensing image enhancement method based on Retinex, the method comprising:

[0005] Based on the decomposition network, the low-light remote sensing image is decomposed into an initial reflection map and an initial illumination map;

[0006] Based on the feature extraction module in the enhancement network, feature extraction is performed on the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map;

[0007] Based on the feature fusion module in the enhancement network, the initial reflection feature map and the initial illumination feature map are subjected to feature fusion and information exchange to obtain an interacted illumination feature map and a reflection feature map;

[0008] Based on the image enhancement module in the enhancement network, the illumination feature map and the reflection feature map after interaction, the mapping curve of the regression illumination map adjustment, and the residual map of the regression reflection map adjustment, the initial illumination map and the initial reflection map are enhanced to obtain enhanced illumination map and reflection map;

[0009] Based on the local enhancement module in the enhancement network, the enhanced illumination map and the reflection map are fused, and the fused image is locally enhanced to obtain the target enhanced image.

[0010] Furthermore, decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map includes:

[0011] Determine a maximum channel value among the three-channel images of R, G, and B corresponding to the low-light remote sensing image as a fourth channel value of a fourth channel image of the low-light remote sensing image;

[0012] Extracting a feature map of a four-channel image corresponding to the low-light remote sensing image to obtain a remote sensing image feature map;

[0013] The channel images of the first three channels of the remote sensing image feature map are stripped off to serve as the initial reflection map, and the channel image of the fourth channel of the remote sensing image feature map is stripped off to serve as the initial illumination map.

[0014] Furthermore, the feature extraction module in the enhancement network extracts features from the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map, including:

[0015] Based on the Transformer and CNN networks, feature extraction is performed on the initial illumination map and the initial reflection map respectively to obtain an initial reflection feature map and an initial illumination feature map.

[0016] Furthermore, based on the Transformer and CNN networks, feature extraction is performed on the initial illumination map and the initial reflection map respectively to obtain an initial reflection feature map and an initial illumination feature map, including:

[0017] Based on the CNN network, feature extraction is performed on the initial reflection map respectively to obtain a preset number of initial reflection feature maps, and based on the preset number of initial reflection feature maps, the initial reflection feature map is generated.

[0018] Furthermore, the Transformer and CNN networks are used to extract features from the initial illumination map and the initial reflection map, respectively, to obtain an initial reflection feature map and an initial illumination feature map, including:

[0019] Based on the Transformer network, global features of the initial illumination map are extracted to obtain the initial illumination feature map.

[0020] Furthermore, before decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map based on the decomposition network, the method further includes:

[0021] Based on a preset training remote sensing image data set, the initial network is trained to generate the decomposition network, wherein the preset training remote sensing image data set includes at least one pair of low-light training images and normal-light images, the low-light training images are images to be enhanced, and the normal-light images are true values.

[0022] Furthermore, before decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map based on the decomposition network, the method further includes:

[0023] Determine whether the resolution of the low-light remote sensing image to be processed is higher than a preset threshold;

[0024] When the resolution of the low-light remote sensing image to be processed is higher than the preset threshold, dividing the low-light remote sensing image into a preset number of image blocks;

[0025] The image blocks are acquired in sequence as the low-light remote sensing image, so as to enhance each image block and then fuse them.

[0026] In a second aspect, the present application also provides a low-light remote sensing image enhancement device based on Retinex, comprising:

[0027] An image decomposition module is used to decompose the low-light remote sensing image into an initial reflection map and an initial illumination map;

[0028] A feature extraction module, used to extract features from the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map;

[0029] A feature fusion module, used to perform feature fusion and information exchange on the initial reflection feature map and the initial illumination feature map to obtain an interacted illumination feature map and a reflection feature map;

[0030] An image enhancement module is used to enhance the initial illumination map and initial reflection map based on the illumination feature map and reflection feature map after interaction, the mapping curve adjusted by the regression illumination map, and the residual map adjusted by the regression reflection map to obtain enhanced illumination map and reflection map;

[0031] The local enhancement module is used to fuse the enhanced illumination map and the reflection map, and to locally enhance the fused image to obtain the target enhanced image.

[0032] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the Retinex-based low-light remote sensing image enhancement method as described above when executing the computer program.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the Retinex-based low-light remote sensing image enhancement method as described above.

[0034] The present application discloses a low-light remote sensing image enhancement method, device, computer equipment and storage medium based on Retinex. The method decomposes the low-light remote sensing image into an initial reflection map and an initial illumination map based on a decomposition network; based on a feature extraction module in the enhancement network, feature extraction is performed on the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map; based on a feature fusion module in the enhancement network, feature fusion and information interaction are performed on the initial reflection feature map and the initial illumination feature map to obtain an illumination feature map and a reflection feature map after interaction; based on the image enhancement module in the enhancement network, the illumination feature map and the reflection feature map after interaction, the mapping curve of the adjusted illumination map, and the residual map of the adjusted reflection map, the initial illumination map and the initial reflection map are enhanced to obtain an enhanced illumination map and a reflection map; based on a local enhancement module in the enhancement network, the enhanced illumination map and the reflection map are fused, and the fused image is locally enhanced to obtain the target enhanced image. Through the above method, not only the interpretability of the algorithm and the image enhancement effect are improved, but also the problems of poor universality and low robustness of the current low-light remote sensing image enhancement methods, as well as noise, blur, insufficient quality, color cast and other problems in the enhanced images are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is a schematic flow chart of a low-light remote sensing image enhancement method based on Retinex provided in the first embodiment of the present application;

[0037] Figure 2A network flow diagram of a low-light remote sensing image enhancement method based on Retinex provided in an embodiment of the present application;

[0038] Figure 3 This is a test data set example in the embodiment of this application;

[0039] Figure 4 A schematic diagram of the inference process in an embodiment of the present application;

[0040] Figure 5 It is a schematic diagram of the inference results in the embodiments of the present application;

[0041] Figure 6 This is a schematic diagram of a feature fusion network module in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of the loss function calculation process in an embodiment of the present application;

[0043] Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0046] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0048] The embodiments of the present application provide a method, device, computer equipment and storage medium for low-light remote sensing image enhancement based on Retinex. The method for low-light remote sensing image enhancement based on Retinex can be applied to a server or a terminal. The server can be an independent server or a server cluster.

[0049] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0050] See also Figure 1 , Figure 1 It is a schematic flow chart of a low-light remote sensing image enhancement method based on Retinex provided in the first embodiment of the present application.

[0051] like Figure 1 As shown, the Retinex-based low-light remote sensing image enhancement method specifically includes steps S10 to S40.

[0052] Step S10: decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map based on a decomposition network;

[0053] Step S20: Based on the feature extraction module in the enhancement network, feature extraction is performed on the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map;

[0054] Step S30: Based on the feature fusion module in the enhancement network, the initial reflection feature map and the initial illumination feature map are subjected to feature fusion and information exchange to obtain an interacted illumination feature map and a reflection feature map;

[0055] Step S40, based on the image enhancement module in the enhancement network, the illumination feature map and the reflection feature map after interaction, the mapping curve of the regression illumination map adjustment, and the residual map of the regression reflection map adjustment, the initial illumination map and the initial reflection map are enhanced to obtain enhanced illumination map and reflection map;

[0056] Step S50: Based on the local enhancement module in the enhancement network, the enhanced illumination map and the reflection map are fused, and the fused image is locally enhanced to obtain the target enhanced image.

[0057] Traditional low-light image enhancement technologies include histogram equalization, grayscale mapping, methods based on Retinex theory, and methods based on atmospheric scattering theory. Traditional enhancement technologies require manual design of features or optimization functions to enhance low-light images, which is difficult to cope with different scenes, and the enhancement results often have problems such as noise, blur, poor quality, and color cast.

[0058] Low-light image enhancement technology based on deep learning continues to develop, achieving more robust results in restoring color details, removing noise, and improving image quality. End-to-end image enhancement methods directly regress the enhanced image, lack theoretical support, and have poor interpretability and effect.

[0059] In order to solve the problems existing in traditional low-light image enhancement technology and the problems existing in low-light image enhancement technology based on deep learning, a low-light remote sensing image enhancement method based on Retinex is provided, that is, a deep learning image enhancement method based on Retinex theory.

[0060] Specifically, Figure 2 As shown, this patent includes a decomposition network, an enhancement network and a feature fusion module. The enhancement method decomposes the image into a reflection map and an illumination map based on the Retinex theory through the decomposition network, wherein the reflection map is a characteristic of the object itself and does not change with changes in illumination. Then, the enhancement network uses the Transformer architecture and the CNN architecture to extract features from the initial illumination map and the initial reflection map respectively. Then, the feature fusion module is used to fuse the information and then enhance it. The enhanced reflection map and illumination map are fused to generate a normal light image, and then local reconstruction is used to fine-tune it to obtain the final target enhanced image.

[0061] The deep learning network in this embodiment is divided into two parts: training and inference. The input images of the training process are pairs of low-light images and normal-light images, and the inference process only inputs low-light images.

[0062] First, you need to train the decomposition network and the enhancement network:

[0063] Before training, data preparation is required. Figure 3 As shown in the figure, two low-light remote sensing image datasets are prepared. First, real low-light satellite remote sensing images are collected and manually adjusted to normal light images of digital product level. Then they are cropped to obtain 512×512 image pairs. Normal light images are used as true values, and low light images are used as images to be enhanced to train the network. A total of 5934 sets of low-light-normal light matching pairs are used, and the dataset is named RSIE.

[0064] The UCMerced remote sensing image classification dataset is selected and adjusted to a dark light image. The combination of linear transformation and gamma transformation can achieve good results in low light image generation. The dataset contains 21 scenes such as airplanes, buildings, and forests, with 100 images for each scene. Each image is simulated using the linear transformation-gamma transformation-linear transformation method, and the image size is 256×256.

[0065] First, train the decomposition network. The training steps are as follows:

[0066] The data input is a low-light-normal light image pair, and the network requires an image input size of 256×256. First, perform data enhancement. When the image pair resolution is greater than 256, randomly crop it, and randomly rotate and flip the cropped image pair to obtain more diverse data. Set the learning rate, loss function, optimization function, training epoch, etc., and input the image pair into the network for training. The input for loss function calculation includes the initial low-light-normal light image pair, the decomposed normal light reflection map and illumination map, and the low-light reflection map and illumination map. Finally, a trained decomposition network is obtained.

[0067] Then the enhanced network is trained, and the training steps are as follows:

[0068] The data input is a low-light reflection image and an illumination image. The network requires an image input size of 256×256. Set the learning rate, loss function, optimization function, training epoch, etc., and input the image pair into the network for training. The loss function calculation input includes the enhanced reflection image, illumination image, and fused enhanced image, normal light image, and normal light reflection image and illumination image obtained by the decomposition network. Finally, a trained enhanced network is obtained.

[0069] Before using the above network for enhanced inference, the above network needs to be tested. The evaluation indicators for measuring the quality of image enhancement mainly include subjective evaluation indicators and objective evaluation indicators. Subjective evaluation indicators are usually based on the subjective feelings of the human eye, and evaluate the color, clarity, saturation, contrast, details, etc. of the image. The evaluation results will vary due to the subjective preferences and individual differences of the observer. The objective evaluation indicators quantify the image quality by calculating the difference or similarity between the output image of the algorithm and the reference image. The objective evaluation indicators used in this embodiment include peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Both can reflect the similarity between the output image and the reference image and the quality of the output image to a certain extent.

[0070] PSNR is used to compare the similarity between two images. The larger the value, the more similar the images are and the better the image quality. The PSNR calculation formula is as follows:

[0071]

[0072] Max I Indicates the maximum pixel value in the image, and its value range is [0,255]. MSE represents the mean square error between the algorithm output image and the reference image, and the calculation formula is as follows:

[0073]

[0074] Where m and n represent the width and height of the image respectively, I(i,j) represents the pixel value of the algorithm output image, and R(i,j) represents the pixel value of the reference image.

[0075] SSIM not only considers the difference between pixel values, but also considers information such as brightness, contrast, and structure. Its value range is [0,1]. The larger the value, the more similar the two images are, and the higher the image quality. The calculation of SSIM usually selects a sliding window of a specific size, slides it continuously on the image, and finally calculates the average value as the final SSIM value. The SSIM calculation formula is as follows:

[0076]

[0077] Where x and y represent two images, μ x and μ y Represent the average value of the pixel values ​​of the two images, and Represents the variance of the pixel values ​​of the two images, σ xy Represents the covariance of the pixel values ​​of the two images. c1 and c2 are two constants.

[0078] After obtaining good results for the above two indicators, inference operations can be performed. If the results are poor, it is necessary to adjust the hyperparameters to fine-tune the network and retrain until a better result is obtained.

[0079] After obtaining the training parameters of the decomposition network and the enhancement network, they can be used to enhance the inference of low-light remote sensing images. The inference flow chart is as follows: Figure 4 As shown, the inference results are Figure 5 shown.

[0080] The inference steps are as follows: input a low-light remote sensing image. If the input image resolution is higher than 256, the image is divided into 256 image blocks, which are enhanced separately and then fused. The decomposition network decomposes the image into a reflection map and an illumination map according to the Retinex theory, where the reflection map is a characteristic of the object itself and does not change with changes in illumination. The enhancement network uses the Transformer architecture and the CNN architecture to extract features from the illumination map and the reflection map, respectively, and uses the feature fusion module proposed in the present invention to fuse information and then enhance it. The enhanced reflection map and illumination map are fused into a normal light image, which is then locally fine-tuned through local reconstruction to obtain the final enhanced image.

[0081] In one embodiment, decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map includes:

[0082] Determine a maximum channel value among the three-channel images of R, G, and B corresponding to the low-light remote sensing image as a fourth channel value of a fourth channel image of the low-light remote sensing image;

[0083] Extracting a feature map of a four-channel image corresponding to the low-light remote sensing image to obtain a remote sensing image feature map;

[0084] The channel images of the first three channels of the remote sensing image feature map are stripped off as the initial reflection map, and the channel image of the fourth channel of the remote sensing image feature map is stripped off as the initial illumination map, wherein the fourth channel is the channel corresponding to the maximum channel value in the remote sensing image feature map.

[0085] In this embodiment, according to the Retinex theory, a U-Net variant network is used to decompose the input low-light image and normal-light image into an initial reflection map and an initial illumination map. The decomposition process is as follows:

[0086] 1) Input the RGB three-channel image corresponding to the low-light remote sensing image to be enhanced into the network, and take the maximum channel value of the three channels of R, G, and B as the fourth channel value of the image.

[0087] This step increases the channels of the image so that more effective information can be obtained in the subsequent feature extraction;

[0088] 2) Using a 9×9 convolution kernel, the number of channels of the image is changed from 4 to 64, and the convolved image is used as the initial feature map G1.

[0089] In this step, more local features can be extracted by using large convolution kernels, and these features are more representative.

[0090] 3) The feature map G1 is processed using a three-layer U-Former structure that combines U-Net and Transformer. After three downsamplings and three upsamplings, the processed feature map G2 is obtained.

[0091] The feature C2 in this step is consistent with the feature map C1 in terms of the number of channels, and integrates the features of downsampling while upsampling, which can effectively solve the problem of feature loss;

[0092] 4) Input feature map C2 into a 3×3 convolution kernel of size , thereby restoring the number of channels of feature map C2 from 64 to 4, and finally constrain all values ​​in feature map C2 to between [0,1] through the Sigmoid function to obtain feature map C3;

[0093] 5) The images corresponding to the first three channels of the feature map G3 are separated as a reflection map, and the image corresponding to the fourth channel is separated separately as an illumination map. The fourth channel is the channel corresponding to the maximum channel value in the remote sensing image feature map.

[0094] In this embodiment, since the illumination map has better global information, the present invention uses Vision Transformer to decompose the illumination map L output by the network. dark Reconstruct. Assuming L dark The size is 256×256×1, and L dark The image is divided into 256 non-overlapping view blocks, each of which is 16×16×1. After convolution, each view block extracts a feature vector of size 1×192. After convolution of 256 view blocks, a feature map G of size 256×192 is obtained. L , as the initial illumination feature map.

[0095] Assume that the input of the Transformer module is Then the output of the Tth Transformer module is The calculation process is as follows:

[0096]

[0097]

[0098]

[0099] Among them, E pos Represents position encoding, MSA, FFN, and LN represent multi-head self-attention mechanism, feedforward network layer, and normalization layer respectively. After T Transformer modules, the illumination feature map is obtained.

[0100] In this embodiment, the present invention uses a CNN network (specifically, a U-Net network) to extract features from the reflectance map. dark , assuming its size is 256×256×3, through the four downsampling layers of the U-Net network from R dark Extract the feature map G of size 16×16×1024 R , and retain the reflection feature map formed after each downsampling, that is:

[0101]

[0102] In this embodiment, feature fusion is to integrate features from different sources but related to each other, so as to obtain features with better expressive ability and improve the performance and robustness of the model. The present invention proposes a bidirectional feature fusion module based on the attention mechanism, such as Figure 6 shown.

[0103] The inference steps are as follows: input a low-light remote sensing image. If the input image resolution is higher than 256, the image is divided into 256 image blocks, which are enhanced separately and then fused. The decomposition network decomposes the image into a reflection map and an illumination map according to the Retinex theory, where the reflection map is a characteristic of the object itself and does not change with changes in illumination. The enhancement network uses the Transformer architecture and the CNN architecture to extract features from the illumination map and the reflection map, respectively, and uses the feature fusion module proposed in the present invention to fuse information and then enhance it. The enhanced reflection map and illumination map are fused into a normal light image, which is then locally fine-tuned through local reconstruction to obtain the final enhanced image.

[0104] Specifically, for the feature map of the illumination map The feature map G with the reflection map R , first transform the feature map Obtained by linear projection and Three matrices are used as the initial illumination map; G is transformed into R Projected to a feature map of size 16×16×192 As the initial reflection map, feature fusion is performed according to the following formulas:

[0105]

[0106]

[0107] For the reflection map output from the above steps Through a 1×1 convolution kernel Project back to a feature map of size 16×16×1024, and then compare it with the previously saved downsampled feature map Input them together into the upsampling network part of U-Net to obtain a feature map of size 256×256×3 Finally, the R output by the decomposition network dark Perform residual linking to obtain the reconstructed reflection map

[0108] For the light map output from the above steps The reshape operation is used to expand the vector to a size of 1×(256*192). Then the global color regression curve is generated by nn.Li near and ReLu function. Finally, the illumination map L output by the decomposition network is analyzed by the global color regression curve. dark Mapping is performed to obtain the reconstructed illumination map

[0109] In this embodiment, the illumination map output in the above steps is (Size is 256×256×1) and reflection map (size is 256×256×3) and perform element-by-element multiplication to obtain the enhanced image. (Size is 256×256×3). The final local reconstruction module is composed of U-Net. Upsampling and downsampling are performed again, and the regression residual graph Finally, the two are added together to obtain the enhanced image I enhance The running formula is as follows:

[0110]

[0111]

[0112] The loss function of the decomposition network is mainly composed of reconstruction loss, reflection loss Figure 1 It is composed of consistency loss and illumination smoothness loss, and the formula is as follows:

[0113]

[0114] Where k1 and k2 are balance coefficients, describing the importance of the loss in the overall loss. Usually k1=0.3 and k2=0.2 are set. Represents the reconstruction loss, that is, the fusion of the decomposed reflection map and illumination map should be consistent with the input image. Here, both low-light images and high-light images are taken into account. The formula is as follows

[0115]

[0116] Where ‖·‖1 represents the L1 norm, also known as the Manhattan norm, which refers to the sum of the absolute values ​​of each element in the vector. Represents reflection consistency loss. In Retinex theory, the reflection map represents the properties of the object itself, so the reflection maps decomposed from images taken at the same location but under different lighting conditions should be exactly the same. The formula is as follows:

[0117]

[0118] Represents the illumination smoothness loss. The illumination map of an image is continuous in smooth areas and discontinuous in areas with steep gradients. The calculation of illumination smoothness loss is usually based on the gradient information of pixels in the image. In a grayscale image, the gradient of each pixel represents the rate of change of the brightness of the pixel. By calculating the average value of the gradient of each pixel, the illumination smoothness of the image can be obtained. The higher the smoothness, the smaller the illumination change between pixels and the smoother the image. The formula is as follows

[0119]

[0120] Among them, |·| represents the absolute value operation, Represents the gradient calculation operation of the image. In areas where the reflection map gradient is large, the penalty for the illumination map is small; in areas where the reflection map is smooth, the penalty for the illumination map is large.

[0121] The enhancement loss function is mainly composed of the consistency loss of the reflection map, the L1 loss of the enhanced image, the perception loss, and the illumination smoothness loss. The formula is as follows:

[0122]

[0123] Among them, λ1, λ2, and λ3 are balance coefficients, which describe the importance of the loss in the overall loss. Usually, the values ​​of all coefficients are set to 0.1. Represents the L1 loss of the enhanced image and the original normal light image, and the formula is as follows:

[0124]

[0125] It is the reflection consistency loss, that is, the reflection map after dark light image enhancement should be consistent with the reflection map obtained by decomposing the normal light image. The calculation formula is as follows:

[0126]

[0127] It is the illumination consistency loss, that is, the illumination map after dark light image enhancement should be consistent with the illumination map obtained by decomposing normal light image. The calculation formula is as follows:

[0128]

[0129] The illumination smoothing loss is calculated by only using the illumination map and reflection map after dark light image enhancement. The calculation formula is as follows:

[0130]

[0131] The loss function calculation process is as follows Figure 7 shown.

[0132] In this embodiment, a low-light remote sensing image enhancement method based on Retinex is proposed, which combines the network design with the Transformer architecture and the convolutional neural network (CNN) architecture. First, the low-light image is decomposed into a reflection map and an illumination map according to the Retinex theory. Considering the characteristics of remote sensing images, the reflection map information comes from local information such as the color, texture, and details of the object. The network design is carried out by residual map regression. The illumination map information basically comes from sunlight, which is global illumination and is less affected by local light sources. It is enhanced by estimating the global color change curve. This method has achieved good results in the task of low-light remote sensing image enhancement, and is also used in the task of low-light natural image data enhancement. It not only improves the interpretability of the algorithm and the image enhancement effect, but also solves the problems of poor universality and low robustness of the current low-light remote sensing image enhancement method, as well as noise, blur, insufficient quality, and color cast in the enhanced image.

[0133] An embodiment of the present application provides a schematic block diagram of a low-light remote sensing image enhancement device based on Retinex.

[0134] The low-light remote sensing image enhancement device based on Retinex includes:

[0135] An image decomposition module is used to decompose the low-light remote sensing image into an initial reflection map and an initial illumination map;

[0136] A feature extraction module, used to extract features from the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map;

[0137] A feature fusion module, used to perform feature fusion and information exchange on the initial reflection feature map and the initial illumination feature map to obtain an interacted illumination feature map and a reflection feature map;

[0138] An image enhancement module is used to enhance the initial illumination map and initial reflection map based on the illumination feature map and reflection feature map after interaction, the mapping curve adjusted by the regression illumination map, and the residual map adjusted by the regression reflection map to obtain enhanced illumination map and reflection map;

[0139] The local enhancement module is used to fuse the enhanced illumination map and the reflection map, and to locally enhance the fused image to obtain the target enhanced image.

[0140] Furthermore, the image decomposition module includes:

[0141] A channel adding unit, used to determine the maximum channel value among the three-channel images of R, G, and B corresponding to the low-light remote sensing image, as the fourth channel value of the fourth channel image of the low-light remote sensing image;

[0142] A channel extraction unit, used to extract a feature map of a four-channel image corresponding to the low-light remote sensing image to obtain a remote sensing image feature map;

[0143] An image extraction unit is used to strip off the channel images of the first three channels of the remote sensing image feature map as the initial reflection map, and to strip off the channel image of the fourth channel of the remote sensing image feature map as the initial illumination map, wherein the fourth channel is the channel corresponding to the maximum channel value in the remote sensing image feature map.

[0144] Furthermore, the feature extraction module is also used for:

[0145] Based on the Transformer and CNN networks, feature extraction is performed on the initial illumination map and the initial reflection map respectively to obtain an initial reflection feature map and an initial illumination feature map.

[0146] Furthermore, the feature extraction module is also used for:

[0147] Based on the CNN network, feature extraction is performed on the initial reflection map respectively to obtain a preset number of initial reflection feature maps, and based on the preset number of initial reflection feature maps, the initial reflection feature map is generated.

[0148] Furthermore, the feature extraction module is also used for:

[0149] Based on the Transformer network, global features of the initial illumination map are extracted to obtain the initial illumination feature map.

[0150] Furthermore, the low-light remote sensing image enhancement device also includes a network training module, which is used to:

[0151] Based on a preset training remote sensing image data set, the initial network is trained to generate the decomposition network, wherein the preset training remote sensing image data set includes at least one pair of low-light training images and normal-light images, the low-light training images are images to be enhanced, and the normal-light images are true values.

[0152] Furthermore, the low-light remote sensing image enhancement device further includes an image segmentation module, which is used to:

[0153] Determine whether the resolution of the low-light remote sensing image to be processed is higher than a preset threshold;

[0154] When the resolution of the low-light remote sensing image to be processed is higher than the preset threshold, dividing the low-light remote sensing image into a preset number of image blocks;

[0155] The image blocks are acquired in sequence as the low-light remote sensing image, so as to enhance each image block and then fuse them.

[0156] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0157] The above-mentioned device can be implemented in the form of a computer program. Figure 8 Runs on the computer device shown.

[0158] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0159] See also Figure 8 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0160] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the low-light remote sensing image enhancement methods based on Retinex.

[0161] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0162] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any low-light remote sensing image enhancement method based on Retinex.

[0163] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the Retinex-based low-light remote sensing image enhancement methods provided in the embodiments of the present application.

[0166] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A low-light remote sensing image enhancement method based on Retinex, characterized in that: The method comprises: Based on the decomposition network, the low-light remote sensing image is decomposed into an initial reflection map and an initial illumination map; Based on the feature extraction module in the enhancement network, feature extraction is performed on the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map; Based on the feature fusion module in the enhancement network, the initial reflection feature map and the initial illumination feature map are subjected to feature fusion and information exchange to obtain an interacted illumination feature map and a reflection feature map; Based on the image enhancement module in the enhancement network, the illumination feature map and the reflection feature map after interaction, the mapping curve of the regression illumination map adjustment, and the residual map of the regression reflection map adjustment, the initial illumination map and the initial reflection map are enhanced to obtain enhanced illumination map and reflection map; Based on the local enhancement module in the enhancement network, the enhanced illumination map and the reflection map are fused, and the fused image is locally enhanced to obtain a target enhanced image; The feature extraction module based on the enhancement network extracts features from the initial reflection map and the initial illumination map respectively, and obtains the initial reflection feature map and the initial illumination feature map, including: Based on the Transformer network, global feature extraction is performed on the initial illumination map to obtain the initial illumination feature map, wherein the calculation formula of the Transformer network is: G L0 =G L +E pos in, is the input of the Transformer module, is the Tth output of the Transformer module, E pos represents position encoding, MSA, FFN, and LN represent multi-head self-attention mechanism, feedforward network layer, and normalization layer respectively; Using a CNN network, extracting features from the initial reflection map to obtain the initial reflection feature map; Wherein, based on the image enhancement module in the enhancement network, the illumination feature map and the reflection feature map after interaction, the mapping curve of the adjusted illumination map, and the residual map of the adjusted reflection map, the initial illumination map and the initial reflection map are enhanced to obtain the enhanced illumination map and reflection map, which includes: Generate a global color regression curve through nn.Linear and ReLu functions, and map the initial illumination map through the global color regression curve based on the illumination feature map after interaction to obtain a reconstructed illumination map as the enhanced illumination map; Based on the residual map and the interactive reflection feature map, the initial reflection map is enhanced to obtain an enhanced reflection map.

2. The low-light remote sensing image enhancement method based on Retinex according to claim 1, characterized in that: Decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map includes: Determine a maximum channel value among the three-channel images of R, G, and B corresponding to the low-light remote sensing image as a fourth channel value of a fourth channel image of the low-light remote sensing image; Extracting a feature map of a four-channel image corresponding to the low-light remote sensing image to obtain a remote sensing image feature map; The channel images of the first three channels of the remote sensing image feature map are stripped off to serve as the initial reflection map, and the channel image of the fourth channel of the remote sensing image feature map is stripped off to serve as the initial illumination map.

3. The low-light remote sensing image enhancement method based on Retinex according to claim 1, characterized in that: The feature fusion module is a bidirectional feature fusion module of the attention mechanism. The feature fusion module based on the enhanced network performs feature fusion and information interaction on the initial reflection feature map and the initial illumination feature map to obtain an interactive illumination feature map and a reflection feature map, including: Based on the bidirectional feature fusion module of the attention mechanism, the initial reflection feature map and the initial illumination feature map are feature fused, and the fused feature map is enhanced to obtain the interactive illumination feature map and reflection feature map, wherein the fusion formula of the cross attention mechanism is: is the fused illumination feature map, is the fused reflection feature map, and The three matrices obtained by linear projection of the initial illumination feature map, is the initial reflection feature map.

4. The low-light remote sensing image enhancement method based on Retinex according to claim 1, characterized in that: The method of extracting features from the initial illumination map and the initial reflection map based on the Transformer and CNN networks to obtain an initial reflection feature map and an initial illumination feature map includes: Based on the CNN network, feature extraction is performed on the initial reflection map respectively to obtain a preset number of initial reflection feature maps, and based on the preset number of initial reflection feature maps, the initial reflection feature map is generated.

5. The low-light remote sensing image enhancement method based on Retinex according to claim 1, characterized in that: The local enhancement module is composed of U-Net. The local enhancement module based on the enhancement network fuses the enhanced illumination map and the reflection map, and locally enhances the fused image to obtain the target enhanced image, which includes: Perform element-by-element multiplication on the enhanced illumination map and the reflection map to obtain an enhanced image map; Based on the first calculation formula, the enhanced image map is upsampled and downsampled to obtain a regression residual map, and based on the second calculation formula and the regression residual map, the enhanced image map is locally enhanced to obtain the target enhanced image; wherein: The first calculation formula is: The second calculation formula is: is the regression residual plot, is the enhanced image, I enhance Enhance the image for the target.

6. The low-light remote sensing image enhancement method based on Retinex according to any one of claims 1 to 5, characterized in that: Before decomposing the low-light remote sensing image into an initial reflection map and an initial illumination map based on the decomposition network, the method further includes: Based on a preset training remote sensing image data set, the initial network is trained to generate the decomposition network, wherein the preset training remote sensing image data set includes at least one pair of low-light training images and normal-light images, the low-light training images are images to be enhanced, and the normal-light images are true values.

7. A low-light remote sensing image enhancement device based on Retinex, characterized in that: include: An image decomposition module is used to decompose the low-light remote sensing image into an initial reflection map and an initial illumination map; A feature extraction module, used to extract features from the initial reflection map and the initial illumination map respectively to obtain an initial reflection feature map and an initial illumination feature map; A feature fusion module, used to perform feature fusion and information exchange on the initial reflection feature map and the initial illumination feature map to obtain an interacted illumination feature map and a reflection feature map; An image enhancement module is used to enhance the initial illumination map and initial reflection map based on the illumination feature map and reflection feature map after interaction, the mapping curve adjusted by the regression illumination map, and the residual map adjusted by the regression reflection map to obtain enhanced illumination map and reflection map; A local enhancement module is used to fuse the enhanced illumination map and the reflection map, and locally enhance the fused image to obtain a target enhanced image; Wherein, the feature extraction module is also used for: Based on the Transformer network, global feature extraction is performed on the initial illumination map to obtain the initial illumination feature map, wherein the calculation formula of the Transformer network is: in, is the input of the Transformer module, is the Tth output of the Transformer module, E pos represents position encoding, MSA, FFN, and LN represent multi-head self-attention mechanism, feedforward network layer, and normalization layer respectively; Using a CNN network, extracting features from the initial reflection map to obtain the initial reflection feature map; Wherein, the image enhancement module is also used for: Generate a global color regression curve through nn.Linear and ReLu functions, and map the initial illumination map through the global color regression curve based on the illumination feature map after interaction to obtain a reconstructed illumination map as the enhanced illumination map; Based on the residual map and the interactive reflection feature map, the initial reflection map is enhanced to obtain an enhanced reflection map.

8. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the Retinex-based low-light remote sensing image enhancement method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the low-light remote sensing image enhancement method based on Retinex as described in any one of claims 1 to 6.

Images