A rain and fog image enhancement method under low illumination based on TF-GAN

Abstract

This paper discloses a TF-GAN-based method for enhancing rain and fog images in low-light conditions. The method comprises the following steps: 1) reconstructing and processing an existing dataset; 2) generating structure estimates using an edge extractor; 3) training an image structure estimation network (ISE block); 4) training a Transformer-based generator; 5) training a Transformer-based discriminator; and 6) testing the Transformer-based generative adversarial network for low-light rain and fog image enhancement. This method can narrow the structural gap between low-light rain and fog images and normal-light images, thereby better enhancing low-light rain and fog images. It comprehensively addresses the problems of existing image enhancement technologies, such as their weak adaptability to complex environments such as climate and poor stability. Furthermore, the network model exhibits good generalization performance.

Description

TECHNICAL FIELD

[0001] The present application relates to edge extraction technology, image enhancement technology, Transformer and generative adversarial network (GAN) technology, in particular to a rain and fog image enhancement method under low illumination based on TF-GAN. BACKGROUND

[0002] Image enhancement technology has a very wide range of applications and plays an important role in intelligent transportation, unmanned driving, security monitoring and smart agriculture. With the continuous development of computer vision and image processing technology, image enhancement technology will be further improved to provide better image quality and visibility for various application scenarios.

[0003] Existing image enhancement methods in all-weather complex environments mainly include low-illumination image enhancement methods and image enhancement methods for rain removal, fog removal and deblurring. Low-illumination image enhancement methods aim to improve image quality under insufficient lighting conditions, including contrast enhancement, noise reduction, color correction and other processing. While rain and fog removal and other enhancement methods aim to eliminate the effects of rain and fog on images, including raindrop and rain streak removal, fog streak removal and other processing. However, in complex environments where low illumination and rain and fog coexist, existing technologies often cannot effectively cope with the situation. Low-illumination image enhancement methods often cannot handle image blurring caused by rain and fog particles, while rain and fog removal enhancement methods often cannot overcome the problem of low contrast and detail loss caused by insufficient lighting. Therefore, there is currently a lack of enhancement technology specifically for low-illumination rain and fog images. Inspired by the Retinex theory, we believe that the main reason why existing methods have poor enhancement effects on low-illumination rain and fog images is that they cannot model the structure of the image, so they cannot obtain the structural details of the image. Since images usually contain rich structures, it is of great significance to explore these structures to promote better structural detail enhancement. To solve this problem and obtain high-quality images with structural details, a structure estimation network ISE Block based on U-shaped transformer is proposed to estimate the structural information of the image. The structure estimation obtained by the ISE Block can better guide the generator to enhance the structural details and guide the discriminator to identify true and false feature information. In addition, it is noted that existing methods mainly rely on local invariance convolution to obtain local features. Since non-local regions contain a large amount of useful feature information, modeling local information has certain defects. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and provides a rain and fog image enhancement method under low illumination based on TF-GAN. This method can efficiently improve the image quality, and improve the visibility and application effect of the image under low illumination and rain and fog weather conditions.

[0005] The technical solution for achieving the object of the present application is:

[0006] A rain and fog image enhancement method under low illumination based on TF-GAN comprises the following steps:

[0007] 1) Reconstructing and processing the existing LOL dataset: the existing LOL dataset is reconstructed and processed to obtain a dataset LOL_RF under low-illumination rain and fog environment, and the process is:

[0008] 1-1) Based on the public LOL dataset, the imaging characteristics under the rain and fog environment are simulated by using the random trajectory generation method to obtain the dataset LOL_RF under the low-illumination rain and fog environment;

[0009] 1-2) The LOL_RF dataset comprises 500 low-illumination rain and fog images and corresponding normal-illumination images, wherein 485 are training sets, and 15 are test sets;

[0010] 2) Generation of structure estimation based on the edge extractor Edge Extractor: the low-illumination rain and fog image and the normal-illumination image are subjected to the edge extractor Edge Extractor to obtain the structure diagram of the low-illumination rain and fog image and the structure diagram of the normal-illumination image, and the process is:

[0011] 2-1) The low-illumination rain and fog image and the corresponding normal-illumination image are input into the edge extractor;

[0012] 2-2) The encoder-decoder composed of the ResBlock module is adopted to obtain the low-frequency feature;

[0013] 2-3) The original image is subtracted by the low-frequency feature to obtain the high-frequency feature information, i.e. the structure diagram of the low-illumination rain and fog image and the structure diagram of the normal-illumination image;

[0014] 3) Training of the image structure estimation network ISE Block: the structure diagram of the low-illumination rain and fog image is input into the image structure estimation network ISE Block to obtain the estimated structure of the normal-illumination image, and the specific process is:

[0015] 3-1) The structure diagram of the low-illumination rain and fog image is mapped into tokens through the Input Project;

[0016] 3-2) The tokens are subjected to feature extraction by an encoder module composed of a multi-convolution channel attention MCCA and a cross-gate feedforward neural network CGFN, and the obtained feature codes are encoded into a deeper embedding space;

[0017] 3-3) The multi-convolution channel attention MCCA is subjected to 1x1 convolution for cross-channel information fusion, and 3x3 convolution for local context information capture, to obtain Q, K, and V matrices, and then channel attention is calculated in the channel dimension, and the cross-gate feedforward neural network CGFN uses 1x1 convolution to expand the channel dimension of the features; then, 3x3 depth separable convolution is used on each branch to effectively extract multi-scale local information of the image; finally, GELU activation function and element-wise multiplication operation are used on the two parallel branches respectively; the extracted features are subjected to a decoder module composed of a multi-convolution channel attention MCCA and a cross-gate feedforward neural network CGFN, the features of the symmetric blocks in the encoder are fused by a skip connection, and the obtained features are decoded into a shallower embedding space;

[0018] 3-4) The tokens obtained in step 3-3) are mapped into normal illumination structure estimation by Output Project;

[0019] 4) Training of the Transformer-based generator: the low-illumination rain and fog image is input into the generator Generator to obtain a high-quality enhanced image, including:

[0020] 4-1) The low-illumination rain and fog image is converted into corresponding tokens by Input Project;

[0021] 4-2) The tokens of the corresponding image obtained in step 4-1) are subjected to feature extraction by an encoder module composed of a multi-convolution channel attention MCCA and a cross-gate feedforward neural network CGFN, and the obtained features are encoded into a deeper embedding space;

[0022] 4-3) The feature vectors are subjected to decoding by a decoder module composed of a multi-convolution channel attention MCCA and a cross-gate feedforward neural network CGFN, the features output from the corresponding encoder are fused before each decoder module, and the structural feature information obtained in the ISEBlock is used to guide the generation process of the generator G;

[0023] 4-4) The tokens obtained in step 4-3) are converted into enhanced images by Output Project, which are our target images;

[0024] 5) Training of the discriminator of the transformer: the obtained enhanced image and normal illumination structure estimation are input into the discriminator D (Discriminator), to distinguish the real / fake of the image, including:

[0025] 5-1) The obtained enhanced image and normal illumination structure estimation are mapped into tokens through Input Project, 5-2) The tokens of the corresponding image obtained in step 5-1) are subjected to feature extraction through a series of encoder modules;

[0026] 5-3) The feature vectors extracted in step 5-2) are subjected to decoding operation through a series of decoder modules, the features of the symmetric blocks in the encoder are fused using the skip connection, and the structure features obtained in the ISE Block are used to guide the discriminator D to better distinguish the true and false of the generated image;

[0027] 5-4) The category [cls] token is attached at the beginning of the tokens obtained in step 5-3), and they are sent into the detection head, and the prediction of the real / fake of the image is output;

[0028] 6) The low-illumination rain and fog images in the LOL_RF test set are sent into the enhancement network for enhancement processing, including:

[0029] 6-1) The low-illumination rain and fog images in the LOL_RF data set are sent into the enhancement network for enhancement;

[0030] 6-2) The enhanced results are visualized.

[0031] The technical scheme solves the problem of poor enhancement effect caused by insufficient light, dark imaging, serious detail loss, and blur caused by rain and mist particles in low-illumination rain and mist environment in image enhancement. First, the low-illumination rain and mist image and the corresponding normal-illumination image are sent into the pre-trained edge extractor network to generate a low-illumination rain and mist structure image and a normal-illumination structure image; second, the input embedding, encode, decode, feature information fusion and out embedding operations are adopted to obtain a normal-illumination structure estimation; third, the low-illumination rain and mist image is input into the generator for training, and three feature maps of different scales are extracted from the image through the encoder stage, and the features are further extracted and fused through the decoder stage to obtain an enhanced low-illumination rain and mist image; finally, the enhanced image and the normal-illumination structure estimation are sent into the discriminator together, a series of feature extraction operations are performed, and multi-scale feature fusion is performed, then the category [cls] token is attached to the beginning of the last obtained tokens, and they are sent into the detection head together to output the prediction of the image real / fake.

[0032] The technical scheme adopts multi-scale connection to fuse features of different levels to model non-local information, and is composed of a structure estimation network based on Transformer, a generator G based on Transformer and a discriminator D based on Transformer, wherein the structure estimation network obtains rich structure information to guide the generator and the discriminator, so as to effectively improve the visibility, contrast and clarity of the image under low-illumination and rain and mist weather conditions, and improve the quality of the image.

[0033] This method can efficiently improve the image quality, and improve the visibility and application effect of the image under low-illumination and rain and mist weather conditions. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 It is an Edge Extractor structure diagram in the embodiment;

[0035] Figure 2 It is a ResNet Block structure diagram in the embodiment;

[0036] Figure 3 It is a Transformer-GAN network structure diagram in the embodiment;

[0037] Figure 4 It is an Encoder Block structure diagram in the embodiment;

[0038] Figure 5 It is a Decoder Block structure diagram in the embodiment;

[0039] Figure 6 Muti-conv Channels Attention structure diagram in the embodiment;

[0040] Figure 7 CGFN structure diagram in the embodiment;

[0041] Figure 8 Skip Connection structure between the encoder and the decoder in the embodiment;

[0042] Figure 9 SGTT module structure in the embodiment. DETAILED DESCRIPTION

[0043] The application will be further described in conjunction with the accompanying drawings and embodiments, but is not limited to the application.

[0044] Embodiment:

[0045] A rain and fog image enhancement method under low illumination based on a TF-GAN, comprising the following steps:

[0046] 1) Reconstructing the existing data set, comprising:

[0047] 1-1) According to the idea of random trajectory generation proposed by Boracchi and Foi, a kernel is generated by adopting a sub-pixel interpolation method for a trajectory vector, each trajectory vector is a complex vector, corresponding to the discrete position of an object following a 2D random motion in a continuous domain, based on the LOL data set, the imaging characteristics in the rain and fog environment are simulated to obtain a data set LOL_RF in the low-illumination rain and fog environment;

[0048] 1-2) The LOL data set contains 500 low-illumination images and their corresponding normal-illumination images, by simulating the imaging characteristics in the rain and fog environment, a data set LOL_RF in the low-illumination rain and fog environment is obtained, of which 485 are used as a training set and 15 are used as a test set, as shown in Table 1, wherein the training set of low-illumination rain and fog images and normal-illumination images generates a low-illumination rain and fog structure diagram and a normal-illumination structure diagram, and trains a structure estimation network ISE Block; the training set of low-illumination rain and fog images and normal-illumination images trains a generator (Generator); the enhanced image generated by the generator is used to train a discriminator D (Discriminator) together with the normal-illumination structure estimation Figure 1

[0049] Table 1 Source of data set

[0050]

[0051] ​

[0052] 2) Generation of structure estimation based on Edge Extractor:

[0053] Edge Extractor is a U-shaped network composed of ResBlock, as shown in Figure 1 , first, the BSDS500 dataset is used to pre-train the edge extractor, wherein the BSDS500 dataset contains a training set (400 images) and a test set (100 images), a total of 500 images, second, the low-illumination rain and fog images and normal-illumination images in the LOL dataset are input into the trained edge extractor Edge Extractor to obtain the structure estimation network (ISE Block) of the image, which requires the use of low-illumination rain and fog image structure and normal-illumination image structure, specifically:

[0054] 2-1) Refer to Figure 1 , the low-illumination rain and fog image and the corresponding normal-illumination image are input into the edge extractor Edge Extractor, which is composed of 1 3*3 convolution, 4 encoders, 3 decoders and 1 3*3 convolution, wherein the encoder and the decoder are composed of ResBlock;

[0055] 2-2) Refer to Figure 2 , the ResBlock module is composed of two 3*3 convolutions, a LeakyReLu activation function and a residual connection, and the image is input into the encoder and the decoder composed of the ResBlock module to obtain the low-frequency feature;

[0056] 2-3) The image is composed of high-frequency features and low-frequency features, wherein the high-frequency features constitute the edges and texture details of the image, and the low-frequency features are related to the brightness and color information of the image, so the high-frequency feature information, i.e. the structure of the image, is obtained by subtracting the low-frequency feature from the original image;

[0057] 3) Training of image structure estimation network ISE Block: ISE Block is a U-shaped network based on Transformer, as shown in Figure 3 , first, the low-illumination structure is input into the ISE Block, first, the rich features of the image are extracted through the Encode and down-sampling operation; second, the size of the image is restored through the Decode and up-sampling operation, and the obtained feature information and the feature information of the corresponding stage of the encoder are multi-scale fused to model the non-local information; finally, the obtained feature information is mapped to the normal-illumination structure estimation through the Out Embedding, and the structure details of the ISE Block refer to Table 2:

[0058] Table 2: Structure details of ISE Block

[0059]

[0060]

[0061] The specific process of training the image structure estimation network ISE Block includes:

[0062] 3-1) Refer to Figure 3 Map the structure of the low-illumination rain and fog image to the tokens of the corresponding image through Input Project, wherein Input Project is composed of a convolution kernel with a size of 3*3, a step of 1, and a LeakyRELU activation function, and the size of the output feature map is 256*256*32;

[0063] 3-2) The feature map obtained in step 3-1) is sent to the encoder, and the encoder is composed of 5 Encoder Blocks, each Encoder Block has two branches, in the first branch, first pass through a Transformer module to extract feature information, and then pass through a convolution with a kernel size of 4*4 and a step of 2 for down-sampling; in the other branch, down-sampling is performed through a Maxpooling with a step of 2, and finally the feature maps obtained from the two branches are concatenated, here there are two forms of down-sampling mode, one is a single branch structure, only a 4*4 convolution with a step of 2 is used, and the other is the above-mentioned double branch structure, as shown in Figure 4 In addition to Encoder Block1 and Encoder Block2, which use the double branch structure, the other Encoder Blocks use the single branch structure for down-sampling, and the purpose of using the double branch down-sampling is to retain important information in the early stage of feature extraction;

[0064] 3-3) The Transformer Block is composed of two modules, namely the multi-convolution channel attention MCCA module and the cross-gate feedforward neural network CGFN module, the main computational complexity in the Transformer comes from the Self-Attention operation, and the time and space complexity of the dot product operation of Key-Query increases quadratically with the size of the input spatial resolution, therefore, it is not advisable to directly apply Self-Attention to high-resolution image tasks, in order to solve this problem, the multi-convolution channel attention MCCA module is used, as shown in Figure 6 ​The feature map is subjected to Layer Normalization, and then 1*1 convolution is performed on three branches to fuse cross-channel pixel-by-pixel information, and then 3*3 convolution is performed to capture the spatial content of each channel, and Q, K, and V containing rich local information are obtained by using formula (2), formula (3), and formula (4). is a weight parameter of 1*1 convolution, is a weight parameter of 3*3 convolution, and then the Key-Query is obtained by point multiplication operation according to formula (5). Then, the value of Attention is obtained according to formula (6), and then 1*1 convolution is performed to fuse channel information, and finally, the obtained feature map and the input feature map are added element by element to further enrich the diversity of the features. The formalization process of the MCCA module is defined as formula (7), wherein a is a learnable parameter.

[0065]

[0066] The obtained feature map is input into the CGFN module, as shown in formula (8). Figure 7 First, Layer Normalization is performed, and then and convolution is performed on the information from spatially adjacent pixel positions, which is very useful for learning local image structures. Then, GLU nonlinear activation functions are performed on the two parallel paths, the obtained feature map and the input feature map are added element by element, and then a convolution is performed to further obtain rich local feature information. The obtained feature map and the feature map subjected to the GLU nonlinear activation function are multiplied element by element, and then the feature maps on the parallel paths and the input feature map are added element by element. 1*1 (W1) convolution is performed to fuse channel information and further enrich the diversity of the features. Finally, Layer Normalization is performed to accelerate the training and improve the performance of the model. For a given input CGFN can be formally defined as formula (9).

[0067]

[0068] Y1 and Y2 are respectively defined as the outputs of the first branch and the second branch in the CGFN module, is defined as element-wise addition, is defined as element-wise multiplication, and Y is the final output.

[0069] The obtained feature map is further processed by 4 Encoder Blocks and 1 separate Transformer Block, and then the feature map is input into the decoder, which also consists of 5 Decoder Block modules, each of which consists of 1 up-sampling module and a Transformer Block module, as shown in Figure 5 First, the input feature map and the corresponding feature map of the decoder are concatenated, as shown in Figure 8 Second, the up-sampling is performed by a deconvolution layer with a kernel size of 2*2 and a step size of 2, and finally a Transformer Block module is executed to obtain a feature map with a size of 256*256*64.

[0070] 3-4) The feature map obtained in step 3-3) is mapped to a normal illumination structure estimation by Out Project, wherein Output Project is composed of a convolution with a kernel size of 3*3 and a step size of 1, and the output size is 256*256*3. In order to improve the quality of the structure estimation, a structure reconstruction loss is used to guide the learning process. The structure reconstruction loss is given by formula (10), which is a hybrid loss including smooth L1 loss and structure loss SSIM loss. λ is a balance weight parameter. The smooth L1 loss and the structure loss SSIM loss are given by formulas (11) and (12):

[0071]

[0072] μ x and μ y are the average pixel values, and are the variances, σ xy is the covariance, and C1 and C2 are constants.

[0073] 4) Training of the Transformer-based generator: The generator G (Generator) is a Transformer-based U-shaped network, and has a similar structure to the ISE Block. Therefore, the entire execution process is similar to that of the ISE Block. The difference is that the input of the generator G is a low-illumination rain and fog image, and the SGTT is used to guide the better learning of the structure features of the generator G in the encoding and decoding modules by using the multi-scale rich structure feature information learned in the ISE Block, and finally an enhanced image is generated. The specific process is as follows:

[0074] 4-1) Refer to Figure 3The low-illumination rain and fog image is mapped into tokens of the corresponding image through Input Project, wherein Input Project is composed of a convolution kernel with a size of 3*3, a step of 1 and a LeakyRELU activation function, and the size of the output feature map is 256*256*32;

[0075] 4-2) Since the generator G and the ISE Block have similar structures, other training processes are similar to the ISE Block;

[0076] 4-3) The structural feature information obtained in the ISE Block guides the generator G, and the structural feature obtained in the ISE Block is fused with the feature extracted in the corresponding stage of the generator G through SGTT. The SGTT module is defined as formula (13), and the structure of the SGTT module is as shown in Figure 9

[0077]

[0078] In order to measure the content difference between the enhanced image and the real image, the MSE loss is used as the content loss, so the loss of the generator is defined as:

[0079]

[0080] Wherein G(R) is the enhanced image of the generator, and I is the image of normal illumination;

[0081] 4-4) The feature map obtained in step 4-3) is mapped into normal-illumination structure estimation through Out Project, wherein Output Project is composed of a convolution kernel with a size of 3*3 and a step of 1, and the output size is 256*256*3 normal-illumination structure estimation;

[0082] 5) Training of the discriminator based on the Transformer: the discriminator D (Discriminator) is also a U-shaped network based on the Transformer, and has a similar structure to the generator G, so the entire execution process is similar to the generator G, and the difference is that the enhanced image obtained by the generator G and the normal-illumination structure estimation obtained by the ISE Block are sent into the discriminator D together, and in the encoding and decoding modules, the multi-scale rich structural feature information learned in the ISE Block is used to guide the discriminator D to better identify the true or false of the image output by the generator G through SGTT. The specific process is as follows:

[0083] ​5-1) The normal illumination structure estimation and the enhanced image obtained in step 4) are combined and mapped into the tokens of the corresponding image through InputProject, where InputProject is composed of a convolution kernel with a size of 3*3, a step of 1 and a LeakyRELU activation function;

[0084] 5-2) Since the discriminator D and the generator G have similar structures, the other training processes are similar to the generator G;

[0085] 5-3) The structural feature information obtained in the ISE Block guides the discriminator D, and the structural features obtained in the ISE Block are fused with the features extracted in the corresponding stage of the discriminator through SGTT. The structure of the SGTT module is as shown in Figure 9

[0086] 5-4) Unlike the generator for synthesizing accurate pixels, the task of the discriminator is to distinguish the real / fake of the image, so it is regarded as a classifier. Therefore, before the Decoder Block4, a class [cls] token is attached at the beginning of the 1D sequence, and then the prediction of real / fake is output by the classification head. Specifically, the size of the feature map output by the Decoder Block3 is 128*128*128. First, the [cls] and the feature map are spliced in dimension 1, and then sent to the Decoder Block4. Finally, the classification head composed of Linear is used, the input dimension is 64, and the output dimension is 1. In order to improve the performance of the network and better help the generator G to generate real images, at the same time, it is hoped that the image can deceive the discriminator D to distinguish the real and fake images, the adversarial loss is defined as formula (15):

[0087]

[0088] Combined with the reconstruction loss, MSE, the overall loss is given by formula (16), where λ m and λ a are balance weight parameters,

[0089] L total =L mix +λ m L mse +λ a L adv (16);

[0090] 6) Test of the transformer-based generative adversarial network low-illumination rain and fog image enhancement network: including:

[0091] ​6-1) Send the low-light rain and fog images in the LOL_RF dataset into the enhancement network for enhancement;

[0092] 6-2) Visualize the enhanced results.

Claims

1. A TF-GAN based rain and fog image enhancement method under low illumination, characterized by: The steps include: 1) Reconstruct the existing LOL dataset: Reconstruct the existing LOL dataset to obtain the dataset LOL_RF in low-light rain and fog environment. The process is as follows: 1-1) Based on the public LOL dataset, we use random trajectory generation to simulate the imaging characteristics of rainy and foggy environments, and obtain the LOL_RF dataset for low-light rainy and foggy environments. 1-2) The LOL_RF dataset includes 500 low-light rain and fog images and their corresponding normal-light images, of which 485 are training images and 15 are test images. 2) Generation of structure estimation based on edge extractor: The edge extractor is used to obtain the structure graph of the low-light rain and fog image and the structure graph of the normal-light image. The process is as follows: 2-1) Input the low-light rain and fog image and the corresponding normal-light image into the edge extractor; 2-2) Use the encoder-decoder composed of ResBlock modules to obtain low-frequency features; 2-3) Subtracting low-frequency features from the original image to obtain high-frequency feature information, namely, the structure diagram of the low-illumination rain and fog image and the structure diagram of the normal illumination image; 3) Training of the Image Structure Estimation Network (ISE) Block: The structure diagram of the low-light rain and fog image is input into the Image Structure Estimation Network (ISE) Block to obtain the estimated structure of the normal-light image. Specifically: 3-1) Map the structure of the low-light rain and fog image into tokens through the Input Project; 3-2) The tokens are passed through the encoder block composed of multi-convolutional channel attention (MCCA) and cross-gate feedforward neural network (CGFN) for feature extraction, and the obtained features are encoded into increasingly deeper embedding spaces; 3-3) Multi-convolutional channel attention (MCCA) implements cross-channel information fusion through 1×1 convolution, and 3×3 convolution captures local context information to obtain Q, K, and V matrices. Then, channel attention is calculated in the channel dimension, and the cross-gate feedforward neural network (CGFN) uses 1×1 convolution to expand the channel dimension of the feature. Then, 3×3 depth-wise separable convolution is used on each branch to effectively extract multi-scale local information of the image. Finally, the GELU activation function and element-wise multiplication operation are used on two parallel branches respectively. The extracted features are decoded by the decoder module composed of multi-convolutional channel attention (MCCA) and cross-gate feedforward neural network (CGFN). Skip connections are used to fuse the features of the symmetric blocks in the encoder, and the obtained features are decoded into increasingly shallow embedding spaces. 3-4) Map the tokens obtained in step 3-3) into a normal illumination structure estimate through the Output Project; 4) Transformer-based generator training: A low-light rain and fog image is input to the generator to obtain a high-quality enhanced image, including: 4-1) Use Input Project to convert low-light rain and fog images into corresponding tokens; 4-2) The tokens of the corresponding image obtained in step 4-1) are passed through an encoder module composed of multi-convolutional channel attention (MCCA) and a cross-gate feedforward neural network (CGFN) for feature extraction, and the obtained features are encoded into increasingly deeper embedding spaces; 4-3) The feature vector is decoded by a decoder module consisting of a multi-convolutional channel attention (MCCA) and a cross-gate feedforward neural network (CGFN). Before each decoder module, the features output by the corresponding encoder are fused, and the structural feature information obtained from ISEBlock is used to guide the generation process of the generator G. 4-4) The tokens obtained in step 4-3) are converted into an enhanced image, that is, the target image, through the Output Project; 5) Training the Transforemr discriminator: The enhanced image and the normal illumination structure estimate are input into the discriminator D (Discriminator) to distinguish between real and fake images, including: 5-1) The enhanced image and the normal illumination structure estimation are mapped into tokens through the input project. 5-2) The tokens of the corresponding image obtained in step 5-1) are passed through a series of encoder modules for feature extraction; 5-3) The feature vector extracted in step 5-2) is decoded through a series of decoder modules, and the features of the symmetric blocks in the encoder are fused using skip connections. The structural features obtained in the ISE block are then used to guide the discriminator D to better distinguish the authenticity of the generated images. 5-4) Append the category [cls] token to the beginning of the tokens obtained in step 5-3) and feed them into the detection head, which outputs the prediction of the image real / fake; 6) The low-light rain and fog images in the LOL_RF test set are fed into the enhancement network for enhancement processing, including: 6-1) Send the low-light rain and fog images in the LOL_RF dataset to the enhancement network for enhancement; 6-2) Visualize the enhanced results.

Images