Multi-channel illumination adaptive network model with Laplacian enhanced pyramid and low-light image enhancement method

Abstract

The invention discloses a multi-channel illumination adaptive network model with a Laplacian enhanced pyramid and a low-light image enhancement method, and belongs to the technical field of low-light image enhancement processing. Frequency information optimization and brightness adaptation are crucial to low-light image enhancement and exposure correction. According to the network provided by the invention, two modules work cooperatively. The method comprises the following steps: firstly, designing a Laplacian enhanced pyramid (LEP), decomposing an image into components with different frequencies, and optimizing the components to perform preliminary enhancement; and then, a multi-channel illumination adaptive Transform (MIAT) is designed, and the color of the image and a multi-channel gamma correction parameter are inquired, so that further fine adjustment is carried out, and an enhanced image is generated. The MIANet is only about 0.17 M in parameter, the average enhancement processing time is only 0.007 seconds when an image with the resolution of 600 * 400 is processed, and the method is superior to the latest advanced method in LLIE and exposure correction benchmark tests.

Description

Technical Field

[0001] The present invention belongs to the technical field of low-light image enhancement processing, and specifically relates to a multi-channel illumination adaptive network model with a Laplace enhancement pyramid and a low-light image enhancement method. Background Art

[0002] Low-Light Image Enhancement (LLIE) [1,2,3,4,5,6,7,8,9,10,11] and exposure correction [12,13,14] It is an important and challenging task in computer vision. Images taken in low-light and overexposed environments have many defects, including reduced contrast and color distortion. Reduced contrast can cause blurring or loss of image details, while color distortion can lead to inaccurate color reproduction and color shift. In addition, these images are usually accompanied by noise problems. These defects not only pose challenges to human visual perception, but also bring difficulties to other visual tasks such as object detection and semantic segmentation under different lighting environments.

[0003] The main function of the low-light image enhancement (LLIE) model is to improve the quality of images taken in low-light environments. [23,24] and gamma correction based methods [25,26] The method adopts a rough global processing method, lacks adaptive processing of local features and noise of the image, and ignores the changes in lighting conditions. The enhanced image usually has a lot of noise and artifacts, and is prone to color distortion. [2,27,28,29] Attempts were made to decompose the image into two components: reflection and illumination. However, due to the assumption that there is no noise and color distortion during the enhancement process, these problems were not effectively addressed, resulting in low quality of the enhanced image. In addition, some methods are sensitive to parameter adjustment, resulting in poor model adaptability.

[0004] With the development of deep learning, more and more methods based on convolutional neural networks (CNN) are [1,3,4,7,10,11,30,31,32] was applied to LLIE. For example, Wang et al.

[11] The relationship between amplitude spectrum and brightness is explored through Fourier transform, and the signal-to-noise ratio (SNR) map is introduced to provide a priori for integrating network information, thereby recovering image details. Wang et al.

[10] In order to cope with ultra-high-definition low-light image enhancement, a Transformer-based low-light enhancement method called Low-Light Transformer (LLFormer) was introduced. However, in order to learn a good mapping function, LLFormer requires a huge amount of parameters and computation. Zheng et al. [8]Lighting and noise priors based on masked autoencoders (MAEs) are designed to improve the performance of the model, but the effectiveness of these priors often requires complex training strategies to be guaranteed.

[0005] The main function of the exposure correction model is to adjust overexposed or underexposed images to achieve the appropriate exposure level. Similar to LLIE, more and more deep learning-based methods are being used to [12 ,14,33,34 ,35] It is applied in the field of exposure correction. Afifi et al. [12 ] proposed a coarse-to-fine deep neural network model to decouple the exposure problem into two sub-problems: color enhancement and detail enhancement. Yuan et al.

[14] Correct the image exposure problem through a nonlinear tone curve. Huang et al. [34 ]With the help of Fourier transform, a Fourier-based deep exposure correction network is proposed, which consists of an amplitude subnet and a phase subnet. Huang et al.

[35] A multi-exposure correction framework based on the exposure normalization and compensation module (ENC) is also proposed. However, the parameters of this framework need to be fine-tuned through a specific regularization strategy to effectively play its role.

[0006] In recent years, illumination adaptive methods [15 ,36 ,37] It has been widely used in image enhancement in harsh lighting environments. Jin et al.

[37] proposed an unsupervised method to adjust image lighting by integrating a layer decomposition network and a light effect suppression network. Yang et al.

[36] Based on the frequency decomposition technique, illumination adaptation is decoupled from multiple light-related tasks, thus solving the general image enhancement problem. Yao et al.

[15] proposed channel-selective normalization (CSNorm), a plug-and-play method that improves the generalization ability of the base network under unknown brightness conditions. However, its effectiveness depends on complex training strategies and data perturbations. In addition, most brightness adaptive methods only consider brightness factors during image enhancement without effectively optimizing frequency information, thus failing to completely solve problems such as noise. This makes the enhanced image still lacking in detail and clarity. Song et al.

[16] Attempts have been made to improve this situation by adding some modules, but the effectiveness of these new modules often requires complex training strategies or additional data enhancement to be guaranteed. In addition, these new modules do not fully consider the importance of frequency information.

[0007] Laplace Pyramid

[17] Originally used for image analysis and compression tasks. With the rapid development of deep learning, many methods integrate the Laplacian pyramid into convolutional neural networks (CNNs). Song et al.

[38] Incorporate the Laplacian pyramid into the decoder architecture to fully exploit the well-encoded features for monocular depth estimation. Qin et al.

[39] and Yinet al.

[40] A Laplacian pyramid enhancement subnetwork was constructed and combined with an object detector to propose a new object detection framework. However, these methods highly couple the Laplacian pyramid with the underlying model of high-level visual tasks, limiting the independent effectiveness of the Laplacian pyramid.

[0008] Transformer

[41] It was originally proposed as a natural language processing model and applied to the field of machine translation.

[42] It is the first application of Transformer in the field of computer vision, successfully transforming the image classification problem into a sequence modeling problem.

[43] The CNN-Transformer cascade was used to apply Transformer to image object detection for the first time. Xie et al.

[44] Combining Transformers with a lightweight multi-layer perceptron (MLP) decoder, a simple yet effective semantic segmentation framework SegFormer is proposed.

[45] By introducing a sliding window-based self-attention mechanism and combining it with a local receptive field, the computational efficiency and accuracy are effectively improved. Scholars further improved the Swin Transformer [46,47] , these models have achieved significant performance improvements in multiple tasks such as computer vision classification, detection, and segmentation.

[0009] In summary, with the development of deep learning, most methods based on convolutional neural networks (CNNs) or Transformer architectures have been proposed and applied in this field, aiming to adjust the illumination while optimizing the image frequency information through carefully designed loss functions and supervised training. However, this usually requires the network to learn a very complex mapping relationship, which is difficult to achieve without increasing the size of the dataset or the size and computation of the model. Summary of the invention

[0010] Frequency information optimization and brightness adaptation are crucial for low-light image enhancement (LLIE) and exposure correction. The former can improve the details, clarity and color reproduction of the image, while the latter can ensure brightness balance and color accuracy to avoid overexposure or underexposure. However, most existing methods find it difficult to achieve these two goals simultaneously and efficiently without significantly increasing the number of parameters and computations. To solve this problem, the present invention proposes a lightweight and fast multi-channel illumination adaptation network.

[0011] To achieve the above purpose, the technical solution adopted by the present invention is:

[0012] A multi-channel illumination adaptive network model with Laplacian enhanced pyramid, including a Laplacian enhanced pyramid (LEP) module and a multi-channel illumination adaptive Transformer (MIAT) module;

[0013] In LEP, the image is first decomposed into low-frequency and multiple high-frequency components through the Laplacian pyramid prior; then, these components of different frequencies are fused and optimized through a multi-scale structure and channel multi-head self-attention (CMSA); finally, the enhanced components are fused and denoised through Gaussian convolution to reconstruct a preliminary enhanced image;

[0014] In MIAT, the Value prior is first used to more efficiently query color and multi-channel gamma correction parameters; then, CMSA is used to reweight channel features so that the network can prioritize key information.

[0015] Finally, illumination space multi-head attention (IS-MA) is designed and integrated into MIAT to capture global dependencies in image space, extract color and multi-channel gamma correction parameters, and further optimize the image to generate enhanced images.

[0016] Frequency information optimization and brightness adaptation are crucial for low-light image enhancement and exposure correction. The proposed network consists of two modules working together, and two modules are designed: Laplacian Enhancement Pyramid (LEP) and Multi-channel Illumination Adaptive Transformer (MIAT), which utilize Laplacian Pyramid prior and Value prior, respectively. These priors not only reduce the computational complexity required for network learning, but also synergistically optimize the frequency information of the image and correct the image illumination through carefully designed modules. In LEP, the image is first decomposed into low-frequency and multiple high-frequency components through Laplacian pyramid prior. Then, these components of different frequencies are fused and optimized through multi-scale structure and channel multi-head self-attention (CMSA). Finally, the enhanced components are fused and denoised through Gaussian convolution to reconstruct the preliminary enhanced image. In MIAT, Value prior is first used to more efficiently query color and multi-channel gamma correction parameters. Subsequently, CMSA is used to reweight channel features so that the network can prioritize key information. Finally, an illumination-space multi-head attention (IS-MA) is designed and integrated into MIAT to capture global dependencies in the image space, extract color and multi-channel gamma correction parameters, and further optimize the image to generate an enhanced image. By combining LEP and MIAT modules, we develop a novel multi-channel illumination adaptation network (MIANet).

[0017] In addition, the present invention has conducted extensive experiments on datasets under multiple different lighting environments, including LLIE datasets LOL-v1, LOL-v2-real, exposure correction datasets, low-light detection datasets EXDark, and low-light segmentation datasets ACDC. Quantitative analysis shows that MIANet achieves state-of-the-art performance in various indicators of various low-level and high-level tasks; while qualitative analysis shows that MIANet can better preserve and optimize the frequency information of images (including texture, structure, brightness, and color), and adjust the brightness of images more naturally.

[0018] Compared with the existing solutions, the beneficial effects of the present invention are:

[0019] 1. This paper proposes MIANet, a lightweight and fast network that optimizes frequency information and naturally adjusts brightness to improve image quality under various lighting conditions.

[0020] 2. The present invention introduces LEP, a new Laplace enhanced pyramid, which enhances and fuses different frequency components through FF-LSA and CMSA, while optimizing the frequency information with minimal computational cost by utilizing a unique inter-layer communication structure.

[0021] 3. We develop MIAT, a new multi-channel illumination adaptive transformer, which uses CMSA to focus the network on brightness and color features. It uses IS-MA to query color and multi-channel gamma correction parameters to effectively correct image brightness.

[0022] 4. The present invention has conducted extensive experiments on datasets of multiple low-level and high-level tasks under different lighting conditions. The results show that MIANet outperforms existing state-of-the-art methods. In addition, the MIANet model of the present invention has only 0.17M parameters and only takes 0.007 seconds on average to process a single image, which reflects its high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the overall architecture of the multi-channel illumination adaptive network model with Laplacian enhanced pyramid. (a) is MIANet, which consists of two parts: (i) Laplacian enhanced pyramid and (ii) multi-channel illumination adaptive Transformer. (b) is the structure of channel multi-head self-attention (CMSA). (c) is the structure of illumination space multi-head attention (IS-MA).

[0024] Figure 2 This is the enhancement process diagram of LEP. H1 to L represent the components before enhancement. arrive Represents the enhanced components, H1 to L and arrive The resolution of

[0025] Figure 3 The structure of FF-LSA. (a) FF-LSA consists of two main parts: (i) Feature Fusion (FF) and (ii) Local Spatial Attention (LSA). (b) The key unit of LSA is Deep Selective Kernel Attention (DSKA), which includes a multi-branch, multi-scale deep convolutional structure and a fully connected layer.

[0026] Figure 4 The visual results of different methods on LOL-v1 show that the method of the present invention not only retains more details, but also adjusts the brightness of the image more naturally.

[0027] Figure 5 The visual results of different methods on LOL-v2-real show that the method of the present invention maintains better contrast while improving the brightness, making the image more natural and clear.

[0028] Figure 6The visual results of the enhanced sub-network of MIANet and IAT on LOL-v2-real show that the method of the present invention not only effectively improves the visibility but also preserves the color, while also showing significant advantages in details.

[0029] Figure 7 The visual results of different methods on Exposure Errors show that the image corrected by the method of the present invention not only has high definition and natural brightness, but also has color restoration that is closer to the real scene.

[0030] Figure 8 The visual results and target detection results of different methods on ExDark. (a) is the input image, (b) is the detection result of the input image, and (c) is the detection result of KinD [7] The enhanced detection results, (d) are obtained through IAT [9] The enhanced detection results,(e) are obtained by Zero-DECE [1] The enhanced detection result, (f) is the detection result enhanced by the method of the present invention. The method of the present invention has significant advantages in brightness adjustment, color restoration and detail retention, and can be closer to the real scene, thereby achieving more accurate target detection. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] Example 1

[0033] Photos taken under harsh lighting conditions often have defects such as unclear texture structure, color distortion and color shift. To solve this problem, the present invention proposes a multi-channel illumination adaptive network (MIANet). MIANet is not only lightweight and fast, but also can significantly improve the quality of images taken under harsh lighting conditions. Its training process only requires one stage. Figure 1 The overall architecture of MIANet is shown. Figure 1 As shown in (a), MIANet consists of two modules: (i) Laplacian Enhanced Pyramid (LEP) and (ii) Multi-channel Illumination Adaptive Transformer (MIAT). Specifically, first, LEP enhances and optimizes the frequency information of the image through feature fusion local spatial attention (FF-LSA) and channel multi-head self-attention (CMSA), thereby generating a preliminary enhanced image. Then, MIANet effectively queries the color transformation matrix and multi-channel gamma correction parameters of the input image through channel multi-head self-attention (CMSA) and illumination space multi-head attention (IS-MA), further corrects the preliminary enhanced image, and finally outputs the enhanced image. Figure 1 (b) and Figure 1 (c) shows the details of CMSA and IS-MA respectively. Their constructions are described in detail below:

[0034] 1 Laplace Augmented Pyramid

[0035] In order to effectively optimize the frequency information, the present invention proposes a Laplacian Enhancement Pyramid (LEP). LEP can preserve and enhance texture and detail information, reduce noise, and correct the brightness and color of the image to a certain extent. Figure 2 The enhancement process of LEP is demonstrated. By decomposing, enhancing and reconstructing the image, a high-quality enhanced image can be obtained. Specifically, with the help of Laplacian pyramid prior, the information of the image at different frequencies can be directly obtained. LEP first uses Laplacian pyramid decomposition to transform the image Decompose into multiple frequency components of different resolutions, denoted as F, including a low-frequency component L and multiple high-frequency components H. The decomposition process of LEP (1≤i≤N) is defined as follows:

[0036] G0=I0 (1)

[0037] G i+1 =Down2(G i *Gaussian) (2)

[0038] H i =G i -Gaussian(Up2(G i+1 )) (3)

[0039] Where I0 is the input image and G0 is the 0th layer Gaussian image. i is the Gaussian image of the i-th layer, Down2 means double downsampling, and Gaussian means Gaussian convolution of size 5×5. i is the Laplacian image of the i-th layer, and Up2 means double upsampling.

[0040] According to formula (2), the Gaussian image G at the bottom layer N As the low-frequency component with the lowest resolution, it is denoted as L. Figure 2 As shown in Figure 2, L retains the most global low-frequency information, reflecting the large-scale structure of the image and the brightness and color distribution of a large area. According to formula (3), each high-frequency component, denoted by H i , which means from the i-th layer Gaussian image G i To the i+1th layer Gaussian image G i+1 The difference between Figure 2 As shown in Figure 2, these high-frequency components capture the image details at different resolutions, but also contain a certain degree of noise. LEP can effectively eliminate this noise when reconstructing the image.

[0041] The present invention adopts CMSA and FF-LSA, and uses the inter-layer communication structure to enhance and fuse L and H to obtain the enhanced components, which are recorded as Includes an enhanced low frequency component and multiple enhanced high frequency components like Figure 2 As shown, the enhanced components not only optimize the frequency information, but also decouple the brightness and color into different components, which helps to adjust the brightness and color of the image. Finally, the enhanced components are reconstructed using recursive addition, Gaussian convolution, and upsampling operations. Gaussian convolution can effectively reduce the noise in high-frequency components by smoothing the image, which helps to suppress random noise and preserve the basic structure. The reconstruction process of LEP (0≤i≤N-1) is defined as follows:

[0042]

[0043] in, is the high-frequency component after the i-th layer enhancement, G N is the low-frequency component after the Nth layer enhancement, G i is the Gaussian image after the i-th layer enhancement and reconstruction, Up2 is double upsampling, and Gaussian represents a Gaussian convolution of size 5×5.

[0044] In order to better combine information of different scales when processing frequency components and help the network better adapt to texture and detail changes in images, the present invention proposes FF-LSA, the details of which are as follows: Figure 3 As shown. Local spatial attention (LSA) is used to enhance the low-frequency component L. In LSA, depth-wise separable convolution is used

[48] and its inverse operation to achieve feature dimension transformation, which not only retains the effective feature extraction capability but also reduces the computational burden of the network. In order to facilitate the direct transmission of information, a skip connection is added between the output and input of FF-LSA.

[49] Based on the design experience of LSA, this paper proposes Deep Selective Kernel Attention (DSKA), which is the key module of LSA. Specifically, DSKA first captures information of different receptive fields through a multi-scale deep convolutional structure while maintaining low computational complexity. Then, the multi-scale feature map is added element by element, and the selection weights are generated through global average pooling and fully connected layers. Finally, the selection weights are normalized by softmax and then weighted summed with the corresponding feature map to obtain the final output feature map. DSKA is similar to the local spatial attention mechanism in that they are both able to focus on different areas of the image and enhance feature representation. The calculation process of DSKA is defined as follows:

[0045] Fk×k =DWC k×k (F in ) (5)

[0046] F sum =FC d (GP(F in +F 3×3 +F 5×5 +F 7×7 )) (6)

[0047]

[0048]

[0049] in, is the input feature map, DWC k×k is a depthwise convolution with a kernel size of k×k, F k×k Yes F in By DWC k×k The processed feature map. GP is global average pooling, FC d is a fully connected layer with output dimension d, F sum is the selection weight. FC c is a fully connected layer with an output dimension of c, softmax is the softmax activation function, concat is the concatenation operation, and Z k×k is the weighted feature, and ⊙ is the element-by-element multiplication.

[0050] Unlike processing the low-frequency component L, the present invention uses FF-LSA for the high-frequency component H. Compared with ordinary LSA, FF-LSA adds a feature fusion (FF) module when processing high-frequency components. Specifically, the FF module first upsamples the enhanced low-resolution component, and then integrates it into the frequency component of the current layer through multiple convolutional layers by pixel-by-pixel multiplication and pixel-by-pixel addition. In this way, the FF module can effectively combine information from different scales, thereby enhancing the feature representation capability.

[0051] In addition, in LEP, although FF-LSA has the ability of local spatial attention and feature fusion, it cannot effectively capture the global dependencies between feature map channels. In order to solve this problem, the present invention introduces CMSA. Because different channels may represent different features of the image (such as edges, textures, colors, etc.), the network should focus on different features in components of different frequencies. By applying CMSA, the weights of each channel can be adaptively adjusted, thereby helping the network to better understand and process the relationship between these features and improve image quality. Figure 2As shown in the figure, the components of different frequencies are not only more detailed during the enhancement process, but also have different color and brightness information. This is because CMSA adaptively adjusts the response intensity of the color channel when adjusting the feature weights, thereby enhancing the color expression. At the same time, the application of CMSA on different frequency components makes the brightness of the reconstructed image more balanced, naturally avoiding the problems of low light and overexposure. Therefore, CMSA not only improves the detail expression of the image, but also effectively adjusts the overall brightness and color of the image, making the image reconstructed by LEP more natural and realistic.

[0052] When using CMSA, there are three design points to note: (i) Add a long jump connection between the input of the first CMSA and the output of the last CMSA in each layer of the pyramid. This is because although the CMSA already has an identity connection inside, the deeper network may still cause information loss. In addition, the prediction input F to the output Compared with directly predicting its mapping, the residual of can make the network training more stable and have better performance. (ii) The enhanced component in each layer of the pyramid Passed to the FF-LSA of the higher resolution layer, and then processed by CMSA. Through this inter-layer communication method, CMSA can not only encode the frequency components of the current layer, but also decode the frequency components of the low resolution, thereby maximizing the enhancement and optimization of the frequency information. (iii) Do not use CMSA for H1. Because the resolution of H1 is too high, using CMSA will result in a large number of parameters and calculations, and the local information captured by the feature map is too little. This will cause CMSA to be ineffective or even lose information due to distraction.

[0053] Feature Map is defined as input and converted into tokens Then the last dimension is divided into h parts, that is, h heads, expressed as in, It should be noted that in Figure 1 (b) only shows the self-attention calculation process of X1. The self-attention calculation process of each head in CMSA is defined as follows:

[0054]

[0055] Among them, X i is the input element of the i-th head, Q i , K i 、V i They are respectively the query element, key element, and value element of the i-th head. Both represent the matrix transpose of the fully connected layer. represents matrix multiplication, r iis a learnable parameter used for adaptively scaling matrix multiplication.

[0056] After calculating self-attention, a feed-forward network (FFN) is needed to enrich and reconstruct the feature representation. 50 ] design experience, the present invention adopts the anti-bottleneck structure to implement FFN. Different from the traditional bottleneck structure, the anti-bottleneck structure can avoid the information compression and loss caused by the transformation of feature spaces in different dimensions. The self-attention outputs of h heads are spliced ​​together to obtain the feature map A∈R hw×c , transforming it into And defined as input, the calculation process of FFN is defined as follows:

[0057] f out =PWC c (DWC 4c (PWC 4c (LN(X out ))))+X out (11)

[0058] Among them, LN stands for layer normalization, PWC 4c It is a point-by-point convolution with an output dimension of 4c, DWC 4c is a depthwise convolution with an output dimension of 4c, PWC c is a point-wise convolution with output dimension c, f out is the output of FFN.

[0059] The spatial attention of FF-LSAi and the channel attention of CMSA cooperate with each other to maximize the enhancement and optimization of frequency information in LEP.

[0060] 2-channel adaptive transformer for lighting

[0061] Applying the same Gamma value to the three channels of RGB in an image for correction may cause color distortion. In addition, it is inappropriate to give the same attention to all channels of the image feature map. When querying the correction parameters, the network should pay more attention to the channels related to color and brightness. In order to solve these problems, the present invention proposes MIAT. In addition, using HSV

[18] The Value prior in the color space assists MIAT to better query the correction parameters. Value represents the brightness of the color and plays a key role in adjusting the color brightness. The calculation process of Value is defined as follows:

[0062] V=max(R,G,B) (12)

[0063] Among them, R, G, and B represent the red, green, and blue components in the RGB color space, respectively.

[0064] MIAT first uses ordinary convolution and depthwise convolution to reduce the amount of parameters and reduce the size of the feature map, and then uses CMSA to adjust the weights of each channel so that the color transformation matrix can be queried more efficiently in subsequent steps. In order to correct image illumination more accurately, the present invention designs IS-MA. IS-MA extracts a 3×3 color transformation matrix and 3 gamma correction parameters from the image through 12 global component queries. These three gamma parameters will act on the R, G, and B channels of the image respectively. Unlike CMSA, in IS-MA, each query Q is initialized to a learnable all-1 matrix. In addition, during the training process, IS-MA adds a unit matrix and an all-1 vector to the color transformation matrix and gamma correction parameters, respectively, to stabilize the training.

[0065] IS-MA operates on the feature map of the input image after 4 times downsampling, and each feature map pixel corresponds to a specific region of the input image. During the training phase, each query will interact globally with all regions of the image. In addition, the query can be dynamically updated in each iteration to better match the reference image.

[0066] In order to avoid generating too many white and black pixels, this paper draws on the idea of ​​label smoothing [ 51 Specifically, the scalar value of each channel of each pixel in the image processed by the color transformation matrix is ​​restricted to a range between ε and 1-ε, where ε = 1e -8 This restriction is because scalar values ​​of 0 or 1 will cause subsequent gamma correction to fail. In this way, the model can reduce its over-reliance on these extreme pixel values. This smoothing approach helps the model better adapt to a variety of different image data and improves its robustness to operations such as color transformations.

[0067] The output image of LEP As input, the calibration process of MIAT is as follows:

[0068]

[0069] in, is the color transformation matrix, is matrix multiplication, ε = 1e -8 I R ,I G ,I B Represent the R, G, and B channels of image I, respectively. R , γ G , γ B Represent the γ correction parameters of the three channels of R, G, and B respectively. Finally, the enhanced images of the three channels are merged in the channel dimension to obtain the final enhanced image. The merging process is defined as follows:

[0070] I e =(I R ,I G ,I B ) T (15)

[0071] MIAT can better maintain the color balance of the original image and improve the performance of image details by querying better color transformation matrix and multi-channel gamma correction parameters. In addition, MIAT can more accurately adjust the brightness of the image to reduce overexposure and underexposure problems, thereby improving the overall image quality.

[0072] Example 2 Experiment

[0073] The performance of MIANet under different lighting conditions is evaluated on multiple benchmark datasets. These benchmark datasets cover two low-level vision tasks and two high-level vision tasks. The low-level vision tasks include low-light image enhancement (LOL-v1 and LOL-v2-real)

[19] and exposure correction (Exposure Errors). Advanced vision tasks include low-light target detection (ExDark)

[20] and Low-light Semantic Segmentation (ACDC)

[21] .

[0074] 1 Low-level visual tasks

[0075] For all low-level vision task experiments, the experimental settings are as follows: the batch size is 8, the Adam optimizer is used, and the initial learning rate is 2e -4 , weight decay is 1e -4 , all training is done on a single GeForce RTX 3090 GPU. The training data is augmented by horizontal and vertical flipping, and a cosine learning strategy is used to prevent overfitting. The evaluation criteria are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0076] 1.1 Low-light image enhancement

[0077] Dataset and implementation details: The LOL dataset contains two versions, LOL-v1 and LOL-v2-real. On the LOL-v1 and LOL-v2-real datasets, the ratio of training set to test set is 485:15 and 689:100 respectively. For the LOL-v1 dataset, the images were first cropped into 256×256 patches and trained for 200 epochs, followed by fine-tuning at 600×400 resolution for 100 epochs. For the LOL-v2-real dataset, the image resolution was kept at 600×400 and trained for 200 epochs. During training, a hybrid loss function was used.

[57] , used to optimize the enhanced image I e and reference image I c The difference between . This loss function combines the smooth L1 loss and the VGG loss

[58] , which is defined as follows:

[0078]

[0079] in, is the smoothed L1 loss, is the VGG loss and λ is set to 0.04.

[0080] Table 1: Quantitative comparison on the LOL (v1, v2-real) dataset. The best and second-best results are marked in bold and italic fonts respectively.

[0081]

[0082]

[0083] Result analysis:

[0084] Quantitative results: The method of the present invention is quantitatively compared with various SOTA enhancement algorithms listed in Table 1. In order to analyze the efficiency of the model, the present invention reports three indicators: FLOPs, model parameters and test time, which are listed in the last column of Table 1. The test time of different models is also listed on the corresponding code platform (M represents Matlab, T represents TensorFlow, and P represents PyTorch). In Table 1, IAT (local) and MIANet (LEP) represent training only by the enhancement subnetwork, while IAT and MIANet represent the entire framework. Compared with the recent SOTA methods FourLLIE and LLFormer, on the LOL-v1 and LOL-v2 datasets, the method of the present invention improves by 3.73dB and 1.57dB, and 0.50dB and 2.21dB, respectively. However, compared with LLFormer, the method of the present invention only uses 0.6% (0.17 / 24.52) of its parameters and 6.1% (1.38 / 22.52) of its computational amount. These results clearly show that the MIANet of the present invention has significant advantages in both image quality and model efficiency.

[0085] Qualitative results: Figure 4 and Figure 5 As shown in Figure 2, MIANet is compared with various algorithms in terms of visual effects. [9] The visual comparison results of the enhanced sub-network on the LOL-v2-real dataset are shown in Figure 6 Please zoom in to get clearer details. Figure 4 Retinexnet and RUAS in the CNN are prone to color distortion or overexposed areas where noise cannot be suppressed. Figure 5 LLFormer in , which easily introduces unnatural artifacts, or Figure 6 The IAT in ,generates blurry images. In contrast, MIANet can better preserve details, adjust image lighting, and accurately restore colors.

[0086] 1.2 Exposure Correction

[0087] Dataset and Implementation Details: Exposure Errors

[12] Contains 24330 images. The training set, validation set, and test set are split in a ratio of 17675:750:5905. The test set has 5 different expert-adjusted results (A / B / C / D / E). The range is from underexposure to overexposure. In Exposure Errors training, according to the settings of Exposure Errors, the training images are cropped into 512×512 patches. When training the Exposure Errors dataset, the loss function is an enhanced image I e and reference image I c The L1 loss function between is defined as follows:

[0088]

[0089] in, is the L1 loss function.

[0090] Table 2: Quantitative comparison on the Exposure Errors dataset. The best and second-best results are marked in bold and italic fonts, respectively.

[0091]

[0092] Result analysis:

[0093] Quantitative results: The correction results of the test images were compared with the standards of five experts. On different test sets, MIANet improved PSNR by 0.69, 0.63, 1.04, 0.58, and 0.67 dB respectively compared to the SOTA method IAT in Table 2, with an average improvement of 0.78 dB. These results clearly show the superior performance of MIANet in image correction under different exposure environments.

[0094] Qualitative results: Since the overexposure phenomenon of the input images in the expert C test set is more serious, the performance of the method of the present invention is evaluated in this overexposure environment. Figure 7 As shown, a visual comparison is made. It can be clearly seen that previous methods, such as MSEC, may over-correct and cause underexposure; while methods like IAT may under-correct, resulting in the exposure problem still existing. Both cases will lead to the loss of image details, and the correction of color brightness will appear unnatural. In contrast, the MIANet of the present invention shows better performance in processing overexposed images. It can retain details and process them properly, so that the image maintains high definition, high color reproduction, and is close to the real scene. In addition, MIANet can naturally adjust the image brightness to avoid the occurrence of too bright or too dark situations.

[0095] 2 High-level visual tasks

[0096] For all high-level vision tasks, images are first processed through MIANet and then the enhanced images are fed into the mmdetection-based

[64] and mmsegmentation

[65] Subsequent recognition algorithm YOLO-v3

[62] and DeepLab-V3+

[66] In order to make a fair comparison, all experiments kept the same settings: including input size, data augmentation method, training cycle and initial weights. The present invention conducted experiments on the original ExDark dataset and the ExDark dataset processed with different enhancement models (including MIANet), used these datasets for YOLOv3 training, and compared their respective training results. The ACDC dataset and DeepLab-V3+ were processed in the same way. In addition, experiments were conducted to jointly train MIANet with YOLO-v3 and DeepLab-V3+ using LOL-v1 pre-trained weights. In order to maintain fairness, only the loss functions of YOLO-v3 and DeepLab-V3+ are still used in joint training.

[0097] 2.1 Low-light object detection

[0098] Dataset and implementation details: ExDark

[20] It includes 7,363 real-world low-light images, covering environments from twilight to extreme darkness, and covering 12 object categories. When training the ExDark dataset, 80% is used for training and 20% is used for testing. The specific division is the same as in the literature. [9,40] After loading the LOL-v1 pre-trained weights for different augmentation models, they are used to augment the ExDark dataset. Then, YOLO-v3 is used to train these augmented datasets and the original ExDark dataset, and the training results are compared. All input images have been cropped and resized to 608×608 pixels. The SGD optimizer is used for training with a batch size of 8 and a total of 25 epochs. The evaluation criteria include mAP and evaluation time per image.

[0099] Table 3: Quantitative comparison on the ExDark dataset. The best and second-best results are marked in bold and italic fonts, respectively.

[0100]

[0101] Result analysis:

[0102] Quantitative results: As shown in Table 3, IAT-YOLO and MIANet-YOLO represent the models jointly trained with IAT and MIANet and YOLO-v3, respectively. Compared with the baseline model and other enhancement methods, MIANet-YOLO performs best in both accuracy and speed. Specifically, MIANet-YOLO ranks first in comprehensive detection performance in 12 categories, and the mAP is improved by 0.4 compared with IAT-YOLO.

[0103] Qualitative results: Figure 8 As shown in the figure, the detection results of YOLO-v3 after using different model methods on the ExDark dataset are visualized. Previous methods either have uneven brightness and unnatural colors, such as Kind, or some details are over-processed, appearing blurred, and the colors appear too a priori, which is unrealistic. The method of the present invention shows significant advantages in brightness adjustment, color restoration and detail retention, and is closer to the real scene, thereby improving the performance of advanced visual tasks. Figure 8 As shown, the image enhanced by the method of the present invention makes the detection of YOLO-v3 more accurate, can locate people more accurately, and effectively avoids misjudging the fishing rod as part of the boat.

[0104] 2.2 Low-light semantic segmentation

[0105] ACDC

[21] The dataset is designed for training and testing semantic segmentation methods under adverse visual conditions (such as fog, night, rain, and snow). This paper extracts a sub-dataset from the ACDC dataset, which contains 1006 night images. Then, using DeepLab-V3+

[66] The model is trained and tested on the training set and validation set of this sub-dataset. The DeepLab-V3+ model is based on the Cityscape dataset.

[67] The pre-trained model is initialized. The SGD optimizer is used to optimize the model, with the batch size set to 8, the number of iterations set to 20,000, the initial learning rate set to 0.05, and the momentum and weight decay set to 0.9 and 5e respectively. -4 In Table 4, the segmentation index mIOU and the evaluation time of each image are shown, where * represents the joint training model. As shown in Table 4, the method of the present invention is 0.6 higher than IAT* in mIOU.

[0106] Table 4: Quantitative comparison on the ACDC dataset. The best and second-best results are marked in bold and italic fonts, respectively. * indicates the joint training model.

[0107]

[0108] 3 Ablation test

[0109] In order to evaluate the contribution of each part of the MIANet model, LOL-V2-real

[19] The dataset was subjected to an ablation study. The results are shown in Table 5, where LP represents the Laplacian pyramid, ILCS represents the inter-layer communication structure, SG represents the single gamma correction parameter, and MG represents the multi-channel gamma correction parameter. The PSNR, SSIM, Params, and FLOPs results of the enhancement task are listed below.

[0110] Table 5: Experiments on the LOL-V2-real dataset show the contribution of each part to MIANet

[0111]

[0112]

[0113] Result analysis:

[0114] LEP: As shown in Table 5(a), when only LP is used to reconstruct the image, the model does not have the ability to train and optimize. However, after the introduction of LL-FSA, as shown in (b), the model not only has the ability to train, but also effectively enhances the components of different frequencies, so that the model achieves a performance level of 16.94dB in the PSNR indicator. Then, based on (b), CMSA and LCS are introduced in turn, and the model achieves performance improvements of 3.17dB and 1.94dB respectively. This shows that CMSA and LCS techniques successfully promote the fusion and enhancement of different frequency components, and by introducing the attention mechanism, the model focuses more on key features.

[0115] MIAT: Based on Table 5(d), (e) and (f) are compared. By using IS-MA to query the gamma correction parameters of multiple channels instead of just querying a single gamma correction parameter, the present invention further obtains a 0.12dB performance improvement on the PSNR indicator. On this basis, CMSA and Value priors are used to assist the query process of the color conversion matrix and gamma correction parameters, further improving the performance by 0.17dB. It should be noted that although MG and Value priors do not significantly improve the performance of the model, they effectively improve the generalization ability of the model. Figure 8 As shown, when applied to untrained datasets, the MIANet of the present invention performs better in detail preservation, brightness adjustment, and color restoration, and can better simulate real scenes.

[0116] In summary, the present invention proposes a multi-channel illumination adaptive network model with Laplacian enhancement pyramid. MIANet optimizes frequency information through LEP to highlight the details in the image, such as edges and textures, while effectively suppressing noise and appropriately adjusting the overall brightness of the image. Subsequently, through the Multi-channel Illumination Adaptive Transformer (MIAT), MIANet can effectively query the color transformation matrix and multi-channel gamma correction parameters to further correct the image and finally produce an enhanced image. MIANet not only shows excellent performance on multiple low-level and high-level tasks, but also has the characteristics of being lightweight and efficient.

[0117] The data sets, models / methods mentioned in this invention come from:

[0118] [1]Guo C, Li C, Guo J, et al. Zero-reference deep curve estimation for low-light image enhancement, Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition, pp.1780-1789(2020)

[0119] [2]Guo X, Li Y, Ling H.LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on image processing 26(2):982-993(2016)

[0121] [3]Lore KG,Akintayo A,Sarkar S.LLNet:A deep autoencoder approach tonatural low-light image enhancement.Pattern Recognition 61:650-662(2017)

[0123] [4]Lv F, Lu F, Wu J, et al. MBLLEN: Low-light image / video enhancement using CNNs, BMVC, Northumbria University: p.4 (2018)

[0125] [5]Moran S,Marza P,McDonagh S,et al.Deeplpf:Deep local parametricfilters for image enhancement,Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition,pp.12826-12835(2020)

[0127] [6]Tu Z,Talebi H,Zhang H,et al.Maxim:Multi-axis mlp for imageprocessing,Proceedings of the IEEE / CVFconference on computer vision andpattern recognition,pp.5769-5780(2022)

[0129] [7]Zhang Y,Zhang J,Guo X Kindling the darkness:A practical low-lightimage enhancer,Proceedings of the 27thACM international conference onmultimedia,pp.1632-1640(2019)

[0131] [8]Zheng N,Zhou M,Dong Y,et al.Empowering low-light image enhancerthrough customized learnable priors,Proceedings of the IEEE / CVF InternationalConference on Computer Vision,pp.12559-12569(2023)

[0133] [9]Cui Z,Li K,Gu L,et al.You only need 90k parameters to adapt light:alight weight transformer for image enhancement and exposure correction.arXivpreprint arXiv:220514871(2022)

[0135]

[10] Wang T,Zhang K,Shen T,et al.Ultra-high-definition low-light imageenhancement:A benchmark andtransformer-based method,Proceedings of the AAAIConference on Artificial Intelligence,pp.2654-2662(2023)

[11] Wang C,Wu H,Jin ZFourllie:Boosting low-light image enhancement by fourier frequencyinformation,Proceedings of the 31st ACM International Conference onMultimedia,pp.7459-7469(2023)

[0138]

[12] Afifi M,Derpanis K G,Ommer B,et al.Learning multi-scale photoexposure correction,Proceedings of theIEEE / CVF Conference on Computer Visionand Pattern Recognition,pp.9157-9167(2021)

[0140]

[13] Nayar,Branzoi Adaptive dynamic range imaging:Optical control ofpixel exposures over space and time,Proceedings Ninth IEEE InternationalConference on Computer Vision,IEEE:pp.1168-1175vol.1162(2003)

[0142]

[14] Yuan L,Sun J Automatic exposure correction of consumerphotographs,Computer Vision–ECCV 2012:12thEuropean Conference on ComputerVision,Florence,Italy,October 7-13,2012,Proceedings,Part IV 12,Springer:pp.771-785(2012)

[0145]

[15] Yao M,Huang J,Jin X,et al.Generalized lightness adaptation withchannel selective normalization,Proceedings of the IEEE / CVF InternationalConference on Computer Vision,pp.10668-10679(2023)

[0147]

[16] Song Y,Cao Z,Xiang W,et al.Troublemaker Learning for Low-LightImage Enhancement.arXiv preprint arXiv:240204584(2024)

[0149]

[17] Burt P J,Adelson E H.The Laplacian pyramid as a compact imagecode[M].Readings in computer vision.Elsevier.1987:671-679.

[0151]

[18] Munsell AH.A color notation[M].Munsell color company,1919.

[0152]

[19] Wei C,Wang W,Yang W,et al.Deep retinex decomposition for low-light enhancement.arXiv preprint arXiv:180804560(2018)

[0154]

[20] Loh Y P,Chan C S.Getting to know low-light images with theexclusively dark dataset.Computer Vision and Image Understanding 178:30-42(2019)

[0156]

[21] Sakaridis C,Dai D,Van Gool L ACDC:The adverse conditions datasetwith correspondences for semantic driving scene understanding,Proceedings ofthe IEEE / CVF International Conference on Computer Vision,pp.10765-10775(2021)

[0157]

[22] Zamir S W,Arora A,Khan S,et al.Restormer:Efficient transformerfor high-resolution image restoration,Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition,pp.5728-5739(2022)

[0158]

[23] Abdullah-Al-Wadud M,Kabir M H,Dewan M AA,et al.A dynamichistogram equalization for image contrast enhancement.IEEE transactions onconsumer electronics 53(2):593-600(2007)

[0159]

[24] Celik T,Tjahjadi T.Contextual and variational contrastenhancement.IEEE Transactions on Image Processing20(12):3431-3441(2011)

[0160]

[25] Huang S-C,Cheng F-C,Chiu Y-S.Efficient contrast enhancement usingadaptive gamma correction with weighting distribution.IEEE transactions onimage processing 22(3):1032-1041(2012)

[0161]

[26] Rahman S,Rahman M M,Abdullah-Al-Wadud M,et al.An adaptive gammacorrection for image enhancement.EURASIP Journal on Image and VideoProcessing 2016:1-13(2016)

[0162]

[27] Land E H.The retinex theory of color vision.Scientific american237(6):108-129(1977)

[0163]

[28] Fu X,Zeng D,Huang Y,et al.A weighted variational model forsimultaneous reflectance and illumination estimation,Proceedings of the IEEEconference on computer vision and pattern recognition,pp.2782-2790(2016)

[0164]

[29] Li M,Liu J,Yang W,et al.Structure-revealing low-light imageenhancement via robust retinex model.IEEE Transactions on Image Processing 27(6):2828-2841(2018)

[0165]

[30] Jiang Y,Gong X,Liu D,et al.Enlightengan:Deep light enhancementwithout paired supervision.IEEE transactions on image processing 30:2340-2349(2021)

[0166]

[31] Wang R,Zhang Q,Fu C-W,et al.Underexposed photo enhancement usingdeep illumination estimation,Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition,pp.6849-6857(2019)

[0167]

[32] Zamir S W,Arora A,Khan S,et al.Learning enriched features forreal image restoration and enhancement,Computer Vision–ECCV 2020:16thEuropean Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XXV 16,Springer:pp.492-511(2020)

[0168]

[33] Park J,Lee J-Y,Yoo D,et al.Distort-and-recover:Color enhancementusing deep reinforcement learning,Proceedings of the IEEE conference oncomputer vision and pattern recognition,pp.5928-5936(2018)

[0169]

[34] Huang J,Liu Y,Zhao F,et al.Deep fourier-based exposure correctionnetwork with spatial-frequency interaction,European Conference on ComputerVision,Springer:pp.163-180(2022)

[0170]

[35] Huang J,Liu Y,Fu X,et al.Exposure normalization and compensationfor multiple-exposure correction,Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,pp.6043-6052(2022)

[0171]

[36] Yang K-F,Cheng C,Zhao S-X,et al.Learning to adapt tolight.International Journal of Computer Vision131(4):1022-1041(2023)

[0172]

[37] Jin Y,Yang W,Tan R T Unsupervised night image enhancement:Whenlayer decomposition meets light-effects suppression,European Conference onComputer Vision,Springer:pp.404-421(2022)

[0173]

[38] Song M,Lim S,Kim W.Monocular depth estimation using laplacianpyramid-based depth residuals.IEEE transactions on circuits and systems forvideo technology 31(11):4381-4393(2021)

[0174]

[39] Qin Q,Chang K,Huang M,et al.DENet:detection-driven enhancementnetwork for object detection under adverse weather conditions,Proceedings ofthe Asian Conference on Computer Vision,pp.2813-2829(2022)

[0175]

[40] Yin X,Yu Z,Fei Z,et al.Pe-yolo:Pyramid enhancement network fordark object detection,International Conference on Artificial Neural Networks,Springer:pp.163-174(2023)

[0176]

[41] Vaswani A,Shazeer N,Parmar N,et al.Attention is all youneed.Advances in neural information processing systems 30(2017)

[0177]

[42] Dosovitskiy A,Beyer L,Kolesnikov A,et al.An image is worth 16x16words:Transformers for image recognition at scale.arXiv preprint arXiv:201011929(2020)

[0178]

[43] Carion N,Massa F,Synnaeve G,et al.End-to-end object detectionwith transformers,European conference on computer vision,Springer:pp.213-229(2020)

[0180]

[44] Xie E,Wang W,Yu Z,et al.SegFormer:Simple and efficient design forsemantic segmentation with transformers. Advances in neural informationprocessing systems 34:12077-12090(2021)

[0182]

[45] Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows,Proceedings of the IEEE / CVF internationalconference on computer vision,pp.10012-10022(2021)

[0184]

[46] Cao H,Wang Y,Chen J,et al.Swin-unet:Unet-like pure transformerfor medical image segmentation,European conference on computer vision,Springer:pp.205-218(2022)

[0186]

[47] Dong X,Bao J,Chen D,et al.Cswin transformer:A general visiontransformer backbone with cross-shaped windows,Proceedings of the IEEE / CVFconference on computer vision and pattern recognition,pp.12124-12134(2022)

[0189]

[48] Chollet F Xception:Deep learning with depthwise separableconvolutions,Proceedings of the IEEE conference on computer vision andpattern recognition,pp.1251-1258(2017)

[0190]

[49] Li X,Wang W,Hu X,et al.Selective kernel networks,Proceedings ofthe IEEE / CVF conference on computer vision and pattern recognition,pp.510-519(2019)

[0192]

[50] Liu Z,Mao H,Wu C-Y,et al.A convnet for the 2020s,Proceedings ofthe IEEE / CVF conference on computer vision and pattern recognition,pp.11976-11986(2022)

[0194]

[51] Szegedy C,Vanhoucke V,Ioffe S,et al.Rethinking the inceptionarchitecture for computer vision,Proceedings of the IEEE conference oncomputer vision and pattern recognition,pp.2818-2826(2016)

[0196]

[52] Yang W,Wang S,Fang Y,et al.From fidelity to perceptual quality:Asemi-supervised approach for low-light image enhancement,Proceedings of theIEEE / CVF conference on computer vision and pattern recognition,pp.3063-3072(2020)

[0199]

[53] Zeng H,Cai J,Li L,et al.Learning image-adaptive 3d lookup tablesfor high performance photo enhancement in real-time.IEEE Transactions onPattern Analysis and Machine Intelligence 44(4):2058-2073(2020)

[0201]

[54] Wang Z,Cun X,Bao J,et al.Uformer:A general u-shaped transformerfor image restoration,Proceedings of the IEEE / CVF conference on computervision and pattern recognition,pp.17683-17693(2022)

[0203]

[55] Chen H,Wang Y,Guo T,et al.Pre-trained image processingtransformer,Proceedings of the IEEE / CVF conference on computer vision andpattern recognition,pp.12299-12310(2021)

[0205]

[56] Kim H,Choi S-M,Kim C-S,et al.Representative color transform forimage enhancement,Proceedings of the IEEE / CVF international conference oncomputer vision,pp.4459-4468(2021)

[0207]

[57] Valanarasu J M J,Yasarla R,Patel V M Transweather:Transformer-based restoration of images degraded by adverse weather conditions,Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,pp.2353-2363(2022)

[0210]

[58] Johnson J,Alahi A,Fei-Fei L Perceptual losses for real-time styletransfer and super-resolution,Computer Vision–ECCV 2016:14th EuropeanConference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part II14,Springer:pp.694-711(2016)

[0213]

[59] B.Digital image processing[M].Springer Science&BusinessMedia,2005.

[0214]

[60] Ignatov A,Kobyshev N,Timofte R,et al.Dslr-quality photos onmobile devices with deep convolutional networks,Proceedings of the IEEEinternational conference on computer vision,pp.3277-3285(2017)

[0216]

[61] Chen Y-S,Wang Y-C,Kao M-H,et al.Deep photo enhancer:Unpairedlearning for image enhancement from photographs with gans,Proceedings of theIEEE conference on computer vision and pattern recognition,pp.6306-6314(2018)

[0219]

[62] Redmon J,Farhadi A.Yolov3:An incremental improvement.arXivpreprint arXiv:180402767(2018)

[0220]

[63] Cui Z,Qi G-J,Gu L,et al.Multitask aet with orthogonal tangentregularity for dark object detection,Proceedings of the IEEE / CVFinternational conference on computer vision,pp.2553-2562(2021)

[0221]

[64] Chen K,Wang J,Pang J,et al.MMDetection:Open mmlab detectiontoolbox and benchmark.arXiv preprint arXiv:190607155(2019)

[0222]

[65] Contributors M.MMSegmentation:Openmmlab semantic segmentationtoolbox and benchmark[Z].2020

[0223]

[66] Chen L-C,Zhu Y,Papandreou G,et al.Encoder-decoder with atrousseparable convolution for semantic image segmentation,Proceedings of theEuropean conference on computer vision(ECCV),pp.801-818(2018)

[0224]

[67] Cordts M,Omran M,Ramos S,et al.The cityscapes dataset forsemantic urban scene understanding,Proceedings of the IEEE conference oncomputer vision and pattern recognition,pp.3213-3223(2016)。

Claims

1. A multi-channel illumination adaptive network model with Laplace enhanced pyramid, characterized in that: Includes Laplacian Enhanced Pyramid (LEP) module and Multi-channel Illumination Adaptive Transformer (MIAT) module; In LEP, the image is first decomposed into low-frequency and multiple high-frequency components through the Laplacian pyramid prior; then, these components of different frequencies are fused and optimized through a multi-scale structure and channel multi-head self-attention (CMSA); finally, the enhanced components are fused and denoised through Gaussian convolution to reconstruct a preliminary enhanced image; In MIAT, the Value prior is first used to more efficiently query color and multi-channel gamma correction parameters; then, CMSA is used to reweight channel features so that the network can prioritize key information. Finally, illumination space multi-head attention (IS-MA) is designed and integrated into MIAT to capture global dependencies in image space, extract color and multi-channel gamma correction parameters, and further optimize the image to generate enhanced images.

2. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 1, characterized in that: In the Laplacian Enhanced Pyramid (LEP) module, the image is first decomposed using the Laplacian pyramid. Decompose into multiple frequency components of different resolutions, denoted as F, including a low-frequency component L and multiple high-frequency components H; the decomposition process of LEP (1≤i≤N) is defined as follows: G0=I0 (1) G i+1 =Down2(G i *Gaussian) (2) H i =G i -Gaussian(Up2(G i+1 )) (3) Among them, I0 is the input image, G0 is the 0th layer Gaussian image; G i is the Gaussian image of the i-th layer, Down z represents double downsampling, Gaussian represents Gaussian convolution of size 5×5; H i is the Laplacian image of the i-th layer, and Up2 means double upsampling; According to formula (2), the Gaussian image G at the bottom layer N As the low-frequency component with the lowest resolution, it is denoted as L; L retains the most global low-frequency information, reflecting the large-scale structure of the image and the brightness and color distribution of a large area; according to formula (3), each high-frequency component is denoted as H i , which means from the i-th layer Gaussian image G i To the i+1th layer Gaussian image G i+1 These high-frequency components capture the image details at different resolutions, but also contain a certain degree of noise; LEP can effectively eliminate this noise when reconstructing the image.

3. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 2, characterized in that: The inter-layer communication structure is used to enhance and fuse L and H to obtain the enhanced components, which are recorded as Includes an enhanced low frequency component and multiple enhanced high frequency components The enhanced components not only optimize the frequency information, but also decouple the brightness and color into different components, which helps to adjust the brightness and color of the image; finally, the enhanced components are reconstructed using recursive addition, Gaussian convolution, and upsampling operations; Gaussian convolution can effectively reduce the noise in the high-frequency components by smoothing the image, which helps to suppress random noise and preserve the basic structure; the reconstruction process of LEP (0≤i≤N-1) is defined as follows: in, is the high-frequency component after the i-th layer enhancement, G N is the low-frequency component after the Nth layer enhancement, G i is the Gaussian image after the i-th layer enhancement and reconstruction, Up2 is double upsampling, and Gaussian represents a Gaussian convolution of size 5×5.

4. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 3, characterized in that: The local spatial attention (LSA) is used to enhance the low-frequency component L, and the depth-separable convolution and its inverse operation are used to achieve feature dimension transformation; in order to promote the direct transmission of information, a jump connection is added between the output and input of FF-LSA, and the deep selective kernel attention (DSKA) is proposed; DSKA first captures the information of different receptive fields through a multi-scale deep convolution structure while maintaining low computational complexity; then, the multi-scale feature map is element-wise added, and the selection weights are generated through global average pooling and fully connected layers; finally, the selection weights are normalized by softmax, and then weighted summed with the corresponding feature map to obtain the final output feature map; DSKA is similar to the local spatial attention mechanism because they can both focus on different areas of the image and enhance feature representation; the calculation process of DSKA is defined as follows: F k×k =DWC k×k (F in ) (5) F sum =FC d (GP(F in +F 3×3 +F 5×5 +F 7×7 )) (6) in, is the input feature map, DWC k×k is a depthwise convolution with a kernel size of k×k, F k×k Yes F in By DWC k×k The processed feature map; GP is global average pooling, FC d is a fully connected layer with output dimension d, F sum is the selection weight; FC c is a fully connected layer with an output dimension of c, softmax is the softmax activation function, concat is the concatenation operation, and Z k×k is the weighted feature, and ⊙ is the element-by-element multiplication.

5. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 4, characterized in that: By applying channel multi-head self-attention (CMSA), the weights of each channel can be adaptively adjusted, thereby helping the network better understand and process the relationship between these features and improve image quality. When using CMSA, there are three design points to note: (i) Add a long-distance jump connection between the input of the first CMSA in each layer of the pyramid and the output of the last CMSA; (ii) Add the enhanced components in each layer of the pyramid. Pass them to the FF-LSA of higher resolution layers, and then process them through CMSA; (iii) do not use CMSA for H1; Feature Map is defined as input and converted into tokens Then the last dimension is divided into h parts, that is, h heads, expressed as in, The self-attention calculation process of each head in CMSA is defined as follows: Among them, X i is the input element of the i-th head, Q i , K i 、V i They are the query element, key element, and value element of the i-th head respectively; Both represent the matrix transpose of the fully connected layer. represents matrix multiplication, r i is a learnable parameter used for adaptively scaling matrix multiplication; After calculating the self-attention, a feed-forward network (FFN) is needed to enrich and reconstruct the feature representation; the self-attention outputs of h heads are concatenated to obtain the feature map A∈R hw×c , transforming it into And defined as input, the calculation process of FFN is defined as follows: f out =PWC c (DWC 4c (PWC 4c (LN(X out ))))+X out (11) Among them, LN stands for layer normalization, PWC 4c It is a point-by-point convolution with an output dimension of 4c, DWC 4c is a depthwise convolution with an output dimension of 4c, PWC c is a point-wise convolution with output dimension c, f out is the output of FFN; The spatial attention of FF-LSA and the channel attention of CMSA cooperate with each other to maximize the enhancement and optimization of frequency information in LEP.

6. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 5, characterized in that: MIAT uses the Value prior in the HSV color space to assist MIAT in better querying the correction parameters; Value represents the brightness of the color and plays a key role in adjusting the color brightness; the calculation process of Value is defined as follows: V=max(R,G,B) (12) Among them, R, G, and B represent the red, green, and blue components in the RGB color space respectively; MIAT first uses ordinary convolution and depthwise convolution to reduce the number of parameters and the size of the feature map, and then uses CMSA to adjust the weights of each channel to more efficiently query the color transformation matrix in subsequent steps.

7. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 6, characterized in that: Illumination Spatial Multi-Head Attention (IS-MA) extracts a 3×3 color transformation matrix and 3 gamma correction parameters from the image through 12 global component queries. These three gamma parameters will act on the R, G, and B channels of the image respectively. Unlike CMSA, in IS-MA, each query Q is initialized as a learnable all-1 matrix. In addition, during training, IS-MA adds a unit matrix and an all-1 vector to the color transformation matrix and gamma correction parameters respectively to stabilize training. IS-MA operates on the feature map of the input image after 4 times downsampling, and each feature map pixel corresponds to a specific area of ​​the input image; during the training phase, each query will globally interact with all areas of the image; in addition, the query can be dynamically updated in each iteration to better match the reference image.

8. The multi-channel illumination adaptive network model with Laplace enhanced pyramid as claimed in claim 7, characterized in that: In order to avoid generating too many white and black pixels, the scalar value of each channel of each pixel in the image processed by the color transformation matrix is ​​restricted to a range between ε and 1-ε, where ε = 1e -8 ; This limitation is because a scalar value of 0 or 1 will cause subsequent gamma correction to fail; The output image of LEP As input, the calibration process of MIAT is as follows: in, is the color transformation matrix, is matrix multiplication, ε = 1e -8 ;I R ,I G ,I B Represent the R, G, and B channels of image I, respectively. R , γ G , γ B Represent the γ correction parameters of the three channels of R, G, and B respectively; finally, the enhanced images of the three channels are merged in the channel dimension to obtain the final enhanced image; the merging process is defined as follows: I e =(I R ,I G ,I B ) T (15)。 9. A low-light image enhancement method based on a multi-channel illumination adaptive network model with a Laplacian enhancement pyramid as claimed in any one of claims 1 to 8, characterized in that: Firstly, LEP enhances and optimizes the frequency information of the image through feature fusion local spatial attention (FF-LSA) and channel multi-head self-attention (CMSA), thereby generating a preliminary enhanced image; then, MIANet effectively queries the color transformation matrix and multi-channel gamma correction parameters of the input image through channel multi-head self-attention (CMSA) and illumination space multi-head attention (IS-MA), further corrects the preliminary enhanced image, and finally outputs the enhanced image.

Images