Transform-based weak light image enhancement method

Abstract

The invention discloses a weak light image enhancement method based on Transform. The method comprises the following steps: extracting local detail information of an input weak light image through a convolutional layer, and converting the local detail information into a local feature map with a specified size; inputting the local feature map into three continuous feature extraction modules constructed based on 3D multi-head attention to extract deeper global features; fusing the three global features through a multi-layer fusion module to generate a final enhanced feature map; the multi-layer fusion module fuses features from different layers by adopting an adaptive weighting method; and the enhanced feature map passes through a convolutional layer and a Sigmoid activation function to obtain a final enhanced image. According to the invention, the quality of an image shot under a low-light condition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a weak light image enhancement method based on Transformer. BACKGROUND

[0002] With the popularity of intelligent devices and the continuous development of science and technology, the way to obtain multimedia data has become simpler. However, in actual application, due to various factors such as devices, environment, improper human operation, etc., it is not easy to obtain high-quality images. Especially in weak light environment, the images taken often have problems such as low brightness, insufficient contrast, dull color, unclear details, and obvious noise. These problems not only affect the visual effect of the image, but also reduce the performance of subsequent advanced visual tasks. Therefore, the weak light image enhancement technology emerges as the times require, aiming to improve the imaging quality of the image, improve the visual effect, and improve the performance of downstream tasks.

[0003] With the rapid development of deep learning, convolutional neural networks (CNN) have been widely used in low-light image enhancement. RetinexNet is a low-light image enhancement model based on Retinex theory, composed of Decom-Net (for decomposing light and reflection images) and Enhance-Net (for light adjustment). EnlightenGAN is an efficient unsupervised generative adversarial network that can be trained without paired low-light / normal-light images. Although these methods can generate high-quality normal-light images, they may still introduce noise during the training phase of low-light image object detection, resulting in performance that is not as good as training directly on low-light images.

[0004] In existing weak light image enhancement methods, there are generally the following defects: (1) methods based on Retinex theory can cause noise and local distortion in the restored image. (2) methods based on convolutional neural networks can only capture local feature information, and the quality of the restored image is low. (3) ViT causes a large amount of computation in the weak light image enhancement task, which cannot be supported by general devices. SUMMARY

[0005] The purpose of the present application is to provide a weak light image enhancement method based on Transformer, the feature extraction module is composed of convolutional neural network and Transformer, and the computational complexity of multi-head attention in Transformer will increase linearly with the input image size.

[0006] The technical scheme adopted by the present application is:

[0007] The weak light image enhancement method based on Transformer comprises the following steps:

[0008] Step 1: The input low-light image is processed through a 3×3 convolutional layer for preliminary feature extraction; the convolutional layer captures local detail information of the image and converts the image into a local feature map of a specified size;

[0009] Step 2: Construct three consecutive feature extraction modules based on 3D multi-head attention. Input the local feature map into the three consecutive feature extraction modules to extract deeper global features. The feature extraction module (CTBlock) is a feature extraction module that combines CNN and Transformer.

[0010] Step 3: Construct a multi-layer fusion module based on depthwise separable convolutions of different sizes. The three global features are fused through a multi-layer fusion module to generate the final enhanced feature map. The multi-layer fusion module uses an adaptive weighting method to fuse features from different layers. The multi-layer fusion module is a multi-level fusion module used to fuse features from other layers to achieve efficient image restoration with minimal computational overhead.

[0011] Step 4: The enhanced feature map is passed through a convolutional layer and a sigmoid activation function to obtain the final enhanced image.

[0012] Specifically, since the pixel values ​​of the input image are between 0 and 1, the Sigmoid activation function ensures that the pixel values ​​of the output image remain within the same range, thereby maintaining numerical consistency and accelerating the convergence of the model during training.

[0013] Further, in step 1, the feature map size is C×H×W; H (height) represents the height of the feature map; W (width) represents the width of the feature map; C (number of channels) represents the number of channels in the feature map, usually used to represent color channels or feature channels. For example, for a tensor of shape (8,3,64,64), 8 represents the batch size, 3 represents the number of channels, 64 represents the height, and 64 represents the width.

[0014] Furthermore, the feature extraction module in step 2 performs the following steps:

[0015] Step 2-1: A local feature map of size C×H×W is input into a 3×3 convolutional layer to effectively extract features while preserving the spatial dimensions of the feature map to obtain the first feature map.

[0016] Step 2-2: The output first feature map is passed through a 1×1 convolutional layer to adjust the number of channels to obtain the second feature map, thereby reducing computational complexity and avoiding the computational overhead of using 3×3 convolution.

[0017] Steps 2-3 involve extracting global features from the second feature map using 3D multi-head attention, and then concatenating the global features with the local features obtained through the convolutional layer along the channel dimension to generate a third feature map containing more comprehensive information. The reason for not concatenating it with the input features is that the relationship between local and global features is more tightly coupled. 3D multi-head attention performs multi-head attention calculations along the height, width, and channel dimensions respectively, effectively capturing global features.

[0018] Steps 2-4 involve using a 1×1 convolution to compress the concatenated third feature map back to the original number of channels C to obtain the global features. Residual connections preserve the initial feature information, which is then fused with the enhanced features to generate a rich and robust output feature map.

[0019] Furthermore, in steps 2-3, the 3D multi-head attention performs the following steps:

[0020] Step 2-3-1: The second feature map integrates information from different channels through a 1×1 convolution to enhance the input features and increase the number of channels to 3C;

[0021] Step 2-3-2: Use a 3×3 depthwise convolution to extract feature maps rich in local information.

[0022] Step 2-3-3 involves decomposing the feature map into query (Q), key (K), and value (V) using a 3×3 depthwise convolution, and then unfolding each into 2×2 patches. Pixels at the same position within four pixels of each patch are grouped together to form four groups. In this way, Q, K, and V are divided into... and The query (Q), key (K), and value (V) are defined as follows: and Where W1 and W2 represent 3×3 depthwise convolutions, W 1×1 This represents a 1×1 convolution. and These represent the 3×3 depthwise convolution kernels used to generate the query (Q), key (K), and value (V) matrices, respectively. Their function is to extract local features and weight the features at each position.

[0023] Specifically, before performing multi-head attention, Q, K, and V need to be unfolded. Because adjacent pixels in low-light images are often very similar and carry redundant feature information, this unfolding operation helps reduce computational redundancy. Specifically, the feature map is divided into 2×2 patches, each containing four pixels. Pixels at the same location within each patch are grouped together, forming four groups. In this way, Q, K, and V are divided into... and Each group of Q, K, and V will undergo multi-head self-attention computation in one dimension of 3D multi-head attention.

[0024] In steps 2-3-4, for each set of query (Q), key (K), and value (V), multi-head attention is performed along the height, width, and channel dimensions to obtain the feature map of the corresponding dimension, so as to generate global features.

[0025] Furthermore, the multi-head attention calculation steps for each dimension in steps 2-3-4 are as follows:

[0026] S1, the self-attention representation of each head (headj) in the group is obtained as follows:

[0027]

[0028] in These are the elements after partitioning the Q matrix, representing the query feature in the i-th row and j-th column of the image patch. These are the elements after partitioning the K matrix, representing the key features of the image patch in the i-th row and j-th column. These are the elements after partitioning the V matrix, representing the value feature of the image patch in the i-th row and j-th column. k represents the number of heads; each head calculates an independent attention score, and the final output is a concatenated set of features; d k Indicates the dimension of the corresponding vector;

[0029] S2, calculate the output feature map for each group, as follows:

[0030]

[0031] Here, Concat represents the concatenation operation. This represents the feature map obtained from the i-th group.

[0032] S3, the output feature maps of each group are merged to obtain the original-sized output map X′, represented as:

[0033]

[0034] Here, Fold represents the folding operation, which can merge the output feature maps of each group into a feature map of the original size C×H×W.

[0035] S4, the original output image is convolved with a 1×1 convolution to obtain the feature map of the corresponding dimension.

[0036] Furthermore, in step 3, the multi-layer fusion module performs the following steps:

[0037] Step 3-1, three consecutive global features A single-layer feature map of size 3C×H×W is generated by splicing along the channel dimension.

[0038] Step 3-2: Use a 1×1 convolution to perform cross-channel information aggregation on the 3C×H×W feature map, and after processing based on the features of each pixel, divide it into three groups and capture spatial information at different scales by using 3×3, 5×5 and 7×7 depth-separable convolutions respectively.

[0039] Step 3-3: The output features of convolutions at different scales are concatenated again along the channel dimension to form a secondary concatenated feature map with a size of 3C×H×W;

[0040] Steps 3-4: The secondary concatenated feature map is fused by a 1×1 convolution to obtain a fused feature map.

[0041] Steps 3-5 involve adding the fused feature map to the sequentially concatenated feature map element by element to re-integrate the feature information, and then using a 1×1 convolution to restore the feature map to the C×H×W dimension to generate the final enhanced feature map.

[0042] Specifically, the multi-layer fusion module fully utilizes the advantages of multi-scale convolution to significantly improve the visual quality of low-light images through multi-level feature fusion, making the enhanced images more natural and with higher detail fidelity.

[0043] Furthermore, as a feasible implementation, the enhanced image in step 4 is input into a pre-trained object detection model (e.g., YOLOv8s) to detect the image enhancement effect.

[0044] Furthermore, as a feasible implementation, step 5 is also included: building a neural network based on the Transformer-based low-light image enhancement method, and training the neural network with the total loss function constructed based on the smoothing loss and structural similarity loss to obtain the image enhancement model.

[0045] Furthermore, step 5 specifically includes the following:

[0046] Step 5-1: Calculate the smoothing loss L for obtaining the low-light enhanced image and the standard illumination image. smooth The corresponding expression is:

[0047]

[0048] Among them, I p and I n These represent low-light enhanced images and standard lighting images, respectively.

[0049] Step 5-2, calculate and obtain the structural similarity score (SSIM). The calculation expression is:

[0050]

[0051] Where, μ p and μ n These are the mean values ​​of the low-light enhanced image and the standard illumination image, respectively, σ p and σ n It is the variance, σ pn This is the covariance, where C1 and C2 are constants, typically set to 0.01. 2 and 0.03 2 .

[0052] Step 5-3: Calculate and obtain the structural similarity loss (SSIM loss), the corresponding expression is:

[0053] L ssim =1-SSIM(I p ,I n (4.6)

[0054] Among them, L ssim Represents structural similarity loss;

[0055] Step 5-4: Combine the smoothing loss and structural similarity loss to obtain the total loss function, which is then used as the final optimization function. The specific expression is as follows:

[0056] L total =L smooth +L ssim (4.7)

[0057] Among them, L total This indicates the total loss.

[0058] This invention, employing the above technical solutions, possesses the following technical advantages: 1. Solving the problem of poor image quality in low-light conditions: In low-light environments, images often suffer from insufficient brightness, significant noise, and blurred details, greatly increasing the visibility and recognition difficulty of targets. The purpose of this invention is to effectively improve image quality and restore detail information in the image through low-light enhancement methods, thereby providing clearer image input for subsequent target detection. 2. Reducing noise and blur: Low-light images are often accompanied by strong noise and blur, affecting the performance of target detection models. Traditional enhancement methods may not be able to effectively reduce noise while preserving image details. The TLLE model, by combining the advantages of convolutional neural networks (CNN) and Transformers, can more accurately reduce noise and blur while improving image quality, enabling target detection models to better identify targets in noisy and blurry low-light images. 3. Improving the contrast between objects and background: In low-light environments, the contrast between objects and background is usually low, making it difficult for target detection models to distinguish between targets and background. The TLLE model improves image contrast through effective enhancement methods, making objects stand out more in the image and improving the recognition ability of the detection model under low-light conditions. 4. Improve the recognition rate and localization accuracy of the target detection model: Low-light environments reduce the recognition rate and localization accuracy of target detection models, especially in complex scenes where targets are difficult to identify and locate accurately. The TLLE model of this invention improves the visibility of targets by enhancing the brightness, clarity, and contrast of the image, thereby improving the recognition rate and localization accuracy of the target detection model under low-light conditions. 5. Reduce the computational complexity introduced by Transformer: A 3D multi-head attention mechanism is designed, performing multi-head attention calculations on the height, width, and channels of the feature map separately, making the computational load increase linearly with the size of the input image.

[0059] This invention significantly improves image brightness and contrast through a series of algorithms and methods, effectively reducing noise and distortion, making details that were previously difficult to discern in low-light environments clearly visible. Whether it's a photograph taken at night, a poorly lit indoor scene, or a dark image caused by improper camera exposure settings, low-light image enhancement technology plays a crucial role, providing a more reliable and high-quality image foundation for image recognition, analysis, processing, and subsequent applications. This invention improves the image quality captured under low-light conditions. Attached Figure Description

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0061] Fig. 1 This is a schematic diagram illustrating the principle architecture of the low-light image enhancement method based on Transformer of the present invention;

[0062] Fig. 2 This is a schematic diagram of the 3D multi-head attention structure of the present invention;

[0063] Fig. 3 This is a schematic diagram of the structure of the multi-level fusion module MFBlock of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0065] like Figs. 1 to 3 As shown in one example, this invention discloses a Transformer-based low-light image enhancement method (TLLE), which includes the following steps:

[0066] Step 1: The input low-light image is processed through a 3×3 convolutional layer for preliminary feature extraction; the convolutional layer captures local detail information of the image and converts the image into a local feature map of a specified size;

[0067] Step 2: Construct three consecutive feature extraction modules based on 3D multi-head attention. Input the local feature map into the three consecutive feature extraction modules to extract deeper global features. The feature extraction module is a feature extraction module that combines CNN and Transformer.

[0068] Step 3: The three global features output by the three feature extraction modules. The final enhanced feature map is generated through a multi-layer fusion module. This module employs an adaptive weighting method to effectively fuse features from different layers. The multi-layer fusion module MFBlock is used to fuse features from other layers, achieving efficient image restoration with minimal computational overhead.

[0069] Step 4: The enhanced feature map is passed through a convolutional layer and a sigmoid activation function to obtain the final enhanced image.

[0070] Specifically, since the pixel values ​​of the input image are between 0 and 1, the Sigmoid activation function ensures that the pixel values ​​of the output image remain within the same range, thereby maintaining numerical consistency and accelerating the convergence of the model during training.

[0071] Further, in step 1, the feature map size is C×H×W; H (height) represents the height of the feature map; W (width) represents the width of the feature map; C (number of channels) represents the number of channels in the feature map, usually used to represent color channels or feature channels. For example, for a tensor of shape (8,3,64,64), 8 represents the batch size, 3 represents the number of channels, 64 represents the height, and 64 represents the width.

[0072] Furthermore, the feature extraction module in step 2 performs the following steps:

[0073] Step 2-1: A local feature map of size C×H×W is input into a 3×3 convolutional layer to effectively extract features while preserving the spatial dimensions of the feature map to obtain the first feature map.

[0074] Step 2-2: The output first feature map is passed through a 1×1 convolutional layer to adjust the number of channels to obtain the second feature map, thereby reducing computational complexity and avoiding the computational overhead of using 3×3 convolution.

[0075] Steps 2-3 involve extracting global features from the second feature map using 3D multi-head attention, and then concatenating the global features with the local features obtained through the convolutional layer along the channel dimension to generate a third feature map containing more comprehensive information. The reason for not concatenating it with the input features is that the relationship between local and global features is more tightly coupled. 3D multi-head attention performs multi-head attention calculations along the height, width, and channel dimensions respectively, effectively capturing global features.

[0076] Steps 2-4 involve using a 1×1 convolution to compress the concatenated third feature map back to the original number of channels C to obtain the global features. Residual connections preserve the initial feature information, which is then fused with the enhanced features to generate a rich and robust output feature map.

[0077] Furthermore, in steps 2-3, the 3D multi-head attention performs the following steps:

[0078] Step 2-3-1: The second feature map integrates information from different channels through a 1×1 convolution to enhance the input features and increase the number of channels to 3C;

[0079] Step 2-3-2: Use a 3×3 depthwise convolution to extract feature maps rich in local information.

[0080] Step 2-3-3 involves decomposing the feature map into query (Q), key (K), and value (V) using a 3×3 depthwise convolution, and then unfolding each into 2×2 patches. Pixels at the same position within four pixels of each patch are grouped together to form four groups. In this way, Q, K, and V are divided into... and The query (Q), key (K), and value (V) are defined as follows: and Where W1 and W2 represent 3×3 depthwise convolutions, W 1×1 This represents a 1×1 convolution. and These represent the 3×3 depthwise convolution kernels used to generate the query (Q), key (K), and value (V) matrices, respectively. Their function is to extract local features and weight the features at each position.

[0081] Specifically, before performing multi-head attention, Q, K, and V need to be unfolded. Because adjacent pixels in low-light images are often very similar and carry redundant feature information, this unfolding operation helps reduce computational redundancy. Specifically, the feature map is divided into 2×2 patches, each containing four pixels. Pixels at the same location within each patch are grouped together, forming four groups. In this way, Q, K, and V are divided into... and Each group of Q, K, and V will undergo multi-head self-attention computation in one dimension of 3D multi-head attention.

[0082] In steps 2-3-4, for each set of query (Q), key (K), and value (V), multi-head attention is performed along the height, width, and channel dimensions to obtain the feature map of the corresponding dimension, so as to generate global features.

[0083] Furthermore, the multi-head attention calculation steps for each dimension in steps 2-3-4 are as follows:

[0084] S1, the self-attention representation of each head (headj) in the group is obtained as follows:

[0085]

[0086] in, These are the elements after partitioning the Q matrix, representing the query feature in the i-th row and j-th column of the image patch. These are the elements after partitioning the K matrix, representing the key features of the image patch in the i-th row and j-th column. These are the elements after partitioning the V matrix, representing the value feature of the image patch in the i-th row and j-th column. k represents the number of heads; each head calculates an independent attention score, and the final output is a concatenated set of features; d k Indicates the dimension of the corresponding vector;

[0087] S2, calculate the output feature map for each group, as follows:

[0088]

[0089] Here, Concat represents the concatenation operation. This represents the feature map obtained from the i-th group.

[0090] S3, the output feature maps of each group are merged to obtain the original-sized output map X. ′ , is represented as:

[0091]

[0092] Here, Fold represents the folding operation, which can merge the output feature maps of each group into a feature map of the original size C×H×W.

[0093] S4, the original output image is convolved with a 1×1 convolution to obtain the feature map of the corresponding dimension.

[0094] Specifically, multi-head attention is an extension of the attention mechanism widely used in deep learning models, especially in the Transformer architecture. It captures multiple semantic associations and complex dependencies in the data more comprehensively by running multiple independent attention mechanisms in parallel, each focusing on a different subspace of the input data, thus improving the model's understanding and expressive capabilities. 3D multi-head attention is a type of multi-head attention where computational complexity increases linearly with spatial size, significantly reducing the complexity of traditional multi-head attention.

[0095] Furthermore, in step 3, the multi-layer fusion module performs the following steps:

[0096] Step 3-1, three consecutive global features A single-layer feature map of size 3C×H×W is generated by splicing along the channel dimension.

[0097] Step 3-2: Use a 1×1 convolution to perform cross-channel information aggregation on the 3C×H×W feature map, and after processing based on the features of each pixel, divide it into three groups and capture spatial information at different scales by using 3×3, 5×5 and 7×7 depth-separable convolutions respectively.

[0098] Step 3-3: The output features of convolutions at different scales are concatenated again along the channel dimension to form a secondary concatenated feature map with a size of 3C×H×W;

[0099] Steps 3-4: The secondary concatenated feature map is fused by a 1×1 convolution to obtain a fused feature map.

[0100] Steps 3-5 involve adding the fused feature map to the sequentially concatenated feature map element by element to re-integrate the feature information, and then using a 1×1 convolution to restore the feature map to the C×H×W dimension to generate the final enhanced feature map.

[0101] Specifically, the multi-layer fusion module fully utilizes the advantages of multi-scale convolution to significantly improve the visual quality of low-light images through multi-level feature fusion, making the enhanced images more natural and with higher detail fidelity.

[0102] Furthermore, as a feasible implementation, the enhanced image in step 4 is input into a pre-trained object detection model (e.g., YOLOv8s) for detection.

[0103] Furthermore, as a feasible implementation, step 5 is also included: building a neural network based on the Transformer-based low-light image enhancement method, and training the neural network with the total loss function constructed based on the smoothing loss and structural similarity loss to obtain the image enhancement model.

[0104] Specifically, smoothing loss is a commonly used loss function that aims to make the output image smoother and reduce noise and artifacts by optimizing model parameters. It is suitable for tasks such as image segmentation, denoising, and super-resolution. Structural similarity is a metric that measures the similarity between two images. It considers the brightness, contrast, and structural information of the images, and can more comprehensively evaluate the differences in image quality.

[0105] Furthermore, step 5 specifically includes the following:

[0106] Step 5-1: Calculate the smoothing loss L for obtaining the low-light enhanced image and the standard illumination image. smooth The corresponding expression is:

[0107]

[0108] Among them, I p and I n These represent low-light enhanced images and standard lighting images, respectively.

[0109] Step 5-2, calculate and obtain the structural similarity score (SSIM). The calculation expression is:

[0110]

[0111] Where, μ p and μ n These are the mean values ​​of the low-light enhanced image and the standard illumination image, respectively, σ p and σ n It is the variance, σ pnThis is the covariance, where C1 and C2 are constants, typically set to 0.01. 2 and 0.03 2 .

[0112] Step 5-3: Calculate and obtain the structural similarity loss (SSIM loss), the corresponding expression is:

[0113] L ssim =1-SSIM(I p ,I n (4.6)

[0114] Among them, L ssim Represents structural similarity loss;

[0115] Step 5-4: Combine the smoothing loss and structural similarity loss to obtain the total loss function, which is then used as the final optimization function. The specific expression is as follows:

[0116] L total =L smooth +L ssim (4.7)

[0117] Among them, L total This indicates the total loss.

[0118] Specifically, low-light image enhancement is an image processing technique that uses specific algorithms to improve the brightness and contrast of images captured under low-light conditions, reduce noise, and thus significantly improve image quality.

[0119] The specific principles of this invention will be explained in detail below:

[0120] like Fig. 1 As shown, this invention presents the principle framework of a low-light image enhancement method based on Transformer. First, the input low-light image undergoes preliminary feature extraction through a convolutional layer. This convolutional layer captures local detail information of the image and converts the image into a feature map of size . Next, the feature map is input into three consecutive feature extraction modules to extract deeper global features. The outputs of these three feature extraction modules are denoted as . Then, features are fused using a multi-layer fusion module. This module employs an adaptive weighting method to effectively fuse features from different layers, generating the final enhanced feature map. Subsequently, the enhanced feature map passes through a convolutional layer and a sigmoid activation function to obtain the final enhanced image, with a size of [size missing]. Since the pixel values ​​of the input image are between 0 and 1, the sigmoid activation function ensures that the pixel values ​​of the output image remain within the same range, thus maintaining numerical consistency and accelerating model convergence during training. Finally, the enhanced image is input into a pre-trained object detection model (e.g., YOLOv8s) for detection. Compared to the low-light image, the improved image provides a higher-quality input to the object detection model, with more accurate details and sharper edges, achieving detection performance close to that of images under normal lighting conditions.

[0121] In the TLLE model, the feature extraction module is similar to the MobileViT module, consisting of multiple sub-modules. First, a feature map of size is input to a π convolutional layer for efficient feature extraction while preserving the spatial dimension of the feature map. Then, the output feature map is passed through a π convolutional layer to adjust the number of channels, thus reducing computational complexity while avoiding the computational overhead of convolution. Next, 3D multi-head attention extracts global features and concatenates them along the channel dimension with the local features obtained from the convolutional layers, generating an output feature map containing more comprehensive information. The reason for not concatenating with the input features is that the relationship between local and global features is more tightly coupled. Finally, a convolution is used to compress the concatenated features back to the original number of channels. Residual connections preserve the initial feature information, which is then fused with the enhanced features to generate a rich and robust output feature map.

[0122] Compared to convolutional neural networks, ViTs (Virtual Attention Modules) can more effectively capture global feature information through multi-head attention, especially when handling long-distance dependencies. ViT is an innovative computer vision method that segments images into small patches and processes them using a Transformer model. It utilizes a self-attention mechanism to capture the dependencies between image patches, thus demonstrating strong performance in multiple vision tasks. However, the computational complexity of self-attention in ViTs is O(n), which leads to very high computational overhead when high-resolution images are input into Transformer-based low-light image enhancement models. To address this issue, this invention proposes a 3D multi-head attention module, whose computational complexity increases linearly with spatial size, significantly reducing the complexity of multi-head attention. 3D multi-head attention performs multi-head attention calculations along the height, width, and channel dimensions respectively. Although each dimension captures local features rather than global features individually, combining attention from all three dimensions effectively captures global features. The architecture of the 3D multi-head attention module is as follows: Fig. 2 As shown. Given an input feature map X∈R C ×H×W First, a 1×1 convolution is used to integrate information from different channels to enhance the input feature X, increasing the number of channels to 3C. Next, a 3×3 depthwise convolution is used to extract features rich in local information. Then, the feature map is decomposed into query (Q), key (K), and value (V) through a 3×3 depthwise convolution, defined as follows: and Where W1 and W2 represent 3×3 depthwise convolutions, W 1×1 This represents a 1×1 convolution. Before performing multi-head attention, Q, K, and V need to be unfolded. Because adjacent pixels in low-light images are often very similar and carry redundant feature information, this unfolding operation helps reduce computational redundancy. Specifically, the feature map is divided into 2×2 patches, each containing four pixels. Pixels at the same location within each patch are grouped together, forming four groups. In this way, Q, K, and V are divided into... and Each group of Q, K, and V will undergo multi-head self-attention computation along one dimension of the 3D multi-head attention model. Taking height-dimensional multi-head self-attention as an example, the self-attention of each head (headj) within the group can be represented as:

[0123]

[0124] in, as well as

[0125] The output feature map for each group is represented as follows:

[0126]

[0127] Here, Concat represents the concatenation operation. This represents the feature map obtained from the i-th group.

[0128] Final output graph X ′ It can be represented as:

[0129]

[0130] Here, Fold represents a folding operation, which merges the output feature maps of each group into a feature map of the original size C×H×W. Then, after a 1×1 convolution, the output feature map is obtained. Subsequently, X′ is input into the width-dimensional multi-head self-attention module to obtain the corresponding feature map. Finally, this feature map is fed into the channel-dimensional multi-head self-attention module to generate the final feature map. The architectures of width-dimensional, channel-dimensional, and height-dimensional multi-head self-attention are very similar. Notably, the output feature map of the channel-dimensional multi-head self-attention module does not use 1×1 convolutions but instead uses fully connected layers. This is because fully connected layers can adaptively adjust and optimize the input features and employ Dropout to prevent overfitting, thereby avoiding overexposure in certain areas of the recovered image and accelerating model convergence.

[0131] The feature fusion module can learn the correlation between features at different levels and adaptively fuse these hierarchical features, thus performing excellently in low-light image enhancement. Therefore, this invention proposes a multi-layer fusion module for fusing features from other layers to achieve efficient image restoration with minimal computational overhead.

[0132] The structure of the multi-layer fusion module is as follows Fig. 3 As shown. First, the enhanced feature maps output by the three consecutive feature extraction modules. The feature map is concatenated along the channel dimension to generate a feature map of size 3C×H×W. Next, a 1×1 convolution is used to aggregate cross-channel information from the feature map, processing features based on each pixel. The resulting map is then divided into three groups, each using 3×3, 5×5, and 7×7 depthwise separable convolutions to capture spatial information at different scales. The outputs of these convolutions at different scales are then concatenated along the channel dimension again to form a new feature map of size 3C×H×W. A 1×1 convolution is then applied to the concatenated feature map to fuse the three scales. Finally, this feature map is added element-wise to the initially concatenated feature map to further integrate the feature information, and a 1×1 convolution is used to restore the feature map to the C×H×W dimension. This design fully leverages the advantages of multi-scale convolution, significantly improving the visual quality of low-light images through multi-level feature fusion, resulting in a more natural enhanced image with higher detail fidelity.

[0133] This invention, TLLE, uses two loss functions: smoothing loss and structural similarity loss. These loss functions are particularly important during the optimization and enhancement process, ensuring that the output image can accurately identify objects in object detection tasks.

[0134] Smooth Loss: To directly minimize the pixel-level difference between the enhanced image and the standard illuminated image, making the enhanced image closer to normal lighting conditions in brightness and color, a smoothing loss is used, the expression of which is shown in Equation 4.4:

[0135]

[0136] Among them, I p and I n These represent the low-light enhanced image and the standard-light image, respectively. Minimizing this difference ensures that the enhanced image is closer to the standard-light image in overall brightness and color, thereby improving image quality and naturalness.

[0137] Structural similarity loss (SSIM loss)

[39] : By measuring the similarity of images in brightness, contrast and structure, SSIM loss can better preserve and restore image details. Using SSIM loss in low-light image enhancement tasks can effectively reduce noise in images, improve sharpness and preserve details, thus providing higher quality input for object detection models. The formula for SSIM is as follows:

[0138]

[0139] Where, μ p and μ n These are the mean values ​​of the low-light enhanced image and the standard illumination image, respectively, σ p and σn It is the variance, σ pn This is the covariance, where C1 and C2 are constants, typically set to 0.01. 2 and 0.03 2 Furthermore, the SSIM loss can be expressed by Equation 4.6 as follows:

[0140] L ssim =1-SSIM(I p ,I n (4.6)

[0141] Combining the two loss functions mentioned above, the final optimization function for TLLE is:

[0142] L total =L smooth +L ssim (4.7)

[0143] By combining these two loss functions, the model enhances the image while maintaining overall brightness and color close to normal lighting conditions and preserving detailed structural information. Therefore, the enhanced image is more accurate and comprehensive, with object features more prominent and easily distinguishable. This significantly improves the accuracy and reliability of subsequent object detection algorithms, providing higher-quality input data for object detection models.

[0144] On the LOLv2-real dataset, the SSIM of the TLLE model of this invention is 1.7% higher than that of the current leading model. In the low-light PASCAL VOC2012 dataset, the low-light image was first enhanced using the TLLE model of this invention, and then the object detection model was used for detection. It was found that the image enhanced by the TLLE model of this invention could be detected more accurately than the image enhanced by the current state-of-the-art technology, with a 0.4% improvement in mAP(50). The computational cost of the TLLE model of this invention increases linearly with the increase of the input image size.

[0145] This invention, employing the above technical solutions, possesses the following technical advantages: 1. Solving the problem of poor image quality in low-light conditions: In low-light environments, images often suffer from insufficient brightness, significant noise, and blurred details, greatly increasing the visibility and recognition difficulty of targets. The purpose of this invention is to effectively improve image quality and restore detail information in the image through low-light enhancement methods, thereby providing clearer image input for subsequent target detection. 2. Reducing noise and blur: Low-light images are often accompanied by strong noise and blur, affecting the performance of target detection models. Traditional enhancement methods may not be able to effectively reduce noise while preserving image details. The TLLE model, by combining the advantages of convolutional neural networks (CNN) and Transformers, can more accurately reduce noise and blur while improving image quality, enabling target detection models to better identify targets in noisy and blurry low-light images. 3. Improving the contrast between objects and background: In low-light environments, the contrast between objects and background is usually low, making it difficult for target detection models to distinguish between targets and background. The TLLE model improves image contrast through effective enhancement methods, making objects stand out more in the image and improving the recognition ability of the detection model under low-light conditions. 4. Improve the recognition rate and localization accuracy of the target detection model: Low-light environments reduce the recognition rate and localization accuracy of target detection models, especially in complex scenes where targets are difficult to identify and locate accurately. The TLLE model of this invention improves the visibility of targets by enhancing the brightness, clarity, and contrast of the image, thereby improving the recognition rate and localization accuracy of the target detection model under low-light conditions. 5. Reduce the computational complexity introduced by Transformer: A 3D multi-head attention mechanism is designed, performing multi-head attention calculations on the height, width, and channels of the feature map separately, making the computational load increase linearly with the size of the input image.

[0146] This invention significantly improves image brightness and contrast through a series of algorithms and methods, effectively reducing noise and distortion, making details that were previously difficult to discern in low-light environments clearly visible. Whether it's a photograph taken at night, a poorly lit indoor scene, or a dark image caused by improper camera exposure settings, low-light image enhancement technology plays a crucial role, providing a more reliable and high-quality image foundation for image recognition, analysis, processing, and subsequent applications. This invention improves the image quality captured under low-light conditions.

[0147] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A low-light image enhancement method based on Transformer, characterized in that: It includes the following steps: Step 1: The input low-light image is processed through a convolutional layer for preliminary feature extraction; Convolutional layers capture local details of an image and convert it into a local feature map of a specified size; Step 2: Construct three consecutive feature extraction modules based on 3D multi-head attention, and input the local feature map into the three consecutive feature extraction modules to extract deeper global features; Step 3: Construct a multi-layer fusion module based on depthwise separable convolutions of different sizes, and fuse the three global features through a multi-layer fusion module to generate the final enhanced feature map; The multi-layer fusion module uses an adaptive weighting method to fuse features from different layers; Step 4: The enhanced feature map is passed through a convolutional layer and a sigmoid activation function to obtain the final enhanced image.

2. The low-light image enhancement method based on Transformer according to claim 1, characterized in that: In step 1, the size of the convolutional layer is 3×3.

3. The low-light image enhancement method based on Transformer according to claim 1, characterized in that: In step 2, the feature extraction module performs the following steps: Step 2-1: A local feature map of size C×H×W is input into a 3×3 convolutional layer to effectively extract features while maintaining the spatial dimension of the feature map to obtain the first feature map. Step 2-2: The output first feature map is passed through a 1×1 convolutional layer to adjust the number of channels to obtain the second feature map; Steps 2-3: Extract global features from the second feature map using 3D multi-head attention, and concatenate the global features with the local features obtained through the convolutional layer along the channel dimension to generate a third feature map containing more comprehensive information; 3D multi-head attention is calculated along the height, width and channel dimensions to capture global features. Steps 2-4: Use a 1×1 convolution to compress the concatenated third feature map back to the original number of channels C to obtain the global features.

4. The low-light image enhancement method based on Transformer according to claim 3, characterized in that: In steps 2-3, the 3D multi-head attention process performs the following steps: Step 2-3-1: The second feature map integrates information from different channels through a 1×1 convolution to enhance the input features and increase the number of channels to 3C; Step 2-3-2: Use a 3×3 depthwise convolution to extract feature maps rich in local information; Step 2-3-3: The feature map is decomposed into query Q, key K and value V by 3×3 depth convolution and expanded into 2×2 patches. Pixels at the same position within 4 pixels of each patch are grouped together to form four groups. In steps 2-3-4, for each set of query Q, key K, and value V, multi-head attention is performed along the height, width, and channel dimensions to obtain the feature map of the corresponding dimension, so as to generate global features.

5. The low-light image enhancement method based on Transformer according to claim 4, characterized in that: The multi-head attention calculation steps for each dimension in steps 2-3-4 are as follows: S1, the self-attention representation of each head in the group is obtained as follows: in, These are the elements after partitioning the Q matrix, representing the query feature in the i-th row and j-th column of the image patch. These are the elements after partitioning the K matrix, representing the key features of the image patch in the i-th row and j-th column. These are the elements after partitioning the V matrix, representing the value feature of the image patch in the i-th row and j-th column. i = 1,...,4, j = 1,...,k; k represents the number of heads; each head calculates an independent attention score, and the final output is a concatenated set of features; d k Indicates the dimension of the corresponding vector; S2, calculate the output feature map for each group, as follows: Here, Concat represents the concatenation operation. This represents the feature map obtained from the i-th group; S3, the output feature maps of each group are merged to obtain the original-sized output map X′, represented as: Wherein, Fold represents the folding operation, which can merge the output feature maps of each group into a feature map of the original size C×H×W; S4, the original output image is convolved with a 1×1 convolution to obtain the feature map of the corresponding dimension.

6. The low-light image enhancement method based on Transformer according to claim 4, characterized in that: In step 3, the multi-layer fusion module performs the following steps: Step 3-1, three consecutive global features A single-layer feature map of size 3C×H×W is generated by splicing along the channel dimension. Step 3-2: Use a 1×1 convolution to perform cross-channel information aggregation on the 3C×H×W feature map, and after processing based on the features of each pixel, divide it into three groups and capture spatial information at different scales by using 3×3, 5×5 and 7×7 depth-separable convolutions respectively. Step 3-3: The output features of convolutions at different scales are concatenated again along the channel dimension to form a secondary concatenated feature map with a size of 3C×H×W; Steps 3-4: The second-time concatenated feature map is fused with the feature maps of the three scales through a 1×1 convolution to obtain the fused feature map; Steps 3-5: The fused feature map is added element by element to the sequentially concatenated feature map to integrate the feature information again, and a 1×1 convolution is used to restore the feature map to the C×H×W dimension to generate the final enhanced feature map.

7. The low-light image enhancement method based on Transformer according to claim 1, characterized in that: The enhanced image in step 4 is input into a pre-trained object detection model to detect the image enhancement effect.

8. The low-light image enhancement method based on Transformer according to claim 1, characterized in that: The method also includes step 5: building a neural network based on the Transformer-based low-light image enhancement method, and using the total loss function constructed based on the smoothing loss and structural similarity loss to train the neural network to obtain the image enhancement model.

9. The low-light image enhancement method based on Transformer according to claim 8, characterized in that: Step 5 specifically includes the following: Step 5-1: Calculate the smoothing loss L for obtaining the low-light enhanced image and the standard illumination image. smooth The corresponding expression is: Among them, I p and I n These represent low-light enhanced image and standard illumination image, respectively; Step 5-2, calculate and obtain the structural similarity score (SSIM). The calculation expression is: Where, μ p and μ n These are the mean values ​​of the low-light enhanced image and the standard illumination image, respectively, σ p and σ n It is the variance, σ pn It is the covariance, where C1 and C2 are constants; Step 5-3: Calculate and obtain the structural similarity loss (SSIM loss), the corresponding expression is: L ssim =1-SSIM(I p ,I n ) (4.6) Among them, L ssim Represents structural similarity loss; Step 5-4: Combine the smoothing loss and structural similarity loss to obtain the total loss function, which is then used as the final optimization function. The specific expression is as follows: L total L smooth +L ssim (4.7) Among them, L total This indicates the total loss.

Images