CNN-Transform-based unsupervised low-illumination image multi-degradation problem recovery method

Abstract

The invention is suitable for the technical field of computer vision, and provides an unsupervised low-illumination image multi-degradation problem recovery method based on CNN-Transform, which constructs a network comprising a low-illumination image enhancement and exposure suppression module and an image denoising module. Adjusting the brightness pixel by pixel and suppressing overexposure by using a low light enhancement and exposure suppression curve; and the latter realizes noise removal through noise addition processing and a C-T module. The design comprises seven unsupervised loss functions, and training can be carried out without pairwise labeling data. The advantages of CNN local feature extraction and Transform global dependence modeling are combined, and the lightweight and real-time performance of the model are ensured. According to the method, the image brightness is effectively enhanced, overexposure is inhibited, noise is removed, a high-quality image basis is provided for the fields of automatic driving, security and protection monitoring, medical images and the like, and meanwhile advanced visual tasks such as target detection and the like are assisted.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an unsupervised low-light image multi-degradation problem restoration method based on CNN-Transformer. Background Art

[0002] In recent years, with the development of deep learning technology, low-light image enhancement methods based on deep learning have been widely used. Compared with traditional methods, deep learning-based methods do not rely too much on hand-designed priors or strategies and have better adaptability and generalization capabilities.

[0003] Among these methods, those based on supervised learning are more commonly used. However, they rely on a large number of paired low-light and normal-light images for training. The high cost of acquiring such data limits their application. In contrast, methods based on unsupervised learning do not require a large amount of labeled paired data for training and have strong generalization capabilities, making them suitable for complex scenes. In addition, most existing methods are based on convolutional neural networks (CNNs), which are limited by the size of their receptive fields and have a weak ability to capture global information. At the same time, most existing methods focus on improving image brightness, but ignore exposure suppression and noise processing. This results in overexposure in local areas of the enhanced image and further amplification of noise, resulting in loss of detail information and reduced visual quality. Summary of the Invention

[0004] The purpose of the present invention is to provide an unsupervised low-light image multi-degradation problem restoration method based on CNN-Transformer, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0005] The present invention is implemented as follows: a CNN-Transformer-based unsupervised low-light image multi-degradation problem restoration method comprises the following steps: Step 1: Build a low-light image augmentation dataset We used existing low-light datasets and nighttime low-light images captured by vehicle-mounted cameras to construct a dataset and randomly split it into training and test sets. The training set images were normalized and scaled to a size of (512, 512, 3), and data augmentation was performed by random flipping.

[0006] Step 2: Build a multi-task recovery network The network consists of two core modules: 1. Low-light image enhancement and exposure suppression module: Structure: 3 3×3 convolutional layers, 2 HardSwish activation functions, 3 CNN-Transformer modules (CT modules), 1 Tanh activation function and low-light enhancement and exposure suppression curve (LES).

[0007] Processing flow: Input low-light image Primary features are extracted through the first two layers of convolution and activation functions 、 ; Fusion of local and global features through the CT module (residual connection for feature fusion) to generate deep features ; The Tanh activation function generates 8 curve parameter maps for each of the RGB channels. By iterating the LES curve, the brightness is enhanced and overexposure is suppressed pixel by pixel to generate a normal lighting image with noise. .

[0008] CT module: This module includes layer normalization (LN), a hybrid attention block (HAB), and a channel-shuffled feedforward neural network (CSFFN). HAB combines spatial attention with self-attention mechanisms to capture local image features and global dependencies. CSFFN dynamically adjusts channel information flow through a gating module to improve feature expression capabilities.

[0009] 2. Image denoising module: Structure: 2 3×3 convolutional layers, 1 GeLU activation function, and 1 CT module.

[0010] Processing flow: Add noise to get and After processing by the convolution and CT modules, noise is removed by element-by-element subtraction to output a clean image. .

[0011] Step 3: Construct an unsupervised loss function Design 7 unsupervised loss functions to ensure multi-task optimization effect: Color preservation loss: constrains the color consistency of input and output images to avoid color distortion; Low-light enhancement and exposure suppression: enhance low-light areas and suppress local overexposure areas through threshold T; Spatial consistency loss: ensures that the average intensity of local areas of the input and output images is consistent, maintaining the consistency of the image spatial structure; Weighted total variation minimization loss: combines the curve parameter map and the gradient of the input image to constrain the spatial smoothness of the output image; Local area consistency loss: ensures a natural and smooth transition between different local areas of the image; Total variation loss: reduces image noise and improves visual clarity; Mean Squared Error Loss: Constraining the consistency of noisy image pairs.

[0012] The total loss function is the weighted sum of the unsupervised loss functions.

[0013] Step 4: Network training and parameter optimization The training set is input into the network, the parameters are iteratively optimized through back propagation, and the network parameters are adjusted and updated using an unsupervised loss function until the model converges.

[0014] Step 5: Test set image restoration The low-light images in the test set are input into the trained network, and are sequentially enhanced, suppressed, and denoised to output high-quality images with appropriate brightness, balanced exposure, and effectively suppressed noise.

[0015] The beneficial effects of the present invention are: The present invention effectively reduces the complexity of the task by converting the low-light image enhancement and exposure suppression problem into a curve estimation problem; by introducing a low-light image enhancement and exposure suppression module and an image denoising module into the network, the network can simultaneously solve the problems of insufficient brightness, overexposure and excessive noise in the image.

[0016] The present invention uses an unsupervised learning method, which can perform training without paired low-light and normal-light images, effectively reducing the difficulty of training.

[0017] This invention combines the respective advantages of CNN and Transformer, solving the problems of the weak ability of existing CNN-based methods to capture global information and the high computational complexity of existing Transformer-based methods. While ensuring the lightweight model and the ability to process images in real time, it effectively enhances image brightness, suppresses overexposure, and removes noise.

[0018] This invention has broad application prospects and can play an important role in fields such as autonomous driving, intelligent transportation, computational photography, security monitoring, and medical imaging. It can also provide a high-quality image information foundation for high-level computer vision tasks such as target detection, semantic segmentation, and face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the network structure of the unsupervised low-light image multi-degradation problem restoration method based on CNN-Transformer proposed in this invention; Figure 2 This is the specific structure of the CNN-Transformer module, hybrid attention module HAB and channel shuffled feedforward neural network CSFFN proposed in this invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0021] A CNN-Transformer-based unsupervised low-light image multi-degradation problem restoration method, the method comprising: Step 1: Construct a low-light image enhancement dataset and randomly divide it into training and test sets. The dataset contains low-light images and does not require paired normal-light image labels. Step 2: Construct a CNN-Transformer-based unsupervised low-light image multi-degradation restoration method network, including a low-light image enhancement and exposure suppression module and an image denoising module; The low-light image enhancement and exposure suppression module extracts features through the CNN-Transformer module and iteratively applies low-light enhancement and exposure suppression curves to generate a noisy normal-light image. The image denoising module performs noise addition and denoising on the noisy normal-light image to output a clean image. Step 3: Construct an unsupervised loss function to ensure the model's low-light enhancement, exposure suppression, image denoising, and color restoration effects without explicit labeling information. The unsupervised loss function includes color preservation loss, low light enhancement and exposure suppression loss, spatial consistency loss, weighted total variation minimization loss, local area consistency loss, total variation loss and mean square error loss; Step 4: Input the training set into the network and use the unsupervised loss function to adjust and update the network parameters until the network converges; Step 5: Input the image of the test set into the trained network and output the restored image.

[0022] In step 2, if Figure 1 As shown in the figure, the network structure of an unsupervised low-light image multi-degradation problem restoration method based on CNN-Transformer includes a low-light image enhancement and exposure suppression module and an image denoising module; the input training set image will first be normalized to fix the pixel value to The image size is then scaled to The size of the data is enhanced by random horizontal and vertical flipping. The low-light image enhancement and exposure suppression module includes 3 Convolutional layer, 2 HardSwish activation functions, 3 CNN-Transformer modules (CT modules), 1 Tanh activation function and low-light enhancement and exposure suppression curve LES; input low-light image First go through the first The convolution layer and HardSwish activation function extract primary features and obtain feature maps ,Then After the second The convolution layer and HardSwish activation function extract features to obtain the feature map ; In order to further extract the global features of the image, alleviate the gradient disappearance and fuse the shallow features, the feature map After the first CT module, and the feature map Perform element-by-element addition to obtain the feature map ; Feature map After the second CT module, the feature map Perform element-by-element addition to obtain the feature map ; Feature map After the third The convolution layer extracts features and obtains the feature map ; Feature map After the third CT module, compared with the feature map and feature maps Perform element-by-element addition operation to combine deep features and initial features to obtain feature map ;Finally, the feature map After the Tanh activation function, 8 curve parameter maps are generated for each of the RGB channels. The size of the elements in the curve parameter map is limited to Within the range, the RGB three channels iterate 8 times for low-light enhancement and exposure suppression curve LES, enhance the low-light area and suppress the high-exposure area, and generate a normal-light image with noise ; The low-light enhancement and exposure suppression curve LES is used to enhance the brightness of the image and suppress overexposure. The formula is as follows: ; Where, represents the image input to the network, It is the weight coefficient used to adjust the degree of curve enhancement. is the image output by the network; Since the above formula enhances the entire image, in order to obtain pixel-level enhancement effect, the above formula is iterated pixel by pixel to obtain the pixel-level iterative enhancement curve. The formula is as follows: ; Where, is the pixel value of the input image, is the number of iterations, is a parameter adjustment matrix consistent with the image size, Iteration n The pixel value output after the second Iteration The pixel value output after the second time; The image denoising module is used to denoise the enhanced image output by the low-light image enhancement and exposure suppression modules. To ensure the lightweight of the model, the image denoising module only uses two Convolutional layer, 1 GeLU activation function and 1 CT module; First, the normal light image with noise output by the low light image enhancement and exposure suppression module Perform noise processing to obtain two noisy images and , the formula is as follows: ; Where, is any reversible matrix, is a random variable randomly sampled from a standard normal distribution; After 1 The convolution layer, GeLU activation function and 1 CT module are then passed through 1 The convolutional layer and Perform element-by-element subtraction to obtain a clean normal lighting image .

[0023] like Figure 2 As shown, the CT module includes layer normalization LN, hybrid attention module HAB and channel shuffled feedforward neural network CSFFN; The hybrid attention module HAB includes 1 spatial attention module, 2 Convolutional layer, 1 The grouped convolution layer, 1 Softmax activation function and 1 learnable position encoding; enter the feature map of HAB First, the feature map is generated by the spatial attention module , and with Perform element-by-element addition to obtain the feature map ; Feature map Then, through the self-attention mechanism, after 1 The convolutional layer and 1 The grouped convolution layer is divided into 3 groups according to the channel, and the , and ; , and After reshaping operations, we generate , and ; and After matrix multiplication, it passes through the Softmax activation function and then Perform matrix multiplication to obtain the feature map ; Feature map After the reshaping operation, then The convolution layer is added element by element with the learnable positional encoding to generate a feature map The hybrid attention module (HAB) combines the spatial attention module and the self-attention mechanism to effectively combine the local and global features of the image. This allows it to accurately restore key information and suppress noise and overexposed areas during low-light image enhancement, significantly improving the detail retention and visual naturalness of the enhanced results. The channel shuffled feedforward neural network CSFFN includes a gating module and a The convolutional layer of the gating module includes two branches, each of which includes 1 Convolutional layer, 1 The grouped convolution layer and channel shuffling operation are used to dynamically control the information flow; the lower branch has one more GeLU activation function than the upper branch; the feature map output by the hybrid attention module HAB The feature maps are generated by the upper and lower branches respectively and feature maps , and After the element-by-element multiplication operation, 1 The convolution layer extracts features and generates feature maps ; In step 3, an unsupervised loss function is constructed to ensure the model's low-light enhancement, exposure suppression, image denoising, and color restoration capabilities without explicit labeling information; the unsupervised loss function includes color preservation loss, low-light enhancement and exposure suppression loss, spatial consistency loss, weighted total variation minimization loss, local area consistency loss, total variation loss, and mean square error loss; The color preservation loss formula is as follows: ; Where, ; and Images and The average pixel value in the channel; ; is the total number of elements in the enhancement matrix; and are the pixel values of input and output respectively; ; The formulas for low-light enhancement and exposure suppression loss are as follows: ; Where, For the image The number of non-overlapping regions of size, For the The average exposure of the local area, is the exposure threshold; The spatial consistency loss formula is as follows: ; Where, is the total number of local regions, Indicates the Local area up, down, left and right The set of adjacent regions, is the local average intensity value of the output image, is the local average intensity value of the image to be enhanced; The weighted total variation minimization loss formula is as follows: ; Where, is the number of iterations of the curve, is the iterative curve parameter diagram, and Represents the gradient of the curve parameter graph in the horizontal and vertical directions, is the weight parameter, and Represent the gradients of the input image in the horizontal and vertical directions, respectively. is the low-light image input to the network; The local area consistency loss formula is as follows: ; Where, For size The total number of local areas, and are the height and width of the image, Indicates the The pixel value after local area enhancement, is the average intensity value of the local area; The total variational loss formula is as follows: ; Where, is the pixel in the image, and Represents the height and width of the image respectively; The mean square error loss formula is as follows: ; Where, is the total number of pixels in the image, and Represents the first of two noisy images pixels; Based on the above unsupervised loss function, the total loss function formula is constructed as follows: ; Where, represents the total loss, is the weight coefficient of each loss function, Indicates color retention loss, Indicates low light enhancement and exposure suppression loss, represents the spatial consistency loss, represents the weighted total variation minimization loss, represents the local area consistency loss, represents the total variational loss, represents the mean squared error loss.

[0024] For step 4: input the training set into the CNN-Transformer-based unsupervised low-light image multi-degradation restoration method network constructed in step 2 for training. Through continuous iterative optimization, use the unsupervised loss function in step 3 to adjust and update the network parameters until the network converges to obtain a trained network model; As for step 5: the low-light images in the test set are input into the trained network, and the images are comprehensively processed using its adaptive enhancement, suppression, and denoising strategies to ultimately generate clear images with rich details, moderate exposure, and effectively suppressed noise.

[0025] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0026] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0027] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A CNN-Transformer-based unsupervised low-light image multi-degradation restoration method, characterized by: The method comprises: Step 1: Construct a low-light image enhancement dataset and randomly divide it into training and test sets. The dataset contains low-light images and does not require paired normal-light image labels. Step 2: Construct a low-light image multi-degradation restoration network, including a low-light image enhancement and exposure suppression module and an image denoising module; The low-light image enhancement and exposure suppression module extracts features through the CNN-Transformer module and iteratively applies low-light enhancement and exposure suppression curves to generate a noisy normal-light image. The image denoising module performs noise addition and denoising on the noisy normal-light image to output a clean image. Step 3: Construct an unsupervised loss function to ensure the model's low-light enhancement, exposure suppression, image denoising, and color restoration effects without explicit labeling information. The unsupervised loss function includes color preservation loss, low light enhancement and exposure suppression loss, spatial consistency loss, weighted total variation minimization loss, local area consistency loss, total variation loss and mean square error loss; Step 4: Input the training set into the network, iteratively optimize the parameters through back propagation, and use the unsupervised loss function to adjust and update the network parameters until the network converges; Step 5: Input the image of the test set into the trained network and output the restored image.

2. The method according to claim 1, characterized in that The dataset described in step 1 contains images from an existing low-light dataset and nighttime low-light images acquired by a vehicle-mounted camera. Before training, the images are normalized, scaled to (512, 512, 3), and randomly flipped for data augmentation.

3. The method according to claim 1, characterized in that The low-light image enhancement and exposure suppression module in step 2 includes: 3 3×3 convolutional layers, 2 HardSwish activation functions, 3 CNN-Transformer modules, 1 Tanh activation function and low-light enhancement and exposure suppression curves LES; The input image is extracted through convolutional layers and activation functions, local and global features are fused through the CNN-Transformer module, a curve parameter map is generated through the Tanh activation function, and a normal illumination image with noise is generated by iterating the LES curve formula. ; The low-light enhancement and exposure suppression curve LES is used to enhance the brightness of the image and suppress overexposure, performing pixel-level iterative enhancement. The curve formula is: ； Where, is the pixel value of the input image, is the number of iterations, is a parameter adjustment matrix consistent with the image size, Iteration n The pixel value output after the second Iteration The pixel value output after the second pass.

4. The method according to claim 3, characterized in that The CNN-Transformer module includes a layer normalization LN, a hybrid attention module HAB and a channel shuffled feedforward neural network CSFFN. The hybrid attention module HAB extracts features by combining spatial attention and self-attention mechanisms, and the channel shuffled feedforward neural network CSFFN dynamically controls the information flow through a gating module.

5. The method according to claim 1, wherein The image denoising module in step 2 includes: 2 3×3 convolutional layers, 1 GeLU activation function, and 1 CNN-Transformer module; The normal illumination image with noise is subjected to noise addition processing, and the noise addition formula is: ； Where, is any reversible matrix, is a random variable randomly sampled from a standard normal distribution; After 1 The convolutional layer, GeLU activation function and 1 CNN-Transformer module are then passed through 1 The convolutional layer and Perform element-by-element subtraction to obtain a clean normal lighting image .

6. The method according to claim 1, characterized in that The total loss of the unsupervised loss function described in step 3 is: ； Where, represents the total loss, is the weight coefficient of each loss function, Indicates color retention loss, Indicates low-light enhancement and exposure suppression loss, represents the spatial consistency loss, represents the weighted total variation minimization loss, represents the local area consistency loss, represents the total variational loss, represents the mean squared error loss.

Images