A multi-task restoration method and system for low-light images based on CNN-Transformer

Abstract

The present invention is applicable to the field of computer vision technology and provides a multi-task restoration method and system for low-light images based on CNN-Transformer. A network consisting of an encoder and a decoder is constructed. The encoder extracts local and global features and combines group iterative convolution to achieve low-light enhancement and exposure suppression. The decoder uses the Decoder CNN-Transformer module and an adaptive deformable hole convolution block to handle non-uniform motion blur. The encoder and decoder use jump connections to fuse shallow features with deep features. The SmoothL1 loss and perceptual loss are designed for joint optimization to enhance image brightness, contrast, and details, generating high-quality results. The method takes into account both lightweight and real-time performance, can handle multiple degradation tasks in the same network, and provides a high-quality image foundation for tasks such as target detection and semantic segmentation, effectively improving the overall image quality.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a CNN-Transformer-based low-light image multi-task restoration method and system. Background Art

[0002] Deep learning-based methods have been widely used in the fields of low-light image enhancement and image deblurring. These methods improve image quality by building deep learning models and using large amounts of data to learn image features.

[0003] However, most existing methods focus on solving a single problem: low-light image enhancement or image deblurring, often ignoring the interaction between the two. In real-world scenarios, images captured in low-light environments are often accompanied by degradation phenomena such as insufficient brightness, overexposure, and motion blur. This makes it difficult to fully restore the overall image quality by addressing each problem separately.

[0004] Although some methods can handle low-light image enhancement and image deblurring simultaneously, they ignore non-uniform motion blur. Non-uniform motion blur is caused by the diverse motion of the object or camera, and the blur degree varies in different areas, making the problem more complex and difficult to solve. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-task restoration method for low-light images based on CNN-Transformer, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0006] The present invention is implemented as follows: a CNN-Transformer-based low-light image multi-task restoration method comprises the following steps:

[0007] Step 1: Obtain training and validation sets

[0008] Low-light blurry images and their corresponding normal-light clear images were collected to construct training and validation sets, respectively. A threshold for the illumination range was defined, and both the training and validation sets consisted of low-light blurry images below the minimum illumination range and their corresponding normal-light clear images within the illumination range. This dataset was used for model training and performance evaluation, providing supervisory signals for subsequent network learning.

[0009] Step 2: Build a multi-task recovery network (encoder-decoder structure)

[0010] The network consists of an encoder, a decoder, and skip connections. The specific structure is as follows:

[0011] Encoder:

[0012] 1) Initial layer: A 3×3 convolutional layer is used to extract shallow features, followed by a PReLU activation function.

[0013] 2) Three CNN-Transformer modules (CT modules), each of which contains: an encoder residual block (EB): composed of three 3×3 convolutional layers and two PReLU activation functions to extract local features; a downsampling layer: reduces the feature map resolution and expands the receptive field; an iterative enhancement module (IEM): including layer normalization (LN), a hybrid attention module (HAB), a channel shuffled feedforward neural network (CSFFN), and a grouped iterative convolution (GIConv). It fuses local and global information through spatial attention and self-attention mechanisms, and combines grouped iterative convolution to achieve low-light enhancement and exposure suppression.

[0014] Decoder:

[0015] 1) 2 encoder residual blocks (EB) and 1 decoder residual block (DB, containing 4 Convolutional layer and 2 PReLU activation functions);

[0016] 2) Three Decoder CNN-Transformer modules (DC-T modules). Each DC-T module contains: Adaptive Deformable Dilated Convolution Block (ADDB): extracts multi-scale features through deformable convolution with dilation rates of 1, 2, and 3, and combines dynamic weights to extract features of non-uniform motion blurred areas; Layer Normalization (LN), Hybrid Attention Module (HAB), Channel Shuffled Feedforward Neural Network (CSFFN), upsampling layers, and Decoder Residual Block (DB) to gradually restore image resolution and integrate deep semantic features;

[0017] 3) Final layer: 1 3×3 convolutional layer outputs the restored image.

[0018] Feature fusion:

[0019] The encoder and decoder directly transfer shallow features through skip connections to achieve deep and shallow feature fusion and retain image detail information.

[0020] Step 3: Construct a joint optimization loss function

[0021] The design includes pixel-level and semantic-level loss functions to jointly optimize low-light enhancement, exposure suppression, and non-uniform motion blur removal tasks:

[0022] SmoothL1 loss: measures the pixel-level difference between the enhanced image and the real image, and is more robust to outliers.

[0023] Perceptual loss: Extracts high-level semantic features based on the pre-trained VGG19 network and calculates feature space differences to ensure enhanced image semantic consistency.

[0024] Total loss function: weighted sum of the above losses.

[0025] Step 4: Network training

[0026] The training set images were randomly flipped horizontally and vertically and randomly cropped (size 256×256×3) for data augmentation before being fed into the network for end-to-end training. The network parameters were optimized using the backpropagation algorithm to minimize the total loss function until the model converged.

[0027] Step 5: Validation set inference

[0028] The low-light blurred images in the validation set are input into the trained network, and the restored normal-light clear images are output to verify the model's processing effect on brightness, contrast and non-uniform motion blur.

[0029] The beneficial effects of the present invention are:

[0030] By designing multiple modules and IEF activation functions, the present invention realizes low-light enhancement, exposure suppression, and non-uniform motion blur removal of images in low-light dynamic environments in the same network, thereby improving the overall image quality.

[0031] This invention combines the respective advantages of CNN and Transformer, solving the problems of weak global information capture ability of existing CNN-based methods and high computational complexity of existing Transformer-based methods. While ensuring the lightweight model and real-time image processing capabilities, the model can accurately model the details and global semantics in the image, providing clearer and higher-quality image input for subsequent tasks.

[0032] This invention has broad application prospects and plays an important role in fields such as autonomous driving, intelligent transportation, and security monitoring. It can also provide a high-quality image information foundation for high-level computer vision tasks such as target detection, semantic segmentation, and face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the network structure of a CNN-Transformer-based multi-task low-light image restoration method proposed in this invention;

[0034] Figure 2 The specific structure of the CNN-Transformer module CT module, encoder residual block EB, iterative enhancement module IEM, hybrid attention module HAB, channel shuffled feedforward neural network CSFFN and grouped iterative convolution GIConv proposed in the present invention;

[0035] Figure 3This is the specific structure of the Decoder CNN-Transformer module DC-T module, adaptive deformable void convolution block ADDB and decoder residual block DB proposed in the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] A multi-task restoration method for low-light images based on CNN-Transformer, characterized in that the method comprises:

[0038] Step 1: Obtain a training set and a validation set, and define a light interval threshold. The training set and validation set are both low-light blurry images below the minimum value of the light interval and corresponding normal-light clear images within the light interval;

[0039] Step 2: Construct a low-light image enhancement, exposure suppression, and non-uniform motion blur removal network, wherein the low-light image enhancement, exposure suppression, and non-uniform motion blur removal network includes an encoder and a decoder;

[0040] Step 3: Construct a joint optimization loss function, which includes SmoothL1 loss and perceptual loss. The total loss function is the weighted sum of SmoothL1 loss and perceptual loss.

[0041] Step 4: Input the training set into the network for training, and update the network parameters by optimizing the total loss function;

[0042] Step 5: Input the validation set into the trained network and output the restored image.

[0043] In step 2, if Figure 1 As shown in the figure, a network structure of a CNN-Transformer-based low-light image multi-task restoration method includes an encoder and a decoder. The encoder and decoder are connected by skipping to fuse shallow features with deep features.

[0044] like Figure 1 As shown, the encoder part includes 1 The convolutional layer is used to extract shallow features, 1 PReLU activation function and 3 CNN-Transformer modules (CT modules);

[0045] like Figure 2 As shown in the figure, the CT module consists of an encoder residual block EB, a downsampling layer and an iterative enhancement module IEM;

[0046] The encoder residual block EB consists of 3 Convolutional layer and 2 PReLU activation functions;

[0047] The iterative enhancement module IEM includes a layer normalization LN, a hybrid attention module HAB, a channel shuffled feedforward neural network CSFFN and a grouped iterative convolution GIConv;

[0048] The hybrid attention module HAB includes 1 spatial attention module, 2 Convolutional layer, 1 The grouped convolution layer, 1 Softmax activation function and 1 learnable position encoding extract the local and global information of the image; enter the feature map of HAB First, the feature map is generated by the spatial attention module , and with Perform element-by-element addition to obtain the feature map ; Feature map After 1 The convolutional layer and 1 The grouped convolution layer is divided into 3 groups according to the channel, and the , and ; , and After reshaping operations, we generate , and ; and After matrix multiplication, it passes through the Softmax activation function and then Perform matrix multiplication operation to obtain feature map ; Feature map After the reshaping operation, then The convolution layer is added element by element with the learnable positional encoding to generate a feature map ;

[0049] The channel shuffled feedforward neural network CSFFN includes a gating module and a The convolutional layer of the gating module includes two branches, each of which includes 1 Convolutional layer, 1 The grouped convolution layer and channel shuffling operation enable information from different components to interact; the lower branch has one more GeLU activation function than the upper branch; the feature map output by the hybrid attention module HAB The feature maps are generated by the upper and lower branches respectively and feature maps , and After the element-by-element multiplication operation, 1 The convolution layer extracts features and generates feature maps ;

[0050] The grouped iterative convolution GIConv can adaptively enhance the information of different groups; first, the input feature map Divided into 4 groups according to the channel dimension, we get and ; After channel attention (CA) and spatial attention (SA), after 1 The convolution layer obtains ;Will Respectively Splicing is performed according to the channel dimension, and each is processed by 1 The convolutional layer generates parameters and ;at the same time, In the top branch, the value is clipped to The interval is divided into 4 groups according to the channel dimension, and the and ;Will and and and The input is fed into the iterative enhancement activation function IEF for low-light enhancement and exposure suppression, and then the results are spliced ​​according to the channel dimension to obtain the enhanced feature map. ;

[0051] The iterative enhancement activation function IEF formula is as follows:

[0052] ;

[0053] Where, , Iterate for IEF The output result after the second , for In the The weight parameter value at the iteration.

[0054] like Figure 1 As shown in Figure 2, the decoder part includes 2 encoder residual blocks EB, 1 decoder residual block DB, 3 Decoder CNN-Transformer modules (DC-T modules) and 1 Convolutional layers;

[0055] like Figure 3 As shown, the decoder residual block DB includes 4 Convolutional layer and 2 PReLU activation functions;

[0056] The DC-T module includes an adaptive deformable dilated convolution block (ADDB), layer normalization LN, a hybrid attention module HAB, a channel shuffled feedforward neural network CSFFN, an upsampling layer and a decoder residual block DB;

[0057] The adaptive deformable dilated convolution block (ADDB) can extract the non-uniform motion blur features in the image; the input feature map First, the void ratios are 1, 2, and 3. Deformable convolution Dconv, extracts feature information of different scales and generates feature maps ， and ; At the same time, input feature map After the average pooling operation, 1 Convolutional layer, 1 GeLU activation function, 1 Convolutional layer and 1 Softmax activation function to generate dynamic weights ， and ; The feature map ， and Respectively with dynamic weight ， and Perform element-by-element multiplication and then element-by-element addition to the input feature map Splicing is done by channel dimension, after 1 The convolutional layer output feature map ;

[0058] In step 3, the loss function includes SmoothL1 loss and perceptual loss;

[0059] The SmoothL1 loss formula is as follows:

[0060] ;

[0061] Where, An image representing network recovery, represents the normal lighting clear image corresponding to the input image;

[0062] The perceptual loss formula is as follows:

[0063] ;

[0064] Where, An image representing network recovery, represents the normal lighting clear image corresponding to the input image, Represents the pre-trained VGG19 network;

[0065] Based on the above loss function, the total loss function formula is constructed as follows:

[0066] ;

[0067] In step 4, the input training set images are first randomly flipped horizontally and vertically for data augmentation, and then randomly cropped to a size of (256, 256, 3);

[0068] The system of the CNN-Transformer-based low-light image multi-task restoration method includes:

[0069] Data acquisition module: used to obtain low-light blurry images and normal-light clear image datasets;

[0070] Network building module: used to build a CNN-Transformer network containing encoders, decoders and skip connections;

[0071] Loss function module: used to calculate SmoothL1 loss, perceptual loss and total loss;

[0072] Training module: used for data enhancement, network training and parameter optimization;

[0073] Reasoning module: used to input the validation set image and output the restoration result.

[0074] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-task restoration method for low-light images based on CNN-Transformer, characterized by: The method comprises: Step 1: Obtain a training set and a validation set, and define a light interval threshold. The training set and validation set are both low-light blurry images below the minimum value of the light interval and corresponding normal-light clear images within the light interval; Step 2: Construct a low-light image enhancement, exposure suppression, and non-uniform motion blur removal network. The low-light image enhancement, exposure suppression, and non-uniform motion blur removal network includes an encoder and a decoder. The encoder and decoder are connected via a skip connection to fuse shallow features with deep features. Step 3: Construct a joint optimization loss function, which includes SmoothL1 loss and perceptual loss. The total loss function is the weighted sum of SmoothL1 loss and perceptual loss. Step 4: Input the training set into the network for training, and update the network parameters by optimizing the total loss function; Step 5: Input the validation set into the trained network and output the enhanced image; In step 2, the encoder includes 1 The convolutional layer, 1 PReLU activation function and 3 CNN-Transformer modules are composed of an encoder residual block EB, a downsampling layer and an iterative enhancement module IEM. The iterative enhancement module IEM includes a layer normalization LN, a hybrid attention module HAB, a channel shuffled feedforward neural network CSFFN and a grouped iterative convolution GIConv; The grouped iterative convolution GIConv includes: dividing the input feature map into four groups according to the channel dimension, combining the channel attention and spatial attention mechanisms to generate enhanced features, obtaining weight parameters through several convolutional layers, and grouping the cropped input feature map, inputting the weight parameters into the iterative enhancement function for low-light enhancement and exposure suppression, and splicing to generate enhanced feature maps; The encoder residual block EB consists of 3 Convolutional layer and 2 PReLU activation functions; The hybrid attention module HAB includes 1 spatial attention module, 2 Convolutional layer, 1 A grouped convolutional layer with a Softmax activation function and a learnable positional encoding; The channel shuffled feedforward neural network CSFFN includes a gating module and a Convolutional layers; The decoder consists of 2 encoder residual blocks EB, 1 decoder residual block DB, 3 Decoder CNN-Transformer modules and 1 Convolutional layers; The decoder residual block DB includes 4 Convolutional layer and 2 PReLU activation functions; The Decoder CNN-Transformer module includes 1 adaptive deformable void convolution block, layer normalization LN, 1 hybrid attention module HAB, 1 channel shuffled feedforward neural network CSFFN, 1 upsampling layer and 1 decoder residual block DB.

2. The method according to claim 1, characterized in that The processing flow of the hybrid attention module HAB is as follows: Input feature map Generated by the spatial attention module , and with Perform element-wise addition to get ; Feature Map through Convolutional layer, After the group convolution layer, it is divided into and , and reshape into and ; and After matrix multiplication, the Softmax activation function and Perform matrix multiplication to obtain feature maps ; Feature Map Reshape, then reshape Convolutional layer, and perform element-wise addition operation with learnable positional encoding to generate feature maps .

3. The method according to claim 1, characterized in that In the grouped iterative convolution GIConv, the input feature map Divided into 4 groups according to the channel, channel attention and spatial attention and Convolutional layer generation , after being spliced ​​with each group of features Convolutional layer generates weight parameters and ,Will After cropping to the [0,1] interval, it is divided into four groups again according to the channel dimension, and we get and , combined with the weight parameter and Enter the iterative enhancement activation function IEF and satisfy the following formula: ； Where: Indicates the The feature map of channel dimension, , Iterate for IEF The output result after the second for In the The weight parameter value at the iteration.

4. The method according to claim 1, wherein In the adaptive deformable hole convolution block, the input feature map , respectively, after the void ratio of 1, 2, 3 Deformable convolution, generating feature maps and , combined with the dynamic weights generated by average pooling, convolution, GeLU, and Softmax activation functions, and then concatenated with the input features. Convolutional layer output feature map .

5. The method according to claim 1, wherein In step 4, the training set images input to the network are also randomly flipped horizontally and vertically and randomly cropped to a size of (256, 256, 3).

6. A CNN-Transformer-based low-light image multi-task restoration system, used to implement the CNN-Transformer-based low-light image multi-task restoration method according to any one of claims 1 to 5, characterized in that: The system comprises: Data acquisition module: used to obtain low-light blurry images and normal-light clear image datasets; Network building module: used to build a CNN-Transformer network containing encoders, decoders and skip connections; Loss function module: used to calculate SmoothL1 loss, perceptual loss and total loss; Training module: used for data enhancement, network training and parameter optimization; Reasoning module: used to input validation set images and output enhanced results.

Images