Low illumination image brightness enhancement and super-resolution method based on dual-channel codec

Abstract

The present invention relates to a low-light image brightness enhancement and super-resolution method based on a dual-path codec, and belongs to the field of computer vision image technology. In the first step, a dark light image is extracted by an encoder with shared parameters to obtain a set of feature maps; in the second step, the feature maps are sent to a super-resolution decoder for decoding to obtain super-resolution feature maps; in the third step, the feature maps output by the encoder in the first step and the super-resolution feature maps in the second step are respectively pooled to obtain two feature vectors, which are weighted and fused using an attention mechanism, and then sent to the low-light decoder for decoding. Finally, the outputs of the two decoders are post-processed to obtain the corresponding image after super-resolution. The present invention is reasonably designed. In response to the defects of the current low-light enhancement method, the low-light enhancement and image super-resolution tasks are combined to improve the visual effect of the reconstructed image, and overall good results are achieved in the low-light enhancement and super-resolution comprehensive tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision images, and in particular to a low-illumination image brightness enhancement and super-resolution method based on a dual-channel codec. Background Art

[0002] With technological advancements, mobile devices are becoming increasingly popular, making image acquisition increasingly convenient. However, during image acquisition, limitations in the camera, shooting environment, and lighting conditions often result in difficult-to-understand low-light images, and low-light image quality is generally low. For example, although images captured by surveillance cameras are labeled as high-resolution, they are actually of very low quality and suffer from significant noise, especially at night. Mobile phone photography, inadequate ambient light, can produce images that are too dark, even leading to noise, blur, and loss of detail. These issues not only affect the visual quality of the image but also create difficulties for subsequent applications. Therefore, post-processing is often required to enhance the image quality while restoring brightness. However, existing low-light enhancement methods often focus solely on restoring image brightness, while neglecting image quality improvements, including detail enhancement. Most existing super-resolution methods assume that the input image is a high-quality, high-resolution ground-truth image obtained through some known or unknown downsampling method. This means that the vast majority of super-resolution methods target input images under normal lighting conditions, which are also noise-free and high-quality. Therefore, when we need to recover high-quality, high-resolution bright-light images from low-light images, simply cascading current low-light methods and super-resolution methods is not enough. Therefore, how to restore image brightness while improving image resolution and further enhance image quality is an important but challenging research topic in the field of computer vision.

[0003] Low-light image enhancement (LLE) addresses low brightness, low contrast, and noise issues found in poorly illuminated images, restoring image brightness and improving visual quality. This technology provides crucial technical support for better computer observation, analysis, and image processing, and has significant application value in many fields, including medical imaging and surveillance systems.

[0004] Traditional low-light image enhancement methods primarily focus on improving image contrast. Histogram equalization (HE) and its variants expand the image's limited dynamic range by adjusting the image's histogram. Retinex theory, which considers an image as the product of a reflective layer and an illumination layer, enhances image brightness by estimating and adjusting the illumination layer. These methods can only restore image brightness to a certain extent and are unable to recover blurred details in low-light images.

[0005] Benefiting from the development of deep learning, a growing number of researchers are applying neural networks to low-light enhancement. For example, there are low-light enhancement techniques such as LLNet, which trains convolutional neural networks using data simulated with random gamma correction; EnlightenGAN, which trains generative adversarial networks using unpaired data; Zero-DCE, which uses a convolutional neural network to learn a mapping from an image to a curve and applies the curve to the original image to adjust the image pixel values; DeepUPE, which estimates the mapping from an image to an illumination map using a convolutional neural network; KinD, which uses a convolutional neural network to decompose an image into a reflection layer and an illumination layer, performs denoising on the reflection layer, and adjusts the brightness of the illumination layer using a scaling factor; and PieNet, which uses a convolutional network to learn a preference vector representing user preferences to guide the brightness enhancement process. Similarly, these existing deep learning-based methods can only restore image brightness but cannot improve image quality, detail, or resolution. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology. In order to address the shortcomings of current low-light enhancement methods, namely the low quality of restored bright-light images, lack of details, and poor subjective visual effects, the present invention cleverly combines low-light enhancement and image super-resolution tasks, and proposes a low-light image brightness enhancement and super-resolution method based on a dual-channel codec. The method has a reasonable design, restores bright-light images of high quality, and is relatively lightweight.

[0007] The present invention solves the technical problem by adopting the following technical solutions:

[0008] A low-light image brightness enhancement and super-resolution method based on a dual-channel codec includes the following steps:

[0009] Step 1: At the image input stage, randomly use data augmentation technology to transform the sample content;

[0010] Step 2: Input the original dark image after data enhancement in step 1 into the network. The encoder, which shares all parameters, extracts features from the input dark image. The encoder includes operations such as convolution and downsampling. The size of the convolution kernel is 3×3, but the number of output channels of the convolution is 32, 64, 128, 256, and 512 respectively. The encoder outputs a set of feature maps.

[0011] Step 3: The output feature map of step 2 is first sent to the super-resolution decoder for decoding. The decoder includes operations such as convolution, transposed convolution, upsampling, and feature map cascade. The feature map after four upsampling and convolution is sent to the detail enhancement module for processing. At the same time, a skip connection structure is used to timely supplement low-level information to obtain a super-resolution feature map.

[0012] Step 4: Send the feature map output by the encoder in step 2 and the super-resolution feature map obtained in step 3 to the low-light decoder for decoding. First, perform pooling operations to obtain the low-light feature vector and the super-resolution feature vector, respectively. Then, use the attention mechanism to perform weighted fusion on the two feature vectors, and then perform four upsampling and convolution on them to finally output the low-light feature map.

[0013] Step 5: Upsample and reduce the dimension of the super-resolution feature map and low-light feature map obtained in steps 3 and 4, respectively. Sub-pixel convolution is used for upsampling, and the number of output channels of the reduced-dimensional convolution is consistent with the number of channels of the input original image, thereby obtaining the super-resolution bright light image and dark light image, respectively.

[0014] Furthermore, the specific method of data enhancement in step 1 includes the following steps:

[0015] (1) If it is in the testing phase, directly proceed to step 2 of claim 1; if it is in the training phase, perform the following operations;

[0016] (2) The data augmentation method of the input dark light image is determined with a probability of 1 / 8, including no flip, vertical flip, 90° counterclockwise rotation, 90° counterclockwise rotation and vertical flip, 180° rotation, 180° rotation and vertical flip, 270° counterclockwise rotation, and 270° counterclockwise rotation and vertical flip;

[0017] (3) In the data-augmented input image, randomly locate an image block of size 128×128 and input it into the network model.

[0018] Furthermore, the specific details of step 2 include the following:

[0019] (1) The network structure of the encoder is shaped like the letter V, which contains 10 convolutional layers. Downsampling is performed after every two convolutional layers, forming a step-by-step downsampling structure to extract features from the input dark light image.

[0020] (2) The convolution kernel size of each convolutional layer is 3×3, without changing the feature map size, but changing the number of output channels to 32, 64, 128, 256, and 512 respectively. Then, the maximum pooling is used to reduce the feature map size to 1 / 2 of the original size, compressing and fusing spatial features, preserving texture content, expanding the receptive field of the convolutional network, and extracting more semantic information.

[0021] (3) All feature maps output by the five groups of convolutions are used as the output of this stage.

[0022] Furthermore, the specific details of step 3 include the following:

[0023] (1) The super-resolution decoder receives all feature maps output by the encoder as input, and gradually convolves and upsamples them. There are four upsamplings in total, which correspond one-to-one to the downsampling in step 2.

[0024] (2) First, the feature map after the fourth downsampling is converted into a 512-dimensional feature vector through global average pooling to extract global information, and then it is expanded into a feature map through dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling;

[0025] (3) Each upsampling first uses transposed convolution to upsample the feature map by a factor of 2, and then concatenates it with the output of the corresponding layer in the encoder in the channel dimension with the same size, so as to timely supplement the spatial information lost due to downsampling when extracting features, and at the same time merge deep semantic feature information;

[0026] (4) Perform two 3×3 convolutions on the cascaded feature map, keeping the number of channels unchanged, to further reconstruct the image information, and repeat this four times;

[0027] (5) The feature map after the fourth upsampling is sent to the detail enhancement module, and the feature map output by the first group of convolutional layers of the encoder (without downsampling) is jump-connected to the output of the detail enhancement module to supplement the global information and obtain a super-resolution feature map.

[0028] Furthermore, the specific details of step (2) include the following:

[0029] ① For the obtained 512-dimensional feature vector, the number of channels is changed to 256 through a fully connected layer;

[0030] ② Expand the 256-dimensional feature vector to a 1×1×256 feature map;

[0031] ③ The feature map obtained in ② is copied by dimension to obtain a feature map of H×W×256, where H and W are the height and width of the feature map before the fourth downsampling of the encoder.

[0032] Furthermore, the specific details of step (5) include the following:

[0033] ① The input feature map is passed through 3×3 convolution, activation function, and 3×3 convolution to obtain enhanced detail information;

[0034] ② Reduce the dimension of the input feature map through 1×1 convolution to obtain global residual information;

[0035] ③Add the feature maps obtained in ① and ② pixel by pixel to obtain the enhanced feature map as the output of the detail enhancement module.

[0036] Furthermore, the specific details of step 4 include the following:

[0037] (1) Perform global average pooling on the feature map obtained by the last set of convolutions in step 2 to obtain the low-light feature vector;

[0038] (2) Global average pooling is also performed on the super-resolution feature map obtained in step 3 to obtain a super-resolution feature vector;

[0039] (3) Use the attention mechanism to perform weighted fusion on the two feature vectors;

[0040] (4) The fused feature vector is gradually convolved and upsampled. There are four upsamplings in total, which correspond one to one with the downsampling in step 2.

[0041] (5) First, the weighted fused feature vector is expanded into a feature map by dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling;

[0042] (6) Each upsampling first uses transposed convolution to upsample the feature map by a factor of 2, and then concatenates it with the output of the corresponding layer in the encoder in the channel dimension with the same size, so as to timely supplement the spatial information lost due to downsampling when extracting features, and at the same time merge deep semantic feature information;

[0043] (7) Perform two 3×3 convolutions on the cascaded feature map, keeping the number of channels unchanged, and further reconstruct the image information. This is repeated four times to obtain a low-light enhanced feature map.

[0044] Furthermore, the specific details of step (3) include the following:

[0045] ① The two eigenvectors are weighted and summed through a learnable parameter α, where α is the weight of the low-light eigenvector and (1-α) is the weight of the super-resolution eigenvector. The sum of the two is the fused eigenvector.

[0046] Furthermore, the specific details of step 5 include the following:

[0047] (1) If it is in the inference and testing stage, sub-pixel convolution and dimensionality reduction convolution are directly performed on the low-light feature map to obtain a super-resolution bright light image;

[0048] (2) If it is in the training stage, sub-pixel convolution and dimensionality reduction convolution are performed on the super-resolution feature map and the low-light feature map respectively to obtain the super-resolution dark light image and bright light image.

[0049] Furthermore, the specific details of steps (1) and (2) include the following:

[0050] ① The input feature map is upgraded through 3×3 convolution. Since it is a 2x super-resolution, the number of pixels is 4 times the original one, so the number of channels of the feature map is also increased by 4 times;

[0051] ② Arrange the extra pixels in the channel dimension of the feature map after dimensionality upgrade to the original channel, so that the width and height of the feature map are doubled respectively, and the number of channels is reduced to the same as the initial number of channels;

[0052] ③Finally, a 3×3 convolution is performed to output a three-channel super-resolution dark or bright image.

[0053] The advantages and positive effects of the present invention are:

[0054] 1. This invention is rationally designed. To address the shortcomings of current low-light enhancement methods—the restored bright-light images are of low quality, lack details, and suffer from poor subjective visual effects—it cleverly combines low-light enhancement and image super-resolution tasks. The result is a rationally designed, high-quality restored bright-light images and relatively lightweight low-light image brightness enhancement and super-resolution method based on a dual-channel codec.

[0055] 2. The present invention divides the low-light image brightness enhancement and super-resolution enhancement process based on a dual-path codec into four stages. In the first stage, a parameter-sharing encoder is used to extract features from the input dark-light image to obtain a set of feature maps. In the second stage, the obtained feature maps are first sent to the super-resolution decoder for decoding, and the decoded feature maps are pooled to obtain super-resolution feature vectors. In the third stage, a pooling operation is performed on the feature maps output by the encoder in the first stage to obtain a low-light feature vector, which is weightedly fused with the super-resolution feature vector obtained in the second stage using an attention mechanism and then sent to the low-light decoder for decoding. In the fourth stage, the outputs of the two decoders are upsampled and convolved respectively to obtain a super-resolution bright-light image and a dark-light image, and the loss is calculated with the corresponding true-value image to constrain the network model. The present invention does not require further processing of the output of the super-resolution encoder during the inference process, so the model complexity is low. While achieving good low-light enhancement and super-resolution effects, it maintains a relatively fast inference speed, and overall achieves good results in the low-light enhancement and super-resolution combined tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the overall flow chart of the low-light image brightness enhancement and super-resolution enhancement network of the present invention;

[0057] Figure 2 It is a structural diagram of the encoder part of the present invention;

[0058] Figure 3 is a structural diagram of a super-resolution decoder portion of the present invention;

[0059] Figure 4 It is a structural diagram of the low-light enhancement decoder part of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0061] A low-light image brightness enhancement and super-resolution method based on a dual-channel codec, such as Figures 1 to 4 As shown, the following steps are included:

[0062] Step S1: During the image input phase, randomly use data augmentation technology to transform the sample content;

[0063] Step S2: Input the original dark image after data enhancement in step S1 into the network. The encoder, which shares all parameters, performs feature extraction on the input dark image. The encoder includes operations such as convolution and downsampling. The size of the convolution kernel is always 3×3, but the number of output channels of the convolution is 32, 64, 128, 256, and 512, respectively. The encoder outputs a set of feature maps.

[0064] Step S3: The output feature map of step 2 is first sent to the super-resolution decoder for decoding. The decoder includes operations such as convolution, transposed convolution, upsampling, and feature map cascade. The feature map after four upsampling and convolution is sent to the detail enhancement module for processing. At the same time, a skip connection structure is used to timely supplement low-level information to obtain a super-resolution feature map.

[0065] Step S4: Send the feature map output by the encoder in step 2 and the super-resolution feature map obtained in step 3 to the low-light decoder for decoding. First, perform a pooling operation to obtain a low-light feature vector and a super-resolution feature vector. Then, use the attention mechanism to perform weighted fusion on the two feature vectors. Then, perform four upsampling and convolution on them, and finally output the low-light feature map.

[0066] Step S5: Upsample and reduce the dimension of the super-resolution feature map and low-light feature map obtained in step S3 and step S4, respectively. The upsampling uses sub-pixel convolution, and the number of output channels of the reduced dimension convolution is consistent with the number of channels of the input original image, thereby obtaining a super-resolution bright light image and a dark light image, respectively.

[0067] The specific implementation method of step S2 is as follows:

[0068] Step S2.1: The encoder's network structure is shaped like the letter V and consists of 10 convolutional layers. Downsampling is performed after every two convolutional layers, forming a progressive downsampling structure to extract features from the input dark-light image.

[0069] In step S2.2, the convolution kernel size of each convolutional layer is 3×3, and the feature map size is not changed. However, the number of output channels is changed to 32, 64, 128, 256, and 512, respectively. Maximum pooling is then used to reduce the feature map size to half of its original size. This compresses and fuses spatial features, preserves texture content, expands the receptive field of the convolutional network, and extracts more semantic information.

[0070] Step S2.3: All feature maps output by the five groups of convolutions are used as the output of this stage.

[0071] The specific implementation method of step S3 is as follows:

[0072] Step S3.1: The super-resolution decoder receives all feature maps output by the encoder as input, and gradually convolves and upsamples them. There are four upsampling steps in total, corresponding one-to-one to the downsampling in step S2.

[0073] Step S3.2: First, the feature map after the fourth downsampling is converted into a 512-dimensional feature vector through global average pooling to extract global information, and then it is expanded into a feature map through dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling;

[0074] In step S3.3, each upsampling step first uses transposed convolution to upsample the feature map by a factor of 2 and concatenate it with the output of the corresponding layer in the encoder in the channel dimension. This replenishes the spatial information lost due to downsampling when extracting features and merges deep semantic feature information.

[0075] Step S3.4: Perform two 3×3 convolutions on the concatenated feature map, keeping the number of channels unchanged, to further reconstruct the image information. Repeat this process four times.

[0076] Step S3.5: Send the feature map after the fourth upsampling and convolution to the detail enhancement module, and jump-connect the feature map output by the first group of convolutional layers of the encoder (without downsampling) to the output of the detail enhancement module to supplement the global information and obtain a super-resolution feature map.

[0077] The specific implementation method of step S3.2 is as follows:

[0078] S3.2.1. Pass the obtained 512-dimensional feature vector through a fully connected layer to change the number of channels to 256.

[0079] S3.2.2. Expand the 256-dimensional feature vector to a 1×1×256 feature map.

[0080] S3.2.3. Copy the feature map obtained in S3.2.2 by dimension to obtain a feature map of H×W×256, where H and W are the height and width of the feature map before the fourth downsampling of the encoder.

[0081] The specific implementation method of step S3.5 is as follows:

[0082] S3.5.1. Apply 3×3 convolution, activation function, and 3×3 convolution to the input feature map to obtain enhanced detail information.

[0083] S3.5.2. Reduce the dimension of the input feature map by 1×1 convolution to obtain global residual information;

[0084] S3.5.3. Add the feature maps obtained in S3.5.1 and S3.5.2 pixel by pixel to obtain the enhanced feature map as the output of the detail enhancement module.

[0085] The specific implementation method of step S4 is as follows:

[0086] Step S4.1, perform global average pooling on the feature map obtained by the last set of convolutions in step S2 to obtain a low-light feature vector;

[0087] Step S4.2: Perform global average pooling on the super-resolution feature map obtained in step S3 to obtain a super-resolution feature vector;

[0088] Step S4.3: Use the attention mechanism to perform weighted fusion on the two feature vectors;

[0089] Step S4.4: gradually convolve and upsample the fused feature vector, with a total of four upsamplings, corresponding one-to-one to the downsampling in step S2;

[0090] Step S4.5: First, the weighted fused feature vector is expanded into a feature map by dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling.

[0091] Step S4.6: Each upsampling step first uses transposed convolution to upsample the feature map by a factor of 2 and concatenate it with the output of the corresponding layer in the encoder in the channel dimension. This replenishes the spatial information lost due to downsampling when extracting features and also incorporates deep semantic feature information.

[0092] Step S4.7: Perform two 3×3 convolutions on the cascaded feature map, keeping the number of channels unchanged, and further reconstruct the image information. Repeat this four times to obtain a low-light enhanced feature map.

[0093] The specific implementation method of step S4.3 is as follows:

[0094] Step S4.3.1. Perform weighted summation on the two eigenvectors using a learnable parameter α, where α is the weight of the low-light eigenvector and (1-α) is the weight of the super-resolution eigenvector. The sum of the two is the fused eigenvector.

[0095] The specific implementation method of step S5 is as follows:

[0096] Step S5.1: If in the inference and testing phase, directly perform sub-pixel convolution and dimensionality reduction convolution on the low-light feature map to obtain a super-resolution bright-light image;

[0097] Step S5.2: If the image is in the training phase, sub-pixel convolution and dimensionality reduction convolution are performed on the super-resolution feature map and the low-light feature map, respectively, to obtain a super-resolution dark-light image and a bright-light image.

[0098] The specific implementation method of step S5.2 is as follows:

[0099] Step S5.2.1: Upscale the input feature map using a 3×3 convolution. Since the super-resolution is 2x, the number of pixels is 4 times the original, so the number of channels in the feature map is also increased by 4 times.

[0100] Step S5.2.2: Arrange the extra pixels in the channel dimension of the feature map after dimensionality increase to the original channel, so that the width and height of the feature map are doubled respectively, and the number of channels is reduced to the same as the initial number of channels;

[0101] Step S5.2.3: Finally, a 3×3 convolution is performed to output the three-channel super-resolution dark and bright images.

[0102] Through the above steps, a normal brightness image after super resolution can be obtained.

[0103] Finally, we train the network with the goal of minimizing the absolute error loss function (L1 loss function) and the mean square error loss function (L2 loss function). We use PSNR (Peak Signal to Noise Ratio) to quantitatively evaluate network performance and qualitatively evaluate network performance through visual effects. The method is as follows:

[0104] Test environment: Python 3.7; TensorFlow framework; Ubuntu 16.04 system; NVIDIA GTX 2080ti GPU

[0105] Test sequence: The selected dataset is the Low-Light dataset (LOL) for low-light enhancement, which contains 500 pairs of low-light-normal-light image pairs; the high-quality dataset (MIT-Adobe FiveK) for image enhancement, which contains 5 groups (A / B / C / D / E) of 5,000 pairs of low-light-normal-light image pairs.

[0106] Testing method: 15 image pairs were selected from League of Legends (LOL) and 500 image pairs were selected from Group C in MIT-Adobe FiveK to quantitatively and qualitatively evaluate the network performance.

[0107] Test indicators: The present invention uses the PSNR indicator for evaluation, calculates the indicator data for different popular algorithms and then compares the results. In addition, the visual effects of the same image after being enhanced by different methods are compared, proving that the present invention can achieve better results in the task of combining low-light enhancement and super-resolution.

[0108] Any matters not described in the present invention are applicable to the prior art.

[0109] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A low-light image brightness enhancement and super-resolution method based on a dual-channel codec, characterized by The following steps are involved: Step 1: At the image input stage, randomly use data augmentation technology to transform the sample content; Step 2: Input the dark-light image after data enhancement in step 1 into the network. The encoder that shares all parameters performs feature extraction on the input dark-light image. The encoder includes convolution and downsampling. The size of the convolution kernel is 3×3, but the number of output channels of the convolution is 32, 64, 128, 256, and 512 respectively. The encoder outputs a set of feature maps. Step 3: The output feature map of step 2 is first sent to the super-resolution decoder for decoding. The decoder includes convolution, transposed convolution, upsampling, and feature map cascade. The feature map after four upsampling and convolution is sent to the detail enhancement module for processing. At the same time, a skip connection structure is used to timely supplement low-level information to obtain a super-resolution feature map. Step 4: Send the feature map output by the encoder in step 2 and the super-resolution feature map obtained in step 3 to the low-light decoder for decoding. First, perform pooling operations on them to obtain the low-light feature vector and the super-resolution feature vector. Then, use the attention mechanism to perform weighted fusion on the two feature vectors. Then, perform four upsampling and convolution on them, and finally output the low-light feature map. Step 5: Upsample and reduce the dimension of the super-resolution feature map and low-light feature map obtained in steps 3 and 4, respectively. Sub-pixel convolution is used for upsampling, and the number of output channels of the reduced-dimensional convolution is consistent with the number of channels of the input original image, thereby obtaining the super-resolution bright light image and dark light image, respectively.

2. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The specific method of data enhancement in step 1 includes the following steps: (1) If it is in the testing phase, go directly to step 2; if it is in the training phase, perform the following operations; (2) The data augmentation method of the input dark light image is determined with a probability of 1 / 8, including no flip, vertical flip, 90° counterclockwise rotation, 90° counterclockwise rotation and vertical flip, 180° rotation, 180° rotation and vertical flip, 270° counterclockwise rotation, and 270° counterclockwise rotation and vertical flip; (3) In the data-augmented input image, randomly locate an image block of size 128×128 and input it into the network model.

3. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The specific details of step 2 include the following: (1) The network structure of the encoder is shaped like the letter V, which contains 10 convolutional layers. Downsampling is performed after every two convolutional layers, forming a step-by-step downsampling structure to extract features from the input dark light image. (2) The convolution kernel size of each convolutional layer is 3×3, without changing the feature map size, but changing the number of output channels to 32, 64, 128, 256, and 512 respectively. Then, the maximum pooling is used to reduce the feature map size to 1 / 2 of the original size, compressing and fusing spatial features, preserving texture content, expanding the receptive field of the convolutional network, and extracting more semantic information. (3) All feature maps output by the five groups of convolutions are used as the output of this stage.

4. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The specific details of step 3 include the following: (1) The super-resolution decoder receives all feature maps output by the encoder as input, and gradually convolves and upsamples them. There are four upsamplings in total, which correspond one-to-one to the downsampling in step 2. (2) First, the feature map after the fourth downsampling is converted into a 512-dimensional feature vector through global average pooling to extract global information, and then it is expanded into a feature map through dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling; (3) Each upsampling first uses transposed convolution to upsample the feature map by a factor of 2, and then concatenates it with the output of the corresponding layer in the encoder in the channel dimension with the same size, so as to timely supplement the spatial information lost due to downsampling when extracting features, and at the same time merge deep semantic feature information; (4) Perform two 3×3 convolutions on the cascaded feature map, keeping the number of channels unchanged, to further reconstruct the image information, and repeat this four times; (5) The feature map after the fourth upsampling and convolution is sent to the detail enhancement module, and the feature map output by the first group of convolutional layers of the encoder is jump-connected to the output of the detail enhancement module to supplement the global information and obtain a super-resolution feature map.

5. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 4, characterized in that: The specific details of step (2) include the following: ① For the obtained 512-dimensional feature vector, the number of channels is changed to 256 through a fully connected layer; ② Expand the 256-dimensional feature vector to a 1×1×256 feature map; ③ The feature map obtained in ② is copied by dimension to obtain a feature map of H×W×256, where H and W are the height and width of the feature map before the fourth downsampling of the encoder.

6. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 4, characterized in that: The detail enhancement module of step (5) includes the following contents: ① The input feature map is passed through 3×3 convolution, activation function, and 3×3 convolution to obtain enhanced detail information; ② Reduce the dimension of the input feature map through 1×1 convolution to obtain global residual information; ③Add the feature maps obtained in ① and ② pixel by pixel to obtain the enhanced feature map as the output of the detail enhancement module.

7. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The specific details of step 4 include the following: (1) Perform global average pooling on the feature map obtained by the last set of convolutions in step 2 to obtain the low-light feature vector; (2) Global average pooling is also performed on the super-resolution feature map obtained in step 3 to obtain a super-resolution feature vector; (3) Use the attention mechanism to perform weighted fusion on the two feature vectors; (4) The fused feature vector is gradually convolved and upsampled. There are four upsamplings in total, which correspond one to one with the downsampling in step 2. (5) First, the weighted fused feature vector is expanded into a feature map by dimension replication. The feature map has the same size and number of channels as the feature map before the fourth downsampling; (6) Each upsampling first uses transposed convolution to upsample the feature map by a factor of 2, and then concatenates it with the output of the corresponding layer in the encoder in the channel dimension with the same size, so as to timely supplement the spatial information lost due to downsampling when extracting features, and at the same time merge deep semantic feature information; (7) Perform two 3×3 convolutions on the cascaded feature map, keeping the number of channels unchanged, and further reconstruct the image information. This is repeated four times to obtain a low-light enhanced feature map.

8. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The attention mechanism of step (3) includes the following: ① The two eigenvectors are weighted and summed through a learnable parameter α, where α is the weight of the low-light eigenvector and (1-α) is the weight of the super-resolution eigenvector. The sum of the two is the fused eigenvector.

9. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 1, characterized in that: The specific details of step 5 include the following: (1) If it is in the inference and testing stage, sub-pixel convolution and dimensionality reduction convolution are directly performed on the low-light feature map to obtain a super-resolution bright light image; (2) If it is in the training stage, sub-pixel convolution and dimensionality reduction convolution are performed on the super-resolution feature map and the low-light feature map respectively to obtain the super-resolution dark light image and bright light image.

10. The low-light image brightness enhancement and super-resolution method based on a dual-channel codec according to claim 9, characterized in that: The sub-pixel convolution and dimensionality reduction convolution in steps (1) and (2) include the following: ① The input feature map is upgraded through 3×3 convolution. Since it is a 2x super-resolution, the number of pixels is 4 times the original one, so the number of channels of the feature map is also increased by 4 times; ② Arrange the extra pixels in the channel dimension of the feature map after dimensionality upgrade to the original channel, so that the width and height of the feature map are doubled respectively, and the number of channels is reduced to the same as the initial number of channels; ③Finally, a 3×3 convolution is performed to output a three-channel super-resolution dark or bright image.

Images