A low-illumination image enhancement method based on a multi-scale fusion U-Net network structure

Abstract

This invention discloses a low-light image enhancement method based on a multi-scale fusion U-Net network structure, comprising: S1, constructing a training dataset and a test sample set; S2, constructing a low-light enhancement network model; S3, jointly calculating multiple loss functions; S4, training the low-light enhancement network model; S5, converting and quantizing the trained low-light enhancement network model for deployment; S6, preprocessing the image data to be enhanced; S7, inputting the preprocessed image data into the low-light enhancement network model for image enhancement; and S8, the low-light enhancement network model outputting an enhanced normal illumination image. This invention reduces the number of model parameters while ensuring high accuracy, enhancing its portability and making low-light image enhancement technology a promising area for development in embedded systems.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and more specifically to a low-light image enhancement method based on a multi-scale fusion U-Net network structure. Background Technology

[0002] In today's information society, digital images are widely used because they can efficiently convey large amounts of information and conform to the intuitive perception of the human eye. However, due to environmental factors (such as rain, fog, and night) and limitations of shooting equipment (such as underexposure and low light), images often suffer from problems such as excessive noise, loss of detail, and poor visual effects, resulting in poor image quality. Therefore, image enhancement technology, especially low-light image enhancement, has become an important research direction in the field of digital image processing.

[0003] In low-light environments, images often appear dark, lack detail, and have poor visual quality, severely impacting information delivery and posing a challenge to image enhancement technologies. While deep learning-based image enhancement models have made significant progress in recent years, they still struggle to achieve ideal visual results under varying lighting conditions. Furthermore, these image enhancement models face difficulties in deployment on edge devices and exhibit low efficiency. Summary of the Invention

[0004] The present invention provides a low-light image enhancement method based on a multi-scale fusion U-Net network structure that can effectively solve the limitations of existing low-light image enhancement methods and greatly improve image quality, and can solve at least one of the above-mentioned technical problems.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A low-light image enhancement method based on a multi-scale fusion U-Net network structure includes the following steps:

[0007] S1. Construct the training dataset and test sample set;

[0008] S2. Construct a low-light enhancement network model, wherein the low-light enhancement network model includes:

[0009] An illumination coefficient estimation module is used to estimate the channel stretching coefficients corresponding to the RGB channels of an image pixel by pixel.

[0010] An encoder network structure is provided, comprising a residual downsampling module and an illumination enhancement module. The residual downsampling module is used for shallow feature extraction and downsampling operations, and the illumination enhancement module is used to improve the illumination of the image by utilizing channel attention.

[0011] The lighting enhancement module includes a multi-scale light extraction block and a local residual layer;

[0012] The illumination extraction block contains three feature extraction units. Each feature extraction unit consists of a convolutional layer, a normalization layer, and a ReLU activation function layer. Each feature extraction unit concatenates the output of the previous feature extraction unit into the input feature map.

[0013] The illumination enhancement module includes two cascaded illumination extraction blocks of different scales. The convolutional kernels of the two illumination extraction blocks are 3×3 and 5×5, respectively. The last layer of the illumination extraction block performs local residual on the input and output of the illumination extraction block. The output of the illumination enhancement module is composed of feature maps of the outputs of the two illumination extraction blocks of different scales spliced ​​together by the channel dimension.

[0014] The decoder network structure includes a residual upsampling module, a skip connection module, and an attention mechanism module. The decoder fuses the feature maps of the corresponding encoders to enhance the image reconstruction quality and generates contextual attention weights to optimize the expressive power of the feature maps.

[0015] S3. Jointly calculate multiple loss functions;

[0016] S4. Train the low-light enhancement network model;

[0017] S5. Quantitatively deploy the trained low-light enhancement network model;

[0018] S6. Preprocess the image data to be enhanced;

[0019] S7. Input the preprocessed image data into the low-light enhancement network model for image enhancement;

[0020] S8, the low-light enhancement network model outputs an enhanced normal-light image.

[0021] Furthermore, the illumination coefficient estimation module includes seven convolutional layers, each of which includes a 3×3 convolutional kernel and a ReLU activation function;

[0022] The first to fourth convolutional layers are connected sequentially. The input to the fifth convolutional layer is a feature map concatenated from the outputs of the third and fourth convolutional layers in the channel dimension. The input to the sixth convolutional layer is a feature map concatenated from the outputs of the second and fifth convolutional layers in the channel dimension. The seventh convolutional layer outputs an illumination enhancement coefficient map.

[0023] Furthermore, the residual downsampling module combines residual and dual convolution design. It reduces the dimensionality of the input feature map and extracts features through the first convolutional layer, while achieving channel alignment through 1×1 convolution. The second convolutional layer further enhances the feature representation. After all convolutional operations, a normalization layer and a ReLU activation function layer are added. Finally, the main path and input path features are fused through addition, achieving efficient spatial dimensionality reduction and feature preservation.

[0024] Furthermore, the residual upsampling module includes a main path and a residual path;

[0025] The main path consists of two convolutional layers, a normalization layer, and an activation function. The convolutional layers are used to extract local information of the input features, and the activation function is used to introduce non-linear characteristics.

[0026] The residual path includes a 1×1 convolutional layer for matching the number of channels and a normalization layer, which is used to superimpose the input features with the output of the main path;

[0027] The outputs of the main path and the residual path are added together and then passed through a deconvolution layer to amplify the spatial resolution, thereby achieving feature map upsampling.

[0028] Furthermore, the skip connection module includes two convolutional layers, each followed by a normalization layer and a ReLU activation function layer. The number of input and output channels of the convolutional layers is consistent, the kernel size is 3×3, and the padding is 1.

[0029] Furthermore, in S3, after each image reconstruction, a joint loss function needs to be calculated to determine the quality of the image reconstruction and optimize the network parameters;

[0030] The joint loss function is a weighted average of the reconstruction loss, perceptual loss, structural similarity loss, and color loss, and is as follows:

[0031]

[0032] Where, λ r , λ p , λ s and λ c These are the weights for reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. and L c These are reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. For joint losses;

[0033] Reconstruction loss The calculation formula is:

[0034]

[0035] Among them, E′ n This is the result of network training enhancement; Y is related to E. ′ n The corresponding expected normal illumination value;

[0036] Extracting E through the VGG network ′ n The features of Y will be defined based on the pre-trained VGG-16 network. Perceived loss The calculation formula is:

[0037]

[0038] Among them, w ij h ij and c ij The dimensions φ of each feature map in the VGG-16 network are described respectively. ij This represents the feature map obtained by the j-th convolutional layer of the i-th block in the VGG-16 network.

[0039] Structural similarity loss The calculation formula is:

[0040]

[0041]

[0042] Where SSIM is the image structural similarity, S and T represent the images to be compared, and μ S and μ T It is the average pixel value of the two images. and It is the variance of the two images, σ ST It is the covariance of the two images, and c1 and c2 are two small constants used to prevent the denominator from being zero;

[0043] Color loss The calculation formula is:

[0044]

[0045]

[0046] Among them, S p and T p Let S and T represent the RGB color vectors of pixel p, respectively. CA(·,·) represents a three-dimensional vector with RGB colors as the vector. The color loss between pixels is the angle between the two color vectors. The total color loss is the sum of the color losses of all pixels. The greater the color deviation between the two images, the greater the loss function.

[0047] Furthermore, in S4, the input to the low-light enhancement network model is a low-light image and a normally exposed image, and the output is a predicted reconstructed image. The training process further includes:

[0048] S41. The images to be trained are randomly divided into several batches, and each batch contains the same number of images;

[0049] S42. Use the batches of images to train and optimize the low-light enhancement network model until the calculated joint loss function reaches the loss threshold or the number of iterations reaches the number threshold.

[0050] Furthermore, S5 further includes:

[0051] S51. Replace the trained low-light image enhancement network model with the ONNX format, and then convert the ONNX format low-light image enhancement network model into a format supported by the hardware device.

[0052] S52. Quantize the low-light image enhancement network model in a format supported by the hardware device;

[0053] S53. Deploy the quantized low-light image enhancement network model on the edge device and load the quantized low-light image enhancement network model.

[0054] S54. Input the preprocessed image into the quantized low-light image enhancement network model, obtain the output result from the quantized low-light image enhancement network model, and save the image in an appropriate format and range using OpenCV.

[0055] Furthermore, the quantization process in S52 further includes:

[0056] S521. Convert the floating-point numbers in the low-light image enhancement network model into integer data to reduce the model's memory footprint and speed up inference. Calculate the scaling factor and offset as follows:

[0057]

[0058]

[0059] S522, For floating-point data X f Perform quantization and convert to uint8 type data X. q The calculation formula is:

[0060]

[0061] Among them, X max and Xmin X represents the maximum and minimum values ​​of a floating-point number, respectively. f Represents a floating-point number, X q This represents quantized uint8 type data. The round function performs rounding operations, and the clamp function is used to ensure that the quantized result X is within the specified range. q Within the interval [0, 255], the definition of clamp is:

[0062]

[0063] The `clamp` function restricts randomly changing values ​​to a given range, where `a` and `b` are constants and `x` is a variable.

[0064] Furthermore, the preprocessing in S6 further includes:

[0065] S61. Read the low-light image to be enhanced and convert it to RGB format;

[0066] S62. Convert the image data into a NumPy array and convert the data format to np.float32 to ensure the accuracy of subsequent processing;

[0067] S63. Convert the image data layout to NHWC format to ensure that the low-light image enhancement network model accepts data format consistent with the required data format.

[0068] The beneficial effects of this invention are reflected in:

[0069] 1. Enhanced low-light image processing capabilities: This invention combines a channel attention mechanism (CAM) and a spatial attention mechanism (SAM) residual module to effectively enhance low-light images during training. The channel attention mechanism can perform weighted correction on the image along the channel dimension, solving the color cast problem in low-light images, while the spatial attention mechanism constrains the regional deviations of the feature map in the spatial dimension, thereby effectively solving the spatial non-uniformity problem.

[0070] 2. This invention constructs an illumination enhancement module, employing a multi-scale illumination extraction block to extract illumination information at different scales from the image. Furthermore, the final layer of the illumination extraction block applies local residual learning to enhance the illumination effect, capturing details and brightness distribution within the image. The module restores the channel count of the feature map through a 1×1 convolutional kernel compression layer, while simultaneously preserving key brightness enhancement information through residual connections, thereby improving the network's ability to model illumination distribution.

[0071] 3. This invention employs a multi-scale fusion mechanism, which can extract global information (such as brightness distribution) and local information (such as texture and details) at different resolutions. It performs well in low-light image enhancement, effectively enhancing the structure and texture of the image, and achieving a good balance between brightness, structure and detail.

[0072] 4. This invention employs a relatively lightweight design scheme in its network design, optimizing the computational load of the model. These advantages make this method widely applicable to embedded devices, particularly suitable for low-light image enhancement tasks requiring low power consumption, high efficiency, and real-time processing. It can effectively improve image quality and adapt to embedded platforms. Attached Figure Description

[0073] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0074] Figure 1 This is a schematic diagram of the overall process of the low-light image enhancement method according to an embodiment of the present invention.

[0075] Figure 2 This is a schematic diagram of a low-light enhancement network model according to an embodiment of the present invention.

[0076] Figure 3 This is a schematic diagram of the lighting enhancement module structure according to an embodiment of the present invention.

[0077] Figure 4 This is a schematic diagram of the 3×3 light extraction block structure according to an embodiment of the present invention.

[0078] Figure 5 This is a set of comparison images showing the visualization effects of image enhancement processing on indoor scenes of the LOL dataset using various algorithms.

[0079] Figure 6 This is another set of comparison images showing the visualization effects of image enhancement processing on indoor scenes of the LOL dataset using various algorithms.

[0080] Figure 7 This is a structural block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] It should be noted that if the embodiments of the present invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied. Additionally, "multiple" refers to two or more.

[0083] See Figure 1 This invention provides a low-light image enhancement method based on a multi-scale fusion U-Net network structure, comprising the following steps:

[0084] S1. Construct the training dataset and test sample set;

[0085] S2. Construct a low-light enhancement network model;

[0086] S3. Jointly calculate multiple loss functions;

[0087] S4. Train the low-light enhancement network model;

[0088] S5. Quantitatively deploy the trained low-light enhancement network model;

[0089] S6. Preprocess the image data to be enhanced;

[0090] S7. Input the preprocessed image data into the low-light enhancement network model for image enhancement;

[0091] S8, the low-light enhancement network model outputs an enhanced normal-light image.

[0092] In this embodiment, in step S1, in order to train the low-light enhancement network model, it is necessary to construct a training dataset and a test sample set. The training dataset is a sequence set composed of multiple data pairs, each data pair consisting of a low-light image and a normal-light image.

[0093] First, multiple low-light images and corresponding normal-light images from various scenarios can be obtained as data pairs. This invention gathers 2289 image pairs from the publicly available datasets LOL, LOL-v1, and LOL-v2. The data content consists of low / normal light image pairs. 90% of the 2289 image pairs are the training set and 10% are the test set. The training set consists of 2074 image pairs, and the test set consists of 215 image pairs. All of these images are converted to PNG format. Before the images are input into the low-light enhancement network model for training, the images are randomly rotated (0°, 90°, 180°, 270°) and flipped as data augmentation.

[0094] See Figure 2 In this embodiment, in step S2, the low-light image is input into a multi-scale fusion U-Net network, and the enhanced image is output. The multi-scale fusion U-Net network includes:

[0095] An illumination coefficient estimation module is used to estimate the channel stretching coefficients corresponding to the RGB channels of an image pixel by pixel.

[0096] An encoder network structure is provided, comprising a residual downsampling module and an illumination enhancement module. The residual downsampling module is used for shallow feature extraction and downsampling operations, and the illumination enhancement module is used to improve the illumination of the image by utilizing channel attention.

[0097] The decoder network structure includes a residual upsampling module, a skip connection module, and an attention mechanism module, which includes an SSA module and a CBAM module. Feature maps from the corresponding encoders are fused in the decoder to enhance image reconstruction quality. Contextual attention weights are generated to optimize the expressive power of the feature maps, and the importance of features is further enhanced through channel attention and spatial attention mechanisms.

[0098] See Figure 3 In this embodiment, the illumination coefficient estimation module includes seven convolutional layers, each of which includes a 3×3 convolutional kernel and a ReLU activation function;

[0099] The first to fourth convolutional layers are connected sequentially. The input to the fifth convolutional layer is a feature map concatenated from the outputs of the third and fourth convolutional layers in the channel dimension. The input to the sixth convolutional layer is a feature map concatenated from the outputs of the second and fifth convolutional layers in the channel dimension. The seventh convolutional layer outputs an illumination enhancement coefficient map.

[0100] The network structure of the illumination coefficient estimation module is shown in Table 1 below. The input image first passes through convolutional layers Conv1 to Conv4 to extract features step by step. The convolutional layers use 3×3 kernels with padding of 1. After the convolutional layers, the ReLU activation function is used. Through multiple convolutional layers, higher-level features are gradually learned. The input of Conv5 is the feature map concatenated from the outputs of Conv3 and Conv4 along the channel dimension. The input of Conv6 is the feature map concatenated from the outputs of Conv2 and Conv5 along the channel dimension. The convolutional layer uses 3×3 kernels with padding of 1. After the convolutional layers, the ReLU activation function is used. The input of the Conv_Out layer is the feature map concatenated from the outputs of Conv1 and Conv6 along the channel dimension. The convolutional layer uses 3×3 kernels with padding of 1. Finally, the illumination enhancement coefficient map is output through Conv_Out. This coefficient is multiplied element-wise with the input image to complete the initial illumination enhancement of the image.

[0101] Table 1 Network of Illuminance Estimation Module

[0102]

[0103] See Figures 3-4 In this embodiment, the lighting enhancement module includes a multi-scale light extraction block and a local residual layer;

[0104] The illumination extraction block contains three feature extraction units. Each feature extraction unit consists of a convolutional layer, a normalization layer, and a ReLU activation function layer. Each feature extraction unit concatenates the output of the previous feature extraction unit into the input feature map.

[0105] The lighting enhancement module includes two cascaded light extraction blocks of different sizes. The convolution kernels of the two light extraction blocks are 3×3 and 5×5, respectively. The last layer of the light extraction block performs local residual on the input and output of the light extraction block.

[0106] The network structure of the illumination enhancement module is shown in Table 2 below. The illumination enhancement module further restores the channel number of the feature map through a compression layer with a 1×1 convolutional kernel, and at the same time maintains the key information of brightness enhancement through residual connection, thereby improving the network's ability to model the illumination distribution. The stitched feature map is further processed through a 3×3 convolutional layer, and then finally adjusted through another 3×3 convolutional layer. The output image is normalized by the Sigmoid activation function.

[0107] Table 2 Lighting Enhancement Module Network

[0108]

[0109] In this embodiment, the residual downsampling module combines residual and dual convolution design. The network structure is shown in Table 3 below. DownConv1 and DownConv2 represent downsampling convolutional layers. The first convolutional layer, DownConv1, reduces the dimensionality of the input feature map and extracts features. The Conv layer enhances feature representation. After all convolutional operations, a normalization layer and a ReLU activation function layer are applied. The DownConv2 layer uses a 1×1 convolutional kernel and a stride of 2 to directly map the resolution and number of channels of the input feature map to the target dimension as a residual term. Finally, the main path and input path features are fused through addition operations to achieve efficient spatial dimensionality reduction and feature preservation.

[0110] Table 3 Residual Subsampling Module Network

[0111]

[0112] In this embodiment, the network structure of the residual upsampling module is shown in Table 4 below. The residual upsampling module includes a main path and a residual path.

[0113] The main path consists of two convolutional layers, a normalization layer, and an activation function. The convolutional layers are used to extract local information of the input features, and the activation function is used to introduce non-linear characteristics.

[0114] The residual path includes a 1×1 convolutional layer for matching the number of channels and a normalization layer, which is used to superimpose the input features with the output of the main path;

[0115] The outputs of the main path and the residual path are added together and then passed through a deconvolution layer to amplify the spatial resolution, thereby upsampling the feature map and reducing the gradient vanishing problem.

[0116] In the residual upsampling and downsampling process, the second branch of the residual passes through a 1×1 convolutional layer and a normalization layer for matching the number of channels. This enables the superposition of input features and main path output, resulting in a relatively lightweight optimization design for the model and reducing the cost of efficiency input.

[0117] Table 4 Residual Upsampling Module Network

[0118]

[0119] In this embodiment, the network structure of the skip connection module is shown in Table 5 below. The skip connection module includes two convolutional layers, Conv1 and Conv2. Each convolutional layer is followed by a normalization layer and a ReLU activation function layer. The number of input and output channels of the convolutional layer is consistent. The kernel size is 3×3 and the padding is 1.

[0120] In the skip connection module, the input feature map and the convolutionally processed feature map are concatenated along the channel dimension, effectively preserving low-level feature information of the input and enhancing the network's feature representation capability. The concatenated feature map is then compressed using a 1×1 convolutional layer to ensure that the number of channels in the output feature map is consistent with that of the input feature map. Finally, the output undergoes a nonlinear transformation using the ReLU activation function to introduce nonlinear characteristics and enhance the model's expressive power.

[0121] Table 5. Jump Connection Module Network

[0122]

[0123] In this embodiment, in step S3, after each image reconstruction, a joint loss function needs to be calculated to determine the quality of the image reconstruction and optimize the network parameters.

[0124] The joint loss function is a weighted average of the reconstruction loss, perceptual loss, structural similarity loss, and color loss, and is as follows:

[0125]

[0126] Where, λ r , λ p , λ s and λ c These are the weights for reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. and L c These are reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. As a joint loss, the four losses are constrained to achieve noise removal, local detail restoration, and color deviation correction in the reconstructed image;

[0127] Joint loss functions play a crucial role in image reconstruction. By combining multiple loss functions, various aspects of image reconstruction can be comprehensively considered, thereby improving the quality of the final image. The weighted combination of these loss functions can be adjusted during training according to actual needs to ensure that the network achieves balanced and optimized results across different task objectives.

[0128] Reconstruction loss Directly measuring the reconstruction error at the pixel level can effectively ensure that the generated image is as visually similar as possible to the original image, avoiding large-scale visual distortion. The calculation formula is as follows:

[0129]

[0130] Among them, E ′ n This is the result of network training enhancement; Y is related to E.′ n The corresponding expected normal illumination value;

[0131] Perceived loss By calculating the differences between images in a high-level semantic feature space, the model captures texture, shape, and high-level semantic information, ensuring that the generated images are perceptually more similar to real images. E is extracted through the VGG network. ′ n The features of Y will be defined based on the pre-trained VGG-16 network. The calculation formula is as follows:

[0132]

[0133] Among them, w ij h ij and c ij The dimensions φ of each feature map in the VGG-16 network are described respectively. ij This represents the feature map obtained by the j-th convolutional layer of the i-th block in the VGG-16 network.

[0134] Structural similarity loss Focusing on the overall structure, edges, contrast, and other features of an image can prevent the generated image from becoming too blurry or losing its original structural features. The calculation formula is as follows:

[0135]

[0136]

[0137] Where SSIM is the image structural similarity, S and T represent the images to be compared, and μ S and μ T It is the average pixel value of the two images. and It is the variance of the two images, σ ST It is the covariance of the two images, and c1 and c2 are two small constants used to prevent the denominator from being zero;

[0138] Color loss To ensure that the color distribution of the generated image with normal illumination matches that of the original image, and to avoid color casts or unnatural tones, the calculation formula is as follows:

[0139]

[0140]

[0141] Among them, S p and T pLet S and T represent the RGB color vectors of pixel p, respectively. CA(·,·) represents a three-dimensional vector with RGB colors as the vector. The color loss between pixels is the angle between the two color vectors. The total color loss is the sum of the color losses of all pixels. The greater the color deviation between the two images, the greater the loss function.

[0142] In this embodiment, in step S4, the input to the low-light enhancement network model is a low-light image and a normally exposed image, and the output is a predicted reconstructed image. The training process further includes:

[0143] S41. The images to be trained are randomly divided into several batches, and each batch contains the same number of images;

[0144] S42. Use the batches of images to train and optimize the multi-scale fusion U-Net network, i.e. the low-light enhancement network model, until the calculated joint loss function reaches the loss threshold or the number of iterations reaches the number threshold.

[0145] As an embodiment of this invention, the selected experimental hardware platform configuration is as follows: CPU processor Intel(R) Xeon(R) CPU E5-2682 v4@2.5GHz, memory 64G, graphics card Nvidia GeForce RTX 3090, video memory 24G; software environment is Ubuntu 20.04, Python version 3.8.0, deep learning framework PyTorch version 1.11.0, CUDA version 11.3.1; the experimental software development environment is PyCharm 2023.2.1 and Matlab R2019a.

[0146] The low-light enhancement network model was trained in batches of 128 and iterated 300 times. This invention uses a learning rate scheduler; the initial learning rate is 0.0001, and the learning rate is adjusted to 0.1 at epochs 100 and 200. In the experiment, the hyperparameter λ in the joint loss function... r , λ p , λ s and λ c The values ​​are 1, 1, 3, and 1, respectively. The training dataset and test sample set are constructed as described above, and the integrated dataset is then input into the low-light enhancement network model for training.

[0147] In this embodiment, step S5 further includes:

[0148] S51. Replace the trained low-light image enhancement network model with the ONNX format, and then convert the ONNX format low-light image enhancement network model into a format supported by the hardware device.

[0149] S52. Quantize the low-light image enhancement network model in a format supported by the hardware device;

[0150] S53. Deploy the quantized low-light image enhancement network model on the edge device and load the quantized low-light image enhancement network model.

[0151] S54. Input the preprocessed image into the quantized low-light image enhancement network model, obtain the output result from the quantized low-light image enhancement network model, and save the image in an appropriate format and range using OpenCV.

[0152] The main quantization process in step S52 above is as follows:

[0153] Floating-point numbers in the low-light image enhancement network model are converted to integer data to reduce the model's memory footprint and speed up inference. The scaling factor and offset are calculated as follows:

[0154]

[0155]

[0156] For floating-point data X f Perform quantization and convert to uint8 type data X. q The formula is:

[0157]

[0158] Among them, X max and X min X represents the maximum and minimum values ​​of a floating-point number. f Represents a floating-point number, X q This represents quantized uint8 type data. The round function performs rounding operations, and the clamp function is used to ensure that the quantized result X is within the specified range. q Within the interval [0, 255], the definition of clamp is:

[0159]

[0160] The `clamp` function constrains randomly varying values ​​within a given interval, where `a` and `b` represent constants and `x` represents a variable. This process prevents overflow and distortion by limiting the data to the given interval, ensuring that the quantization results are suitable for hardware inference.

[0161] Through the quantization steps described above, the floating-point model can be converted into an integer model that is more suitable for hardware accelerators, thereby optimizing the model's running efficiency and memory usage, and improving inference performance.

[0162] In this embodiment, the preprocessing process in S6 further includes:

[0163] S61. Read the low-light image to be enhanced and convert it to RGB format;

[0164] S62. Convert the image data into a NumPy array and convert the data format to np.float32 to ensure the accuracy of subsequent processing;

[0165] S63. Convert the image data layout to NHWC format to ensure that the low-light image enhancement network model accepts data format consistent with the required data format.

[0166] See Figures 5-6 This invention provides comparative experiments on multiple low-light datasets, demonstrating the comparative results of different algorithms on the LOL dataset. (a) is the input low-light image, (l) is the corresponding real image (normal exposure image), (b)-(j) are the results of other comparative methods, and (k) is the result of the method proposed in this invention. (b) The HE method enhances the image with dense noise and severe color cast. (c) The images enhanced by AHE and (i) RUASNet methods are too dark, indicating that neither method can effectively enhance image brightness under low-light conditions, and is accompanied by a small amount of noise. (d) The LIME method has insufficient optimization of local brightness in low-light image processing, resulting in uneven processing effects in different areas, thus affecting the overall visual consistency of the image. (e) As can be seen from the figure, the MSR method significantly improves the brightness of low-light images, with better gradient effects and improved color saturation. However, the image sharpness is insufficient, and in high-brightness areas of the image, such as... Figure 5 The contrast of the calligraphy and paintings in the middle and later parts is insufficient. Figure 6 The background texture is unclear, and over-enhancement occurs, causing loss of image details. (f) EFINet and (h) SCINet methods are insufficient in brightness optimization, resulting in insufficient contrast between light and dark areas, making it difficult to discern details in dark areas of the image, and the image is accompanied by a lot of noise, resulting in poor visual quality. (j) The EFINet method largely restores the color reproduction of the image, but often causes loss of image details or the presence of random noise, and the image smoothness is insufficient. (g) URetinexNet and (k) the method of this invention... Figure 5 The enhancement effect is good, and the image clarity is high, but... Figure 6 In the previous method, the (g)URetinexNet method resulted in severe color cast in the enhanced image, and the contrast and color saturation of the wood grain texture of the background were low. In contrast, the image enhanced by the method of this invention is more natural, improves contrast and brightness, and restores the color and detail information of the image.

[0167] The comparison of the above experimental results shows that this method has a more obvious advantage over the other nine methods in terms of subjective visual effect.

[0168] In the LOL test dataset, the image quality of the enhanced images of the present invention and the enhanced images of existing methods is evaluated. The evaluation metrics are: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0169] Table 6 Comparison of Quantitative Data in LOL Test Dataset

[0170]

[0171] As shown in Table 6 above, the method of this invention is compared in detail with the current mainstream low-light image enhancement methods HE, AHE, LIME, MSR, Zero-DEC, UREtinexNet, SCINet, RUASNet, and EFINet on the LOL dataset using the two evaluation metrics PSNR and SSIM. In the evaluation metrics, a higher PSNR value and a higher SSIM value closer to 1 indicate better low-light enhancement and a better restoration of the image effect under normal lighting. Therefore, it can be seen that the method of this invention has a significant performance advantage over the current mainstream low-light image enhancement methods, with both PSNR and SSIM values ​​exceeding those of existing methods.

[0172] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the low-light image enhancement method based on the multi-scale fusion U-Net network structure described above.

[0173] See Figure 7 The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the low-light image enhancement method based on the multi-scale fusion U-Net network structure described above.

[0174] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the low-light image enhancement method based on the multi-scale fusion U-Net network structure described above.

[0175] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the low-light image enhancement method based on the multi-scale fusion U-Net network structure described above.

[0176] It should be noted that those skilled in the art will understand that all or part of the steps implemented in the embodiments of the present invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in hardware, it can be implemented entirely or partially by purchasing standard parts or modifications. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0177] In summary, the low-light image enhancement algorithm proposed in this invention can effectively preserve the color information of the image and non-uniformly enhance the image illumination, thus significantly improving the image enhancement effect compared to existing methods.

[0178] It should be understood that the examples and embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various modifications or changes based on them. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A low-light image enhancement method based on a multi-scale fusion U-Net network structure, characterized in that, Includes the following steps: S1. Construct the training dataset and test sample set; S2. Construct a low-light enhancement network model, wherein the low-light enhancement network model includes: An illumination coefficient estimation module is used to estimate the channel stretching coefficients corresponding to the RGB channels of an image pixel by pixel. An encoder network structure is provided, comprising a residual downsampling module and an illumination enhancement module. The residual downsampling module is used for shallow feature extraction and downsampling operations, and the illumination enhancement module is used to improve the illumination of the image by utilizing channel attention. The lighting enhancement module includes a multi-scale light extraction block and a local residual layer; The illumination extraction block contains three feature extraction units. Each feature extraction unit consists of a convolutional layer, a normalization layer, and a ReLU activation function layer. Each feature extraction unit concatenates the output of the previous feature extraction unit into the input feature map. The illumination enhancement module includes two cascaded illumination extraction blocks of different scales. The convolutional kernels of the two illumination extraction blocks are 3×3 and 5×5, respectively. The last layer of the illumination extraction block performs local residual on the input and output of the illumination extraction block. The output of the illumination enhancement module is composed of feature maps of the outputs of the two illumination extraction blocks of different scales spliced ​​together by the channel dimension. The decoder network structure includes a residual upsampling module, a skip connection module, and an attention mechanism module. The decoder fuses the feature maps of the corresponding encoders to enhance the image reconstruction quality and generates contextual attention weights to optimize the expressive power of the feature maps. S3. Jointly calculate multiple loss functions; S4. Train the low-light enhancement network model; S5. Quantitatively deploy the trained low-light enhancement network model; S6. Preprocess the image data to be enhanced; S7. Input the preprocessed image data into the low-light enhancement network model for image enhancement; S8, the low-light enhancement network model outputs an enhanced normal-light image.

2. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, The illumination coefficient estimation module includes seven convolutional layers, each of which includes a 3×3 convolutional kernel and a ReLU activation function; The first to fourth convolutional layers are connected sequentially. The input to the fifth convolutional layer is a feature map concatenated from the outputs of the third and fourth convolutional layers in the channel dimension. The input to the sixth convolutional layer is a feature map concatenated from the outputs of the second and fifth convolutional layers in the channel dimension. The seventh convolutional layer outputs an illumination enhancement coefficient map.

3. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, The residual downsampling module combines residual and double convolution design. It reduces the dimensionality of the input feature map and extracts features through the first convolutional layer, and achieves channel alignment through 1×1 convolution. Then, it further enhances the feature expression through the second convolutional layer. After all convolutional operations, it is followed by a normalization layer and a ReLU activation function layer. Finally, it fuses the main path and input path features through addition operation to achieve efficient spatial dimensionality reduction and feature preservation.

4. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, The residual upsampling module includes a main path and a residual path; The main path consists of two convolutional layers, a normalization layer, and an activation function. The convolutional layers are used to extract local information of the input features, and the activation function is used to introduce non-linear characteristics. The residual path includes a 1×1 convolutional layer for matching the number of channels and a normalization layer, which is used to superimpose the input features with the output of the main path; The outputs of the main path and the residual path are added together and then passed through a deconvolution layer to amplify the spatial resolution, thereby achieving feature map upsampling.

5. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, The skip connection module contains two convolutional layers, each followed by a normalization layer and a ReLU activation function layer. The number of input and output channels of the convolutional layers is consistent, the kernel size is 3×3, and the padding is 1.

6. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, In S3, after each image reconstruction, the joint loss function needs to be calculated to determine the quality of the image reconstruction and optimize the network parameters. The joint loss function is a weighted average of the reconstruction loss, perceptual loss, structural similarity loss, and color loss, and is as follows: Where, λ r , λ p , λ s and λ c These are the weights for reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. , , and L c These are reconstruction loss, perceptual loss, structural similarity loss, and color loss, respectively. For joint losses; Reconstruction loss The calculation formula is: in, It is the result of network training enhancement, where Y is related to... The corresponding expected normal illumination value; Extracted via VGG network The features of Y will be defined based on the pre-trained VGG-16 network. Perceived loss The calculation formula is: in, , and The dimensions of each feature map in the VGG-16 network are described separately. This represents the feature map obtained by the j-th convolutional layer of the i-th block in the VGG-16 network. Structural similarity loss The calculation formula is: Where SSIM represents image structural similarity, and S and T represent the images to be compared. and It is the average pixel value of the two images. and It is the variance of the two images. It is the covariance of the two images. and These are two small constants used to prevent the denominator from being zero; Color loss The calculation formula is: in, and Let S and T represent the RGB color vectors of pixel p, respectively. This means that RGB colors are used as three-dimensional vectors, the color loss between pixels is the angle between the two color vectors, and the total color loss is the sum of the color losses of all pixels. The greater the color deviation between the two images, the greater the loss function.

7. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 6, characterized in that, In step S4, the input to the low-light enhancement network model is a low-light image and a normally exposed image, and the output is a predicted reconstructed image. The training process further includes: S41. The images to be trained are randomly divided into several batches, and each batch contains the same number of images; S42. Use the batches of images to train and optimize the low-light enhancement network model until the calculated joint loss function reaches the loss threshold or the number of iterations reaches the number threshold.

8. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 7, characterized in that, S5 further includes: S51. Replace the trained low-light image enhancement network model with the ONNX format, and then convert the ONNX format low-light image enhancement network model into a format supported by the hardware device. S52. Quantize the low-light image enhancement network model in a format supported by the hardware device; S53. Deploy the quantized low-light image enhancement network model on the edge device and load the quantized low-light image enhancement network model. S54. Input the preprocessed image into the quantized low-light image enhancement network model, obtain the output result from the quantized low-light image enhancement network model, and save the image in an appropriate format and range using OpenCV.

9. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 8, characterized in that, The quantization process in S52 further includes: S521. Convert the floating-point numbers in the low-light image enhancement network model into integer data to reduce the model's memory footprint and speed up inference. Calculate the scaling factor and offset as follows: ； S522, Floating-point data Perform quantization and convert to uint8 data type. The calculation formula is: in, and These represent the maximum and minimum values ​​of the floating-point number, respectively. Represents floating-point numbers. This represents quantized uint8 type data. The round function performs rounding operations to the nearest integer. The function is used to ensure the quantization result Within the interval [0, 255], The definition of is: in, A function restricts randomly changing values ​​to a given interval, where both a and b are constants. It is represented as a variable.

10. The low-light image enhancement method based on a multi-scale fusion U-Net network structure as described in claim 1, characterized in that, The preprocessing process in S6 further includes: S61. Read the low-light image to be enhanced and convert it to RGB format; S62. Convert the image data into a NumPy array and convert the data format to np.float32 to ensure the accuracy of subsequent processing; S63. Convert the image data layout to NHWC format to ensure that the low-light image enhancement network model accepts data format consistent with the required data format.

Images