Low-illumination image enhancement method based on multi-branch feature aggregation

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a low-illumination image enhancement method based on multi-branch feature aggregation. The method comprises the following steps: designing a feature enhancement mapping network model, wherein the model comprises a convolutional layer, a symmetric multi-feature aggregation module and a multi-dimensional enhancement module; and dividing a training set and a test set by using a non-uniform low-illumination data set, inputting the training set and the test set into the constructed network model for training, and performing fine tuning to obtain a final model. The invention designs a low-illumination image enhancement method based on multi-branch feature aggregation, which is used for improving the performance of image features. On the basis of the overall feature enhancement mapping network, a symmetric multi-feature aggregation module and a multi-dimensional enhancement module are innovatively provided, and the features and channel dimensions of the symmetric multi-feature aggregation module and the multi-dimensional enhancement module are modulated together, so that the features of the low-illumination image are effectively enhanced. According to the method, the quality and definition of the image can be remarkably improved, and illumination and reflection details are more distinct.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, specifically to a low-light image enhancement method based on multi-branch feature aggregation. Background Technology

[0002] Image enhancement techniques under non-uniform lighting conditions are an important research topic in computer vision. Their core objective is to recover rich, high-quality visual information from single or multiple frames of low-light input images using advanced computational methods. With breakthroughs in deep learning technology, convolutional neural network-based enhancement methods have significantly outperformed traditional Retinex decomposition and histogram equalization algorithms in terms of detail preservation and visual quality. However, existing enhancement algorithms still have room for improvement in areas such as adaptability to real-world scenarios, computational efficiency optimization, and the construction of objective evaluation systems. Specifically, these include insufficient robustness to complex lighting conditions, limited model generalization ability, and a lack of unified quality assessment standards. These bottlenecks restrict the widespread adoption of this technology in practical applications.

[0003] Chinese patent publication number "CN117455811B", entitled "A Low-Light Image Enhancement Method Based on a Transformer-Based Structure-Aware Dual-Stream Network", proposes a dual-path collaborative network architecture consisting of a main enhancement branch and a structure-aware branch. The main enhancement branch achieves basic illumination correction and detail reconstruction through multi-scale feature learning, while simultaneously generating multi-level structural cues. The structure-aware branch is equipped with a feature parsing module to perform structured representation learning on the deep semantic features output by the main branch encoder. Specifically, a gated structural attention mechanism is introduced to achieve dynamic structural feature selection and enhancement within the Transformer framework. Furthermore, an innovative cross-modal feature interaction module is designed, utilizing a cross-attention mechanism to establish feature associations between the two branches, achieving complementary fusion of enhancement and structural features, thereby improving the structural integrity and detail clarity of the reconstructed image. This method fails to capture detailed features comprehensively during image enhancement, resulting in severe loss of edge detail information. The image enhancement method used is too simplistic and superficial, failing to consider the influence of numerous uncontrollable factors in the real environment on the image. This leads to blurry and unclear image enhancement results in environments affected by non-uniform low-light scenes. Therefore, we propose a low-light image enhancement method with multi-branch feature aggregation to solve the above problems. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a low-light image enhancement method based on multi-branch feature aggregation. This method extracts deep features from non-uniform low-light images layer by layer, and then cleverly aggregates and fuses various rich structural and detail information to give the enhanced non-uniform low-light image better visual clarity and structural quality, which is more in line with human visual observation, thus solving the problems mentioned in the background technology.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0008] A low-light image enhancement method based on multi-branch feature aggregation includes the following steps:

[0009] Step 1, Design the Network Model: Design a feature enhancement mapping network model to extract image features from the non-uniform low-light dataset through a multi-dimensional enhancement module and numerous convolutional layers. Then, a pixel reconstruction layer is used to increase image resolution and reduce the number of channels, making image features more prominent and obvious. While performing several deep feature mining operations, each layer passes through a convolutional layer and undergoes a pass-through and concatenation operation. After passing through the multi-dimensional enhancement module and an inverse pixel reconstruction layer, the image resolution is reduced, the number of channels is increased, and overfitting is prevented. Simultaneously, the data output from each layer is upsampled again through a convolutional layer. Finally, all images are input into a symmetric multi-feature aggregation module, whose output is added to the original input for fusion, resulting in a high-quality final output image.

[0010] Step 2, Prepare the dataset: Prepare the first non-uniform low-light image dataset, divide it into training and test sets and preprocess it; prepare the second non-uniform low-light image dataset, preprocess it and then fine-tune the model.

[0011] Step 3, train the network model: input the dataset processed in step 2 into the network model built in step 1 for training until the number of training iterations reaches the initial set threshold, at which point the network model is considered to have been trained and the network model parameters are saved.

[0012] Step 4, fine-tuning the model: The network model is retrained and fine-tuned using the processed second non-uniform low-light image dataset to optimize the network model parameters, improve image enhancement performance, and obtain a network model that can achieve the best results.

[0013] Step 5, Save the model: Solidify the finalized model parameters and save the model. The model can output a corresponding high-quality image after inputting a non-uniform low-light image.

[0014] Furthermore, in step 1, the multi-dimensional enhancement module inputs the image into the multi-attention module. After multi-branch processing, while preserving the original features and extracting deeper image features, it performs splicing, addition, and multiplication operations on the texture feature information and background information of the image, so that a large number of features can be obtained in a more balanced way without causing feature loss or overfitting risk. The symmetric multi-feature aggregation module sends multiple inputs to the symmetric multi-feature aggregation module. Through multi-input parallel processing, it preserves the spatial structure, reduces the risk of overfitting, and maximizes the preservation of the enhanced image features, making them vivid.

[0015] Furthermore, the preprocessing of the prepared dataset in step 2 includes: adjusting the size of each image in the dataset and fixing the size of the input image as input to the network model.

[0016] Furthermore, in the training process of step 3, a loss function value that minimizes and an appropriate evaluation metric are selected. A composite loss function is used for the loss function, and the feature enhancement mapping network adopts a smoothness loss function, a structural similarity loss function, and a mean squared error loss function. The choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly reflect the quality of the model.

[0017] Furthermore, the appropriate evaluation metrics selected are peak signal-to-noise ratio (PSNR) and structural similarity, which can evaluate the quality and degree of image distortion of non-uniform low-light image enhancement results and play a role in measuring the non-uniform low-light image enhancement network model. PNR is based on the error between corresponding pixels, that is, it is an error-sensitive image quality evaluation, while structural similarity measures image similarity from three aspects: brightness, contrast and structure.

[0018] (III) Beneficial Effects

[0019] Compared with existing technologies, this invention provides a low-light image enhancement method based on multi-branch feature aggregation, which has the following beneficial effects:

[0020] 1. The method of this invention can be used to enhance the representation of image features. It modulates the features and channel dimensions of the image, thereby effectively enhancing the features of non-uniform low-light images. This method can significantly improve image quality and sharpness, making lighting and reflection details more vivid.

[0021] 2. This invention proposes a multi-dimensional enhancement module, which improves the connectivity of the network and enhances the feature capture ability. It enhances low-light images at different scales, preserves the original features, and improves the image representation effect in terms of structural and background details, making the enhanced image details and structural background more convincing.

[0022] 3. This invention proposes a symmetric multi-feature aggregation module, which fuses multi-branch features channel by channel. During fusion, the multi-path outputs are weighted and superimposed according to the channel dimension based on pixel-level dynamic weights. Through feature reuse between layers, the synergistic optimization of texture detail enhancement and noise suppression is achieved, effectively improving the image signal-to-noise ratio while maintaining edge sharpness.

[0023] 4. This invention proposes a composite loss function that integrates mean squared error loss, smoothness loss, and structural similarity loss. By synergistically optimizing pixel-level accuracy, structural preservation capability, and perceptual quality, it comprehensively improves the visual realism and detail fidelity of low-light images. This composite loss function constructs optimization objectives from three dimensions: minimizing numerical error, geometric feature stability, and adaptation to the human visual system. While ensuring the mathematical fidelity of the image, it enhances the visual naturalness and spatial coherence of the generated results. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention;

[0025] Figure 2 This is a schematic diagram of the feature enhancement mapping network model structure of the present invention;

[0026] Figure 3 This is a schematic diagram of the multi-dimensional enhancement module structure of the present invention;

[0027] Figure 4 This is a schematic diagram of the symmetric multi-feature aggregation network structure of the present invention;

[0028] Figure 5 A comparison of relevant indicators between existing technologies and the method proposed in this invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1

[0031] like Figure 1-5 As shown in the figure, an embodiment of the present invention proposes a low-light image enhancement method based on multi-branch feature aggregation, which specifically includes the following steps:

[0032] Step 1: Design the network model: Design a feature enhancement mapping network model; such as... Figure 2As shown, a schematic diagram of the feature enhancement mapping network model structure is provided. The feature enhancement mapping network consists of a multi-dimensional enhancement module, a symmetric multi-feature aggregation module, a pixel reconstruction layer, an inverse pixel reconstruction layer, convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, convolutional layer 7, convolutional layer 8, and an R-type activation function. The multi-dimensional enhancement module and the symmetric multi-feature aggregation module are used to connect the network and fuse feature maps. The pixel reconstruction layer and the inverse pixel reconstruction layer are used to preserve image details and avoid loss of image information. The kernel size of convolutional layer 1, convolutional layer 5, convolutional layer 6, convolutional layer 7, and convolutional layer 8 is 3×3, and the stride is 1. The kernel size of convolutional layer 2, convolutional layer 3, and convolutional layer 4 is 1×1, the stride is 1, and the activation function is an R-type function.

[0033] like Figure 3 As shown, a schematic diagram of the multi-dimensional enhancement module structure is provided. This module consists of a global average pooling layer 1, a global average pooling layer 2, a global max pooling layer 1, a global max pooling layer 2, a convolutional layer 9, a convolutional layer 10, a convolutional layer 11, a convolutional layer 12, a convolutional layer 13, a convolutional layer 14, and an R-type activation function. The convolutional kernels of convolutional layers 9, 10, 11, 12, 13, and 14 have a size of 3×3, a stride of 1, and an R-type activation function.

[0034] like Figure 4 As shown, a schematic diagram of the symmetric multi-feature aggregation module structure is provided, which consists of global average pooling layer 3, global average pooling layer 4, convolutional layer 17, convolutional layer 18, convolutional layer 19, convolutional layer 20, convolutional layer 21, convolutional layer 22, convolutional layer 23, convolutional layer 24 and R-type function; the kernel size of all convolutional layers is 3×3, the stride is 1, and the activation function is R-type function.

[0035] The R-type function is expressed as f(x). ReLU =max(0,x).

[0036] Step 2: Prepare the dataset: Use LOL-v1 as the first non-uniform low-light image dataset, divide it into training and test sets and preprocess it; use LOL-v2 as the second non-uniform low-light image dataset, preprocess it and then fine-tune the model.

[0037] Step 3: Train the network model: Train the feature enhancement mapping network model by inputting the dataset processed in Step 2 into the network model built in Step 1 for training.

[0038] During training, the optimal loss function value and evaluation metric are selected. Specifically, the loss function between the network output image and the ground truth image is minimized until 400 training iterations are completed, at which point the model parameters are considered pre-trained and saved. Simultaneously, the optimal evaluation metric is selected to measure the algorithm's accuracy and evaluate the system's performance. During training, a composite loss function is used; the feature enhancement mapping network employs a smoothness loss function, a structural similarity loss function, and a mean squared error loss function.

[0039] During training, a smoothness loss function is chosen. The smoothness loss function typically assumes that changes in a natural image are gradual; its working principle is to minimize the gradient between a pixel and its neighborhood.

[0040]

[0041] Where N is the number of iterations; and These represent the gradient values ​​in the horizontal and vertical directions, respectively, and out_img represents the output image.

[0042] During training, the structural similarity loss function is selected. The structural similarity loss function is also a frequently used loss function, generally measuring the difference between the ground truth image and the output image. The structural similarity loss evaluates image quality by comprehensively considering brightness, contrast, and structure, while removing the brightness dimension. This helps to better preserve structure and texture during model training.

[0043]

[0044] Where M represents different scales; μ p ,μ g σ represents the mean of the output image and the ground truth, respectively; p ,σ g σ represents the standard deviation of the output image and the ground truth, respectively; pg β represents the covariance between the two. m and γ m c1 and c2 are both constants.

[0045] During training, the mean squared error loss function is used. The principle of the mean squared error loss function formula is as follows:

[0046]

[0047] Among them, y i It is the actual output data. This is the output data of the model fitting. From this, we can see that the closer the loss is to 0, the better the model and the better the fit, and the more successful the data prediction.

[0048] Therefore, the loss function is defined as:

[0049] L=λ1*L Smoothness-Loss +λ2*L SSIM +λ3*L MSE

[0050] Where λ1 = 0.5, λ1 = 0.3, and λ1 = 0.2.

[0051] The choice of loss function affects the quality of the model, accurately reflecting the difference between the predicted and true values, and providing correct feedback on the model's quality.

[0052] Choosing appropriate evaluation metrics, such as peak signal-to-noise ratio and structural similarity, can effectively assess the quality and degree of image distortion in non-uniform low-light image enhancement results, and serve as a measure of the non-uniform low-light image enhancement network model.

[0053] Peak signal-to-noise ratio (PSNR) is based on the error between corresponding pixels, i.e., it is an error-sensitive image quality assessment; structural similarity measures image similarity from three aspects: brightness, contrast, and structure, and is an indicator used to measure the degree of similarity between two digital images; image perceptual similarity is a method for measuring image similarity, which uses a deep learning model to evaluate the perceptual differences between two images.

[0054] Peak signal-to-noise ratio, structural similarity, and image perceptual similarity are defined as follows:

[0055]

[0056] Where MSE is the mean square error between the original image and the processed image, and MAX is the mean square error between the two images. I This represents the maximum value of the image color; 255 is represented by 8 bits of sampling.

[0057]

[0058] Where, μ x μ y Let x and y represent the mean values ​​of the image, respectively. and Let σ represent the variances of the images x and y, respectively. xy Let C1 and C2 represent the covariance of the images x and y, where C1 and C2 are constants.

[0059]

[0060] Specifically, the feature stack is extracted from layer L, and cell normalization is performed in the channel dimension. For layer l, the result is denoted as... Using vectors The active channel is scaled and the L2 distance is calculated. Finally, the average value is calculated spatially and the sum is calculated across the channels.

[0061] The training iterations were set to 400. The learning rate was set to 0.0001 for the first 200 iterations, and then gradually decreased to 0 for the next 200 iterations. The maximum number of images input to the network per iteration was determined by the computer's graphics processing unit (GPU) performance; 16 images were used per iteration to ensure more stable training and better results, guaranteeing rapid network fitting. The Adam optimizer was chosen as the network parameter optimizer. Its advantages include simple implementation, high computational efficiency, low memory requirements, and parameter updates unaffected by gradient scaling, resulting in relatively stable parameters.

[0062] Step 4: Fine-tune the model: Train and fine-tune the model using the processed second non-uniform low-light image dataset to obtain stable and usable model parameters, further improving the model's ability to enhance non-uniform low-light images; ultimately resulting in better image quality enhanced by the model.

[0063] Step 5: Save the model: Solidify the finalized model parameters.

[0064] When performing non-uniform low-light image enhancement, the final high-quality image can be obtained by directly inputting the image into the network.

[0065] This invention constructs a low-light image enhancement method based on multi-branch feature aggregation, which can directly generate high-quality images from non-uniform low-light images without intermediate steps. Under the same conditions, the feasibility and superiority of this method are further verified by calculating the relevant indices of the images obtained by existing methods. The relevant indices of existing technologies and the method proposed in this invention are compared in the table below;

[0066] from Figure 5 It can be seen that the method proposed in this invention has better peak signal-to-noise ratio and structural similarity than existing methods. These indicators further demonstrate that the method proposed in this invention has better image enhancement quality in non-uniform low-light conditions.

[0067] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on multi-branch feature aggregation, characterized in that, Includes the following steps: Step 1, Design the Network Model: Design a feature enhancement mapping network model to extract image features from the non-uniform low-light dataset through a multi-dimensional enhancement module and numerous convolutional layers. Then, a pixel reconstruction layer is used to increase image resolution and reduce the number of channels, making image features more prominent and obvious. While performing several deep feature mining operations, each layer passes through a convolutional layer and undergoes a pass-through and concatenation operation. After passing through the multi-dimensional enhancement module and an inverse pixel reconstruction layer, the image resolution is reduced, the number of channels is increased, and overfitting is prevented. Simultaneously, the data output from each layer is upsampled again through a convolutional layer. Finally, all images are input into a symmetric multi-feature aggregation module, whose output is added to the original input for fusion, resulting in a high-quality final output image. Step 2, Prepare the dataset: Prepare the first non-uniform low-light image dataset, divide it into training and test sets and preprocess it; Prepare a second non-uniform low-light image dataset, preprocess it, and then fine-tune the model; Step 3, train the network model: input the dataset processed in step 2 into the network model built in step 1 for training until the number of training iterations reaches the initial set threshold, at which point the network model is considered to have been trained and the network model parameters are saved. Step 4, fine-tuning the model: The network model is retrained and fine-tuned using the processed second non-uniform low-light image dataset to optimize the network model parameters, improve image enhancement performance, and obtain a network model that can achieve the best results. Step 5, Save the model: Solidify the finalized model parameters and save the model. The model can output a corresponding high-quality image after inputting a non-uniform low-light image.

2. The low-light image enhancement method based on multi-branch feature aggregation according to claim 1, characterized in that: In step 1, the multi-dimensional enhancement module inputs the image into the multi-attention module. After multi-branch processing, while preserving the original features and extracting deeper image features, it performs splicing, addition, and multiplication operations on the texture features and background information of the image. This allows for a more balanced acquisition of a large number of features without causing feature loss or overfitting. The symmetric multi-feature aggregation module sends multiple inputs to the symmetric multi-feature aggregation module. Through multi-input parallel processing, it preserves the spatial structure, reduces the risk of overfitting, and maximizes the preservation of the enhanced image features, ensuring their vividness.

3. The low-light image enhancement method based on multi-branch feature aggregation according to claim 1, characterized in that: The preprocessing of the prepared dataset in step 2 involves adjusting the size of each image in the dataset and fixing the size of the input image as input to the network model.

4. The low-light image enhancement method based on multi-branch feature aggregation according to claim 1, characterized in that: In the training process of step 3, a loss function value that is minimized and an appropriate evaluation metric are selected. A composite loss function is used, and the feature enhancement mapping network adopts a smoothness loss function, a structural similarity loss function, and a mean squared error loss function. The choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly reflect the quality of the model.

5. The low-light image enhancement method based on multi-branch feature aggregation according to claim 4, characterized in that: The appropriate evaluation metrics chosen are peak signal-to-noise ratio (PSNR) and structural similarity. These metrics can evaluate the quality and degree of image distortion of non-uniform low-light image enhancement results, and serve as a measure of the non-uniform low-light image enhancement network model. PNR is based on the error between corresponding pixels, which is an error-sensitive image quality evaluation. Structural similarity measures image similarity from three aspects: brightness, contrast, and structure.

Images