Lightweight low-light image enhancement method and device

Abstract

The invention relates to a lightweight low-light image enhancement method and device. The method comprises the steps that an image data set comprises an image pair composed of a low-light image and a normal-light image; carrying out key feature extraction and image size dimension reduction on the image pairs in the image data set to obtain dimension-reduced image pairs; performing forward diffusion on the normal light image in the dimension-reduced image pair based on a diffusion model to obtain a diffusion image, and performing reverse denoising on the low-light image through the diffusion image to obtain a denoised image; performing detail enhancement and color correction on the denoised image to obtain a target image; a diffusion model is constructed to perform forward diffusion and reverse denoising processing on a dimensionality-reduced image, so that the calculation efficiency is ensured, the feature expression capability is effectively improved, the noise diffusion process is accurately controlled, the denoising effect is stabilized, detail enhancement and color correction are combined, the authenticity of image details and colors is maintained, the stability of the denoising effect is improved, and the method is suitable for large-scale popularization and application. And meanwhile, image details and color authenticity are kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image enhancement, in particular to a lightweight low-light image enhancement method and device. BACKGROUND

[0002] In the visual system, night and low-light conditions often lead to a significant decline in the quality of the captured image, which is manifested as unclear details, more noise in dark places, and overall darkness, which seriously affects the execution of various visual tasks. Therefore, low-light image enhancement technology has a wide range of applications in many fields, such as night monitoring, autonomous driving, target detection, etc.

[0003] Existing low-light image enhancement methods still have many shortcomings. Traditional methods rely on image prior knowledge or simple physical models, such as gray scale transformation, histogram equalization, and Retinex theory decomposition method, which can improve brightness and contrast to some extent, but lack adaptability to complex scenes and noise, and have poor generalization ability and robustness. Although deep learning methods have made progress, CNN-based methods have poor global feature enhancement due to the size limitation of convolution kernels. Transformer-based methods consume a lot of computing resources and have slow inference speed. Although diffusion models have strong generation and generalization ability, existing methods have problems such as high computational cost, limited detail recovery, and unstable results.

[0004] Therefore, there is an urgent need to propose a lightweight low-light image enhancement method and device to solve the technical problems of high computational cost, insufficient detail recovery, and poor color performance of existing low-light image enhancement methods, resulting in unstable results. SUMMARY

[0005] Therefore, it is necessary to provide a lightweight low-light image enhancement method and device to solve the technical problems of high computational cost, insufficient detail recovery, and poor color performance of existing low-light image enhancement methods, resulting in unstable results.

[0006] To solve the above problems, in a first aspect, the present application provides a lightweight low-light image enhancement method, comprising: obtaining an image dataset and constructing a diffusion model; the image dataset comprises image pairs composed of low-light images and normal-light images; performing key feature extraction and image size dimension reduction on the image pairs in the image dataset to obtain a reduced dimension image pair; performing forward diffusion on the normal-light images in the reduced dimension image pair based on the diffusion model to obtain a diffusion image, and performing reverse denoising on the low-light images through the diffusion image to obtain a denoised image; performing detail enhancement and color correction on the denoised image to obtain a target image.

[0007] In a possible implementation, the key feature extraction and image size dimension reduction on the image pair in the image dataset are performed to obtain a dimension-reduced image pair, including: segmenting each image in the image pair to obtain a plurality of sub-features; performing feature extraction on each sub-feature through different convolution branches to obtain an output result of each convolution branch; inputting all the output results into a splicing channel to perform feature splicing to obtain a splicing result; performing dimension reduction on the splicing result through standard convolution to obtain a dimension-reduced image pair.

[0008] In a possible implementation, the forward diffusion on the normal light image in the dimension-reduced image pair based on the diffusion model is performed to obtain a diffusion image, including: setting a fixed variance and a time step; the fixed variance increases with the increase of the time step; in the diffusion model, the Gaussian noise of the fixed variance at each time step is gradually added to the normal light image in the dimension-reduced image pair to obtain a diffusion image close to pure noise.

[0009] In a possible implementation, the reverse denoising on the low-light image through the diffusion image is performed to obtain a denoised image, including: based on a denoising network, performing reverse prediction on the diffusion image, the time step and the low-light image to obtain reverse prediction noise of the next time step, and performing denoising on the low-light image according to the reverse prediction noise to obtain an initial denoised image; calculating the Gaussian noise gradually added with the fixed variance and the reverse prediction noise to obtain a mean square error; after updating the denoising formula of the denoising network according to the mean square error, performing reverse iterative denoising on the subsequent time step, the initial denoised image and the low-light image according to the updated denoising network, and obtaining a denoised image when the time step is completed.

[0010] In a possible implementation, the detail enhancement and color correction on the denoised image are performed to obtain a target image, including: performing detail enhancement on the denoised image based on a lightweight SR module to obtain a reconstructed image; performing color correction on the reconstructed image based on a color enhancement module to obtain a target image.

[0011] In a possible implementation, the detail enhancement on the denoised image based on the lightweight SR module is performed to obtain a reconstructed image, including: After inputting the denoising image into the lightweight SR module, shallow feature extraction is performed on the denoising image to obtain shallow features; Deep feature extraction is performed on the shallow features based on a plurality of separable distillation blocks to obtain feature information of different depths; Multi-layer feature fusion is performed on all feature information to obtain refined fusion features; Feature reconstruction is performed on the refined fusion features to obtain a reconstructed image.

[0012] In a possible implementation, after inputting the denoising image into the lightweight SR module, shallow feature extraction is performed on the denoising image to obtain shallow features, including: The denoising image is copied to obtain a plurality of copied images; The plurality of copied images are spliced along the channel dimension to obtain a spliced image; The spliced image is subjected to shallow feature extraction according to a convolution module to obtain shallow features.

[0013] In a possible implementation, the color correction module is used to correct the color of the reconstructed image to obtain a target image, including: After inputting the reconstructed image into the color enhancement module, the reconstructed image is subjected to normalization processing to obtain a processed image; The processed image is subjected to coarse-grained feature extraction through deep convolution to obtain a coarse-grained feature map; The coarse-grained feature map is subjected to activation operation based on a SimpleGate unit to obtain an enhanced feature map; The enhanced feature map is input into an SCA attention module for channel feature information enhancement to obtain enhanced features; The enhanced features are input into a color correction module to obtain a target image.

[0014] In a possible implementation, the enhanced features are input into the color correction module to obtain a target image, including: The RGB channel of the enhanced features is adjusted through a three-dimensional weight vector of the color correction module to obtain a target image.

[0015] In a second aspect, the present application further provides a lightweight low-light image enhancement device, comprising: A data acquisition module is configured to acquire an image dataset and construct a diffusion model; the image dataset comprises an image pair composed of a low-light image and a normal-light image; A feature extraction module is configured to extract key features and reduce the dimension of an image size of the image pair in the image dataset to obtain a reduced dimension image pair; an image diffusion module configured to perform forward diffusion on the normal light image in the reduced dimension image pair based on the diffusion model to obtain a diffusion image, and perform reverse de-noising on the low light image based on the diffusion image to obtain a de-noised image; a detail enhancement module configured to perform detail enhancement and color correction on the de-noised image to obtain a target image.

[0016] The present application has the advantages that: an image dataset is obtained, and a diffusion model is constructed; the image dataset comprises an image pair composed of a low light image and a normal light image; key feature extraction and image size reduction are performed on the image pair in the image dataset to obtain a reduced dimension image pair; forward diffusion is performed on the normal light image in the reduced dimension image pair based on the diffusion model to obtain a diffusion image, and reverse de-noising is performed on the low light image based on the diffusion image to obtain a de-noised image; detail enhancement and color correction are performed on the de-noised image to obtain a target image; forward diffusion and reverse de-noising are performed on the reduced dimension image pair by constructing the diffusion model, the feature expression capability is effectively improved while the calculation efficiency is ensured, the noise diffusion process is accurately controlled, the de-noising effect is stabilized, and the image details and color authenticity are maintained by combining the detail enhancement and color correction, so that the advantages of taking into account the calculation efficiency and the feature expression capability, improving the de-noising effect stability, and maintaining the image details and color authenticity are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An embodiment flowchart of a lightweight low light image enhancement method provided by the present application is shown in the figure; Figure 2 An embodiment flowchart of step S102 in the present application is shown in the figure; Figure 1 Figure 3 An embodiment structure diagram of an IDC module provided by the present application is shown in the figure; Figure 4 An embodiment flowchart of detail enhancement provided by the present application is shown in the figure; Figure 5 An embodiment structure diagram of a BSRN provided by the present application is shown in the figure; Figure 6 An embodiment structure diagram of a color enhancement module (CEM) provided by the present application is shown in the figure; Figure 7 An embodiment structure diagram of a lightweight low light image enhancement device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, wherein the accompanying drawings form a part of the present application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.​

[0019] As Figure 1 shown, one specific embodiment of the present application discloses a lightweight low-light image enhancement method, comprising: S101, acquiring an image data set and constructing a diffusion model; the image data set comprises an image pair composed of a low-light image and a normal-light image.

[0020] The lightweight low-light image enhancement method provided by the embodiments of the present application can be applied to a lightweight low-light image enhancement system, wherein the lightweight low-light image enhancement can be based on a software system running on a terminal device, and the terminal device can be a server, a tablet computer, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, or the like. The embodiments of the present application do not limit the specific type of the terminal device.

[0021] Among them, the data set required for selecting the network model can be selected according to actual needs, the data set can be divided into a training set and a test set, the network model selects the LOLv1 data set as the training and test, the LOLv1 data set contains 500 low-light and normal-light image pairs, wherein the training set contains 485 images, and the test set contains 15 images. The low-light image contains noise generated during photographing, most of the images are indoor scenes, and the resolution of all images is 600x400.

[0022] S102, key feature extraction and image size dimension reduction are performed on the image pair in the image data set to obtain a reduced dimension image pair.

[0023] Among them, the key feature extraction and image size dimension reduction refer to reducing the image dimension through feature segmentation, multi-branch convolution extraction and splicing operation, for example, after different sub-features are extracted by multi-convolution branch, the dimension is reduced by splicing, and the calculation amount of the subsequent diffusion process is reduced.

[0024] S103, based on the diffusion model, the normal-light image in the reduced dimension image pair is subjected to forward diffusion to obtain a diffusion image, and the low-light image is subjected to reverse denoising through the diffusion image to obtain a denoised image.

[0025] Wherein, the forward diffusion refers to converting a normal light image into a diffusion image close to pure noise by adding Gaussian noise step by step, for example, setting a fixed variance time step control noise superposition process. The reverse denoising refers to predicting noise based on the diffusion image and the time step and iteratively denoising, for example, updating the denoising formula after predicting the noise through the denoising network.

[0026] S104, detail enhancement and color correction are performed on the denoised image to obtain a target image.

[0027] Wherein, the detail enhancement refers to recovering high-frequency details through a lightweight module, for example, using a separable distillation block to extract deep features. The color correction refers to correcting color deviation through a channel adjustment module, for example, adjusting the RGB channel distribution by using a three-dimensional weight vector.

[0028] Specifically, first, paired low-light and normal light images are obtained from the data set, and key features are extracted through segmentation and multi-branch convolution, and then dimensionality reduction is performed to reduce the data amount for subsequent processing. In the diffusion model, noise is added to the dimensionality-reduced normal light image step by step to generate a diffusion image, and the diffusion image is used to guide the denoising process of the low-light image. By predicting noise and iteratively updating the denoising network parameters, the noise in the low-light image is gradually removed. Subsequently, multi-scale feature fusion is performed on the denoised image using a lightweight module to enhance details, and then the channel weight is adjusted through a color correction module, and finally a clear and color-accurate target image is output.

[0029] Compared with the prior art, the embodiment provided by the present embodiment provides an image data set, and a diffusion model is constructed. The image data set includes image pairs composed of low-light images and normal light images. Key features are extracted from the image pairs in the image data set, and the size of the image is reduced to obtain a dimensionality-reduced image pair. Based on the diffusion model, the normal light image in the dimensionality-reduced image pair is subjected to forward diffusion to obtain a diffusion image, and the low-light image is subjected to reverse denoising through the diffusion image to obtain a denoised image. The denoised image is subjected to detail enhancement and color correction to obtain a target image. By constructing the diffusion model, the dimensionality-reduced image pair is subjected to forward diffusion and reverse denoising processing, which effectively improves the feature expression capability while ensuring the calculation efficiency, accurately controls the noise diffusion process, stabilizes the denoising effect, and maintains the image details and color authenticity by combining the detail enhancement and color correction. It has the advantages of balancing the calculation efficiency and the feature expression capability, improving the stability of the denoising effect, and maintaining the image details and color authenticity.

[0030] In some embodiments of the present application, as shown in Figure 2 The step S102 includes: S201, segmenting each image in the image pair to obtain a plurality of sub-features.

[0031] Wherein, the segmentation refers to dividing the input image into multiple local regions, and specifically, a sliding window or an adaptive region division algorithm can be used to achieve the division, so as to decompose the whole image into sub-regions which are easier to process. The different convolution branches refer to parallel convolution layers with different kernel sizes or receptive fields, and specifically, a 5*5, 1*11, 11*1 or separable convolution structure can be used to achieve the parallel convolution layers, which are used to extract local information of sub-features from different scales or angles.

[0032] S202, feature extraction is performed on each sub-feature through different convolution branches respectively to obtain output results of each convolution branch.

[0033] S203, all the output results are input into a splicing channel for feature splicing to obtain a splicing result.

[0034] Wherein, the splicing channel refers to merging multiple feature maps along the channel dimension, and specifically, a channel splicing function or a self-defined splicing layer can be used to achieve the merging, so as to integrate diversified features extracted by different convolution branches.

[0035] S204, a dimension reduction operation is performed on the splicing result through a standard convolution to obtain a dimension reduction image pair.

[0036] Wherein, the dimension reduction operation refers to reducing the channel number or spatial size of the feature map, and specifically, a standard convolution or a pooling layer with a step size greater than 1 can be used to achieve the operation, so as to reduce the subsequent calculation complexity.

[0037] Specifically, in the key feature extraction stage, first, the image pair composed of the low-light image and the normal-light image is subjected to a segmentation processing to generate multiple sub-regions. Each sub-region is subjected to feature extraction through an independent convolution branch, for example, one branch uses a 3*3 convolution kernel to capture detailed textures, and another branch uses a 5*5 convolution kernel to extract context information in a larger range. The output results of the branches are sent to a splicing channel for fusion to form a splicing result containing multi-scale features. Subsequently, a standard convolution layer is used to perform channel compression and spatial downsampling on the spliced high-dimensional features, and finally, a dimension reduction image pair with reduced dimension but complete information is obtained.

[0038] In the embodiment of the application, an Inception deep convolution (IDC) module is used to extract key features and reduce the size of the image pair in the data set. As shown in Figure 3 , a structure diagram of the IDC module is shown. The IDC module includes four parallel convolution branches, one of which is a depth separable convolution with a kernel size of 5*5, and two one-dimensional strip convolutions (kernel sizes are 1*11 and 11*1) and an identity mapping. Specifically, the input feature X is first subjected to a segmentation operation along the channel dimension to divide it into four sub-features X hw , X w , Xh and X id Then, 4 sub-features are respectively extracted in different branches, as shown in formula (1): (1) In the formula, c represents the number of channels, that is, the number of channels of the 4 branches [:c], [c:2c], [2c:3c] and [3c:] is c. Next, X hw , X w , X h and X id are respectively extracted in different branches, and the process is shown in formula (2): (2) In the formula, k n represents the kernel size of the depth separable convolution, which is set to 5 or 7 by default in the embodiments of the present application, k m represents the kernel size of the one-dimensional strip convolution, which is set to 11 by default in the embodiments of the present application, X ’ hw , X ’ w , X ’ h and X ’ id respectively represent the output results of the 4 branches.

[0039] Next, the outputs of the four branches are concatenated and then subjected to a standard 1x1 convolution for dimension reduction, which is beneficial to improve the calculation efficiency and reduce resource consumption in the diffusion model in the later stage. Finally, the final output Y of the characteristics after the IDC module (the image after dimension reduction) can be obtained. The process is shown in formula (3): (3) In some embodiments of the present application, step S103 comprises: setting a fixed variance and a time step; the fixed variance increases with the increase of the time step.

[0040] Fixed variance refers to the noise variance being pre-set before training and not adjusted during training. This can be achieved using a linear growth strategy, ensuring that the noise intensity increases with each time step, thus avoiding the computational complexity associated with dynamic variance adjustment. The time step refers to the total number of stages in the diffusion process, which can be determined using either uniform or exponential partitioning. This controls the gradual nature of noise addition, ensuring that the noise gradually covers the image information.

[0041] In the diffusion model, Gaussian noise with fixed variance at each time step is gradually added to the normal light image in the dimensionality reduced image pair to obtain a diffusion image close to pure noise.

[0042] Gaussian noise refers to random noise that conforms to the normal distribution. Specifically, it can be generated in a parameterized manner with a mean of zero and a variance determined by the time step. It is used to simulate the image degradation process and ensure that the noise distribution is associated with the time step.

[0043] Specifically, in the diffusion model, a fixed variance is pre-set and gradually increased over time. At each time step, Gaussian noise of corresponding intensity is generated based on the current variance and superimposed on the normal light image. As time steps accumulate, the noise intensity gradually increases, and the normal light image gradually degenerates into a state close to pure noise. This process simplifies the modeling of the noise distribution through the linear relationship between the fixed variance and the time step, making the noise addition process controllable and repeatable, and providing a stable noise distribution foundation for subsequent inverse denoising.

[0044] In the embodiment of the present invention, the low-light image and the normal-light image after dimensionality reduction are simultaneously fed into the diffusion model to complete the process of gradually adding noise and denoising recovery. In the forward diffusion process, the fixed variance increases with the increase of time steps. Gaussian noise to the input data X 0 (normal light image), after T steps of noise addition, the input X 0 is transformed into a data close to pure noise X T (i.e., diffuse image), as shown in formulas (4) and (5):

[0045] Where, and Represents the time step t Noisy data and predefined variance at , along with t increases with the increase of N represents Gaussian distribution, noise X T Obey the Gaussian distribution, that is X T~ N (I, 0).

[0046] In some embodiments of the application, the low-light image is inversely denoised by the diffusion image to obtain a denoised image, comprising: Based on the denoising network, the diffusion image, the time step and the low-light image are inversely predicted to obtain the inverse prediction noise of the next time step, and the low-light image is denoised according to the inverse prediction noise to obtain the initial denoised image.

[0047] Wherein, the denoising network refers to a neural network model used to recover the original image features from noise, which can be implemented by a deep learning model based on UNet architecture, which captures noise distribution features at different scales through multiple layers of convolution and skip connection. The inverse prediction noise refers to the noise component that needs to be removed in the current time step obtained by model inference, which can be realized by joint prediction of input diffusion image, time step encoding and low-light image features, and is used to guide the gradual elimination of noise in the denoising process.

[0048] The fixed variance step-by-step added Gaussian noise is calculated with the inverse prediction noise to obtain the mean square error.

[0049] Wherein, the mean square error refers to the difference between the Gaussian noise and the predicted noise, which can be realized by calculating the square difference of each pixel and taking the average value, and is used to optimize the parameter update direction of the denoising network.

[0050] After updating the denoising formula of the denoising network according to the mean square error, the subsequent time step, the initial denoised image and the low-light image are inversely iteratively denoised according to the updated denoising network, and when the time step is completed, the denoised image is obtained.

[0051] Wherein, the inverse iterative denoising refers to the process of gradually removing noise in the order of decreasing time step, which can be realized by combining the updated denoising formula and the current image state in each time step and performing multiple inferences, and finally outputting a clear image when the time step is zero.

[0052] Specifically, in the inverse denoising stage, the noise distribution information carried by the diffusion image and the degradation characteristics of the low-light image are input into the denoising network at the same time, and the network dynamically adjusts the noise prediction strategy according to the time step parameter. The initial denoised image is obtained by subtracting the predicted noise component from the diffusion image, and this process is iteratively performed in multiple time steps. The mean square error calculation part compares the fixed variance noise added by the forward diffusion process with the inverse prediction noise, forming a positive feedback signal for network training. The denoising formula updates the parameters according to the error calculation result, so that the noise prediction accuracy of the subsequent time step is gradually improved. After all the time steps are completed, the accumulated prediction error is effectively suppressed, and finally a denoised image with higher fidelity is output.

[0053] In specific embodiments of the present application, in the reverse denoising process, a neural network (such as U-Net) is learned to reverse the forward diffusion process, which is called a denoising network epsilon θ , to predict how to reverse X t from X t-1 , that is, to learn the Gaussian denoising conversion. In each step t of the reverse denoising, the network task is to predict the noise added to the image X t in the forward diffusion process by diffusing the image X t-1 , the time step t and the low-light image (as conditional information to guide the denoising process), to obtain the initial denoised image. After iterative optimization of the initial denoised image, the random Gaussian noise is converted into clear data , and the process is as shown in equation (6): (6) In the formula, is the noise predictor in the denoising process, which is learned by the U-Net network. The calculation is as shown in equation (7): (7) In the formula, , is the noise vector predicted by the model by optimizing the network parameters .

[0054] Finally, by calculating the mean square error between the noise predicted by the network and the real added noise ε, the denoising formula in the denoising network is gradually updated, and the network parameters are optimized by back propagation and gradient descent. In the inference process, the noise in the low-light image is gradually removed to generate the target normal light image (i.e. the denoised image). This conditional diffusion model architecture enables the model to learn the complex mapping relationship from the low-light image to the normal light image, while taking advantage of the powerful generation ability of the diffusion model to produce high-quality images.

[0055] In some embodiments of the present application, step S104 comprises: performing detail enhancement on the denoised image based on a lightweight SR module to obtain a reconstructed image.

[0056] The lightweight SR module refers to a structure that realizes multi-level feature extraction and fusion through a separable distillation block, and can be specifically implemented by shallow feature extraction, deep feature extraction of multiple separable distillation blocks, and multi-layer feature fusion to solve the problem of blurred details in the low-light image.

[0057] The color correction module is used for color correction of the reconstructed image to obtain a target image.

[0058] The color correction module is a structure for channel feature enhancement through normalization processing, deep convolution feature extraction, and attention mechanism, and can be implemented by using a normalization layer, a deep convolution layer, a SimpleGate activation unit, and an SCA attention module, and is used for solving the problems of color distortion and insufficient saturation in a low-light scene.

[0059] Specifically, the denoised image is first input into the lightweight SR module, initial features are extracted through shallow convolution, then multiple separable distillation blocks perform layered processing on the features to extract detailed information of different depths, then multi-scale information is integrated into refined features through feature fusion, and finally a high-resolution reconstructed image is generated through a reconstruction layer. The reconstructed image is further input into the color correction module, after normalization processing, coarse-grained features are extracted through deep convolution, the SimpleGate unit is used for nonlinear activation of the features to enhance effective information, and the SCA attention module is used for weight distribution of channel features to highlight key color information, and finally the color correction module adjusts the RGB channel ratio based on a three-dimensional weight vector to output a target image after color correction.

[0060] In some embodiments of the present application, as shown in Figure 4 The color correction module is used for color correction of the reconstructed image to obtain a target image. S401, inputting the denoised image into the lightweight SR module, and performing shallow feature extraction on the denoised image to obtain shallow features; S402, performing deep feature extraction on the shallow features based on multiple separable distillation blocks to obtain feature information of different depths; S403, performing multi-layer feature fusion on all feature information to obtain refined fusion features; S404, performing feature reconstruction on the refined and fused features to obtain a reconstructed image.

[0061] The separable distillation block is a module composed of a deep separable convolution and a feature distillation operation, and can be implemented by using a multi-branch deep separable convolution and a channel splitting operation, and is used for extracting multi-scale deep features and reducing the number of parameters. The multi-layer feature fusion refers to integrating features of different depths, and can be implemented by using channel splicing and convolution operation, and is used for retaining feature information of different levels and eliminating redundancy. The feature reconstruction refers to restoring the fused features to a high-resolution image, and can be implemented by using deconvolution or sub-pixel convolution operation, and is used for generating an enhanced image with clear details.

[0062] In some embodiments of the present application, step S401 comprises: copying the denoised image to obtain a plurality of copied images; splicing the plurality of copied images along a channel dimension to obtain a spliced image; performing shallow feature extraction on the spliced image according to the convolution module to obtain shallow features.

[0063] In specific embodiments of the present application, the image generated by the diffusion model is further enhanced in detail by a lightweight SR module. The module is constructed based on a blueprint separable residual network (BSRN) and is divided into four stages: shallow feature extraction, deep feature extraction, multi-layer feature fusion, and feature reconstruction. As shown in Figure 5 , the overall structure of BSRN is shown. BSRN is divided into four stages: shallow feature extraction, deep feature extraction, multi-layer feature fusion, and feature reconstruction. In the shallow feature extraction stage, the input image is copied n times to obtain a plurality of copied images, and spliced along the channel dimension to obtain a spliced image, and then a BSRN block composed of a 1x1 standard convolution and a deep convolution is used for shallow feature extraction, which maps the input image to a higher-dimensional feature space, as shown in equation (8): (8) In the formula, represents the input low-resolution image, n represents the number of input images Concat.

[0064] In the deep feature extraction process, a plurality of efficient separable distillation blocks (ESDB) are used to gradually refine the features extracted in the shallow layer, wherein each ESDB includes three stages of feature distillation, feature concentration, and feature enhancement. The process is shown in equation (9): (9) In the formula, represents the deep features extracted by the first ESDB block, represents the kth ESDB block, and represent the input and output of the kth ESDB block, respectively. k

[0065] In order to fully utilize the feature information of different depths, in the multi-layer feature fusion process, first, the features extracted at different depths (i.e. the features output by each ESDB) are concatenated, then they are sequentially fused through a 1x1 convolution and a GELU activation function, and finally the fused features are further refined using a BSRN. The process is shown in equation (10): (10) ​Finally, in the feature reconstruction stage, an upsampling operation is used to complete, as shown in equation (11): (11) In the formula, indicates the super-resolution output result graph (i.e. the reconstructed image) through the lightweight SR module.

[0066] In some embodiments of the present application, the color correction module is used to correct the color of the reconstructed image to obtain a target image, including: After the reconstructed image is input into the color enhancement module, the reconstructed image is normalized to obtain a processed image.

[0067] Wherein, the normalization processing refers to adjusting the pixel value of the input image to a preset range to stabilize the training process, which can be realized by batch normalization or layer normalization method, which helps to eliminate the influence of light difference on subsequent processing.

[0068] The processed image is extracted by deep convolution to obtain a coarse feature map.

[0069] Wherein, the deep convolution refers to the operation of extracting features by using a single layer convolution kernel, which can be realized by using a 3x3 or 5x5 convolution kernel size, which can capture spatial local features while reducing the number of parameters.

[0070] The coarse feature map is activated by the SimpleGate unit to obtain an enhanced feature map.

[0071] Wherein, the SimpleGate unit refers to the operation of nonlinear filtering of features by gating mechanism, which can be realized by ReLU activation of part of the channels after channel segmentation, which can enhance the feature expression ability and suppress redundant information.

[0072] The enhanced feature map is input into the SCA attention module to enhance the channel feature information to obtain an enhanced feature.

[0073] Wherein, the SCA attention module refers to a hybrid module combining channel attention and spatial attention mechanism, which can be realized by combining channel weight calculation and spatial position weighting, which can strengthen the features of important channels and improve the ability to retain spatial details.

[0074] The enhanced feature is input into the color correction module to obtain a target image.

[0075] Specifically, the reconstructed image is first normalized to eliminate brightness differences, and then coarse-grained features containing global structures are extracted by deep convolution. After the SimpleGate unit is activated, the effective features are retained and the noise interference is suppressed. The activated features further strengthen the spatial information of the key channels through the SCA attention module, and finally input the color correction module to independently adjust the RGB components through the learnable channel weight to achieve accurate correction of the color distribution. For example, in a low-light scene, the red channel may be significantly attenuated due to insufficient illumination, and at this time the color correction module can automatically enhance the intensity of the red component to restore the natural color tone.

[0076] In specific embodiments of the present application, color correction is performed on the image enhanced by the lightweight SR module. This function is completed by the color enhancement module (CEM), which is composed of the NAFNet network and the color correction module (CCM). As shown in Figure 6 , a color enhancement module (CEM) structure diagram is shown. First, the input feature map X 0 is normalized by LayerNorm to obtain the processed image, and then the feature map is subjected to a standard 1×1 convolution and a 3×3 deep convolution to extract coarse-grained features, obtaining a coarse-grained feature map, as shown in formula (12): (12) In the formula, LN represents the LayerNorm operation, represents a standard 1×1 convolution and a 3×3 deep convolution operation in sequence.

[0077] Then, the feature map is activated by the SimpleGate unit to obtain the enhanced feature map. In SimpleGate, the feature map is divided into two parts along the channel dimension, and then element-wise multiplication is performed to obtain the enhanced feature Xenhanced. This process can be represented as formula (13): (13) In the formula, X conv_1 and X conv_2 represent the feature maps with a size of (C / 2, H, W), respectively, represents element-wise multiplication.

[0078] Next, the enhanced feature map X enhancedThe input features are first subjected to a global average pooling operation to compress the original spatial information into channels, and then the channel attention is calculated through a multi-layer perceptron to weight the original input feature map. This process can be represented as shown in equation (14): (14) In the formula, pool represents a global average pooling operation, W represents a fully connected layer, represents a channel product operation.

[0079] Finally, the output of the first stage of NAFNet is: , represents a 1x1 convolution operation used to increase the channel dimension. As shown in Figure 6 , repeating most of the operations of the first stage of NAFNet (removing the 3x3 depth convolution and the SCA attention module) can obtain the output feature Y 2. Y 2 is sent to the final color correction module (CCM) for final color tuning, and the final high-light image close to the standard Y can be obtained.

[0080] In some embodiments of the present application, the enhanced features are input to the color correction module to obtain the target image, including: The RGB channels of the enhanced features are adjusted by the three-dimensional weight vector of the color correction module to obtain the target image.

[0081] The color correction module refers to a structure for adjusting the color distribution of an image based on learnable parameters, which can be implemented by independent scaling and translation operations on the RGB channels through a three-dimensional weight vector, and can adaptively correct color deviation for different scenes. The three-dimensional weight vector refers to a vector structure containing three independent weight parameters, which can be implemented by a learnable neural network parameter matrix, and each weight parameter corresponds to a color channel in the RGB channel for independently adjusting the color intensity of each channel. The RGB channel adjustment refers to a linear or nonlinear transformation of the numerical values of the red, green, and blue color channels in the image, which can be implemented by matrix multiplication combined with an activation function to correct color deviation and enhance color saturation.

[0082] Specifically, in the color correction module, a three-dimensional weight vector is applied to the RGB channels of the enhanced feature. First, the enhanced feature is decomposed into independent red, green, and blue channel components, and then the pixel values of each channel are weighted and calculated by a three-dimensional weight vector. For example, the pixel value of the red channel can be multiplied by the corresponding weight parameter, and the same operation is performed on the green and blue channels. Further, the weighted channel data is recombined into a complete image through normalization processing, and finally the target image after color correction is output. This process can fine-tune the brightness distribution difference of different channels, thereby solving the common color distortion problem in low-light images.

[0083] In specific embodiments of the present application, the color correction module adjusts the RGB channels of the input image by learning a 3-dimensional weight vector, which determines the degree of enhancement or weakening of each channel. In this way, the model can automatically learn the optimal weight configuration during the training process to achieve color correction. The process is shown in equation (15): (15) In the formula, Y denotes the image after color correction, w is the learned weight vector, which is adjusted to 1x3x1x1 to adapt to the broadcasting rule, denotes the element-wise multiplication operation, Z is a tensor of all 1s, which has the same shape as w is used to calculate the remaining part.

[0084] The embodiment of the present application reduces the image resolution through the Inception deep convolution module (IDC), significantly improves the efficiency of the diffusion model, greatly reduces the consumption of computing resources, and solves the problem of high computational cost of existing diffusion models. The lightweight super-resolution enhancement module (SR) is specifically used to optimize the detail recovery of the image generated by the diffusion model, which can accurately reconstruct the local key features of the image, and makes up for the defects of the existing method in insufficient detail recovery. The self-adaptive color enhancement module (CEM) is innovatively constructed, which effectively reduces the difference between low-light images and normal light images in the color representation level through multi-channel feature fusion and color correction, makes the enhanced image color more natural, and improves the overall visual effect. Combining the powerful generation ability of the diffusion model and the advantages of the SR and color enhancement module, not only can effectively enhance the quality of the low-light image, but also can recover or even infer the lost detail information, and the overall effect is more stable and reliable.

[0085] In order to better implement the lightweight low-light image enhancement method in the embodiment of the present application, on the basis of the lightweight low-light image enhancement method, correspondingly, the embodiment of the present application also provides a lightweight low-light image enhancement device, such as Figure 7As shown, the lightweight low-light image enhancement device 700 comprises: a data acquisition module 701, configured to acquire an image dataset, and construct a diffusion model; the image dataset comprises an image pair composed of a low-light image and a normal-light image; a feature extraction module 702, configured to perform key feature extraction and image size dimension reduction on the image pair in the image dataset, to obtain a reduced dimension image pair; an image diffusion module 703, configured to perform forward diffusion on the normal-light image in the reduced dimension image pair based on the diffusion model, to obtain a diffusion image, and perform reverse denoising on the low-light image through the diffusion image, to obtain a denoised image; a detail enhancement module 704, configured to perform detail enhancement and color correction on the denoised image, to obtain a target image.

[0086] The lightweight low-light image enhancement device 700 provided by the above embodiment can implement the technical solutions described in the above lightweight low-light image enhancement method embodiment, and the principles of the implementation of the above modules or units can be referred to the corresponding content in the above lightweight low-light image enhancement method embodiment, which will not be described here again.

[0087] The lightweight low-light image enhancement method and device provided by the present application are described in detail above, and the principles and implementation modes of the present application are described by applying specific examples in this paper. The above embodiment is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A lightweight low-light image enhancement method, characterized in that: include: Acquire an image dataset and construct a diffusion model; the image dataset includes image pairs consisting of low-light images and normal-light images; Performing key feature extraction and image size dimensionality reduction on the image pairs in the image dataset to obtain reduced-dimensionality image pairs; performing forward diffusion on the normal light image in the dimensionality reduction image pair based on the diffusion model to obtain a diffused image, and performing reverse denoising on the low light image using the diffused image to obtain a denoised image; Perform detail enhancement and color correction on the denoised image to obtain a target image.

2. The lightweight low-light image enhancement method according to claim 1, wherein: The step of performing key feature extraction and image size dimensionality reduction on the image pairs in the image dataset to obtain reduced-dimensionality image pairs includes: Segmenting each image in the image pair to obtain a plurality of sub-features; Each sub-feature is extracted through different convolution branches to obtain the output result of each convolution branch; Input all output results into the splicing channel for feature splicing to obtain the splicing results; A dimensionality reduction operation is performed on the splicing result through standard convolution to obtain a dimensionality-reduced image pair.

3. The lightweight low-light image enhancement method according to claim 1, wherein: The step of performing forward diffusion on the normal light image in the dimensionality reduction image pair based on the diffusion model to obtain a diffused image includes: Setting a fixed variance and a time step; wherein the fixed variance increases as the time step increases; In the diffusion model, the Gaussian noise with the fixed variance at each time step is gradually added to the normal light image in the dimensionality reduction image pair to obtain a diffusion image close to pure noise.

4. The lightweight low-light image enhancement method according to claim 3, wherein: The performing reverse denoising on the low-light image by using the diffuse image to obtain a denoised image includes: performing reverse prediction on the diffuse image, the time step, and the low-light image based on a denoising network to obtain reverse predicted noise for a next time step, and denoising the low-light image based on the reverse predicted noise to obtain an initial denoised image; Calculating the Gaussian noise gradually added with the fixed variance and the reverse prediction noise to obtain a mean square error; After updating the denoising formula of the denoising network according to the mean square error, reverse iterative denoising is performed on the subsequent time steps, the initial denoised image, and the low-light image according to the updated denoising network, and a denoised image is obtained when the time step is completed.

5. The lightweight low-light image enhancement method according to claim 1, wherein: The performing detail enhancement and color correction on the denoised image to obtain a target image includes: Performing detail enhancement on the denoised image based on a lightweight SR module to obtain a reconstructed image; The reconstructed image is color corrected based on a color enhancement module to obtain a target image.

6. The lightweight low-light image enhancement method according to claim 5, wherein: The step of performing detail enhancement on the denoised image based on a lightweight SR module to obtain a reconstructed image includes: After inputting the denoised image into the lightweight SR module, shallow feature extraction is performed on the denoised image to obtain shallow features; Performing deep feature extraction on the shallow features based on multiple separable distillation blocks to obtain feature information at different depths; Perform multi-layer feature fusion on all feature information to obtain refined fusion features; Feature reconstruction is performed on the refined and fused features to obtain a reconstructed image.

7. The lightweight low-light image enhancement method according to claim 6, wherein: After the denoised image is input into the lightweight SR module, shallow features are extracted from the denoised image to obtain shallow features, including: Copying the denoised image to obtain multiple copied images; splicing the multiple replicated images along a channel dimension to obtain a spliced ​​image; Perform shallow feature extraction on the spliced ​​image according to the convolution module to obtain shallow features.

8. The lightweight low-light image enhancement method according to claim 5, wherein: The color correction of the reconstructed image based on the color enhancement module to obtain a target image includes: After inputting the reconstructed image into the color enhancement module, normalizing the reconstructed image to obtain a processed image; Performing coarse-grained feature extraction on the processed image through deep convolution to obtain a coarse-grained feature map; Performing an activation operation on the coarse-grained feature map based on a SimpleGate unit to obtain an enhanced feature map; Inputting the enhanced feature map into the SCA attention module to enhance the channel feature information to obtain enhanced features; The enhanced features are input into a color correction module to obtain a target image.

9. The lightweight low-light image enhancement method according to claim 8, wherein: The step of inputting the enhanced features into a color correction module to obtain a target image comprises: The RGB channels of the enhanced features are adjusted using the three-dimensional weight vector of the color correction module to obtain a target image.

10. A lightweight low-light image enhancement device, characterized in that: include: A data acquisition module is used to acquire an image dataset and construct a diffusion model; the image dataset includes an image pair consisting of a low-light image and a normal-light image; A feature extraction module, configured to extract key features and reduce the image size of the image pairs in the image dataset to obtain reduced-dimensional image pairs; an image diffusion module, configured to perform forward diffusion on the normal-light image in the dimensionality-reduced image pair based on the diffusion model to obtain a diffused image, and perform reverse denoising on the low-light image using the diffused image to obtain a denoised image; The detail enhancement module is used to perform detail enhancement and color correction on the denoised image to obtain a target image.

Images