Low-illumination image enhancement modeling method, enhancement method, device and storage medium

Abstract

The application discloses a low-illumination image enhancement modeling method and device, an enhancement method and a storage medium, and relates to the field of image enhancement. The method comprises training a low-illumination image enhancement model by using training images, and the training process comprises the following steps: inputting the training images into a color gamut conversion module to obtain a linear rawRGB image and a reconstructed sRGB image; inputting the linear rawRGB image into a down-sampling module to obtain a first sub-image and a second sub-image; performing enhancement processing on the second sub-image; extracting a frequency domain brightness guide signal of the first sub-image and the enhanced second sub-image to generate a guide signal; based on the guide signal, using an illumination enhancement decoupling network to estimate a reflectivity component, an illumination component and a brightness enhancement parameter in parallel; and calculating a multi-task loss function based on a model output result and updating model parameters. The application solves the dependence on paired training data which is difficult to obtain, and maintains excellent brightness improvement and detail restoration effects.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a low-illumination image enhancement modeling method and an enhancement method based on multi-domain joint processing, a device, and a storage medium. BACKGROUND

[0002] Low-illumination images exist widely in practical applications such as night monitoring, intelligent driving, and unmanned aerial vehicle aerial photography. Due to insufficient environmental illumination, camera sensor gain improvement, and atmospheric scattering, such images generally have low overall brightness, significantly reduced contrast, a large amount of lost detail information, and significantly enhanced noise. These degradation phenomena seriously restrict the performance of subsequent visual tasks (such as target detection, tracking, and identification), and directly affect the reliability and practicality of related systems. Especially on embedded platforms such as unmanned aerial vehicles, vehicle-mounted devices, or mobile terminals, real-time processing requirements and limited computing resources further exacerbate the technical challenges of low-illumination image enhancement.

[0003] Early low-illumination image enhancement methods are mainly based on traditional image processing techniques, typical representatives of which include histogram equalization, gamma correction, and decomposition methods based on the Retinex theory. Histogram equalization enhances the global contrast by redistributing pixel intensity, but it easily leads to overexposure in local regions, loss of details, and amplification of noise; gamma correction only adjusts the overall brightness through nonlinear mapping, which is simple but lacks local adaptive ability and is difficult to improve dark areas while protecting bright area information; decomposition methods based on the Retinex theory attempt to decompose the image into an illumination component and a reflection component, and restore color and details through a multi-scale strategy, but such methods rely heavily on artificially designed priors and parameters, and are prone to color distortion, halos, and detail distortion under extreme low-light or complex noise conditions.

[0004] In recent years, with the rapid development of deep learning technology, data-driven image enhancement methods have shown great potential. Such methods can automatically learn the complex mapping relationship between low-illumination images and normally exposed images from a large number of samples, and integrate semantic information to achieve more intelligent and adaptive enhancement. According to whether the training process relies on paired low-light-normal-light images, existing deep learning methods can be divided into two categories: supervised and unsupervised. Supervised methods jointly learn brightness enhancement, detail reconstruction, and noise suppression in an end-to-end manner by constructing an accurate paired training dataset, and have superior performance. However, obtaining high-quality and paired low-illumination and normally exposed images is costly and difficult in practice, which limits their generalization ability in real-world scenarios.

[0005] To overcome the dependence of supervised learning on paired data, unsupervised or self-supervised methods have emerged. These methods achieve image enhancement without relying on paired reference images through mechanisms such as perceptual loss, structure preservation, and prior modeling, significantly reducing the cost of training data acquisition and improving the adaptability of the model to unknown scenes. However, under extreme low-light conditions, existing unsupervised and self-supervised methods still face many severe challenges:

[0006] (1) In scenes with extremely uneven light distribution, local regions of the image may appear underexposed or overexposed. Unsupervised or self-supervised methods usually enhance under global consistency constraints, making it difficult to achieve local adaptive brightness adjustment, which can lead to saturation distortion in high-light areas and insufficient recovery of details in dark areas, affecting the visual consistency and structural fidelity of the enhanced results.

[0007] (2) In outdoor dynamic environments (such as road monitoring and unmanned aerial vehicle aerial photography), environmental light sources often have sparsity, abruptness, and diversity (such as mixed lighting of car lights, streetlights, and natural light), making it difficult for unsupervised loss functions based on statistical or physical priors to stably capture global light patterns. In scenes with frequent light source flickering or variable direction, the network may have inconsistent mapping strategies, resulting in brightness jitter, color drift, or artifact enhancement in the output image, severely affecting the stability and reliability of the enhancement effect.

[0008] (3) In embedded deployment environments (such as unmanned aerial vehicles, mobile terminals, or edge computing devices), due to limited GPU power and memory resources, most unsupervised enhancement models based on complex deep networks cannot achieve real-time inference on high-resolution inputs. Existing methods are mostly designed and validated based on server-level GPUs, without fully considering model lightweight and inference efficiency, making it difficult to meet real-time processing requirements on resource-constrained platforms.

[0009] In summary, existing low-light image enhancement methods, especially unsupervised and self-supervised paradigms, still struggle to achieve robust brightness enhancement, effective noise and artifact suppression, and real-time processing on embedded platforms under extreme lighting conditions. Therefore, there is an urgent need for a self-supervised low-light image enhancement solution that does not require paired reference images, can handle complex lighting changes, and balances enhancement quality and computational efficiency, to promote its practical application in low-resource embedded vision systems. SUMMARY

[0010] The present application aims to provide a low-light image enhancement modeling method, an enhancement method, a device, and a storage medium to solve the problems of the need for paired reference images in supervised methods, the inability of unsupervised or self-supervised methods to handle complex lighting changes, and the difficulty in balancing enhancement quality and computational efficiency.

[0011] This invention solves the above-mentioned technical problems through the following technical solution: a low-light image enhancement modeling method, comprising:

[0012] A low-light image enhancement model is constructed, the model including a color gamut conversion module and an enhanced backbone network;

[0013] The color gamut conversion module includes:

[0014] The non-linear transformation module is used to map the input image from the sRGB color gamut to the linear rawRGB color gamut;

[0015] The affine color mapping module is used to perform feature analysis on a linear rawRGB image and generate affine transformation parameters, and reconstruct an sRGB image based on the affine transformation parameters.

[0016] The enhanced backbone network includes:

[0017] The downsampling module is used to sample the linear rawRGB image and generate a first sub-image and a second sub-image with independent noise distribution and consistent pixel mean.

[0018] An image enhancement submodule is used to perform brightness enhancement processing on the second sub-image;

[0019] The frequency domain luminance guidance module is used to extract frequency domain luminance guidance signals from the first sub-image and the enhanced second sub-image respectively, and generate guidance signals;

[0020] The illumination enhancement decoupling network is a parameter-sharing multi-task network, and is used to estimate the reflectance component, illuminance component and brightness enhancement parameter of the first sub-map and the second sub-map respectively according to the guiding signal;

[0021] The low-light image enhancement model is trained using training images. The training process includes:

[0022] The training image is input into the color gamut conversion module to obtain a linear rawRGB image and a reconstructed sRGB image;

[0023] The linear rawRGB image is input into the downsampling module to obtain the first sub-image and the second sub-image.

[0024] Enhance the second subgraph;

[0025] Extract the frequency domain brightness guidance signals from the first sub-image and the enhanced second sub-image to generate a guidance signal;

[0026] Based on the guiding signal, a light enhancement decoupling network is used to estimate the reflectivity component, illuminance component, and brightness enhancement parameters in parallel.

[0027] The multi-task loss function is calculated based on the model output result, and the model parameters are updated through back propagation.

[0028] Further, the nonlinear transformation module is a nonlinear activation-free network module, which realizes nonlinear mapping from sRGB color gamut to rawRGB color gamut by stacking a double residual structure and introducing a simplified channel attention mechanism.

[0029] Further, the affine color mapping module generates the affine transformation parameters through a lightweight attention mechanism, specifically including:

[0030] The linear rawRGB image is expanded in the channel dimension and inserted into the luminance channel to construct a multi-channel feature map;

[0031] The multi-channel feature map is subjected to a deep convolution and a pooling operation to generate a query vector, a key vector and a value vector;

[0032] The color mapping matrix, the bias vector and the luminance correction parameter are output through the interaction calculation of the query vector, the key vector and the value vector.

[0033] Further, the down-sampling module is a neighborhood down-sampling module, which generates the first sub-graph and the second sub-graph by randomly sampling adjacent pixels within each pixel block, so that the noise distribution of the first sub-graph and the second sub-graph is independent and the pixel mean is consistent, to construct an image pair required for self-supervised training.

[0034] Further, the specific process of the frequency domain luminance guide module for generating a guide signal includes:

[0035] The image is subjected to a discrete cosine transform to obtain a frequency spectrum in the frequency domain;

[0036] The frequency spectrum in the frequency domain is divided into a plurality of sub-bands, and each sub-band is converted back to the spatial domain through a discrete cosine inverse transform to obtain a plurality of frequency domain sub-graphs;

[0037] The frequency domain sub-graphs and the gray scale graph of the original image are spliced in the channel dimension to form a multi-channel fusion feature;

[0038] Based on the multi-channel fusion feature, a guide signal for reflectance estimation and illumination estimation is generated through a trainable parameter combination.

[0039] Further, the illumination enhancement decoupling network is composed of a reflectance network, an illumination network and an enhancement network with the same structure and shared parameters;

[0040] The reflectance network is configured to estimate a reflectance component based on a reflectance guide signal.

[0041] The illumination network is configured to estimate an illumination component based on the illumination guidance signal.

[0042] The enhancement network is configured to estimate a brightness enhancement parameter.

[0043] The illumination component is multiplied pixel by pixel with the reflectance component after being corrected by the brightness enhancement parameter to obtain a final enhanced output image.

[0044] Further, an expression of the multi-task loss function is as follows:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] wherein, denotes a total multi-task loss; denote weight coefficients corresponding to respective losses; denotes a color gamut conversion loss; denotes a reflectance loss; denote illumination losses of the first sub-image and the enhanced second sub-image respectively; denotes a consistency loss; denotes a brightness enhancement loss; denotes an original input sRGB image; denotes a color mapping matrix; denotes a linear rawRGB image; denotes a bias vector; denotes a brightness correction parameter; denotes a two-norm; denotes a reflectance component estimated from the first sub-image; denotes a reflectance component estimated from the enhanced second sub-image; denotes a weight coefficient of a noise regularization term denotes a noise map estimated from the first sub-image; ​​​​​​​​​denotes a noise map estimated from the enhanced second subgraph; denotes a reflectance network; denotes the brightness information of the enhanced second subgraph; denotes element-wise multiplication; denotes the illumination component estimated from the first subgraph; denotes the first subgraph; denotes the RGB channel maximum in the illumination component estimated from the first subgraph; denotes a gradient clipping operation; denotes the gradient of the illumination component of the first subgraph; denotes the illumination component estimated from the enhanced second subgraph; denotes the RGB channel maximum in the illumination component estimated from the enhanced second subgraph; denotes the enhanced second subgraph; denotes the brightness enhancement parameter estimated by the enhancement network; denotes the gradient of the illumination component of the enhanced second subgraph; N denotes the total number of pixels of the image; denotes a certain pixel point in the image ; denotes the pixel point surrounding neighborhood of the pixel point ; denotes the brightness value at the pixel point in the final enhanced sRGB image; denotes the brightness value at the pixel point in the final enhanced sRGB image; denotes the brightness value at the pixel point in the original input sRGB image; denotes the brightness value at the pixel point in the original input sRGB image; denotes a preset brightness enhancement ratio; and denote the brightness values of the enhanced image at adjacent pixels p and q, respectively; denotes a set of all adjacent pixel pairs in the final enhanced sRGB image.

[0054] Based on the same concept, the present application also provides a low-illumination image enhancement method, comprising:

[0055] inputting a low-illumination sRGB image to be processed into a pre-trained low-illumination image enhancement model; wherein the low-illumination image enhancement model is pre-trained using the low-illumination image enhancement modeling method as described above;

[0056] The nonlinear conversion module is used for mapping a low-illumination sRGB image to be processed from an sRGB color gamut to a linear rawRGB color gamut, to obtain a linear rawRGB image;

[0057] The frequency domain brightness guide module is used for extracting a frequency domain brightness guide signal from the linear rawRGB image, to generate a guide signal;

[0058] The illumination enhancement decoupling network is used for generating an enhanced image according to the guide signal and the linear rawRGB image;

[0059] The affine color mapping module is used for converting the enhanced image from the rawRGB color gamut back to the sRGB color gamut, to obtain a final enhanced sRGB image.

[0060] Based on the same concept, the present application also provides an electronic device, comprising a memory, a processor and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the low-illumination image enhancement modeling method or the low-illumination image enhancement method as described above.

[0061] Based on the same concept, the present application also provides a computer readable storage medium, which stores a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the low-illumination image enhancement modeling method or the low-illumination image enhancement method as described above.

[0062] Compared with the prior art, the present application has the following beneficial effects:

[0063] The present application can directly generate a high-quality self-supervised training pair from a single low-illumination image by using the downsampling module, and completely eliminates the dependence on paired training data which is difficult to obtain; in combination with the Retinex decomposition mechanism of the frequency domain brightness guide, the model can still achieve excellent brightness enhancement and detail recovery effects in extreme low light, noise complexity and other no-reference real scenes;

[0064] The present application creatively integrates self-supervised image enhancement and self-supervised noise reduction tasks into a unified framework, isolates noise in the reflectance component by using a noise priori regular term in the reflectance loss, and increases a strong gradient smoothing constraint in the illumination loss, so that the model can effectively suppress noise amplification and color distortion while greatly enhancing the brightness of the image, avoiding the common problems of amplification and blocky artifacts in traditional methods, and outputting a result that is more clear and natural in subjective vision.

[0065] The present application aims at the bottleneck of limited computing power of embedded platforms, and adopts multiple lightweight designs. First, through the parameter sharing illumination enhancement decoupling network, the same set of network parameters can complete the estimation task of reflectivity, illumination and enhancement parameters with different guide vectors, greatly reducing the model parameter amount. Secondly, the non-linear activation-free module (NAF) and affine color mapping module (ACMB) in the color gamut conversion module all adopt depth separable convolution and lightweight attention mechanism, which is high in calculation efficiency. Finally, the whole model has extremely low calculation complexity and memory occupation while maintaining high enhancement quality, and the inference speed is extremely fast, fully meeting the real-time processing requirements of embedded devices such as unmanned aerial vehicles, autonomous driving and mobile phones.

[0066] The frequency domain brightness guide module introduced in the present application provides strong frequency domain prior knowledge, so that the Retinex decomposition process is more in line with the physical law, and has stronger adaptability to complex scenes such as uneven illumination and mixed multi-light sources. At the same time, the consistency loss ensures that the local structure of the image before and after enhancement is maintained, and the enhancement of the brightness loss restricts the global naturalness of the enhancement result. This multi-angle and multi-task joint optimization strategy improves the generalization ability of the algorithm in unknown scenes and the stability of the output result, avoiding overfitting. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0068] Figure 1 It is a low-illumination image enhancement modeling method flowchart in the embodiment of the present application;

[0069] Figure 2 It is a low-illumination image enhancement model architecture diagram in the embodiment of the present application;

[0070] Figure 3 It is an architecture diagram of the non-linear activation-free network module in the embodiment of the present application;

[0071] Figure 4 It is an ACMB module architecture diagram in the embodiment of the present application;

[0072] Figure 5 It is an enhancement backbone network architecture diagram in the embodiment of the present application;

[0073] Figure 6 It is an FLGM module architecture diagram in the embodiment of the present application;

[0074] Figure 7is a low-illumination image enhancement method flowchart in the embodiment of the present application. DETAILED DESCRIPTION

[0075] The technical solutions in the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0076] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0077] Embodiment one

[0078] The present application provides a low-illumination image enhancement modeling method. In order to solve the problem of lack of high-quality reference image pairs in low-illumination scenes, an end-to-end self-supervised learning framework is constructed, internal image pairs are constructed and mapped to learn, the generalization of the model under the condition of no reference is expanded, and the processing demand of large-scale unpaired data in actual engineering is met. In order to solve the problem that the enhancement algorithm based on deep learning is easy to amplify noise and generate artifacts in the brightness improvement process, the self-supervised image enhancement and self-supervised image denoising modules are fused in the same inference process, the noise amplification and artifact generation are effectively inhibited through joint loss constraint and shared feature extraction, and the processing efficiency is improved. In order to solve the problem of excessive consumption of computing resources caused by pixel-by-pixel processing of high-resolution images, the lightweight network design and convolution attention replacement technology are adopted, the model complexity and inference delay are significantly reduced through parameter sharing and module reuse.

[0079] Figure 1 A low-illumination image enhancement modeling method flowchart provided by the present application is shown. As shown in Figure 1 The low-illumination image enhancement modeling method of the present application includes the following steps:

[0080] Step 1: Construct a low-illumination image enhancement model.

[0081] Step 2: Train the low-illumination image enhancement model using training images.

[0082] As Figure 2As shown, the low-illumination image enhancement model includes a color space conversion module (Color Space Conversion Framework, CSCF) and an enhancement backbone network (Self-Supervised Retinex Decomposition Framework, SSRDF); the color space conversion module is responsible for the rapid conversion of images between different color spaces, and includes a nonlinear transformation module and an affine color mapping module (Affine Color Mapping Block, ACMB); the enhancement backbone network is based on the Retinex decomposition theory, and includes a down-sampling module, an image enhancement sub-module, a frequency domain luminance guidance module (Frequency-Luminance Guidance Module, FLGM), and a lighting enhancement decomposition network (Decomposition Lighting and Enhancement Network, DeLightEn-Net).

[0083] In order to avoid introducing a large amount of noise by directly performing Retinex decomposition-based image enhancement on sRGB images, the sRGB images need to be converted to rawRGB images. In this embodiment, the nonlinear transformation module is a nonlinear activation free block (Nonlinear Activation Free Block, NAF), which realizes nonlinear mapping of the sRGB color space to the rawRGB color space by stacking a double residual structure and introducing a simplified channel attention mechanism (Simplified Channel Attention, SCA). Figure 3 The architecture diagram of the nonlinear activation free block is shown, wherein (a) represents the NAF module architecture, (b) represents the SCA simplified channel attention mechanism, and (c) represents the simplified gate architecture in the NAF module. C, H, and W represent the channel number, height, and width of the input feature map, respectively. As shown in (a), the NAF module is composed of a nonlinear transformation layer, a simplified gate layer, and a residual connection layer. Figure 3 As shown, the specific process of the nonlinear activation free block for mapping the input image from the sRGB color space to the linear rawRGB color space is as follows:

[0084] The input image is first normalized by a LayerNorm layer to stabilize the training process and accelerate convergence, and then a standard convolution layer is used to expand the channel number of the input image to twice the original number, so that subsequent channel separation can be performed; then the features are extracted from the expanded feature map through deep convolution (DWConv) and sent to SCA for channel attention calculation; in SCA, the spatial features of each channel are first compressed into a global scalar through global average pooling to generate a channel-level global feature, and then the channel-level global feature passes through a lightweight gating mechanism (usually composed of two convolution layers or fully connected layers) to calculate a set of channel attention weights, thereby achieving precise color and brightness correction while keeping the model lightweight; the channel attention weight is multiplied with the feature map output by DWConv to generate an attention-weighted feature map; then the attention-weighted feature map is connected in residual with the normalized feature map, and then connected in residual with the input image; finally, a convolution layer is used to restore the channel number of the processed feature map to 3 channels to obtain the final linear rawRGB image.

[0085] After completing the mapping from the sRGB color gamut to the linear rawRGB color gamut, the color gamut conversion module also needs to perform reverse color mapping using the affine color mapping module to ensure that the enhanced image is compatible with general terminal devices and can be directly used for subsequent reasoning. The affine color mapping module of the present application automatically estimates the ISP affine matrix , bias vector and brightness correction parameter through an end-to-end feature extraction network, and applies them to the linear rawRGB image to realize color correction and brightness adjustment of the enhanced image, thereby ensuring visual consistency and reasoning robustness on different hardware platforms.

[0086] The affine color mapping module generates affine transformation parameters through a lightweight attention mechanism, as shown in Figure 4 , and the specific process includes:

[0087] The input 3-channel linear rawRGB image is copied three times for each channel to generate an initial feature map containing 9 channels, which enhances the expression ability of color information through channel replication; the channel average value of the 3-channel linear rawRGB image is calculated to generate a single-channel grayscale image as a scene brightness reference and as a new channel; the initial feature map is spliced with the grayscale channel, and other auxiliary channels are introduced to form a multi-channel feature map (at least 10 channels, 13 channels in this embodiment), which integrates original color, brightness information and context information, providing rich perceptual support for subsequent parameter estimation; the multi-channel feature map is input into a deep separable convolution (DWConv) layer for feature extraction and scale sampling, and then the output feature map is globally averaged pooled (GAP) to generate a query vector (Q) after compressing the spatial information; the multi-channel feature map processed by DWConv is divided into two parts in the channel dimension: one part of the feature map is input into a CBS module (Conv-BN-SiLU activation function sequence) for further feature transformation to extract and generate a key vector (K); the other part of the feature map is directly channel-averaged pooled to generate a value vector (V); based on the standard attention mechanism, the generated Q, K and V vectors are used for interactive operation, and the result of the interactive operation is reshaped to finally analyze the affine transformation parameters to be estimated, including the ISP affine matrix , bias vector and brightness correction parameter The affine transformation parameters constitute an affine mapping and brightness adjustment scheme from the rawRGB color gamut to the sRGB color gamut, providing a key basis for color restoration and brightness consistency of the enhanced image, ensuring the reliability and accuracy of subsequent inference tasks.

[0088] The ISP affine matrix , bias vector is used to perform affine transformation on the input linear RawRGB image, and the specific formula is:

[0089] (1)

[0090] Where, · represents matrix multiplication. This operation realizes linear color mapping from the rawRGB color gamut to the sRGB color gamut.

[0091] The brightness correction parameter is used to perform final brightness fine-tuning on the image after affine transformation to prevent overexposure or underexposure and ensure visual consistency:

[0092] (2)

[0093] Where,​ representing a reconstructed sRGB image.

[0094] The design idea of the enhanced backbone network is to deeply integrate self-supervised image enhancement and self-supervised noise reduction based on Retinex theory: in the training stage, three subnetworks, namely a reflectance network (R-Net), an illumination network (L-Net), and an enhancement network (E-Net), are constructed, and the three subnetworks are collectively referred to as DeLightEn-Net, and the network is driven to learn brightness correction and noise separation through a joint self-supervised loss; in the inference stage, DeLightEn-Net can simultaneously complete brightness enhancement and noise suppression of low-illumination images, ensuring that the output image is both bright and clear. The reflectance network estimates the reflectance component based on the reflectance guide signal; the illumination network estimates the illumination component based on the illumination guide signal; and the enhancement network estimates the brightness enhancement parameter. The structures of the reflectance network (R-Net), the illumination network (L-Net), and the enhancement network (E-Net) are completely identical, and the only difference is the input guide vector, which is generated by the frequency domain brightness guide module. By changing the guide vector, the same network can output different image analysis components.

[0095] In this embodiment, the downsampling module is a neighborhood downsampling module. As shown in Figure 5 , the neighborhood downsampling module performs random adjacent pixel sampling based on a 2x2 pixel block on the linear rawRGB image output by the nonlinear activation-free network module to generate a first subgraph and a second subgraph. The specific operation is as follows:

[0096] The linear rawRGB image is divided into several 2x2 pixel blocks, and two adjacent pixels are randomly selected in each pixel block to splice into two subgraphs, namely the first subgraph and the second subgraph. Since the first subgraph and the second subgraph are relatively independent in noise distribution and have consistent pixel means, noise estimation and removal can be performed through the Neighbor2Neighbor self-supervised noise reduction algorithm. The first subgraph and the second subgraph are simultaneously subjected to enhancement and noise reduction processing. First, the image enhancement submodule is used to perform nonlinear mapping on the pixel values in the second subgraph to achieve preliminary brightness enhancement processing, and the enhanced second subgraph is obtained.

[0097] The frequency domain brightness guide module introduces a frequency domain decomposition and reconstruction method to make the components of Retinex decomposition more accurate through frequency domain decomposition, thereby improving the accuracy of the model in restoring and reconstructing the details of the enhanced image. Figure 6 The architecture diagram of the frequency domain brightness guide module is shown. As shown in Figure 5 and Figure 6 , the specific process of generating a guide signal by the frequency domain brightness guide module includes:

[0098] For the first subgraph Or the enhanced second subgraph The image is subjected to Discrete Cosine Transform (DCT) to obtain the frequency domain spectrum; the frequency domain spectrum is divided into multiple sub-bands, and each sub-band is transformed back to the spatial domain through Inverse Discrete Cosine Transform to obtain multiple frequency domain sub-images.

[0099] Given that the DCT spectrum exhibits a continuous low-frequency to high-frequency distribution from the upper left to the lower right, a linear segmentation method is used to divide the frequency domain, retaining only the coefficients of each sub-band and setting the rest to zero. This divides the spectrum into four sub-bands, with the segmentation formula as follows:

[0100] (3)

[0101] (4)

[0102] (5)

[0103] (6)

[0104] in, These represent four frequency domain subgraphs; Represents a spectral image; Represents the pixels in a spectrum image; Represents the intercept of the spectrum image; and This indicates the height and width of the spectral image.

[0105] Compare the four frequency domain subgraphs with the original image (first subgraph). Or the enhanced second subgraph The grayscale images are stitched together along the channel dimension to form a five-channel fusion feature; based on the five-channel fusion feature, through trainable parameters ( The four frequency domain sub-images and the weight parameters of the grayscale image of the original image are combined to generate guiding signals for estimating the reflectance component R and the illuminance component L, respectively. Based on Retinex decomposition theory, R and L correspond to low-frequency and high-frequency components in the frequency domain, respectively, while the noise introduced by the enhancement mainly originates from the reflectance component itself, since L is determined by ambient light and does not contain material and texture noise.

[0106] For the first subgraph Using DeLightEn-Net, the corresponding reflectivity components can be obtained separately. , illumination component and brightness enhancement parameters For the enhanced second subgraph Using DeLightEn-Net, the corresponding reflectivity components can be obtained separately. , an illumination component , and a brightness enhancement parameter . Thus, the first sub-image and the enhanced second sub-image can be disassembled as follows:

[0107] (7)

[0108] (8)

[0109] wherein, R denotes a reflectance component , and n denotes noise in the reflectance component . Since the ambient light of the first sub-image and the second sub-image is consistent, the illumination component remains unchanged, and after the preliminary brightness enhancement on the second font, it can be represented as:

[0110] (9)

[0111] wherein, k denotes a preset brightness enhancement ratio. Formula (9) makes an approximate treatment of approximately equal to 1. First, the exponential Taylor expansion of the reflectance component and the noise is performed, and since the noise is small in value, all terms except the first-order term can be discarded. Meanwhile, since k is very close to 1 in value, k is approximately equal to 1, and thus:

[0112] (10)

[0113] Formula (7) and formula (10) jointly constitute a set of self-supervised image enhancement reference pairs. In the training process, a corresponding loss function is constructed to establish a self-supervised constraint between the two, and simultaneously guide the optimization of the self-supervised denoising process.

[0114] In the specific embodiments of the present application, the low-illumination image enhancement model is trained using training images, and the training process includes:

[0115] Step 2.1: input the training image into the color gamut conversion module to obtain a linear rawRGB image and a reconstructed sRGB image.

[0116] A plurality of low-illumination images are captured by a camera, and the low-illumination images are used as training images. The low-illumination images refer to images taken in an insufficient light or weak light environment. The non-linear transformation module of the color gamut conversion module is used to map the training image from the sRGB color gamut to the linear rawRGB color gamut to obtain a linear rawRGB image; the affine color mapping module of the color gamut conversion module is used to reconstruct the linear rawRGB image to generate a reconstructed sRGB image.

[0117] ​​​Step 2.2: input the linear rawRGB image into the downsampling module to obtain a first sub-image and a second sub-image;

[0118] Step 2.3: perform enhancement processing on the second sub-image;

[0119] Step 2.4: extract the frequency domain brightness guide signal of the first sub-image and the enhanced second sub-image to generate a guide signal (including a reflectance guide signal and an illumination guide signal);

[0120] Step 2.5: based on the guide signal, use the illumination enhancement decoupling network to estimate the reflectance component, the illumination component and the brightness enhancement parameter in parallel;

[0121] Step 2.6: calculate the multi-task loss function based on the model output result, and update the model parameters through back propagation.

[0122] In this embodiment, the multi-task loss function includes the loss function of the color gamut conversion module, the Retinex loss function (i.e. the reflectance loss function and the illumination loss function), the consistency loss function and the brightness enhancement loss function. The expression of the multi-task loss function is:

[0123] (11)

[0124] wherein, Ltotal represents the total loss of the multi-task; Lcolor, Lretinex, Lconsistency and Lbrightnessenhance represent the weight coefficients corresponding to each loss, respectively; Lcolor represents the color gamut conversion loss; Lretinex represents the reflectance loss; Lillumination1 and Lillumination2 represent the illumination loss of the first sub-image and the enhanced second sub-image, respectively; Lconsistency represents the consistency loss; Lbrightnessenhance represents the brightness enhancement loss. The calculation formula of the color gamut conversion loss is:

[0125] (12)

[0126] wherein, LsRGB represents the original input sRGB image; Lcolor represents the color mapping matrix; LrawRGB represents the linear rawRGB image; Lbias represents the bias vector; Lgamma represents the brightness correction parameter; L2 represents the two-norm. The specific calculation formula of the Retinex loss function is:

[0127] (13)

[0128] (14)

[0129] (15)

[0130] (16)

[0131] wherein, represents the reflectance component estimated from the first sub-image; represents the reflectance component estimated from the enhanced second sub-image; represents the weight coefficient of the noise regularization term is designed to compensate for the brightness disturbance error caused by neighborhood sampling, and is inspired by the noise learning theory to constrain the non-structural differences between image pairs; represents the noise map estimated from the first sub-image; represents the noise map estimated from the enhanced second sub-image; represents the reflectance network; represents the brightness information of the enhanced second sub-image; represents element-wise multiplication; represents the illumination component estimated from the first sub-image; represents the first sub-image; represents the maximum value of the RGB channel in the illumination component estimated from the first sub-image; represents the gradient clipping operation; represents the gradient of the illumination component of the first sub-image; represents the illumination component estimated from the enhanced second sub-image; represents the maximum value of the RGB channel in the illumination component estimated from the enhanced second sub-image; represents the enhanced second sub-image; represents the brightness enhancement parameter estimated by the enhancement network; represents the gradient of the illumination component of the enhanced second sub-image. Both formula (15) and formula (16) are composed of four parts: first, the two-norm loss between the sub-image (first sub-image or enhanced second sub-image) and the illumination component, reflectance component and brightness enhancement parameter is used to constrain the consistency of the image after decomposition and reconstruction; second, it is used to constrain the stability of training; third, the compensation term is used to prevent the gradient propagation of the image by detach operation; fourth, the gradient constraint of the illumination component is used to ensure the smooth continuity of the illumination, avoid learning unnatural high-frequency illumination changes, and conform to the naturalness of physical light distribution.

[0132]

[0133] ​The self-supervised loss is composed of consistency loss and brightness enhancement loss, and the principle of the consistency loss is that the pixel points in the current image and the pixel points in the neighborhood of the pixel points should have a certain range of brightness difference between the images before and after enhancement, so as to avoid unnatural brightness fluctuation in the image enhancement process, which is helpful to stabilize the training process and inhibit exposure abnormalities and enhancement artifacts. The specific formula of the consistency loss is:

[0134] (17)

[0135] Wherein, N represents the total number of pixels of the image; represents a certain pixel point in the image ; represents the pixel points in the neighborhood of the pixel point ; ; represents the brightness value of the pixel point in the sRGB image after final enhancement; represents the brightness value of the pixel point in the sRGB image after final enhancement; represents the brightness value of the pixel point in the original input sRGB image; represents the brightness value of the pixel point in the original input sRGB image.

[0136] The design of the brightness enhancement loss is mainly to realize the effective estimation of the brightness enhancement parameter, and the specific formula is:

[0137] (18)

[0138] Wherein, represents the preset brightness enhancement ratio; and respectively represent the brightness values of the enhanced images at adjacent pixels p and q; represents the set of all adjacent pixel pairs in the sRGB image after final enhancement. The first term of the brightness enhancement loss is that there is a mathematical logical relationship between the brightness enhancement parameter after the decomposition of the first subgraph and the enhanced second subgraph and the preset brightness enhancement ratio, and the second term is the consistency constraint of the RGB channel enhancement value, wherein .

[0139] The present application aims at the problems of lack of reference images in current actual engineering, easy amplification of noise in low-illumination enhancement, and difficulty in lightweight deployment of the model, and proposes a low-illumination image enhancement modeling method based on Retinex image decomposition theory and compatible with self-supervised denoising, and the main innovation points are:

[0140] Self-supervised image pair construction: Directly generate two frames of self-supervised image pairs with independent noise distribution and consistent pixel mean from a single low-illumination image using down-sampling technology, and obtain enhanced sub-images through preliminary enhancement to realize mapping learning under no-reference condition;

[0141] Frequency domain brightness guided Retinex decomposition: Introduce frequency domain information in the Retinex decomposition framework, accurately estimate the reflectance component and the illumination component through frequency domain sub-band decomposition and reconstruction, improve the accuracy of image decomposition, and enhance the detail fidelity and noise suppression effect of the reconstructed image;

[0142] Lightweight network and parameter sharing: In view of the limited embedded GPU computing power, the design of sharing parameters of DeLightEn-Net and trainable guide vector driving the same network to generate different decomposition components greatly reduces the model parameter quantity and speeds up the forward inference, and meets the real-time and resource constraint requirements.

[0143] The modeling method of the application is trained and tested on the public data sets SICE and LOLv1 / v2 data sets. The SICE data set is a large-scale multi-exposure data set for single image contrast enhancement, which contains 4413 images and covers 589 indoor and outdoor scenes. The reference image is generated by 13 kinds of multi-exposure fusion and HDR algorithm (high dynamic range imaging), and is subjected to subjective screening to ensure quality. The data set provides low-illumination images under multiple exposure levels and corresponding synthetic reference images, which are suitable for end-to-end supervised learning. The LOLv1 data set is the first real low-light and normal-light paired image data set, containing 500 strictly aligned images, mainly covering indoor scenes such as residential buildings, teaching buildings and gymnasiums. LOLv2 introduces complex degradation such as compression noise, motion blur and artifacts, generates diverse samples combined with optimized alignment and synthesis process, and covers typical scenes such as indoor, natural and human, which is an important benchmark for low-light enhancement under complex degradation conditions.

[0144] In the experiment, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are selected as evaluation indexes of low-illumination image enhancement. The higher the PSNR, the higher the image restoration degree; the closer the SSIM to 1, the stronger the structural consistency.

[0145] In addition to the above image enhancement quality indicators, the processing time RTime is used as the main basis for evaluating the computing efficiency of the current algorithm. The entire experiment is completed on Nvidia Geforce RTX 3090 GPU.

[0146] In this experiment, representative low-light image enhancement methods in recent years are selected, covering unsupervised, self-supervised and partially supervised algorithms, a total of 8 kinds: URetinexNet and SNR-aware are typical supervised methods, Zero-DCE, EnlightenGAN, RUAS, SCI, GDP, PairLIE, etc. are unsupervised or self-supervised algorithms. The experimental results are shown in Table 1.

[0147] Table 1 Summary of experimental results

[0148]

[0149] Table 1 lists the PSNR and SSIM comparison results of the method of the present application and the comparison method on LOLv1, LOLv2 and SICE data sets, and supplements the inference time comparison on the SICE data set. From Table 1, although the method of the present application cannot surpass the supervised enhancement algorithms such as URetinexNet and SNR-aware in image quality indicators, it is superior to existing unsupervised methods in most indicators on three data sets, fully proving the significant effect of the proposed algorithm in noise suppression and detail reconstruction. In addition, although the inference speed of the supervised method leads the unsupervised method, the average inference time of the present application 0.0025s compared with the fastest 0.0015s is also in an acceptable range, effectively balancing real-time and image quality. In summary, the present application realizes excellent enhancement effect and higher running efficiency without relying on high-quality reference image pairs, providing a practical solution for real-time low-light image enhancement on embedded platforms.

[0150] Example Two

[0151] As shown in Figure 7 , the low-light image enhancement method provided by the present application comprises the following steps:

[0152] Step 1: input the low-light sRGB image to be processed into a pre-trained low-light image enhancement model; wherein the low-light image enhancement model is pre-trained using the low-light image enhancement modeling method in Example One of the present application;

[0153] Step 2: use a nonlinear transformation module to map the low-light sRGB image to be processed from the sRGB color gamut to the linear rawRGB color gamut, to obtain a linear rawRGB image;

[0154] Step 3: use a frequency domain brightness guide module to extract a frequency domain brightness guide signal from the linear rawRGB image, to generate a guide signal, including a reflectance guide signal and an illumination guide signal;

[0155] Step 4: generating an enhanced image from the guidance signal and the linear rawRGB image using the illumination enhancement disentangled network;

[0156] Step 5: converting the enhanced image from the rawRGB color gamut back to the sRGB color gamut using the affine color mapping module to obtain the final enhanced sRGB image.

[0157] Embodiment Three

[0158] The embodiments of the present application also provide an electronic device, which comprises a memory, a processor and a computer program or instructions stored in the memory, and the processor executes the computer program or instructions to implement the low-light image enhancement modeling method or the low-light image enhancement method in the embodiments of the present application.

[0159] Although not shown, the electronic device comprises a processor, which can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or loaded from a storage section into a random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), etc. In the RAM, various programs and data required for device operation are also stored. The processor, the ROM and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0160] The above processor and memory are used together to execute programs / instructions stored in the memory, and the programs / instructions are executed by a computer to implement the methods, steps or functions described in the above embodiments.

[0161] Although not shown, the embodiments of the present application also provide a computer readable storage medium having a computer program or instructions stored thereon, and the computer program or instructions are executed by a processor to implement the low-light image enhancement modeling method or the low-light image enhancement method in the embodiments of the present application.

[0162] Read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information which can be accessed by a computing device. According to the definition used herein, computer readable medium does not include transitory media, such as modulated data signals and carrier waves.

[0163] The above descriptions are only specific embodiments of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or modifications within the technical scope disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A low-light image enhancement modeling method, characterized in that, The modeling method includes: A low-light image enhancement model is constructed, the model including a color gamut conversion module and an enhanced backbone network; The color gamut conversion module includes: The non-linear transformation module is used to map the input image from the sRGB color gamut to the linear rawRGB color gamut; The affine color mapping module is used to perform feature analysis on a linear rawRGB image and generate affine transformation parameters, and reconstruct an sRGB image based on the affine transformation parameters. The enhanced backbone network includes: The downsampling module is used to sample the linear rawRGB image and generate a first sub-image and a second sub-image with independent noise distribution and consistent pixel mean. An image enhancement submodule is used to perform brightness enhancement processing on the second sub-image; A frequency domain luminance guidance module is used to extract frequency domain luminance guidance signals from the first sub-image and the enhanced second sub-image, respectively, and generate guidance signals. Its specific implementation process includes: performing a discrete cosine transform on the image to obtain a frequency domain spectrum; dividing the frequency domain spectrum into multiple sub-bands, and converting each sub-band back to the spatial domain through an inverse discrete cosine transform to obtain multiple frequency domain sub-images; concatenating the frequency domain sub-images with the grayscale image of the original image in the channel dimension to form a multi-channel fusion feature; and generating guidance signals for reflectivity estimation and illuminance estimation respectively through trainable parameter combinations based on the multi-channel fusion feature. The illumination enhancement decoupling network is a parameter-sharing multi-task network, and is used to estimate the reflectance component, illuminance component and brightness enhancement parameter of the first sub-map and the second sub-map respectively according to the guiding signal; The low-light image enhancement model is trained using training images. The training process includes: The training image is input into the color gamut conversion module to obtain a linear rawRGB image and a reconstructed sRGB image; The linear rawRGB image is input into the downsampling module to obtain the first sub-image and the second sub-image. Enhance the second subgraph; Extract the frequency domain brightness guidance signals from the first sub-image and the enhanced second sub-image to generate a guidance signal; Based on the guiding signal, a light enhancement decoupling network is used to estimate the reflectivity component, illuminance component, and brightness enhancement parameters in parallel. The multi-task loss function is calculated based on the model output, and the model parameters are updated through backpropagation.

2. The low-light image enhancement modeling method according to claim 1, characterized in that, The nonlinear transformation module is a nonlinear activation-free network module. The nonlinear activation-free network module achieves nonlinear mapping from sRGB color gamut to rawRGB color gamut by stacking a double residual structure and introducing a simplified channel attention mechanism.

3. The low-light image enhancement modeling method according to claim 1, characterized in that, The affine color mapping module generates the affine transformation parameters through a lightweight attention mechanism, specifically including: A multi-channel feature map is constructed by expanding the linear rawRGB image in the channel dimension and inserting the luminance channel. The multi-channel feature map is subjected to depthwise convolution and pooling operations to generate query vector, key vector and value vector; By interactively calculating the query vector, key vector, and value vector, the color mapping matrix, bias vector, and brightness correction parameters are output.

4. The low-light image enhancement modeling method according to claim 1, characterized in that, The downsampling module is a neighborhood downsampling module. The neighborhood downsampling module generates the first sub-image and the second sub-image by randomly sampling adjacent pixels within each pixel block, so that the noise distribution of the first sub-image and the second sub-image are independent and the pixel mean is consistent, so as to construct the image pair required for self-supervised training.

5. The low-light image enhancement modeling method according to claim 1, characterized in that, The illumination enhancement decoupling network consists of a reflectivity network, an illuminance network, and an enhancement network with identical structures and shared parameters; The reflectivity network is used to estimate reflectivity components based on the reflectivity guiding signal; The illuminance network is used to estimate illuminance components based on the illuminance guidance signal; The enhancement network is used to estimate the brightness enhancement parameters; The illuminance component, after being corrected by the luminance enhancement parameter, is multiplied pixel by pixel with the reflectance component to obtain the final enhanced output image.

6. The low-light image enhancement modeling method according to any one of claims 1 to 5, characterized in that, The expression for the multi-task loss function is: ； ； ； ； ； ； ； ； in, Indicates the total loss from multiple tasks; These represent the weighting coefficients corresponding to each type of loss; Indicates the color gamut conversion loss; Indicates reflectivity loss; These represent the illuminance loss in the first sub-image and the enhanced second sub-image, respectively. This indicates a loss of consistency. This indicates a loss in brightness enhancement; This represents the original input sRGB image; Represents the color mapping matrix; Represents a linear rawRGB image; Represents the bias vector; Indicates brightness correction parameters; Represents the L2 norm; This represents the reflectance component estimated from the first subgraph; This represents the reflectance component estimated from the enhanced second submap; Represents the noise regularization term Weighting coefficients; This represents the noise map estimated from the first subgraph; This represents the noise map estimated from the enhanced second submap; Represents a reflectivity network; This represents the brightness information of the enhanced second sub-image; This indicates element-wise multiplication; This represents the illuminance component estimated from the first subgraph; This represents the first subgraph; This represents the maximum value of the RGB channels in the illuminance component estimated from the first submap; This indicates the gradient truncation operation; This represents the gradient of the illuminance component in the first subplot; This represents the illuminance component estimated from the enhanced second submap; This represents the maximum value of the RGB channels in the illuminance component estimated from the enhanced second submap; This represents the enhanced second subgraph; This represents the brightness enhancement parameters estimated by the enhancement network; This represents the gradient of the illuminance component in the enhanced second sub-image; N represents the total number of pixels in the image. Represents a pixel in an image ; Represents pixels Pixels in the surrounding neighborhood ; Represents the pixels in the final enhanced sRGB image. The brightness value at that location; Represents the pixels in the final enhanced sRGB image. The brightness value at that location; Represents the pixels in the original input sRGB image. The brightness value at that location; Represents the pixels in the original input sRGB image. The brightness value at that location; This indicates the preset brightness enhancement ratio; and These represent the brightness values ​​of the enhanced image at adjacent pixels p and q, respectively; This represents the set of all adjacent pixel pairs in the final enhanced sRGB image.

7. A low-light image enhancement method, characterized in that, The enhancement method includes: The low-light sRGB image to be processed is input into a pre-trained low-light image enhancement model; wherein, the low-light image enhancement model is pre-trained using the low-light image enhancement modeling method as described in any one of claims 1 to 6; The low-light sRGB image to be processed is mapped from the sRGB color gamut to the linear rawRGB color gamut using a non-linear transformation module to obtain a linear rawRGB image. The frequency domain luminance guidance signal is extracted from the linear rawRGB image using the frequency domain luminance guidance module to generate a guidance signal; An enhanced image is generated using an illumination enhancement decoupling network based on the guiding signal and the linear rawRGB image; The enhanced image is converted from the rawRGB color gamut back to the sRGB color gamut using the affine color mapping module, resulting in the final enhanced sRGB image.

8. An electronic device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The processor executes the computer program or instructions to implement the low-light image enhancement modeling method as described in any one of claims 1 to 6 or the low-light image enhancement method as described in claim 7.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by the processor, they implement the low-light image enhancement modeling method as described in any one of claims 1 to 6 or the low-light image enhancement method as described in claim 7.

Images