A low-light enhancement method and system based on curve family functions

Abstract

The present invention discloses a low-light enhancement method and system based on a curve family function. The method comprises the following steps: 1) using a dataset consisting of low-light images to train a low-light enhancement network model, which includes a feature extractor, a parameter estimator, and a curve family function; the feature extractor is used to downsample the input image and then perform feature extraction on the resulting downsampled image; the parameter estimator is used to calculate the curve family parameter values of the curve family function corresponding to the input image based on the features extracted by the feature extractor; 2) inputting the low-light image to be enhanced into the trained low-light enhancement network model, the feature extractor extracting features from the low-light image to be enhanced, obtaining corresponding image features and inputting these features into the parameter estimator, obtaining corresponding curve family parameters α and assigning them to the curve family function; and 3) applying the curve family function obtained in step 2) to each pixel of the low-light image to be enhanced to obtain a light-enhanced image.

Description

Technical Field

[0001] The present invention belongs to the field of low-light enhancement of digital images, and relates to an implementation method and a training method of an unsupervised low-light enhancement model based on an enhancement curve family. Background Art

[0002] Low light is a common source of image degradation, often caused by low-light shooting environments, incorrect parameter settings, camera malfunction, and improper user operation. Improving the brightness of low-light images has long been a focus of academic and industrial research. Currently, popular enhancement methods mostly utilize deep learning, using neural networks to directly modify and synthesize individual pixels. These existing methods employ paired training data, contrastive learning, or retinal color constancy theory to ensure the quality of the training results. Furthermore, these brightening methods often employ operations such as noise reduction and contrast enhancement to improve image quality at the human visual level.

[0003] However, these methods require network models with numerous parameters, which carries the risk of overfitting. Due to the model's complexity, training often consumes considerable time and resources. These methods also have limited robustness and are prone to noise. Furthermore, to improve image quality at the human visual level, these methods sacrifice much of the high-dimensional semantics that computers can extract. Summary of the Invention

[0004] To address the above technical issues, this paper proposes an unsupervised low-light enhancement method and system based on a family of enhancement curves. This method can achieve image light enhancement with shorter runtimes and smaller network models, while also preserving computer-extractable high-dimensional semantics.

[0005] The technical solution adopted in the present invention is as follows:

[0006] A low-light enhancement method based on a curve family includes the following steps:

[0007] Using a dataset consisting of unpaired and unannotated low-light images to train a low-light enhancement network model; the low-light enhancement network model includes a feature extractor, a parameter estimator, and a family of curve functions; the feature extractor includes a downsampling layer and a number of consecutive convolutional layers, and each convolutional layer is followed by a rectified linear unit function. The downsampling layer reduces the size of the original image and decreases the number of parameters required for the entire model. The role of the convolutional layer and the rectified linear unit function is to extract the features of the image and ensure that the feature values are non-negative. The features of different convolutional layers will be added together and used as the input for the next convolutional layer; the parameter estimator includes a global average pooling layer and a rectified linear unit function. The global average pooling layer calculates the parameter values of the family of curves based on the obtained feature vectors, and the rectified linear unit function ensures that the parameter values are non-negative; the family of curve functions is a manually defined family of functions used to map each pixel of the input image.

[0008] Input the low-light image to be enhanced into the trained low-light enhancement model for the following processing steps:

[0009] Input the image to be enhanced into the feature extractor to obtain image features;

[0010] Input the image features into the parameter estimator to obtain appropriate parameter values for the family of curves;

[0011] Apply the established family of curve functions and the parameters from the previous step to each pixel of the image to be enhanced to obtain the final image after light enhancement;

[0012] Furthermore, the convolutional layer of the feature extractor has n (n≥3) layers, and each convolutional layer is accompanied by a rectified linear unit function. The i-th convolutional layer extracts features from its input and passes through the rectified linear unit function to obtain the output result of this layer. The first convolutional layer takes the original low-light image as input, the input of the i-th (2≤i<n) convolutional layer is the output of the previous convolutional layer, and the input of the n-th convolutional layer is the sum of the output results of the previous n-1 convolutional layers.

[0013] Furthermore, the parameter estimator includes a global average pooling layer and a rectified linear unit function.

[0014] A low-light enhancement system based on a family of curve functions includes a low-light enhancement network model, which, after training, processes the input low-light image to be enhanced to obtain an image with increased brightness; the network model includes:

[0015] A feature extractor, including a downsampling layer and a number of consecutive convolutional layers, each convolutional layer is followed by a rectified linear unit function, and the sum of the output results of the first two rectified linear unit functions is used as the input for the third convolutional layer together; this module is used to extract the features of the image.

[0016] The parameter estimator includes a global average pooling layer and a linear rectification function; this module determines the parameters of the curve family based on the characteristics of the image.

[0017] The curve family function is an artificially determined function used to map the pixels of the original image according to the parameters obtained by the parameter estimator to obtain the final image with improved brightness.

[0018] Compared with the prior art, the present invention has the following positive effects:

[0019] By adjusting the distribution of the neural network's feature space and channel dimensions, this method achieves effective low-light enhancement with only a single parameter forward pass. Compared to existing technologies, this method achieves superior overall results in terms of neural network robustness, number of network model parameters, training speed, and processing speed, while also preserving the high-dimensional semantics of the enhanced image at the computational level. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a structural diagram of the low-light enhancement network used in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] To further enhance the aforementioned features and advantages of the present invention, the following embodiments are provided with accompanying drawings for detailed description. It should be noted that the specific number of layers, modules, functions, and layer configurations given in the following embodiments are merely preferred implementations and are not intended to be limiting. Persons skilled in the art will appreciate that they can select the number and configuration of layers based on actual needs.

[0022] This embodiment discloses a low-light enhancement method based on an enhancement curve family. Taking low-light enhancement of a target content image as an example, the specific description is as follows:

[0023] Step 1: Collect a large number of unpaired and unlabeled low-light images to form a low-light dataset H.

[0024] Step 2: Build a low-light enhancement network model.

[0025] The network structure is as follows Figure 1 As shown, the model is divided into feature extractor E f , parameter estimator M and curve family function F. Feature extractor E f First, the input image I selected from H is downsampled to a 16×16 image. Three convolutional layers are then used to extract features from the downsampled image. Each convolutional layer is followed by a rectified linear unit (ReLU). The network model adds the outputs of the first and second rectified linear units (ReLUs) and uses this as the input to the third convolutional layer.

[0026] The parameter estimator M consists of a global average pooling layer and a linear rectification function (ReLU). This module calculates the value of the curve group parameter α based on the feature matrix extracted by the previous module and ensures that the value is non-negative.

[0027] The global average pooling layer calculation formula is:

[0028]

[0029] Where, α global is the parameter estimated by the global average pooling layer operation, S is the feature matrix output by the feature extractor, C, H, W are the total number of channels, total length and total width of the feature matrix, S i,j,k is the value of the i-th channel (j, k) position of the feature matrix.

[0030] The linear rectification function calculation formula is: α=max(0,α global )

[0031] Where α is the final estimated parameter value.

[0032] The curve family function F is a given family of functions f(x, α), whose specific output is determined by the input value x and the parameter α. After the parameter estimator M calculates α, the network model applies the function mapping to each pixel of the input image I to obtain the output image O.

[0033] When determining the curve family function f, in order to ensure the stability and continuity of the generated image, f must satisfy the following four properties:

[0034] 1. Contrast unchanged: f(0,α)=0,

[0035] 2. Content consistency: f is monotonic and differentiable;

[0036] 3. Full domain coverage: For any x0, y0, there exists α such that f(x0, α) = y0;

[0037] 4. Strict monotonicity: the derivative of f for any x in the domain is non-zero

[0038] The following curve function families all meet the above four conditions:

[0039]

[0040] in,

[0041] In the above function, the independent variable x is a positive real value, which is the specific value of each pixel of the image in the application scenario of the present invention; the independent variable α is the parameter of the curve family, which is the parameter of the curve family to be determined by the network model in the application scenario of the present invention.

[0042] Figure 1 Reciprocal function family For example, it can also be replaced by other function families.

[0043] Step 3: Train the low-light enhancement model.

[0044] The total loss function of the model is:

[0045] L=λ exp L exp +λ spa L spa + color L color ,

[0046] Where λ exp ,λ spa ,λ color is the weight term, usually λ exp Set to 10, λ spa Set to 48, λ color Set to 5.

[0047] L exp Is the enhancement loss function:

[0048] L exp =∑ x∈I |E(x)-e|,

[0049] Where E(·) is the network model in step 2, I is the low-light image to be enhanced, x is the pixel in image I, and e is the manually set hyperparameter value used to control the degree of light enhancement.

[0050] L spa is the spatial distribution consistency loss function:

[0051] L spa =∑ j∈Ω (|E(I)-E(I) j |-|II j |) 2 ,

[0052] Where Ω is the offset set of four pixels adjacent to a pixel, i.e., up, down, left, and right; I j The image E(I) is obtained by shifting the image I by one pixel in a certain direction. j The output image E(I) is shifted by one pixel in a certain direction.

[0053] L color is the color stability loss function:

[0054] L color =(E(I) R -E(I) G ) 2 +(E(I) G -E(I) B ) 2 +(E(I) B -E(I) R ) 2 ,

[0055] E(I) R , E(I) G , E(I) B It is the component of the output image E(I) on the three RGB channels.

[0056] After calculating the total loss function term, the parameters of each convolutional layer in the feature extractor are updated through the back-propagation method for training.

[0057] Step 4: In the inference stage, for the low-light image to be enhanced, input it into the low-light enhancement model trained in step 3 to obtain the result.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A low-light enhancement method based on a curve family function, the steps of which include: 1) Using a dataset composed of low-light images to train a low-light enhancement network model; The low-light enhancement network model includes a feature extractor, a parameter estimator, and a curve family function; the feature extractor is used to downsample the input image and then extract features from the obtained downsampled image; the parameter estimator is used to calculate the curve family parameter values of the curve family function corresponding to the input image according to the features extracted by the feature extractor; 2) Input the low-light image to be enhanced into the trained low-light enhancement network model, the feature extractor extracts features from the low-light image to be enhanced, obtains the corresponding image features and inputs them into the parameter estimator, obtains the corresponding curve family parameter α and assigns it to the curve family function; 3) Apply the curve family function obtained in step 2) to each pixel of the low-light image to be enhanced to obtain a light-enhanced image; The curve family function is a set function family f(x,α), which is used to map each pixel value x of the input image to obtain the corresponding output image; among them, the function family f(x,α) satisfies four properties: a) For any α, f(0,α)=0, f(x,α)=1; b) The function family f(x,α) is monotonic and differentiable; c) For any positive real numbers x0, y0, there exists α such that f(x0,α)=y0; d) For any α, the derivative of the function family f(x,α) with respect to any x in the domain is non-zero; The curve family function is: an exponential function family, a reciprocal function family, a power function family, or an arctangent function family; The expression of the exponential function family is f(x,α)=(α x -1) / (α-1), where the parameter range is α∈[0,1)∪(1,+∞); the expression of the reciprocal function family is f(x,α)=(α+1)x / (x+α), where the parameter range is α∈(-∞,-1)∪(0,+∞); the expression of the power function family is f(x,α)=AT(x α ,x1,x2), where the parameter range is 0 <x1<x2,α> 0; the expression of the inverse tangent function family is f(x,α)=AT(arctan(x),x1,x2), where the parameter range is x1 <x2; The parameter estimator includes a global average pooling layer and a rectified linear unit; wherein, the global average pooling layer calculates the curve family parameter value α according to the features extracted by the feature extractor.

2. The method according to claim 1, characterized in that The parameter estimator includes a global average pooling layer and a linear rectification function; wherein the global average pooling layer calculates the curve family parameter value α according to the features extracted by the feature extractor, and the linear rectification function is used to ensure that the curve family parameter value α is non-negative; the operation formula of the global average pooling layer is α global is the parameter estimated by the global average pooling layer operation, S is the feature matrix output by the feature extractor, C is the total number of channels of the feature matrix, H is the total length of the feature matrix, W is the total width of the feature matrix, S i,j,k is the value of the position of the ith channel (j, k) of the feature matrix; the linear rectification function operation formula is α=max(0,α global ).

3. The method according to claim 1, characterized in that The total loss function used to train the low-light enhancement network model is L = λ exp L exp +λ spa L spa +λ color L color ; Among them, λ exp ,λ spa ,λ color is the weight term, the enhanced degree loss function term L exp =∑ x∈I |E(x)-e|, E(x) is the pixel value of the low-light enhancement network model after enhancing the pixel x in the low-light image I, and e is the manually set hyperparameter value; the spatial distribution consistency loss function term L spa =∑ j∈Ω (|E(I)-E(I) j |-|II j |) 2 , Ω is the offset set of four pixels adjacent to a pixel, I j The low-light image I is shifted by one pixel in one direction, E(I) j The output image E(I) is shifted by one pixel in one direction; the color stability loss function term L color =(E(I) R -E(I) G ) 2 +(E(I) G -E(I) B ) 2 +(E(I) B -E(I) R ) 2 , where E(I) R 、E(I) G 、E(I) B They are the components of the output image E(I) on the three RGB channels.

4. The method according to claim 1, wherein The feature extractor includes a downsampling layer and n consecutive convolutional layers, and each convolutional layer is followed by a rectified linear unit; wherein, the i-th convolutional layer extracts features from the input image and then passes through the rectified linear unit to obtain the output result of the i-th layer, the input of the i-th convolutional layer is the output of the (i-1)-th convolutional layer, 2≤i<n; the input of the n-th convolutional layer is the sum of the output results of the previous n-1 convolutional layers.

5. A low-light enhancement system based on a curve family function applicable to the method of claim 1, characterized in that: Includes a feature extractor, a parameter estimator, and a curve family function; wherein, the feature extractor is used to downsample the input image and then extract features from the obtained downsampled image; the parameter estimator is used to calculate the curve family parameter values of the curve family function corresponding to the input image according to the features extracted by the feature extractor and assign them to the curve family function; The curve family function is used to map and enhance each pixel of the input image to obtain a light-enhanced image.

6. A server, characterized in that: Includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of any one of claims 1 to 4 are implemented.

Images