Lightweight low-light image enhancement device and method based on zero-shot learning
By using a lightweight low-light image enhancement device based on zero-order learning, combined with lightweight curve parameter estimation, Fourier frequency domain transform and fitness function modules, the problems of lack of dataset, noise processing and uneven illumination in low-light image enhancement are solved, and efficient real-time edge enhancement effect is achieved.
Patent Information
- Application Number
- CN202411260396.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-10
AI Technical Summary
Existing low-light image enhancement methods suffer from problems such as lack of datasets, poor noise handling, insufficient handling of uneven lighting, and excessive model computational resources, making real-time deployment on the edge difficult.
A lightweight low-light image enhancement device based on zero-learning is adopted, including a lightweight curve parameter estimation module, a Fourier frequency domain transform module, and a fitness function module. Through iterative enhancement and region block adjustment, pixel-by-pixel enhancement effect is achieved.
It reduces reliance on datasets, improves model generalization performance and robustness, reduces computational resource requirements, is suitable for real-time deployment on the board, and effectively alleviates uneven lighting problems.
Smart Images

Figure CN118781034B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a lightweight low-light image enhancement device and method based on zero-shot learning. Background Art
[0002] In real life, due to specific lighting conditions such as at night, or limitations of photography technology (such as exposure time), the quality of the images (including pictures or videos) taken is often poor, dark, and unable to see the real scene clearly. This not only seriously affects the understanding and perception of the real world, but also seriously affects the analysis of more high-level algorithm tasks on the pictures, such as face detection and recognition, object detection and recognition, etc. Therefore, low-light (dark light) image enhancement technology has been widely studied, which is very important for scenes with strict requirements for video surveillance, autonomous driving night scenes, and scenes that need to run related algorithms in dark light.
[0003] Currently popular low-light image enhancement methods mainly include traditional histogram equalization-based methods, Retinex (retina + cortex) model-based methods, and deep learning-based methods. Deep learning-based methods have been more developed and successful due to their better accuracy and robustness. However, current low-light image enhancement methods still face many problems and challenges:
[0004] (1) Datasets are relatively scarce and difficult to construct. For supervised methods based on deep learning, a large amount of paired labeled data is required, including a set of paired images with poor low-light imaging quality and images with good normal light imaging quality. Such real-scene data is very difficult to obtain; although many scholars have proposed some synthetic data sets, because they are synthetic data, there will be differences from real data, resulting in poor generalization ability in real scenes.
[0005] (2) Noise problem. Existing methods cannot handle the noise problem well when processing and enhancing low-light images, and may even amplify the original noise. Although some methods will try to add some Gaussian white noise during training, this artificially added noise is often quite different from the real noise.
[0006] (3) Uneven lighting problem. In real-life scenes, in low-light conditions, there will be many dark and unclear areas, but there will also be some small areas that are overexposed or have strong light. This causes the lighting of the entire image to be uneven. Existing methods tend to treat all areas equally and enhance all areas. The best approach is to enhance the dark areas while suppressing the overexposed areas.
[0007] (4) The existing deep learning model architecture and methods, including the latest transformer (Retinexformer) and Mamba (RetinexMamba) proposed in 2024, generally have large networks and require more computing resources and computing power to run. They are difficult to deploy in real time on the terminal side, so lightweight models are urgently needed.
[0008] In view of this, the present invention is conceived in depth to address the above-mentioned problems existing in the existing low-light image enhancement methods, and actively studies and improves the methods to develop the present invention. Summary of the invention
[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a lightweight low-light image enhancement device and method based on zero-shot learning that can enhance the low-light image effect, thereby enhancing the generalization and robustness of the model.
[0010] In order to achieve the above object, the solution of the present invention is:
[0011] A lightweight low-light image enhancement device based on zero-shot learning, using a deep learning model, including a lightweight curve parameter estimation module based on a zero-shot learning method, a Fourier frequency domain transformation module, a fitness function module and a merging module;
[0012] The lightweight curve parameter estimation module based on the zero-shot learning method includes a lightweight CNN to predict the brightness enhancement curve function of the input image. For each pixel in the image, iterative enhancement is performed according to the curve function.
[0013] The Fourier frequency domain transform module gives the Fourier transform spectrum of the input original image, and calculates the amplitude of each pixel based on the spectrum. The larger the amplitude, the stronger the brightness.
[0014] The fitness function module calculates the return value of the fitness function for each area according to the distribution of pixel values in the image. Different areas have different brightness values and are divided into dark areas, bright areas, edge areas and central areas. The return value of the fitness function for different areas is different, and the enhancement method is dynamically adjusted by area.
[0015] The merging module combines the results of the lightweight curve parameter estimation module, the Fourier frequency domain transform module and the fitness function module based on the zero-shot learning method to calculate each pixel, and uses the final merged result as the final image enhancement adjustment result.
[0016] Furthermore, the lightweight curve parameter estimation module based on the zero-shot learning method includes three parts: the definition of the brightness enhancement curve function, the curve parameter estimation network, and the no-reference loss function;
[0017] Definition of the brightness enhancement curve function: a curve that automatically maps low-light images to normal lighting. The parameters of this curve depend entirely on the input image. The formula for the curve parameters is as follows:
[0018] ;
[0019] It represents the enhancement result of the given input pixel x at the nth iteration. Its value is based on the n-1th enhancement result, superimposed on the input pixel x on the curve parameter weight The adjusted value, Represents the curve parameter function at each pixel position, ranging from [-1, 1];
[0020] The curve parameter estimation network is a convolutional neural network, which stacks Conv+BN+Relu modules. The final output value range is [-1,1], and the final output function is the Tanh function.
[0021] The no-reference loss function divides an image into K regions and designs several dimensional losses for each region, including spatial consistency loss, exposure control loss, color consistency loss, and lighting smoothness loss.
[0022] Furthermore, in the definition of the brightness enhancement curve function, the curve needs to meet three conditions:
[0023] First, the range of each pixel value of the enhanced image is normalized to [0,1];
[0024] Second, the curve should be monotonic to ensure the difference between adjacent pixels;
[0025] Third, since the parameter estimation of the neural network learning curve is required, the design of the curve should be as simple and differentiable as possible.
[0026] Furthermore, the specific method of the Fourier frequency domain transform module is: the input image is transformed into a Fourier spectrum through the Fourier transform function. The spectrum contains the amplitude value of each pixel. The amplitude value is normalized for the entire image, and the range is [0,1]. Therefore, the output of the Fourier frequency domain transform module is the function value after amplitude normalization. , for each pixel value of the image, its value range is [0,1], the specific formula is:
[0027] ;
[0028] Input represents the input image, dft represents Fourier transform, norm represents the normalization operation based on amplitude of the Fourier transformed image, and the final output is the normalized value of each pixel in the amplitude spectrum diagram after Fourier transform of the original image, which represents the brightness of each pixel.
[0029] Furthermore, the specific method of the fitness function module is: the input low-light image has been divided into K regions. Considering the different imaging quality of each region of the image under different environments, the image is divided into dark area, bright area, edge area and center area. The entropy information, edge content information and standard deviation of gray value in each region are obtained. Based on this information, the local brightness of a certain area is obtained. , the specific formula is as follows:
[0030] ;
[0031] Represents the entropy of region K. The larger the value, the more information it contains. Represents the edge content of region K. The larger the value, the more edge information there is, and the better the contrast of this area. Represents the standard deviation of the grayscale value of area K. The larger the value, the greater the average pixel change in this area, and the better the contrast of the area. There are three weight parameters in front of the three pieces of information. , , , three weight parameters , , According to the entropy of the input image , Edge Content And the standard deviation of the gray value The value distribution ranges of the three information are dynamically adjusted to ensure that the final values of the three information are summed in the same dimension.
[0032] Furthermore, the specific method of merging the modules is to take the curve prediction value of the lightweight curve parameter estimation module based on the zero-time learning method as the main factor, and on this basis consider the values of the Fourier frequency domain transform module and the fitness function module, and make fine adjustments to the corresponding proportions, and finally output Better low-light image enhancement effect, the specific formula is as follows:
[0033] /
[0034] in, The final output for better low-light image enhancement effect, Represents the enhancement result of the given input pixel x at the nth iteration, is the function value after amplitude normalization, It represents the local brightness in the K area, and x is the value of each pixel in the image to be enhanced.
[0035] A lightweight low-light image enhancement method based on zero-shot learning comprises the following steps:
[0036] Step S1, input the low-light image into a lightweight curve parameter estimation module based on a zero-shot learning method, predict the brightness enhancement curve function of the input image through CNN, and for each pixel in the image, perform iterative enhancement according to the curve function, and output the result to step S4;
[0037] Step S2, input the low-light image into the Fourier frequency domain transform module, obtain the Fourier transform spectrum of the low-light image, calculate the amplitude of each pixel according to the spectrum, and output the result to step S4;
[0038] Step S3, input the low-light image into the fitness function module, and divide the image into dark area, bright area, edge area and center area according to the distribution of pixel values of the image. The fitness function return value of each area is calculated. The fitness function return value of different areas is different, and the enhancement mode is dynamically adjusted by area, and the result is output to step S4;
[0039] Step S4: perform merging calculation on each pixel through a merging module according to the output results of step S1, step S2, and step S3, and use the final merging result as the final image enhancement adjustment result.
[0040] Further, in step S1, the lightweight curve parameter estimation module based on the zero-shot learning method includes three parts: the definition of the brightness enhancement curve function, the curve parameter estimation network, and the no-reference loss function;
[0041] Definition of the brightness enhancement curve function: a curve that automatically maps low-light images to normal lighting. The parameters of this curve depend entirely on the input image. The formula for the curve parameters is as follows:
[0042] ;
[0043] It represents the enhancement result of the given input pixel x at the nth iteration. Its value is based on the n-1th enhancement result, superimposed on the input pixel x on the curve parameter weight The adjusted value, Represents the curve parameter function at each pixel position, ranging from [-1, 1];
[0044] The curve parameter estimation network is a convolutional neural network, which stacks Conv+BN+Relu modules. The final output value range is [-1,1], and the final output function is the Tanh function.
[0045] The no-reference loss function divides an image into K regions and designs several dimensional losses for each region, including spatial consistency loss, exposure control loss, color consistency loss, and lighting smoothness loss.
[0046] Furthermore, the specific method of the Fourier frequency domain transform module is: the input image is transformed into a Fourier spectrum through the Fourier transform function. The spectrum contains the amplitude value of each pixel. The amplitude value is normalized for the entire image, and the range is [0,1]. Therefore, the output of the Fourier frequency domain transform module is the function value after amplitude normalization. , for each pixel value of the image, its value range is [0,1], the specific formula is:
[0047] ;
[0048] Input represents the input image, dft represents Fourier transform, norm represents the normalization operation based on amplitude of the Fourier transformed image, and the final output is the normalized value of each pixel in the amplitude spectrum diagram after Fourier transform of the original image, which represents the brightness of each pixel.
[0049] Furthermore, the specific method of the fitness function module is: the input low-light image has been divided into K regions. Considering the different imaging quality of each region of the image under different environments, the image is divided into dark area, bright area, edge area and center area. The entropy information, edge content information and standard deviation of gray value in each region are obtained. Based on this information, the local brightness of a certain area is obtained. , the specific formula is as follows:
[0050] ;
[0051] Represents the entropy of region K. The larger the value, the more information it contains. Represents the edge content of region K. The larger the value, the more edge information there is, and the better the contrast of this area. Represents the standard deviation of the grayscale value of area K. The larger the value, the greater the average pixel change in this area, and the better the contrast of the area. There are three weight parameters in front of the three pieces of information. , , , three weight parameters , , According to the entropy of the input image , Edge Content And the standard deviation of the gray value The value distribution ranges of the three information are dynamically adjusted to ensure that the final values of the three information are summed in the same dimension.
[0052] Furthermore, the specific method of merging the modules is to take the curve prediction value of the lightweight curve parameter estimation module based on the zero-time learning method as the main factor, and on this basis consider the values of the Fourier frequency domain transform module and the fitness function module, and make fine adjustments to the corresponding proportions, and finally output Better low-light image enhancement effect, the specific formula is as follows:
[0053] /
[0054] in, The final output for better low-light image enhancement effect, Represents the enhancement result of the given input pixel x at the nth iteration, is the function value after amplitude normalization, It represents the local brightness in the K area. x is the value of each pixel in the image to be enhanced. x must belong to a certain area K. A dark-light image is a pixel value x of a dark-light image. It is predicted by the three modules respectively and then combined to get the final enhancement amplitude.
[0055] After adopting the above scheme, the present invention proposes a lightweight parameter estimation network and module based on zero-time learning, combined with the method of Fourier transform and regional block fitness function, to complete the final pixel-by-pixel enhancement adjustment. Based on the method of zero-time learning, there is no need to rely on paired supervised data images during training, which greatly reduces the dependence on the data set and improves the generalization performance of the model. The lightweight network requires better computing resources and is more suitable for real-time deployment on the board. Combining Fourier transform and fitness function can effectively alleviate the problem of uneven lighting. Each pixel has an adjustment coefficient suitable for itself, avoiding the defects of previous methods.
[0056] Compared with the prior art, the lightweight low-light image enhancement device and method based on zero-shot learning of the present invention have the following advantages:
[0057] The lightweight curve parameter estimation module based on the zero-shot learning method of the present invention adopts the zero-shot learning method, does not require paired data like supervised learning, solves the problem of lack of data, and at the same time, because the training data will not be fitted, the robustness of the model will be better.
[0058] The lightweight curve parameter estimation module based on the zero-shot learning method of the present invention adopts the task of deep curve estimation based on the zero-shot learning method, which only requires a conventional lightweight network, so the requirement for computing resources is very low and can be run and deployed in real time on the board-side device.
[0059] The Fourier frequency domain transform module of the present invention introduces the transform function of the amplitude spectrum of the Fourier frequency domain transform, and gives the amplitude of each pixel as a reference value of the brightness, so that dynamic enhancement adjustment with differences at the pixel level can be achieved according to the brightness reference value.
[0060] The merging module of the present invention merges the depth curve estimation function, the fitness function and the amplitude change function in the Fourier frequency domain, and finally achieves a better low-light image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 This is a module diagram of the lightweight low-light image enhancement method based on zero-shot learning of the present invention. DETAILED DESCRIPTION
[0062] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.
[0063] like Figure 1 As shown, the present invention discloses a lightweight low-light image enhancement device based on zero-shot learning, which adopts a deep learning model. The model includes a lightweight curve parameter estimation module based on the zero-shot learning method, a Fourier frequency domain transform module, a fitness function module based on image content, grayscale value, etc., and a merging module of the three functions.
[0064] The lightweight curve parameter estimation module based on the zero-shot learning method includes an ordinary lightweight CNN (convolutional neural network), which is used to predict the brightness enhancement curve function of the input image. For each pixel in the image, it is continuously iterated and enhanced according to the curve function; the Fourier frequency domain transform module gives the Fourier transform spectrum of the input original image. According to the spectrum, the amplitude of each pixel can be calculated, that is, the amplitude function. Generally speaking, the larger the amplitude, the stronger the brightness; the fitness function module can calculate the fitness values of some areas according to the pixel value distribution of the image. Different areas have different brightness values and can be divided into dark areas, bright areas, edge areas and center areas. The fitness function return values of different areas are different, so the enhancement method can be dynamically adjusted by area. The merging module of the three functions will merge and calculate each pixel based on the results of the three functions, and use the final merged result as the final enhancement adjustment result.
[0065] The lightweight curve parameter estimation module based on the zero-shot learning method introduces the existing method and research result Zero-DCE (see reference 1). In the existing methods of low-light image enhancement, in order to solve the problem of limited generalization ability of existing supervised learning-related methods and the problem of unstable training of reinforcement learning or unsupervised learning, some scholars have proposed a zero-shot learning method. This method only learns enhancement from the test image and does not need to collect paired supervised data (i.e., a dark light image and a normal light image). The currently popular zero-shot learning method generally transforms the low-light enhancement task into a deep curve estimation task, inputs a low-light image, generates a high-order curve as output, and adjusts the dynamic range of the input at the pixel level according to the curve to finally obtain the enhanced image. Unlike the image reconstruction-based method that requires massive computing resources, the zero-shot learning method learns the mapping from image to curve, which only requires a lightweight network and can be deployed on the end-side device in a friendly manner.
[0066] The present invention proposes a new low-light image enhancement method. Instead of image reconstruction, the image enhancement task is defined as a specific curve estimation problem of the image. The original low-light image is iteratively enhanced and adjusted through the estimated curve function. No paired supervised images are required. Due to the transformation of the task problem, only a lightweight network is needed to learn the curve function, so a very fast reasoning speed can be achieved. The lightweight curve parameter estimation module based on the zero-shot learning method includes three parts: the definition of the brightness enhancement curve function, the curve parameter estimation network, and the reference-free loss function.
[0067] (1) Design and definition of brightness enhancement curve function: The design goal is to automatically map low-light images to normal-light curves, and the parameters of this curve are completely dependent on the input image. To design this curve, three conditions must be met: First, the range of each pixel value in the enhanced image is normalized to [0,1], so as to avoid information loss caused by overflow truncation and other problems; second, the curve should be monotonic, so as to ensure the difference between adjacent pixels; third, since the neural network learning curve parameter estimation is required, the curve design should be as simple and differentiable as possible. Based on this, the designed curve parameter formula is as follows:
[0068] ;
[0069] It represents the enhancement result of a given input pixel x at the nth iteration. Its value is based on the n-1th enhancement result, superimposed on the input pixel x in the curve parameter weight (i.e. ) value after adjustment, that is Represents the curve parameter (weight) function at each pixel position, ranging from [-1, 1]. For example, when When the value is -1, ;when When the value is 0, ;when When the value is 1, , -1, 0, 1 are Three special values.
[0070] (2) Lightweight curve parameter estimation network. This network is an ordinary convolutional neural network, stacking Conv+BN+Relu modules. Conv+BN+ReLU is a commonly used convolutional neural network structure, where Conv is the abbreviation of the convolution module, and the full name is Convolution; BN is the abbreviation of Batch Normalization, which is used to normalize the results of the convolution layer; Relu is a common activation function. Conv+BN+Relu is a basic module of convolutional neural network in the field of deep learning, which includes convolution operation, normalization and activation function. Considering that the final output result range is [-1,1], the final output function is designed to be the Tanh function.
[0071] (3) No-reference loss function: Assume that an image is divided into K regions. Several dimensional losses can be designed for each region, including spatial consistency loss, exposure control loss, color consistency loss, and illumination smoothness loss. Since this is an existing design method, please refer to reference 1 for details.
[0072] The reason why the present invention introduces the Fourier frequency domain transform module is that the frequency domain information of Fourier has been proved to have great potential in the field of image enhancement. According to the properties of Fourier transform, the phase information in the Fourier spectrum represents the content structure of the image, and the amplitude information represents the brightness information of the image. The amplitude value is proportional to the brightness. Therefore, the present invention introduces the Fourier transform, and transforms the input image into a Fourier spectrum diagram through the Fourier transform function. The spectrum diagram contains the amplitude value of each pixel. The amplitude value is normalized for the entire image, and the range is [0,1]. Therefore, the output of the Fourier frequency domain transform module is the function value after amplitude normalization. For each pixel value of the image, its value range is [0,1]. The specific formula is:
[0073] ;
[0074] Input represents the input image, dft represents Fourier transform, norm represents the normalization operation based on amplitude of the Fourier transformed image, and the final output is the normalized value of each pixel in the amplitude spectrum diagram after Fourier transform of the original image, which represents the brightness of each pixel.
[0075] The present invention adopts the fitness function module to take into account the different imaging quality of each area of the image under different environments. There are often dark areas, bright areas, edge areas, central areas, etc. When doing image enhancement, a theoretically good method should achieve different adjustment amplitudes for different areas. How can we know which area each point in the image belongs to? The present invention designs a fitness function. The fitness function returns the fitness function value of each area. An image is divided into K areas on average. Each area can obtain the fitness function return value according to the pixel value. The size of the fitness function return value reflects the situation of this area, such as whether it is a bright area or a dark area, at the edge, or in the center, etc. The fitness function return value is different, and the final data enhancement result is also different. If the calculated fitness function value is relatively large, it may be in a bright area, so the enhancement amplitude can be smaller, otherwise it may be in a dark area, and the adjustment amplitude can be larger. The present invention adopts the fitness function module to consider the pixel value situation in each area, and can obtain the entropy information, edge content information and related information of the standard deviation of the gray value in each area. Based on this information, the local brightness situation in a certain area can be known. , the specific formula is as follows:
[0076] ;
[0077] Represents the entropy of region K. The larger the value, the more information it contains. Represents the edge content of region K. The larger the value, the more edge information there is, and the better the contrast of this area. Indicates the standard deviation of the grayscale value of area K. The larger the value, the greater the average pixel change in this area, and the better the contrast of the area. There are three weight parameters in front of the three pieces of information. , , , three weight parameters , , According to the entropy of the input image , Edge Content And the standard deviation of the gray value The value distribution ranges of the three information are dynamically adjusted to ensure that the final values of the three information are summed in the same dimension.
[0078] The three weight parameters are dynamically adjusted. One possible implementation method is that for K regions, assuming that the input low-light image has three columns, K×3 values can be calculated. The first column is That is, the value of information entropy. Normalize this column, such as the maximum value normalization, then According to all The reciprocal of the maximum value inside is 1 ; For different pictures, the calculated values of the three basic information are different, and dynamic normalization is performed based on the unique values of each picture.
[0079] The fitness function module itself does not actively indicate which area is a dark area, a bright area, etc. Instead, it will count some indicators based on the pixel values of each area, including entropy information (the specific method is to count the histogram and probability distribution of the frequency of occurrence of each pixel in this area, and calculate the entropy information based on the probability distribution), edge information (using common image edge detection operators such as the Sobel operator to calculate the edge information of the image), and the standard deviation of the grayscale value (reflecting the contrast of this area). For each area, a specific fitness value can be calculated through the formula. This value reflects some characteristics of this image, such as brightness, edge information, contrast, etc. Therefore, for areas with different fitness values, different enhancement methods should be dynamically adopted.
[0080] The merging module fuses the results of the three functions to obtain the final enhanced value. The fusion method is based on the curve prediction value of the lightweight curve parameter estimation module based on the zero-time learning method. On this basis, the values of the Fourier frequency domain transform module and the fitness function module are considered, and the corresponding proportions are fine-tuned to finally output Better low-light image enhancement effect, the specific formula is as follows:
[0081] / ;
[0082] For the fitness function, a value can be calculated for each area, and for the entire image, it can be divided into K areas. When the entire image is finally enhanced, pixel enhancement must be performed for each point X. If the pixel point X is in area K, that is, , then the value of the fitness function is The dimension of fitness function calculation is area. For pixels in the same area, the returned fitness function value is the same, while the fitness function values in different areas are different. In this way, the pixel values in different areas can be treated differently during the final image enhancement, rather than uniformly brightening or darkening. For example, the dark ones can be enhanced more to make them brighter, while the bright ones can be enhanced less to maintain the current state.
[0083] The present invention proposes a lightweight parameter estimation network and module based on zero-shot learning, combined with the method of Fourier transform and regional block fitness function, to complete the final pixel-by-pixel enhancement adjustment. Based on the method of zero-shot learning, there is no need to rely on paired supervised data images during training, which greatly reduces the dependence on the data set and improves the generalization performance of the model. The lightweight network requires better computing resources and is more suitable for real-time deployment on the board side. Combining Fourier transform and fitness function can effectively alleviate the problem of uneven lighting. Each pixel has an adjustment coefficient suitable for itself, avoiding the defects of previous methods.
[0084] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.
[0085] Reference 1: “Zero-reference deep curve estimation for low-light image enhancement” by C. Guo, C. Li, J. Guo, CC Loy, J. Hou, S. Kwong, and R. Cong, 2020 CVPR (IEEE / CVF Conference on Computer Vision and Pattern Recognition), pp. 1780–1789.
Claims
1. A lightweight low-light image enhancement device based on zero-shot learning, characterized in that: A deep learning model is used, including a lightweight curve parameter estimation module based on a zero-shot learning method, a Fourier frequency domain transform module, a fitness function module, and a merging module; The lightweight curve parameter estimation module based on the zero-shot learning method includes a lightweight CNN to predict the brightness enhancement curve function of the input image. For each pixel in the image, iterative enhancement is performed according to the curve function. The Fourier frequency domain transform module gives the Fourier transform spectrum of the input original image, and calculates the amplitude of each pixel based on the spectrum. The larger the amplitude, the stronger the brightness. The fitness function module is divided into dark area, bright area, edge area and center area according to the pixel value distribution of the image. Different areas have different brightness values. The fitness function return value of each area is calculated. The fitness function return value of different areas is different, and the enhancement method is dynamically adjusted by area. The specific method of the fitness function module is: The input low-light image has been divided into K regions. Considering the different imaging quality of each region in different environments, the image is divided into dark area, bright area, edge area and center area. The entropy information, edge content information and standard deviation of gray value in each region are obtained. Based on this information, the local brightness of a certain area is obtained. , the specific formula is as follows: ; Indicates the local brightness situation; Represents the entropy of region K. The larger the value, the more information it contains. Represents the edge content of region K. The larger the value, the more edge information there is, and the better the contrast of this area. Represents the standard deviation of the grayscale value of area K. The larger the value, the greater the average pixel change in this area, and the better the contrast of the area. There are three weight parameters in front of the three pieces of information. , , , three weight parameters , , According to the entropy of the input image , Edge Content And the standard deviation of the gray value The value distribution range of the three information is dynamically adjusted to ensure that the final values of the three information are summed in the same dimension; The merging module is a lightweight curve parameter estimation module based on the zero-shot learning method, the Fourier frequency domain transform module and the fitness function module to merge and calculate each pixel, and use the final merged result as the final image enhancement adjustment result; The specific method of merging modules is to take the curve prediction value of the lightweight curve parameter estimation module based on the zero-time learning method as the main factor, and then consider the values of the Fourier frequency domain transform module and the fitness function module, and make fine adjustments to the corresponding proportions, and finally output Better low-light image enhancement effect, the specific formula is as follows: / in, The final output for better low-light image enhancement effect, Represents the enhancement result of the given input pixel x at the nth iteration, is the function value after amplitude normalization, It represents the local brightness in the K area, and x is the value of each pixel in the image to be enhanced.
2. The lightweight low-light image enhancement device based on zero-shot learning as claimed in claim 1, characterized in that: The lightweight curve parameter estimation module based on the zero-shot learning method includes three parts: the definition of the brightness enhancement curve function, the curve parameter estimation network and the no-reference loss function; Definition of the brightness enhancement curve function: a curve that automatically maps low-light images to normal lighting. The parameters of this curve depend entirely on the input image. The formula for the curve parameters is as follows: ; It represents the enhancement result of the given input pixel x at the nth iteration. Its value is based on the n-1th enhancement result, superimposed on the input pixel x on the curve parameter weight The adjusted value, Represents the curve parameter function at each pixel position, ranging from [-1, 1]; The curve parameter estimation network is a convolutional neural network, which stacks Conv+BN+Relu modules. Conv represents the convolution module, BN represents the normalization of the result of the convolution layer; Relu is the activation function; the final output result range is [-1,1], and the final output function is the Tanh function; The no-reference loss function divides an image into K regions and designs several dimensional losses for each region, including spatial consistency loss, exposure control loss, color consistency loss, and lighting smoothness loss.
3. The lightweight low-light image enhancement device based on zero-shot learning as claimed in claim 1, characterized in that: The specific method of the Fourier frequency domain transform module is: the input image is transformed into a Fourier spectrum through the Fourier transform function. The spectrum contains the amplitude value of each pixel. The amplitude value is normalized for the entire image, and the range is [0,1]. Therefore, the output of the Fourier frequency domain transform module is the function value after amplitude normalization. , for each pixel value of the image, its value range is [0,1], the specific formula is: ; Input represents the input image, dft represents Fourier transform, norm represents the normalization operation based on amplitude of the Fourier transformed image, and the final output is the normalized value of each pixel in the amplitude spectrum diagram after Fourier transform of the original image, which represents the brightness of each pixel.
4. A lightweight low-light image enhancement method based on zero-shot learning, characterized in that: The following steps are involved: Step S1, input the low-light image into a lightweight curve parameter estimation module based on a zero-shot learning method, predict the brightness enhancement curve function of the input image through CNN, and for each pixel in the image, perform iterative enhancement according to the curve function, and output the result to step S4; Step S2, input the low-light image into the Fourier frequency domain transform module, obtain the Fourier transform spectrum of the low-light image, calculate the amplitude of each pixel according to the spectrum, and output the result to step S4; Step S3, input the low-light image into the fitness function module, according to the pixel value distribution of the image, different areas have different brightness values, divided into dark area, bright area, edge area and central area, calculate the fitness function return value of each area, the fitness function return value of different areas is different, dynamically adjust the enhancement method by area, and output the result to step S4; the specific method of the fitness function module is: the input low-light image has been divided into K areas, considering the different imaging quality of each area of the image under different environments, the image is divided into dark area, bright area, edge area and central area, the entropy information, edge content information and gray value standard deviation related information in each area are calculated, and the local brightness situation in a certain area is obtained based on this information , the specific formula is as follows: ; Indicates the local brightness situation; Represents the entropy of region K. The larger the value, the more information it contains. Represents the edge content of region K. The larger the value, the more edge information there is, and the better the contrast of this area. Represents the standard deviation of the grayscale value of area K. The larger the value, the greater the average pixel change in this area, and the better the contrast of the area. There are three weight parameters in front of the three pieces of information. , , , three weight parameters , , According to the entropy of the input image , Edge Content And the standard deviation of the gray value The value distribution range of the three information is dynamically adjusted to ensure that the final values of the three information are summed in the same dimension; Step S4, performing merging calculation on each pixel through a merging module according to the output results of step S1, step S2, and step S3, and taking the final merging result as the final image enhancement adjustment result; The specific method of merging modules is to take the curve prediction value of the lightweight curve parameter estimation module based on the zero-time learning method as the main factor, and then consider the values of the Fourier frequency domain transform module and the fitness function module, and make fine adjustments to the corresponding proportions, and finally output Better low-light image enhancement effect, the specific formula is as follows: / in, The final output for better low-light image enhancement effect, Represents the enhancement result of the given input pixel x at the nth iteration, is the function value after amplitude normalization, It represents the local brightness in the K area, and x is the value of each pixel in the image to be enhanced.
5. The lightweight low-light image enhancement method based on zero-shot learning as claimed in claim 4, characterized in that: In step S1, the lightweight curve parameter estimation module based on the zero-shot learning method includes three parts: the definition of the brightness enhancement curve function, the curve parameter estimation network and the no-reference loss function; Definition of the brightness enhancement curve function: a curve that automatically maps low-light images to normal lighting. The parameters of this curve depend entirely on the input image. The formula for the curve parameters is as follows: ; It represents the enhancement result of the given input pixel x at the nth iteration. Its value is based on the n-1th enhancement result, superimposed on the input pixel x on the curve parameter weight The adjusted value, Represents the curve parameter function at each pixel position, ranging from [-1, 1]; The curve parameter estimation network is a convolutional neural network, which stacks Conv+BN+Relu modules. The final output value range is [-1,1], and the final output function is the Tanh function. The no-reference loss function divides an image into K regions and designs several dimensional losses for each region, including spatial consistency loss, exposure control loss, color consistency loss, and lighting smoothness loss.
6. The lightweight low-light image enhancement method based on zero-shot learning as claimed in claim 4, characterized in that: The specific method of the Fourier frequency domain transform module is: the input image is transformed into a Fourier spectrum through the Fourier transform function. The spectrum contains the amplitude value of each pixel. The amplitude value is normalized for the entire image, and the range is [0,1]. Therefore, the output of the Fourier frequency domain transform module is the function value after amplitude normalization. , for each pixel value of the image, its value range is [0,1], the specific formula is: ; Input represents the input image, dft represents Fourier transform, norm represents the normalization operation based on amplitude of the Fourier transformed image, and the final output is the normalized value of each pixel in the amplitude spectrum diagram after Fourier transform of the original image, which represents the brightness of each pixel.
Citation Information
Patent Citations
Low-light image enhancement method and system based on deep learning
CN116258653A
No-reference low-illumination image enhancement method based on high-order curve iteration
CN116912114A
Low-illumination image enhancement method based on semantic information guidance and illumination curve estimation
CN117474819A
Low-light image enhancement method capable of balancing details and brightness of different regions simultaneously
CN117611467A
Low-light image enhancement and night fatigue driving detection method based on deep learning zero reference depth curve estimation algorithm
CN117934349A
