Low-exposure image enhancement method and system based on embedding maclaurin and convolution

Abstract

This disclosure provides a method and system for enhancing low-exposure images based on embedded McLaurin and convolution, relating to the field of image processing technology. The method involves acquiring a low-exposure image to be enhanced, processing and decomposing the low-exposure image, inputting the low-exposure image into a backbone network to extract multi-scale features, introducing a dynamic adjustment factor into the backbone network, using the dynamic adjustment factor to learn a convolution weight calibration map, calibrating the convolution weights of the backbone network with the learned weight calibration map, performing convolution operations on the features using the calibrated convolution kernels to obtain the luminance component of the low-exposure image, and outputting a long-exposure enhanced image that varies with the dynamic adjustment factor using the luminance component.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and specifically to a method and system for enhancing low-exposure images based on embedded McLaughlin and convolution. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Currently, the core idea of ​​most convolutional neural networks for low-exposure image enhancement is to learn the luminance component of the image from the low-exposure image and then remove the luminance component to achieve low-exposure image enhancement. However, this enhancement method has two problems:

[0004] (1) The mathematical relationship between low-exposure images and their corresponding luminance components is extremely complex, requiring a deep network to fit the mathematical relationship between low-exposure images and luminance components, resulting in a slow forward derivation speed.

[0005] (2) After the network training is completed, the network weights are static values, resulting in a fixed brightness level of the output image, which cannot meet the needs of consumer electronics such as smartphones and digital cameras for dynamically adjustable brightness levels. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a low-exposure image enhancement method and system based on embedding the McLaurin expansion formula and convolution weight calibration. By embedding the McLaurin expansion formula, the fitting performance of the network is enhanced, thereby achieving network lightweighting. Furthermore, a weight calibration method is designed to enable dynamic adjustment of the brightness level of the output image.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] Low-exposure image enhancement methods based on embedded McLaurin and convolution include:

[0009] Obtain the low-exposure image to be enhanced, and process and decompose the low-exposure image;

[0010] The low-exposure image is input into the backbone network to extract multi-scale features. A dynamic adjustment factor is introduced into the backbone network. The dynamic adjustment factor is used to learn a convolution weight calibration map. The learned weight calibration map is used to calibrate the convolution weights of the backbone network. The calibrated convolution kernels are used to perform convolution operations on the features to obtain the luminance component of the low-exposure image. The luminance component is used to output a long-exposure enhanced image that can change with the dynamic adjustment factor.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions:

[0012] The low-exposure image enhancement system based on embedded Maclaurin and convolution comprises:

[0013] A preprocessing module is configured to acquire a low-exposure image to be enhanced and perform processing decomposition on the low-exposure image.

[0014] An image enhancement module is configured to input the low-exposure image into a backbone network to extract multi-scale features, introduce a dynamic adjustment factor in the backbone network, learn a convolution weight calibration map by using the dynamic adjustment factor, calibrate the convolution weight of the backbone network by using the learned weight calibration map, perform convolution operation on the features by using the calibrated convolution kernel, obtain a brightness component of the low-exposure image, and output a long-exposure enhanced image that can change with the dynamic adjustment factor by using the brightness component.

[0015] According to some embodiments, the present disclosure adopts the following technical solutions:

[0016] A computer-readable storage medium stores a plurality of instructions, which are adapted to be loaded and executed by a processor of a terminal device to implement the low-exposure image enhancement method based on embedded Maclaurin and convolution.

[0017] According to some embodiments, the present disclosure adopts the following technical solutions:

[0018] A terminal device comprises a processor and a computer-readable storage medium, the processor is configured to implement instructions, and the computer-readable storage medium is configured to store a plurality of instructions, which are adapted to be loaded and executed by the processor to implement the low-exposure image enhancement method based on embedded Maclaurin and convolution.

[0019] Compared with the prior art, the present disclosure has the following beneficial effects:

[0020] The present disclosure proposes a low-exposure image enhancement network embedded with Maclaurin expansion formula and convolution weight calibration, which strengthens the fitting performance of the network by embedding the Maclaurin expansion formula, realizes network lightweight, and designs a weight calibration method to realize dynamic adjustment of the brightness level of the output image.

[0021] The network of the present disclosure can effectively improve the visual quality of the low-exposure image. When different dynamic adjustment factors are input, the brightness level of the output image can be changed, indicating the effectiveness of the dynamic calibration method. Since the Maclaurin expansion formula is embedded in the network, a complex nonlinear transformation function can be fitted with fewer network layers. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of the present disclosure, are used to provide a further understanding of the present disclosure, and the schematic embodiments of the present disclosure and the description thereof are used to explain the present disclosure, and do not constitute improper limitations on the present disclosure.

[0023] Figure 1 A method flowchart of an embodiment of the present disclosure;

[0024] Figure 2 A backbone network structure schematic diagram of an embodiment of the present disclosure;

[0025] Figure 3 A convolution weight calibration network structure schematic diagram of an embodiment of the present disclosure;

[0026] Figure 4 A low-exposure image enhancement experimental effect schematic diagram of an embodiment of the present disclosure. DETAILED DESCRIPTION:

[0027] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0029] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0030] Embodiment 1

[0031] In an embodiment of the present disclosure, a low-exposure image enhancement method based on embedded Maclaurin and convolution is provided, comprising:

[0032] Obtaining a low-exposure image to be enhanced, processing and decomposing the low-exposure image;

[0033] Inputting the low-exposure image into a backbone network to extract multi-scale features, introducing a dynamic adjustment factor in the backbone network, learning a convolution weight calibration map using the dynamic adjustment factor, calibrating the convolution weight of the backbone network using the learned weight calibration map, performing convolution operation on the features using the calibrated convolution kernel, obtaining the brightness component of the low-exposure image, and outputting a long-exposure enhanced image that can change with the dynamic adjustment factor using the brightness component.

[0034] As an embodiment, the way of obtaining a low-exposure image to be enhanced and processing and decomposing the low-exposure image comprises:

[0035] The low-exposure image is decomposed into a luminance component and a product of the long-exposure image, and a residual term is added during the decomposition to obtain the luminance component.

[0036] Specifically, the low-exposure image is decomposed into a luminance component and a product of the long-exposure image, as shown in formula (1):

[0037] D c (x) = E c (x) · H c (x) (1)

[0038] where x represents a pixel coordinate, c represents three channels of red, green and blue of the image, H c (x) represents an enhanced long-exposure image, E c (x) represents an illumination component, D c (x) is a low-exposure image, and ‘·’ represents pixel point multiplication.

[0039] The enhanced long-exposure image is represented as:

[0040]

[0041] In order to accelerate the network convergence, a residual term is added in formula (2), and the following formula (3) can be obtained:

[0042]

[0043] In formula (3), Er c (x) is the luminance component after adding the residual term, which is simply referred to as the luminance component, and is represented as:

[0044]

[0045] According to the multi-scale Retinex theory, Er c (x) is a linear combination of multi-scale features, but a simple linear combination method leads to an enhanced image that is too dark or too bright. Therefore, Er c (x) is represented as a nonlinear combination of multi-scale features, and is represented as:

[0046] Er c (x) = g {T [D c (x)]} (5)

[0047] In formula (5), T[.] is a multi-scale feature extraction module, and g{.} represents a nonlinear function.

[0048] Since g{.} is relatively complex, if the function is directly learned from the low-exposure image, hundreds of convolution layers are required to accurately fit Er c (x) and D cThe functional relationship between (x) and the model is described. To reduce model complexity, this disclosure innovatively embeds the McLaurin expansion formula, which transforms the functional relationship between the luminance component and the low-exposure image into a combination of multi-scale features and polynomial coefficients, as shown in formula (6).

[0049]

[0050] Let g(0) = a0, g'(0) = a1, g"(0) / 2! = a2, ..., g n (0) / n! = a n Formula (6) is transformed into:

[0051]

[0052] In the formula, a0, a1, a2, ..., a n Let a0, a1, a2, ..., a be the coefficients of the Maclaurin expansion formula. Calculate a0, a1, a2, ..., a n and multi-scale features T[D c After (x)], the luminance component Er can be calculated according to formula (7). c (x), and then the enhanced long exposure image can be derived according to formula (3), as shown in formula (8).

[0053] H c (x)=

[0054]

[0055] To further reduce the complexity of the model, n is set to 5 in this patent, and the expansion remainder R with eigenvalues ​​approaching zero is omitted. n+1 [D c From (x)], we can obtain:

[0056]

[0057] Furthermore, according to formula (9), a backbone network is constructed to enhance the low-exposure image. The structure of the backbone network is as follows: Figure 2 As shown, the backbone network consists of 11 convolutional layers. The first 7 convolutional layers, 2 downsampling layers, and 2 upsampling layers constitute a multi-scale convolutional module used to generate multi-scale features.

[0058] In order to compress the multi-scale features to 3 channels, 2 convolutional layers are arranged after the multi-scale convolutional module to gradually compress the multi-scale features to 3 channels. In order to accelerate the convergence of the model, the features are weighted and calibrated by using the jump connection. The above 9 convolutional layers constitute the multi-scale feature generation module T[.] in formula (5). In order to obtain the coefficients {a0, a1, a2, a3, a4, a5} of the Maclaurin expansion formula, another branch is introduced on the multi-scale convolutional module, which includes 2 convolutional layers and 1 global average pooling layer. The global average pooling layer is used to calibrate the image features in the channel dimension, which is used to improve the learning ability of the network. The branch outputs a weight map of 18 channels, and the weight map of 18 channels is decomposed to obtain 6 3-channel coefficients {a0, a1, a2, a3, a4, a5}.

[0059] The function of the backbone network is to learn the low exposure image D c (x) to the luminance component Er c (x) by using the transform function, and substituting the Er c (x) output by the backbone network into formula (3), the enhanced long exposure image H c (x) can be obtained.

[0060] When the backbone network is trained, all weights are fixed values, resulting in the luminance of the output long exposure image H c (x) being a fixed value, which cannot achieve the purpose of dynamic adjustment. In order to realize the adjustable luminance level of the long exposure image H c (x), the disclosure innovatively learns the convolution weight calibration map from a dynamic adjustment factor, and calibrates the convolution weight of the backbone network with the learned weight calibration map, so as to realize the dynamic adjustment of the weight of the backbone network. The convolution weight calibration network structure is as shown in Figure 3 .

[0061] The convolution weight calibration network is composed of 11 one-dimensional convolutional layers (the number of one-dimensional convolutional layers is consistent with the number of convolutional layers of the backbone network), and the input of the convolution weight calibration network is the dynamic adjustment factor φ. The 11 one-dimensional convolutional layers learn 11 groups of weight calibration maps from the dynamic adjustment factor φ, and the size of the weight calibration map is the same as the size of the convolution kernel of the backbone network. The learned 11 groups of weight calibration maps are used to calibrate the convolution kernel of the backbone network, so as to obtain the calibrated convolution kernel, and then the convolution operation is performed on the feature by using the calibrated convolution kernel, so as to obtain the luminance component. Since the dynamic adjustment factor φ is an adjustable parameter, the weight calibration map will change accordingly, and the convolution kernel of the backbone network will also change dynamically, so that the dynamic adjustment of the luminance component can be realized. The forward derivation process of the network is as shown in formula (10) to formula (14).

[0062]

[0063] In the formula, Net T {Q T1 Q T2 ,……,Q T11 ; φ} represents the convolutional weight calibration network, where φ is a dynamic adjustment factor and is the input of the convolutional weight calibration network, Q T1 Q T2 ,……,Q T11 These are the weights of the 11 one-dimensional convolutional layers in the convolutional weight calibration network. {B1,B2,……,B 11} represents the weight calibration map output by the convolutional weight calibration network, used to calibrate the weights of the backbone network.

[0064] Φ={Q S1 Q S2 ,......,Q S11} (11)

[0065] In the formula, Φ represents the set of weights from the 11 convolutional layers in the backbone network. The weight calibration map {B1, B2, ..., B...} output by the convolutional weight calibration network is then used to calibrate the weights. 11 The weights Φ for the backbone network are Φ = {Q} S1 Q S2 ,……,Q S11 Dynamic calibration yields the following:

[0066] Φ P ={B1·Q S1 B2·Q S2 ,......,B 11 ·Q S11} (12)

[0067] In the formula, Φ P The weights of the calibrated backbone network are used to perform a convolution operation on the input low-exposure image to obtain the luminance component, as shown below:

[0068] Er c (x)=Net S {Φ P ;D c (x)} (13)

[0069] In the formula, Net S {Φ P ;D c (x)} represents the backbone network, where the low-exposure image D is... c (x) is used as input, and the luminance component Er c (x) is the output.

[0070] Substituting formula (13) into formula (3) yields the enhanced long-exposure image H. c(x), is denoted as:

[0071]

[0072] The forward propagation process of the network shows that the dynamic adjustment factor φ in formula (10) is dynamically variable, and then the weight calibration graph {B1, B2, …, B 11} is dynamically variable, and the weight Φ P of the calibrated backbone network is dynamically variable, and then the output luminance component Er c (x) is also dynamically variable, and therefore the enhanced long-exposure image H c (x) can change with the dynamic adjustment factor φ.

[0073] The loss function of the network is shown in formula (15), which includes two items: a data preservation item and a feature similarity item. The data preservation item is used to minimize the difference between each pixel of the output image and the reference image. The feature similarity item is used to minimize the difference between the color, brightness, and saturation of the output image and the reference image.

[0074]

[0075] In the formula, is the reference image, and ||.||2 is the L2 norm. λ is an adjustment factor used to balance the weights of the two loss items, which is set to 0.01 in this patent. respectively represent the color, brightness, and saturation feature similarities, and are calculated as follows:

[0076]

[0077]

[0078]

[0079] In the formula, σ H and σ GT respectively represent the variances of H c (x) and H (x). u H and u GT respectively represent the means of H c (x) and H (x). σ H_GT represents the covariance of H c (x) and H (x). τ1, τ2, and τ3 are small constants used to avoid a denominator of 0, which are set to 0.01, 0.05, and 0.05, respectively.

[0080] As one embodiment, in network training, this disclosure uses long-exposure images captured by a camera as reference images for the output results, and short-exposure images captured in the same scene as input to the backbone network. The exposure time difference between the long-exposure and short-exposure images is used as input to the convolutional weight calibration network. The backbone network and the convolutional weight calibration network are trained synchronously. During training, the Adam optimizer is used to optimize the network weights, with a weight decay rate of 0.0002 and an initial learning rate of 0.0001, which decays by a factor of 10 every 20 epochs. The training batch and training epoch are set to 2 and 50, respectively. The weights of the backbone network are initialized using a Gaussian distribution.

[0081] Low-exposure image enhancement results as follows Figure 4 As shown, the original image has a short exposure time and extremely low brightness. After enhancement by the network proposed in this patent, the image brightness and contrast are significantly improved, and details are clearer. Experimental results show that this network can effectively improve the visual quality of low-exposure images. When different dynamic adjustment factors are input, the brightness level of the output image can be changed, demonstrating the effectiveness of the dynamic calibration method proposed in this patent. Because the network incorporates the McLaurin expansion formula, complex nonlinear transformation functions can be fitted with fewer network layers. Therefore, enhancing a 600*600 image on a computer with an Intel-i5 CPU and an NVIDIA GTX 2080Ti GPU takes only 0.017 seconds.

[0082] Example 2

[0083] One embodiment of this disclosure provides a low-exposure image enhancement system based on embedded McLaurin and convolution, comprising:

[0084] The preprocessing module is used to acquire the low-exposure image to be enhanced and to process and decompose the low-exposure image.

[0085] The image enhancement module is used to input the low-exposure image into the backbone network to extract multi-scale features, introduce a dynamic adjustment factor into the backbone network, use the dynamic adjustment factor to learn a convolution weight calibration map, and use the learned weight calibration map to calibrate the convolution weights of the backbone network. The calibrated convolution kernels are used to perform convolution operations on the features to obtain the luminance component of the low-exposure image, and the luminance component is used to output a long-exposure image that can change with the dynamic adjustment factor.

[0086] Example 3

[0087] One embodiment of this disclosure provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the low-exposure image enhancement method based on embedded McLaurin and convolution.

[0088] Example 4

[0089] In one embodiment of the present disclosure, a terminal device is provided, comprising a processor and a computer readable storage medium, the processor is configured to implement instructions; the computer readable storage medium is configured to store a plurality of instructions, the instructions are adapted to be loaded and executed by the processor to implement the low-exposure image enhancement method based on embedded Maclaurin and convolution.

[0090] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable data processing device to produce a computer-implemented process, so that the instructions executed by the computer or other programmable data processing device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0092] Although the specific embodiments of the present disclosure are described above with reference to the accompanying drawings, the present disclosure is not limited to the above-described embodiments, and various modifications or changes can be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. A low-exposure image enhancement method based on embedded McLaurin and convolution, characterized in that, include: Obtain the low-exposure image to be enhanced, and process and decompose the low-exposure image; The low-exposure image is input into the backbone network to extract multi-scale features. A dynamic adjustment factor is introduced into the backbone network. The dynamic adjustment factor is used to learn a convolution weight calibration map. The learned weight calibration map is used to calibrate the convolution weights of the backbone network. The calibrated convolution kernel is used to perform convolution operations on the features to obtain the luminance component of the low-exposure image. The luminance component is used to output a long-exposure image that can change with the dynamic adjustment factor. The brightness component is a nonlinear combination of multi-scale features. The McLaurin expansion formula is embedded in the nonlinear function. The functional relationship between the brightness component and the low-exposure image is transformed into a combination of multi-scale features and polynomial coefficients through the McLaurin expansion formula, as shown in the following formula: in, It is a long exposure image. It is a low-exposure image. It is a multi-scale feature generation module. These are the coefficients of the Maclaurin expansion formula; The backbone network includes a multi-scale convolution module, from which another branch is derived to obtain the coefficients of the McLaurin expansion formula.

2. The low-exposure image enhancement method based on embedded McLaurin and convolution as described in claim 1, characterized in that, The low-exposure image to be enhanced is obtained, and the low-exposure image is processed and decomposed in the following ways: the low-exposure image is decomposed into the product of the luminance component and the long-exposure image, and a residual term is added during the decomposition process to obtain the luminance component.

3. The low-exposure image enhancement method based on embedded McLaurin and convolution as described in claim 1, characterized in that, Two convolutional layers are set after the multi-scale convolutional module to gradually compress the multi-scale features into three channels, and a skip connection layer is used to weight and calibrate the features.

4. The low-exposure image enhancement method based on embedded McLaurin and convolution as described in claim 1, characterized in that, The additional branch derived from the multi-scale convolution module includes two convolutional layers and one global average pooling layer, wherein the global average pooling layer calibrates image features in the channel dimension.

5. The low-exposure image enhancement method based on embedded McLaurin and convolution as described in claim 1, characterized in that, The process of introducing a dynamic adjustment factor into the backbone network, using the dynamic adjustment factor to learn a convolutional weight calibration map, and calibrating the convolutional weights of the backbone network with the learned weight calibration map includes learning a convolutional weight calibration map from a dynamic adjustment factor, calibrating the convolutional weights of the backbone network with the learned weight calibration map, realizing that the weights of the backbone network are dynamically adjustable. The size of the weight calibration map is the same as the size of the convolutional kernel of the backbone network. The learned weight calibration map is used to calibrate the convolutional kernel of the backbone network to obtain the calibrated convolutional kernel. Then, the calibrated convolutional kernel is used to perform convolution operation on the features to obtain the luminance component.

6. A low-exposure image enhancement system based on embedded McLaurin and convolution, characterized in that, include: The preprocessing module is used to acquire the low-exposure image to be enhanced and process the low-exposure image. break down; The image enhancement module is used to input the low-exposure image into the backbone network to extract multi-scale features, introduce a dynamic adjustment factor into the backbone network, learn a convolution weight calibration map using the dynamic adjustment factor, calibrate the convolution weights of the backbone network using the learned weight calibration map, perform convolution operations on the features using the calibrated convolution kernels to obtain the luminance component of the low-exposure image, and output a long-exposure image that can change with the dynamic adjustment factor using the luminance component. The brightness component is a nonlinear combination of multi-scale features. The McLaurin expansion formula is embedded in the nonlinear function. The functional relationship between the brightness component and the low-exposure image is transformed into a combination of multi-scale features and polynomial coefficients through the McLaurin expansion formula, as shown in the following formula: in, It is a long exposure image. It is a low-exposure image. It is a multi-scale feature generation module. These are the coefficients of the Maclaurin expansion formula; The backbone network includes a multi-scale convolution module, from which another branch is derived to obtain the coefficients of the McLaurin expansion formula.

7. A computer-readable storage medium, characterized in that, It stores multiple instructions, which are adapted to be loaded and executed by the processor of a terminal device, according to any one of claims 1-5, the low-exposure image enhancement method based on embedded McLaurin and convolution.

8. A terminal device, characterized in that, The method includes a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions adapted to be loaded by the processor and executed as described in any one of claims 1-5, a low-exposure image enhancement method based on embedded McLaurin and convolution.

Images