A Lossless Enhancement Method for Image Exposure Correction Based on Decoupled and Aggregated Convolutions

Abstract

The present invention discloses a method for lossless enhancement of image exposure correction based on decoupled and aggregated convolution, which is a new type of convolution unit considering both image contrast and detail features. Its steps include: 1. The contrast perception decoupling unit branch enhances the contrast statistical feature perception ability of the network; 2. The detail perception decoupling unit branch enhances the detail feature perception ability of the network; 3. The hybrid unit branch performs weighted mixing of the detail and contrast perception features. The present invention can enhance the detail and contrast perception abilities based on convolutional neural networks, thereby enabling lossless enhancement of the image exposure correction effect.

Description

Technical Field

[0001] The present invention belongs to the field of image enhancement and processing, and specifically relates to a method for lossless enhancement of image exposure correction based on decoupled and aggregated convolutions. Background Art

[0002] When capturing or processing images in real scenarios, due to factors such as light, camera settings, or others, the finally captured images often tend to be overexposed or underexposed, thus affecting the quality and visual effect of the images. These underexposed / overexposed images not only seriously deteriorate the quality of visual perception, but also affect the performance of subsequent high-level computer vision tasks, such as intelligent computer vision tasks like image classification and segmentation.

[0003] In recent years, with the development of deep learning, especially the wide application of convolutional neural networks (CNNs) in computer vision and image processing tasks, a large number of CNN-based methods for correcting poor image exposure have been proposed. CNNs perform a series of convolutional and non-linear operations, and automatically optimize the overall loss function through the gradient descent algorithm, avoiding the traditional manual feature extraction steps during the training process, and the model has stronger robustness.

[0004] Underexposed / overexposed images are affected by reduced contrast and detail distortion. Contrast degradation changes the statistical distribution of low-frequency components, while detail distortion disrupts the structural characteristics of high-frequency components; existing CNN-based methods often design an end-to-end architecture to learn contrast enhancement and detail restoration in the shared feature space. However, contrast-related features are mainly distributed in low-frequency components, while detail-related features are mainly distributed in high-frequency components. Since low-frequency components are statistically superior to high-frequency components, these methods mainly focus on contrast enhancement and cannot guarantee that high-frequency details can be effectively restored. Summary of the Invention

[0005] Aiming at the problem that the contrast enhancement and detail restoration of deep learning-based image exposure correction methods cannot be balanced, the present invention provides a method for lossless enhancement of image exposure correction based on decoupled and aggregated convolutions, in order to improve the network's perception ability of details and contrast, and thus can losslessly enhance the image exposure correction effect.

[0006] To achieve the above object of the invention, the following technical solutions are adopted:

[0007] The method for lossless enhancement of image exposure correction based on decoupled and aggregated convolutions of the present invention is characterized in that it is carried out according to the following steps:

[0008] Step 1: Obtain the training image dataset S and construct the network M;

[0009] Step 1.1: Synthesize and collect a dataset S of poorly exposed pairs in real scenarios, S = {s1, s2,..., s j ,..., s n}, where s j is the data pair in the j-th scenario, and s j = {I j , gt j}, I j ∈ R C×H×W represents the poorly exposed image in the j-th scenario, and gt j ∈ R C×H×W represents the correctly exposed image in the j-th scenario, where C represents the number of channels of the image, H represents the height of the image, W represents the width of the image, and n represents the number of data pairs in the dataset;

[0010] Step 1.2: Build an image poorly exposed correction neural network M = {E, F, T} using decoupled and aggregated convolutions, where E represents the feature extraction layer, F represents the feature fusion layer, and T represents the image mapping layer; among them, the feature fusion layer F contains m decoupled and aggregated convolutions {C1, C2,..., C i ,..., C m}, and C i represents the i-th layer of decoupled and aggregated convolution; and C i = {X i , D i , A i}, X i represents the detail-aware decoupling unit branch of the i-th layer, D i represents the contrast-aware decoupling unit branch of the i-th layer, A i represents the mixing unit branch of the i-th layer, and m represents the number of decoupled and aggregated convolutions in the feature fusion layer F;

[0011] Step 2: Obtain the image contrast feature F c and the image detail feature F d ;

[0012] Step 2.1: The feature extraction layer E processes the poorly exposed image I j in the j-th scenario to obtain the image feature e j ∈ R C×H×W ;

[0013] Step 2.2: The image feature e j is input into the feature fusion layer F for processing to obtain the image pair mixing feature F j_x_m of the m-th layer in the j-th scenario;

[0014] Step 2.3: The image pair hybrid feature F of the m-th layer j_x_m is input into the feature image mapping layer T, and an exposure correction image O in the j-th scenario is generated j = M(Q m , I j ); Q m represents all convolution parameters of the image exposure correction neural network M;

[0015] Step 3: Train the model weight parameter Q of the image exposure correction neural network M M ;

[0016] Step 3.1: Construct an exposure correction loss function L using Equation (6):

[0017]

[0018] Step 3.2: Based on the exposure correction paired dataset S, use the Adam optimization iterator to train the exposure correction paired dataset S, and calculate the exposure correction loss function L to optimize and update the model weight parameter Q M , until the exposure correction loss function L converges, so as to obtain the trained image exposure correction neural network M' and its optimal convolution parameter Q' M ;

[0019] Step 4: Fuse the optimal convolution parameter Q' in the trained image exposure correction neural network M' M with the optimal convolution parameter W' of the feature fusion layer F F in it;

[0020] Step 4.1: Let the optimal convolution parameter W' of the feature fusion layer F F = {W'1, W'2,..., W' i ,..., W' m}, W' i represents the optimal parameter of the decoupling and aggregation convolution of the i-th layer, and represents the optimal convolution kernel parameter at the p-th position in the detail perception decoupling unit branch X i of the i-th layer; represents the convolution kernel parameter at the p-th position in the contrast perception decoupling unit branch D i of the i-th layer;

[0021] Step 4.2: Use Equation (7) to transform to obtain the equivalent convolution parameter of the i-th layer

[0022]

[0023] In Equation (7), W sc_i represents the residual term equivalent convolution parameter of the contrast perception decoupling unit branch D of the i-th layer with a convolution receptive field of 1, and there is: i the residual term equivalent convolution parameter of

[0024]

[0025] In Equation (8), R represents the set of positions of all images with poor exposure;

[0026] Step 4.3: Use Equation (9) to perform a conversion to obtain the converted convolution kernel parameter at the p-th position in the contrast perception decoupling unit branch D of the i-th layer i

[0027]

[0028] In Equation (9), W sd_i represents the residual term equivalent convolution parameter of the detail perception decoupling unit branch X of the i-th layer with a convolution receptive field of 1, and there is: i the residual term equivalent convolution parameter of

[0029]

[0030] Step 4.4: According to the linear superposition principle of convolution, use Equation (11) to obtain the final equivalent mixed convolution parameter W' z , so as to obtain the equivalent mixed convolution parameter set W' M ={W'1, W'2,..., W' z ,..., W' m}:

[0031]

[0032] Step 4.5: Use W' M to replace the optimal convolution parameter Q' in the trained image exposure correction neural network M' M the optimal convolution parameter W” of the feature fusion layer F in F , so as to obtain the optimal image exposure correction neural network M”, which is used to correct images with poor exposure in any real scene.

[0033] The feature of the image exposure correction lossless enhancement method based on decoupled and aggregated convolution according to the present invention also lies in that the step 2.2 is carried out according to the following steps:

[0034] Step 2.2.1: The detail perception decoupling unit branch X of the i-th layer i uses Equation (2) to extract the image details F of the i-th layer j_c_i ∈R​BxC×H×W ; B represents the number of mini-batch samples in the training stage

[0035]

[0036] In Equation (2), e j_p represents the local feature value at the p-th position in the image feature e j ; represents the local feature value at the central position in the image feature e j ; represents the learnable convolution kernel parameter at the p-th position in the detail-aware decoupling unit branch X i of the i-th layer; R j represents the set of image positions in the j-th scenario;

[0037] Step 2.2.2: The contrast-aware decoupling unit branch D i of the i-th layer extracts the contrast feature F j_d_i ∈R BxC×H×W ;

[0038]

[0039] In Equation (3), represents the learnable convolution kernel parameter at the p-th position in the contrast-aware decoupling unit branch D i of the i-th layer;

[0040] Step 2.2.3: The hybrid unit branch A i of the i-th layer performs weighted weight fusion using Equation (4) to obtain the image pair hybrid feature F j_x_i ∈R BxC×H×W of the i-th layer in the j-th scenario:

[0041] F j_x_i = Sigmoid(α c_i ) · F j_c_i + Sigmoid(α d_i ) · F j_d_i (4)

[0042] In Equation (4), α c_i and α d_i are two learnable weighted weight coefficients in the hybrid unit branch A i of the i-th layer; Sigmoid is the activation function;

[0043] Step 2.2.4: Use the image pair hybrid feature F j_x_i in the j-th scenario as the detail-aware decoupling unit branch X i of the (i + 1)-th layerThe input, and is processed according to the process of Step 2.2.1 - Step 2.2.3 to obtain the image pair hybrid feature F of the (i + 1)-th layer in the j-th scenario j_x_i+1 ; thus, from the hybrid unit branch A of the m-th layer m the image pair hybrid feature F of the m-th layer in the j-th scenario is obtained j_x_m .

[0044] An electronic device of the present invention includes a memory and a processor, characterized in that the memory is used to store a program for supporting the processor to execute the image exposure correction and lossless enhancement method, and the processor is configured to execute the program stored in the memory

[0045] A computer-readable storage medium of the present invention stores a computer program, characterized in that when the computer program is run by a processor, it executes the steps of the image exposure correction and lossless enhancement method

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows

[0047] By decoupling the learning process in the existing convolutional neural network, the present invention proposes a new type of decoupling and aggregation convolution, which can explicitly guide the contrast and detail modeling of the network, improve the contrast perception and detail perception capabilities in the network learning stage, combine the two through feature mixing to obtain a robust feature representation, overcome the problem of poor contrast perception and detail perception capabilities in the prior art, and utilize the linear additivity of convolution to bring about an improvement in the network's contrast perception and detail perception capabilities, without introducing additional computational overhead, and improve the performance of the existing image exposure correction neural network model, thereby improving the visual effects of detail and contrast restoration of the existing poor-exposure network in image exposure correction BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a specific usage flowchart of the decoupling and aggregation convolution of the present invention DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] In this embodiment, an image exposure correction and lossless enhancement method based on decoupling and aggregation convolution enhances the contrast through a contrast perception decoupling unit and enhances the details through a detail perception decoupling unit, and then simultaneously adjusts the feature response degrees of the two through learnable weighting coefficients. Finally, the model topologies of the detail perception decoupling unit branch and the contrast perception decoupling unit branch are fused through reparameterization aggregation conversion to be equivalently transformed into a single convolution unit, so as to losslessly improve the existing image exposure correction effect. Specifically, as Figure 1 shown, the method includes the following steps

[0050] Step 1: Obtain the dataset S of defective exposure training images and construct the network M;

[0051] Step 1.1: Synthesize and collect the defective exposure paired dataset S = {s1, s2,..., s j ,..., s n} in the real scene, where s j is the data pair in the j-th scene, and s j = {I j , gt j}, I j ∈ R C×H×W represents the defective exposure image in the j-th scene, and gt j ∈ R 3×H×W represents the correctly exposed image in the j-th scene, where the input image is an RGB image, C represents the number of channels of the image, H represents the height of the image, W represents the width of the image, and n represents the number of data pairs in the dataset;

[0052] Step 1.2: Use decoupled and aggregated convolutions to build the image defective exposure correction neural network M = {E, F, T}, where E represents the feature extraction layer, F represents the feature fusion layer, and T represents the image mapping layer; among them, the feature fusion layer F contains m decoupled and aggregated convolutions {C1, C2,..., C i ,..., C m}, and C i represents the decoupled and aggregated convolution of the i-th layer; and C i = {X i , D i , A i}, X i represents the detail perception decoupled unit branch of the i-th layer, D i represents the contrast perception decoupled unit branch of the i-th layer, A i represents the mixing unit branch of the i-th layer, and m represents the number of decoupled and aggregated convolutions in the feature fusion layer F.

[0053] Step 2: Obtain the image contrast feature F c and the image detail feature Fd;

[0054] Step 2.1: The feature extraction layer E processes the defective exposure image I j in the j-th scene to obtain the image feature e j ∈ R C×H×W ;

[0055] Step 2.2: The feature fusion layer F processes the image feature e j using Equation (1);

[0056] e j = E(Ij ) (1)

[0057] Step 2.2.1: Detail perception decoupling unit branch X of the i-th layer i Extract the image details Fi of the i-th layer using Equation (2) j_c_i ∈R BxC×H×W ; where B represents the number of mini-batch samples in the training phase

[0058]

[0059] In Equation (2), e j_p represents the local feature value at the p-th position in the image feature e j ; represents the local feature value at the central position in the image feature e j ; represents the learnable convolution kernel parameter at the p-th position in the detail perception decoupling unit branch X i of the i-th layer; R j represents the set of image positions in the j-th scenario

[0060] Step 2.2.2: Contrast perception decoupling unit branch D of the i-th layer i Extract the contrast feature Fi using Equation (3) j_d_i ∈R BxC×H×W ;

[0061]

[0062] In Equation (3), represents the learnable convolution kernel parameter at the p-th position in the contrast perception decoupling unit branch D i of the i-th layer;

[0063] Step 2.2.3: Hybrid unit branch A of the i-th layer i Perform weighted weight fusion using Equation (4) to obtain the hybrid feature Fi of the i-th layer in the j-th scenario j_x_i ∈R BxC×H×W :

[0064] Fi j_x_i = Sigmoid(α c_i )·Fi j_c_i + Sigmoid(α d_i )·Fi j_d_i (4)

[0065] In Equation (4), α c_i and α d_i are two learnable weighted weight coefficients in the hybrid unit branch A i of the i-th layer; Sigmoid is the activation function

[0066] Step 2.2.4: Use the hybrid feature F of the image pair in the j-th scenario j_x_i as the input of the detail-aware decoupling unit branch X of the (i + 1)-th layer, and process it according to the process of Step 2.2.1 - Step 2.2.3 to obtain the hybrid feature F of the image pair in the (i + 1)-th layer in the j-th scenario i ; thus, obtain the hybrid feature F of the image pair in the m-th layer in the j-th scenario from the hybrid unit branch A of the m-th layer j_x_i+1 ; m ; j_x_m

[0067] Step 2.3: Input the hybrid feature F of the image pair in the m-th layer into the input feature image mapping layer T, and use Equation (5) to generate the exposure correction image O in the j-th scenario j_x_m = M(Q j , I m ) ; Q j represents all convolution parameters of the image exposure defect correction neural network M; m

[0068] O j = M(Q m , I j ) (5)

[0069] Step 3: Train the model weight parameters Q of the image exposure defect correction neural network M M ;

[0070] Step 3.1: Use Equation (6) to construct the exposure defect correction loss function L:

[0071]

[0072] Step 3.2: Based on the exposure defect pairing dataset S, use the Adam optimization iterator to train the exposure defect pairing dataset S, and calculate the exposure defect correction loss function L to optimize and update the model weight parameters Q M , until the exposure defect correction loss function L converges, so as to obtain the trained image exposure defect correction neural network M' and its optimal convolution parameters Q' M ;

[0073] Step 4: Fuse the optimal convolution parameters W' of the feature fusion layer F in the optimal convolution parameters Q' of the trained image exposure defect correction neural network M'; this step is to reduce the computational overhead of network operation; M ; this step is to reduce the computational overhead of network operation; F

[0074] Step 4.1: Let the optimal convolution parameter W' of the feature fusion layer F F={W'1, W'2,..., W' i ,..., W' m}, where W' i represents the optimal parameters of the decoupled and aggregated convolution of the i-th layer, and represents the optimal convolution kernel parameter at the p-th position in the detail-aware decoupling unit branch X i of the i-th layer; represents the convolution kernel parameter at the p-th position in the contrast-aware decoupling unit branch D i of the i-th layer;

[0075] Step 4.2: Use Equation (7) to transform to obtain the equivalent convolution parameters

[0076]

[0077] In Equation (7), W sc_i represents the equivalent convolution parameter of the contrast-aware branch residual term with a convolution receptive field of 1, and there is:

[0078]

[0079] In Equation (8), R represents the set of positions of all underexposed images;

[0080] Step 4.3: Use Equation (9) to transform to obtain

[0081]

[0082] In Equation (9), W sd_i represents the equivalent convolution parameter of the detail-aware branch residual term with a convolution receptive field of 1, and there is:

[0083]

[0084] Step 4.4: According to the principle of linear superposition of convolution, use Equation (11) to obtain the final equivalent hybrid convolution parameter W‘ z , thus obtaining the set of equivalent hybrid convolution parameters W' M ={W'1, W'2,..., W' z ,..., W' m}:

[0085]

[0086] Step 4.5: Use W' M to replace the optimal convolution parameter Q' in the trained image underexposure correction neural network M'M Optimal convolution parameter W” of the feature fusion layer F F Therefore, the optimal image under-exposure correction neural network M” is obtained, which is used to correct the under-exposed images in any real scene.

[0087] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0088] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is run by a processor, it executes the steps of the above method.

[0089] In summary, the present invention proposes to decouple contrast enhancement and detail restoration during the convolution process of a convolutional neural network. This method is based on statistical observations that the feature responses in local regions can be decomposed into low-frequency and high-frequency components through differential operations. Based on this statistical observation, the present invention introduces a new type of convolutional unit with parallel contrast perception and detail perception units, called decoupled and aggregated convolution, to guide the network to model image contrast and detail features. Different from traditional convolutional neural networks, this scheme injects addition / differential operations into the convolution process to explicitly guide contrast and detail feature modeling. In addition, to balance contrast enhancement and detail restoration, a dynamic coefficient is introduced for each branch to adjust the degree of feature response. The decoupled and aggregated convolution proposed by the present invention can be used as a general unit to replace the convolution kernels in existing exposure correction networks based on convolutional neural networks to promote contrast enhancement and detail restoration. Finally, through the reparameterized aggregation transformation technology, the two parallel units are equivalently transformed into one convolutional unit, greatly reducing the computational overhead of the network.

Claims

1. An image exposure correction lossless enhancement method based on decoupled and aggregated convolution, characterized in that, It is carried out according to the following steps: Step 1: Obtain the training image dataset and construct a network ; Step 1.1: Synthesize and collect a dataset of poorly exposed pairs in real scenarios , where is the data pair in the j-th scenario, and , represents the poorly exposed image in the j-th scenario, represents the correctly exposed image in the j-th scenario, where C represents the number of channels of the image, H represents the height of the image, W represents the width of the image, and n represents the number of data pairs in the dataset; Step 1.2: Build an image exposure defect correction neural network using decoupled and aggregated convolutions , where represents the feature extraction layer, represents the feature fusion layer, represents the image mapping layer; among them, the feature fusion layer contains m decoupled and aggregated convolutions , represents the i-th layer of decoupled and aggregated convolution; and , represents the i-th layer of detail perception decoupled unit branch, represents the i-th layer of contrast perception decoupled unit branch, represents the i-th layer of mixing unit branch, and m represents the number of decoupled and aggregated convolutions in the feature fusion layer ; Step 2: Obtain the image contrast feature and the image detail feature ; Step 2.1: The feature extraction layer processes the defective exposure image in the j-th scenario to obtain the image features in the j-th scenario ; Step 2.2: The image features are input into the feature fusion layer for processing to obtain the hybrid features of the image pair at the m-th layer in the j-th scenario ; Step 2.3: The image pair hybrid features of the m-th layer are input into the image mapping layer to generate an exposure correction image in the j-th scenario ; denotes all convolutional parameters of the image exposure defect correction neural network Step 3: Train the model weight parameters of the neural network for correcting poor exposure of images of the ; Step 3.1: Construct an exposure defect correction loss function using Equation (6) :[[]]END]] (6) Step 3.2 Based on the dataset of defective exposure pairs , use the Adam optimization iterator to train the dataset of defective exposure pairs , and calculate the defective exposure correction loss function to optimize and update the model weight parameters , until the defective exposure correction loss function converges, thereby obtaining the trained neural network for defective image exposure correction ' and its optimal convolution parameters ; Step 4: Correct the trained neural network for image underexposure The optimal convolution parameters in the feature fusion layer in the convolution parameters are fused; Step 4.1: Let the optimal convolution parameters of the feature fusion layer be = , where represents the optimal parameters of the decoupling and aggregation convolution of the i-th layer, and , represents the optimal convolution kernel parameters at the p-th position in the detail-aware decoupling unit branch of the i-th layer; represents the convolution kernel parameters at the p-th position in the contrast-aware decoupling unit branch of the i-th layer. Step 4.2: Use Equation (7) to perform conversion to obtain the equivalent convolution parameters of the i-th layer : (7) In formula (7), represents the residual term equivalent convolution parameter of the i-th layer contrast perception decoupling unit branch with a convolution receptive field of 1, and there is: ​ (8) In formula (8), R represents the set of positions of all images with poor exposure; Step 4.3: Use Equation (9) to perform conversion to obtain the converted convolutional kernel parameters at the p-th position in the contrast perception decoupling unit branch of the i-th layer : ​ (9) In formula (9), represents the residual equivalent convolution parameter of the detail perception decoupling unit branch of the i-th layer with a convolution receptive field of 1, and there is:[[]] ​ (10) Step 4.4: According to the principle of linear superposition of convolution, the final equivalent hybrid convolution parameters are obtained using Equation (11) , thereby obtaining the equivalent hybrid convolution parameter set = : (11) Step 4.5: Use to replace the optimal convolution parameters in the trained neural network for correcting poor image exposure ' and the optimal convolution parameters of the feature fusion layer in ' so as to obtain the optimal neural network for correcting poor image exposure ' for correcting poorly exposed images in any real scenario.

2. The method for lossless enhancement of image exposure correction based on decoupled and aggregated convolution according to claim 1, characterized in that The said step 2.2 is carried out according to the following steps: Step 2.2.1: The detail perception decoupling unit branch of the i-th layer Extract the image details of the i-th layer using Equation (2) ; B represents the number of mini-batch samples in the training phase = (2) In formula (2), represents the local feature value at the p-th position in the image feature represents the local feature value at the central position in the image feature; represents the convolutional kernel parameter to be learned at the p-th position in the detail-aware decoupling unit branch of the i-th layer; represents the set of image positions in the j-th scenario; Step 2.2.2: The contrast perception decoupling unit branch of the i-th layer Extract the contrast feature using Equation (3) ; = (3) In Equation (3), represents the learnable convolutional kernel parameter at the p-th position in the contrast perception decoupling unit branch of the i-th layer ; Step 2.2.3: Hybrid unit branch of the i-th layer Perform weighted weight fusion using Equation (4) to obtain the hybrid feature of the image pair at the i-th layer in the j-th scenario :[[]]END]] = + (4) In formula (4), and are the two weighted weight coefficients to be learned in the hybrid unit branch of the i-th layer ; is the activation function; Step 2.2.4: Use the mixed features of the image pair in the j-th scenario as the input of the detail-aware decoupling unit branch of the (i + 1)-th layer and process it according to the procedures in Steps 2.2.1 - 2.2.3 to obtain the mixed features of the image pair in the (i + 1)-th layer in the j-th scenario ; thus, the mixed unit branch of the m-th layer is used to obtain the mixed features of the image pair in the m-th layer in the j-th scenario .

3. An electronic device, comprising a memory and a processor, characterized in that, The said memory is used to store a program for supporting the processor to execute the image exposure correction and lossless enhancement method described in claim 1 or 2, and the said processor is configured to execute the program stored in the said memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the said computer program is run by the processor, it executes the steps of the image exposure correction and lossless enhancement method described in claim 1 or 2.

Images