Optimized Zero-dce network raw domain contrast enhancement-based method

Abstract

The invention provides an optimization-based raw domain contrast enhancement method for a Zero-dce network, and the method comprises the following steps: S1, data preparation and implementation: A, collecting a low-contrast raw map, namely raw data; b, converting the original Raw data into the size of (1080, 1920); s2, implementing CLAHE enhancement and cutting: performing CLAHE enhancement, cutting the Raw data with the changed size and a result after CLAHE enhancement into a plurality of (540, 960) fragments, taking a cutting result at a corresponding position as a training data pair, and performing a step S3.3; s3, constructing an optimized Raw domain contrast enhancement Zero-dceR model, and inputting the model by using (1, 540, 960), and S4, updating parameters. According to the method, deep learning contrast enhancement is carried out on a low-contrast Raw domain obtained by a sensor, a single channel is used as input of the network, a Zero-dce network suitable for single-channel Raw data is constructed, a good self-adaptive contrast enhancement effect is achieved for Raw domain scenes with different characteristics, and finally AIISP performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for Raw domain contrast enhancement based on an optimized Zero-DCE network. Background Art

[0002] In the prior art, contrast enhancement plays an important role in improving the visual quality of computer vision, pattern recognition, and digital image processing. In actual scenes, the image contrast quality is reduced due to lack of photography experience, inherent defects of imaging equipment, or capturing scenes with low or high illumination intensity. Raw images are the original data obtained from the sensor, which retains more information. Contrast enhancement in the Raw domain can obtain better imaging effects later. However, there are few studies on contrast enhancement in the Raw domain, most of which are concentrated in the image RGB domain and mainly based on traditional algorithm enhancement. The actual application effects of these methods in the Raw domain are poor.

[0003] The disadvantages of the prior art are:

[0004] There are many classic enhancement methods in the current contrast enhancement research, such as gamma transformation and histogram equalization for global grayscale adjustment. Although these methods are efficient, they cannot achieve adaptive contrast enhancement. Grayscale adjustment based on local conditions, such as contrast-limited adaptive histogram equalization and adaptive contrast enhancement, can alleviate the problems caused by global grayscale adjustment and improve local contrast effects, but these methods can only be used for single image processing, and the same parameters cannot be adapted to different scenes.

[0005] In addition, common terms in the prior art include:

[0006] Contrast: A measurement of the difference in brightness between the brightest white and the darkest black in the light and dark areas of an image.

[0007] CLAHE: Contrast-constrained adaptive histogram equalization algorithm.

[0008] RGB: The three optical primary colors, R, G, and B stand for red, green, and blue respectively.

[0009] Raw data: The image sensor converts the captured light source signal into raw data of digital signal. Summary of the invention

[0010] In order to solve the above problems, the purpose of this application is: to address the problems of low contrast, low transparency and low saturation in the collected Raw domain, the present invention proposes a method of converting Raw data into a single-channel space for processing, obtaining a suitable high-contrast Raw domain through CLAHE enhancement, and using a single channel as the input of the network to construct a Zero-DCE network adapted to single-channel Raw data, so as to obtain an efficient and generalized Raw domain contrast enhancement technical method.

[0011] Specifically, the present invention provides a method for Raw domain contrast enhancement based on optimized Zero-DCE network, the method comprising the following steps:

[0012] S1, data preparation, implementation:

[0013] C, collect low contrast Raw images, i.e. Raw data;

[0014] D, convert the original Raw data into (1080,1920) size;

[0015] S2, implement CLAHE enhancement and cropping: perform CLAHE enhancement, crop the raw data after resizing and the results after CLAHE enhancement into multiple (540,960) segments, and use the cropping results at corresponding positions as training data pairs to proceed to step S3.3;

[0016] S3, builds an optimized Raw domain contrast enhancement Zero-dceR model with (1,540,960) input to achieve:

[0017] S3.1, obtain Zero-dceR network;

[0018] S3.2, prediction;

[0019] S3.3, calculating the training loss according to the results of S2 and S3.2, i.e., the cropping result after CLAHE enhancement in step S2 and the prediction result of step S3.2;

[0020] S4, parameter update.

[0021] The step S2 further comprises:

[0022] S2: Perform CLAHE enhancement on the Raw data after data morphology transformation, set the local mapping window size M and the local mapping function slope limit threshold T, and obtain the result of local equalization, as shown in formula (1), and finally construct a low-contrast-high-contrast data set:

[0023] img_clahe=CLAHE(img,M,T) (1).

[0024] The method step S3 further comprises:

[0025] S3: Build an optimized Raw domain contrast enhancement Zero-dceR model, adjust the model structure by cropping and post-processing the low-contrast-high contrast dataset, so as to adapt to the Raw domain contrast enhancement; the specific modifications in the Zero-dceR network algorithm are as follows: set the number of convolution channels of the first layer input and the last layer output to 1 channel, and use (1,540,960) as the input image scale of the encoder network, set one downsampling and one upsampling layer, and use L1loss and SSIM contrast loss as the training loss for learning.

[0026] The specific structure of the network is:

[0027] The step S3.1 further comprises:

[0028] ① The optimized Raw domain contrast enhancement Zero-dceR model mainly consists of a downsampling module to extract key features while reducing the amount of training parameters; a seven-layer deep convolution module to extract deep features; a one-layer upsampling module to restore to the input size; and finally an enhancement module to fuse the input information;

[0029] ② The optimized Raw domain contrast enhancement Zero-dceR model downsampling module includes one layer of convolution and one layer of pooling processing, and the upsampling module includes one layer of convolution and one layer of upsampling processing;

[0030] ③ The deep convolution module of the optimized Raw domain contrast enhancement Zero-dceR model mainly includes two layers of convolution. The convolution kernel size of the first layer is 3, and the convolution kernel size of the second layer is 1. The result of the first layer of convolution is the input of the second layer of convolution, and the result of the second layer of convolution is the output of the deep convolution module; the first deep convolution layer to the fourth deep convolution layer are connected to the input and output in sequence; residual connection is performed from the fifth layer, that is, the output of the third deep convolution layer and the fourth deep convolution layer are spliced ​​as the input of the fifth deep convolution layer, the output of the second deep convolution layer and the fifth deep convolution layer are spliced ​​as the input of the sixth deep convolution layer, and finally the output of the first deep convolution layer and the sixth deep convolution layer are spliced ​​as the input of the seventh deep convolution layer;

[0031] That is, step S3.1 corresponds to steps ①-③;

[0032] The step S3.2 further comprises:

[0033] ④ The enhancement module of the optimized Raw domain contrast enhancement Zero-dceR model fuses the input information into the output of the upsampled deep convolution module through equations (2)-(3).

[0034] o=Up(Depth_conv(Down(x))) (2)

[0035] t i+1 =t i +o*(t i 2 -t i ), i=0,1...7 (3)

[0036] Among them, x represents the input of the network, Down() represents the downsampling module of the network, Depth_conv() represents the seven-layer deep convolution module, Up() represents the upsampling module, o is the upsampling output, when i=0, t0=o;

[0037] That is, step S3.2 corresponds to this step ④;

[0038] The step S3.3 further comprises:

[0039] ⑤ The optimized Raw domain contrast enhancement Zero-dceR model uses L1loss, as shown in formula (4), and SSIM contrast loss, as shown in formula (5), as the final loss function of the network.

[0040] L1(x,y)=|x-Zero-dceR(x)| (4)

[0041]

[0042] Loss(x,y)=1-C(x,y)+L1(x,y) (6)

[0043] Where x represents the predicted output of the network, y represents the high-contrast raw image corresponding to the low-contrast raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively;

[0044] That is, step S3.3 corresponds to step ⑤.

[0045] The step S4 further comprises:

[0046] S4: The network parameters are updated using the optimizer Adam, which is an adaptive optimization algorithm that can adaptively adjust the learning rate; the fixed learning rate is 0.0001 and the training cycle is 100.

[0047] Therefore, the advantages of this application are:

[0048] Deep learning contrast enhancement is performed on the low-contrast Raw domain obtained by the sensor. A single channel is used as the input of the network to build a Zero-DCE network that is adapted to single-channel Raw data. It has a good adaptive contrast enhancement effect for Raw domain scenes with different characteristics, ultimately improving the AIISP performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0050] Figure 1 It is a flow chart of an embodiment of the method of the present application.

[0051] Figure 2 It is a simplified schematic diagram of the process of this application method.

[0052] Figure 3 In the embodiment of the present application method Zero-dceR Schematic diagram of the specific modifications in the algorithm. DETAILED DESCRIPTION

[0053] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0054] The embodiment of the present application proposes a method for Raw domain contrast enhancement based on optimized Zero-DCE network, the steps are as follows: Figure 1 As shown, the following steps are included:

[0055] S1, data preparation, implementation:

[0056] A, collect low contrast Raw images, i.e. Raw data;

[0057] B, convert the original Raw data into (1080,1920) size;

[0058] S2, implement CLAHE enhancement and cropping: perform CLAHE enhancement, crop the raw data after resizing and the results after CLAHE enhancement into multiple (540,960) segments, and use the cropping results at corresponding positions as training data pairs to proceed to step S3.3;

[0059] S3, builds an optimized Raw domain contrast enhancement Zero-dceR model with (1,540,960) input to achieve:

[0060] S3.1, obtain Zero-dceR network;

[0061] S3.2, prediction;

[0062] S3.3, calculating the training loss according to the results of S2 and S3.2, i.e., the cropping result after CLAHE enhancement in step S2 and the prediction result of step S3.2;

[0063] S4, parameter update, using the optimizer Adam to update the network parameters. The Adam optimizer is an adaptive optimization algorithm that can adaptively adjust the learning rate; the fixed learning rate is 0.0001 and the training cycle is 100.

[0064] Specifically, Figure 2 As shown, the steps further include the following:

[0065] The step S2: CLAHE enhancement is performed on the Raw data after the data morphology is transformed, and the local mapping window size M and the local mapping function slope limit threshold T are set to obtain the result of local equalization, as shown in formula (1), and finally a low-contrast-high-contrast data set is constructed:

[0066] img_clahe=CLAHE(img,M,T) (1). The step S3: construct an optimized Raw domain contrast enhancement Zero-dceR model. By cropping and post-processing the low-contrast-high-contrast data set, the model structure is adjusted to adapt to the Raw domain contrast enhancement. In the Zero-dceR algorithm, the specific modification is: the number of convolution channels of the first layer input and the last layer output is set to 1 channel, and (1,540,960) is used as the input image scale of the encoder network, a downsampling and upsampling layer is set, and the contrast loss of L1loss and SSIM is used as the training loss for learning.

[0067] like Figure 3 As shown, the specific structure of the network is:

[0068] The step S3.1 further comprises:

[0069] ① The optimized Raw domain contrast enhancement Zero-dceR model mainly consists of a downsampling module to extract key features while reducing the amount of training parameters; a seven-layer deep convolution module to extract deep features; a one-layer upsampling module to restore the input size; and finally an enhancement module to fuse the input information.

[0070] ② The downsampling module of the optimized Raw domain contrast enhancement Zero-dceR model includes one layer of convolution and one layer of pooling processing, and the upsampling module includes one layer of convolution and one layer of upsampling processing.

[0071] ③ The deep convolution module of the optimized Raw domain contrast enhancement Zero-dceR model mainly includes two layers of convolution. The convolution kernel size of the first layer is 3, and the convolution kernel size of the second layer is 1. The result of the first layer of convolution is the input of the second layer of convolution, and the result of the second layer of convolution is the output of the deep convolution module. The input and output of the first to fourth deep convolution layers are connected front and back in sequence. Residual connection is performed from the fifth layer, that is, the output of the third deep convolution layer and the fourth deep convolution layer are spliced ​​as the input of the fifth deep convolution layer, the output of the second deep convolution layer and the fifth deep convolution layer are spliced ​​as the input of the sixth deep convolution layer, and finally the output of the first deep convolution layer and the sixth deep convolution layer are spliced ​​as the input of the seventh deep convolution layer.

[0072] That is, step S3.1 corresponds to steps ①-③;

[0073] The step S3.2 further comprises:

[0074] ④ The enhancement module of the optimized Raw domain contrast enhancement Zero-dceR model fuses the input information into the output of the upsampled deep convolution module through equations (2)-(3).

[0075] o=Up(Depth_conv(Down(x))) (2)

[0076] t i+1 =t i +o*(t i 2 -t i ), i=0,1...7 (3)

[0077] Among them, x represents the input of the network, Down() represents the downsampling module of the network, Depth_conv() represents the seven-layer deep convolution module, Up() represents the upsampling module, o is the upsampling output, and when i=0, t0=o.

[0078] That is, step S3.2 corresponds to this step ④;

[0079] The step S3.3 further comprises:

[0080] ⑤ The optimized Raw domain contrast enhancement Zero-dceR model uses L1loss, as shown in formula (4), and SSIM contrast loss, as shown in formula (5), as the final loss function of the network.

[0081] L1(x,y)=|x-Zero-dceR(x)| (4)

[0082]

[0083] Loss(x,y)=1-C(x,y)+L1(x,y) (6)

[0084] Where x represents the predicted output of the network, y represents the high-contrast raw image corresponding to the low-contrast raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively.

[0085] That is, step S3.3 corresponds to step ⑤.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for Raw domain contrast enhancement based on optimized Zero-DCE network, characterized in that: The method comprises the following steps: S1, data preparation, implementation: A, collect low contrast Raw images, i.e. Raw data; B, convert the original Raw data into (1080,1920) size; S2, implement CLAHE enhancement and cropping: perform CLAHE enhancement, crop the raw data after resizing and the results after CLAHE enhancement into multiple (540,960) segments, and use the cropping results at corresponding positions as training data pairs to proceed to step S3.3; S3, builds an optimized Raw domain contrast enhancement Zero-dceR model with (1,540,960) input to achieve: S3.1, obtain Zero-dceR network; S3.2, prediction; S3.3, calculating the training loss according to the results of S2 and S3.2, i.e., the cropping result after CLAHE enhancement in step S2 and the prediction result of step S3.2; S4, parameter update.

2. According to claim 1, a method for Raw domain contrast enhancement based on optimized Zero-DCE network is characterized in that: The step S2 further comprises: S2: Perform CLAHE enhancement on the Raw data after data morphology transformation, set the local mapping window size M and the local mapping function slope limit threshold T, and obtain the result of local equalization, as shown in formula (1), and finally construct a low-contrast-high-contrast data set: img_clahe=CLAHE(img,M,T) (1).

3. According to claim 1, a method for Raw domain contrast enhancement based on optimized Zero-DCE network is characterized in that: The method step S3 further comprises: S3: Build an optimized Raw domain contrast enhancement Zero-dceR model, adjust the model structure by cropping and post-processing the low-contrast-high contrast dataset, so as to adapt to the Raw domain contrast enhancement; the specific modifications in the Zero-dceR network algorithm are as follows: set the number of convolution channels of the first layer input and the last layer output to 1 channel, and use (1,540,960) as the input image scale of the encoder network, set one downsampling and one upsampling layer, and use L1loss and SSIM contrast loss as the training loss for learning.

4. The method for Raw domain contrast enhancement based on optimized Zero-DCE network according to claim 3 is characterized in that: The specific structure of the network is: The step S3.1 further comprises: ① The optimized Raw domain contrast enhancement Zero-dceR model mainly consists of a downsampling module to extract key features while reducing the amount of training parameters; a seven-layer deep convolution module to extract deep features; a one-layer upsampling module to restore to the input size; and finally an enhancement module to fuse the input information; ② The optimized Raw domain contrast enhancement Zero-dceR model downsampling module includes one layer of convolution and one layer of pooling processing, and the upsampling module includes one layer of convolution and one layer of upsampling processing; ③ The deep convolution module of the optimized Raw domain contrast enhancement Zero-dceR model mainly includes two layers of convolution. The convolution kernel size of the first layer is 3, and the convolution kernel size of the second layer is 1. The result of the first layer of convolution is the input of the second layer of convolution, and the result of the second layer of convolution is the output of the deep convolution module; the first deep convolution layer to the fourth deep convolution layer are connected to the input and output in sequence; residual connection is performed from the fifth layer, that is, the output of the third deep convolution layer and the fourth deep convolution layer are spliced ​​as the input of the fifth deep convolution layer, the output of the second deep convolution layer and the fifth deep convolution layer are spliced ​​as the input of the sixth deep convolution layer, and finally the output of the first deep convolution layer and the sixth deep convolution layer are spliced ​​as the input of the seventh deep convolution layer; That is, step S3.1 corresponds to steps ①-③; The step S3.2 further comprises: ④ The enhancement module of the optimized Raw domain contrast enhancement Zero-dceR model fuses the input information into the output of the upsampled deep convolution module through equations (2)-(3). o=Up(Depth_conv(Down(x))) (2) t i+1 =t i +o*(t i 2 -t i ),i=0,1...7 (3) Among them, x represents the input of the network, Down() represents the downsampling module of the network, Depth_conv() represents the seven-layer deep convolution module, Up() represents the upsampling module, o is the upsampling output, when i=0, t0=o; That is, step S3.2 corresponds to this step ④; The step S3.3 further comprises: ⑤ The optimized Raw domain contrast enhancement Zero-dceR model uses L1loss, as shown in formula (4), and SSIM contrast loss, as shown in formula (5), as the final loss function of the network. L1(x,y)=|x-Zero-dceR(x)| (4) Loss(x,y)=1-C(x,y)+L1(x,y) (6) Where x represents the predicted output of the network, y represents the high-contrast raw image corresponding to the low-contrast raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively; That is, step S3.3 corresponds to step ⑤.

5. The method for Raw domain contrast enhancement based on optimized Zero-DCE network according to claim 4, characterized in that: The step S4 further comprises: S4: The network parameters are updated using the optimizer Adam, which is an adaptive optimization algorithm that can adaptively adjust the learning rate; the fixed learning rate is 0.0001 and the training cycle is 100.

Images