Optimization-based raw domain contrast enhancement method for Unet network

Abstract

The invention provides an optimization-based raw domain contrast enhancement method for a Unet network, and the method comprises the steps: S1, data preparation and implementation: A, collecting a low-contrast raw map, namely raw data; b, converting a single channel into four rgb channels; then the step S2 and the step S3 are carried out respectively; s2, implementing: enhancing the CLAHE; s3, cutting is carried out, and then the step S4.3 is continued; s4, an optimized Raw domain contrast enhancement UnetR model is constructed; s4.1, a UnetR network is obtained; s4.2, performing prediction; s4.3, calculating the training loss according to the results of the S2 and the S4.2; and S5, updating parameters. A proper high-contrast Raw graph is obtained through CLAHE enhancement, and an end-to-end contrast enhancement network is trained, so that an efficient and generalization Raw domain contrast enhancement technical method is obtained.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method for Raw domain contrast enhancement based on an optimized Unet network. Background Art

[0002] In the prior art, contrast enhancement plays an important role in improving the visual quality of computer vision, pattern recognition, and digital image processing. In actual scenes, the image contrast quality is reduced due to lack of photography experience, inherent defects of imaging equipment, or capturing scenes with low or high illumination intensity. Raw images are the original data obtained from the sensor, which retains more information. Contrast enhancement in the Raw domain can obtain better imaging effects later. However, there are few studies on contrast enhancement in the Raw domain, most of which are concentrated in the image RGB domain and mainly based on traditional algorithm enhancement. The actual application effects of these methods in the Raw domain are poor.

[0003] The disadvantages of the prior art are:

[0004] There are many classic enhancement methods in the current contrast enhancement research, such as gamma transformation and histogram equalization for global grayscale adjustment. Although these methods are efficient, they cannot achieve adaptive contrast enhancement. Grayscale adjustment based on local conditions, such as contrast-limited adaptive histogram equalization and adaptive contrast enhancement, can alleviate the problems caused by global grayscale adjustment and improve local contrast effects, but these methods can only be used for single image processing, and the same parameters cannot be adapted to different scenes.

[0005] In addition, common terms in the prior art include:

[0006] Contrast: A measurement of the difference in brightness between the brightest white and the darkest black in the light and dark areas of an image.

[0007] CLAHE: Contrast-constrained adaptive histogram equalization algorithm.

[0008] RGB: three optical primary colors, R, G, and B stand for red, green, and blue respectively.

[0009] Raw data: The image sensor converts the captured light source signal into raw data of digital signal. Summary of the invention

[0010] In order to solve the above problems, the purpose of this application is to: in view of the problems of low contrast, low transparency and low saturation in the collected Raw domain, the present invention proposes a method of converting Raw data into a four-channel space for processing, obtaining a suitable high-contrast Raw domain through CLAHE enhancement, and training an end-to-end contrast enhancement network to obtain an efficient and generalized Raw domain contrast enhancement technical method.

[0011] Specifically, the present invention provides a method for Raw domain contrast enhancement based on an optimized Unet network, the method comprising the following steps:

[0012] S1, data preparation, implementation:

[0013] A, collect low contrast Raw images, i.e. Raw data;

[0014] B, convert single channel to RGB four channels; then proceed to steps S2 and S3 respectively;

[0015] S2, implementation: CLAHE enhancement;

[0016] S3, perform cropping; crop each channel of the original data and the enhanced four-channel data to a size of (256, 256), concatenate the cropping results at the corresponding positions to obtain (4, 256, 256), and obtain several cropping results for each original Raw data as training data; proceed to step S4.3;

[0017] S4, build an optimized Raw domain contrast enhancement UnetR model;

[0018] S4.1, obtain the UnetR network;

[0019] S4.2, prediction;

[0020] S4.3, calculate the training loss based on the results of S3 and S4.2;

[0021] S5, parameter update.

[0022] The method further comprises:

[0023] S1: Data preparation: Collect N low-contrast original Raw images, and map the single-channel Raw data to the rggb four-channel, as shown in formula (1):

[0024] img_rggb=to_rggb(img_raw) (1)

[0025] Among them, to_rggb() represents the mapping function from single channel to four channels;

[0026] S2: High contrast Raw image generation: Perform CLAHE enhancement on each channel of img_rggb, set the local mapping window size M and the local mapping function slope limit threshold T, and obtain the local equalization result, as shown in formula (2). Then, concatenate the transformed results of the four channels, as shown in formula (3). Different M and T are set for different scenes, and the best effect is selected as gt.

[0027] img_clahe[i]=clahe(img_rggb[i], M, T) (2)

[0028] img_gt=stack(img_clahe[i]), i=0, 1, 2, 3 (3)

[0029] S3: Construct a low-contrast-high-contrast training data set: crop the low-contrast and high-contrast Raw data rggb four-channel space with a cropping size of (4,256,256). After cropping, the low-contrast and high-contrast local positions correspond one to one to obtain each set of training data;

[0030] S4: Building an optimized Raw domain contrast enhancement UnetR model:

[0031] The model structure is adjusted through post-cropping of low-contrast-high-contrast datasets and four-channel spatial features to adapt to Raw domain contrast enhancement; the specific modifications in the UnetR network are as follows: the number of convolution channels of the first layer input and the last layer output is set to 4 channels, (4,256,256) is used as the input image scale of the encoder network, and the contrast loss of L1loss and SSIM is used as the training loss for learning;

[0032] S5: The network parameters are updated using the optimizer Adam, which is an adaptive optimization algorithm that can adaptively adjust the learning rate; the learning rate is fixed to 0.0001 and the training cycle is 100.

[0033] In step S4, the specific steps of the network are:

[0034] The step S4.1 further comprises:

[0035] ① The optimized Raw domain contrast enhancement UnetR model encoder includes a primary encoder and a secondary encoder, which extracts features of different scales from the input low-contrast Raw domain;

[0036] Each encoder contains two layers of convolution processing and one layer of downsampling module. After feature extraction, the size is reduced and input to the next level encoder, as shown in equations (4) and (5):

[0037] E1(x)=poolMax(conv(conv(x))) (4)

[0038] E2(x)=poolMax(conv(conv(E1(x)))) (5)

[0039] ② The decoder of the optimized Raw domain contrast enhancement UnetR model includes a primary decoder and a secondary decoder, which aggregates the feature information of different scales obtained by the encoder to obtain a feature map;

[0040] Each decoder includes upsampling operation, two layers of convolution processing and residual connection with the encoder, as shown in equations (6) and (7):

[0041] D1(x)=conv(conv(upsample(conv(E2(x))))) (6)

[0042] D2(x)=conv(conv(upsample(conv(D1(x))))) (7)

[0043] That is, step S4.1 corresponds to steps ① and ②;

[0044] The step S4.2 further comprises:

[0045] ③The last layer of the optimized Raw domain contrast enhancement UnetR model is the spatial information fusion layer, which includes pooling processing, two convolution modules and a spatial attention layer;

[0046] The input is the output of the second-level decoder, and the maximum pooling and average pooling are performed on it to extract the key spatial features as shown in equations (8) and (9). The spatial features are convolved in two layers to obtain the spatial feature attention and dot product with the input to obtain the feature map of fused spatial information, as shown in equation (10):

[0047] M=poolMax(conv(conv(D2(x)))) (8)

[0048] A=poolAvg(conv(conv(D2(x)))) (9)

[0049]

[0050] That is, step S4.2 corresponds to step ③, and O in formula (10) represents the predicted result;

[0051] The step S4.3 further comprises:

[0052] ④ The optimized Raw domain contrast enhancement UnetR model uses L1loss, as shown in formula (11), and SSIM contrast loss, as shown in formula (12), as the final loss function of the network:

[0053] L1(x, y) = |x - UnetR(x)| (11)

[0054]

[0055] Loss(x,y)=1-C(x,y)+L1(x,y) (13)

[0056] Where x represents the predicted output of the network, y represents the high-contrast Raw image corresponding to the low-contrast Raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively;

[0057] The calculation of training loss in step S4.3 corresponds to step ④.

[0058] In the step S1, N=300.

[0059] Therefore, the advantages of this application are:

[0060] Deep learning contrast enhancement is performed on the low-contrast Raw domain obtained by the sensor, which has a good adaptive contrast enhancement effect on Raw domains with different characteristics, ultimately improving the AIISP performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0062] Figure 1 It is a flow chart of an embodiment of the method of the present application.

[0063] Figure 2 It is a simplified schematic diagram of the process of this application method.

[0064] Figure 3 It is a schematic diagram of specific modifications in the UnetR algorithm in the embodiment of the method of the present application. DETAILED DESCRIPTION

[0065] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0066] The embodiment of the present application proposes a method for Raw domain contrast enhancement based on Unet, the steps are as follows: Figure 1 As shown, including:

[0067] S1, data preparation, implementation:

[0068] A, collect low contrast Raw images, i.e. Raw data;

[0069] B, convert single channel to RGB four channels; then proceed to steps S2 and S3 respectively;

[0070] S2, implementation: CLAHE enhancement;

[0071] S3, perform cropping; crop each channel of the original and enhanced four-channel data to a size of (256, 256), concatenate the cropping results at the corresponding positions to obtain (4, 256, 256), and obtain several cropping results for each original Raw data as training data; proceed to step S4.3;

[0072] S4, build an optimized Raw domain contrast enhancement UnetR model;

[0073] S4.1, obtain the UnetR network;

[0074] S4.2, prediction;

[0075] S4.3, calculate the training loss based on the results of S3 and S4.2;

[0076] S5, parameter update, using the optimizer Adam to update the network parameters. The Adam optimizer is an adaptive optimization algorithm that can adaptively adjust the learning rate; the fixed learning rate is 0.0001, and the training cycle is 100.

[0077] like Figure 2 Specifically, the steps are as follows:

[0078] Step S1: Data preparation. Collect 300 low-contrast original Raw images, and map the single-channel Raw data to the rggb four-channel, as shown in formula (1):

[0079] img_rggb=to_rggb(img_raw) (1)

[0080] Among them, to_rggb() represents the mapping function from single channel to four channels.

[0081] Step S2: High contrast Raw image generation. Perform CLAHE enhancement on each channel of img_rggb, set the local mapping window size M and the local mapping function slope limit threshold T, and obtain the local equalization result, as shown in formula (2). Then, concatenate the transformed results of the four channels, as shown in formula (3). Different M and T are set for different scenes, and the best effect is selected as gt.

[0082] img_clahe[i]=clahe(img_rggb[i], M, T) (2)

[0083] img_gt=stack(img_clahe[i], i=0, 1, 2, 3 (3)

[0084] Step S3: Construct a low-contrast-high-contrast training data set. The low-contrast and high-contrast Raw data rggb four-channel space is cropped with a cropping size of (4, 256, 256). After cropping, the low-contrast and high-contrast local positions correspond one to one to obtain each set of training data.

[0085] Step S4: Construct an optimized Raw domain contrast enhancement UnetR model.

[0086] Through post-cropping of low-contrast-high-contrast datasets and four-channel spatial features, the model structure is adjusted to adapt to Raw domain contrast enhancement. Figure 3 As shown in the figure, the specific modification in the UnetR algorithm is: the number of convolution channels of the first layer input and the last layer output is set to 4 channels, (4,256,256) is used as the input image scale of the encoder network, and the contrast loss of L1loss and SSIM is used as the training loss for learning. The specific steps of the network are:

[0087] The step S4.1 further comprises:

[0088] ① The encoder part of the optimized Raw domain contrast enhancement UnetR model includes a first-level encoder and a second-level encoder, which extract features of different scales from the input low-contrast Raw domain. Each encoder contains two layers of convolution processing and one layer of downsampling module. After feature extraction, the size is reduced and input to the next level encoder, as shown in equations (4) and (5):

[0089] E1(x)=poolMax(conv(conv(x))) (4)

[0090] E2(x)=poolMax(conv(conv(E1(x)))) (5)

[0091] ② The decoder of the optimized Raw domain contrast enhancement UnetR model includes a first-level decoder and a second-level decoder, which aggregates the feature information of different scales obtained by the encoder to obtain a feature map. Each decoder contains an upsampling operation, two layers of convolution processing, and a residual connection with the encoder, as shown in equations (6) and (7):

[0092] D1(x)=conv(conv(upsample(conv(E2(x))))) (6)

[0093] D2(x)=conv(conv(upsample(conv(D1(x))))) (7);

[0094] The step S4.2 further comprises:

[0095] ③ The last layer of the optimized Raw domain contrast enhancement UnetR model is the spatial information fusion layer, which includes pooling processing, two-layer convolution modules and spatial attention layer. The input is the output of the second-level decoder, and the maximum pooling and average pooling are performed on it to extract the key spatial features as shown in equations (8) and (9). The spatial features are convolved in two layers to obtain the spatial feature attention and dot product with the input to obtain the feature map of fused spatial information, as shown in equation (10):

[0096] M=poolMax(conv(conv(D2(x)))) (8)

[0097] A=poolAvg(conv(conv(D2(x)))) (9)

[0098]

[0099] The step S4.3 further comprises:

[0100] ④ The optimized Raw domain contrast enhancement UnetR model uses L1loss, as shown in formula (11), and SSIM contrast loss, as shown in formula (12), as the final loss function of the network:

[0101] L1(x, y) = |x - UnetR(x)| (11)

[0102]

[0103] Loss(x,y)=1-C(x,y)+L1(x,y) (13)

[0104] Where x represents the predicted output of the network, y represents the high-contrast Raw image corresponding to the low-contrast Raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for Raw domain contrast enhancement based on an optimized Unet network, characterized in that: The method comprises the following steps: S1, data preparation, implementation: A, collect low contrast Raw images, i.e. Raw data; B, convert single channel to RGB four channels; then proceed to steps S2 and S3 respectively; S2, implementation: CLAHE enhancement; S3, perform cropping; crop each channel of the original data and the enhanced four-channel data to a size of (256, 256), concatenate the cropping results at the corresponding positions to obtain (4, 256, 256), and obtain several cropping results for each original Raw data as training data; proceed to step S4.3; S4, build an optimized Raw domain contrast enhancement UnetR model; S4.1, obtain the UnetR network; S4.2, prediction; S4.3, calculate the training loss based on the results of S3 and S4.2; S5, parameter update.

2. According to claim 1, a method for Raw domain contrast enhancement based on an optimized Unet network is characterized in that: The method further comprises: S1: Data preparation: Collect N low-contrast original Raw images, and map the single-channel Raw data to the rggb four-channel, as shown in formula (1): img_rggb=to_rggb(img_raw) (1) Among them, to_rggb() represents the mapping function from single channel to four channels; S2: High contrast Raw image generation: Perform CLAHE enhancement on each channel of img_rggb, set the local mapping window size M and the local mapping function slope limit threshold T, and obtain the local equalization result, as shown in formula (2). Then, concatenate the transformed results of the four channels, as shown in formula (3). Different M and T are set for different scenes, and the best effect is selected as gt. img clahe[i]=clahe(img_rggb[i], M, T) (2) img_gt=stack(img_clahe[i]), i=0, 1, 2, 3 (3) S3: Construct a low-contrast-high-contrast training data set: crop the low-contrast and high-contrast Raw data rggb four-channel space with a cropping size of (4,256,256). After cropping, the low-contrast and high-contrast local positions correspond one to one to obtain each set of training data; S4: Building an optimized Raw domain contrast enhancement UnetR model: The model structure is adjusted through post-cropping of low-contrast-high-contrast datasets and four-channel spatial features to adapt to Raw domain contrast enhancement; the specific modifications in the UnetR network are as follows: the number of convolution channels of the first layer input and the last layer output is set to 4 channels, (4,256,256) is used as the input image scale of the encoder network, and the contrast loss of L1loss and SSIM is used as the training loss for learning; S5: The network parameters are updated using the Adam optimizer, which is an adaptive optimization algorithm that can adaptively adjust the learning rate. The learning rate is fixed to 0.0001 and the training cycle is 100.

3. According to claim 1, a method for Raw domain contrast enhancement based on an optimized Unet network is characterized in that: In step S4, the specific steps of the network are: The step S4.1 further comprises: ① The optimized Raw domain contrast enhancement UnetR model encoder includes a primary encoder and a secondary encoder, which extracts features of different scales from the input low-contrast Raw domain; Each encoder contains two layers of convolution processing and one layer of downsampling module. After feature extraction, the size is reduced and input to the next level encoder, as shown in equations (4) and (5): E1(x)=poolMax(conv(conv(x))) (4) E2(x)=poolMax(conv(conv(E1(x)))) (5) ② The decoder of the optimized Raw domain contrast enhancement UnetR model includes a primary decoder and a secondary decoder, which aggregates the feature information of different scales obtained by the encoder to obtain a feature map; Each decoder includes upsampling operation, two layers of convolution processing and residual connection with the encoder, as shown in equations (6) and (7): D1(x)=conv(conv(upsample(conv(E2(x))))) (6) D2(x)=conv(conv(upsample(conv(D1(x))))) (7) That is, step S4.1 corresponds to steps ① and ②; The step S4.2 further comprises: ③The last layer of the optimized Raw domain contrast enhancement UnetR model is the spatial information fusion layer, which includes pooling processing, two convolution modules and a spatial attention layer; The input is the output of the second-level decoder, and the maximum pooling and average pooling are performed on it to extract the key spatial features as shown in equations (8) and (9). The spatial features are convolved in two layers to obtain the spatial feature attention and dot product with the input to obtain the feature map of fused spatial information, as shown in equation (10): M=poolMax(conv(conv(D2(x)))) (8) A=poolAvg(conv(conv(D2(x)))) (9) That is, step S4.2 corresponds to step ③, and O in formula (10) represents the predicted result; The step S4.3 further comprises: ④ The optimized Raw domain contrast enhancement UnetR model uses L1loss, as shown in formula (11), and SSIM contrast loss, as shown in formula (12), as the final loss function of the network: L1(x, y) = |x - UnetR(x)| (11) Loss(x,y)=1-C(x,y)+L1(x,y) (13) Where x represents the predicted output of the network, y represents the high-contrast Raw image corresponding to the low-contrast Raw image, C() represents the SSIM contrast calculation function, and μ x With μ y are the average values ​​of x and y, σ x With σ y are the standard deviations of x and y respectively; The calculation of training loss in step S4.3 corresponds to step ④.

4. According to claim 1, a method for Raw domain contrast enhancement based on an optimized Unet network is characterized in that: In the step S1, N=300.

Images