An unsupervised automatic correction method for image exposure

Abstract

An unsupervised automatic correction algorithm for image exposure includes the following steps: (1) taking the brightness channel of the image to be corrected as the input of the optimal S-curve estimation network; (2) the network is unsupervisedly trained based on the input image to obtain the optimal S-curve of the input image and the corrected image brightness channel; (3) the image color channel is proportionally corrected and merged with the brightness channel to obtain the corrected image. This method performs targeted training on the test image, which can better overcome the problem of weak generalization ability of the existing correction method, meet the requirements of practical applications for image exposure correction, and perform exposure correction on real images collected in the field without reference.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to correction of image exposure. Background Art

[0002] Exposure is the total amount of light allowed to fall on the photographic medium during the photographic process, but improper exposure often reduces the quality of the captured image under poor lighting conditions such as backlighting. Although most modern imaging sensors can automatically adjust the relevant hardware parameters according to the lighting conditions of the target, satisfactory results cannot be obtained under most backlighting conditions. In order to restore the existing low-quality backlit images, users can manually adjust the image's tone curve to adjust the shadow area, midtones, and highlight areas of the image. However, since the optimal adjustment curve for each image is different, users cannot manually batch process exposure-distorted images. Therefore, a method for automatically correcting image exposure is needed in the industrial field. For example, for monitoring systems, when the lighting conditions are poor, the visibility of dark details in the captured images or videos is low. In order to restore the details of the image, it is very necessary to automatically correct the exposure.

[0003] Traditional image enhancement methods such as histogram equalization and image enhancement methods based on Retinex theory, when used to correct image exposure, are not only not targeted, but also easily cause halo effects or destroy the smoothness of image tones.

[0004] At present, the existing explorations for image exposure correction include a series of heuristic algorithms and machine learning-based algorithms. Representative heuristic algorithms include: Tsai and Yeh in "Contrast compensation by fuzzy classification and image illumination analysis for back-lit and front-lit color face images" use the method of segmenting the image highlight area and the backlight area to adjust the image, but the region segmentation results in the above method are unreliable; Yuan and Sun in "Automatic exposure correction of consumer photographs" model image exposure correction as an undirected graph labeling problem, and obtain the optimal solution through brute force, which is time-consuming and laborious. Representative algorithms in the field of machine learning include: Dale et al. established a million-level image dataset in "Image restoration using online photo collections", in which they searched for the image closest to the test image as a correction reference, which required a lot of storage space; Kang et al. established a dataset through manual interactive correction in "Personalization of image enhancement", and corrected the test image by matching the closest image and using its correction parameters, but the dataset size was too small, resulting in poor robustness to the test image; Li and Wu divided the highlight area and the backlight area through semantic segmentation in "Learning-based restoration of backlit images" and then adjusted them separately, but if the segmentation accuracy cannot be guaranteed, it may directly lead to poor correction effect. The accuracy of the above detection methods is highly dependent on low-level visual features and the size of the dataset, and is easily affected by changes in image content. In other words, the difficulty in the current research on automatic correction methods for image exposure lies in designing an algorithm with scene consistency. Summary of the invention

[0005] The purpose of the present invention is to provide an unsupervised automatic correction method for image exposure based on a neural network, which solves the shortcomings of the exposure correction algorithm in the field of machine learning that it depends on the size of the data set and is easily affected by the image content, and meets the requirements of practical applications for automatic correction of image exposure.

[0006] To achieve the above object, the solution of the present invention is:

[0007] An unsupervised automatic correction method for image exposure comprises the following steps:

[0008] (1) The brightness channel of the exposure-distorted image is used as the input of the optimal S-curve estimation network ExCNet (Exposure Correction Network);

[0009] (2) The network performs unsupervised training based on the input image to obtain the optimal S-curve of the input image and the corrected image brightness channel;

[0010] (3) The color channel of the image is proportionally corrected and merged with the brightness channel to obtain the corrected image.

[0011] Further, step (1) includes the following steps:

[0012] (1-1) The image to be corrected is separated into YIQ channels to obtain brightness channel and color channel. The brightness channel refers to the Y component, which represents the brightness information of the image; the color channel refers to the I component and Q component, which carry the color information of the image and describe the properties of the image color and saturation;

[0013] (1-2) The brightness channel is used as the input of the optimal S-curve estimation network ExCNet.

[0014] Step (2) includes the following steps:

[0015] (2-1) ExCNet performs a series of convolution and pooling operations on the image to calculate the two parameters φ of the S curve that control the degree of brightness adjustment of the shadow area and the highlight area respectively. s and φ h ,φ s is the shadow area adjustment magnitude, φ h ExCNet uses a convolutional neural network to estimate the S-curve parameters: it performs convolution and pooling operations on the input image to extract the image tone information, and finally changes the number of units in the last layer of the fully connected layer so that the output is two values.

[0016] The two parameters of the S-curve above are φs and φ h It is the result of parameterizing the S-curve so that the shape of the S-curve can be estimated through a neural network. The mathematical expression of the parameterized S-curve is:

[0017] f(x:φs,φ h )=x+φs×f Δ (x)-φ h ×f Δ (1-x)

[0018] Where x is the input brightness value, f(x:φ s ,φ h) is the output brightness value.

[0019] In the above formula Its parameters are set as: 1 =5, k 2 =14, k 3 =1.6;

[0020] (2-2) ExCNet adjusts the input image according to the calculated S-curve to obtain the correction result of the current model;

[0021] (2-3) Calculate the loss of the adjusted image and update the ExCNet weights. The loss consists of two parts:

[0022] (a) Image content visibility

[0023] (b) Contrast change before and after correction

[0024] The model loss function is:

[0025]

[0026] Where E i is the single region visibility loss term, and its value is negatively correlated with the corrected single region content visibility. ij is the change in contrast between two regions before and after correction, Ω(i) represents the region adjacent to a certain region, and λ is a parameter for adjusting the weights of the above two effects (a) and (b).

[0027] (2-3-1) In order to make every pixel in the image bright and clear enough and improve the visibility of the image content, the Ei in the loss function is negatively correlated with the visibility of the corrected single area content. The mathematical expression is as follows:

[0028]

[0029] Among them l i and are the average brightness values ​​of the region before and after correction, respectively. Minimize E i will make Try to get as close to 0.5 as possible, so both underexposed and overexposed areas are adjusted toward a well-exposed brightness value.

[0030] In minimizing E i The process will keep the brightness value unchanged from 0.5 before and after the update. That is, if l i Greater than 0.5, It will also be slightly larger than 0.5. This property ensures that brighter areas in the original image remain brighter in the corrected image, and vice versa.

[0031] (2-3-2) In order to keep the contrast between adjacent areas in the corrected image as close as possible to the original Figure 1 E in the loss function ij is the change in contrast between two areas before and after correction. The mathematical expression is as follows:

[0032]

[0033] Among them l j and l i represents the average brightness of two adjacent areas, and represents the average brightness of the two regions after correction. Minimize E ij This will keep the brightness difference between adjacent areas as constant as possible.

[0034] (2-3-3) After calculating the loss function, the error back propagation algorithm is used to update the model weights.

[0035] (2-3-4) Repeat steps (2-1)-(2-3-3) until the loss function converges to a stable value. Experiments show that most images can converge within 200 iterations. After convergence, the image adjusted in step (2-2) is the brightness channel of the image with corrected exposure.

[0036] Step (3) comprises the following steps:

[0037] (3-1) The correction ratio is calculated based on the image brightness channels before and after correction, that is, the correction parameter matrix is ​​obtained by dividing the brightness channel after correction by the brightness channel before correction.

[0038] (3-2) Multiply the two color channels of the image to be corrected by the correction parameter matrix to obtain the corrected image color channels.

[0039] (3-3) The corrected brightness channel and color channel are merged into a corrected YIQ channel image, which is then converted into an RGB image to obtain a corrected image.

[0040] Through the coordination of the above three steps, an unsupervised automatic correction method for image exposure based on a neural network is shown in the present invention. This method adopts a neural network as a regression means and uses an unsupervised learning method, which better overcomes the problem that the existing correction method is easily affected by the image content and is suitable for various image scenes. The accuracy and efficiency of this method have also been confirmed in real image data experiments, and can meet the requirements of practical applications for automatic correction methods for image exposure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall correction flow chart of the present invention.

[0042] Figure 2 This is the network structure diagram of the optimal S-curve estimation network ExCNet.

[0043] Figure 3 It is a schematic diagram of an S-curve used for adjusting an image in the present invention.

[0044] Figure 4 These are the four convolution kernels used to calculate the loss function in the present invention.

[0045] Figure 5 It is a comparison chart showing an example of the effect of using the present invention to correct the exposure of an image.

[0046] Figure 6 It is a bar chart of user preferences in Experiment 1 of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation details of the present invention are further described below in conjunction with the accompanying drawings:

[0048] The solution of the unsupervised automatic correction method of image exposure based on neural network shown in the present invention is:

[0049] (1) The brightness channel of the exposure-distorted image is used as the input of the optimal S-curve estimation network;

[0050] (2) The network performs unsupervised training based on the input image to obtain the optimal S-curve of the input image and the corrected image brightness channel;

[0051] (3) The color channel of the image is proportionally corrected and merged with the brightness channel to obtain the corrected image.

[0052] The following is a flowchart of the overall correction of the present invention. Figure 1 Describe each step in detail:

[0053] 1. Channel separation

[0054] The image to be corrected needs to be separated into YIQ channels to obtain brightness channel and color channel. The brightness channel refers to the Y component, which represents the brightness information of the image; the color channel refers to the I component and Q component, which carry the color information of the image and describe the properties of the image color and saturation; the brightness channel is used as the input of the optimal S-curve estimation network ExCNet.

[0055] The following is the conversion equation from RGB image to YIQ component:

[0056]

[0057] (II) Optimal S-curve estimation network ExCNet estimates the optimal S-curve of the input image and corrects the brightness channel

[0058] ExCNet performs unsupervised training based on the input image, estimates its optimal S-curve and obtains the corrected brightness channel image. The network structure of ExCNet in the experiment is shown in Table 1:

[0059] Table 1 ExCNet network structure

[0060]

[0061]

[0062] The specific steps include:

[0063] (1) Input: The brightness channel of the image to be corrected is uniformly resized to 128×128 as the input I of ExCNet l ;

[0064] (2) Use the estimated S-curve to correct the input image:

[0065] (2-1) Convolution + Pooling: ExCNet performs a series of convolution and pooling operations on the image to extract information such as the image’s hue. 5 ;

[0066] (2-2) Full connection: Use the fully connected layer to convolve the image information extracted by convolution 5 The two parameters converted into S-curves are shaded area adjustment magnitude φ s and highlight area adjustment magnitude φ # ;

[0067] (2-3) Use S-curve parameterized mapping:

[0068] f(x:φ s ,φ h )=x+φ s ×f Δ (x)-φ h ×f Δ (1-x)

[0069] For the input image I l Correction is performed to obtain the corrected image Where x is the input brightness value I l ,f(x:φ s ,φ h ) is the output brightness value, in the above formula Its parameters are set as: 1 =5, k 2 =14, k 3=1.6;

[0070] (3) Calculate the loss function:

[0071] (3-1) Average pooling: Use the average pooling layer to parallelly calculate the image I before correction l And the corrected image The average brightness of each area block I lb , The block size is set to 4×4, so the pooling result size of the 128×128 image is 32×32, corresponding to the brightness mean of each of the 32×32 regions;

[0072] (3-2) Convolutional layer consisting of four non-trainable convolution kernels: using Figure 4 The four convolution kernels shown can calculate the image I before correction respectively. lb And the corrected image The average brightness difference between each area block and the adjacent areas on the left, right, top, and bottom Abbreviated as

[0073] (3-3) Calculate the loss term E that is negatively correlated with the regional contrast data :

[0074] According to the definition of the negative correlation of visibility of a single area in the loss function definition:

[0075]

[0076] It can be seen that the average brightness I of each area before and after correction calculated in (3-1) is lb , Substitute l in the above formula i and Then summing all elements of the result matrix can obtain the loss term E which is negatively correlated with the contrast of the entire image area. data .

[0077] (3-4) Calculate the loss term E of the difference between the contrast before and after correction between regions smoot# :

[0078] According to the definition of the difference between the contrast before and after correction between regions in the loss function definition:

[0079]

[0080] It can be seen that the image before correction I calculated in (3-2) lb And the corrected image The average brightness difference between each area block and the adjacent areas on the left, right, top, and bottom down}} replace (l j -l i )and Then summing all elements of the result matrix can obtain the loss term E which is the difference between the contrast before and after correction of the entire image area. smoot# .

[0081] (3-5) Calculate the total loss:

[0082]

[0083] In the experiment, it is set to λ=12;

[0084] (4) After calculating the loss function, the error back propagation algorithm is used to update the model weights and learning rate;

[0085] (5) Repeat steps (2-4) 200 times. The result obtained in step (2) is the estimated optimal S curve.

[0086] (III) Using the estimated optimal S-curve, correct the brightness channel of the image to be corrected:

[0087] Parameterize the S-curve:

[0088] f(x:φ s ,φ h )=x+φ s ×f Δ (x)-φ h ×f Δ (1-x)

[0089] Correct the brightness channel of the image to be corrected to obtain the brightness channel of the corrected image, where x is the input brightness value, f(x:φ s ,φ # ) is the output brightness value, in the above formula The parameter k 1 =5, k 2 =14, k 3 =1.6;

[0090] (iv) dividing the brightness channel after correction by the brightness channel before correction to obtain a correction parameter matrix representing the correction ratio of each pixel;

[0091] (5) performing proportional correction on the color channels of the image to be corrected, i.e., the I component and the Q component, using a correction parameter matrix;

[0092] (VI) Combine the corrected brightness channel and color channel and convert them into RGB image by the following conversion equation:

[0093]

[0094] The converted RGB image is the image after exposure correction.

[0095] The present invention is further described below through specific experiments:

[0096] Experimental data set and comparison method: This experiment was conducted on the image exposure data set, using 1512 photos taken in different scenes and lighting conditions. These photos were roughly divided into 3 groups, with about 500 photos in each group: Group A has severe pathological exposure, Group B has slightly pathological exposure, and Group C has good exposure. This experiment compares the proposed ExCNet with 8 other representative automatic exposure correction methods, including: [1] "JC Russ. The Image Processing Handbook. CRC Press, Inc., 7th edition, 2015.", that is, "histogram equalization", [2] "K. Zuiderveld. Contrast limited adaptive histogram equalization. In Graphics Gems IV, pages 474-485. Academic Press Professional, Inc., 1994.", that is, "contrast-preserving adaptive histogram equalization", [3] "DJ Jobson, Z. Rahman, and GAWoodell. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Processing, 6(7):965–976, 1997”, that is, “A multiscale retina,method for bridging the gap between color images and human observation of scenes”, [4] “Google.Picasa.http: / / picasa.google.com / .”, that is, “Google’s automatic contrast adjustment algorithm”, [5] “Z.Farbman,R.Fattal,D.Lischinski,and R.Szeliski.Edge-preserving decompositions for multi-scaletone and detail manipulation.ACM Trans.Graph.,27(3):67, 2008.”, that is, “edge-preserving decomposition for multi-scale tone and detail manipulation”, [6] “S.Paris,SWHasinoff,and J.Kautz.Local Laplacianfilters:Edge-aware image processing with a laplacian pyramid.926ACMTrans.Graph.,30(4):68, 2011.” That is, “Edge-aware image processing using Laplacian pyramid”, [7] “L.Yuan and J.Sun.Automatic exposure correction of consumer photographs.InEuropean Conference on Computer Vision,pages 771–785,2012.” That is, “Automatic exposure correction of photos”, [8] “Z.Li and X.Wu.Learning-based restoration of backlit images.IEEE Trans.Image Processing,27(2):976–986,2018.” That is, “Learning-based restoration of backlit images”.

[0097] Experiment 1 is a user preference comparison experiment. The correction results of the present invention are randomly mixed with the correction results of the other 8 methods in pairs, that is, each group of pictures has 2 correction results. Ten evaluators (6 males and 4 females) choose their preferences. The options include "the correction effect on the left is better", "the correction effect on the right is better", and "no obvious preference". Figure 6 The experimental results shown in the figure show that users have a clear preference for the ExCNet proposed in the present invention. In the experiments of Group A and Group B, ExCNet has a good correction effect on both extremely poorly exposed and slightly poorly exposed pictures; in the experiments of Group C, ExCNet does not make the visual effect of pictures with normal exposure levels worse.

[0098] Experiment 2 is a visual quality comparison. The correction results obtained by using ExCNet and other 8 methods for the same image are visually compared. It is concluded that the other 8 methods have color difference, no ability to repair underexposed areas, side effects on normally exposed images, reduced contrast of original images, and introduced halo effects. The ExCNet proposed in this invention has high robustness to input images. Whether it is an image with extremely poor exposure level or a normal image, it can ensure high visual quality of the output image, which meets the requirements of practical applications.

[0099] Experiment 3 is an objective indicator experiment, which uses two objective indicators to compare different methods: contrast distortion image quality assessment CDIQA and brightness ordinal distortion LOD.

[0100] The experimental results are shown in Table 2. ExCNet has the highest CDIQA index value, indicating that ExCNet has the strongest ability to restore details, and the LOD value is low, indicating that the difference between the correction result of ExCNet and the original image is small, that is, the fidelity is good. The above properties of ExCNet show that the present invention can not only perform significant exposure correction on the image, but also pay attention to the fidelity of image enhancement, and there will be no over-correction phenomenon. Therefore, the visual quality of the corrected image is the best among the 9 compared methods.

[0101] Table 2 Objective index experimental results

[0102]

[0103] The above description of the embodiments is to facilitate the understanding and use of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. An unsupervised automatic correction method for image exposure, characterized in that: The following steps are involved: (1) The YIQ channel of the image to be corrected is separated to obtain the brightness channel and the color channel, and the brightness channel is used as the input of the optimal S-curve estimation network ExCNet; (2) ExCNet performs a series of convolution and pooling operations on the image to calculate the two parameters φ of the S-curve that control the degree of brightness adjustment of the shadow area and the highlight area respectively. s and φ h ; (3) ExCNet adjusts the input image according to the calculated S-curve; (4) Calculate the loss of the adjusted image and use back propagation to update the ExCNet weights; (5) repeating steps (2) to (4) a certain number of times until the loss function converges to a stable value, and after convergence, the image adjusted in step (3) is the brightness channel of the image with the exposure corrected; (6) calculating a correction ratio based on the brightness channels of the image before and after correction, and correcting the color channels of the image to be corrected; (7) merging the corrected brightness channel and color channel to obtain a corrected image; Among them, the loss in step (4) consists of two parts: (4-1) Image content visibility; (4-2) Contrast change before and after correction; The model loss function is: Where E i is the single region content visibility loss term, and its value is negatively correlated with the corrected single region content visibility. ij is the change in contrast between two regions before and after correction, Ω(i) represents the region adjacent to a certain region, and λ is the parameter for adjusting the weights of the above two effects (4-1) and (4-2); In order to make each pixel in the image bright and clear enough and improve the visibility of the image content, E in step (4) i It is negatively correlated with the corrected visibility of the content in a single region, and the mathematical expression is as follows: Among them l i and are the average brightness values ​​of the region before and after correction respectively; minimize E i will make As close to 0.5 as possible, so both underexposed and overexposed areas are adjusted towards a well-exposed brightness value; In minimizing E i The process will keep the brightness value unchanged from 0.5 before and after the update. That is, if l i Greater than 0.5, It will also be greater than 0.5; this property ensures that the brighter areas in the original image remain brighter in the corrected image, and vice versa; In order to keep the contrast between adjacent regions in the corrected image as consistent as possible with the original image, E ij is the change in contrast between two areas before and after correction. The mathematical expression is as follows: Among them l j and l i represents the average brightness of two adjacent areas, and represents the average brightness of the two regions after correction; minimize E i This will keep the brightness difference between adjacent areas as constant as possible.

2. The unsupervised automatic correction method for image exposure according to claim 1, characterized in that: The brightness channel in step (1) refers to the Y component, which represents the brightness information of the image, and the color channel refers to the I component and the Q component, which carry the color information of the image and describe the properties of the image color and saturation.

3. The unsupervised automatic correction method for image exposure according to claim 1, characterized in that: Use S-shaped nonlinear curve to adjust the image quality of shadow area and highlight area of ​​the image, and map the original inappropriate image exposure level to a better exposure level; Parameterize the S-curve into two parameters φ in step (2) s and φ h , so that it can be estimated through a neural network; the mathematical expression of the S-curve parameterization is: f(x:φ s ,f h )=x+φ s ×f Δ (x)-φ h ×f Δ (1-x) Where x is the input brightness value, f(x:φ s ,φ h ) is the output brightness value; In the above formula The parameters are set as follows: k1=5, k2=14, k3=1.

6.

4. The unsupervised automatic correction method for image exposure according to claim 1, characterized in that: ExCNet uses a convolutional neural network to estimate the S-curve parameters: convolution and pooling operations are performed on the input image to extract the image tone information, and finally the output is two values ​​by modifying the number of units in the fully connected layer.

5. The unsupervised automatic correction method for image exposure according to claim 1, characterized in that: The average brightness of the area before and after the required correction l i and It is calculated as follows: Perform 4×4 average pooling operations on the input image of ExCNet and the image corrected by the S curve, respectively, and obtain that each pixel value in the two new feature maps represents the average brightness of a 4×4 area; calculate The term only needs to subtract the two feature maps obtained above.

6. The unsupervised automatic correction method for image exposure according to claim 5, characterized in that: The difference in average brightness between adjacent areas before and after the correction is calculated as follows: Based on the average brightness of each area that has been calculated, it is only necessary to calculate the difference between adjacent areas. Here, four different convolution kernels are used to perform convolution operations on the feature map representing the average brightness of each area to achieve this; The four convolution kernels correspond to the adjacent areas in the upper, lower, left and right directions respectively. The size of the convolution kernel used to calculate the adjacent area in the upper direction is 3×1, and the value is (-1,1,0), which means subtracting the brightness value of the upper area from the brightness value of the middle area; the size of the convolution kernel used to calculate the adjacent area in the lower direction is 3×1, and the value is (0,1,-1), which means subtracting the brightness value of the lower area from the brightness value of the middle area; the size of the convolution kernel used to calculate the adjacent area in the left direction is 1×3, and the value is (-1,1,0), which means subtracting the brightness value of the left area from the brightness value of the middle area; the size of the convolution kernel used to calculate the adjacent area in the right direction is 1×3, and the value is (0,1,-1), which means subtracting the brightness value of the right area from the brightness value of the middle area.

7. The image exposure correction method according to claim 1, characterized in that: In step (6), the correction ratio is calculated based on the brightness channels of the image before and after correction. The correction parameter matrix is ​​obtained by dividing the brightness channel after correction by the brightness channel before correction. This matrix is ​​then multiplied by the two color channels of the image to be corrected to obtain the color channels of the corrected image.

8. The image exposure correction method according to claim 1, characterized in that: In step (7), the corrected brightness channel and color channel are merged into a corrected YIQ channel image, which is then converted into an RGB image to obtain a corrected image.

Images