Camera-aware characteristic-guided real exposure correction method and system

Abstract

The application discloses a camera perception characteristic guided real exposure correction method and system, and belongs to the technical field of computer photography and image processing. The application is used for exposure correction of real data, a channel guided convolutional neural network is designed according to camera perception characteristics, and a real paired data acquisition system is built. Meanwhile, a knowledge distillation strategy used for exposure correction is designed, a real paired exposure correction data acquisition system is proposed, a data set is acquired, an exposure correction neural network is trained by using the acquired data set, real image exposure correction can be completed with high quality, and the quality of real image exposure correction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a camera-aware characteristic guided real-world exposure correction method and system, belonging to the technical field of computer photography and image processing. BACKGROUND

[0002] Exposure affects the intensity and quality of the captured image. Modern digital cameras can automatically control exposure through automatic exposure mode, or manually control exposure by users to obtain images of appropriate quality. However, there are some important factors that cause exposure errors, including errors measured through the lens, user manual errors, brightness changes and difficult lighting conditions. Exposure errors occur in the early stage of image capture, which will greatly reduce the contrast and visual quality of the final image.

[0003] In order to solve the above problems, many model-driven exposure correction techniques have been proposed by technical personnel. For example, histogram-based methods, Retinex theory-based methods, etc. In recent years, data-driven methods using deep learning technology have achieved good performance, and such methods learn the non-linear mapping function between the error exposure and the good exposure image. However, most methods only deal with overexposure or underexposure problems, which limits the practical application of correcting various exposures. More and more researches have begun to focus on solving the exposure correction problem in an end-to-end network. However, these methods either ignore the requirement of learning different prior knowledge, or only design a simple module to implicitly learn various prior knowledge for overexposure and underexposure correction. Since the correction process of overexposure and underexposure is obviously different, correcting overexposure and underexposure is different. Therefore, it is a challenge to correct these two kinds of unreasonable exposure with one network. At present, most of the digital camera sensors are designed to have higher sensitivity to the green channel than to the red and blue channels, because the human eye is more sensitive to green than to other colors. This camera-aware characteristic means that the red and blue channels are more advantageous for overexposure correction, and the green channel is more advantageous for underexposure correction.

[0004] In addition, most of the existing methods perform exposure correction on 8-bit standard color images (sRGB), which are generated from RAW data by a highly nonlinear operation, i.e. camera image signal processing (ISP). Due to the compression and quantization operations of ISP, sRGB has information loss, which leads to a potential bottleneck for exposure correction. In fact, previous studies have shown that RAW data contains more abundant information, higher bit values, and maintains a linear relationship between exposure and scene brightness, which is beneficial for image restoration. However, there is currently a lack of real paired RAW domain exposure correction datasets. SUMMARY

[0005] The purpose of the present application is to create a camera perception characteristic guided real exposure correction method and system for real data exposure correction, design a channel guided convolutional neural network according to the camera perception characteristic, build a real paired data acquisition system, construct a high-quality real paired exposure correction dataset, train an accurate channel characteristic guided exposure correction network, and complete real image exposure correction with high quality and improve the imaging quality.

[0006] The present application analyzes the perception characteristics of different channels of the camera, designs a channel guided exposure correction network according to the different perception characteristics of different channels, and further designs a knowledge distillation strategy for exposure correction, proposes a real paired exposure correction data acquisition system and acquires a dataset, trains an exposure correction neural network using the acquired dataset, realizes high-precision image exposure correction, and improves the quality of real image exposure correction.

[0007] Advantages

[0008] Compared with the prior art, the present application has the following advantages:

[0009] 1. The present application builds a real data acquisition system, which is used to acquire real paired over / underexposed image and normally exposed image datasets. The data acquisition system can efficiently acquire real paired exposure correction datasets, and the acquired datasets can effectively improve the accuracy of the exposure correction method on real images.

[0010] 2. The present application designs a channel guided exposure correction network according to the perception characteristics of different color channels of the camera, adopts a knowledge distillation training strategy, can adaptively excavate the prior knowledge required for different exposure correction tasks, adaptively generate modulation parameters, and effectively improve the modeling ability of the exposure correction network for overexposure and underexposure correction tasks.

[0011] 3. The present application uses a convolutional neural network to learn the mapping relationship between over / underexposed images and normally exposed images, combines beneficial effects 1 and 2 to improve the generalization of the convolutional neural network, and can improve the quality of real image exposure correction and ensure the fidelity of real exposure correction images. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is the general flowchart of the method of the present application;

[0013] Figure 2 is the composition schematic diagram of the system of the present application;

[0014] Figure 3 is the schematic diagram of different color channels of the camera having different perception characteristics;

[0015] Figure 4 is a schematic diagram of a channel-guided exposure correction network in the present application;

[0016] Figure 5 is a schematic diagram of exposure correction knowledge distillation in the present application. DETAILED DESCRIPTION

[0017] The present application will be further described below with reference to the accompanying drawings.

[0018] As Figure 1 shown, in one aspect, the present application proposes a camera perception characteristic guided real exposure correction method, comprising the following steps:

[0019] Step 101: Build a real data acquisition system, and use the system to acquire a real paired exposure correction dataset. The real paired exposure correction dataset contains paired over / underexposed images and normally exposed images.

[0020] In order to support exposure correction on RAW images, the error exposure and correct exposure RAW images are acquired in pairs to obtain a real RAW exposure correction dataset.

[0021] Specifically, the camera can be fixed on a tripod to constitute a real data acquisition system. When shooting, the system first adjusts the aperture, focal length, exposure time, etc. to acquire high-quality normally exposed images. Then, using the control software, the exposure time is controlled at a ratio of 1 / 100 to 10. Finally, underexposed or overexposed images are acquired. Through the above process, a real paired exposure correction dataset can be acquired.

[0022] Step 102: Analyze the camera perception characteristics, and design a channel-guided exposure correction network to learn different prior knowledge required for underexposure and overexposure in the same network.

[0023] The camera perception characteristic is that the green channel has higher sensitivity than the red channel and the blue channel, as Figure 3 shown. This means that the red channel and the blue channel are more advantageous for overexposure correction and the green channel is more advantageous for underexposure correction. According to the camera perception characteristic, the present application designs a channel-guided exposure correction network. The overall structure of the network includes a main branch, a color channel guided branch and a guided channel selection module, as Figure 4 shown.

[0024] To make full use of the useful information in the input RAW image, the RAW image is decomposed into four channels of RGGB (Red Green Green Blue). The red and blue channels are more informative for overexposure correction, while the two green channels are more informative for underexposure correction. Since different guide channels are helpful for different exposure corrections, a guide channel selection module is first used to automatically select the required color channels for overexposure or underexposure correction. Then, the selected color channels are input into the color channel guide branch to generate guide features. Finally, the guide features are used to modulate the corresponding features in the main branch. In this way, the guide information can be automatically selected and effectively used for overexposure and underexposure correction, respectively.

[0025] Further, the main branch includes 4 encoder stages and 4 corresponding decoder stages. At the end of each encoder stage, a 4x4 kernel size and 2-step convolution downsample the feature map to 1 / 2 scale. Before each decoder stage, the features are upsampled to 2 scales using bilinear interpolation. The shallow feature maps are passed from each encoder stage to the corresponding decoder stage using skip connections. Further, to simplify training, residual learning is introduced in the exposure correction network, and residual blocks are used as basic blocks to construct the encoder and decoder, where the residual block consists of two 3x3 convolutions, followed by an activation function and a 1x1 convolution.

[0026] To automatically select the required channels for overexposure and underexposure correction, a guide channel selection module is designed. First, the four-channel RAW image is input into a global average pooling (GAP) layer to obtain an average vector. Subsequently, to further obtain the weight information of different channels, two fully connected layers (FC) are used to learn the non-linear relationship between each channel. Then, the Sigmoid gating mechanism is used to limit the final attention weight of each channel, and the two channels corresponding to the maximum value are selected as the selected channels.

[0027] As mentioned above, when overexposed, the red and blue channels are easier to recover, while when underexposed, the green channel is easier to recover. Inspired by this, when training the network for overexposure and underexposure correction, the guide channel selection module is constrained to select specific channels for the corresponding correction task, which is specifically represented as:

[0028] L s ||I s -I d ||1 (1)

[0029] where L s represents the channel selection loss, I s and I d represent the selected channel and the target channel, respectively, i.e., the blue and red channels for overexposure correction, and the green channel for underexposure correction.

[0030] To make the main branch feature adapt to different exposure corrections, the present application designs a color channel guided module. Inspired by guided filtering, the color channel guided module modulates the feature by guidance information. First, the selected color channel is input into the color channel guided branch to generate a guided feature. Then, the main branch feature is modulated by the guided feature with corresponding spatial resolution.

[0031] Specifically, the pixel-level scaling value and bias value for enhancing the main branch feature F g can be generated from the guided feature F m , denoted as:

[0032]

[0033] wherein, is the enhanced feature, and α(F g ) and β(F g ) are two learnable modulation parameters.

[0034] So far, the design of the channel-guided exposure correction network is completed, which can learn different prior knowledge required for underexposure and overexposure in the same network.

[0035] Step 103: Use knowledge distillation to make the channel-guided exposure correction network learn the prior knowledge required for overexposure and underexposure correction more effectively, as shown in Figure 5 .

[0036] wherein, knowledge distillation is to distill the prior knowledge required for overexposure and underexposure correction to the channel-guided exposure correction network. Before using knowledge distillation, first train two networks respectively for overexposure and underexposure correction, and the target loss function L of the two networks is denoted as:

[0037] L=L+L r (3)

[0038] wherein, L s and L r are the guided channel selection loss and the reconstruction loss, respectively.

[0039] Further, the reconstruction loss L r is denoted as:

[0040] L r =||O-T||1 (4)

[0041] wherein, O and T represent the output of the exposure correction network and the corresponding target image, respectively.

[0042] When the overexposure and underexposure correction network is well trained, the knowledge of overexposure and underexposure correction is extracted into a unified network by using the method of knowledge distillation.

[0043] In the aspect of knowledge distillation, the two well-trained overexposure and underexposure correction networks are teacher networks, and the unified overexposure and overexposure correction network is a student network. For overexposure and underexposure correction, the features of the student network should be close to the corresponding features of the teacher network.

[0044] The knowledge distillation process is represented as:

[0045] L=L s +L r +L kd (5)

[0046] Wherein, L kd is the error between the features of the teacher and student networks, and is represented as:

[0047]

[0048] Wherein, N is the number of features used, and represent the features in the student network and the teacher network, respectively. It can be a feature in the overexposure correction network or a feature in the underexposure network, which is used to supervise the features in the student network.

[0049] So far, the construction of the knowledge distillation training strategy has been completed, which can distill different prior knowledge into a unified exposure correction network.

[0050] Step 104: input the learning rate, optimization method, iteration number and collected real paired data set, train the network parameters according to the knowledge distillation strategy, and obtain the mapping relationship f between the over / underexposure image and the normal exposure image.

[0051] The optimized network parameters are obtained by optimizing the objective functions 3 and 5, and the training of the exposure correction network is completed, and the optimized mapping relationship f between the over / underexposure image and the normal exposure image is obtained.

[0052] Step 105: input the over / underexposure image to be tested, and input the mapping relationship f between the over / underexposure image and the normal exposure image obtained in step 104. Through the mapping relationship f between the over / underexposure image and the normal exposure image, the over / underexposure image is mapped into a normal exposure image, so as to realize efficient and high-precision image exposure correction and improve the imaging quality.

[0053] Further, the normal exposure image is I and over / under-exposed image and a normally exposed image, respectively.

[0054] Preferably, the training process of the network of step 104 and the image exposure correction process of step 105 are completed using GPU, and the cuDNN library is used to speed up the running speed of the convolutional neural network.

[0055] On the other hand, as Figure 2 As shown in the figure, the present application proposes a camera-aware characteristic guided real exposure correction system, which comprises a data acquisition module, a channel-guided exposure correction network design module, a knowledge distillation strategy design module and a network training and testing module.

[0056] The data acquisition module is used to acquire real paired over / under-exposed image and normally exposed image dataset. The output end of the data acquisition module is connected with the input end of the network training and testing module.

[0057] The channel-guided exposure correction network design module is used to design the exposure correction network to make full use of the prior information of different channels. The output end of the channel-guided exposure correction network design module is connected with the input end of the knowledge distillation strategy design module.

[0058] The knowledge distillation strategy design module is used to design the network training strategy to distill the knowledge of over-exposure and under-exposure correction networks into a unified exposure correction network model. The output end of the knowledge distillation strategy design module is connected with the input end of the network training and testing module.

[0059] The network training and testing module is used to train the above-mentioned exposure correction network and test its effectiveness.

[0060] The above specific description further details the purpose, technical scheme and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A camera-sensing characteristic-guided true exposure correction method, characterized in that, Includes the following steps: Step 101: Build a real data acquisition system and use the system to acquire real paired exposure correction datasets, which contain pairs of overexposed / underexposed images and normally exposed images. Step 102: Analyze the camera's perception characteristics, design a channel-guided exposure correction network, and learn different prior knowledge required for underexposure and overexposure in the same network; The overall network structure includes a main branch, a color channel guide branch, and a guide channel selection module. The RAW image is decomposed into four channels: red, green, green, and blue. First, the guide channel selection module automatically selects the required color channel for overexposure or underexposure correction. Then, the selected color channel is input into the color channel guide branch to generate guide features. Finally, the guide features are used to modulate the corresponding features in the main branch. Design a guided channel selection module. First, input the four-channel RAW image into a global average pooling layer to obtain the average vector. Then, to further obtain the weight information of different channels, use two fully connected layers to learn the nonlinear relationship between each channel. Then, use the Sigmoid gating mechanism to limit the final attention weight of each channel, and the channel corresponding to the two maximum values ​​is the selected channel. When training the network to correct overexposure and underexposure, the constraint-guided channel selection module selects specific channels for the corresponding correction task, as shown below: （1） in, Indicates channel selection loss. and These represent the selected channel and the target channel, respectively: overexposure correction is done in the blue and red channels, and underexposure correction is done in the green channel; Design a color channel guidance module to modulate features using guidance information. First, input the selected color channel into the color channel guidance branch to generate guidance features. Then, use the guidance features with corresponding spatial resolution to modulate the main branch features. Step 103: Use knowledge distillation to enable the channel-guided exposure correction network to learn the prior knowledge required for overexposure and underexposure correction more effectively; Knowledge distillation involves distilling the prior knowledge required for overexposure and underexposure correction into a channel-guided exposure correction network. Before using knowledge distillation, two networks are first trained, one for overexposure correction and the other for underexposure correction, with their objective loss functions... Represented as: （3） in, and These are the guide channel selection loss and reconstruction loss, respectively; Reconstruction losses Represented as: （4） in, and These are represented as the output of the exposure correction network and the corresponding target image, respectively. When the overexposure and underexposure correction networks are well trained, the knowledge distillation method is used to extract the knowledge of overexposure and underexposure correction into a unified network; In terms of knowledge distillation, the two well-trained overexposure and underexposure correction networks are the teacher network, and the unified overexposure and underexposure correction network is the student network; for both overexposure and underexposure correction, the characteristics of the student network should be close to the corresponding characteristics of the teacher network. The knowledge distillation process can be represented as: （5） in, The error between the network characteristics of teachers and students is represented as: （6） in, The number of features used. and These represent the characteristics in the student network and the teacher network, respectively. Used to monitor characteristics within student networks; Step 104: Input the learning rate, optimization method, number of iterations, and the collected real pairwise dataset. Train the network parameters according to the knowledge distillation strategy to obtain the mapping relationship between overexposed / underexposed images and normally exposed images. ; The optimized network parameters are obtained by optimizing the objective functions Equations 3 and 5 during training, thus completing the training of the exposure correction network and obtaining the mapping relationship between the optimized overexposed / underexposed images and the normally exposed images. ; Step 105: Input the overexposed / underexposed image to be tested, and input the mapping relationship between the overexposed / underexposed image and the normally exposed image obtained in Step 104. ; through the mapping relationship between overexposed / underexposed images and normally exposed images It maps overexposed / underexposed images to properly exposed images, thereby achieving high-efficiency and high-precision image exposure correction and improving image quality; Normal exposure image is , and These are overexposed / underexposed images and normally exposed images, respectively.

2. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, In step 101, the real data acquisition system first adjusts the aperture, focal length, and exposure time during shooting to acquire a high-quality, properly exposed image; then, it uses control software to control the exposure time at a magnification of 1 / 100 to 10; finally, it acquires underexposed or overexposed images.

3. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, In step 102, the main branch includes 4 encoder stages and 4 corresponding decoder stages; At the end of each encoder stage, a 4×4 kernel size and 2-step convolution downsample the feature map to a ratio of 1 / 2. Before each decoder stage, the feature map is upsampled to two scales using bilinear interpolation; Skip connections pass shallow feature maps from each encoder stage to the corresponding decoder stage.

4. The camera-sensing characteristic-guided true exposure correction method as described in claim 1 or 3, characterized in that, Residual learning is introduced into the exposure correction network, using residual blocks as basic blocks to construct the encoder and decoder. Each residual block consists of two... The convolution consists of an activation function and a... convolution.

5. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, In step 102, two convolutional layers are used to extract guiding features. Pixel-level scaling and bias values ​​are generated to enhance the main branch features. , represented as: in, For the enhanced features, and These are two learnable modulation parameters.

6. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, In step 103, These are features in overexposure correction networks.

7. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, In step 103, This is a characteristic of underexposed networks.

8. The camera-sensing characteristic-guided true exposure correction method as described in claim 1, characterized in that, The training process of the network in step 104 and the image exposure correction process in step 105 are completed using a GPU, and the cuDNN library is used to accelerate the running speed of the convolutional neural network.

Images