ISP plug-and-play brightness adjustment module based on neural network

Abstract

The present invention belongs to the field of image processing technology, and is specifically a neural network-based ISP plug-and-play brightness adjustment module. The brightness adjustment module of the present invention is mathematically represented as a linear mapping function: y = w*x + b, where the input x is the RAW image data I separated by color channel. rggb The product of the brightness adjustment coefficient α, that is, x = I rggb *α, the output y is the RAW image under the desired exposure conditions; w and b are the learnable coefficients of the brightness adjustment module. The brightness adjustment module is trained using the same dataset as the neural network image signal processor; the mapping coefficients w and b are updated via a backpropagation optimization algorithm. This invention enables the use of camera exposure coefficients to control the output image brightness level without changing the structure of the neural network image signal processor, enabling adaptive control of the output image brightness.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an ISP plug-and-play brightness adjustment module. Background Art

[0002] The goal of an ISP (Image Signal Processor) is to process the RAW image data from the image sensor, ultimately producing a beautiful and natural color image with the goal of human-perceived quality. Traditional ISPs employ a divide-and-conquer strategy, consisting of multiple hand-designed algorithm processing modules that each address related image quality issues. Existing convolutional neural network-based ISPs mostly employ a customized U-Net structure, trained on a dataset of RAW-RGB image pairs, enabling the neural network to complete end-to-end processing of input RAW images and output RGB images. Compared to traditional modular designs, image processors based on convolutional neural networks have the potential to jointly handle multiple image processing tasks and excel in complex imaging conditions such as low light and high dynamic range.

[0003] However, the output image brightness of existing convolutional neural network image signal processors is related to the brightness of the model training dataset and cannot adaptively control the brightness of the output image based on the parameters provided by the camera exposure system (exposure time, ISO, aperture size). Summary of the Invention

[0004] The object of the present invention is to provide an ISP plug-and-play brightness adjustment module based on a neural network that can adaptively control the brightness of an output image.

[0005] The present invention uses RAW-RGB data pairs under different exposure conditions during the training process of the neural network to design a plug-and-play brightness adjustment module to accept the exposure coefficient provided by the exposure system to control the brightness of the neural network output image without changing the original neural network structure.

[0006] The design idea of ​​the brightness adjustment module: Considering that the exposure system parameters include aperture size, exposure time, and ISO three key indicators, and there is a linear relationship between the brightness level (pixel value) of the RAW image and the aforementioned three exposure coefficients, the brightness adjustment module can be a linear function. It linearly maps the brightness level of the input RAW image to the desired brightness level based on the proportional relationship between the output and input exposure coefficients. The RAW image output by the brightness adjustment module is then processed through the neural network image signal processor to obtain the RGB image under the desired output exposure conditions.

[0007] Specifically, the neural network-based ISP plug-and-play brightness adjustment module designed by the present invention is a linear mapping function in the following form:

[0008] y=w*x+b, (1)

[0009] Its input x is the RAW image data I separated by color channel rggb The product of the brightness adjustment coefficient α, that is, x = I rggb *α, the output y is the RAW image under the desired exposure conditions; w and b are the learnable coefficients of the brightness adjustment module; the brightness adjustment module is trained using the same dataset as the neural network image signal processor;

[0010] During the training phase, the RAW image separated by color channels and the brightness adjustment coefficient α are used as input x. After the linear mapping function y=w*x+b, the RAW image y under the desired exposure conditions is output. The L2 loss function is calculated by pairing y with the real label RGB image of the real label RAW image, and the mapping coefficients w and b are updated through the back-propagation optimization algorithm.

[0011] The specific steps are as follows (see Figure 3 ):

[0012] Step 1: Based on the exposure factor when the RAW image was taken: aperture size a si , exposure time a ti and ISOa gi , and the exposure factor a of the desired output image so 、a to 、a go , calculate the brightness adjustment coefficient α according to the linear proportional relationship:

[0013]

[0014] Separate the standard Bayer format RAW by color channel (unpack) into RGGB image data I rggb .

[0015] Step 2: train the brightness adjustment module; the brightness adjustment coefficient α and the RGGB image data I rggb As the input of the brightness adjustment module, the linear mapping output is the RGGB image I under the desired exposure condition. orggb :

[0016] I Orggb =w*a*(I rggb +b), (3)

[0017] The training of the brightness adjustment module is based on the coefficients w and b of the linear mapping function fitted by the current data set. Deep learning technology can be used to calculate I orggb And the L2 loss is calculated for the output image GT RAW graph of the true label:

[0018] L2 loss(I orggb , I gt(raw) )=||I orggb -I gt(raw) ||2, (4)

[0019] Update w and b using the backpropagation optimization algorithm based on the loss results.

[0020] Step 3: Fine-tune the existing neural network image signal processor. The existing neural network image signal processor outputs the RGGB image I with the brightness adjustment module. orggb As input, it outputs an RGB image under the desired exposure conditions, and calculates the loss with the GTRGB image of the real label. The neural network model parameters are updated through the back-propagation optimization algorithm to achieve integration with the brightness adjustment module.

[0021] Step 2 and step 3 can be performed independently or simultaneously.

[0022] Step 4: After completing steps 2 and 3, the input brightness adjustment coefficient α and RGGB image I are obtained through preprocessing in step 1. rggb , the trained brightness adjustment module and neural network image signal processor can output RGB images under desired exposure conditions.

[0023] The present invention provides the ability to control the brightness level of an output image using a camera exposure coefficient without changing the structure of a neural network image signal processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the use of the plug-and-play brightness adjustment module in the present invention.

[0025] Figure 2 This is a diagram illustrating the brightness adjustment module training method of the present invention.

[0026] Figure 3 This is a flowchart illustrating the function implementation of the plug-and-play brightness adjustment module in the present invention.

[0027] Figure 4 This is an example of the processing effect of the brightness adjustment module in the present invention. DETAILED DESCRIPTION

[0028] Step 1: Based on the exposure factor when the RAW image was taken: aperture size a si , exposure time a ti and ISOa gi , and the exposure factor a of the desired output image so 、a to 、a go , calculate the brightness adjustment coefficient α according to the linear proportional relationship:

[0029]

[0030] Generally, the aperture size provided by RAW data is expressed in F number n. It is approximately expressed as the aperture size, so

[0031]

[0032] n i and n o Represents the F number of the input RAW and the desired output image respectively.

[0033] The standard Bayer format RAW image input is separated (unpacked) by color channel into 4-channel RGGB image data I rggb .

[0034] Step 2: train the brightness adjustment module; the brightness adjustment coefficient α and the RGGB image data I rggb As the input of the brightness adjustment module, the linear mapping output is the RGGB image I under the desired exposure condition. orggb :

[0035] I Orggb =w*a*(I rggb +b), (3)

[0036] The training of the brightness adjustment module is based on the coefficients w and b of the linear mapping function fitted by the current data set. Deep learning technology can be used to calculate the linear mapping output result I orggb And the L2 loss of the output image GT RAW map of the true label,

[0037] L2 loss(I orggb ,I gt(raw) )=||I arggb -I gt(raw) ||2, (4)

[0038] The back propagation optimization algorithm is used to update the w and b of the brightness adjustment module according to the loss results.

[0039] Step 3: Fine-tune the existing neural network image signal processor. The existing neural network image signal processor outputs the RGGB image I with the brightness adjustment module. orggb As input, it outputs an RGB image under the desired exposure conditions, and calculates the loss with the GTRGB image of the real label. The neural network model parameters are updated through the back-propagation optimization algorithm to achieve integration with the brightness adjustment module.

[0040] Steps 2 and 3 can be performed independently or simultaneously. Performing step 2 independently first and then performing steps 2 and 3 together after the L2 loss of the brightness adjustment module converges usually achieves better results.

[0041] Step 4: After completing steps 2 and 3, the input brightness adjustment coefficient α and RGGB image I are obtained through preprocessing in step 1. rggb , the trained brightness adjustment module and neural network image signal processor can output RGB images under desired exposure conditions.

[0042] The processing effect of the brightness adjustment module can be seen in Figure 4 When the existing neural network ISP is combined with the brightness adjustment module, the following operation can be achieved: input the RAW image under exposure condition B, and output the RGB image under the desired exposure condition (exposure condition A or C). It can be seen that the image output by the neural network ISP is quite close to the ground truth RGB image in the dataset.

[0043] The advantage of the present invention is that it provides the ability to control the brightness level of the output image using the camera exposure coefficient without changing the neural network ISP structure.

Claims

1. An ISP plug-and-play brightness adjustment module based on a neural network, characterized in that: is a linear mapping function of the following form: y=w*x+b, (1) Its input x is the RAW image data I separated by color channel rggb The product of the brightness adjustment coefficient α, that is, x = I rggb *α, the output y is the RAW image under the desired exposure conditions; w and b are the learnable coefficients of the brightness adjustment module; the brightness adjustment module is trained using the same dataset as the neural network image signal processor; In the training phase, the RAW image separated by color channels and the brightness adjustment coefficient α are used as input x. After the linear mapping function y=w*x+b, the RAW image y under the desired exposure condition is output. The L2 loss function is calculated by pairing y with the real label RGB image of the real label RAW image, and the mapping coefficients w and b are updated through the back-propagation optimization algorithm. The training process is: (1) According to the exposure factor when the RAW image was taken: aperture size a si , exposure time a ti and ISOa gi , and the exposure factor a of the desired output image so 、a to 、a go , calculate the brightness adjustment coefficient α according to the linear proportional relationship: Separate the standard Bayer format RAW into RGGB image data by color channel I rggb ; (2) The brightness adjustment coefficient α and the RGGB image data I rggb As the input of the brightness adjustment module, the linear mapping output is the RGGB image I under the desired exposure condition. orggb : I Orggb =w*a*(I rggb +b), (3) The training of the brightness adjustment module is based on the coefficients w and b of the linear mapping function fitted by the current data set, and deep learning technology is used to calculate I orggb Calculate the L2 loss of the output image GT RAW image with the true label, and use the backpropagation optimization algorithm to update w and b; (3) Fine-tune the existing neural network image signal processor to output the RGGB image I with the brightness adjustment module orggb As the input of the neural network image signal processor, it outputs an RGB image under the desired exposure conditions and calculates the loss with the ground truth RGB image. The neural network model parameters are updated through the back-propagation optimization algorithm to achieve integration with the brightness adjustment module. (4) After completing process (2) and process (3), the input brightness adjustment coefficient α and RGGB image I are obtained through preprocessing in process (1). rggb , the trained brightness adjustment module and neural network image signal processor can output RGB images under desired exposure conditions.

Images