Method for tone mapping and purple edge removing of HDR image

Abstract

The invention provides a method for tone mapping and purple edge removal of an HDR image. The method comprises the steps of S1, data preparation; s2, performing network pre-processing on the image data, namely a data processing part; s3, designing a convolutional neural network for tone mapping and purple edge removal; S3.1, designing a basic feature extraction module; s3.2, designing a global condition vector module; s3.3, designing a scaling offset module; s3.4, designing a compression parameter module; s3.5, carrying out postprocessing on the design network; and S4, training by using the convolutional neural network in the S3, using Adam as an optimizer, setting the initial learning rate to be 0.0001, setting the training period to be 100, reducing the learning rate by 0.1 time every 30 training periods, setting loss as a combination of L1loss and vgloss, and setting the loss weight to be 10: 1. The tone mapping output of scenes in various dynamic ranges is stable and does not flicker, and complex parameter adjustment of a traditional image method is avoided; the problem that the tone mapping result of a scene with a very large dynamic range is difficult to fit is solved, and the tone mapping robustness is relatively good; meanwhile, purple edge removal is accurate, and missing detection and false detection are avoided.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent monitoring video processing, and in particular relates to a method for simultaneously removing purple fringes from HDR image tone mapping. Background Art

[0002] In the existing technology, although HDR images have a larger dynamic range and can reflect the real scene more delicately, they require larger storage space and transmission bandwidth than LDR images, and HDR is difficult to output and display. Currently, the dynamic range of most graphics output devices such as monitors and printers is much smaller than that of ordinary HDR images. In the process of image dynamic range compression, namely tone mapping, purple fringing will also appear on ordinary monitors. It is a very necessary and meaningful research to be able to achieve the function of removing purple fringing while tone mapping.

[0003] The current tone mapping methods are mainly divided into traditional tone mapping methods and tone mapping methods based on convolutional neural networks; traditional tone mapping methods are further divided into global and local methods; methods based on convolutional neural networks usually select relatively good LDR images as labels and HDR images as input through a variety of traditional methods. In terms of removing purple fringing, you can usually choose a relatively high-end optically optimized lens / sensor to remove purple fringing. If conditions permit, you can also use post-processing software such as PS and Lightroom to remove purple fringing. In the ISP process, if there is no CAC module in the ISP process or the CAC module does not work well, you need to use a depurple algorithm to achieve the function of removing purple fringing. At present, most depurple algorithms use traditional ideas and are divided into two parts: purple fringing detection and purple fringing correction.

[0004] However, among traditional tone mapping methods, the global method is fast but has poor results, while the local method is slow and has relatively good results but has more parameters, and it is difficult to adjust the parameters for scenes with different dynamic ranges. Existing methods based on convolutional neural networks have poor fitting capabilities for scenes with very large dynamic ranges, and the brightness of LDR images is difficult to stabilize. When the dynamic range of the scene changes drastically, the LDR image will flicker and other unstable conditions.

[0005] In addition, the existing purple fringing removal algorithm uses a fixed parameter combination, so it is difficult to simultaneously achieve purple fringing detection in different scenes, and missed detection and false detection are inevitable.

[0006] Moreover, the tone mapping and purple fringing removal algorithms in existing ISPs are often functionally divided into two modules, which have higher requirements on hardware and computing power.

[0007] In addition, common terminology explanations in the prior art include:

[0008] HDR: high dynamic range, dynamic range refers to the brightness ratio between the brightest object and the darkest object in the scene. The larger the dynamic range, the richer the levels that can be expressed. The dynamic range in real scenes reaches 109:1, the dynamic range that the human visual system can perceive is about 105:1, and the dynamic range of general image sensors is about 102:1.

[0009] LDR: low dynamic range, low dynamic range.

[0010] Convolutional neural network: A convolutional neural network is a deep neural network with a convolutional structure. Images can be directly used as input to the network, avoiding the complex feature extraction and data reconstruction process in traditional algorithms. It has great advantages in the processing of two-dimensional images. For example, the network can extract image features including color, texture, shape and image topology by itself, and is robust and efficient in processing two-dimensional images.

[0011] Tone mapping: Tone mapping enables HDR images to be displayed on our low dynamic range displays and to be as consistent as possible with the human eye's visual experience. Its essence is to convert HDR images into LDR images.

[0012] Purple fringing: The cause of purple fringing is relatively complex and is generally believed to be caused by lens chromatic aberration (the lens refracts light of different spectra to different degrees, resulting in the inability to image light of different spectra onto one point). The imaging system generally focuses on the green channel accurately. Due to lens chromatic aberration, the blue and red channels cannot be completely focused, resulting in purple-red fringing on the edge of the object.

[0013] The optimizer Adam is named after Adaptive Moment Estimation. Summary of the invention

[0014] In order to solve the above problems, the purpose of this application is: the present invention combines a tone mapping algorithm based on deep learning and uses a network to simultaneously realize the functions of tone mapping and purple fringing removal.

[0015] Specifically, the present invention proposes a method for simultaneously removing purple fringes from HDR image tone mapping, wherein the main implementation steps of the method are as follows:

[0016] S1, data preparation: prepare 400 sets of data pairs, each set of data contains HDR images of the same scene I HDR , HDR corresponding long frame image I long , and the LDR image I after removing the purple edge LDR_gt , where I LDR_gtAs a GT, its purple edge removal method uses traditional methods to adjust parameters and select the best one;

[0017] S2, network pre-processing of image data:

[0018] S2.1, design network pre-processing, first normalize the HDR image and the long frame image I corresponding to HDR long , as shown in the following formulas (1) and (2):

[0019]

[0020]

[0021] Among them I HDR For 16-bit HDR images, I HDRN is the normalized image, I longn is the long frame image corresponding to the normalized HDR;

[0022] S2.2, for the long frame image I corresponding to the normalized HDR longn Perform binarization processing, set I longn_i For any point on the image, if the point is overexposed, that is, I longn_i The corresponding pixel value is 1, then the pixel value of the point remains unchanged. If the point is not overexposed, that is, I longn_i If the corresponding pixel value is less than 1, the pixel value of the point is changed to 0, and the image after binarization is recorded as I longnb ,

[0023] I longn_i For image I longn At any point, after the binarization and replacement of the following formula (3), the new I longn On I longn_i The value of changes, and the resulting image after the change is I longnb ; The processing is shown in formula (3):

[0024]

[0025] S2.3, for the long frame image I corresponding to the normalized and binarized HDR longnb Further processing is performed by setting the R and G values ​​of the four components RGGB to 0, as shown in formula (4), and the resulting image is recorded as I longnb_B :

[0026]

[0027] Among them I longnb_R Represents the R component pixel in the image, I longnb_G Represents the G component pixel in the image; I longnb_Rand I longnb_G For image I longnb The corresponding components of the points on R and G are processed and replaced by formula (4) and the new I longnb On I longnb_R and I longnb_G The value of changes, and the resulting image after the change is I longnb_B ;

[0028] S2.4, in the task of dynamic range compression, the transformation in the log domain is more in line with the visual characteristics of the human eye, so it is converted to the log domain, and the HDR image is converted to the log2 logarithmic domain through the following formula (5);

[0029]

[0030] I HDRN_log = -1×log2(I′ HDRN ) Formula (5)

[0031] The negative sign is because the base of the log function used in this method is 2, and the value of the independent variable is 0-1. At this time, the function value is negative, so it is negated;

[0032] S2.5, converting the HDR image I into the log2 domain HDRN_log And the long frame after pre-processing image I longnb_B The concatenation is performed in the channel dimension and the result is recorded as I HDRN_long ;

[0033] Among them, the dimensions of the network training are 4 dimensions, namely B, C, H, W: Batch_size is the number of batches, Channel is the number of channels, Height is the image height, and Width is the image width. Splicing in the channel dimension can ensure the consistency of size during the network operation, and at the same time ensure that the image information is not lost, so as to facilitate the next step of feature extraction;

[0034] S3, design convolutional neural network for tone mapping and purple edge removal:

[0035] S3.1, the beginning of the convolutional neural network module, the first submodule is the basic feature extraction module: design the basic feature extraction module, use three consecutive convolution kernels of 3x3, stride 1, pad 1 convolution series, each convolution is followed by a relu activation function, I HDRN_long The image is passed through the basic feature extraction module to obtain basic features;

[0036] S3.2, design a global conditional vector module, use three consecutive convolutions with a kernel of 3x3, a stride of 2, and a pad of 1, and connect them in series. Each convolution is followed by a relu activation function. After the basic features pass through the global conditional vector module, the average value in the w and h directions is calculated to obtain a global conditional vector of size 32;

[0037] S3.3, design a scaling offset module, use 6 fully connected layers to form the scaling offset module, the global condition vector generated in step S3.2 passes through 6 identical fully connected layers to obtain 6 groups of parameters, which are divided into a first group of scaling parameters and offset parameters, a second group of scaling parameters and offset parameters, and a third group of scaling parameters and offset parameters in turn; the basic features of step S3.1 are convolved with a convolution kernel of 3x3, a stride of 1, and a pad of 1, and then mapped with the first group of scaling parameters and offset parameters to obtain the first global feature; the first global feature is convolved with a convolution kernel of 3x3, a stride of 1, and a pad of 1, and then mapped with the second group of scaling parameters and offset parameters to obtain the second global feature; the second global feature is convolved with a convolution kernel of 3x3, a stride of 1, and a pad of 1, and then mapped with the third group of scaling parameters and offset parameters to obtain the third global feature;

[0038] S3.4, design a compression parameter module, concat the third global feature and the basic feature extracted in step S3.1, and then perform a convolution with a convolution kernel of 3x3, a stride of 2, and a pad of 1, and then perform a tanh function to obtain a dynamic range compression parameter scale. The hyperbolic tangent function tanh is one of the activation functions in deep learning tasks, and its expression is: The operation here is the same as step S2.5. Concat is the concatenation in step S2.5, and is also performed in the Channel dimension.

[0039] S3.5, design network post-processing, I HDRN_long The image is multiplied by the dynamic range compression parameter scale, and then exponentially multiplied by 2 to obtain the LDR image, calculated as shown in formula (6), where I LDR This is the image after tone mapping and purple fringing removal;

[0040]

[0041] S4, network training:

[0042] The convolutional neural network designed in step S3 is used for training, and the optimizer uses Adam. The Adam (Adaptive Moment Estimation) optimizer is an adaptive optimization algorithm used to update the weights of the neural network. It can adaptively adjust the learning rate to improve the convergence speed and generalization ability of the model, and is widely used in the neural network training process; the initial learning rate is 0.0001, the training cycle is 100, the weight decay parameter is 0.0001, the loss is a combination of L1 loss and vggloss, and the loss weight is 10:1.

[0043] In the step S1, the purple edge removal method uses a traditional method to adjust parameters and select the best one, further comprising:

[0044] There is a CAC module (Chromatic Aberration Correction) in the ISP process: by correcting the lens, the purple fringing caused by the optical chromatic aberration of the lens is optimized (optical chromatic aberration includes longitudinal chromatic aberration and lateral chromatic aberration); if the CAC correction is incomplete, or purple fringing is still caused by reasons other than the optical chromatic aberration of the lens, the subsequent ISP will include a depurple module, which includes two parts: purple fringing detection and purple fringing correction;

[0045] The screening is preferably by human visual inspection, including:

[0046] ①The degree of cleanliness of purple fringing removal, aiming to solve the problem of missed detection;

[0047] ② Whether the color is abrupt after purple fringing correction, aim to solve the color problem after correction;

[0048] ③Whether the non-purple edge parts are affected, aiming to solve the problem of false detection.

[0049] In step S2.1, the normalization refers to image normalization, which converts the pixel values ​​of an image into values ​​within a certain range in order to better perform the next step of processing. In this method, the image I HDR and I long The initial data type is uint16, whose theoretical maximum value is 65535. The goal is to normalize it to float32 data with a maximum value of 1, so I HDR and I long Divide by 65535, then convert to float32, normalize uint16, and then convert to float32, no data loss will occur; assuming it is expressed in python3 code:

[0050] img_hdrn=(img_hdr / 65535).astype(np.float32)

[0051] img_longn=(img_long / 65535).astype(np.float32);

[0052] In step S2.2, the binarization refers to image binarization, which is to set the grayscale value of the pixel points on the image to the maximum or minimum, that is, the process of making the entire image present an obvious black and white effect, wherein the division is based on the threshold value. In this method, the threshold value is 1, that is, the area that was originally overexposed remains overexposed, and the grayscale value of the area that was originally not overexposed is set to 0;

[0053] The code using python3 is:

[0054] img_longn[img_longn<1]=0

[0055] img_longn[img_longn! =0]=1

[0056] img_longnb = img_longn;

[0057] In step S2.3, the RAW data format adopts the Bayer arrangement, the ratio of the R\G\B components in the Bayer is 1:2:1, the BAYER format is divided into four modes: GBRG, GRBG, BGGR, and RGGB, and this method adopts the RGGB mode. In order to implement this step, the target pixel is directly processed according to the specific position information of the RGGB mode;

[0058] The code using python3 is:

[0059] img_longnb[0::2,0::2]=0 #R

[0060] img_longnb[1::2,0::2]=0 #G

[0061] img_longnb[0::2,1::2]=0 #G;

[0062] In step S2.4,

[0063] The code using python3 is:

[0064] I HDRN_log = -1*torch.log2(I' HDRN );

[0065] In the step S2.5,

[0066] The code using python3 is:

[0067] I HDRN_long = torch.cat([I HDRN_log ,I longnb_B ].

[0068] In step S3.5,

[0069] The python3 code is: LDR =2**(scale*I HDRN_long ).

[0070] In the step S4,

[0071] The parameters passed into the interface in pytorch include: network parameters, initial learning rate lr, weight decay parameter weight_decay;

[0072] The code using python3 is:

[0073] optimizer=torch.optim.Adam(NET.parameters(),

[0074] lr=config.lr,

[0075] weight_decay=config.weight_decay).

[0076] Therefore, the advantages of the present application are: the present invention uses a set of networks to simultaneously complete the functions of tone mapping and purple fringing removal, that is, by designing a convolutional neural network to perform tone mapping and purple fringing removal, the network is allowed to learn the degree of tone mapping for scenes with different dynamic ranges in a data-driven manner; it ensures that the tone mapping output is stable and does not flicker in scenes with various dynamic ranges, avoiding the complex parameter adjustment of traditional image methods; it solves the problem that the tone mapping results of scenes with very large dynamic ranges are difficult to fit, and the tone mapping has good robustness; at the same time, purple fringing is accurately removed, avoiding the problems of missed detection and false detection that often occur in traditional depurple algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0078] Figure 1 It is a schematic diagram of the process of this method. DETAILED DESCRIPTION

[0079] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0080] like Figure 1 As shown, the present application proposes a method for simultaneously removing purple fringes from HDR image tone mapping, wherein the main implementation steps of the method are as follows:

[0081] S1, data preparation: prepare 400 sets of data pairs, each set of data contains HDR images of the same scene I HDR , HDR corresponding long frame image I long , and the LDR image I after removing the purple edge LDR_gt , where I LDR_gt As a GT, its purple edge removal method uses traditional methods to adjust parameters and select the best one;

[0082] The method for removing purple fringing uses a traditional method to adjust parameters and select the best one, and further includes: a CAC module (Chromatic Aberration Correction) in the ISP process: by correcting the lens, the purple fringing caused by the optical chromatic aberration of the lens is optimized (optical chromatic aberration includes longitudinal chromatic aberration and lateral chromatic aberration); if the CAC correction is incomplete, or purple fringing is still generated due to reasons other than the optical chromatic aberration of the lens, the subsequent ISP will include a depurple module, which includes two parts: purple fringing detection and purple fringing correction;

[0083] The screening is preferably by human visual inspection, including:

[0084] ①The degree of cleanliness of purple fringing removal, aiming to solve the problem of missed detection;

[0085] ② Whether the color is abrupt after purple fringing correction, aim to solve the color problem after correction;

[0086] ③Whether the non-purple edge parts are affected, aiming to solve the problem of false detection;

[0087] S2, network pre-processing of image data, that is, data processing part:

[0088] S2.1, design network pre-processing, first normalize the HDR image and the long frame image I corresponding to HDR long , as shown in the following formulas (1) and (2):

[0089]

[0090]

[0091] Among them I HDR For 16-bit HDR images, I HDRN is the normalized image, I longn is the long frame image corresponding to the normalized HDR;

[0092] Normalization refers to image normalization, which converts the pixel values ​​of an image into values ​​within a certain range in order to better perform the next step of processing. In this method, the image I HDR and I long The initial data type is uint16, whose theoretical maximum value is 65535. The goal is to normalize it to float32 data with a maximum value of 1, so I HDR and I long Divide by 65535, then convert to float32, normalize uint16, and then convert to float32, no data loss will occur; assuming it is expressed in python3 code:

[0093] img_hdrn=(img_hdr / 65535).astype(np.float32)

[0094] img_longn=(img_long / 65535).astype(np.float32);

[0095] S2.2, for the long frame image I corresponding to the normalized HDR longn Perform binarization processing, set I longn_i For any point on the image, if the point is overexposed, that is, I longn_i The corresponding pixel value is 1, then the pixel value of the point remains unchanged. If the point is not overexposed, that is, I longn_i If the corresponding pixel value is less than 1, the pixel value of the point is changed to 0, and the image after binarization is recorded as I longnb ,

[0096] I longn_i For image I longn At any point, after the binarization and replacement of the following formula (3), the new I longn onI longn_i The value of changes, and the resulting image after the change is I longnb ;

[0097] The processing is shown in formula (3):

[0098]

[0099] The binarization refers to image binarization, which is to set the grayscale value of the pixel points on the image to the maximum or minimum, that is, the process of making the entire image present an obvious black and white effect. The basis for the division is the threshold. In this method, the threshold is 1, that is, the area that was originally overexposed remains overexposed, and the grayscale value of the area that was originally not overexposed is set to 0;

[0100] The code using python3 is:

[0101] img_longn[img_longn<1]=0

[0102] img_longn[img_longn! =0]=1

[0103] img_longnb = img_longn;

[0104] S2.3, for the long frame image I corresponding to the normalized and binarized HDR longnb For further processing, the RAW data format adopts the Bayer arrangement, the ratio of R\G\B components in Bayer is 1:2:1, the BAYER format is divided into four modes: GBRG, GRBG, BGGR, and RGGB, and this method adopts the RGGB mode. In order to implement this step, the target pixel is directly processed according to the specific position information of the RGGB mode;

[0105] The R and G values ​​of the four components of RGGB are set to 0, as shown in formula (4), and the resulting image is recorded as I longnb_B :

[0106]

[0107] Among them I longnb_R Represents the R component pixel in the image, I lognb_G Represents the G component pixel in the image; I longnb_ R and I longnb_G For image I longnb The corresponding components of the points on R and G are processed and replaced by formula (4) and the new I longnb On I longnb_R and I longnb_G The value of changes, and the resulting image after the change is I longnb_B ;

[0108] The code using python3 is:

[0109] img_longnb[0::2,0::2]=0 #R

[0110] img_longnb[1::2,0::2]=0 #G

[0111] img_longnb[0::2,1::2]=0 #G;

[0112] S2.4, in the task of dynamic range compression, the transformation in the log domain is more in line with the visual characteristics of the human eye, so it is converted to the log domain, and the HDR image is converted to the log2 logarithmic domain through the following formula (5);

[0113]

[0114] I HDRN_log = -1×log2(I′ HDRN ) Formula (5)

[0115] The negative sign is because the base of the log function used in this method is 2, and the value of the independent variable is 0-1. At this time, the function value is negative, so it is negated;

[0116] The code using python3 is:

[0117] I HDRN_log = -1*torch.log2(I′ HDRN );

[0118] S2.5, converting the HDR image I into the log2 domain HDRN_log And the long frame after pre-processing image I longnb_B The concatenation is performed in the channel dimension and the result is recorded as I HDRN_log ;

[0119] Among them, the dimensions of the network training are 4 dimensions, namely B, C, H, W: Batch_size is the number of batches, Channel is the number of channels, Height is the image height, and Width is the image width. Splicing in the channel dimension can ensure the consistency of size during the network operation, and at the same time ensure that the image information is not lost, so as to facilitate the next step of feature extraction;

[0120] The code using python3 is:

[0121] I HDRN_long = torch.cat([I HDRN_log , I longnb_B ];

[0122] S3, design convolutional neural network for tone mapping and purple edge removal:

[0123] S3.1, the beginning of the convolutional neural network module, the first submodule is the basic feature extraction module: design the basic feature extraction module, use three consecutive convolution kernels of 3x3, stride 1, pad 1 convolution series, each convolution is followed by a relu activation function, I HDRN_long The image is passed through the basic feature extraction module to obtain basic features;

[0124] S3.2, design a global conditional vector module, use three consecutive convolutions with a kernel of 3x3, a stride of 2, and a pad of 1, and connect them in series. Each convolution is followed by a relu activation function. After the basic features pass through the global conditional vector module, the average value in the w and h directions is calculated to obtain a global conditional vector of size 32;

[0125] S3.3, design a scaling offset module, use 6 fully connected layers to form the scaling offset module, the global condition vector generated in step S3.2 passes through 6 identical fully connected layers to obtain 6 groups of parameters, which are divided into the first group of scaling parameters and offset parameters, the second group of scaling parameters and offset parameters, and the third group of scaling parameters and offset parameters in turn; the basic features of step S3.1 pass through a convolution with a convolution kernel of 3x3, stride of 1, and pad of 1, and then pass through the first group of scaling parameters and offset parameters to obtain the first global feature; the first global feature passes through 1 The convolution kernel is 3x3, stride is 1, and pad is 1, and then the second global feature is obtained through the second set of scaling parameters and offset parameters. The second global feature is convolved with a convolution kernel of 3x3, stride is 1, and pad is 1, and then the third global feature is obtained through the third set of scaling parameters and offset parameters. The sources of the 3 groups of 6 parameters here are all obtained by processing the same fully connected layer, but the network uses and usage orders are different, so the names are different, in order to ensure the complexity and diversity of the network in the fitting process and the robustness of the trained model.

[0126] S3.4, design a compression parameter module, concat the third global feature and the basic feature extracted in step S3.1, and then perform a convolution with a convolution kernel of 3x3, a stride of 2, and a pad of 1, and then perform a tanh function to obtain a dynamic range compression parameter scal; the operation here is the same as step S2.5, and concat is the concatenation in step S2.5, which is also performed in the Channel dimension;

[0127] S3.5, design network post-processing, I HDRN_long The image is multiplied by the dynamic range compression parameter scale, and then exponentially multiplied by 2 to obtain the LDR image, calculated as shown in formula (6), where I LDR This is the image after tone mapping and purple fringing removal;

[0128]

[0129] The python3 code is: LDR =2**(scale*I HDRN_long );

[0130] S4, network training:

[0131] The convolutional neural network designed in step S3 is used for training, and the optimizer uses Adam. The Adam (Adaptive Moment Estimation) optimizer is an adaptive optimization algorithm used to update the weights of the neural network. It can adaptively adjust the learning rate to improve the convergence speed and generalization ability of the model, and is widely used in the neural network training process;

[0132] The parameters passed into the interface in pytorch include: network parameters, initial learning rate lr, weight decay parameter weight_decay;

[0133] The code using python3 is:

[0134] optimizer=torch.optim.Adam(NET.parameters(),

[0135] lr=config.lr,

[0136] weight_decay=config.weight_decay);

[0137] The initial learning rate is 0.0001, the training cycle is 100, the weight decay parameter is 0.0001, the loss is a combination of L1 loss and vggloss, and the loss weight is 10:1.

[0138] In summary, this method goes through the following steps:

[0139] S1, Data preparation: Image I HDR and I long The initial data type is uint16;

[0140] S2, network pre-processing: data I HDR and I long According to step S2.1, the data is normalized to float32 data with a maximum value of 1. Then, the normalized long frame data is binarized according to step S2.2. Then, according to step S2.3, the R and G values ​​of the four components of RGGB are set to 0, and the normalized I HDR After conversion to the log domain in step S2.4, the processed HDR and its long frame corresponding images are spliced ​​in the channel dimension in S2.5 to complete the pre-network data processing;

[0141] S3, Convolutional Neural Network: Set the training parameters and configure the Adam optimizer according to step S4. LDR_gt Participate in loss calculation for labels;

[0142] In the network processing process: the result of the pre-processing is subjected to basic feature extraction in step S3.1, wherein the channels of three consecutive convolutions are all set to 32 to obtain basic features, and then the basic features are subjected to step S3.2 to obtain a global vector, and the third global feature is obtained through the offset scaling module in step S3.3, and the third global feature and the basic feature are processed by the parameter compression module in step S3.4 to obtain the compression coefficient scale, and finally the network post-processing in step S3.5 is performed to obtain the result image that realizes dynamic range compression and purple edge removal at the same time;

[0143] S4, conduct training.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for simultaneously removing purple fringing from HDR image tone mapping, characterized in that: The method comprises the following steps: S1, data preparation: prepare 400 sets of data pairs, each set of data contains HDR images of the same scene I HDR , HDR corresponding long frame image I long , and the LDR image I after removing the purple edge LDR_gt , where I LDR_gt As a GT, its purple edge removal method uses traditional methods to adjust parameters and select the best one; S2, network pre-processing of image data, that is, data processing part: S2.1, design network pre-processing, first normalize the HDR image and the long frame image I corresponding to HDR long , as shown in the following formulas (1) and (2): Among them I HDR For 16-bit HDR images, I HDRN is the normalized image, I longn is the long frame image corresponding to the normalized HDR; S2.2, for the long frame image I corresponding to the normalized HDR longn Perform binarization processing, set I longn_i For any point on the image, if the point is overexposed, that is, I longn_i The corresponding pixel value is 1, then the pixel value of the point remains unchanged. If the point is not overexposed, that is, I longn_i If the corresponding pixel value is less than 1, the pixel value of the point is changed to 0, and the image after binarization is recorded as I longnb , I longn_i For image I longn At any point, after the binarization and replacement of the following formula (3), the new I longn onI longn_i The value of changes, and the resulting image after the change is I longnb ; The processing is shown in formula (3): S2.3, for the long frame image I corresponding to the normalized and binarized HDR longnb Further processing is performed by setting the R and G values ​​of the four components RGGB to 0, as shown in formula (4), and the resulting image is recorded as I longnb_B : Among them I longnb_R Represents the R component pixel in the image, I longnb_G Represents the G component pixel in the image; I longnb_R and I longnb_G For image I longnb The corresponding components of the points on R and G are processed and replaced by formula (4) and the new I longnb onI longnb_R and I longnb_G The value of changes, and the resulting image after the change is I longnb_B ; S2.4, in the task of dynamic range compression, the transformation in the log domain is more in line with the visual characteristics of the human eye, so it is converted to the log domain, and the HDR image is converted to the log2 logarithmic domain through the following formula (5); I HDRN_log = -1 × log2(I′ HDRN ) Formula (5) The negative sign is because the base of the log function used in this method is 2, and the value of the independent variable is 0-1. At this time, the function value is negative, so it is negated; S2.5, converting the HDR image I into the log2 domain HDRN_log And the long frame after pre-processing image I longnb_B The concatenation is performed in the channel dimension and the result is recorded as I HDRN_log ; Among them, the dimensions of the network training are 4 dimensions, namely B, C, H, W: Batch_size is the number of batches, Channel is the number of channels, Height is the image height, and Width is the image width. Splicing in the channel dimension can ensure the consistency of size during the network operation, and at the same time ensure that the image information is not lost, so as to facilitate the next step of feature extraction; S3, design convolutional neural network for tone mapping and purple edge removal: S3.1, the beginning of the convolutional neural network module, the first submodule is the basic feature extraction module: design the basic feature extraction module, use three consecutive convolution kernels of 3x3, stride 1, pad 1 convolution series, each convolution is followed by a relu activation function, I HDRN_long The image is passed through the basic feature extraction module to obtain basic features; S3.2, design a global conditional vector module, use three consecutive convolutions with a kernel of 3x3, a stride of 2, and a pad of 1, and connect them in series. Each convolution is followed by a relu activation function. After the basic features pass through the global conditional vector module, the average value in the w and h directions is calculated to obtain a global conditional vector of size 32; S3.3, design a scaling offset module, use 6 fully connected layers to form the scaling offset module, the global condition vector generated in step S3.2 passes through 6 identical fully connected layers to obtain 6 groups of parameters, which are divided into the first group of scaling parameters and offset parameters, the second group of scaling parameters and offset parameters, and the third group of scaling parameters and offset parameters in turn; the basic features of step S3.1 are convolved with a convolution kernel of 3x3, stride of 1, and pad of 1, and then mapped with the first group of scaling parameters and offset parameters to obtain the first global feature; the first global feature is convolved with a convolution kernel of 3x3, stride of 1, and pad of 1, and then mapped with the second group of scaling parameters and offset parameters to obtain the second global feature; the second global feature is convolved with a convolution kernel of 3x3, stride of 1, and pad of 1, and then mapped with the third group of scaling parameters and offset parameters to obtain the third global feature; S3.4, design a compression parameter module, concat the third global feature and the basic feature extracted in step S3.1, and then perform a convolution with a convolution kernel of 3x3, a stride of 2, and a pad of 1, and then perform a tanh function to obtain a dynamic range compression parameter scale. The hyperbolic tangent function tanh is one of the activation functions in deep learning tasks, and its expression is: The operation here is the same as step S2.

5. Concat is the concatenation in step S2.5, and is also performed in the Channel dimension. S3.5, design network post-processing, I HDRN_long The image is multiplied by the dynamic range compression parameter scale, and then exponentially multiplied by 2 to obtain the LDR image, calculated as shown in formula (6), where I LDR This is the image after tone mapping and purple fringing removal; S4, network training: The convolutional neural network designed in step S3 is used for training, and the optimizer uses Adam. The Adam optimizer is an adaptive optimization algorithm for updating the weights of the neural network, and it can adaptively adjust the learning rate; The initial learning rate is 0.0001, the training cycle is 100, the weight decay parameter is 0.0001, the loss is a combination of L1 loss and vggloss, and the loss weight is 10:

1.

2. The method for simultaneously removing purple fringing of HDR image tone mapping according to claim 1, characterized in that: In the step S1, the purple edge removal method uses a traditional method to adjust parameters and select the best one, further comprising: There is a CAC module in the ISP process: by correcting the lens, the purple fringing caused by the optical chromatic aberration of the lens is optimized; if the CAC correction is incomplete, or purple fringing is still caused by reasons other than the optical chromatic aberration of the lens, the subsequent ISP will include a depurple module, which includes two parts: purple fringing detection and purple fringing correction; The screening is preferably by human visual inspection, including: ①The degree of cleanliness of purple fringing removal, aiming to solve the problem of missed detection; ② Whether the color is abrupt after purple fringing correction, aim to solve the color problem after correction; ③Whether the non-purple edge parts are affected, aiming to solve the problem of false detection.

3. The method for simultaneously removing purple fringing of HDR image tone mapping according to claim 1, characterized in that: In step S2.1, the normalization refers to image normalization, which converts the pixel values ​​of an image into values ​​within a certain range in order to better perform the next step of processing. In this method, the image I HDR and L long The initial data type is uint16, whose theoretical maximum value is 65535. The goal is to normalize it to float32 data with a maximum value of 1, so I HDR and I long Divide by 65535, then convert to float32, normalize uint16, and then convert to float32, no data loss will occur; assuming it is expressed in python3 code: img_hdrn=(img_hdr / 65535).astype(np.float32) img_longn=(img_long / 65535).astype(np.float32); In step S2.2, the binarization refers to image binarization, which is to set the grayscale value of the pixel points on the image to the maximum or minimum, that is, the process of making the entire image present an obvious black and white effect, wherein the division is based on the threshold value. In this method, the threshold value is 1, that is, the area that was originally overexposed remains overexposed, and the grayscale value of the area that was originally not overexposed is set to 0; The code using python3 is: img_longn[img_longn<1]=0 img_longn[img_longn! =0]=1 img_longnb = img_longn; In step S2.3, the RAW data format adopts the Bayer arrangement, the ratio of the R\G\B components in the Bayer is 1:2:1, the BAYER format is divided into four modes: GBRG, GRBG, BGGR, and RGGB, and this method adopts the RGGB mode. In order to implement this step, the target pixel is directly processed according to the specific position information of the RGGB mode; The code using python3 is: img_longnb[0::2,0::2]=0 #R img_longnb[1::2,0::2]=0 #G img_longnb[0::2,1::2]=0 #G; In step S2.4, The code using python3 is: I HDRN _ log =-1*torch.log2(I′ HDRN ); In the step S2.5, The code using python3 is: I HDRN_long =torch.cat([I HDRN_log ,I longnb_B ]。 4. The method for simultaneously removing purple fringing of HDR image tone mapping according to claim 1, characterized in that: In step S3.5, The python3 code is: LDR ×2 ** (scale*I HDRN_long ).

5. The method for simultaneously removing purple fringing of HDR image tone mapping according to claim 1, characterized in that: In the step S4, The parameters passed into the interface in pytorch include: network parameters, initial learning rate lr, weight decay parameter weight_decay; The code using python3 is: optimizer=torch.optim.Adam(NET.parameters(), lr = config.lr, weight_decay=config.weight_decay).

Images