Method for generating raw domain noise pairing data under extremely dark light

Abstract

The invention provides a raw domain noise pairing data generation method under extremely dark light. The method comprises the following steps: S1, acquiring noise modeling data and calibrating noise parameters; s2, collecting and making clean data; s3, designing a random brightness mapping strategy; s4, design of mosaic data enhancement is carried out; s5, designing overturning rotation data enhancement; s6, generating noise data; s7, designing a color cast disturbance strategy; and S8, the step S3 to the step S7 are executed in sequence in training, and the final noise data IN2 and the normalized data Iorm form paired data. According to an existing noise modeling method, clean data easy to obtain is processed, the tedious generation mode of real pairing data in an extremely dark light scene and the possible problems of pseudo textures, residual noise and color cast are avoided, meanwhile, data diversity is enhanced and expanded through data, and the model generalization ability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent monitoring video processing, and in particular relates to a method for generating raw domain noise pairing data under extremely dark light. Background Art

[0002] In the existing technology, in extremely dark light scenes, the denoising method based on neural network is significantly better than the traditional denoising method, with less pseudo texture and residual noise. However, training a better neural network requires a large amount of data as support.

[0003] Currently, there are two main types of noise paired data. The first is real paired data, which is usually the direct output of the image sensor as noise data, and then the clean data is obtained through the denoising algorithm; the second is to generate false paired data. There are currently methods to model the noise formation process, including photon shot noise, row noise, readout noise, etc., and add these noises to the clean data to obtain the noise data.

[0004] However, since it is difficult to obtain clean data in extremely dim light scenes, it is usually necessary to collect many frames and use weighted average to reduce noise, and then remove it through a denoising algorithm. The real paired data requires the scene to be completely still, and denoising may produce pseudo texture, residual noise and color cast problems, so it is difficult to obtain real paired data. Generating false paired data does not require complete stillness, but obtaining clean data in extremely dim light scenes may still produce pseudo texture, residual noise and color cast problems.

[0005] In addition, the commonly used terms in the prior art include:

[0006] aw domain: The original data output by the image sensor is called raw data. Image Signal Processing (ISP) performs signal processing on the original raw image data output by the image sensor. After the demosaic interpolation algorithm, it will be converted to the rgb domain. The domain before converting to the rgb domain is called the raw domain. Image noise: Image noise mainly refers to the rough part of the image generated in the process of CCD or CMOS receiving light as a signal and outputting it. It also refers to foreign pixels that should not appear in the image, usually caused by electronic interference.

[0007] Noisy paired data: Neural network training requires paired data, generally input and label. The input of noisy paired data is noisy data, and the label is clean data without noise. Summary of the invention

[0008] In order to solve the above problems, the purpose of this application is to generate pseudo-pairing data according to the existing noise modeling method, darken the bright image through linear transformation to simulate the extremely dark light scene, reduce the pseudo texture and residual noise of obtaining clean data, and at the same time enhance and expand the data diversity through series data to improve the generalization ability of neural networks.

[0009] Specifically, the present invention provides a method for generating raw domain noise paired data under extremely dark light, the method comprising the following steps:

[0010] S1, collect noise modeling data and calibrate noise parameters: collect the data required for noise modeling, including a black frame with black tape completely covering the lens and a flat frame with flat white paper, calibrate the noise generation related parameters p, and randomly generate noise according to the related parameters. The formula for adding noise is shown in formula (1), where I N is the noise image, I C is a clean image, G is the noise generation function, p is the calibrated noise generation parameter,

[0011] I N =I C +G(I C ,p) Formula (1)

[0012] S2, collect and create clean data: In indoor and outdoor scenes with rich details and good lighting conditions, collect raw data with less original noise, and process it to obtain clean raw data. C ;

[0013] S3, design random brightness mapping strategy:

[0014] S3.1, the original raw data is normalized after subtracting the black level, as shown in formula (2), where I is the original raw data, I norm is the normalized data, BLC is the black level, and n is the bit width;

[0015]

[0016] S3.2, light source area mask generation, binarize the normalized data I with a threshold of 0.8, and dilate the binarized data by 5×5 to obtain the light source area mask, as shown in formula (3), where T is the binarization function and D is the dilation function;

[0017] mask = D(T(0.8)) Formula (3);

[0018] S3.3, random brightness mapping, divide the target brightness mean interval into 4 different intervals, namely [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], [0.2, 0.4], and randomly select with probability of 0.5, 0.35, 0.15, 0.05, and uniformly sample the selected intervals to obtain the target brightness mean M c , linear compression is performed according to formula (4), where M t is the mean of the normalized data in step S3.1, I t The data after brightness mapping;

[0019]

[0020] S3.4 Generate brightness mapping data I according to formula (5) f ,

[0021] I f =I t (1-mask)+I norm ·Mask formula (5)

[0022] S4, design mosaic data enhancement: set the mosaic area and the number of areas, randomly extract and randomly crop each enhancement to complete the mosaic data enhancement;

[0023] S5, design flip rotation data enhancement: randomly select one of multiple methods during operation to increase diversity;

[0024] S6, generate noise data: map the brightness data I f The clean data I generated after the operations of step S4 and step S5 are performed in sequence C Then add noise according to formula (1) to obtain the noise data I N ;

[0025] S7, design color cast perturbation strategy:

[0026] S7.1, for the noise data I in step S6 N To perform color cast perturbation, firstly N The Gr channel is sampled with a uniform distribution from -0.0001 to 0.0001, and the sampling result is added to the noise data I N The Gr channel sampling result is multiplied by the uniformly distributed sampling from -0.02 to 0.02 as the Gb channel noise and added to the noise data I N The Gb channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the R channel noise and added to the noise data I NThe R channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the B channel noise and added to the noise data I N On the B channel, the color cast disturbance noise data I is obtained N1 ;

[0027] S7.2, the noise data I after the color cast disturbance in step S7.1 N1 Perform threshold segmentation, add 0.000096, 0.000072, 0.000048, 0.000024 color cast compensation to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], [0.0036, 0.0048] respectively, and finally obtain the noise data I N2 ;

[0028] S8, execute steps S3 to S7 in sequence, step S7.2 final noise data I N2 and the normalized data I from step S3.1 norm Constitute paired data.

[0029] The step S2 further comprises:

[0030] S2.1, in indoor and outdoor lighting conditions above 10 lux, use the image sensor to collect 2500 raw data with relatively small original noise LN Note that data including light sources need to be collected. The relatively small noise refers to scenes with lighting conditions above 10 lux, which are considered to have relatively small noise.

[0031] S2.2, using the traditional denoising algorithm BM3D to denoise the raw data collected in step S2.1 LN Process to get clean raw data I C .

[0032] The step S4 further comprises:

[0033] S4.1 Set the size of the mosaic area to 512×512, set the number of areas to 4, and randomly select a pixel point with an even width and height within 128 pixels from the center point. This point divides the mosaic area into upper left, upper right, lower left, and lower right.

[0034] S4.2 Each time enhancement is performed, 4 data are randomly selected from the data set, and the 4 data are randomly cropped into corresponding shapes and sizes at the upper left, upper right, lower left, and lower right. The cropping starting point pixels are all even numbers, and they are filled into the corresponding areas to complete the mosaic data enhancement.

[0035] The step S5 further comprises:

[0036] When S5.1 is flipped upside down or rotated 180° clockwise, the second to last rows of the original data are intercepted and filled into the first to the second to last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second to last row of the original data, and then the pixels of the odd and even columns of the last row are swapped.

[0037] S5.2 When flipping left and right or rotating 90° clockwise, the second to last columns of the original data are intercepted and filled into the first to the second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped.

[0038] S5.3 randomly selects any one of steps S5.1 and S5.2 to generate data, that is, when generating data, randomly selects one of the upside-down flipping, left-right flipping, 90° clockwise rotation, and 180° clockwise rotation each time.

[0039] The image sensor used in the method is imx327.

[0040] Therefore, the advantages of this application are: based on the existing noise modeling method, the easily obtained clean data is processed to avoid the cumbersome generation of real paired data in extremely dark light scenes and the possible existence of pseudo texture, residual noise and color cast problems, while using data enhancement to expand data diversity and improve the generalization ability of the model. This method is applicable to Beijing Ingenic's chip T41AIISP, in the field of image signal processing, to improve the noise level in extremely dark light scenes and improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0042] Figure 1 It is a schematic diagram of the process of the present application method. DETAILED DESCRIPTION

[0043] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0044] like Figure 1 As shown, this application proposes a method for generating raw domain noise paired data under extremely dark light, the image sensor used is imx327, and the main implementation steps of the method are as follows:

[0045] Step S1, collecting noise modeling data and calibrating noise parameters: collecting the data required for noise modeling, the required data includes a black frame using black tape to completely cover the lens and a flat frame of flat white paper, calibrating the noise generation related parameters p, and randomly generating noise according to the related parameters, as shown in formula (1) for adding noise, where I N is the noise image, I C is a clean image, G is the noise generation function, and p is the calibrated noise generation parameter.

[0046] I N =I C +G(I C ,p) Formula (1)

[0047] Step S2, collect and create clean data:

[0048] S2.1, in scenes with rich details and good lighting conditions, such as indoor and outdoor scenes with lighting conditions above 10 lux, use the image sensor to collect 2500 raw data with relatively small original noise. LN , please note that data including light sources need to be collected; the relatively small noise refers to scenes with lighting conditions above 10lux, which are considered to have relatively small noise;

[0049] S2.2, use the traditional denoising algorithm BM3D to denoise the raw data collected in step 2.1 LN Process to get clean raw data I C .

[0050] Step S3, design a random brightness mapping strategy:

[0051] S3.1, the original raw data is normalized after subtracting the black level, as shown in formula (2), where I is the original raw data, I norm is the normalized data, BLC is the black level, and n is the bit width;

[0052]

[0053] S3.2, light source area mask generation, binarize the normalized data I with a threshold of 0.8, and dilate the binarized data by 5×5 to obtain the light source area mask, as shown in formula (3), where T is the binarization function and D is the dilation function;

[0054] mask=D(T(0.8)) Formula (3)

[0055] S3.3, random brightness mapping, divide the target brightness mean interval into 4 different intervals, namely [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], [0.2, 0.4], and randomly select with probability of 0.5, 0.35, 0.15, 0.05, and uniformly sample the selected intervals to obtain the target brightness mean M c , linear compression is performed according to formula (4), where M t is the mean of the normalized data in step 3.1, I t The data after brightness mapping;

[0056]

[0057] S3.4 Generate brightness mapping data I according to formula (5) f ,

[0058] I f =I t (1-mask)+I norm ·Mask formula (5)

[0059] Step S4, design mosaic data enhancement:

[0060] S4.1, set the size of the mosaic area to 512×512, set the number of areas to 4, and randomly select a pixel point with an even width and height within 128 pixels from the center point. This point divides the mosaic area into upper left, upper right, lower left, and lower right.

[0061] S4.2, each time the enhancement is performed, 4 data are randomly selected from the data set, and the 4 data are randomly cropped into corresponding shapes and sizes at the upper left, upper right, lower left, and lower right. The cropping starting point pixels are all even numbers, and they are filled into the corresponding areas to complete the mosaic data enhancement;

[0062] Step S5, design flip rotation data enhancement:

[0063] When S5.1 is flipped upside down or rotated 180° clockwise, the second to last rows of the original data are intercepted and filled into the first to the second to last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second to last row of the original data, and then the pixels of the odd and even columns of the last row are swapped.

[0064] S5.2 When flipping left and right or rotating 90° clockwise, the second to last columns of the original data are intercepted and filled into the first to the second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped.

[0065] S5.3 randomly selects any one of steps S5.1 and S5.2 to generate data, that is, when generating data, randomly selects one of the upside-down flip, left-right flip, 90° clockwise rotation, and 180° clockwise rotation each time;

[0066] Step S6, generating noise data: mapping the brightness data I f The clean data I generated after the operations of step S4 and step S5 are performed in sequence C Then add noise according to formula (1) to obtain the noise data I N ;

[0067] Step S7, designing a color cast disturbance strategy:

[0068] S7.1, for the noise data I in step S6 N To perform color cast perturbation, firstly N The Gr channel is sampled with a uniform distribution from -0.0001 to 0.0001, and the sampling result is added to the noise data I N The Gr channel sampling result is multiplied by the uniformly distributed sampling from -0.02 to 0.02 as the Gb channel noise and added to the noise data I N The Gb channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the R channel noise and added to the noise data I N The R channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the B channel noise and added to the noise data I N On the B channel, the color cast disturbance noise data I is obtained N1 ;

[0069] S7.2, the noise data after color cast disturbance in S7.1 I N1 Perform threshold segmentation, add 0.000096, 0.000072, 0.000048, 0.000024 color cast compensation to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], [0.0036, 0.0048] respectively, and finally obtain the noise data I N2 .

[0070] Step S8, execute steps S3 to S7 in sequence, step S7.2 the final noise data I N2 and the normalized data I from step S3.1 norm Constitute paired data.

[0071] This application does not involve training models, only data generation methods.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating raw domain noise paired data under extremely dark light, characterized in that: The method comprises the following steps: S1, collect noise modeling data and calibrate noise parameters: collect the data required for noise modeling, including a black frame with black tape completely covering the lens and a flat frame with flat white paper, calibrate the noise generation related parameters p, and randomly generate noise according to the related parameters. The formula for adding noise is shown in formula (1), where I N is the noise image, I C is a clean image, G is the noise generation function, p is the calibrated noise generation parameter, I N = I C + G(I C , p) Equation (1); S2, collect and create clean data: In indoor and outdoor scenes with rich details and good lighting conditions, collect raw data with less original noise, and process it to obtain clean raw data. C ; S3, design random brightness mapping strategy: S3.1, the original raw data is normalized after subtracting the black level, as shown in formula (2), where I is the original raw data, I norm is the normalized data, BLC is the black level, and n is the bit width; S3.2, light source area mask generation, binarize the normalized data I with a threshold of 0.8, and dilate the binarized data by 5×5 to obtain the light source area mask, as shown in formula (3), where T is the binarization function and D is the dilation function; mask=D(T(08)) Formula (3); S3.3, random brightness mapping, divide the target brightness mean interval into 4 different intervals, namely [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], [0.2, 0.4], and randomly select with probability of 0.5, 0.35, 0.15, 0.05, and uniformly sample the selected intervals to obtain the target brightness mean M c , linear compression is performed according to formula (4), where M t is the mean of the normalized data in step S3.1, I t It is the data after brightness mapping; S3.4 Generate brightness mapping data I according to formula (5) f , I f = I t · (1 - mask)+I norm · mask Equation (5); S4, design mosaic data enhancement: set the mosaic area and the number of areas, randomly extract and randomly crop each enhancement to complete the mosaic data enhancement; S5, design flip rotation data enhancement: randomly select one of multiple methods during operation to increase diversity; S6, generate noise data: map the brightness data I f The clean data I generated after the operations of step S4 and step S5 are performed in sequence C Then add noise according to formula (1) to obtain the noise data I N ; S7, design color cast perturbation strategy: S7.1, for the noise data I in step S6 N To perform color cast perturbation, firstly N The Gr channel is sampled with a uniform distribution from -0.0001 to 0.0001, and the sampling result is added to the noise data I N The Gr channel sampling result is multiplied by the uniformly distributed sampling from -0.02 to 0.02 as the Gb channel noise and added to the noise data I N The Gb channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the R channel noise and added to the noise data I N The R channel of the Gr channel is multiplied by the uniformly distributed sampling of 0.45 to 0.65 as the B channel noise and added to the noise data I N On the B channel, the color cast disturbance noise data I is obtained N1 ; S7.2, the noise data I after the color cast disturbance in step S7.1 N1 Perform threshold segmentation, add color cast compensation of 0.000096, 0.000072, 0.000048, and 0.000024 to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], and [0.0036, 0.0048], respectively, and finally obtain the noise data I N2 ; S8, execute steps S3 to S7 in sequence, step S7.2 final noise data I N2 and the normalized data I from step S3.1 norm Constitute paired data.

2. The method for generating raw domain noise paired data under extremely dark light according to claim 1, characterized in that: The step S2 further comprises: S2.1, in indoor and outdoor lighting conditions above 10 lux, use the image sensor to collect 2500 raw data with relatively small original noise LN Note that data including light sources need to be collected. The relatively small noise refers to the relatively small noise when the lighting condition is above 10 lux. S2.2, using the traditional denoising algorithm BM3D to denoise the raw data collected in step S2.1 LN Process to get clean raw data I C .

3. The method for generating raw domain noise paired data under extremely dark light according to claim 1, characterized in that: The step S4 further comprises: S4.1 Set the size of the mosaic area to 512×512, set the number of areas to 4, and randomly select a pixel point with an even width and height within 128 pixels from the center point. This point divides the mosaic area into upper left, upper right, lower left, and lower right. S4.2 Each time enhancement is performed, 4 data are randomly selected from the data set, and the 4 data are randomly cropped into corresponding shapes and sizes at the upper left, upper right, lower left, and lower right. The cropping starting point pixels are all even numbers, and they are filled into the corresponding areas to complete the mosaic data enhancement.

4. The method for generating raw domain noise paired data under extremely dark light according to claim 1, characterized in that: The step S5 further comprises: When S5.1 is flipped upside down or rotated 180° clockwise, the second to last rows of the original data are intercepted and filled into the first to the second to last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second to last row of the original data, and then the pixels of the odd and even columns of the last row are swapped. S5.2 When flipping left and right or rotating 90° clockwise, the second to last columns of the original data are intercepted and filled into the first to the second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped. S5.3 randomly selects any one of steps S5.1 and S5.2 to generate data, that is, when generating data, randomly selects one of the upside-down flipping, left-right flipping, 90° clockwise rotation, and 180° clockwise rotation each time.

5. The method for generating raw domain noise paired data under extremely dark light according to claim 1, characterized in that: The image sensor used in the method is imx327.

Images