A method for generating raw domain noise pairing data under extremely dark light

Abstract

This invention provides a method for generating raw domain noise-paired data under extremely low light conditions, comprising: S1, collecting noise modeling data and calibrating noise parameters; S2, collecting and creating clean data; S3, designing a random brightness mapping strategy; S4, designing mosaic data augmentation; S5, designing flip-rotation data augmentation; S6, generating noise data; S7, designing a color distortion perturbation strategy; S8, during training, executing steps S3 to S7 sequentially, resulting in the final noise data I. N2 and normalized data I norm This method constructs paired data. Based on existing noise modeling methods, it processes easily obtainable clean data to avoid the cumbersome generation process of real paired data in extremely low light scenes and the possible problems of pseudo textures, residual noise, and color bias. At the same time, it uses data augmentation to expand the diversity of data and improve the generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of intelligent surveillance video processing technology, and specifically relates to a method for generating raw domain noise-paired data under extremely low light conditions. Background Technology

[0002] In current technologies, neural network-based denoising methods are significantly better than traditional denoising methods in extremely low-light scenes, with fewer pseudo-textures and residual noise. However, training a good neural network requires a large amount of data.

[0003] Currently, there are two main types of noise-paired data acquisition. The first is real-paired data, which is usually the direct output of the image sensor as noise data, and then clean data is obtained through denoising algorithms. The second is generating fake-paired data. There are already methods to model the noise formation process, including photon shot noise, line noise, readout noise, etc., and noise data is obtained by adding these noises to clean data.

[0004] However, obtaining clean data in extremely low-light scenes is difficult. It usually requires collecting many frames, weighting and averaging to reduce noise, and then removing it through a denoising algorithm. True pairing data requires the scene to be completely still, and denoising may produce false textures, residual noise, and color cast issues, so obtaining true pairing data is quite difficult. Generating false pairing data does not require the scene to be completely still, but obtaining clean data in extremely low-light scenes may still produce false textures, residual noise, and color cast issues.

[0005] In addition, commonly used terms in the prior art include:

[0006] Raw domain: The raw data output by the image sensor is called raw data. Image Signal Processing (ISP) performs signal processing on the raw image data output by the image sensor. After the demosaic interpolation algorithm, it will be converted to the RGB domain. The domain before being converted to the RGB domain is called the raw domain.

[0007] Image noise: Image noise mainly refers to the rough parts in the image generated during the process of CCD or CMOS receiving light as a signal and outputting it. It also refers to foreign pixels that should not appear in the image, usually caused by electronic interference.

[0008] Noisy paired data: Neural network training requires paired data, usually input and label. The input of noisy paired data is noisy data, and the label is clean data without noise. Summary of the Invention

[0009] To address the aforementioned issues, the purpose of this application is to generate pseudo-paired data based on existing noise modeling methods, darken bright images through linear transformation to simulate extremely dark light scenes, reduce pseudo-textures and residual noise in obtaining clean data, and enhance the generalization ability of neural networks by expanding data diversity through a series of data enhancements.

[0010] Specifically, the present invention provides a method for generating raw domain noise-paired data under extremely low light conditions, the method comprising the following steps:

[0011] S1, Collect noise modeling data and calibrate noise parameters: Collect the data required for noise modeling, including a black frame with the lens completely covered by black tape and a flat frame on a flat piece of white paper, and calibrate the relevant parameters for noise generation. Noise can be randomly generated based on relevant parameters, as shown in equation (1), which is the formula for adding noise. For noisy images, For a clean image, Let be the noise generation function. For the calibrated noise generation parameters,

[0012] Equation (1);

[0013] S2, Acquire and Process Clean Data: In indoor and outdoor scenes with rich detail and good lighting conditions, acquire raw data with low noise and process it to obtain clean raw data. ;

[0014] S3, Design a random brightness mapping strategy:

[0015] S3.1, the original raw data is normalized after subtracting the black level, as shown in equation (2), where This is the original raw data. For the normalized data, Black level. For bit width;

[0016] Equation (2);

[0017] S3.2, Light source region mask generation, normalized data with a threshold of 0.8. Binarization is performed, and the binarized data is dilated by 5×5 to obtain the light source region mask, as shown in equation (3), where It is a binary function. It is the expansion function;

[0018] Equation (3);

[0019] S3.3, Random Brightness Mapping: The target brightness mean range is divided into four different intervals: [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], and [0.2, 0.4]. These intervals are randomly selected with probabilities of 0.5, 0.35, 0.15, and 0.05. The target brightness mean M is obtained by uniformly sampling from each selected interval. c Linear compression is performed according to equation (4), where The mean of the data after normalization in step S3.1, This is the data after brightness mapping;

[0020] Equation (4);

[0021] S3.4 Generate luminance mapping data I according to formula (5) f ,

[0022] Equation (5);

[0023] S4, Design mosaic data augmentation: Set the mosaic region and the number of regions, randomly select and crop the region for each augmentation to complete the mosaic data augmentation;

[0024] S5 features enhanced flip and rotation data design: during operation, one of multiple methods is randomly selected to increase diversity;

[0025] S6, Generate noise data: For the brightness mapping data I... f Clean data generated after sequentially performing steps S4 and S5. Then, noise is added according to equation (1) to obtain noise data. ;

[0026] S7, designing a color cast perturbation strategy:

[0027] S7.1, regarding the noise data from step S6 To perform color distortion perturbation, first, the noisy data... The Gr channel is sampled using a uniform distribution from -0.0001 to 0.0001, and the sampling results are added to the noise data. On the Gr channel; multiply the Gr channel sampling result by a uniformly distributed sample of -0.02 to 0.02 and add it as Gb channel noise to the noise data. On the Gb channel; the Gr channel sampling result is multiplied by a uniformly distributed sample of 0.45 to 0.65 and added to the noise data as R channel noise. On the R channel; multiply the Gr channel sampling result by a uniformly distributed sample of 0.45 to 0.65 and add it as B channel noise to the noise data. On the B channel, color distortion noise data was obtained. ;

[0028] S7.2, the noise data after color distortion perturbation in step S7.1 Threshold segmentation is performed, and color compensation of 0.000096, 0.000072, 0.000048, and 0.000024 is added to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], and [0.0036, 0.0048] respectively, to finally obtain the noisy data. ;

[0029] S8, execute steps S3 to S7 sequentially, step S7.2 final noise data. and the normalized data from step S3.1 This constitutes paired data.

[0030] Step S2 further includes:

[0031] S2.1, In scenes with indoor or outdoor lighting conditions above 10 lux, use an image sensor to acquire 2500 raw data images with relatively low noise. Note that data including light sources needs to be collected. The term "relatively low noise" refers to a scenario where the illumination conditions are above 10 lux.

[0032] S2.2, use the traditional denoising algorithm BM3D to process the raw data acquired in step S2.1. Process to obtain clean raw data .

[0033] Step S4 further includes:

[0034] S4.1 Set the size of the mosaic region to 512×512 and the number of regions to 4. Randomly select a pixel with an even number of width and height within 128 pixels away from the center point. This pixel divides the mosaic region into upper left, upper right, lower left, and lower right.

[0035] S4.2 For each augmentation, four data points are randomly selected from the dataset. These four data points are then randomly cropped into corresponding shapes and sizes for the top left, top right, bottom left, and bottom right regions, with the starting pixel number being an even number. These cropping points are then filled into the corresponding regions to complete the mosaic data augmentation.

[0036] Step S5 further includes:

[0037] When S5.1 flips vertically or rotates 180° clockwise, the second to last rows of the original data are extracted and filled into the first to second-to-last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second-to-last row of the original data, and then the odd and even columns of the last row are swapped.

[0038] When S5.2 flips left or right or rotates 90° clockwise, the second to last columns of the original data are extracted and filled into the first to second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped.

[0039] S5.3 Randomly select any one of steps S5.1 and S5.2 to generate data, that is, when generating data, each time, randomly select one of the following: upside down flip, left and right flip, 90° clockwise rotation, or 180° clockwise rotation.

[0040] The image sensor used in the method is an IMX327.

[0041] Therefore, the advantages of this application are: based on existing noise modeling methods, it processes easily obtainable clean data, avoiding the cumbersome generation of real-world paired data in extremely low-light scenes and potential issues such as pseudo-textures, residual noise, and color cast. Simultaneously, it uses data augmentation to expand data diversity and improve the model's generalization ability. This method is applicable to the Beijing Junzheng T41 AIIPS chip, in the field of image signal processing, aiming to improve noise levels and enhance image quality in extremely low-light scenes. Attached Figure Description

[0042] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0043] Figure 1 This is a schematic diagram of the method flow of this application. Detailed Implementation

[0044] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0045] like Figure 1 As shown, this application proposes a method for generating raw domain noise-paired data under extremely low light conditions. The image sensor used is an IMX327. The main implementation steps of the method are as follows:

[0046] Step S1: Collect noise modeling data and calibrate noise parameters: Collect the data required for noise modeling, including a black frame with the lens completely covered by black tape and a flat frame on a flat piece of white paper, and calibrate the relevant parameters for noise generation. Noise can be randomly generated based on relevant parameters, as shown in equation (1), which is the formula for adding noise. For noisy images, For a clean image, Let be the noise generation function. These are the calibrated noise generation parameters.

[0047] Equation (1)

[0048] Step S2, collect and prepare clean data:

[0049] S2.1, In scenes with rich detail and good lighting conditions, such as indoor and outdoor environments (i.e., indoor and outdoor lighting conditions above 10 lux), 2500 raw data images with relatively low noise are acquired using an image sensor. Note that data including light sources needs to be collected; the term "relatively low noise" refers to scenarios where the illumination conditions are above 10 lux.

[0050] S2.2, use the traditional denoising algorithm BM3D to process the raw data acquired in step 2.1. Process to obtain clean raw data .

[0051] Step S3, design a random brightness mapping strategy:

[0052] S3.1, The raw data is normalized after subtracting the black level, as shown in equation (2), where This is the original raw data. For the normalized data, Black level. For bit width;

[0053] Equation (2)

[0054] S3.2, Light source region mask generation, normalized data with a threshold of 0.8. Binarization is performed, and the binarized data is dilated by 5×5 to obtain the light source region mask, as shown in equation (3), where It is a binary function. It is the expansion function;

[0055] Equation (3)

[0056] S3.3, Random Brightness Mapping: The target brightness mean range is divided into four different intervals: [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], and [0.2, 0.4]. These intervals are randomly selected with probabilities of 0.5, 0.35, 0.15, and 0.05. The target brightness mean M is obtained by uniformly sampling from each selected interval. c Linear compression is performed according to equation (4), where The mean of the data after normalization in step 3.1, This is the data after brightness mapping;

[0057] Equation (4)

[0058] S3.4 Generate luminance mapping data I according to formula (5) f ,

[0059] Equation (5);

[0060] Step S4, Design mosaic data augmentation:

[0061] S4.1, set the size of the mosaic region to 512×512, set the number of regions to 4, and randomly select a pixel with an even number of width and height within a range of 128 pixels from the center point. This pixel divides the mosaic region into upper left, upper right, lower left, and lower right.

[0062] S4.2, Each augmentation involves randomly selecting 4 data points from the dataset, randomly cropping the 4 data points into corresponding shapes and sizes (top left, top right, bottom left, bottom right), with the starting pixel number being an even number, and filling them into the corresponding regions to complete the mosaic data augmentation.

[0063] Step S5, Design Flip and Rotation Data Augmentation:

[0064] When S5.1 flips vertically or rotates 180° clockwise, the second to last rows of the original data are extracted and filled into the first to second-to-last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second-to-last row of the original data, and then the odd and even columns of the last row are swapped.

[0065] When S5.2 flips left or right or rotates 90° clockwise, the second to last columns of the original data are extracted and filled into the first to second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped.

[0066] S5.3 Randomly select any one of steps S5.1 and S5.2 to generate data, that is, when generating data, each time, randomly select one of the following: vertical flip, horizontal flip, 90° clockwise rotation, or 180° clockwise rotation.

[0067] Step S6, Generate noise data: For the brightness mapping data I f Clean data generated after sequentially performing steps S4 and S5. Then, noise is added according to equation (1) to obtain noise data. ;

[0068] Step S7, Design a color deviation perturbation strategy:

[0069] S7.1, regarding the noise data from step S6 To perform color distortion perturbation, first, the noisy data... The Gr channel is sampled using a uniform distribution from -0.0001 to 0.0001, and the sampling results are added to the noise data. On the Gr channel; multiply the Gr channel sampling result by a uniformly distributed sample of -0.02 to 0.02 and add it as Gb channel noise to the noise data. On the Gb channel; the Gr channel sampling result is multiplied by a uniformly distributed sample of 0.45 to 0.65 and added to the noise data as R channel noise. On the R channel; multiply the Gr channel sampling result by a uniformly distributed sample of 0.45 to 0.65 and add it as B channel noise to the noise data. On the B channel, color distortion noise data was obtained. ;

[0070] S7.2, Noise data after color distortion perturbation in S7.1 Threshold segmentation is performed, and color compensation of 0.000096, 0.000072, 0.000048, and 0.000024 is added to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], and [0.0036, 0.0048] respectively, to finally obtain the noisy data. .

[0071] Step S8: Execute steps S3 to S7 sequentially. Step S7.2: Final noise data. and the normalized data from step S3.1 This constitutes paired data.

[0072] This application does not involve training models, but only includes data generation methods.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating raw domain noise-paired data under extremely low light conditions, characterized in that, The method includes the following steps: S1, Collect noise modeling data and calibrate noise parameters: Collect the data required for noise modeling, including a black frame with the lens completely covered by black tape and a flat frame on a flat piece of white paper, and calibrate the noise generation parameters. Noise can be randomly generated based on relevant parameters, as shown in equation (1), which is the formula for adding noise. For noisy data, For clean raw data, Let be the noise generation function. For the calibrated noise generation parameters, Equation (1); S2, Acquire and Process Clean Data: In indoor and outdoor scenes with rich detail and good lighting conditions, acquire raw data with low noise and process it to obtain clean raw data. ; Step S2 further includes: S2.1, In scenes with indoor or outdoor lighting conditions above 10 lux, use an image sensor to acquire 2500 raw data images with relatively low noise. It is necessary to collect data including light source, and the noise is considered to be relatively low when the illumination conditions are above 10 lux; S2.2, use the traditional denoising algorithm BM3D to process the raw data acquired in step S2.

1. Process to obtain clean raw data ; S3, Design a random brightness mapping strategy: S3.1, the original raw data is normalized after subtracting the black level, as shown in equation (2), where This is the original raw data. For the normalized data, Black level. For bit width; Equation (2); S3.2, Light source region mask generation, normalized data with a threshold of 0.

8. Binarization is performed, and the binarized data is dilated by 5×5 to obtain the light source region mask, as shown in equation (3), where It is a binary function. It is the expansion function; Equation (3); S3.3, Random Brightness Mapping: The target brightness mean range is divided into four different intervals: [0.001, 0.01], [0.01, 0.1], [0.1, 0.2], and [0.2, 0.4]. These intervals are randomly selected with probabilities of 0.5, 0.35, 0.15, and 0.

05. The target brightness mean M is obtained by uniformly sampling from each selected interval. c Linear compression is performed according to equation (4), where The mean of the data after normalization in step S3.1, This is the data after brightness mapping; Equation (4); S3.4 Generating luminance mapping data I according to formula (5) f , Equation (5); S4, Design mosaic data augmentation: Set the mosaic region and the number of regions, randomly select and crop the region for each augmentation to complete the mosaic data augmentation; S5, designing data enhancement for flipping and rotation: randomly selecting one of multiple methods during operation to increase diversity; step S5 further includes: When S5.1 flips vertically or rotates 180° clockwise, the second to last rows of the original data are extracted and filled into the first to second-to-last rows of the enhanced data. The last row of the enhanced data is first supplemented with the second-to-last row of the original data, and then the odd and even columns of the last row are swapped. When S5.2 flips left or right or rotates 90° clockwise, the second to last columns of the original data are extracted and filled into the first to second to last columns of the enhanced data. The last column of the enhanced data is first supplemented with the second to last column of the original data, and then the pixels of the odd and even rows of the last column are swapped. S5.3 Randomly select any one of steps S5.1 and S5.2 to generate data, that is, when generating data, each time, randomly select one of the following: vertical flip, horizontal flip, 90° clockwise rotation, or 180° clockwise rotation. S6, Generate noise data: For the brightness mapping data I... f The clean raw data generated after sequentially performing steps S4 and S5 Then, noise is added according to equation (1) to obtain noise data. ; S7, designing a color cast perturbation strategy: S7.1, regarding the noise data from step S6 To perform color distortion perturbation, first, the noisy data... The Gr channel is sampled using a uniform distribution from -0.0001 to 0.0001, and the sampling results are added to the noise data. On the Gr channel; multiply the Gr channel sampling result by a uniformly distributed sample of -0.02 to 0.02 and add it as Gb channel noise to the noise data. On the Gb channel; the Gr channel sampling result is multiplied by a uniformly distributed sample of 0.45 to 0.65 and added to the noise data as R channel noise. On the R channel; multiply the Gr channel sampling result by a uniformly distributed sample of 0.45 to 0.65 and add it as B channel noise to the noise data. On the B channel, color distortion noise data was obtained. ; S7.2, the noise data after color distortion perturbation in step S7.1 Threshold segmentation is performed, and color compensation of 0.000096, 0.000072, 0.000048, and 0.000024 is added to the data distributed in [0, 0.001], [0.001, 0.0024], [0.0024, 0.0036], and [0.0036, 0.0048] respectively, to finally obtain the noisy data. ; S8, execute steps S3 to S7 sequentially, step S7.2 final noise data. and the normalized data from step S3.1 This constitutes paired data.

2. The method for generating raw domain noise-paired data under extremely low light conditions according to claim 1, characterized in that, Step S4 further includes: S4.1 Set the size of the mosaic region to 512×512 and the number of regions to 4. Randomly select a pixel with an even number of width and height within 128 pixels away from the center point. This pixel divides the mosaic region into upper left, upper right, lower left, and lower right. S4.2 For each augmentation, four data points are randomly selected from the dataset. These four data points are then randomly cropped into corresponding shapes and sizes for the top left, top right, bottom left, and bottom right regions, with the starting pixel number being an even number. These cropping points are then filled into the corresponding regions to complete the mosaic data augmentation.

3. The method for generating raw domain noise-paired data under extremely low light conditions according to claim 1, characterized in that, The image sensor used in the method is an IMX327.

Images