Optimization method for large noise points in spatial domain denoising under extremely dark light

Abstract

The invention provides an optimization method for large noise points in airspace denoising under extremely dark light. The optimization method comprises the following steps: S1, acquiring noise modeling data and calibrating noise parameters; s2, calibrating fixed mode noise; s3, preparing clean training data; s4, designing a bright and dead pixel generation method; s5, designing a dark and dead pixel generation method; s6, generating pairing data; s7, training a neural network model; and S8, model reasoning. According to the method, an existing noise modeling method is optimized, dead pixel correction and fixed pattern noise removal are combined according to the characteristics of the large noise in the extremely dark light scene, more real pairing data containing the large noise are generated, the neural network can better deal with the real large noise scene, and the denoising capability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent monitoring video processing, and in particular relates to an optimization method for denoising large noise points in the airspace under extremely dark light. Background Art

[0002] In the existing intelligent surveillance video processing technology, in extremely dark light scenes, due to the limited number of photons, in order to achieve the target brightness, the image sensor analog gain is usually maximized, and the image sensor faces serious noise problems. Some image sensors will produce many large noise points due to process problems. Generally, bad pixel correction and spatial denoising are used to remove large noise points.

[0003] At present, bad pixel correction usually uses traditional methods. For static bad pixels, data is used for calibration, and for dynamic bad pixels, the brightness difference between bad pixels and surrounding pixels is generally judged for correction.

[0004] At present, traditional spatial denoising methods include bilateral filtering, non-local mean filtering, BM3d algorithm, etc.; At present, spatial denoising algorithms based on neural networks include methods based on real paired data and generated paired data.

[0005] However, the current bad pixel correction requires static calibration in advance for static bad pixels, and dynamic bad pixels are prone to detail loss due to threshold setting; the current traditional spatial denoising is prone to produce pseudo textures and detail loss in extremely dark light scenes. Although the neural network-based spatial denoising has obvious effects, the cost of producing real paired data is relatively high, and the noise of the existing generated paired data is different from the real data, which makes it difficult to remove large noise points.

[0006] In addition, the commonly used terms in the prior art include:

[0007] Image noise: Image noise mainly refers to the rough part of the image produced in the process of CCD or CMOS receiving light as a signal and outputting it. It also refers to foreign pixels that should not appear in the image, usually caused by electronic interference.

[0008] Spatial denoising: denoising a single frame image in the spatial domain.

[0009] Large noise points: In extremely dark light scenes, the analog gain of the image sensor is usually maximized. When the analog gain is maximized, a large number of large noise points will be generated. The large noise points may be static bad pixels or dynamic bad pixels, or fixed pattern noise caused by deviations in the size of the photodiode, the parameters of the MOS field effect tube, etc. Summary of the invention

[0010] In order to solve the above problems, the purpose of this application is to: based on a neural network method, combine the bad pixel correction principle and the causes of large noise points, use noise modeling to generate more realistic noise, so that the neural network can complete the bad pixel correction function while completing spatial denoising, and effectively remove large noise points.

[0011] Specifically, the present invention provides an optimization method for denoising large noise points in the spatial domain under extremely dark light, and the method comprises the following steps:

[0012] S1, collect the data required for noise modeling, including the black frame with black tape completely covering the lens and the flat frame of flat white paper, calibrate the noise generation related parameters p, and randomly generate noise according to the related parameters. The formula for adding noise is shown in formula (1), where I N0 is the noise data, I C For clean data, G p is the Poisson noise generating function, G r is the readout noise generation function, p is the calibrated noise generation parameter,

[0013] I N0 =I C +G p (I C , p)+G γ (I C , p) Formula (1);

[0014] S2, calibrated fixed pattern noise:

[0015] S2.1, collect fixed pattern noise data I d , seal the image sensor with black tape in a dark room without light, and collect 500 frames of data at an analog gain of 54 again;

[0016] S2.2, according to formula (2), the 500 frames of fixed pattern noise data collected under the analog gain again of 54 are weighted averaged to obtain the fixed pattern noise under the analog gain, where fpn is the fixed pattern noise, I di is the i-th frame data,

[0017]

[0018] S3, prepare clean training data, crop the clean data to a size of 512×512, with a width and height step of 256, and normalize it according to formula (3), where I is the cropped clean data, I norm is the normalized data, BLC is the black level, and n is the bit width;

[0019]

[0020] S4, design bright and bad pixel generation, including:

[0021] S4.1, for I norm The mean is segmented to determine the number of bright and bad points to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5;

[0022] S4.2, for each bad pixel, first randomly select the bad pixel coordinate position, and calculate the mean value within 10 of the neighborhood around the bad pixel coordinate position. If the mean value is greater than 0.05, reselect the bad pixel coordinate position. If the reselection is repeated for more than 5 times, the bad pixel generation will be discarded. If the mean value is less than 0.05, sample the bad pixel value uniformly in the range of 0.1 to 1, and fill the bad pixel value into the bad pixel coordinate position.

[0023] S5, design dark pixel generation, including:

[0024] S5.1, for I norm The mean is segmented to determine the number of dark spots to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5;

[0025] S5.2, for each dark bad point, first randomly select the dark bad point coordinate position, and calculate the mean value within 10 neighborhoods around the bright bad point coordinates. If the mean value is less than 0.05, reselect the dark bad point coordinates. If the reselection is repeated for more than 5 times, the dark bad point generation will be abandoned. If the mean value is greater than 0.05, sample the dark bad point values ​​uniformly between 0 and 0.001, and fill the dark bad point into the dark bad point coordinate position;

[0026] S6, generating paired data, including:

[0027] S6.1, with a probability of 0.5, select the normalized data I norm As clean data, the noise data I is obtained by equation (1) N0 , and then generate the final noise data I through steps S4 and S5 N2 , I norm and I N2 Constitute paired data;

[0028] S6.2, with a probability of 0.5, the black frame data I d Through formula (3), we can get Idnorm , the noise data I is obtained through formula (4) NI , and then generate the final noise data I through steps S4 and S5 N2 , I norm and I N2 Constitute paired data,

[0029] I NI =I C +G p (I C , p)+I dborm Formula (4);

[0030] S7, training the neural network model. During the training, paired data is generated online according to step S6. The optimizer uses Adam, the learning rate is 0.0001, the training cycle is 120, the learning rate is reduced by 0.1 times every 20 cycles, and the training loss is L1loss;

[0031] S8, model reasoning, as shown in formula (5), where I O is the model inference result, M is the trained model, I IN To include input noise data,

[0032] I O =M(I IN -fpn) Formula (5).

[0033] The black frames in step S1 include black frames with analog gains of 1, 2, 4, 8, 32, and 54.

[0034] The image sensor used in the method is sc2210.

[0035] Therefore, the advantages of this application are: based on the optimization of the existing noise modeling method, the characteristics of large noise points in extremely dark light scenes are combined with bad pixel correction and fixed pattern noise removal to generate more realistic paired data containing large noise points, so that the neural network can better cope with real large noise scenes and improve the denoising ability. This method is applicable to Beijing Junzheng's chip T41 AIISP, the field is image signal processing, improving the noise level in extremely dark light scenes and improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0037] Figure 1 This is a schematic diagram of the application process. DETAILED DESCRIPTION

[0038] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, this application proposes an optimization method for spatial denoising of large noise points under extremely dark light, and the image sensor used is sc2210, wherein the main implementation steps of the method are as follows:

[0040] Step S1 collects the data required for noise modeling and calibrates the noise generation related parameters p. Noise can be randomly generated according to the related parameters, as shown in formula (1), where I N0 is the noise data, I c For clean data, G p is the Poisson noise generating function, G r is the readout noise generation function, and p is the calibrated noise generation parameter.

[0041] L N0 =I C +G p (I C , p)+G γ (I C , p) Formula (1);

[0042] Step S2 calibrates the fixed pattern noise:

[0043] S2.1 Collect black frame data I d , seal the image sensor with black tape in a dark room without light, and collect 500 frames of data at an analog gain of 54 again;

[0044] S2.2 According to formula (2), 500 frames of fixed pattern noise data collected under analog gain again of 54 are weighted averaged to obtain the fixed pattern noise under analog gain, where fpn is the fixed pattern noise, I di is the i-th frame data,

[0045]

[0046] Step S3 prepares clean training data, crops the clean data to a size of 512×512 with a width and height step of 256, and normalizes it according to formula (3), where I is the cropped clean data, I norm is the normalized data, BLC is the black level, and n is the bit width;

[0047]

[0048] Step S4 designs a method for generating bright and bad pixels:

[0049] S4.1 to I normThe mean is segmented to determine the number of bright and bad points to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5;

[0050] S4.2 For each bright bad point, first randomly select the coordinate position of the bright bad point, and then calculate the mean value within 10 neighborhoods around the bright bad point coordinates. If the mean value is greater than 0.05, reselect the coordinate position of the bright bad point. If the reselection is repeated for more than 5 times in a row, the generation of the bright bad point will be discarded. If the mean value is less than 0.05, sample the bright bad point values ​​uniformly in the range of 0.1 to 1, and fill the bright bad point values ​​into the coordinate position of the bright bad point.

[0051] Step S5 designs a dark pixel generation method:

[0052] S5.1 to I norm The mean is segmented to determine the number of dark spots to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5;

[0053] S5.2 For each dark bad point, first randomly select the dark bad point coordinate position, and then calculate the mean value within 10 neighborhoods around the bright bad point coordinates. If the mean value is less than 0.05, reselect the dark bad point coordinates. If the reselection is repeated more than 5 times in a row, the generation of the dark bad point will be abandoned. If the mean value is greater than 0.05, sample the dark bad point values ​​uniformly between 0 and 0.001, and fill the dark bad point into the dark bad point coordinate position.

[0054] Step S6 generates pairing data:

[0055] S6.1 selects the normalized data I with a probability of 0.5 norm As clean data, the noise data I is obtained by equation (1) N0 , and then generate the final noise data I through steps S4 and S5 N2 , I norm and I N2 Constitute paired data;

[0056] S6.2 With a probability of 0.5, the black frame data Id is obtained by equation (3) dnorm , the noise data IN1 is obtained through formula (4), and then the final noise data I is generated through steps 4 and 5 N2 , I norm and IN2 Constitute paired data;

[0057] I N1 =I C +G p (I C , p)+I dborm Mode( 4 ).

[0058] Step S7 trains the neural network model. During the training, paired data is generated online according to step S6. The optimizer uses Adam, the learning rate is 0.0001, the training cycle is 120, the learning rate is reduced by 0.1 times every 20 cycles, and the training loss is L1loss.

[0059] Step S8: Model inference, as shown in formula (5), where I O is the model inference result, M is the trained model, I IN To include input noise data,

[0060] I O =M(I IN -fpn) Formula (5).

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An optimization method for denoising large noise points in the spatial domain under extremely dark light, characterized in that: The method comprises the following steps: S1, collect the data required for noise modeling, including the black frame with black tape completely covering the lens and the flat frame of flat white paper, calibrate the noise generation related parameters p, and randomly generate noise according to the related parameters. The formula for adding noise is shown in formula (1), where I N0 is the noise data, I C For clean data, G p is the Poisson noise generating function, G r is the readout noise generation function, p is the calibrated noise generation parameter, I N0 = I C + G p (I C , p) + G r (I C , p) Equation (1); S2, calibrated fixed pattern noise: S2.1, collect fixed pattern noise data I d , seal the image sensor with black tape in a dark room without light, and collect 500 frames of data at an analog gain of 54 again; S2.2, according to formula (2), the 500 frames of fixed pattern noise data collected under the analog gain again of 54 are weighted averaged to obtain the fixed pattern noise under the analog gain, where fpn is the fixed pattern noise, I di is the i-th frame data, S3, prepare clean training data, crop the clean data to a size of 512×512, with a width and height step of 256, and normalize it according to formula (3), where I is the cropped clean data, I norm is the normalized data, BLC is the black level, and n is the bit width; S4, design bright and bad pixel generation, including: S4.1, for I norm The mean is segmented to determine the number of bright and bad points to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5; S4.2, for each bad pixel, first randomly select the bad pixel coordinate position, and calculate the mean value within 10 of the neighborhood around the bad pixel coordinate position. If the mean value is greater than 0.05, reselect the bad pixel coordinate position. If the reselection is repeated for more than 5 times, the bad pixel generation will be discarded. If the mean value is less than 0.05, sample the bad pixel value uniformly in the range of 0.1 to 1, and fill the bad pixel value into the bad pixel coordinate position. S5, design dark pixel generation, including: S5.1, for I norm The mean is segmented to determine the number of dark spots to be generated: data with a mean less than 0.1 are randomly sampled from integers uniformly distributed between 5 and 25; data with a mean less than 0.2 and greater than 0.1 are randomly sampled from integers uniformly distributed between 3 and 15; data with a mean less than 0.3 and greater than 0.2 are randomly sampled from integers uniformly distributed between 1 and 10; data with a mean greater than 0.3 are randomly sampled from integers uniformly distributed between 0 and 5; S5.2, for each dark bad point, first randomly select the dark bad point coordinate position, and calculate the mean value within 10 neighborhoods around the bright bad point coordinates. If the mean value is less than 0.05, reselect the dark bad point coordinates. If the reselection is repeated for more than 5 times, the dark bad point generation will be abandoned. If the mean value is greater than 0.05, sample the dark bad point values ​​uniformly between 0 and 0.001, and fill the dark bad point into the dark bad point coordinate position; S6, generating paired data, including: S6.1, with a probability of 0.5, select the normalized data I norm As clean data, the noise data I is obtained by equation (1) N0 , and then generate the final noise data I through steps S4 and S5 N2 , I norm and I N2 Constitute paired data; S6.2, with a probability of 0.5, the black frame data I d Through formula (3), we can get I dnorm , the noise data I is obtained through formula (4) N1 , and then generate the final noise data I through steps S4 and S5 N2 , I norm and I N2 Constitute paired data, I NI = I C + G p (I C , p) + I dnorm Equation (4); S7, training the neural network model. During the training, paired data is generated online according to step S6. The optimizer uses Adam, the learning rate is 0.0001, the training cycle is 120, the learning rate is reduced by 0.1 times every 20 cycles, and the training loss is L1loss; S8, model reasoning, as shown in formula (5), where I O is the model inference result, M is the trained model, I IN To include input noise data, I O = M(I IN - fpn) Equation (5).

2. The optimization method for removing large noise points in the spatial domain under extremely dark light according to claim 1, characterized in that: The black frames in step S1 include black frames with analog gains of 1, 2, 4, 8, 32, and 54.

3. The optimization method for removing large noise points in the spatial domain under extremely dark light according to claim 1, characterized in that: The image sensor used in the method is sc2210.

Images