A neuromorphic camera denoising method for general lighting conditions

Abstract

The application discloses a neuromorphic camera denoising method for general lighting conditions, and belongs to the field of video denoising. According to the relationship between the intensity of the pulse stream noise and the scene lighting, the continuous space-time pulse stream is converted into the picture feature representation under a specific window, and the input of the traditional neural network is adapted; the dark current noise and the response inconsistency noise are removed through the calibrated dark current matrix and the response inconsistency matrix; the image features are converted into the frequency domain features through the two-dimensional discrete wavelet transform, the low-cost noise removal is realized by using the frequency domain noise separation characteristics; the shot noise and the clipping noise are removed by fusing the frequency domain features of adjacent windows; the residual noise on the frequency domain is thoroughly removed through the deep self-attention network; finally, in order to inhibit the excessive smoothing caused by the denoising, the fusion features on the frequency domain are used to refine the denoising result. The application is suitable for the field of video denoising, is used for reconstructing a clean image, reducing the noise in the image and reducing the influence of the noise on the downstream task.

Description

TECHNICAL FIELD

[0001] The application relates to a general-purpose light condition-oriented neuromorphic camera denoising method and belongs to the field of video denoising. BACKGROUND

[0002] In recent years, according to the principle of the visual sampling mechanism of the biological retina, a series of new cameras that collect pulse array signals have become a pioneering innovation in the field of vision sensors. Compared with traditional frame-based cameras, these new sensors have significant advantages such as low motion blur, high dynamic range, high time resolution, etc., which make neuromorphic cameras show great potential in various applications such as unmanned aerial vehicles, industrial monitoring, autonomous driving, etc.

[0003] Due to the special sampling mechanism and data storage method, the application of neuromorphic cameras often needs to be matched with a suitable image reconstruction method. The existing reconstruction method focuses on scenes with sufficient light intensity to obtain more accurate and sufficient sampling information. In this case, the noise form of the neuromorphic camera is single and weak, and the influence of noise on the reconstruction result is limited. However, the scenes in the real world are not always under the best lighting conditions, and the data captured by neuromorphic cameras under low-light conditions will contain more complex noise distribution and stronger noise level, which means that the results of the existing reconstruction method are always accompanied by a large amount of noise, which is not conducive to the performance of downstream tasks. SUMMARY

[0004] In view of the problem that the existing neuromorphic camera reconstruction method cannot well remove noise under different lighting conditions, the main purpose of the present application is to provide a general-purpose light condition-oriented neuromorphic camera denoising method, which reconstructs a clean image from the original data of the neuromorphic camera, reduces the noise in the image, and reduces the influence of noise on downstream tasks.

[0005] The purpose of the present application is achieved by the following technical scheme:

[0006] The application discloses a general-purpose light condition-oriented neuromorphic camera denoising method, which converts continuous space-time pulse flow into picture feature representation under a specific window to adapt to the input of a traditional neural network according to the relationship between pulse flow noise intensity and scene light; dark current noise and response inconsistency noise are removed through a calibrated dark current matrix and a response inconsistency matrix; image features are converted into frequency domain features through two-dimensional discrete wavelet transform, and low-cost noise removal is realized by using the frequency domain noise separation characteristics; time domain noise, mainly including shot noise and truncation noise, is removed by fusing the frequency domain features of adjacent windows; residual noise in the frequency domain is thoroughly removed through a deep self-attention network; finally, the denoising result is refined by using the fused features in the frequency domain to suppress excessive smoothing caused by denoising, and a clean image is finally reconstructed, so that the noise in the image is reduced and the influence of the noise on downstream tasks is reduced.

[0007] The general-purpose light condition-oriented neuromorphic camera denoising method disclosed by the application comprises the following steps:

[0008] Step 1: The total noise N is decoupled into the sum of multiple noise components by physically modeling the noise of the pulse camera.

[0009] The total noise N of the pulse flow reconstructed image captured by the pulse camera:

[0010]

[0011] Wherein I represents a clean image reconstructed in an ideal case; Q r is a relative charge matrix, defined as CΔV / α, C is the capacitance size in the camera integration circuit, ΔV is the voltage difference required to complete an integration, and α is the photoelectric conversion coefficient, which represents the number of relative electrons contained in a single pixel sensor and can be indirectly calculated by shooting data to determine the light intensity; L is the light intensity of the scene; N p is shot noise; N d is dark current noise; N rnu is response inconsistency noise; N q is quantization noise; N c is truncation noise.

[0012] Quantization noise N q derived from the hysteresis of the readout circuit, the integration circuit outputs the pulse signal to the readout circuit after completing an integration and waits for the clock signal to read out, so that the output time of the pulse signal is always lagging behind the generation time, which will introduce additional quantization noise on the integration time:

[0013] N q ~ U(-t0, t0)

[0014] where U represents uniform distribution, t0 is the unit time of high-speed readout timing.

[0015] Truncated noise N c From the process of recovering image from pulse stream, for still scene, using longer pulse stream means introducing smaller truncated noise in the recovered image. The average time interval of pulse emission of a certain pixel unit is d, using pulse stream with time length l (kd < l ≤ (k + 1)d, k ∈ N) to reconstruct image, the size of truncated noise introduced is represented as:

[0016]

[0017] where P represents the probability of the value.

[0018] In order to measure the noise parameters of the camera, three groups of uniform light scenes with known light intensity need to be shot for calibration, one group is shot in complete dark environment, and the other two groups are shot in uniform light environment with different light intensity L1 and L2. According to photoelectric effect, current definition formula and capacitance definition formula, then:

[0019]

[0020] It is derived that:

[0021]

[0022] where T d , T1 are the average pulse emission time interval of dark scene and low light scene respectively, L d is the equivalent scene brightness of dark current effect.

[0023] The response inconsistency matrix R represents the response inconsistency noise N rnu , which represents the ratio of the response degree of the reference pixel to each pixel of the sensor array. In order to ensure the consistency of the average brightness before and after correction, the pixel (x m , y m ) with the average pulse emission time interval closest to the mean value is taken as the reference pixel, then:

[0024]

[0025] where T2 is the average pulse emission time interval of bright light scene, (x m , y m ) is the reference pixel position.

[0026] At the same time:

[0027]

[0028] Then for the pulse emission time considering dark current noise, there is:

[0029]

[0030] where T is the ideal pulse emission time under light intensity L. Equation (8) can be used to add dark current noise when synthesizing data.

[0031] Step 2, using the physical noise model in Step 1, synthesize noisy pulse streams conforming to the real noise distribution using the dark current matrix L d , the relative charge matrix Q r , and the response inconsistency matrix R to form clean image G-noisy pulse stream S pair data for deep network model training based on full supervision learning.

[0032] The specific process is as follows:

[0033] a) Determine parameters: clean grayscale image I, light intensity scaling factor θ, and synthesized pulse stream length l;

[0034] b) Scale light intensity using θ: I = I * θ;

[0035] c) Set window size: wins = l;

[0036] d) Add Poisson noise: is the Poisson function;

[0037] e) Convert light intensity to time interval representation: I max is the dynamic range of the processed grayscale image;

[0038] f) Add dark current noise and response inconsistency noise:

[0039] g) Add quantization noise: D = D + Rand(-1, 1), where Rand is a random function that returns a random floating-point number in the interval;

[0040] h) Convert time interval back to light intensity representation:

[0041] i) Simulate the integration stage: Ig = Ig + I', where Ig is the integrator, which has the same dimensions as the reconstructed image, and is initialized using a random Gaussian function;

[0042] j) Simulate the readout stage: when Ig ≥ thres, the pulse stream S[l-wins] is set to 1, otherwise it is set to 0, where thres is the response threshold;

[0043] k) Reset the integrator: Ig = Ig - S[l-wins] * thres;

[0044] l) process next time, wins = wins - 1;

[0045] m) if wins ≠ 0, jump to step c, otherwise, output pulse stream S, stop.

[0046] Build the neural network model in the following steps, initialize time t = 0 for continuous generation of clean images in time scale.

[0047] Step 3, according to the sampling principle of neuromorphic camera and the relationship between noise intensity and pulse density, adaptively generate pulse stream representation, and calculate the image feature A represented by the pulse stream on the representation t .

[0048] Adaptive pulse conversion contains a function of length l to pulse density s, which maintains the same noise level of output image by setting different sampling window length corresponding to different light intensity, the specific form is:

[0049]

[0050] Step 4, establish the relationship between image feature A t and relative charge matrix Q r , dark current matrix L d , response inconsistency matrix R, and based on the time sequence characteristics of pulse data, construct deep neural network to remove dark current noise and response inconsistency noise of pulse stream, get I t .

[0051] The inconsistency correction network includes two stages, the first stage removes the dark current noise on A t using relative charge matrix Q r , dark current matrix L d , get I t,0 , the second stage removes the response inconsistency noise on I t,0 using response inconsistency matrix R, get I t , the specific formula of the whole process is:

[0052]

[0053]

[0054] Among them, and are the two parts of the inconsistency correction network.

[0055] Step 5, according to the time sequence characteristics of pulse data, establish a multi-scale decomposition network based on I t and I t-1 , when t = 0, let I t-1 = I t. The feature I t The domain conversion is performed, which is represented as:

[0056]

[0057]

[0058]

[0059] wherein, represents a two-dimensional image discrete wavelet transform, {LL, LH, HL, HH} represents a frequency domain subgraph set, L is a low-pass filter, H is a high-pass filter, and LH represents a result of performing low-pass filtering first and then high-pass filtering.

[0060] Step 6, fusing I t i and I t-1 i The time domain information is used to remove the truncation noise and other time domain noises, and the time domain fusion starts from the lowest layer feature, and a convolutional neural network is used to predict a fusion mask from the low-frequency subgraph of the current frame and the previous frame, and the process is represented as:

[0061]

[0062] wherein represents a convolutional neural network of the time domain fusion module, mask 2 represents a fusion mask at scale 2, and the frequency domain features of two consecutive frames are fused according to the fusion mask, and the specific formula is:

[0063]

[0064] wherein is a frequency domain fusion feature at scale 2, and compared with the current frequency domain feature temporal noise is suppressed. In addition, the time domain fusion at a higher scale (0, 1) needs to refer to the mask of the lower layer when predicting the mask:

[0065]

[0066] The frequency domain fusion feature at scale 0 After inverse discrete wavelet transform, the image domain feature F t is obtained.

[0067] Step 7, constructing a denoising network module at each scale, realizing further removal of noise in the frequency domain, and obtaining a denoising result D t iMeanwhile, high-level feature denoising requires guidance from the low-level denoising results to achieve fine-grained noise removal.

[0068] Thorough noise removal of frequency domain features at multiple scales is achieved using deep self-attention networks. This process can be represented as follows:

[0069]

[0070] in This represents a deep self-attention network, which effectively removes spatial noise using its self-attention and cross-attention mechanisms. t 2 For the denoised frequency domain features at scale 2, higher-scale frequency domain denoising requires guidance from lower-scale denoised features. This process is represented as:

[0071]

[0072] Step 8: Use a convolutional layer to extract the output F from step 6. t 0 and the output D of step 7 t 0 The fusion mask is used to refine the denoising result and obtain R. t To address the over-smoothing problem caused by denoising, the inverse discrete wavelet transform is used to transform the frequency domain features R. t Convert to image features O t O t This is the final denoising result at time t.

[0073] To avoid over-smoothing in frequency domain denoising, the fusion feature F needs to be utilized. t 0 Refined Denoising Feature D t 0 The specific formula is as follows:

[0074]

[0075] Here, mask is the fusion mask that guides the refinement process, which is achieved through a refinement network. The prediction is obtained. The denoising features are further refined using a mask, preserving texture details while removing noise. The specific formula for this process is:

[0076] R t 0 =mask*F t 0 +(1-mask)*D t 0

[0077] Among them, R t0 For the refined frequency domain features, O t :

[0078]

[0079] O t That is, at time t, the real pulse stream S t is used to reconstruct the clean image.

[0080] Step 9, stop when all pulse data is processed, get the reconstruction result O t of the final time, if there is still pulse data, continue to perform image reconstruction of the next time, let t = t + 1, I t-1 = F t-1 , jump to step 3 to continue execution.

[0081] Step 10, through steps 3-9, the neural network model is built, and the paired data generated in step 2 is used for full supervision training and testing, and the network model for image reconstruction is generated.

[0082] Also includes step 11, according to the network model obtained in step 10, the image taken by the neuromorphic camera is processed, and the clean image is reconstructed, the noise in the image is reduced, and the influence of the noise on the downstream task is reduced.

[0083] Advantages:

[0084] 1. Compared with the prior art which only refers to limited time pulse stream, the neuromorphic camera denoising method disclosed in the application uses long-time high-density pulse stream as the input of the reconstruction model, can generate reconstruction results at any time, and the visual quality of the reconstruction results increases with time until it is stable, improving the quality of the image.

[0085] 2. Compared with the prior art which does not analyze the source and distribution of neuromorphic camera noise, the neuromorphic camera denoising method disclosed in the application is based on physical modeling of the noise generation process of the neuromorphic camera and noise parameter calibration of the specific camera, fully considers the types and sources of noise, and uses corresponding noise reduction measures to reconstruct the image, reduce the noise in the image, and improve the image quality.

[0086] 3. Compared with the prior art which lacks a data synthesis method consistent with the real noise distribution, the neuromorphic camera denoising method disclosed in the application is based on a data synthesis method for neuromorphic camera noise modeling. The paired data generated by the noise synthesis method provides training basis for full supervision deep learning method, makes the model more suitable for image reconstruction, and reduces the noise of the reconstructed image.

[0087] 4、The neural morphological camera denoising method disclosed in the present application has poor interpretability and strong fitting ability compared with the prior art, is based on multi-scale fusion, realizes feature fusion and denoising from coarse to fine, realizes low-cost feature fusion and denoising based on the frequency domain noise separation characteristics of discrete wavelet transform, realizes fitting of frequency domain denoising mapping based on the strong fitting ability of the deep self-attention mechanism network, and improves the image quality. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 is a flowchart of the neural morphological camera denoising method disclosed in the present application;

[0089] Figure 2 is a reconstruction result graph of the neural morphological camera denoising method disclosed in the present application for a synthetic noisy pulse stream input;

[0090] Figure 3 is a reconstruction result graph of the neural morphological camera denoising method disclosed in the present application for a real noisy pulse stream input;

[0091] Figure 4 is a reconstruction result graph of the neural morphological camera denoising method disclosed in the present application in a high-light and high-speed scene. DETAILED DESCRIPTION

[0092] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0093] Example 1:

[0094] Figure 1 The denoising process provided by the embodiments of the present application is shown in Figure 1 The neural morphological camera denoising method disclosed in the present embodiment includes the following steps:

[0095] Step 101: Decouple the total noise N into the sum of a plurality of noise components by physically modeling the noise of the pulse camera.

[0096] The total noise N of the pulse stream reconstructed image captured by the pulse camera:

[0097]

[0098] Shot noise Np The randomness of the number of photons hitting the photoelectric sensor at a certain moment is described, which represents the noise error and the sum of the ideal value E of the number of photons as a whole obeys the Poisson distribution:

[0099] E+N p ~P(E)

[0100] Where P represents the Poisson distribution.

[0101] Dark current noise N d Originating from the thermal diffusion of carriers and defects inside and on the surface of the PN junction, even if the scene light intensity is 0, the existence of dark current noise will make the photosensitive circuit release electrons and form an output equivalent to "dark current" with "photocurrent". Considering the randomness of dark current generation and ignoring the influence of temperature, the dark current noise is modeled as:

[0102] N d (x,y)~P(μ d (x,y))

[0103] Where μ d (x,y) represents the mean of the dark signal. Due to the limitations of manufacturing process, the dark current shows pixel heterogeneity, and the dark current difference between different pixels is large.

[0104] Response inconsistency noise N rnu Describe the inconsistency of different pixel sensor units in response to the same light intensity, which is caused by the mismatch of capacitance and photodiode. Specifically, the pulse firing process of a single integration circuit considering the response inconsistency noise is represented as:

[0105]

[0106] Where T is the integration time required for the integration circuit of the reference pixel to reach the pulse firing threshold under the same photocurrent I ph (τ); δC and δV are the differences in capacitance and voltage between the integration circuit of a single pixel and the integration circuit of the reference pixel, respectively.

[0107] Quantization noise N q Originating from the hysteresis of the readout circuit, the integration circuit outputs the pulse signal to the readout circuit after completing an integration and waits for the clock signal to read out, so the output time of the pulse signal is always lagging behind the generation time, which will introduce additional quantization noise on the integration time:

[0108] N q ~U(-t0,t0)

[0109] Where U represents the uniform distribution, and t0 is the unit time of the high-speed readout timing.

[0110] Truncation noise N cThe longer the pulse stream used for recovering the image from a static scene, the less noise is introduced in the recovered image. Let the average time interval of a pixel unit emitting a pulse be d, and the length of the pulse stream used for reconstructing the image be l (kd < l ≤ (k + l)d, k ∈ N), the size of the truncation noise introduced by the pulse stream is represented as:

[0111]

[0112] where P represents the probability of the value.

[0113] To measure the noise parameters of the camera, three groups of uniform light scenes with known light intensity are shot for calibration. One group is shot in a completely dark environment, and the other two groups are shot in uniform light environments with different light intensities (measured using a light meter, denoted as L1, L2). According to the photoelectric effect, the current definition formula, and the capacitor definition formula, we have:

[0114]

[0115] It is derived that:

[0116]

[0117] where T d , T1 are the average pulse emission time intervals of the dark scene and the low light scene, respectively, and L d is actually the equivalent scene brightness of the dark current effect.

[0118] The response inconsistency matrix R represents the response inconsistency noise N rnu , which represents the ratio of the response degree of the reference pixel to each pixel of the sensor array. To ensure that the average brightness before and after correction is consistent, take the pixel (x m , y m ) with the average pulse emission time interval closest to the mean as the reference pixel, then we have:

[0119]

[0120] where T2 is the average pulse emission time interval of the bright light scene, and (x m , y m ) is the reference pixel position.

[0121] At the same time, we have:

[0122]

[0123] Then for the pulse emission time considering the dark current noise, we have:

[0124]

[0125] where T is the ideal spike firing time under light intensity L. The above equation can be used to add dark current noise when synthesizing data.

[0126] Step 102: According to the physical noise model, use the dark current matrix L measured by calibration d and the response inconsistency matrix R to synthesize a noisy pulse stream that conforms to the real noise distribution, forming clean image G-noisy pulse stream S paired data for deep network model training based on full supervision learning.

[0127] Dark current matrix L measured by calibration d , relative response matrix Q r and response inconsistency matrix R, use the clean grayscale image to generate a noisy pulse stream, the specific steps are as follows:

[0128] A. Determine the parameters: clean grayscale image I, light intensity scaling factor θ = 0.125, synthesized pulse stream length l = 300;

[0129] B. Use θ to scale the light intensity: I = I * θ;

[0130] C. Set the window size: wins = l;

[0131] D. Add Poisson noise:

[0132] E. Convert light intensity to time interval representation:

[0133] F. Add dark current noise and response inconsistency noise:

[0134] G. Add quantization noise: D = D + Rand(-1,1);

[0135] H. Convert time interval back to light intensity representation:

[0136] I. Simulate the integration stage: Ig = Ig + I';

[0137] J. Simulate the readout stage: when Ig ≥ thres, pulse stream S[l-wins] is set to 1, otherwise 0, where thres is the response threshold, set to 255;

[0138] K. Reset the integrator: Ig = Ig - S[l-wins]*thres;

[0139] L. Process the next time, wins = wins - 1;

[0140] M. If wins ≠ 0, jump to process c, otherwise, output pulse stream S and stop.

[0141] The neural network model is built in the following steps, and the time t=0 is initialized to generate clean images continuously in the time scale.

[0142] Step 103: The pulse stream window size is adaptively generated according to the relationship between the noise intensity and the pulse density, and the mean value of the pulse stream is calculated on the window to form the image feature A t .

[0143] Pulse density calculation method:

[0144]

[0145] Where h and w are the height and width of the sensor respectively, the pulse density represents the scene light intensity, the greater the scene light intensity, the shorter the time required for the sensor to integrate once, and the smaller the output pulse time interval, then the average pulse density of the sensor array is larger, and in this example, h=250 and w=400.

[0146] Adaptive pulse conversion includes a length l-pulse density s function, which maintains the same noise level of the output image by setting different light intensity corresponding to the sampling window length, and the specific form is:

[0147]

[0148] After calculating the sampling window length from the above, the image feature A t is further calculated:

[0149]

[0150] Step 104: A t is spliced with the relative charge matrix Q r , the dark current matrix L d , and the response inconsistency matrix R is sent into the non-uniform correction network to remove the dark current noise and the response inconsistency noise, and I t is obtained, which greatly suppresses the influence of sensor dead pixels compared with A t .

[0151] The non-uniform correction network includes a two-stage process, the first stage removes the dark current noise on A t using the relative charge matrix Q r , the dark current matrix L d , and obtains I t,0 , the second stage removes the response inconsistency noise on I t,0 using the response inconsistency matrix R, and obtains I t , and the specific formula of the whole process is:

[0152]

[0153]

[0154] wherein and are two parts of the non-uniformity correction network respectively, both of which are composed of four layers of convolution, and the size of the convolution kernel is set to 1 considering the pixel heterogeneity of the dark current noise and the response non-uniformity noise.

[0155] Step 105: when t = 0, let I t-1 = I t , and use the multi-scale discrete wavelet transform to decompose I t and I t-1 into subgraphs I t 0,1,2 , I t-1 0,1,2 in different sizes, and realize low-cost noise removal by using the characteristics that the noise and the detail texture frequency are different.

[0156] Use the multi-scale discrete wavelet transform to perform domain conversion on the feature It that has removed the dark current noise and the response non-uniformity noise, and the process is represented as:

[0157]

[0158]

[0159]

[0160] wherein, represents the two-dimensional image discrete wavelet transform, {LL, LH, HL, HH} represents a set of frequency domain subgraphs, L is a low-pass filter, H is a high-pass filter, and LH represents the result of performing low-pass filtering first and then high-pass filtering. Since the noise is generally high frequency and the texture details belong to low frequency, in theory, the converted frequency domain subgraphs can separate the residual noise and the texture details on I t , and the frequency domain denoising guided by the low-frequency subgraph LL is efficient for the network model.

[0161] Step 106: fuse I t i and I t-1 i on multiple scales, use a convolution layer to extract a fusion mask mask i and weighted sum to obtain a fusion result F t i of each scale, and use the time domain information to remove the truncated noise and other time domain noise.

[0162] Temporal fusion starts from the lowest level feature, first uses a convolutional neural network to predict the fusion mask from the low-frequency subgraph of the current frame and the previous frame, the process is represented as:

[0163]

[0164] Wherein represents the neural network of the temporal fusion module, which is composed of four layers of convolutional layers, mask 2 represents the fusion mask on scale 2, according to which the frequency domain features of the two consecutive frames are weighted and fused, and the specific formula is:

[0165]

[0166] Wherein is the frequency domain fusion feature on scale 2, compared with the current frequency domain feature suppresses the temporal noise. In addition, the temporal fusion on a higher scale (0, 1) needs to refer to the mask of the lower layer when predicting the mask:

[0167]

[0168] The frequency domain fusion feature on scale 0 After inverse discrete wavelet transform, the image domain feature F is obtained t .

[0169] Step 107: the fusion feature of step 106 There is still some noise, which is further removed by the deep self-attention network module in the frequency domain to obtain the denoising result D of each scale t i At the same time, the denoising of high-level features needs to be guided by the bottom-level denoising result to achieve fine-grained noise removal.

[0170] Through the deep self-attention network, the frequency domain features on multiple scales are thoroughly removed, and the process is represented as:

[0171]

[0172] Wherein represents the deep self-attention network Transformer, which can well remove the spatial noise by using its self-attention and cross-attention mechanism, and in order to extract the multi-dimensional features in the frequency domain, The encoder front end also includes multiple residual convolutional blocks. D t 2 The frequency domain feature on scale 2 after denoising, the frequency domain denoising on a higher scale needs to be guided by the denoising feature on a lower scale, and the process is represented as:

[0173]

[0174] Step 108: Extract the output F from step 106 using a convolutional layer. t 0 And step 107 output D t 0 The fusion mask is used to refine the denoising result and obtain R. t To address the over-smoothing problem caused by denoising, the inverse discrete wavelet transform is used to transform the frequency domain features R. t Convert to image features O t O t This is the final denoising result at time t.

[0175] To avoid over-smoothing in frequency domain denoising, the fusion feature F needs to be utilized. t 0 Refined Denoising Feature D t 0 The specific formula is as follows:

[0176]

[0177] Here, mask is a fusion mask that guides refinement, which is achieved through a refinement network consisting of four convolutional layers. The prediction was obtained. The denoising features are further refined using a mask, preserving texture details as much as possible while removing noise. The specific formula for this process is:

[0178] R t 0 =mask*F t 0 +(1-mask)*D t 0

[0179] Among them, R t 0 The refined frequency domain features are obtained by performing an inverse discrete wavelet transform to obtain O. t :

[0180]

[0181] O t That is, at time t, using the real pulse flow S t A clean, reconstructed image.

[0182] Step 109: Stop when all pulse data has been processed, and obtain the final reconstruction result O. t If there is still pulse data, continue with the image reconstruction for the next time step, let t = t + 1, I t-1 =F t-1 Proceed to step 103 to continue execution.

[0183] Step 110: The neural network model built via steps 103-109 is fully supervised trained using the paired data generated in step 102, guided by gradient descent using Adam optimizer, learning rate set to 1e-3, and the optimization objective is as follows:

[0184] loss = ‖O t ,G t ‖1+‖R t 0 ,G t 0 ‖1+‖D t 1 ,G t 1 ‖1+‖D t 1 ,G t 2 ‖1

[0185] where G t is the paired clean image, G t 0、1、2 represent different scale frequency subgraphs of the paired clean image after discrete wavelet transform, respectively. In testing, the trained neural network model is applied to the test data, and the process is completely consistent with steps 3-9, finally realizing the reconstruction of the clean image.

[0186] The effectiveness of the present application will be illustrated by the reconstruction results of the pulse stream.

[0187] 1. Experimental conditions

[0188] The hardware test conditions of this experiment are: Intel Xeon Gold 6240C, memory 32G. The GPU is NVIDIA GeForce RTX 3090, with 24G of video memory, and the CUDA version is 11.1; the camera used is a pulse camera (Vidar). In addition to the reconstruction method on the contrast pulse modality, the experiment also compares the image and video denoising methods. In order to ensure the fairness of the comparative experiment, the TFP is used to reconstruct the gray-scale image of the continuous pulse, so as to adapt the picture modality input of the image and video denoising method. The pulse reconstruction method uses the original weight for testing, and the image and video denoising methods are retrained and tested on the synthetic data set.

[0189] 2. Experimental results

[0190] In order to quantitatively measure the noise removal effect of different methods, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are used to measure the visual quality of the pulse stream reconstruction result, and the scene is divided into low-light and high-light scenes according to the pulse density of the pulse stream reflecting the brightness characteristics of the scene.

[0191] Table 1 shows the index results of the spatio-temporal pulse stream reconstruction method and the image and video denoising method on the simulation data set, and it can be seen from the table that the method of the present application has obvious improvement compared with all the comparison methods, and good visual effect reconstruction results can be obtained in both high-light scenes and low-light scenes.

[0192] Table 1 shows the index results of the spatio-temporal pulse stream reconstruction method and the image and video denoising method on the simulation data set, and it can be seen from the table that the method of the present application has obvious improvement compared with all the comparison methods, and good visual effect reconstruction results can be obtained in both high-light scenes and low-light scenes.

[0193]

[0194]

[0195] Figure 2 The results of the comparison methods and the method of the present application on the simulation data set are shown, and the method of the present application can obtain good visual effect reconstruction images in both high-light scenes and low-light scenes.

[0196] Figure 3 The results of the comparison methods and the method of the present application on the real data set are shown, and it can be seen from the figure that the neuromorphic camera denoising method of the present application can effectively remove the temporal noise and spatial noise, especially in extremely dark scenes, and still ensure sufficient details while removing as much noise as possible, greatly widening the application scenarios of the neuromorphic camera and having high application value.

[0197] Figure 4 The results of the method of the present application in high-light and high-speed are shown, verifying that the method is suitable for a wide range of scenarios, and can not only process static scenes, but also be compatible with the performance of the neuromorphic camera in high-speed scenes.

[0198] In summary, the neuromorphic camera denoising method for general lighting conditions proposed in the embodiment can adaptively convert pulse data to process noise levels under different lighting conditions, correct non-uniform noise using calibrated noise parameters, separate noise using discrete wavelet transform, and remove temporal noise and spatial noise using a frequency domain fusion network and a deep self-attention network, significantly improving the utilization of continuous spatio-temporal pulse streams and the information recovery capability of low-light response pulse streams, and successfully solving the problem that existing neuromorphic camera reconstruction algorithms rely on high-light scene data. The neuromorphic camera denoising method for general lighting conditions proposed in the present example greatly widens the use scenarios of the neuromorphic camera, and plays a significant role in downstream tasks such as target detection and recognition, and object tracking.

[0199] The above detailed description makes further detailed explanation on the purpose, technical scheme and beneficial effects of the present application. It should be understood that the above description is only specific embodiments of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A denoising method for neuromorphic cameras under general lighting conditions, characterized in that: Includes the following steps, Step 1: By physically modeling the noise of the pulse camera, the total noise is... Decoupling results in the sum of multiple noise components; Step 2: Based on the physical noise model in Step 1, use the dark current matrix measured from the calibration data. Relative charge matrix and response inconsistency matrix Synthesize a noisy pulse stream that conforms to the real noise distribution to form a clean image. - Noisy pulse stream Paired data for training deep network models based on fully supervised learning; Step 3: Based on the sampling principle of neuromorphic cameras and the relationship between noise intensity and pulse density, adaptively generate a pulse flow representation, and calculate the image features represented by the pulse flow on this representation. ; Step 4: Establish image features With relative charge matrix Dark current matrix Response Inconsistency Matrix Based on the temporal characteristics of pulse data, a deep neural network is constructed to remove dark current noise and response inconsistency noise from the pulse flow, resulting in... ; Step 5: Based on the timing characteristics of the pulse data, establish a system based on... Multi-scale decomposition network, when season ; Multi-scale discrete wavelet transform was used to analyze features that had already had dark current noise and response inconsistency noise removed. Performing a domain conversion is represented as follows: in, Represents the discrete wavelet transform of a two-dimensional image. Represents the set of frequency domain subgraphs. It is a low-pass filter. For high-pass filters, This represents the result after performing low-pass filtering first and then high-pass filtering; Step 6: Fusion at multiple scales and Temporal noise is removed using temporal information. Temporal fusion starts from the lowest-level features, first using a convolutional neural network to predict the fusion mask from the low-frequency sub-images of the current frame and the previous frame. This process is represented as: in A convolutional neural network representing the temporal fusion module. The fusion mask at scale 2 is weighted and fused based on the frequency domain features of two consecutive frames. The specific formula is as follows: in This represents the frequency domain fusion feature at scale 2, compared to the current frequency domain features. Temporal noise was suppressed; furthermore, temporal fusion at higher scales (0,1) improved prediction. At the same time, it is necessary to refer to the lower level. : Frequency domain fusion features at scale 0 Image domain features are obtained after discrete wavelet inverse transform. ; Step 7: Construct denoising network modules at various scales to further remove noise in the frequency domain, obtaining denoising results for each scale. Meanwhile, high-level feature denoising needs to be guided by the results of low-level denoising to achieve fine-grained noise removal; Step 8: Use a convolutional layer to extract the output from step 6. and the output of step 7 Fusion mask Weighted fusion is then used to refine the denoising results. To address the over-smoothing problem caused by denoising, inverse discrete wavelet transform is used to extract frequency domain features. Convert to image features , That is The final denoising result at any given moment; Step 9: Stop when all pulse data has been processed, and obtain the final reconstruction result. If there is still pulse data, continue with the image reconstruction for the next time step. , Proceed to step 3 and continue execution; Step 10: Following steps 3-9, the neural network model is built, and the paired data generated in step 2 is used to train and test it under full supervision to generate a network model for image reconstruction.

2. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 1, characterized in that: It also includes step 11: based on the network model obtained in step 10, the images captured by the neuromorphic camera are processed to reconstruct clean images, reduce noise in the images, and reduce the impact of noise on downstream tasks.

3. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 1, characterized in that: The implementation method for step 1 is as follows: Total noise in images reconstructed from pulse streams captured by a pulse camera : in This represents a clean image reconstructed under ideal conditions; Let be the relative charge matrix, defined as , The value of the capacitor in the camera's integrating circuit. The voltage difference required to complete one integration step The photoelectric conversion coefficient represents the relative number of electrons contained in a single pixel sensor and can be indirectly calculated by capturing data to determine light intensity. The light intensity of the scene; In response to non-uniform noise; To quantize noise; To cut off noise; Quantization noise Due to the hysteresis of the readout circuit, the integrator outputs a pulse signal to the readout circuit after completing one integration and waits for the clock signal to be read out. Therefore, the output time of the pulse signal always lags behind the generation time, which introduces additional quantization noise into the integration time. in Represents a uniform distribution. This refers to the unit of time for high-speed timing readout. Cut-off noise This originates from the process of recovering an image from a pulse stream. For static scenes, using a longer pulse stream means introducing less noise into the recovered image. The average pulse interval of a certain pixel unit is Length of use The magnitude of the truncation noise introduced by the pulse stream reconstructed image is expressed as: in Represents the probability of taking a value; To measure the camera's noise parameters, three sets of uniformly lit scenes with known light intensities need to be photographed for calibration: one set in complete darkness, and the other two sets at different light intensities. and The image was taken under uniform lighting conditions; according to the photoelectric effect, the definition of current, and the definition of capacitance, we have: We derive that: in , These represent the average pulse firing time intervals for dark scenes and low-light scenes, respectively. It is the scene brightness equivalent to the dark current effect; Response Inconsistency Matrix Characterizing the response to non-uniform noise It represents the ratio of the response level of the reference pixel to that of each pixel in the sensor array; To ensure consistent average brightness before and after correction, the pixel whose average pulse firing time interval is closest to the mean is selected. As a reference pixel, we have: in The average pulse firing time interval for bright light scenes. Reference pixel position; Simultaneously we obtain: Then, considering the pulse firing time with regard to dark current noise, we have: in The light intensity is The ideal pulse firing time is given; Equation (8) can be used to add dark current noise when synthesizing data.

4. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 3, characterized in that: The implementation method for step 2 is as follows: a) Determine parameters: Clean grayscale image Light intensity scaling factor Synthetic pulse length ; b) Use Scaling light intensity: ; c) Set the window size: ; d) Add Poisson noise: , It is the Poisson function; e) Convert light intensity into time interval representation: , The dynamic range of the grayscale image being processed; f) Add dark current noise and response inconsistency noise: ; g) Add quantization noise: , This is a random function that returns a random floating-point number within a specified interval. h) Convert the time interval back to light intensity representation: ; i) Simulation integration stage: , The integrator has the same dimensions as the reconstructed image and is initialized using a random Gaussian function; j) Simulated readout stage: when At that time, pulse flow Set to 1 otherwise to 0, where For response threshold; k) Reset the integrator: ; l) Process the next moment. ; m) If Then proceed to step c; otherwise, output pulse stream. ,stop.

5. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 4, characterized in that: The implementation method for step 3 is as follows: Adaptive pulse conversion includes a length For pulse density The function maintains the same noise level in the output image by setting the sampling window length corresponding to different light intensities. Specifically, it takes the following form: in Sampling window length.

6. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 5, characterized in that: The implementation method for step 4 is as follows: Inconsistency correction networks consist of a two-stage process, the first stage being... The above utilizes the relative charge matrix Dark current matrix Remove dark current noise to obtain The second phase is in The above utilizes the response inconsistency matrix After removing non-uniform noise from the response, we obtain The specific formula for the entire process is as follows: in and These are the two parts of the inconsistency correction network.

7. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 6, characterized in that: The implementation method for step 7 is as follows: Thorough noise removal of frequency domain features at multiple scales is achieved using deep self-attention networks. This process can be represented as follows: in This represents a deep self-attention network, which can effectively remove spatial noise by utilizing its self-attention and cross-attention mechanisms. For the denoised frequency domain features at scale 2, higher-scale frequency domain denoising requires guidance from lower-scale denoised features. This process is represented as: 。 8. The denoising method for neuromorphic cameras under general lighting conditions as described in claim 7, characterized in that: The implementation method for step 8 is as follows: To avoid over-smoothing in frequency domain denoising, fusion features need to be utilized. Refined noise reduction features The specific formula is as follows: in, To guide the refinement of the fusion mask, it uses a refinement network. Predicted; utilized Further refine and detail the denoising features, preserving texture details while removing noise. The specific formula for this process is as follows: in, The refined frequency domain features are obtained after undergoing inverse discrete wavelet transform. : That is Utilizing real pulse flow at all times A clean, reconstructed image.

Images