Noise quality assessment method for event cameras

Abstract

The present invention discloses a noise quality evaluation method for event cameras, comprising the following steps: 1. Using different degrees of dimming lenses to obtain event stream data with different noise levels; 2. Preprocessing the event stream data and equally dividing it into multiple event packages, projecting the event packages to obtain event frames; 3. Calculating the square sum of each pixel in the event frame to obtain the overall square sum expectation of the event frame; 4. Extracting a fixed number of reference events from the event package and calculating the number of valid pixels in the event frame; 5. Repeating steps 2 to 4 until the event effective structure ratio of each event package is calculated, and then obtaining the average noise level score of the event stream. The present invention can obtain event streams with different noise levels, and can effectively evaluate the degree of noise influence on the event stream without relying on additional information assistance and manual annotation.

Description

Technical Field

[0001] The present invention belongs to the field of event camera denoising, and specifically relates to a multi-noise level data shooting method and a reference-free event noise level evaluation method. Background Art

[0002] Event-based Camera is a new type of biological visual sensor. Different from the working principle of traditional cameras, event cameras output asynchronously by sensing the dynamic changes in the brightness of the scene, so its output signal is sparse. Thanks to this, event cameras can achieve higher sampling rate, dynamic range and lower power consumption than traditional cameras, thus showing great advantages in high-speed motion or low-light scenes. However, since event cameras lack the process of integrating brightness input like traditional cameras, they are more susceptible to noise, which seriously damages and affects the performance of event cameras in various tasks. In addition, since they are affected by the coupling of multiple factors such as the environment, scene and sensor, such noise is difficult to filter out.

[0003] In order to solve the noise problem of event cameras, relevant noisy event data and corresponding noise level evaluation indicators have been proposed to explore the noise generation mechanism of event cameras and introduce relevant denoising algorithms. However, the existing shooting methods cannot meet the requirements of simulating complex noise levels in scenes, and the existing evaluation indicators need to rely on image information and manual annotation, which have poor reliability. Therefore, a method is needed to simulate the shooting of real event data with different noise levels in the scene, and at the same time judge the noise level of the event data based only on event information. Summary of the invention

[0004] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes a noise quality evaluation method for event cameras, which aims to capture noisy event streams with various noise levels in the same scene and quantitatively evaluate the noise level without using any additional information and manual labeling, so as to effectively evaluate the degree to which the event stream is affected by noise.

[0005] The present invention does not achieve the above-mentioned invention object, and adopts the following technical solution:

[0006] The noise quality evaluation method for an event camera of the present invention is characterized in that it comprises the following steps:

[0007] Step 1: Install different types of ND filters in front of the event camera lens to simulate different ambient light intensities, so as to obtain event stream data with multiple noise levels. e k represents the kth event, N e Represents the event stream data The total number of events included; Initialize the quality evaluation parameters, including: the kth event e k The pixel weight b k , the number of reference events for a single evaluation N, the number of reference events used for interpolation M;

[0008] Step 2: Evenly divide the event stream data with multiple noise levels by the number of reference events N evaluated in a single time. Get E event packets and use the reference time t ref Perform alignment to obtain an alignment event frame;

[0009] Step 3: Calculate the alignment event frame IWE corresponding to the j-th event packet j The total sum of squares of events is expected to be NTSS j ;

[0010] Step 4: Calculate the alignment event frame IWE corresponding to the j-th event packet according to the reference event number M. j The number of effective pixels on L j ;

[0011] Step 5: Expected NTSS based on the total sum of squares of the number of events j and the number of effective pixels L j , calculate the alignment event frame IWE corresponding to the j-th event packet j Effective event structure ratio And according to the process of step 3 to step 4, the effective event structure ratio of all event packets is obtained, thereby obtaining the multi-noise level event stream data The average noise level score MESR is used as the quality evaluation result, where := represents the assignment operation.

[0012] The noise quality evaluation method for event cameras according to the present invention is also characterized in that step 2 comprises:

[0013] Step 1.1: Use formula (1) to obtain the event data D contained in the jth event packet: j :

[0014]

[0015] In formula (1), (x, y, t) is the space-time coordinate position, which represents the pixel coordinate of the event camera at time t, (x k ,y k ,t k ) is the kth event e k The space-time coordinate location of the occurrence, c k is the kth event e k The polarity of , δ is the Dirac function, j represents the sequence number of the event packet, E is the number of event packets, and satisfies:

[0016]

[0017] In formula (2), Indicates a round-down operation;

[0018] Step 1.2: The event data D in the jth event packet j Projection to a given reference time And the event data D j The coordinates of satisfy the alignment equation shown in formula (3):

[0019]

[0020] In formula (3), W represents the mapping relationship, Indicates that the two satisfy the mapping relationship W, Represents the kth event e k At reference time The projection coordinates of k represents the kth event e k Projection events under the mapping relation W;

[0021] Using formula (4), we can get the alignment event frame IWE corresponding to the j-th event packet: j :

[0022]

[0023] In formula (4), (x i,j ,y i,j ) represents the alignment event frame IWE j At any i-th pixel position on k Represents the projection event e′ k The weight of .

[0024] The step 3 comprises:

[0025] Step 1.3: Calculate the alignment event frame IWE corresponding to the j-th event packet using formula (5): j Any i-th pixel (x i,j ,y i,j )The probability of generating a valid event p i,j :

[0026]

[0027] In formula (5), n i,j The i-th pixel (x i,j ,y i,j ) in the jth alignment event frame IWE j The number of events generated on

[0028] Step 1.4: Calculate the alignment event frame IWE corresponding to the j-th event packet according to formula (6): j The expected total sum of squares of events is NTSS j :

[0029]

[0030] In formula (6), K represents the alignment event frame IWE j The total number of pixels.

[0031] The step 4 is to calculate using formula (7):

[0032]

[0033] The effective pixel number L j is the aligned event frame IWE corresponding to the jth event packet j It is obtained when M events are selected for interpolation.

[0034] The step 5 comprises:

[0035] Step 1.5: Calculate the effective structure ratio ESR of the j-th event packet according to formula (8): j :

[0036]

[0037] Step 1.6: Calculate event stream data using formula (9) The average event effective information structure ratio MESR is:

[0038]

[0039] The mean event effective information structure ratio MESR represents the event stream data The average noise level score.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention obtains event stream data with different noise levels by adding different types of ND lenses to simulate different levels of ambient light intensity in the scene, thereby solving the shortcoming that event cameras are difficult to obtain data of different lighting scenes.

[0042] 2. The present invention obtains the structural contrast in the scene by measuring the degree of alignment between events and edges in the event frame, and extracts scene invariants therefrom to obtain relative structural information of the scene. This information is independent of the number of projection events, and can thus measure the noise level of event data well. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a principle flow chart of obtaining noise event data of different degrees according to an embodiment of the present invention;

[0044] Figure 2 4 is a flowchart of a reference-free quality evaluation method according to an embodiment of the present invention. Specific implementation methods

[0045] In this embodiment, Figure 1 As shown, a noise quality evaluation method for event cameras includes the following steps:

[0046] Step 1: Install different types of ND filters in front of the event camera lens to simulate different ambient light intensities, so as to obtain event stream data with multiple noise levels. e k represents the kth event, N e Representing event stream data The total number of events included N e ; Initialize quality evaluation parameters, including: the kth event e k The pixel weight b k , the number of reference events N for a single evaluation, and the number of reference events M used for interpolation.

[0047] The specific implementation process is as follows Figure 1 As shown in the figure, first select the event scene to be photographed. The movement mode in the fixed scene is almost the same and the number of moving objects remains unchanged. Then, the camera is assumed to be on a tripod and ready to shoot. During the shooting process, different types of ND filters are installed in front of the lens according to the noise level to be simulated. Figure 2 Shooting situations simulating four noise levels are shown;

[0048] Step 2: Evenly divide the event stream data with multiple noise levels by the number of reference events N evaluated in a single time Get E event packets and use the reference time t ref Perform alignment to obtain an alignment event frame;

[0049] Step 2.1: Use formula (1) to obtain the event data D contained in the jth event packet: j :

[0050]

[0051] In formula (1), (x, y, t) is the space-time coordinate position, which represents the pixel coordinate of the event camera at time t, (x k ,y k ,t k ) is the kth event e k The space-time coordinate location of the occurrence, ck is the kth event e k The polarity of , δ is the Dirac function, j represents the sequence number of the event packet, E is the number of event packets, and satisfies:

[0052]

[0053] In formula (2), Indicates a rounding down operation. According to the initialization step of step 1, the number of events in a single event package is set to N = 30000. The number of event packages obtained by this setting is not likely to lose too much scene structure information, and can include the main effective time as much as possible, so as to more accurately evaluate the event flow noise.

[0054] Event polarity c k Meet c k =sgn((log(I t+Δt )-log(I t )) / σ), where, I t represents the scene brightness at time t, σ represents the contrast threshold of the camera for this event, and sgn(·) represents the sign function;

[0055] Step 2.2: The event data D in the jth event packet j Projection to a given reference time Take the time when the first event in the jth event package is generated as the reference time The value of event data D j The coordinates of satisfy the alignment equation shown in formula (3):

[0056]

[0057] In formula (3), W represents the mapping relationship, Indicates that the two satisfy the mapping relationship W, represents the kth event e k At reference time The projection coordinates of k′ represents the kth event e k Projection events under the mapping relation W;

[0058] Using formula (4), we can get the aligned event frame IWE corresponding to the jth event packet: j :

[0059]

[0060] In formula (4), (x i,j ,y i,j ) represents the alignment event frame IWE j At any i-th pixel position onk Represents the projection event e′ k The weight is set to b in step 1 k =1, under this setting, only the number of events generated along the trajectory is counted, that is, the aligned event frame IWE can be regarded as an event statistics graph obtained along the motion trajectory. The aligned event frame IWE obtained on this basis is conducive to subsequent derivation.

[0061] Step 3: Calculate the aligned event frame IWE corresponding to the jth event packet j The total sum of squares of events is expected to be NTSS j ;

[0062] Step 3.1: Calculate the aligned event frame IWE corresponding to the jth event packet j Any i-th pixel (x i,j ,y i,j )The probability of generating a valid event p i,j :

[0063]

[0064] In formula (5), n i,j The i-th pixel (x i ,y i ) in the jth alignment event frame IWE j The number of events generated on . The probability of a valid event p i,j , by Poisson distribution It is derived that, where λ x,y Indicates the alignment event frame IWE corresponding to the jth event packet j Any i-th pixel (x i ,y i )The average rate at which events are generated.

[0065] Step 3.2: According to formula (6), calculate the aligned event frame IWE corresponding to the jth event packet j The expected total sum of squares of events is NTSS:

[0066]

[0067] In formula (6), K represents the alignment event frame IWE j The total number of pixels in the event. The total sum of squares is calculated based on The calculated result is that the aligned event frame IWE corresponding to the j-th event packet is j The total sum of squares of events TSS j satisfy Total sum of squares (TSS) j Reflects the contrast information of the scene structure.

[0068] Step 4: Calculate the alignment event frame IWE corresponding to the jth event packet based on the number of reference events M j The number of effective pixels on L j , the number of effective pixels reflects the number of pixels that conform to a specific rule, such as the simplest, which statistically satisfies the jth alignment event frame IWE j The number of pixels where the cumulative events at any pixel in exceeds 1 can be obtained:

[0069]

[0070] Step 5: Expected NTSS based on the total sum of squares of the number of events j and the number of effective pixels L j , calculate the aligned event frame IWE corresponding to the jth event packet j Effective event structure ratio Repeat steps 3 to 4 to obtain the effective event structure ratio of all event packets, thereby obtaining multi-noise level event stream data. The average noise level score MESR is used as the quality evaluation result, where: = represents the assignment operation.

[0071] Step 5.1: Calculate the effective structure ratio ESR of the jth event packet according to formula (8): j :

[0072]

[0073] Step 5.2: Calculated ESR j , can only reflect the noise level of the current j-th event packet. In order to obtain the noise level contained in the entire event stream, it is only necessary to repeat steps 2 to 4 and use formula (9) to calculate the effective structure ratio ESR of the j-th event j , until all event packages are traversed and the event stream data is finally calculated The average event effective information structure ratio MESR is:

[0074]

[0075] Mean event effective information structure ratio MESR represents event stream data The average noise level score. Figure 2 As shown, the noise evaluation method is a calculation process of MESR of a single event stream. The quality evaluation scores of different ND filter event data can be obtained by repeating the process to evaluate the noise level of the ND filter.

Claims

1. A noise quality evaluation method for event cameras, characterized in that: The following steps are involved: step , different types of ND filters are installed in front of the event camera lens to simulate different ambient light intensities, so as to obtain event stream data with multiple noise levels ; Indicates events, Represents the event stream data The total number of events included; Initialize the quality evaluation parameters, including: Events The pixel weight , the number of reference events for a single evaluation , the number of reference events used for interpolation ; step , based on the number of reference events for a single evaluation Evenly split the event stream data with multiple noise levels ,get event packets, and reference time Perform alignment to obtain an alignment event frame; step , calculate the Aligned event frames corresponding to event packets The total expected sum of squares of events ; step , according to the number of reference events , calculate the Aligned event frames corresponding to event packets The number of effective pixels on ; step , according to the total expected sum of squares of the events and the number of effective pixels , calculate the first Aligned event frames corresponding to event packets Effective event structure ratio , and obtain the effective event structure ratio of all event packets according to the process of steps 3 to 4, thereby obtaining the multi-noise level event stream data Average noise level score , and as the quality evaluation result, among which, Represents an assignment operation.

2. According to the claim The noise quality evaluation method for event-oriented cameras is characterized in that: The steps include: Step 2.1: Use formula (1) to get Event data contained in the event package : (1) In formula (1), is the space-time coordinate position, indicating that the event camera is The pixel coordinates at the time, For the Events The space-time coordinates of the occurrence, It is Events The polarity of is the Dirac function, Represents the sequence number of the event package, is the number of event packets, and satisfies: (2) In formula (2), Indicates a round-down operation; Step 2.2: Event data in event packages Projection to a given reference time , and the event data The coordinates of satisfy the alignment equation shown in formula (3): (3) In formula (3), Represents the mapping relationship, Indicates that the two satisfy the mapping relationship , Representing the Events At reference time The projection coordinates of Representative Events In the mapping relationship The projection event below; Using formula (4), we can get the Aligned event frames corresponding to event packets : (4) In formula (4), Represents an alignment event frame Any The pixel position, Represents the projection event The weight of .

3. According to the claim The noise quality evaluation method for event-oriented cameras is characterized in that: The steps include: Step 3.1: Calculate the first Aligned event frames corresponding to event packets Any Pixels The probability of generating a valid event : (5) In formula (5), It is the Pixels In the Alignment event frames The number of events generated on Step 3.2: Calculate the first Aligned event frames corresponding to event packets The expected sum of squares of events : (6) In formula (6), Represents the alignment event frame The total number of pixels.

4. According to the claim The noise quality evaluation method for event-oriented cameras is characterized in that: The steps It is calculated using formula (7): (7) The number of effective pixels It is in Aligned event frames corresponding to event packets Select The value obtained when interpolating events.

5. According to the claim The noise quality evaluation method for event-oriented cameras is characterized in that: The steps include: Step 5.1: Calculate the first The effective structure ratio of event packages : (8) Step 5.2: Calculate event stream data using formula (9) The average proportion of effective information structure of events : (9) The average event effective information structure ratio Represents the event stream data The average noise level score.

Images