Multi-scale voting mechanism denoising method for event camera

Abstract

The application discloses a multiscale voting mechanism denoising method for an event camera, and comprises the following steps: event stream data slice processing of event camera output, forming event stream data segments; according to the event camera pixel / resolution, the spatial position of the received event stream data segment is equally divided into multiple filtering areas according to the selected division scale, and each pixel area corresponding to each scale is a filtering area; the difference between the real event number and the average event number in each filtering area is calculated; noise voting of the filtering area pixel is carried out according to the positive and negative of the difference, noise voting results of the filtering area pixel under different scales are obtained, the noise voting results of the filtering area pixel under different scales are summed, and the final pixel noise voting result is obtained, and the noise event of the received event stream data segment is removed according to the final pixel noise voting result. The application directly processes event stream data, saves storage space and improves algorithm speed.

Description

Technical Field

[0001] This invention relates to the field of signal processing and image denoising technology, and in particular to a multi-scale voting mechanism denoising method for event cameras. Background Technology

[0002] Event-based cameras differ from traditional frame-based cameras in that they only output events when they detect changes in light intensity. Their data format is an event stream, containing temporal, spatial, and event polarity information. Event stream data has three key characteristics compared to traditional frame-based image data: first, at the same frame rate, its data volume is significantly smaller than that of a frame-based image; second, the time intervals between events are much shorter, often down to the microsecond level; and third, it has a wider dynamic range. Due to these characteristics, event cameras have great potential for development in the fields of autonomous driving and robotics.

[0003] In practical use of event cameras, various interferences from the real environment often generate unnecessary noise events. Because noise and valid signal characteristics are quite similar, it is often difficult for the human eye to accurately distinguish them. Such noise events hinder subsequent image processing tasks and may also interfere with the accuracy of target detection or tracking. Therefore, denoising for event cameras is essential.

[0004] Currently, there are two main methods for denoising event stream data. The first method converts the event stream data into image frame format and then filters it using traditional denoising algorithms. For example, Chinese patent CN112053290A, entitled "Unsupervised Event Camera Denoising Method and Apparatus Based on Convolutional Denoising Autoencoder," first converts the event stream into frame format and then trains it using a neural network to achieve denoising. The second method primarily denoises based on the spatiotemporal attributes of the event stream data itself. For example, Chinese patent CN114885074A, entitled "An Event Camera Denoising Method Based on Spatiotemporal Density," first acquires the event stream and then filters events that are far removed from the target event in time.

[0005] Both methods can achieve good denoising results. However, frame-based denoising algorithms impose certain computational and storage burdens, while event-stream-based denoising algorithms may suffer from poor denoising performance because they fail to consider the overall image features. Therefore, it is essential to consider both the computational speed of event-based camera denoising algorithms and the overall image features to achieve better denoising results. Summary of the Invention

[0006] The purpose of this invention is to address the problems in the prior art by providing a multi-scale voting mechanism denoising method for event cameras. This method sets up multi-scale filtering zones to evaluate both overall and local features in the event image. By using a voting mechanism between different scales, the influence of each scale on the final denoising result is determined, thereby improving the denoising accuracy. The entire process directly operates on the event stream data, reducing the computational and storage pressure on the computing device.

[0007] This invention is implemented as follows: a multi-scale voting mechanism denoising method for event cameras, comprising the following steps:

[0008] The event stream data output from the event camera is sliced ​​and processed to form event stream data fragments;

[0009] The spatial location of the received event stream data segments is divided into multiple filtering zones based on the event camera pixels / resolution and a selected partitioning scale, with each pixel area corresponding to each scale being a filtering zone; the difference between the number of actual events and the average number of events in each filtering zone is calculated.

[0010] Based on the sign of the difference, noise voting is performed on the pixels in the filtering area to obtain noise voting results for pixels in the filtering area at different scales. The noise voting results for pixels in the filtering area at different scales are summed to obtain the final pixel noise voting result. Noise events in the received event stream data segments are removed based on the final pixel noise voting result.

[0011] The slicing process is performed based on equal time intervals or equal event intervals.

[0012] The event stream data segment output by the event camera is shown below:

[0013] E v =(E vx E vy E vp E vt v = 1, 2, ..., E all ;

[0014] Among them, E v E represents the v-th event in a segment of event stream data output by the event camera. all This represents the total number of events in this event stream data segment, with each event containing a horizontal coordinate E. vx , ordinate E vy Polarity E vp Time E vt .

[0015] The space is divided into equal-area sections based on the selected partitioning scale. The resulting multiple filtering zones are calculated using the following formula:

[0016]

[0017] Where L and H are the maximum horizontal and maximum vertical pixel values ​​of the event camera, respectively, and m i and n i These are the preset maximum horizontal and maximum vertical pixel values ​​for the filter area, respectively, N i The number of filtering zones is denoted by i, where i is the scale number and n is the number of scales required. The size of the filtering zones is the same for each scale.

[0018] The average number of events within each filtering zone is calculated using the following formula:

[0019]

[0020] Among them, E mij Let be the average number of events in the j-th filter zone at the i-th scale, where j is the filter zone number at a certain scale.

[0021] The number of real events in each filtering zone is calculated using the following formula:

[0022]

[0023] Where E represents the original event, and D... ij E represents the range of the j-th filter region at the i-th scale. sij Let be the number of real events in the j-th filter region at the i-th scale.

[0024] The step of performing noise voting on the pixels in the filter region based on the sign of the difference to obtain noise voting results for the pixels in the filter region at different scales is shown below:

[0025]

[0026] (x,y)∈D ij ,

[0027] i = 1, 2, ..., n; j = 1, 2, ..., N i ;

[0028] Among them, I i,(x,y) Let (x, y) be the voting result at the i-th scale, and (x, y) be the coordinates of each pixel of the event camera.

[0029] The noise voting results of the pixels in the filter area at different scales are summed to obtain the final pixel noise voting result, as shown in the following formula:

[0030]

[0031] Among them, I (x,y) The accumulated voting results across n scales are calculated from event stream data segments and used as a filter template for subsequent filtering operations on the event stream data segments.

[0032] The step of removing noisy events from the received event stream data segments based on the final pixel noise voting result includes:

[0033] The voting result I after summing across n scales (x,y) Each event in the event stream data segment output by the event camera is sequentially determined to be noise. If it is noise, it is deleted, and finally all noise is removed.

[0034] Specifically, when determining whether each event in the event stream data segment output by the event camera is noise, if the number of pixels in the event is less than 0, it is considered noise; otherwise, it is considered non-noise. This is expressed as follows:

[0035]

[0036] v = 1, 2, ..., E all ;

[0037] In the formula, Represents the pixels in the event.

[0038] The denoising method disclosed in this invention considers signal-noise discrimination at different scales, and the scale is adjustable, which has better robustness for the denoising process of event stream data output by event cameras in different scenarios; at the same time, it adopts a voting mechanism to determine whether an event is noise, which improves the accuracy of the denoising algorithm; it processes directly based on event stream data, without the need to convert the event stream into image frames first, which saves storage space and improves the algorithm speed, which has positive significance for the application of event cameras. Attached Figure Description

[0039] Figure 1 This is a flowchart of a multi-scale voting mechanism denoising method for event cameras according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the overall processing architecture of an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the calculation process for the denoising process in an embodiment of the present invention.

[0042] Figure 4 This is an example of the denoising effect at various scales, using an indoor segment from the msecv dataset as an example in this embodiment of the invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0044] The embodiments of the present invention first establish a multi-scale denoising template, then establish a voting mechanism based on the multi-scale denoising template, and finally perform denoising processing on the event stream data of the event camera according to the voting results of the voting mechanism, thus completing the final denoising process of the event stream data.

[0045] The processing flow includes: The event camera acquires event stream data, and the event stream data is segmented into multiple segments by a computer or other computing device, with the segmentation method being equal time intervals or equal event count intervals. Multiple scales are defined based on the event camera's resolution, and each pixel area corresponding to each scale is called a filtering area. The difference between the average number of events in the filtering area and the actual number of events in the filtering area at each scale is calculated. If the difference is greater than or equal to 0, the filtering area receives a positive vote; otherwise, it receives a negative vote. The voting results at the same spatial location at each scale are linearly added to obtain the final voting result. If the voting result is greater than or equal to 0, all events within the filtering area are considered valid; otherwise, they are considered noise events and are removed, ultimately achieving noise reduction.

[0046] The event stream data output by the event camera is sliced ​​into event stream data segments during the denoising process. The event stream data segments formed after denoising are then denoised. Each event stream data segment may include multiple events, and each event contains information such as horizontal and vertical coordinates (x, y), polarity p, and time t.

[0047] Below, using an indoor segment of the MVSEC general event dataset as an example, the calculation process will be explained in detail. This event stream data segment contains a total number of events E. all The value is 9063. The maximum value of the event camera's pixel ordinate L is 260, and the maximum value of its lateral coordinate W is 346.

[0048] Please see Figure 1 as well as Figure 2 , Figure 3 As shown, the multi-scale voting mechanism denoising method for event cameras provided in this embodiment of the invention is implemented by the following steps:

[0049] S1: Denoising Template Creation

[0050] The system uses a computer or other computing device to receive event stream data segments output by the event camera and performs multi-scale filtering calculations. Specifically, the event camera pixels are divided into multiple filtering zones according to different scales, with n scales and N filtering operations per scale. iThere are several filter zones; specifically, the average number of events and the actual number of events are calculated within the j-th filter zone at the i-th scale. The actual number of events within a filter zone refers to the total number of events in a slice that appear in that filter zone, and the average total number of events (or mean event count) within a filter zone refers to the average number of events in a slice that appear in each filter zone.

[0051] Assuming two scales are chosen for filtering, the m corresponding to each scale... i and n i As shown in the following formula:

[0052]

[0053] i = 1, 2;

[0054] Where, m i and n i Let be the length and width of the filter region at the i-th scale.

[0055] N i The specific calculation for the number of filter zones is as follows:

[0056]

[0057] i = 1, 2;

[0058] N1 = 1, N2 = 4;

[0059] The number of filter zones for scale 1 is 1, and the number of filter zones for scale 2 is 4.

[0060] Next, it is necessary to count the number of events in the filtering zone at each scale, that is, to calculate the average number of events and the number of real data in the j-th filtering zone at the i-th scale.

[0061] The average number of events in each filter region is:

[0062]

[0063] Knowing the total number of events, the average number of events in each filtering zone can be calculated. This average number of events serves as a standard for the presence of noise, providing a certain condition for subsequent judgment.

[0064] The number of real events in each filter region is:

[0065]

[0066] Through the above processing, the original event representation is partitioned for calculation, and the number of real events and the average number of events in each filtering zone are obtained.

[0067] S2. Establish and vote on the noise voting mechanism in the filtering region:

[0068] In this invention, the noise voting mechanism refers to voting in the filtering area at each scale, and summing the votes for each spatial location (filtering area, i.e., pixel area) at all scales to obtain the final voting result.

[0069] Once the average number of events and the number of real events in each filter region at each scale are obtained, noise voting can be performed on the filter region at that scale. Taking the two scales mentioned above as examples, the voting result is obtained by comparing the average number of events and the number of real events in each filter region at each scale, based on the difference, as follows:

[0070]

[0071] If the number of real events is greater than the average number of events (i.e., a positive value), then the filter or voting result for this region's pixels is counted as 1 vote; otherwise, -1 vote is counted. This comparison continues until all pixels in multiple filtering regions across multiple scales have completed their voting, resulting in multiple voting results, i.e., filters or filtering templates.

[0072] Since the density of noise data in time and space is less than that of signal data, comparing it with the average number of events within the filtering zone can initially filter out noise signals. However, this method can lead to errors depending on the size of the filtering zone. If only the first scale of the filtering zone is used for judgment, all events will be considered noise and removed, resulting in a large error. If only the filtering zone at a small scale is used for judgment, some areas with more noise in certain small regions cannot be effectively removed. Therefore, multi-scale voting can effectively achieve the above-mentioned denoising purpose.

[0073] S3: The computer or other computing device accumulates the voting results of multiple filtered filtering regions (filters at each scale) to obtain the final denoising result (final filter), and then uses the final denoising result to filter and denoise the events in the event stream data segment.

[0074] S31. Voting results of the cumulative filtering region:

[0075] By adding the corresponding filters (noise voting results) at the two scales, the final filter (noise voting result) is obtained.

[0076] The filter contains filtering information at two different scales, as follows:

[0077]

[0078] S32. Denoising based on voting results:

[0079]

[0080] v = 1, 2, ..., E all ;

[0081] The final denoising process only requires inputting the x and y coordinates of each event in the event stream data segment into the judgment. If it is noise, then the removal operation is performed, and finally the denoising process of the entire event stream data segment is completed.

[0082] It should be noted that in this invention, the division of the filtering area is based on the resolution of the event camera. The number of scales and the size of each scale are set based on experience, and can be manually set according to the actual situation. After this setting, the entire space of the event camera pixels can be divided into multiple filtering areas of equal area according to the resolution information / pixel information of the event camera. This facilitates subsequent voting on the filtering area, and then noise reduction processing is performed using the voting results according to the aforementioned method.

[0083] The specific operation processes of steps two and three in this embodiment of the invention are as follows: Figure 3 As shown in the previous explanation, the process will not be repeated here.

[0084] To further illustrate the effectiveness of the method of the present invention, based on the above calculation process, the present invention also experimented with the denoising process at scale numbers 3, 4, 5, 6, and 7, and the results are as follows. Figure 4 As shown.

[0085] It can be clearly observed that as the number of scales i increases, the area of ​​erroneous denoising gradually decreases. When i=7, the background gate part in the mvsec dataset image is already very complete, and the noise signal in the surrounding open area is also filtered out. It can be considered that the signal and noise are well distinguished. The denoising algorithm has achieved good results and has a positive effect on the further development and application of event camera data streams.

[0086] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-scale voting mechanism denoising method for event cameras, characterized in that, Including the following steps: The event stream data output from the event camera is sliced ​​and processed to form event stream data fragments; The spatial location of the received event stream data segments is divided into multiple filtering zones based on the event camera pixels and the selected division scale, with each pixel area corresponding to each scale being a filtering zone. Calculate the difference between the number of actual events and the average number of events in each filter region; Based on the sign of the difference, noise voting is performed on the pixels in the filtering region to obtain the noise voting results of the pixels in the filtering region at different scales, as shown below: (x, y) e D ij , i = 1, 2,..., n; j = 1, 2,..., N i ; Among them, I i,(x,y) The voting results are for the i-th scale, where x and y are the coordinates of each pixel of the event camera; D ij E represents the range of the j-th filter region at the i-th scale. sij E represents the number of real events in the j-th filter region at the i-th scale. mij N represents the average number of events within the j-th filter region at the i-th scale; i The number of filter zones is defined, and the size of the filter zones is equal at each scale. The noise voting results of pixels in the filter region at different scales are summed to obtain the final pixel noise voting result. Based on this final pixel noise voting result, noisy events in the received event stream data segments are removed, including: The voting results accumulated at n scales are used to sequentially determine whether each event in the event stream data segment output by the event camera is noise. If it is noise, it is deleted, and finally all noise is removed. If the pixel value in the event is less than 0, it is noise; otherwise, it is not noise.

2. The method of claim 1, wherein, The slicing process is performed based on equal time intervals or equal event intervals.

3. The method of claim 1, wherein, The event stream data segment output by the event camera is shown below: E v =(E vx ,E vy ,E vp ,E vt ),v=1,2,...,E all ; Among them, E v E represents the v-th event in a segment of event stream data output by the event camera. all This represents the total number of events in this event stream data segment, with each event containing a horizontal coordinate E. vx , ordinate E vy Polarity E vp Time E vt .

4. The method of claim 1, wherein, The space is divided into equal-area zones based on the selected partitioning scale. The resulting multiple filtering zones are calculated using the following formula: Where L and H are the maximum horizontal and maximum vertical pixel values ​​of the event camera, respectively, and m i and n i These are the preset maximum horizontal pixel value and maximum vertical pixel value of the filter area, respectively.

5. The method of claim 3, wherein, The average number of events in each filtering zone is calculated using the following formula:

6. The method of claim 1, wherein, The number of real events in each filtering zone is calculated using the following formula: Where E represents the original event.

7. The method of claim 1, wherein, The noise voting results of the pixels in the filter region at different scales are summed to obtain the final pixel noise voting result, as shown in the formula: where I (x,y) is the accumulated voting result under n scales, which is calculated through the event stream data segment, and is used as a filter template for subsequent filtering operation on the event stream data segment.

8. The method of claim 3, wherein, The process of sequentially determining whether each event in the event stream data segment output by the event camera is noise is as follows: In the formulae, represents a pixel in an event.

Images