A superpixel-based event data reconstruction and quality assessment method

Abstract

The present invention provides a method for event data reconstruction and quality assessment based on superpixels, comprising the following steps: S1. Acquiring event data stream; S2. Constructing superpixels; S3. Single superpixel assessment; S4. Superpixel candidates; S5. Superpixel fusion; S6. Superpixel image stitching; S7. Image candidates; S8. Superpixel image whole picture assessment; and S9. Outputting assessment results. The method of the present invention mainly utilizes the characteristics of superpixels and event cameras to reconstruct and assess event data, so that the reconstructed event data can be well applied and assessed, reducing the application difficulty, solving the problems of the original data of the event camera being difficult to process and assess, improving the efficiency of event data use, and improving the ease of use and versatility of event data.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a superpixel-based event data reconstruction and quality assessment method. Background Art

[0002] An event camera is a new type of sensor that, unlike traditional cameras, does not capture complete images, but only outputs changes in pixel brightness, i.e., event data. Event cameras have the characteristics of low latency, high dynamic range, and extremely low power consumption, making them suitable for scenes with high-speed motion and dramatic changes in lighting.

[0003] The main problem with existing technologies is that event camera data, as a new type of non-traditional data, is not compatible with existing methods and applications. For event camera data, due to its unique structure, many current applications and data layers have barriers and cannot fully take advantage of it.

[0004] Superpixels are an image segmentation technology that aggregates pixels with similar color, texture, edge and other features into small areas, called superpixels. Unlike ordinary pixels, superpixels are composed of multiple ordinary pixels. They are divided according to the content of the image. Superpixels can retain the structure and semantic information of the image. Compared with ordinary pixels, superpixels can effectively reduce redundant information in images and improve the efficiency and accuracy of subsequent processing. Superpixels have become an established intermediate-level image representation method and are widely used in traditional image processing and computer vision fields.

[0005] The difficulty of solving the above problems and defects is: due to the novelty and uniqueness of event camera data, there is currently no universal solution, which shows that it is difficult to process.

[0006] The significance of solving the above problems is to provide a solution to difficult problems such as discrete asynchrony of event data, noise problems, resolution problems and visualization, so as to improve the efficiency of event data use and improve the usability and versatility of event data. Summary of the invention

[0007] The present invention provides an event data reconstruction and quality assessment method based on superpixels, which mainly utilizes the characteristics of superpixels and event cameras to reconstruct and assess event data, so that the reconstructed event data can be well applied and assessed, reducing the application difficulty, solving the problems of the original data of event cameras being difficult to process and assess, improving the utilization efficiency of event data, and improving the usability and versatility of event data.

[0008] The technical solution of the present invention is as follows:

[0009] The method for event data reconstruction and quality assessment based on superpixels of the present invention comprises the following steps: S1. Acquiring event data stream: using an event camera to acquire an asynchronous event stream in a scene; S2. Constructing superpixels: constructing superpixels one by one from the data in the asynchronous event stream; S3. Single superpixel evaluation: judging whether the construction of a single superpixel is completed according to the number of event camera pixels constituting the superpixel, and inputting the constructed superpixel into step S4; S4. Superpixel candidate: storing the constructed superpixel in a superpixel candidate area and indexing it in sequence; S5. Superpixel fusion: combining superpixels The overlapping superpixels in the candidate area are merged; S6. Superpixel image stitching: image stitching of superpixels in the superpixel candidate area; S7. Image candidate: a certain number of event superpixels are combined to generate a complete image, and whether the stitching is complete in the image candidate area is determined, and the complete superpixel event image is input into step S8 for evaluation; S8. Superpixel image whole picture evaluation: the complete superpixel event image is evaluated; and S9. Output evaluation results: the evaluation results of the event camera superpixel image are output to provide a basis for subsequent algorithm application.

[0010] Optionally, in the above-mentioned superpixel-based event data reconstruction and quality assessment method, in step S2, constructing superpixels one by one from the data in the asynchronous event stream includes: for event camera data, taking a timestamp as a reference, setting three constraints of distance constraint, polarity constraint, and time interval constraint to perform clustering constraint restrictions, generating event data superpixels one by one according to the similarities and continuity of event points, and limiting the number of ordinary event camera pixels contained in each superpixel.

[0011] Optionally, in the above-mentioned superpixel-based event data reconstruction and quality assessment method, in step S5, the superpixel fusion method is: when the overlapping area of ​​two superpixels is greater than or equal to half, the overlapping part is cut out to generate a new superpixel; when the overlapping area of ​​two superpixels is less than half, the overlapping part is assigned to the superpixel with a later timestamp.

[0012] Optionally, in the above-mentioned superpixel-based event data reconstruction and quality assessment method, in step S5, the superpixel image stitching method is: aggregate the superpixels that are aggregated and fused within an image space range into a complete image, obtain the current event camera chip resolution information from the event camera initial value, and construct a matrix consistent with the event camera chip resolution space; fill the superpixels in the matrix according to the spatial information, and set all the unfilled matrix parts to zero, wherein the splicing of superpixels is subject to the constraints of timestamps.

[0013] Optionally, in the above-mentioned superpixel-based event data reconstruction and quality assessment method, in step S8, the reciprocal of the boundary recall rate and the weighted average of the edge average distance are used as evaluation methods, and the reciprocal weighted average of the under-segmentation error and the variation interpretation is used as the evaluation index. The evaluation result directly reflects the quality of the event superpixel and indirectly reflects the quality of the event camera data.

[0014] According to the technical solution of the present invention, the beneficial effects produced are:

[0015] The superpixel-based event data reconstruction and quality assessment method of the present invention reconstructs and assesses the quality of event camera data by utilizing superpixel technology. The method can build a good intermediate layer, break through the direct barriers between data and applications, reduce application difficulty, expand the application scope of event data, and effectively improve the quality and application effect of event camera data.

[0016] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0018] Figure 1 It is a flow chart of the superpixel-based event data reconstruction and quality assessment method of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0020] The working principle of the event data reconstruction and quality assessment method based on superpixels of the present invention is as follows: the present invention uses a new type of sensor event camera to obtain scene data, and then uses a superpixel method to reconstruct the event camera data in view of the problems that event camera data is not universal, difficult to apply, and lacks quality assessment. The pixels of the event camera are aggregated one by one in superpixels, and then a complete image is reconstructed from multiple superpixel blocks. The reconstructed event data can be well applied and evaluated, which reduces the difficulty of application and improves the application efficiency.

[0021] like Figure 1 As shown, the superpixel-based event data reconstruction and quality assessment method of the present invention comprises the following steps:

[0022] S1. Get event data stream: First use the event camera to get the asynchronous event stream in the scene. The event stream can record the changes in the scene.

[0023] S2. Construct superpixels: Construct superpixels one by one from the data in the asynchronous event stream.

[0024] The event camera outputs independent asynchronous event camera pixels according to the different motion states of the scene. Different from the information contained in traditional pixels, each asynchronous event camera pixel contains four kinds of information: the horizontal axis x and vertical axis y of the spatial coordinates of the pixel, the time information t when the pixel is triggered, and the polarity information p of the pixel, where the value of p is usually (-1,1). The pixels of the event camera are continuously updated on the time axis. Due to the asynchronous nature of event data, multiple event superpixels can be aggregated in parallel at the same time.

[0025] Constructing superpixels from the data in the asynchronous event stream one by one includes: for event camera data, using a timestamp as a reference, setting three constraints, namely distance constraint, polarity constraint, and time interval constraint, to perform clustering constraint restrictions, generating event data superpixels based on the similarities and continuity of event points, and limiting the number of ordinary event camera pixels contained in each superpixel.

[0026] S3. Single superpixel evaluation: Determine whether the construction of a single superpixel is completed based on the number of event camera pixels that constitute the superpixel, and input the constructed superpixel into step S4.

[0027] S4. Superpixel candidate: The constructed superpixels are stored in the superpixel candidate area and indexed in order.

[0028] S5. Superpixel fusion: fuse the superpixels that overlap in the superpixel candidate regions.

[0029] Since the event camera has a high sampling frequency, when fusing superpixels, there may be a phenomenon of superpixel overlap in space. To address this problem, the superpixel fusion method of the present invention is as follows: when the overlap area of ​​two superpixels exceeds or equals half, the overlapped part is cut out to generate a new superpixel; when the overlap area of ​​two superpixels is less than half, the overlapped part is assigned to the pixel with the larger timestamp.

[0030] S6. Superpixel image stitching: perform image stitching on the superpixels in the superpixel candidate area.

[0031] The method of superpixel image stitching includes: aggregating superpixels that are aggregated and fused within an image space into a complete image, obtaining the current event camera chip resolution information from the event camera initial value, and constructing a matrix that is consistent with the event camera chip resolution space; filling the superpixels in the matrix according to the spatial information, and setting all the unfilled matrix parts to zero, wherein the stitching of superpixels is subject to the constraints of timestamps.

[0032] S7. Image candidate: combine a certain number of event superpixels to generate a complete image, determine whether the image candidate area is completely spliced, and input the complete superpixel event image into step S8 for evaluation.

[0033] S8. Whole-image evaluation of superpixel images: Evaluate the complete superpixel event image.

[0034] According to the method of evaluating superpixels in different application scenarios, the image reconstructed by superpixels can be continuously optimized using the evaluation results. For tasks such as target detection, target segmentation, and semantic segmentation, it is necessary to select a superpixel method that can maintain the boundary information of objects in the image, and to select an evaluation index that can measure the degree of overlap between the superpixel boundary and the true segmentation boundary. The present invention uses the reciprocal of the boundary recall rate and the weighted average of the average edge distance as the evaluation method. Both methods calculate the difference between the superpixel boundary and the true boundary. For tasks such as stereo matching, optical flow estimation, and target tracking, the present invention uses the reciprocal weighted average of the under-segmentation error and the change interpretation as the evaluation index. The evaluation result directly reflects the quality of the event superpixel and indirectly reflects the quality of the event camera data.

[0035] S9. Output evaluation results: Output the evaluation results of the event camera superpixel image to provide a basis for subsequent algorithm application.

[0036] The present invention proposes superpixels for event cameras, and for the first time introduces superpixel technology into event data, providing a low-level event data feature that can retain the boundary information of objects, reduce the complexity of event data, and improve processing efficiency and performance. The method of the present invention mainly utilizes the characteristics of superpixels and event cameras to reconstruct and evaluate event data, solving the problems that the original data of event cameras is difficult to process and evaluate, so that the reconstructed event data can be well applied and evaluated, reducing the difficulty of application, improving the efficiency of event data use, and improving the ease of use and versatility of event data, so that event camera data can be applied in multiple fields, such as detection, tracking, semantic segmentation and other fields.

[0037] Based on the method of the present invention and other methods for event camera reconstruction [1][2][3] The results of the method on the target tracking task are shown in Table 1.

[0038] Table 1 Method of the present invention and other methods for event camera reconstruction [1][2][3] Comparison of results of the methods on target tracking tasks

[0039]

[0040]

[0041] The proposed method is compared with similar international methods on target tracking tasks, and the comparison data set uses the internationally renowned data set FE108 [4] We use the internationally accepted comparison method, representative success rate (RSR) and overlap precision (OP). The higher the two evaluation values, the better the results. The experimental results show that our method is better than other similar methods.

[0042] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

[0043] References:

[0044] [1]Ramesh, B., Yang, H., Orchard, G., Le Thi, NA, Zhang, S., & Xiang, C. (2019). Dart: distribution aware retinal transform for event-based cameras. IEEE transactions on pattern analysis and machine intelligence, 42(11), 2767-2780.

[0045] [2] Liu, Q., Yuan, D., Fan, N., Gao, P., Li, X., & He, Z. (2022). Learning dual-level deep representation for thermal infrared tracking. IEEE Transactions on Multimedia, 25, 1269-1281.

[0046] [3]Zhang,J.,Zhao,K.,Dong,B.,Fu,Y.,Wang,Y.,Yang,X.,&Yin,B.(2021).Multi-domain collaborative feature representation for robust visual objecttracking.The Visual Computer,37(9-11),2671-2683.

[0047] [4]Zhang,J.,Yang,X.,Fu,Y.,Wei,X.,Yin,B.,&Dong,B.(2021).Objecttracking by jointly exploiting frame and event domain.In Proceedings of theIEEE / CVF International Conference on Computer Vision(pp.13043-13052)。

Claims

1. A superpixel-based event data reconstruction and quality assessment method, characterized in that: The following steps are involved: S1. Obtain event data stream: Use the event camera to obtain the asynchronous event stream in the scene; S2. Constructing superpixels: constructing superpixels one by one from the event pixel data in the asynchronous event stream; S3. Single superpixel evaluation: judging whether the construction of a single superpixel is completed according to the number of event camera pixels constituting the superpixel, and inputting the constructed superpixel into step S4; S4. Superpixel candidate: store the constructed superpixels in the superpixel candidate region and create indexes in order; S5. Superpixel fusion: fusion of superpixels that overlap in the superpixel candidate regions; S6. Superpixel image stitching: stitching the superpixels in the superpixel candidate area; S7. Image candidate: combine a certain number of event superpixels to generate a complete image, determine whether the splicing is complete in the image candidate area, and input the complete superpixel event image into step S8 for evaluation; S8. Whole-image evaluation of superpixel images: Evaluate the complete superpixel event image; as well as S9. Output evaluation results: Output the evaluation results of the event camera superpixel image to provide a basis for subsequent algorithm application.

2. The method for event data reconstruction and quality assessment based on superpixels according to claim 1, characterized in that: In step S2, constructing superpixels one by one from the data in the asynchronous event stream includes: for the event camera data, taking a timestamp as a reference, setting three constraints of distance constraint, polarity constraint, and time interval constraint to perform clustering constraint restrictions, generating event data superpixels one by one according to the similarities and continuity of event points, and limiting the number of ordinary event camera pixels contained in each superpixel.

3. The method for event data reconstruction and quality assessment based on superpixels according to claim 1, characterized in that: In step S5, the superpixel fusion method is as follows: when the overlapping area of ​​two superpixels is greater than or equal to half, the overlapping part is cut out to generate a new superpixel; when the overlapping area of ​​two superpixels is less than half, the overlapping part is assigned to the superpixel with a later timestamp.

4. The method for event data reconstruction and quality assessment based on superpixels according to claim 1, characterized in that: In step S5, the superpixel image stitching method is: aggregating the superpixels that are aggregated and fused within an image space into a complete image, obtaining the current event camera chip resolution information from the event camera initial value, and constructing a matrix that is consistent with the event camera chip resolution space; The superpixels are filled in the matrix according to the spatial information, and the unfilled matrix parts are all set to zero, wherein the splicing of the superpixels is subject to the constraints of the timestamp.

5. The method for event data reconstruction and quality assessment based on superpixels according to claim 1, characterized in that: In step S8, the reciprocal of the boundary recall rate and the weighted average of the edge average distance are used as evaluation methods, and the reciprocal weighted average of the under-segmentation error and the variation explanation are used as evaluation indicators. The evaluation results directly reflect the quality of the event superpixel and indirectly reflect the quality of the event camera data.

Images