A method and system for event camera data quality assessment

Abstract

A method for event camera data quality evaluation, comprising the following steps: S1. binocular setting; S2. data synchronization; S3. camera calibration; S4. data set acquisition; S5. data alignment and registration; S6. data reconstruction; S7. model design training; S8. model deployment; S9. data evaluation. The method and system for event camera data quality evaluation can effectively solve the binocular design of the existing event camera and high-speed camera, the synchronization and data calibration and correction alignment of the event camera and high-speed camera, the contrast annotation of event data, and the model training and deployment question, and improve the quality and availability of event camera data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular, to a method and system for evaluating the quality of event camera data. Background Technology

[0002] Event Camera [1] An event camera is a novel type of visual sensor that does not acquire images at a fixed frame rate but outputs events based on changes in pixel brightness. Event cameras offer advantages such as high temporal resolution, low latency, low power consumption, and high dynamic range, making them suitable for high-speed motion scenes, low-light environments, and multi-source conditions. Event camera data quality assessment refers to the analysis and evaluation of the data output by the event camera to determine whether it meets the expected effects and requirements. Event camera data quality assessment is crucial for the design, debugging, optimization, and application of event cameras. However, due to the unique characteristics of event camera data, such as noise, sparsity, and directionality, event camera data quality assessment faces numerous challenges. For example, how to define and quantify the quality indicators of event camera data, and how to utilize the relationship between event camera data and traditional camera data for evaluation, are all challenges.

[0003] The industry has long lacked effective methods for assessing and evaluating the quality of event camera data. This has resulted in a lack of reference points for the iterative updates of event cameras and the processing of downstream applications, thus hindering the development of event cameras. Technical issues such as the binocular design of event cameras and high-speed cameras, synchronization between event cameras and high-speed cameras, data calibration and alignment, comparative annotation of event data, and model training and deployment are all absent in existing technologies. Therefore, the lack of effective data quality assessment methods for event cameras not only affects the iterative updates of event camera hardware but also impacts downstream tasks such as target recognition based on event cameras. [2] .

[0004] The difficulty in solving the above problems and defects lies in the fact that, as an emerging information acquisition sensor, the event camera differs greatly from traditional cameras. Current traditional methods and technical approaches are no longer sufficient to solve this problem. It is necessary to propose new methods from scratch and continuously improve them. Therefore, solving the above problems is very difficult.

[0005] The significance of solving the above problems and shortcomings is that by proposing a new method to solve the problem of event camera data quality assessment, it provides an important foundation for the hardware design of event cameras and downstream tasks of event cameras, and has great research and application significance. Summary of the Invention

[0006] This invention provides a method and system for evaluating the quality of event camera data. Through the binocular design of event camera and high-speed camera proposed in this invention, the synchronization of event camera and high-speed camera, the calibration and correction alignment of data, the comparison and annotation of event data, and the model training and deployment, the data collected by event camera can be effectively evaluated, providing an important foundation for the hardware development of event camera and downstream tasks of event camera.

[0007] The technical solution of the present invention is as follows:

[0008] According to one aspect of the present invention, a method for evaluating the quality of event camera data is provided, comprising the following steps: S1. Binocular setup: designing data acquisition and deploying the binocular camera; S2. Data synchronization: simultaneously acquiring data from a high-speed camera and an event camera using the binocular camera, and employing a synchronization trigger to ensure data synchronization between the two cameras; S3. Camera calibration: designing the calibration of the event camera and the high-speed camera using an electronic calibration board, and deriving the calibration results; S4. Data acquisition: determining the scene and conditions for acquiring the dataset, and acquiring data according to different lighting conditions, shooting distances, and shooting angles to ensure the diversity of the acquired dataset; S5. Data alignment and registration: processing the acquired data from the high-speed camera and the event camera, and utilizing... The calibration results obtained in step S3 are used to correct and register the event camera data and high-speed camera data; S6. Data Reconstruction: The event camera data is reconstructed so that the reconstructed event camera data and high-speed camera data have the same frame rate, ensuring that the event camera data and high-speed camera data are consistent at the same time; S7. Model Design and Training: A referenced evaluation network model and a non-referenced evaluation network model are constructed and trained; S8. Model Deployment: The referenced evaluation model and the non-referenced evaluation network model trained in step S7 are deployed on the server; S9. Data Evaluation: The data quality of the event camera is evaluated using the referenced evaluation network and the non-referenced evaluation network trained in step S7, and the score value of the event camera data quality is output and used to specify the scheme for downstream tasks.

[0009] Optionally, in the above method for evaluating the quality of event camera data, in step S1, the data acquisition design includes: using a binocular camera consisting of an event camera and a high-speed camera to acquire data. The event camera and the high-speed camera are respectively set at both ends of a fixed axis and connected to a synchronization trigger and a high-speed data memory respectively. The synchronization trigger uses a clock pulse to simultaneously trigger the event camera and the high-speed camera to align the timelines of the event camera and the high-speed camera. The high-speed data memory is used to store the data collected by the high-speed camera, and the fiber optic transmission device is used to transmit the high-speed camera data.

[0010] Optionally, in the above method for evaluating the quality of event camera data, in step S3, the calibration results include the intrinsic parameter matrix, distortion parameter, rotation matrix, and translation matrix of the event camera and the high-speed camera.

[0011] Optionally, in the above method for evaluating the quality of event camera data, in step S6, the event camera data reconstruction method is to convert the event camera data that occurs at the same time as the high-speed camera output frame into a grayscale image with a brightness value of 0.255.

[0012] Optionally, in the above method for evaluating the quality of event camera data, in step S7, the model design includes constructing a referenced evaluation network model and a non-referenced evaluation network model. The referenced evaluation network model is used to compare event data and high-speed camera data to generate evaluation values ​​of event data quality, which can be used as annotation information. The non-referenced evaluation network model is trained on the event data after obtaining the annotation information.

[0013] Optionally, in the above method for quality assessment based on event camera data, in step S7, a referenced evaluation network model is first trained. The training requires both event camera data and high-speed camera data. After training, an evaluation value for the event camera data is generated. The evaluation value is then used as a label value to train a non-referenced evaluation network model.

[0014] Optionally, in the above method for evaluating the quality of event camera data, in step S9, if the data acquired by the event camera and the high-speed camera binoculars can be evaluated using a referenced evaluation network model, then if the acquired data only contains event camera data, then a non-referenced evaluation network model is used for evaluation.

[0015] According to one aspect of the present invention, a system for evaluating the quality of event camera data is provided, characterized in that it comprises: a binocular camera, a synchronization trigger, a high-speed data storage device, and an optical fiber transmission device, wherein the binocular camera consists of an event camera and a high-speed camera, used for data acquisition, wherein the event camera and the high-speed camera are respectively disposed at opposite ends of a fixed axis and respectively connected to the synchronization trigger and the high-speed data storage device, the synchronization trigger using a clock pulse to simultaneously trigger the event camera and the high-speed camera to align the timelines of the event camera and the high-speed camera, the high-speed data storage device is used to store the data collected by the high-speed camera, and the optical fiber transmission device is used to transmit the high-speed camera data.

[0016] The beneficial effects of the technical solution of the present invention are as follows:

[0017] This invention implements a method and system for evaluating the quality of event camera data. It can utilize the relationship between event camera data and traditional high-speed camera data to evaluate the quality of event camera data from multiple angles and levels using a transformers network model, thereby improving its comprehensiveness and accuracy. It can also utilize the characteristics of event camera data itself and use traditional high-speed camera data as a reference to conduct effective evaluations of event cameras, and design a series of new methods to interface event cameras and traditional cameras.

[0018] To better understand and illustrate the concept, working principle, and effects of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments: Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a flowchart of the method for evaluating the quality of event camera data according to the present invention;

[0021] Figure 2 This is the calibration plate used in this invention;

[0022] Figure 3 This is a schematic diagram of the system for evaluating the data quality of event camera data according to the present invention and its data acquisition process. Detailed Implementation

[0023] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the scope of the invention.

[0024] The method and system for evaluating the quality of event camera data utilizes a binocular design combining an event camera and a high-speed camera, along with tools such as event camera calibration. Using high-speed camera data as a reference, the quality of event camera data is labeled, and the network model is trained and optimized by collecting data from different scenarios. This invention, as a novel data acquisition device, effectively evaluates the quality of event camera data through the collection and labeling of event data and the use of deep learning technology.

[0025] The working principle of this invention is based on the assumption that there are certain relationships or similarities between event camera data and traditional high-speed camera data. For example, edges in event camera data are aligned with edges in traditional camera images; brightness variations in event camera data are consistent with brightness values ​​in traditional camera images; and optical flow in event camera data matches optical flow in traditional camera images. The invention utilizes these relationships or similarities to construct a network model using transformers to solve for unknown variables in the event camera data, such as noise, optical flow, depth, and motion parameters. These variables can reflect the quality or usability of the event camera data and can also be used to reconstruct high-resolution images or videos. Then, evaluation metrics such as noise ratio, contrast, sharpness, and stability are used to quantify the quality of the event camera data and compare or analyze it with traditional camera data. These metrics can reflect the performance and effectiveness of event camera data under different application scenarios and requirements.

[0026] This invention designs a binocular data acquisition system. After acquiring a large amount of effective data, it effectively reconstructs, aligns, and labels data from event cameras and high-speed cameras. Then, it uses deep learning methods for data training. After training, it achieves effective evaluation of the event data. Figure 1 As shown, the specific implementation steps are as follows:

[0027] S1. Binocular Setup: Design the data acquisition process and configure the deployment of the binocular cameras.

[0028] The data acquisition design includes using a binocular data acquisition unit (i.e., a binocular camera) consisting of an event camera and a high-speed camera to acquire data. The event camera and high-speed camera are positioned at opposite ends of a fixed axis, respectively connected to a synchronization trigger and a high-speed data storage device. The synchronization trigger uses clock pulses to simultaneously trigger both the event camera and the high-speed camera, aligning their timelines. The data generated by the high-speed camera far exceeds the capacity of ordinary network and storage systems. Therefore, this invention designs a dedicated high-speed data storage device and fiber optic transmission equipment. The high-speed data storage device stores the data collected by the high-speed camera, and the fiber optic transmission equipment transmits the high-speed camera data.

[0029] S2. Data Synchronization: Use a stereo camera to simultaneously acquire data from a high-speed camera and an event camera, and employ a synchronization trigger to ensure that the data acquired by the event camera and the high-speed camera in the stereo camera are synchronized.

[0030] S3. Camera Calibration: Design the calibration of event cameras and high-speed cameras using an electronic calibration board, and export the calibration results, including the intrinsic parameter matrix, distortion parameters, rotation matrix, translation matrix, etc. of the event cameras and high-speed cameras.

[0031] Since event cameras can only capture dynamic data, traditional static calibration boards are unsuitable for calibrating event cameras. Therefore, this invention employs a dynamic electronic calibration board for dual-target calibration of event cameras and high-speed cameras, wherein, for example... Figure 2 As shown, the black and white colors in the checkerboard of the electronic calibration board (electronic display) alternate at a certain frequency, that is, black and white are displayed alternately. In this way, the event camera can capture the data presented by the calibration board. At the same time, the calibration board can move left and right to complete the calibration task at different angles.

[0032] S4. Dataset Acquisition: Determine the scene and conditions for data acquisition, and collect data according to different lighting conditions, shooting distances, shooting angles, etc., to ensure the diversity of the acquired dataset. This step involves acquiring the dataset for model training.

[0033] S5. Data Alignment and Registration: Process the acquired high-speed camera and event camera data, and use the calibration results (i.e., camera parameters) obtained in step S3 to perform correction and registration of the event camera data and high-speed camera data.

[0034] S6. Data Reconstruction: Reconstruct the event camera data so that the reconstructed event camera data and high-speed camera data have the same frame rate, ensuring that the event camera data and high-speed camera data are consistent at the same time.

[0035] The event camera data reconstruction method is as follows: This invention uses the data captured by the high-speed camera as the reference image. Since the event camera data output is in the form of pulse data, it is necessary to reconstruct the pulse data. The reconstruction strategy is to convert the event camera data that is in the same time as the high-speed camera output frame into a grayscale image with a brightness value of 0.255.

[0036] S7. Model Design and Training: Construct and train both a referenced evaluation network model and a non-referenced evaluation network model. First, train the referenced evaluation network model, which requires both event camera data and high-speed camera data. After training, evaluation values ​​for the event camera data will be generated. These evaluation values ​​will then be used as labels to train the non-referenced evaluation network model. The labels (i.e., event data labels) are generated by evaluating the data using the referenced evaluation network model, obtaining a score, and then using this score as a label to train the non-referenced evaluation network model.

[0037] Specifically, regarding model design and training: This invention proposes two types of evaluation network model construction: one is a referenced evaluation network model, used to compare event data and high-speed camera data to generate evaluation values ​​for the quality of the event data, which can be used as annotation information; the other is a non-referenced evaluation network model, trained on event data after obtaining annotation information. Both types of evaluation network models are constructed using transformer models. One is a referenced evaluation network model, and the second is a non-referenced evaluation network model. Because the calculated loss functions are different, the referenced evaluation method requires calculating the loss function of the direct similarity between the evaluation image and the reference image, while the non-referenced model requires calculating the loss function of the difference between the evaluation image and the annotation value. However, the features learned by the two network models are relatively consistent, so the backbone network can be the same.

[0038] S8. Model Deployment: Deploy the referenced evaluation model and the unreferenced evaluation network model trained in step S7 on the server.

[0039] S9. Data Evaluation: The trained referenced evaluation network and unreferenced evaluation network from step S7 are used to evaluate the data quality of the event camera, outputting a score for the event camera data quality, which is then used to specify the approach for downstream tasks. Data acquired using both the event camera and high-speed camera binoculars can be evaluated using the referenced evaluation network model; if the acquired data only includes event camera data, then the unreferenced evaluation network model is used.

[0040] like Figure 3 As shown, the system for evaluating the data quality of event cameras according to the present invention includes: a stereo camera, a synchronization trigger 3, a high-speed data storage device 4, and an optical fiber transmission device 5. The stereo camera consists of an event camera 1 and a high-speed camera 2 for data acquisition. Event camera 1 and high-speed camera 2 are respectively positioned at opposite ends of a fixed axis and connected to the synchronization trigger 3 and the high-speed data storage device 4. The synchronization trigger 3 uses clock pulses to simultaneously trigger event camera 1 and high-speed camera 2 to align their timelines. The data generated by high-speed camera 2 far exceeds the limits that ordinary network and storage systems can handle. The high-speed data storage device 4 stores the data collected by the high-speed camera, and the optical fiber transmission device 5 transmits the high-speed camera data.

[0041] This invention addresses the problem of evaluating the output data quality of a novel sensor; it implements a method for evaluating the quality of event camera data; and, in response to the current lack of quality evaluation for event data, this invention utilizes a model training strategy that uses high-speed camera data as a reference. It employs a binocular camera setup, designs camera calibration and data alignment schemes, collects data from multiple scenes, and designs an evaluation network model based on a transformer architecture. Using high-speed camera data as a reference, it trains the data quality labeling and evaluation algorithm model for event cameras. The trained model can be applied to evaluate event camera data collected in different scenarios.

[0042] This invention primarily verifies its effectiveness in two ways: firstly, by comparing the event data before and after processing with the filtering algorithm; and secondly, by using the evaluation method of this invention to screen high-quality event data for target identification.

[0043] Experiment 1: Spatial convolution filtering method is used to process event cameras. [3] It is a proven effective event camera data filtering method that utilizes the spatial correlation of event camera data to perform spatial convolution filtering, remove isolated noise points, and smooth edge information.

[0044] Event1-Event5 in the dataset represent datasets from different scenarios. Five different scenarios were used in this experiment. The MOS score is the result value of the evaluation output of this invention, ranging from 1 to 5 points. The results are shown in Table 1. After processing with the spatial convolution filtering method, the quality of the event data was significantly improved, verifying the effectiveness of this invention.

[0045] Table 1

[0046] Dataset Before filtering (MOS) After filtering (MOS) Event 1 2 3 Event2 4 4 Event 3 3 4 Event4 2 4 Event 5 2 3 Mean 2.6 3.6

[0047] Experiment 2: After evaluation using the method of this invention, for continuously collected event data, high-scoring event data from Event1 to Event5 are selected for target recognition tasks to improve the accuracy of recognition.

[0048] Table 2 Comparison of the accuracy of target recognition using screened data and randomly screened data in this invention.

[0049]

[0050] This experiment selected data from 5 scenarios for target recognition. By comparing the accuracy of target recognition with that of randomly selected data, the results show that the accuracy of the high-quality data selected in this invention is improved by 6.3%-7.7%, proving the effectiveness of the method of this invention.

[0051] The above description represents the preferred embodiment based on the inventive concept and working principle. The above embodiments should not be construed as limiting the scope of protection of these claims; other embodiments and combinations of implementations based on the inventive concept are all within the scope of protection of this invention.

[0052] References:

[0053] [1] Gallego, Guillermo, et al. "Event-based vision: A survey." IEEE transactions on pattern analysis and machine intelligence 44.1 (2020): 154-180.

[0054] [2] Kim, Junho, Inwoo Hwang, and Young Min Kim. "Ev-tta: Test-timeadaptation for event-based object recognition." Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022.

[0055] [3] Scheerlinck, Cedric, Nick Barnes, and Robert Mahony. "Asynchronousspatial image convolutions for event cameras." IEEE Robotics and Automation Letters 4.2 (2019): 816-822.

Claims

1. A method for evaluating the quality of event-based camera data, characterized in that, Includes the following steps: S1. Binocular Setup: Design the data acquisition process and configure the deployment of the binocular cameras; S2. Data Synchronization: The binocular camera is used to simultaneously acquire data from the high-speed camera and the event camera, and a synchronization trigger is used to ensure that the data acquired by the event camera and the high-speed camera in the binocular camera are synchronized; S3. Camera Calibration: Design the calibration of event cameras and high-speed cameras using an electronic calibration board, and export the calibration results; S4. Data Collection: Determine the scene and conditions for collecting the data, and collect data according to different lighting, shooting distance and shooting angle to ensure the diversity of the collected data; S5. Data Alignment and Registration: The acquired data from the high-speed camera and the event camera are processed, and the calibration results obtained in step S3 are used to correct and register the event camera data and the high-speed camera data. S6. Data Reconstruction: Reconstruct the event camera data so that the reconstructed event camera data and high-speed camera data have the same frame rate, ensuring that the event camera data and high-speed camera data are consistent at the same time. S7. Model Design and Training: Construct and train a referenced evaluation network model and an unreferenced evaluation network model; S8. Model Deployment: Deploy the referenced evaluation network model and the unreferenced evaluation network model trained in step S7 on the server; S9. Data Evaluation: The trained referenced evaluation network model and unreferenced evaluation network model from step S7 are used to evaluate the data quality of the event camera, output the data quality score of the event camera, and use it to specify the scheme for downstream tasks.

2. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S1, the data acquisition design includes: using a binocular camera consisting of an event camera and a high-speed camera to acquire data. The event camera and the high-speed camera are respectively set at both ends of a fixed axis and connected to a synchronization trigger and a high-speed data memory respectively. The synchronization trigger uses a clock pulse to simultaneously trigger the event camera and the high-speed camera to align their timelines. The high-speed data memory is used to store the data collected by the high-speed camera, and the fiber optic transmission device is used to transmit the high-speed camera data.

3. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S3, the calibration results include the intrinsic parameter matrix, distortion parameter, rotation matrix, and translation matrix of the event camera and the high-speed camera.

4. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S6, the event camera data reconstruction method is to convert the event camera data that occurs at the same time as the output frame of the high-speed camera into a grayscale image.

5. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S7, the model design includes constructing a referenced evaluation network model and a non-referenced evaluation network model. The referenced evaluation network model is used to compare event camera data and high-speed camera data to generate evaluation values ​​of event data quality, which are used as annotation information. The non-referenced evaluation network model is trained on the event data after obtaining the annotation information.

6. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S7, the referenced evaluation network model is first trained. The training requires the use of both the event camera data and the high-speed camera data. After training, an evaluation value for the event camera data is generated. The evaluation value is then used as a label value to train the unreferenced evaluation network model.

7. The method for evaluating the quality of event-based camera data according to claim 1, characterized in that, In step S9, when the data acquired by the event camera and the high-speed camera are used, a referenced evaluation network model is used for evaluation. If the acquired data only contains event camera data, a non-referenced evaluation network model is used for evaluation.

8. A system for evaluating the quality of event-based camera data, used to implement the method for evaluating the quality of event-based camera data according to any one of claims 1 to 7, characterized in that, include: Binocular camera, synchronization trigger, high-speed data storage and fiber optic transmission equipment, among which, The binocular camera consists of an event camera and a high-speed camera, used for data acquisition. The event camera and the high-speed camera are respectively positioned at opposite ends of a fixed axis and connected to the synchronization trigger and the high-speed data memory, respectively. The synchronization trigger uses clock pulses to simultaneously trigger the event camera and the high-speed camera, aligning their timelines. The high-speed data storage device is used to store the data collected by the high-speed camera. The fiber optic transmission equipment is used to transmit high-speed camera data.

Images