A first-view data acquisition method and system based on binocular event camera and depth camera

Abstract

The application discloses a kind of based on binocular event camera and depth camera first view angle data acquisition method and system, the system includes two Prophesee evk4 event camera, a Realsense D435 depth camera, 3D printing support, helmet, voltage selection chip, direct current source and connecting line. All components are fixed on the helmet by 3D printing support, and the voltage selection chip and the direct current source provide stable voltage for the camera synchronously. The depth camera can synchronously output two grayscale images, an RGB image and a depth image. Calibration is required before use to ensure the accuracy of data acquisition. After calibration, the labeled information on the depth camera can be automatically projected onto the event camera, solving the problem of difficult event data labeling and obtaining 3D position information and color information corresponding to event points. The application realizes efficient and stable first view angle data acquisition and is suitable for various computer vision applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical structure and computer, and particularly relates to a first-view data acquisition method and system based on binocular event camera and depth camera. BACKGROUND

[0002] In the field of computer vision and robotics, first-view data acquisition is of great significance for research and development. First-view, that is, data acquisition from the user's perspective, can provide more realistic and intuitive visual information, and is widely used in robot navigation, augmented reality, virtual reality and other fields. However, there are few existing first-view data acquisition devices, which has not fully released the potential of this field. Traditional camera systems perform poorly in dynamic scenes and low lighting conditions, while event cameras can provide better data in these conditions due to their high temporal resolution and high dynamic range. However, existing systems often lack integration of event cameras and depth cameras, making it difficult to achieve efficient data acquisition.

[0003] In addition, the labeling of event data is very difficult because event cameras capture pixel-level changes rather than traditional frame images. To solve this problem, we introduce a labeling method that can automatically project the labeling information on the grayscale image output by the depth camera onto the event data, thereby obtaining the 3D position information and color information corresponding to the event points. This method greatly simplifies the labeling process of event data and improves the efficiency and accuracy of data labeling. SUMMARY

[0004] The present application aims to provide a first-view data acquisition method and system based on binocular event camera and depth camera to solve the problems in the prior art.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] The present application first provides a first-view data acquisition method based on binocular event camera and depth camera, comprising the following steps:

[0007] 1) Assemble a first-view device for simulating user's first-view shooting, the first-view device comprising a binocular event camera and a depth camera;

[0008] 2) Calibrate the binocular event camera and the depth camera in the first-view device so that the event points captured by the binocular event camera and the same object in the picture captured by the depth camera can correspond;

[0009] 3) Start the first-view device to shoot, the depth camera sends a synchronization signal, and the synchronization signal is processed by step-up and step-down before being input into the binocular event camera as a trigger signal;

[0010] 4) export the picture taken by the depth camera and the event point taken by the binocular event camera, and match the timestamp of the time when the binocular event camera receives the trigger signal with the timestamp of the time when the depth camera sends the synchronization signal, so as to realize the synchronization of the picture taken by the depth camera and the event point taken by the binocular event camera;

[0011] 5) label the target object in the picture taken by the depth camera, and project the labeled picture on the event point taken by the binocular event camera, so as to label the event point taken by the binocular event camera synchronously;

[0012] 6) output the labeled event point taken by the binocular event camera synchronously, obtain the three-dimensional position information and color information corresponding to the event point, and realize the collection of data.

[0013] As a preferred scheme of the present application, the first perspective device comprises:

[0014] a binocular event camera, used for capturing visual data with high time resolution and high dynamic range;

[0015] a depth camera, used for capturing depth images, grayscale images and RGB images of a scene, and capable of sending a synchronization signal;

[0016] a helmet, used for being worn on the head of a user, and providing a first perspective of data collection;

[0017] a support, used for fixing the binocular event camera and the depth camera on the helmet, and ensuring the stability and consistency of the perspective of the cameras.

[0018] a voltage selection chip, used for performing step-up and step-down processing on the synchronization signal sent by the depth camera and outputting a trigger signal.

[0019] As a preferred scheme of the present application, in the step 5), the target object in the picture taken by the depth camera is recognized by a depth camera labeling tool and labeled in the picture taken by the depth camera. Further, the labeling tool adopts Segment-Anything, MediaPipe-master, OpenPose, etc.

[0020] The present application further provides a system for realizing the above-mentioned first perspective data collection method, comprising:

[0021] a first perspective device, comprising a binocular event camera, a depth camera and a voltage selection chip, the binocular event camera and the depth camera being used for simulating a first perspective of a user and taking a scene within the perspective, and the voltage selection chip being used for performing step-up and step-down processing on a synchronization signal sent by the depth camera;

[0022] A camera calibration module is configured to calibrate parameters of the binocular event camera and the depth camera.

[0023] A camera signal processing module is configured to control the depth camera to send a synchronization signal; the synchronization signal is input into the binocular event camera as a trigger signal after being processed by a voltage selection chip of the first-view device;

[0024] A camera synchronization module is configured to record a timestamp of the binocular event camera receiving the trigger signal, and match the timestamp with a timestamp of the depth camera sending the synchronization signal, so as to realize synchronization between the two cameras.

[0025] A picture labeling module is configured to label pictures taken by the depth camera, and project the labeled pictures onto the binocular event camera, so as to label pictures taken by the binocular event camera synchronously.

[0026] Compared with the prior art, the present application has the following beneficial effects:

[0027] The present application integrates the binocular event camera and the depth camera, realizes high-time-resolution and dynamic-range visual data acquisition, simplifies the labeling process, improves data synchronization accuracy, improves data practicability, enhances multi-scene application potential, and promotes the development of computer vision and robot technology.

[0028] The present application uses a low-cost NetFPGA hardware device to realize flexible, fair and controllable allocation of network resources, reduces the implementation cost of network resource allocation, and has simple implementation method and flexible means.

[0029] The present application can greatly improve the data consistency and accuracy in multi-camera shooting configuration, and provides strong technical support for advanced visual analysis and processing. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The present application is based on a binocular event camera and a depth camera.

[0031] Figure 2 The present application is based on a binocular event camera and a depth camera.

[0032] Figure 3 The present application is based on a binocular event camera and a depth camera. DETAILED DESCRIPTION

[0033] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0034] like Figure 1 As shown, this invention provides a first-view data acquisition method based on a binocular event camera and a depth camera, comprising the following steps:

[0035] 1) Equip with a first-person perspective device for simulating the user's first-person view during shooting; such as Figure 2 and Figure 3 As shown, in a specific embodiment of the present invention, the first-view device includes two event cameras (left and right), a depth camera, a helmet, a 3D-printed bracket, and a voltage selection chip, used to capture visual data with high temporal resolution and high dynamic range. The depth camera is used to capture depth information of the scene and can simultaneously output two grayscale images, one RGB image, and one depth image. The helmet is worn on the user's head to provide first-view data acquisition. The 3D-printed bracket is used to fix the two event cameras and the depth camera to the helmet, ensuring camera stability and consistency of viewpoint. The voltage selection chip is used to perform step-up / step-down processing on the synchronization signal emitted by the depth camera and output a trigger signal. All components are connected by wires to ensure data and power transmission.

[0036] 2) The binocular event camera and depth camera in the first-view device are calibrated using the Zhang Zhengyou calibration method so that the event points captured by the binocular event camera and the same object in the image captured by the depth camera can correspond.

[0037] 3) Activate the first-view device. The depth camera emits a synchronization signal in square wave form, with a frame rate range of 6Hz to 300Hz. The emitted synchronization signal is processed by a voltage selection chip on the first-view device. When the amplitude of the synchronization signal is less than a preset threshold, the voltage selection chip does not output a trigger signal. When the input signal exceeds the preset threshold, the voltage selection chip outputs a trigger signal with the same amplitude as the voltage selection chip's power supply voltage and the same frequency and duty cycle as the synchronization signal, which is then input to the binocular event camera.

[0038] 4) Export the images captured by the depth camera and the event points captured by the stereo event camera. Match the timestamp of the trigger signal received by the stereo event camera with the timestamp of the synchronization signal emitted by the depth camera to achieve synchronization between the images captured by the depth camera and the event points captured by the stereo event camera.

[0039] 5) Label the target object in the picture taken by the depth camera, set the time interval T for the depth camera to take two pictures in succession, the time point t for the depth camera to take pictures, segment the time sequence of the event points taken by the binocular event camera in the time period [t-T / 2, t+T / 2], and match the event points in each time period with the pictures taken by the depth camera in the same time period; and project the labeled pictures taken by the depth camera after matching onto the event points taken by the binocular event camera, thereby labeling the event points taken synchronously by the binocular event camera; in a specific embodiment of the present application, image labeling tools such as Segment-Anything, MediaPipe-master, OpenPose, etc. are used to label the target object in the picture taken by the depth camera.

[0040] 6) Output the labeled event points taken synchronously by the binocular event camera, obtain the three-dimensional position information and color information corresponding to the event points, and realize data collection.

[0041] The present application also provides a first-view data collection system based on a binocular event camera and a depth camera, comprising:

[0042] A first-view device comprising a binocular event camera, a depth camera and a voltage selection chip, the binocular event camera and the depth camera being used to simulate a user's first view and take pictures of the scene within the view, and the voltage selection chip being used to perform step-up and step-down processing on the synchronization signal sent by the depth camera;

[0043] A camera calibration module for calibrating the parameters of the binocular event camera and the depth camera;

[0044] A camera signal processing module for controlling the depth camera to send a synchronization signal; the synchronization signal is input into the binocular event camera as a trigger signal after being processed by the voltage selection chip of the first-view device;

[0045] A camera synchronization module for recording the timestamp of the trigger signal received by the binocular event camera, and matching it with the timestamp of the synchronization signal sent by the depth camera, thereby realizing the synchronization between the two cameras;

[0046] A picture labeling module for labeling the pictures taken by the depth camera, and projecting the labeled pictures onto the binocular event camera, thereby labeling the pictures taken synchronously by the binocular event camera.

[0047] The system of the present application can realize the synchronization between cameras, ensuring that the synchronization error of the collected data is within 0.5 ms. The synchronization process is that the depth camera sends a synchronization signal, which is transmitted to the event camera after passing through a voltage boosting chip, so as to obtain the timestamp of the event camera when the depth camera is imaging. Before use, the system needs to be calibrated to ensure the accuracy of data collection. The calibration process includes adjusting the position and angle of the camera, and calibrating the intrinsic and extrinsic parameters of the camera to ensure the accuracy of the collected data.

[0048] After labeling, the grayscale image labeling information on the depth camera can be automatically projected onto the event camera, solving the problem of difficult labeling of event data, and obtaining 3D position information and color information corresponding to the event points.

[0049] In one embodiment of the present application, the depth camera controlled by the computer is of the Realsense D435 model, and the two event cameras are of the Prophesee EVK4 model. The system of the present application uses the computer to send a synchronization signal, which is sent in the form of a square wave with a frequency range from 6 Hz to 300 Hz and an amplitude of about 2 V by precisely controlling the frame rate of the signal. The synchronization signal is first processed by a voltage selection chip powered by a direct current source. The function of the voltage selection chip is to output 0 V when the input signal is less than the preset threshold, and to output a signal with an amplitude of the direct current source voltage, a frequency and a duty cycle same as the input signal when the input signal exceeds the threshold. Subsequently, the output signal is sent to the Prophesee EVK4 camera as a trigger signal. By recording the timestamp of the trigger signal received by the EVK4 camera and matching it with the timestamp of the signal sent by the Realsense camera, the synchronization between the two cameras is realized. The preset threshold in the voltage selection chip is 1 V, and the direct current source voltage is set to 5 V.

[0050] The system of the present application realizes an efficient and accurate camera synchronization mechanism by precisely sending and adjusting the computer control signal, combined with the intelligent threshold judgment and signal conversion of the voltage boosting chip. In addition, the system of the present application supports extensive frame rate adjustment capability, making it adaptable to different shooting requirements and environmental conditions. Through the system of the present application, the data consistency and accuracy in multi-camera shooting configuration can be greatly improved, providing strong technical support for advanced visual analysis and processing. By integrating binocular event cameras and depth cameras, efficient and stable first-view data collection is realized, which is suitable for various computer vision applications such as robot navigation, augmented reality and virtual reality, etc.

[0051] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation to the patent scope of the present application. For ordinary skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which shall all fall into the protection scope of the present application.

Claims

1. A first view data acquisition method based on binocular event camera and depth camera, characterized in that, The method comprises the following steps: 1) assembling a first perspective device for shooting from the first perspective of a user, the first perspective device comprising: a binocular event camera for capturing visual data with high temporal resolution and high dynamic range; a depth camera for capturing depth images, grayscale images and RGB images of a scene, and capable of emitting a synchronization signal; a helmet for being worn on the head of the user, providing the first perspective of data acquisition; a support for fixing the binocular event camera and the depth camera on the helmet, ensuring the stability of the cameras and the consistency of the perspective; a voltage selection chip for step-up and step-down processing of the synchronization signal emitted by the depth camera and outputting a trigger signal; 2) calibrating the binocular event camera and the depth camera in the first perspective device, so that the same object in the event points shot by the binocular event camera and the pictures shot by the depth camera can correspond; 3) starting the first perspective device to shoot, the depth camera emitting a synchronization signal, and the emitted synchronization signal being step-up and step-down processed and then input into the binocular event camera as a trigger signal; 4) exporting the pictures shot by the depth camera and the event points shot by the binocular event camera, and matching the timestamp of the trigger signal received by the binocular event camera with the timestamp of the synchronization signal emitted by the depth camera, so as to realize the synchronization of the pictures shot by the depth camera and the event points shot by the binocular event camera; 5) labeling the target object in the pictures shot by the depth camera, and projecting the labeled pictures onto the event points shot by the binocular event camera, so as to label the event points shot by the binocular event camera synchronously; 6) outputting the labeled event points shot by the binocular event camera synchronously, obtaining the three-dimensional position information and color information corresponding to the event points, and realizing the acquisition of data.

2. The first view data acquisition method of claim 1, wherein, In the step 2), the binocular event camera and the depth camera are calibrated by Zhang Zhengyou's calibration method.

3. The first perspective data acquisition method of claim 1, wherein, In the step 3), the depth camera emits a synchronization signal in the form of a square wave, and the frame rate of the synchronization signal ranges from 6 Hz to 300 Hz.

4. The first perspective data collection method of claim 1, wherein, In the step 4), when the synchronization signal emitted by the depth camera is step-up and step-down processed, the processing is performed by the voltage selection chip on the first perspective device; when the amplitude of the synchronization signal is less than a preset threshold, the voltage selection chip does not output a trigger signal; When the input signal exceeds the preset threshold, the voltage selection chip outputs a trigger signal with the same amplitude as the supply voltage of the voltage selection chip, and the same frequency and duty cycle as the synchronization signal.

5. The first perspective data collection method of claim 1, wherein, In the step 4), matching the timestamp of the trigger signal received by the binocular event camera with the timestamp of the synchronization signal emitted by the depth camera specifically comprises: setting the time interval for the depth camera to continuously shoot two pictures as T, setting the time point for the depth camera to shoot a picture as t, segmenting the time sequence of the event points shot by the binocular event camera in the time period [t-T / 2, t+T / 2], and matching the event points in each time period with the pictures shot by the depth camera in the same time period.

6. The first perspective data collection method of claim 1, wherein, In the step 5), the target object in the pictures shot by the depth camera is identified by a depth camera labeling tool and labeled in the pictures shot by the depth camera.

7. A system for implementing the first view data acquisition method of claim 1, characterized by, The method comprises: The first perspective device comprises a binocular event camera, a depth camera and a voltage selection chip, the binocular event camera and the depth camera are used for simulating a first perspective of a user and taking a scene in the perspective, and the voltage selection chip is used for performing step-up and step-down processing on a synchronization signal sent by the depth camera; The camera calibration module is used for calibrating parameters of the binocular event camera and the depth camera; The camera signal processing module is used for controlling the depth camera to send a synchronization signal; the synchronization signal is input into the binocular event camera as a trigger signal after being processed by the voltage selection chip of the first perspective device; The camera synchronization module is used for recording a timestamp of the trigger signal received by the binocular event camera, and matching the timestamp with a timestamp of the synchronization signal sent by the depth camera, so as to realize synchronization between the two cameras; The picture labeling module is used for labeling pictures taken by the depth camera, and projecting the labeled pictures onto event points taken by the binocular event camera, so as to label the event points taken by the binocular event camera synchronously.

Images