Event camera calibration device and method based on LED adjustable light source and LED frame synchronization

Abstract

The invention discloses an event camera calibration device and method based on an LED adjustable light source and LED frame synchronization. A light source module is formed by a plurality of small LED lamp beads in a circular arrangement mode from inside to outside. The light source module is fixed at the black and white intersection points in the Zhang Zhengyou chessboard through the specially-made light source module mounting bracket. An LED dot matrix module is adopted as an event camera frame synchronization device. A ROS (Root Operating System) is utilized to program a control node so as to control the single chip microcomputer and the Internet of Things module, and then the turn number of on-off turns of LED lamp beads in the light source module, the flicker frequency, the brightness of the LED lamp beads and the counting frequency of the LED dot matrix module are controlled. The event camera calibration method and system can adapt to calibration of various environments, various types of checkerboards and various types of event cameras, the calibration points are extracted through an algorithm, the accuracy of event camera calibration is effectively improved, and the applicability of various checkerboards is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to an event camera calibration device and method based on an LED adjustable light source and LED frame synchronization. BACKGROUND

[0002] An event camera is a relatively novel camera, which is significantly different from a traditional frame camera. Its working principle is based on triggering events by pixel-level brightness changes, rather than continuously capturing images. This design makes the event camera perform well in low light, high dynamic range and high-speed motion environments. The event camera can capture motion information in the scene in real time and accurately through pixel-level brightness changes, which is particularly suitable for high-speed motion or rapidly changing scenes. In low light and high dynamic range conditions, the event camera can still provide reliable information. Since the event camera only triggers acquisition when an event occurs, compared with the frame camera which continuously captures images, the event camera greatly reduces the amount of data generated and transmitted, thereby saving energy and improving overall efficiency. The event camera is widely used in many fields. In the field of robots, it can be used to perceive the surrounding environment and obtain motion information in real time, providing important data for robot navigation and obstacle avoidance. In autonomous vehicles, the event camera can provide real-time motion information on highways, helping vehicles drive safely and react quickly. In monitoring systems, it can efficiently detect abnormal behavior, target tracking, and reduce unnecessary data storage and transmission, improving the overall performance of the monitoring system.

[0003] It is necessary to accurately calibrate the camera when it is applied, especially in the field of computer vision. Camera calibration is a necessary step for tasks such as target tracking, object recognition, and three-dimensional reconstruction. Accurate camera parameters can improve the performance and accuracy of computer vision algorithms. In the fields of autonomous driving and robotics, multiple sensors are often deployed and calibrated to the same coordinate system through joint calibration to enhance the ability to perceive the environment. Current calibration of event cameras often uses a screen to display a checkerboard image, and adjusts the brightness of the screen or moves the screen as an event trigger event camera. However, the black and white intersection points of the checkerboard in the captured image are often not clear enough, and it is difficult to achieve precise frame synchronization for multiple groups of event cameras of different brands, reducing the accuracy of event camera calibration. SUMMARY

[0004] To solve the above problems, the present application uses a light source module with variable light source range, a single-chip microcomputer, an Internet of Things module, and a Zhang Zhengyou checkerboard to form a calibration board. The calibration board control node programmed with ROS controls the size of the light source range and the flashing frequency of the light beads as the trigger condition of the event camera to realize the calibration of the event camera. The LED dot matrix frame synchronization module is used for frame synchronization of multiple groups of event cameras.

[0005] The application provides an event camera calibration device and method based on an LED adjustable light source and LED frame synchronization, and specifically comprises the following steps:

[0006] Step 1: Design a light source module based on a variable light source range of LED lamp beads: the light source module is composed of 38 small LED lamp beads, a light source module lamp cover and a bottom circuit board. The 38 LED lamp beads are arranged in a 4-layer annular arrangement, wherein the first layer is composed of 1 LED lamp bead, the second layer is composed of 6 LED lamp beads, the third layer is composed of 12 LED lamp beads, and the fourth layer is composed of 19 LED lamp beads. Each layer of lamp beads is connected in series with each other and provides a positive and negative electrode interface to independently control each layer of lamp beads through the circuit board. The bottom circuit board is composed of a microcontroller and an Internet of Things module, and has a longitudinal groove at the upper and lower ends for positioning the installation position of the light source module. When installed, the upper and lower grooves need to be aligned with the intersection lines of the checkerboard grid. A button cell is installed on the back of the circuit board as the power supply of the light source module.

[0007] Step 2: Design an adjustable light source module fixing device: the light source module fixing device is made of transparent acrylic material, and the front surface is a horizontal clamping groove for installing the light source module. The installation interval and position of the light source module can be adjusted according to the size of the checkerboard grid of different calibration boards. The back surface of the light source module fixing device has a longitudinal clamping groove at each end, and the light source module fixing device can be installed on the calibration board through the longitudinal clamping grooves at both ends. The specific installation position is determined by the size of the checkerboard grid of different calibration boards. The front surface of the light source module fixing device has a horizontal groove for positioning the installation position of the device. When installed, the groove needs to be aligned with the intersection lines of the checkerboard grid.

[0008] Step 3: Frame synchronization device based on LED dot matrix: the frame synchronization device is composed of an LED dot matrix, a microcontroller and an Internet of Things module. When calibrating multiple event cameras, the frames used for calibration need to be synchronized. The LED dot matrix has two clamping groove assemblies installed at the top end and the left side for fixing the device. The device is fixed to the upper left corner of the calibration board. When the light source module in step 1 flashes, the LED dot matrix displays a number synchronously. The number starts from 1 and the counting frequency depends on the flashing frequency of the light source module in step 1. When the event camera completes an event acquisition process, such as the process of turning off, flashing and turning off again of an LED light source module, the LED dot matrix frame synchronization module number is incremented by 1 after the process is completed.

[0009] Step 4: ROS-based calibration board control node: Event cameras are often used in ROS-based autonomous driving environment perception. The light source module described in step 1 and the LED dot matrix frame synchronization device described in step 3 are controlled in a ROS programming manner. The control process mainly sends instructions from the host computer to the Internet of Things module built into the light source module bottom circuit board and the LED dot matrix frame synchronization device, respectively. The Internet of Things module communicates with the microcontroller through a serial port, and sends control instructions to the microcontroller through the Internet of Things module to control the flashing frequency of the light source module, the light source size of the light source module, and the digital display interval of the LED dot matrix frame synchronization device.

[0010] Step 5: Event camera data acquisition: Multiple event cameras are fixed at one end, and the calibration board is placed at different angles and distances in a manual manner. The calibration board control node described in step 3 adjusts the flashing frequency of the light source, the size of the light source, and the digital update interval of the LED dot matrix frame synchronization module according to different event cameras and chessboard sizes. Larger calibration boards with larger chessboard sizes use larger light source ranges, and smaller calibration boards with smaller chessboard sizes use smaller light source ranges to improve the quality of event camera frames and thus improve the accuracy of calibration.

[0011] Step 6: Event camera calibration: The event camera ROS data packet obtained in step 5 acquires the original data frame, and according to the preset time interval, the events collected within a certain time range are superimposed into the image to generate an event image for calibration. Then, an edge detection algorithm is used to detect the event image, and the numbers displayed by the LED dot matrix in the calibration board and the edge features of the light source module can be obtained:

[0012]

[0013] where G(x, y) is the gray value of the pixel coordinates of the binary image after Gaussian filtering, and I(x, y) is the gray value of the pixel coordinates of the input binary image. By performing Gaussian filtering on the input binary image, the edge detection accuracy is improved.

[0014] I x (x, y) = G(x, y) * (I(x + 1, y) - I(x - 1, y))

[0015] I y (x, y) = G(x, y) * (I(x, y + 1) - I(x, y - 1))

[0016]

[0017] where I x (x, y), I y(x, y) respectively represent the gradient of the image in the horizontal direction and the vertical direction, G(x, y) represents the gradient amplitude of the image at the pixel coordinate (x, y), and θ(x, y) represents the gradient direction of the image at the pixel coordinate (x, y).

[0018]

[0019] In the formula, M(x, y) is the image after non-maximum suppression, and the non-boundary region features are weakened by non-maximum suppression to obtain more accurate edge features.

[0020] The edge features are sorted according to the position of the coordinate system in the two-dimensional image where the extracted features are located. Since the LED dot matrix module is located at the top left corner of the image, the edge feature with the order of 1 is the number displayed by the LED:

[0021]

[0022] argsort(R iy )i∈(1,n),y∈(1,n)

[0023] In the formula, R iy represents the average value of the pixel coordinates in the region numbered i in the edge detection in the y-axis, i represents the number of regions, and j represents the number of pixel points in the region i. argsort(R iy ) is in ascending order, and after the ascending order of R iy , the region with the order of 1 is the topmost edge region in the binary image, representing the number feature.

[0024] The obtained number edge feature is matched with the locally stored number template to obtain the actual number:

[0025]

[0026] In the formula, T(x, y) represents the pixel point coordinate in the template image, and I(x, y) represents the image pixel point coordinate containing the number edge feature. Since it is a binary image, the gray value of the non-edge region is 0, and the value is larger, which represents a higher matching degree, and the matching score Score is obtained after normalization processing. The similarity between images is determined by the matching score.

[0027] According to the actual number in the multi-event camera frame, the events are synchronized. At the same time, the median filter is used for noise reduction processing of the frame, and then RANSAC is used for iterative estimation of other edge features to obtain the center coordinates of the LED light source module. Through the frame synchronization processing of the LED dot matrix module and the extraction of the center of the LED light source module, the intrinsic parameters of the event camera and the relative position relationship of multiple groups of event cameras are calculated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a schematic diagram of an event camera calibration device based on LED adjustable light source and LED frame synchronization.

[0029] Figure 2 is a schematic diagram of a light source module fixing device.

[0030] Figure 3 is a schematic diagram of a light source module structure.

[0031] Figure 4 is a schematic diagram of an LED dot matrix module.

[0032] Figure 5 is a whole flow chart of the patent. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0034] As shown in Figures 1-4 , the present application provides an event camera calibration device and method based on LED adjustable light source and LED frame synchronization, which specifically includes the following steps:

[0035] Step 1: Design a light source module with variable light source range based on LED lamp beads: as shown in Figure 4 , the light source module is composed of 38 small LED lamp beads, a light source module lamp cover and a bottom circuit board. The 38 LED lamp beads form a light source module through 4-layer annular arrangement. The first layer is composed of 1 LED lamp bead, the second layer is composed of 6 lamp beads, the third layer is composed of 12 lamp beads, and the fourth layer is composed of 19 lamp beads. Each layer of lamp beads is connected in series with each other and provides a positive and negative terminal interface to independently control each layer of lamp beads through the bottom circuit board. The bottom circuit board is composed of a microcontroller and an Internet of Things module, and has a longitudinal groove at the top and bottom ends for positioning the installation position of the light source module. When installing, it is necessary to ensure that the upper and lower grooves coincide with the intersection lines of the checkerboard grid. A button cell is installed on the back of the circuit board as the power supply of the light source module.

[0036] Step 2: Design an adjustable light source module fixing device: as shown in Figure 2As shown, the light source module fixing device is made of transparent acrylic material, and its front surface is a horizontal clamping groove for installing the light source module. The installation interval and position of the light source module can be adjusted according to the size of the chessboard grid of different calibration boards. The back surface of the light source module fixing device has a longitudinal clamping groove at each end, and the light source module fixing device can be installed on the calibration board through the longitudinal clamping grooves at both ends. The specific installation position is determined by the size of the chessboard grid of different calibration boards. The front surface of the light source module fixing device has a horizontal groove for positioning the installation position of the device, and the groove needs to be aligned with the intersection line of the chessboard grid during installation.

[0037] Step 3: LED dot matrix-based frame synchronization device: as shown in Figure 4 The frame synchronization device is composed of an LED dot matrix, a microcontroller, and an Internet of Things module. During multi-event camera calibration, the frames used for calibration need to be synchronized. The LED dot matrix has two clamping groove assemblies installed at the top and left side for fixing the device, and the device is fixed to the upper left corner of the calibration board. As shown in Figure 1 When the light source module flashes, the LED dot matrix displays the number synchronously, and the number starts from 1 and increases by 1 each time the light source module flashes.

[0038] Step 4: ROS-based calibration board control node: control the light source module and LED dot matrix frame synchronization device through ROS programming. The control process mainly sends instructions from the host computer to the Internet of Things modules built into the light source module bottom circuit board and LED dot matrix frame synchronization device respectively. The Internet of Things module communicates with the microcontroller through a serial port, and sends control instructions to the microcontroller through the Internet of Things module to control the flashing frequency of the light source module, the size of the light source module, and the digital display interval of the LED dot matrix frame synchronization device.

[0039] Step 5: Event camera data acquisition: multiple event cameras are fixed at one end, and the calibration board is placed at different angles and distances in a manual manner. The calibration board control node adjusts the flashing frequency of the light source, the size of the light source, and the digital update interval of the LED dot matrix frame synchronization module according to different event cameras and chessboard grid sizes. Larger calibration boards with larger chessboard grid sizes use larger light sources, and smaller calibration boards with smaller chessboard grid sizes use smaller light sources to improve the quality of the event camera frame and thus improve the accuracy of the calibration.

[0040] Step 6: Event camera calibration: obtain the original data frame from the event camera ROS data packet obtained in step 5, superimpose the events collected within a certain time range into the image according to the preset time interval, generate an event image for calibration, and then use an edge detection algorithm to detect the event image. The numbers displayed by the LED dot matrix and the edge features of the light source module in the calibration board can be obtained:

[0041]

[0042] In the formula, G(x, y) is the gray value of the pixel coordinate of the binary image after Gaussian filtering, and I(x, y) is the gray value of the pixel coordinate of the input binary image. The input binary image is filtered by Gaussian filtering to improve the accuracy of edge detection.

[0043] I x (x, y) = G(x, y) * (I(x + 1, y) - I(x - 1, y))

[0044] I y (x, y) = G(x, y) * (I(x, y + 1) - I(x, y - 1))

[0045]

[0046] In the formula, I x (x, y) and I y (x, y) represent the gradients of the image in the horizontal and vertical directions, respectively, G(x, y) represents the gradient amplitude of the image at pixel coordinate (x, y), and θ(x, y) represents the gradient direction of the image at pixel coordinate (x, y).

[0047]

[0048] In the formula, M(x, y) is the image after non-maximum suppression, and the non-maximum suppression weakens the non-boundary region features to obtain more accurate edge features.

[0049] According to the position of the coordinate system in the two-dimensional image where the extracted features are located, the edge features are sorted. Since the LED dot matrix module is located at the top left corner of the image, the edge feature with the order of 1 is the number displayed by the LED:

[0050]

[0051] argsort(R iy )i∈(1,n),y∈(1,n)

[0052] In the formula, R iy represents the average value of the pixel coordinates in the region numbered i in the edge detection in the y-axis, i represents the number of regions, and j represents the number of pixel points in region i. argsort(R iy ) is in ascending order. After sorting R iy in ascending order, the region with the order of 1 is the topmost edge region in the binary image, representing the number feature.

[0053] The obtained number edge feature is matched with the locally stored number template to obtain the actual number:

[0054]

[0055] In the formula, T(x, y) represents the pixel point coordinates in the template image, and I(x, y) represents the image pixel point coordinates containing the digital edge features. Since it is a binary image, the gray value of the non-edge region is 0. The greater the value, the higher the matching degree, and the matching score Score is obtained through normalization processing. The similarity between images is determined by the matching score.

[0056] Synchronization of events is performed according to the actual digital events in the multi-event camera frame. At the same time, median filtering is used to reduce noise in the frame, and then RANSAC is used to iteratively estimate other edge features to obtain the LED light source module center coordinates. The intrinsic parameters of the event camera and the relative position relationship of multiple event cameras are calculated after the frame synchronization processing of the LED dot matrix module and the extraction of the LED light source module center.

[0057] Although the present application has been disclosed in detail with reference to the accompanying drawings, it is understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims, and can include various modifications, improvements and equivalent solutions made to the application without departing from the scope and spirit of the present application.

Claims

1. An apparatus and method for calibrating an event camera based on LED adjustable light source and LED frame synchronization, characterized in that, It comprises the following steps: Step 1, design the light source module based on the variable light source range of LED lamp beads: the light source module comprises 38 small LED lamp beads (1), a light source module lamp cover (2), and a bottom circuit board (3). The small LED lamp beads (1) are arranged in four layers in a ring shape, each layer of lamp beads is connected in series with each other, and a positive and negative electrode interface is provided to independently control each layer of lamp beads through the bottom circuit board (3). The light source module lamp cover (2) covers the 38 LED lamp beads. The bottom circuit board (3) comprises a microcontroller (4) and an Internet of Things module (5). The upper and lower ends of the bottom circuit board each have a longitudinal groove for positioning the installation position of the light source module. When installed, the upper and lower grooves need to be aligned with the intersection lines of the chessboard grid. A button cell is installed on the back of the bottom circuit board as the power supply of the light source module; Step 2, adjustable light source module fixing device: the light source module fixing device is made of transparent acrylic material. The front surface is a horizontal clamping groove for installing the light source module. The installation interval and position of the light source module can be adjusted according to the size of the chessboard grid of different calibration boards. The back surface of the light source module fixing device has a longitudinal clamping groove at each end, and the light source module fixing device can be installed on the calibration board through the longitudinal clamping grooves at both ends. The front surface of the light source module fixing device has a horizontal groove for positioning the installation position of the device. When installed, the groove needs to be aligned with the intersection lines of the chessboard grid; Step 3, frame synchronization device based on LED dot matrix: the frame synchronization device comprises a microcontroller (4), an Internet of Things module (5), and an LED dot matrix (6) for frame synchronization of multi-event camera calibration; The LED dot matrix (6) has two clamping groove assemblies installed at the top and left side for fixing the device. The device is fixed to the upper left corner of the calibration board. When the light source module flashes as described in step 1, the LED dot matrix (6) displays a number synchronously. The number starts from 1 and is counted. The counting frequency depends on the flashing frequency of the light source module described in step 1. When the event camera completes an event acquisition process, such as the process of turning off, flashing, and turning off again of a LED light source module, the LED dot matrix frame synchronization module number is incremented by 1 after a process is completed. Step 4, calibration board control node based on ROS: the light source module described in S1 and the LED dot matrix frame synchronization device described in S3 are controlled by using ROS programming. The control process mainly sends ROS instructions from the upper computer to the light source module bottom circuit board (3) and the LED dot matrix frame synchronization device. The microcontroller (4) and the Internet of Things module (5) are used for serial communication to adjust the flashing frequency, light source size, and LED dot matrix frame synchronization device number update frequency of the light source module. Step 5, event camera data acquisition: a plurality of event cameras are fixed at one end, and the placement of the calibration board at different angles and distances is adjusted manually, the calibration board control node dynamically adjusts the light source module parameters, including the flashing frequency of the light source, the size of the light source and the digital update interval of the LED dot matrix frame synchronization module; in order to improve the quality of the event camera frame and the accuracy of the calibration, according to the size of the chessboard of different calibration boards, the method flexibly selects a larger light source range for a larger calibration board during the acquisition process, and a smaller light source range is used for a smaller calibration board, thereby optimizing the calibration conditions and improving the accuracy and efficiency of data acquisition; Step 6, event camera calibration: the event camera ROS data packet obtained in step 5 is the original event data generated in the field of view during the recording process, the obtained event data is divided according to the digital update interval of the frame synchronization device set in step 3, the events in the same time period are superimposed on the image according to their pixel positions, the original event image is obtained, the original event image is processed by applying Gaussian filtering to obtain a high-precision binary event image; the binary event image after Gaussian filtering is processed by using an edge detection algorithm to extract the digital displayed by the LED dot matrix on the calibration board and the edge features of the light source module; The extracted features are subjected to non-maximum suppression to obtain more accurate edge feature images; the coordinates of the edge features in the binary event image are sorted, and the edge features with a sort of 1 are the numbers displayed by the LED; the obtained digital edge features are matched with the locally stored digital templates, the matching scores are obtained through normalization processing, the similarity between the images is judged, and the actual numbers are obtained; the actual numbers in the frames of the plurality of event cameras are used to realize the synchronization processing of the events; the frame is subjected to noise reduction processing by using median filtering, and the center coordinates of the LED light source module are obtained by iteratively estimating other edge features by RANSAC, and the intrinsic parameters of the event camera and the relative position relationship of the plurality of event cameras are calculated.

2. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 1, the 38 LED lamp beads of the light source module can be independently controlled through the microcontroller and the Internet of Things module on the bottom circuit board, including the adjustment of brightness and flashing mode, and the flashing frequency depends on different brands and types of event cameras, and the flashing frequency needs to ensure that the event camera can at least capture the complete process from extinguishing to flashing to extinguishing within the time interval of one superposition.

3. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 2, the size, shape and position of the front transverse clamping groove and the longitudinal clamping groove of the light source module fixing device can be flexibly adjusted according to the size and shape of the chessboard of different calibration boards to adapt to different calibration requirements.

4. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 3, the LED dot matrix frame synchronization device displays numbers during data acquisition, and the numbers start counting from 1, and the timing frequency depends on the time required for the event camera to complete an event acquisition process, and the numbers synchronize the frames of the plurality of event cameras.

5. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 4, the calibration board control node sends instructions to the light source module bottom circuit board and LED dot matrix frame synchronization device through the host computer, realizes dynamic real-time remote control of the light source module and the frame synchronization device, and adapts to the actual needs of different calibration processes.

6. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 5, during the event camera data acquisition process, the parameters of the light source module are adjusted according to the instructions of the calibration board control node, including the light source size, the flashing frequency and the LED dot matrix digital display interval.

7. The LED adjustable light source and LED frame synchronization based event camera calibration device and method of claim 1, wherein, In step 6, through LED dot matrix module frame synchronization processing and LED light source module center extraction, the relative position relationship of multiple event cameras is calculated, and the joint calibration of multiple event cameras is realized. In the calibration frame preprocessing process, median filtering is used for noise reduction processing of the frame, and RANSAC is used for iterative estimation of the edge features of the LED light source module, which can improve the robustness of the edge features and ensure more accurate estimation of the center of the LED light source module.

Images