Binocular event camera calibration method and device based on scintillation calibration board

Abstract

The application provides a kind of binocular event camera calibration method and device based on flicker calibration board, belongs to computer vision-camera calibration technical field, builds binocular event camera calibration system, generates virtual calibration board picture, and is made into flicker video with black background picture and is played in equal ratio;Different calibration poses under the calibration board event data are collected using binocular event camera;Calibration board event data is reconstructed into grayscale image, feature point coordinate information is obtained by feature extraction, and initial values of binocular event camera internal and external parameters and distortion coefficient are obtained using traditional calibration algorithm;Global optimization function and cost function are constructed, and iterative solution is carried out, so that the result of cost function is less than the set threshold value, and the final camera parameter is obtained.The device comprises a modeling module, a preprocessing module, an initial calibration module and a parameter optimization module.The application realizes high-precision calibration of binocular event camera in static state, effectively improves the calibration precision, and lays a foundation for better serving industrial measurement.

Description

Technical Field

[0001] This invention belongs to the field of computer vision-camera calibration technology, specifically relating to a binocular event camera calibration method and apparatus based on a scintillation calibration board. Background Technology

[0002] An event camera is a novel type of dynamic vision sensor. Unlike standard cameras, it does not capture instantaneous light intensity information of a scene, but only dynamically responds to parts of the scene where light intensity changes. It employs a novel Address Event Protocol (AER) encoding protocol to encode events as (x, y, z, p). It features high dynamic range, high temporal resolution, and low data redundancy, and is therefore considered to have the potential to better replace standard cameras in certain special environments to perform related tasks.

[0003] Like traditional cameras, event cameras can be simplified to pinhole imaging. When using binocular event cameras for depth perception tasks, establishing the transformation relationship between pixels in the image and their corresponding spatial points is crucial. This transformation relationship is the intrinsic and extrinsic parameter matrix of the binocular camera, which needs to be obtained in advance through camera calibration. Some event cameras, such as DAVIS and ATIS, can generate instantaneous grayscale information of the scene, which can be calibrated using traditional frame camera calibration algorithms. However, simultaneously acquiring instantaneous grayscale frames and asynchronous event streams of the scene imposes many hardware limitations. Integrating the active pixel sensor (APS) circuit and the AER circuit on the same pixel results in a large pixel size, making it difficult to improve resolution, which limits the application of event cameras to some extent. To fully utilize the characteristics of event cameras, newly manufactured event cameras no longer provide instantaneous grayscale output, which also means that grayscale output cannot be directly used for calibration. Therefore, in applications such as environmental perception and 3D reconstruction that acquire depth information, achieving high-precision calibration has become a new challenge.

[0004] The object distance is tens, hundreds, or even thousands of times greater than the focal length. A tiny error in the system parameters can be amplified several times over, and the calibration result directly affects the accuracy and precision of 3D perception. Currently, event camera calibration methods can be divided into direct event-based calibration and reconstructed frame-based calibration. The first type of method moves a calibration board in front of the camera, directly extracting event features triggered by feature edges during the movement, and then combining this with calibration board event frames captured at different positions along the trajectory for calibration and optimization. The second type of method also moves the calibration board in front of the camera, reconstructs grayscale frames using intensity reconstruction algorithms, and then combines this with traditional calibration algorithms for calibration. However, regardless of the calibration method, all rely on the movement of the calibration board to trigger events, which introduces movement errors to some extent. Especially in binocular calibration, theoretically, the two cameras need to maintain strict time synchronization for the same set of calibration images. However, reconstructed frames require the accumulation of events over a period of time. Due to hardware limitations, timestamps are difficult to align perfectly and exhibit different zero-drift effects, resulting in the actual reconstructed frames from the same set of calibration boards under the binocular viewpoint not being strictly spatiotemporally consistent, introducing additional errors. This means that binocular event camera calibration can never achieve the same calibration accuracy as a camera operating on the same frame. Other scholars have proposed other calibration methods, such as the invention patent CN202110031239.5, which discloses a method, device, computer equipment, and storage medium for calibrating the intrinsic parameters of an event camera. However, its main purpose is to achieve ease of operation. As a preliminary step in 3D perception, a single calibration can complete many experiments in that state; therefore, the efficiency of calibration does not actually affect the reconstruction efficiency. Therefore, the calibration process should prioritize accuracy over ease of operation. The limited accuracy of current calibration methods significantly restricts the industrial application of event cameras. Summary of the Invention

[0005] To address the aforementioned deficiencies and improvement needs of existing technologies, this invention provides a binocular event camera calibration method and apparatus based on a scintillation calibration board. This method excites events through a scintillation virtual calibration board, achieving high-precision calibration of a binocular event camera in a static state, effectively improving the calibration accuracy of the binocular event camera system and laying the foundation for it to better serve industrial measurement.

[0006] A method for calibrating a binocular event camera based on a scintillation calibration board, the method comprising:

[0007] Step 1: Build a binocular event camera calibration system, including a monitor and a binocular event camera; establish the relationship between virtual size and physical size based on the monitor resolution and calibration board size, and generate a virtual calibration board image of a specific physical size, wherein the virtual calibration board image has several circular feature points;

[0008] Step 2: Use the virtual calibration board image generated in Step 1 and the black background image to create a video with a flickering frequency of 100Hz, and play it on the monitor using a player at the same ratio;

[0009] Step 3: Place the monitor in the common field of view of the binocular event camera, fix the pose of the binocular event camera, and in a scenario where the binocular camera and the monitor remain relatively stationary, play the video from Step 2 to trigger the calibration board image to generate event information. Use the binocular event camera to collect a set of calibration board event data under this calibration pose. Rotate the monitor or change the monitor's pitch angle to the next calibration pose and collect the next set of calibration board event data. Collect a total of 10-20 sets of calibration board event data.

[0010] Step 4: Use an accumulator to reconstruct the calibration board event data. Each set of calibration board event data is reconstructed into a grayscale image, resulting in a total of 10-20 grayscale images.

[0011] Step 5: Extract features from a series of reconstructed grayscale images to obtain 10-20 sets of calibration board feature point coordinate information. Use traditional calibration algorithms to obtain the initial values ​​of the binocular event camera's intrinsic and extrinsic parameters and distortion coefficients. Each grayscale image yields a set of calibration board feature point coordinate information.

[0012] Step 6: Construct a global optimization function and a cost function. Substitute the intrinsic and extrinsic parameters of the stereo event camera and the initial values ​​of the distortion coefficients under different calibration poses, and iteratively solve the global optimization function to make the cost function result less than the set threshold, thus obtaining the final camera parameters.

[0013] Furthermore, the binocular event camera calibration system in step 1 includes a display and a binocular event camera. The binocular event camera consists of two event cameras (left and right), two camera platforms, two sliding and rotatable supports, and a sliding rail. The two event cameras are symmetrically arranged on both sides of the display and are fixedly connected to the two camera platforms. Three sets of parallel through holes are provided on the platforms to adjust the camera positions. The two camera platforms are fixedly connected to the two supports, which are connected to the sliding rail using shaped sliders and bolts that match the sliding rail, used to adjust the positions of the two event cameras. The event camera is the main body of the binocular event camera calibration system. The supports connect the event camera to the sliding rail and simultaneously enable rotation of the event camera in the xy plane. The sliding rail enables translation of the event camera along the baseline, i.e., the x-axis direction, thereby changing the binocular distance. In the binocular event camera calibration system, the x-axis direction of the world coordinate system is the direction of the line connecting the optical centers of the left and right event cameras, i.e., the baseline direction; the y-axis direction is perpendicular to the baseline and points towards the display plane; and the z-axis direction is perpendicular to the system platform and upwards.

[0014] Furthermore, in step 3, the two event cameras are DVXplorer models, and the 10-20 sets of positioning poses cannot have two sets of coplanar situations, and the display cannot have large-scale reflections.

[0015] Furthermore, step 4 specifically involves: dividing each set of calibration board event data into windows containing multiple events, and converting each window into a 3D event tensor; generating each 3D event tensor and using an accumulator to generate a new image reconstruction and a new tensor, and combining them into a set; after accumulating for a certain period of time, reconstructing the final grayscale image from the set of all tensors and image reconstructions in that set.

[0016] In order to improve the detail information of the reconstructed image in a relatively static scene, the accumulator's event accumulation time should be no less than the reciprocal of the video flicker frequency in step 2 (for example, if the flicker frequency is 100Hz, the event accumulation time should be no less than 1 / 100s, i.e., 10ms).

[0017] Furthermore, step 5 specifically involves: performing feature point detection and sorting on the reconstructed grayscale image, using a traditional calibration algorithm for initial calibration, and obtaining initial values ​​for camera intrinsic and extrinsic parameters and distortion coefficients, including the camera intrinsic parameter matrix, translation vector, rotation matrix, and distortion coefficients.

[0018] Step 5.1: Denoise the reconstructed grayscale image using a Gaussian smoothing template and median filtering to obtain a noise-free grayscale image.

[0019] Step 5.2: Apply the edge detection (Canny) algorithm to the denoised grayscale image to find the brightness gradient of the grayscale image, and perform non-maximum suppression based on the gradient to obtain a binarized image;

[0020] Step 5.3: For the binarized image, according to the 8-neighborhood connectivity criterion, replace the arc with multiple polylines, connect the polylines, and obtain the image of continuous boundaries;

[0021] Step 5.4: For images with continuous boundaries, use the least squares method to fit elliptical features and obtain parameter information such as the center position of the ellipse, the radii of the major and minor axes, and the rotation angle.

[0022] Step 5.5: Calculate the ellipticity based on the elliptic center position and parameter information, select feature points that meet the requirements (ellipticity less than 0.075) for conic section reconstruction, extract coordinate information from the reconstructed feature points, and obtain the coordinate values ​​of the feature point center.

[0023] Step 5.6: Starting from the top left feature point, traverse the columns until the bottom right feature point, sort the coordinates of the feature point centers to obtain the feature point sorting result;

[0024] Step 5.7: Using the obtained coordinate information of the center of the feature points on the calibration board and the sorting results, the traditional calibration algorithm is used to establish four coordinate systems for object imaging (world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system) based on the pinhole imaging model. The image coordinates and three-dimensional coordinates of the feature points on the calibration board are solved to obtain the camera intrinsic parameter matrix, rotation matrix, translation vector, and distortion coefficient.

[0025] Furthermore, step 6 specifically includes:

[0026] Step 6.1: Using the camera's intrinsic and extrinsic parameters, the initial values ​​of the distortion coefficients, and the feature point coordinate information obtained in step 5, project a set of virtual ellipses onto the image plane.

[0027] Step 6.2: Replace the original feature points with virtual ellipses and recalibrate to obtain new camera intrinsic and extrinsic parameters and distortion coefficients;

[0028] Step 6.3: Using the new camera intrinsic and extrinsic parameters and distortion coefficients, and the ellipse parameters and coordinates obtained from the previous projection, project a new set of virtual ellipses onto the image plane, establish a global optimization function and a cost function, and calculate the distance between the center point of the virtual ellipses and the actual ellipse.

[0029] The global optimization function and the cost function are respectively:

[0030]

[0031] In the formula, ξ represents the distance between the virtual ellipse center and the actual projected ellipse center, where the actual projected ellipse center is calculated from the physical dimensions of the calibration plate; l and ξ r Indicates the initial values ​​of the left and right camera parameters; The three-dimensional coordinates of the first to the nth feature points; and These are the parameter values ​​for the left and right cameras obtained in this iteration; These are the three-dimensional coordinates of the first to nth feature points obtained from this reprojection; The virtual ellipse coordinates are calculated using the left and right camera parameters obtained in this iteration; Ψ l Ψ r A function that projects 3D points onto a 2D plane using left and right camera parameters; These are the virtual ellipse coordinates calculated using the initial values ​​of the left and right camera parameters (first iteration) or the left and right camera parameters obtained from the previous iteration (second and subsequent iterations); Λl and Λ r These are the essential matrices of the left and right camera parameters, respectively; d l and d r These are the distortion coefficients for the left and right cameras, respectively; and Let Γ be the transformation matrix between the left and right camera coordinate systems in the extrinsic parameters of the m-th iteration. S This is the matrix used to transform the world coordinate system to the left camera coordinate system in the extrinsic parameters of the m-th iteration; Let M and N be the three-dimensional coordinates of the nth feature point; M and N are M images in different poses and N feature points on each image, respectively.

[0032] Step 6.4 Iterate over steps 6.2 and 6.3. Each iteration yields a new set of virtual ellipse and camera parameters, as well as the distance between the ellipse centers. Repeat the above iteration process until the cost function result is less than the set threshold, and obtain the final camera parameters.

[0033] A binocular event camera calibration device based on a scintillation calibration plate is used to implement the above-mentioned binocular event camera calibration method. The binocular event camera calibration device includes:

[0034] The modeling module is used to build a binocular event camera calibration system and generate virtual calibration board images with specific physical dimensions, and to create a video by combining the images with a black background.

[0035] The preprocessing module is used to play videos, stimulate the generation of binocular event information, adjust the calibration pose, and collect calibration board event data; the accumulator reconstructs the calibration board event data into a grayscale image.

[0036] The initial calibration module is used to extract features from the reconstructed grayscale image and use traditional calibration algorithms to obtain the initial values ​​of the intrinsic parameters of the left and right event cameras, the extrinsic parameters between the two cameras, and the distortion coefficients.

[0037] The parameter optimization module is used to construct a global optimization function and a cost function. By substituting the camera's intrinsic and extrinsic parameters and initial values ​​of distortion coefficients under different poses, the global optimization function is iteratively solved to make the cost function value less than a set threshold, thereby obtaining the final camera parameters and improving calibration accuracy.

[0038] The innovative aspects of this invention:

[0039] First, this invention creates a virtual calibration board image and plays it back as a video. Existing camera calibration methods, whether for ordinary cameras or event cameras, use physical calibration boards. Physical calibration boards may have drawbacks such as poor surface quality, inaccurate feature point positions, and blurred feature point edge information. Furthermore, the transportation and storage of physical calibration boards are also problematic. In contrast, the virtual calibration board is played back on a monitor, where surface quality depends only on the monitor screen. The feature point positions are more precise, edge features are clearer, and transportation and storage issues are eliminated.

[0040] Secondly, this invention eliminates the need to move the camera or calibration board to generate event data. Most existing event camera calibration methods require moving the camera or calibration board to generate event information. During camera or calibration board movement, image distortion or feature point misalignment inevitably occurs, affecting the final calibration results. In this invention, movement is limited to moving the display to different poses, and event information is generated by flashing in a fixed pose. Therefore, the reconstructed image in each pose has definite image features and positional information, avoiding movement errors.

[0041] Meanwhile, this invention offers higher calibration accuracy compared to other event camera calibration methods. Existing event camera calibration methods typically use the center of a feature point in the detected distorted 2D image as the ground truth. However, after perspective projection and lens distortion, circular feature points become elliptical on the image plane, causing the true center to deviate from the feature point's center, thus affecting the calibration results. This invention proposes an optimized method that uses the initial calibration parameters to first project the calibration board's feature points onto the camera's image plane, obtaining a distortion-free feature point image. Then, the distortion-free feature points are used for calibration, and the camera parameters are recalculated. This process is repeated until the optimization function converges, resulting in an error lower than existing calibration methods.

[0042] The beneficial effects of this invention are:

[0043] This invention does not rely on motion-triggered events. Compared to calibration using a moving calibration board or camera, it can accumulate more events to increase the feature point information of the calibration board without introducing additional motion errors. The binocular event camera system uses an accumulator to reconstruct calibration images in real time, exhibiting strict spatiotemporal consistency. Calibration can be performed using mature traditional camera calibration algorithms, eliminating the need for designing new algorithms for processing and feature extraction. This significantly reduces the difficulty of binocular event camera calibration, enabling it to meet the requirements of high-precision environmental perception. Furthermore, calibration can be completed using only a common electronic display screen as the calibration object. The calibration board image can be scaled for different camera lens field of view, flexibly adjusting the size of the virtual calibration board, demonstrating good adaptability. Regarding camera parameter optimization, this invention does not use the extracted feature points as fixed ground truth values ​​but rather iteratively updates the camera parameters, thus remaining unaffected by the final initial values ​​and improving calibration accuracy. Attached Figure Description

[0044] Figure 1 This is a flowchart of the binocular event camera calibration method based on a scintillation calibration board in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the binocular event camera calibration in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram illustrating the specific process of center-sorting in an embodiment of the present invention;

[0047] Figure 4 This is an image of the virtual calibration board used in the embodiments of the present invention;

[0048] Figure 5 This is a schematic diagram of the internal structure of a computer device in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be described in detail below with reference to the embodiments. However, the implementation of the present invention is not limited thereto. Obviously, the embodiments described below are only some embodiments of the present invention. For those skilled in the art, other similar embodiments can be obtained without creative effort and all fall within the protection scope of the present invention.

[0050] Example 1

[0051] This embodiment provides a binocular event camera calibration method based on a scintillation calibration board, such as... Figure 1 As shown, the specific implementation process is as follows:

[0052] The first step is to construct a binocular event camera calibration system, including a monitor and binocular event cameras. The binocular event camera system consists of two event cameras (left and right), two camera platforms, two sliding and rotating supports, and a sliding rail. The two event cameras are symmetrically arranged on both sides of the monitor and are fixedly connected to the two camera platforms. Each platform has three sets of parallel through holes to adjust the camera's position. The two camera platforms are fixedly connected to the two supports, which are connected to the sliding rail using shaped sliders and bolts that match the sliding rail, used to adjust the position of the two event cameras. The event cameras are the main body of the binocular event camera calibration system. The supports connect the event cameras to the sliding rail and allow for rotation of the event cameras in the xy plane. The sliding rail allows for translation of the event cameras along the baseline, i.e., the x-axis, thereby changing the binocular distance. In the binocular camera system, the x-axis direction of the world coordinate system is the direction of the line connecting the optical centers of the left and right event cameras, i.e., the baseline direction; the y-axis direction is perpendicular to the baseline and points towards the monitor plane; and the z-axis direction is perpendicular to the system platform and upwards.

[0053] A virtual calibration board image with 11 rows and 9 columns, containing 99 circular feature points, is generated on a display with a resolution of 2550×1440. The calibration board image is a circular array with a center-to-center spacing of 15mm. Figure 4 As shown.

[0054] The second step is to create a video by combining the virtual calibration board image from the first step with the black background image, which flashes alternately at a frequency of 100Hz, and then play it on the monitor.

[0055] The third step involves adjusting the distance between the display and the binocular event camera, as well as the poses of the left and right cameras, so that the display is positioned within the common field of view of the binocular event camera. The camera poses are then fixed, and event information is generated by playing the video produced in the second step. The binocular event camera is then used to collect calibration board event data under this calibration pose. The event camera model is DVXplorer, with a resolution of 640×480 pixels. The process of placing the display within the relatively stationary common field of view of the binocular event camera includes:

[0056] Adjust the slider so that the distance between the system baseline, i.e. the distance between the optical centers of the left and right event cameras, is 400mm. At this time, the distance between the lenses of the left and right event cameras in the x direction is 400mm.

[0057] Adjust the camera platform and rotate the camera so that the angle between the optical axes of the left and right cameras is 50° (for ease of calculation, in this embodiment, the baselines of the left and right event cameras are adjusted to be equal to the angles between the optical axes of their respective cameras, i.e.) ),in This represents the angle between the optical axis of the left camera and the baseline. This represents the angle between the optical axis of the right camera and the baseline;

[0058] Adjust the left and right event camera lenses so that their focal lengths are both 25mm;

[0059] Adjust the distance between the binocular event camera and the monitor to approximately 900mm, so that the calibration plate occupies 1 / 4 to 1 / 2 of the field of view in the image. Make fine adjustments at this distance until all feature points can be displayed completely and clearly in the image.

[0060] In this embodiment, the binocular event camera calibration system is as follows: Figure 2 As shown, a binocular event camera with a fixed pose is used. Under the same baseline length, optical axis angle, lens focal length, and distance between the lens and the display, the display angle is adjusted to different calibration poses. At each calibration pose, the binocular event camera continuously captures images of the flashing calibration board to obtain calibration board event data. One set of calibration board event data is collected for each calibration pose, and 12 sets of calibration board event data are collected continuously. A preliminary assessment is then made to determine whether the event data meets the requirements for feature detection and ranking.

[0061] If the event stream information is scarce or scattered, or if the calibration board image is tilted at too large an angle, discard the data set and collect a new set of data that meets the requirements.

[0062] If the event stream information meets the detection and sorting requirements, and the calibration board pose is normal, then the data set is retained.

[0063] The fourth step involves inputting the calibration board event data collected in the previous step into the accumulator. The accumulator is then used to reconstruct the calibration board event data. Each set of calibration board event data is reconstructed into a grayscale image, resulting in a total of 12 grayscale images. The reconstruction process includes:

[0064] (1) Divide each set of calibration board event data into windows containing multiple events, and convert the windows into a 3D event tensor;

[0065] (2) For each 3D event tensor generated, it is combined with the previously segmented and transformed 3D event tensors in the same group through an accumulator to generate a new image reconstruction and a new tensor, and then assembled into a set;

[0066] (3) After accumulating for a certain period of time, the final grayscale image is reconstructed from the set of all tensors and image reconstructions in this group. In this embodiment, the flashing frequency is 100Hz and the time of the accumulated event should not be less than 1 / 100s, i.e. 10ms.

[0067] The fifth step involves feature point detection and sorting of the reconstructed grayscale image, followed by initial calibration using a traditional calibration algorithm to obtain the camera intrinsic parameter matrix, translation vector, rotation matrix, and initial values ​​of distortion coefficients. The process of extracting calibration board feature points from the grayscale image includes:

[0068] (1) The reconstructed grayscale image is denoised using a Gaussian smoothing template and median filtering to obtain a noise-free grayscale image;

[0069] (2) The Canny algorithm is used on the denoised grayscale image to find the brightness gradient of the grayscale image. Non-maximum suppression is performed according to the gradient. The threshold grayscale value is set to 40. The grayscale values ​​above this value are averaged to 255, and the grayscale values ​​below this value are averaged to 0, so as to obtain a binary image containing only black and white pixels.

[0070] (3) For the binarized image, according to the 8-neighborhood connectivity criterion, the boundary pixels of the feature points of the binary image are connected, and the arc is replaced by a multi-segment polyline. The line segments are extracted and the rotation direction is unified to obtain the image of continuous boundary.

[0071] (4) For images with continuous boundaries, use the least squares method to fit elliptical features and obtain parameters such as the center position of the ellipse, the radius of the major and minor axes, and the rotation angle.

[0072] (5) Calculate the ellipticity based on the elliptic center position and parameter information. Reconstruct the feature points with an ellipticity less than 0.075 (the smaller the ellipticity, the closer it is to a circle) using conic sections to obtain a calibration plate image with feature point parameter information.

[0073] (6) First, using the coordinates of the four points (top left, bottom left, top right, and bottom right) of the calibration plate image, and the number of rows and columns of the calibration plate, the distance between the center points is calculated to be 15mm. Then, by comparing the distance with the minor axis length of the central feature point, the positions of the five larger circles are found, and the coordinates of the center of the larger circles are determined. Figure 3 The following describes the specific process of sorting the large circles: ① Define the point with the smallest distance from the edge of the calibration board as circle 1; ② Calculate the distance between each large circle, find the two closest large circles, solve the equation of the straight line connecting the centers of these two circles, and define the point with the largest distance from the line as the center of circle 2; ③ Define the two circles on the straight line in step 2 as circle 4 and circle 5, and define the one with the smaller distance from circle 1 as circle 4; ④ After determining circles 1, 2, 4, and 5, the remaining circle is circle 3. Due to the affine transformation (linear transformation + translation) of the calibration board, the positions of circle 1 and circle 3 may be opposite. The distances of the two intersection points formed by the lines connecting circle 4 and circle 2, and circle 5 and circle 2, and the lines connecting circle 1 and circle 3, relative to the center point of the calibration board image can be compared to determine whether the positions of circle 1 and circle 3 need to be swapped. After completing the sorting of the large circles, based on the average interval between rows and columns of the calibration plate, several sets of straight lines (9 horizontal lines and 11 vertical lines) parallel or perpendicular to the lines connecting circles 1 and 3 and circles 2 and 4 can be calculated. Based on the intersections of these pairs of lines, the small circles on the calibration plate can be sorted. This completes the process of detecting and sorting the circle centers.

[0074] (7) Using traditional calibration methods, the image coordinates and 3D coordinates of the feature points on the calibration board are solved to obtain the camera intrinsic parameter matrix, rotation matrix, translation vector, and distortion coefficients. Specifically:

[0075] The pinhole camera imaging model used in traditional calibration algorithms can represent the mapping relationship between two-dimensional image points and three-dimensional points as follows:

[0076] sm=K[R|t]M

[0077] Where m is the coordinate of the two-dimensional image point, M is the spatial coordinate of the three-dimensional point, and s is the scale factor. Let a be the camera intrinsic parameter matrix. x and a y K, R, and t are the normalized focal lengths in the horizontal and vertical directions, respectively; γ is the tilt angle of the two image axes; (u0, v0) are the image pixel coordinates of the principal point; and R and t are the rotation matrix and translation vector between the camera coordinate system and the world coordinate system, respectively. Calibration is the process of solving for K, R, and t.

[0078] First, the camera's intrinsic parameters are solved using traditional calibration methods, with a planar calibration plate used as a calibration reference. The above formula can be expressed as follows:

[0079]

[0080] Where r1, r2, and r3 are the first three columns of the rotation matrix, X is the x-coordinate of the feature point in the world coordinate system, Y is the y-coordinate of the feature point in the world coordinate system, and H is the homography matrix between the image coordinate plane and the world coordinate plane, which contains the camera's intrinsic and extrinsic parameters and can be solved through the relationship between four or more corresponding points. From the above equation, it can be seen that...

[0081] [h1h2h3]=λK[r1r2 t]

[0082] Where h1, h2, and h3 are column vectors of the homography matrix, λ is a scaling factor, and the rotation matrix R is an orthogonal matrix, we have:

[0083]

[0084] Let B = K -T K -1 To solve for matrix B, it is necessary to calculate but:

[0085]

[0086] remember

[0087] v ij =[H 1i H 1j H 1i H2j +H 2i H 1j H 2i H 2j H 1i H 3j +H 3i H 1j H 2i H 3j +H 3i H 2j H 3i H 3j ] T

[0088] b = [B 11 B 12 B 22 B 13 B 23 B 33 ] T

[0089] but It can be transformed into:

[0090] H i T BH j =v ij T b

[0091] Where b is a matrix composed of column vectors of matrix B, B 11 B 12 B 22 B 13 B 23 B 33 These are the column vectors of matrix B, H i Let v be the column vectors of the homography matrix. ij The polynomial composed of the column vectors of matrix H;

[0092] The orthogonality relation of the rotation matrix can be rewritten as:

[0093]

[0094] During camera calibration, images are acquired at n calibration plate poses. By combining the equations for each pose, the following can be obtained:

[0095]

[0096] Solving the above equation will yield the intrinsic and extrinsic parameters of a single camera.

[0097] For stereo event cameras, it is also necessary to calculate the rotation and translation matrices of the left and right cameras relative to the same coordinate system, which can be obtained by the following formula:

[0098] R = R r (R l )T

[0099] T = T r -RT l

[0100] Where R is the rotation matrix between the two cameras, and T is the translation matrix between the two cameras. r T is the rotation matrix of the right camera relative to the calibration plate obtained by the traditional calibration algorithm. r R is the translation vector of the right camera relative to the calibration board obtained using a traditional calibration algorithm. l T is the rotation matrix of the left camera relative to the calibration plate obtained by the traditional calibration algorithm. l The translation vector of the left camera relative to the calibration board is obtained using a traditional calibration algorithm.

[0101] Step 6: Collect optimized camera parameter values ​​from multiple grayscale images, construct a global optimization function and a cost function, and solve them. Ensure the cost function result is less than a set threshold to obtain the optimized camera parameters. Specifically:

[0102] s1 uses the camera's intrinsic and extrinsic parameters, initial values ​​of distortion coefficients, and fitted ellipse parameters and coordinates to project a set of virtual ellipses onto the image plane;

[0103] s2 uses a virtual ellipse to replace the ellipse obtained from the original feature extraction, and recalibrates to obtain new camera intrinsic and extrinsic parameters and distortion coefficients;

[0104] s3 uses the new camera's intrinsic and extrinsic parameters and distortion coefficients, along with the ellipse parameters and coordinates obtained from the previous projection, to project a new set of virtual ellipses onto the image plane. It then establishes a global optimization function and a cost function to calculate the distance between the center point of the virtual ellipses and the actual ellipse.

[0105] S4 iterates over S2 and S3, obtaining a new set of virtual ellipse and camera parameters, as well as the distance between the ellipse centers, in each iteration. This iterative process is repeated until the error value is less than the set threshold (0.1 pixels).

[0106] The global optimization function is:

[0107]

[0108] The cost function is:

[0109]

[0110] In calculating the virtual ellipse projection error, the above formula is used... Replace with virtual elliptical coordinates That's all.

[0111] Example 2

[0112] This embodiment provides a binocular event camera calibration device based on a stroboscopic calibration plate, including:

[0113] The modeling module is used to build a binocular event camera calibration system and generate virtual calibration board images with specific physical dimensions, which are then combined with a black background image to create a video.

[0114] The preprocessing module is used to play videos, stimulate the generation of binocular event information, adjust the calibration pose, and collect calibration board event data; the accumulator reconstructs the calibration board event data into a grayscale image.

[0115] The initial calibration module is used to extract features from the reconstructed grayscale image and use traditional calibration algorithms to obtain the intrinsic parameters of the left and right event cameras, the extrinsic parameters between the two cameras, and the initial values ​​of the distortion coefficients.

[0116] The parameter optimization module is used to construct a global optimization function and a cost function. By substituting the camera's intrinsic and extrinsic parameters and initial values ​​of distortion coefficients under different poses, the global optimization function is iteratively solved to make the cost function value less than a set threshold, thereby obtaining the final camera parameters and improving calibration accuracy.

[0117] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0118] In further embodiments, the following is also provided:

[0119] An electronic device, such as Figure 5 As shown, it includes a memory and a processor, as well as computer instructions stored in the memory and on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For simplicity, further details are omitted here.

[0120] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0121] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0122] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0123] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0124] This invention provides a binocular event camera calibration method based on a scintillation calibration board. This method achieves relatively static calibration between the binocular event camera and the calibration board, avoiding spatiotemporal errors introduced by event frame capture due to calibration board movement and binocular timestamp alignment. This significantly improves the accuracy and stability of the calibration, laying a solid foundation for high-precision 3D perception and reconstruction based on binocular event cameras. This method is not limited to high-precision calibration between binocular event cameras; it can also be applied to hybrid calibration between event cameras and frame cameras.

[0125] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for calibrating a binocular event camera based on a scintillation calibration board, characterized in that, The binocular event camera calibration method includes: Step 1: Build a binocular event camera calibration system, including a monitor and a binocular event camera; establish the relationship between virtual size and physical size based on the monitor resolution and calibration board size, and generate a virtual calibration board image of a specific physical size, wherein the virtual calibration board image has several circular feature points; Step 2: Use the virtual calibration board image generated in Step 1 and the black background image to create a video with a certain flickering frequency, and play it on the monitor at the same ratio; Step 3: Place the monitor in the common field of view of the binocular event camera, fix the pose of the binocular event camera, and in a scenario where the binocular camera and the monitor remain relatively stationary, play the video from Step 2 to trigger the calibration board image to generate event information. Use the binocular event camera to collect a set of calibration board event data under this calibration pose. Rotate the monitor or change the monitor's pitch angle to the next calibration pose and collect the next set of calibration board event data. Collect a total of 10-20 sets of calibration board event data. Step 4: Use an accumulator to reconstruct the calibration board event data, and reconstruct each set of calibration board event data into a grayscale image; Step 5: Extract features from the series of reconstructed grayscale images to obtain the coordinate information of the calibration board feature points. Use the traditional calibration algorithm to obtain the initial values ​​of the intrinsic and extrinsic parameters and distortion coefficients of the binocular event camera; wherein, each grayscale image yields a set of calibration board feature point coordinate information. Step 6: Construct a global optimization function and a cost function. Substitute the intrinsic and extrinsic parameters of the stereo event camera under different calibration poses and the initial values ​​of the distortion coefficients, and iteratively solve the global optimization function until the cost function result is less than a set threshold to obtain the final camera parameters; specifically: Step 6.1: Using the camera's intrinsic and extrinsic parameters, the initial values ​​of the distortion coefficients, and the feature point coordinate information obtained in step 5, project a set of virtual ellipses onto the image plane. Step 6.2: Replace the original feature points with virtual ellipses and recalibrate to obtain new camera intrinsic and extrinsic parameters and distortion coefficients; Step 6.3: Using the new camera intrinsic and extrinsic parameters and distortion coefficients, and the ellipse parameters and coordinates obtained from the previous projection, project a new set of virtual ellipses onto the image plane, establish a global optimization function and a cost function, and calculate the distance between the center point of the virtual ellipses and the actual ellipse. The global optimization function and the cost function are respectively: In the formula, The distance between the virtual ellipse center and the actual projected ellipse center is represented by the distance between the virtual ellipse center and the actual projected ellipse center, where the actual projected ellipse center is calculated from the physical dimensions of the calibration plate. and Indicates the initial values ​​of the left and right camera parameters; The three-dimensional coordinates of the first to the nth feature points; and These are the parameter values ​​for the left and right cameras obtained in this iteration; These are the three-dimensional coordinates of the first to nth feature points obtained from this reprojection; , The virtual ellipse coordinates are calculated using the left and right camera parameters obtained in this iteration; , A function that projects 3D points onto a 2D plane using left and right camera parameters; , The virtual ellipse coordinates are calculated using the initial values ​​of the left and right camera parameters (first iteration) or the left and right camera parameters obtained from the previous iteration (second and subsequent iterations), respectively. and These are the essential matrices of the parameters for the left and right cameras, respectively. and These are the distortion coefficients for the left and right cameras, respectively; and Let be the transformation matrix between the left and right camera coordinate systems in the extrinsic parameters of the m-th iteration. This is the matrix used to transform the world coordinate system to the left camera coordinate system in the extrinsic parameters of the m-th iteration; Let M and N be the three-dimensional coordinates of the nth feature point; M and N are M images in different poses and N feature points on each image, respectively. Step 6.4 Iterate over steps 6.2 and 6.

3. Each iteration yields a new set of virtual ellipse and camera parameters, as well as the distance between the ellipse centers. Repeat the above iteration process until the cost function result is less than the set threshold, and obtain the final camera parameters.

2. The binocular event camera calibration method based on a scintillation calibration board according to claim 1, characterized in that, The binocular event camera calibration system in step 1 includes a display and a binocular event camera. The binocular event camera consists of two event cameras, left and right. The two event cameras are symmetrically arranged on both sides of the display and can adjust the binocular distance and rotate in the xy plane. In the binocular event camera calibration system, the x-axis of the world coordinate system is the direction of the line connecting the optical centers of the left and right event cameras, i.e., the baseline direction; the y-axis is the direction perpendicular to the baseline and pointing towards the display plane; and the z-axis is the direction perpendicular to the system platform and upward.

3. The binocular event camera calibration method based on a scintillation calibration board according to claim 1, characterized in that, Step 4 specifically involves: dividing each set of calibration board event data into windows containing multiple events, and converting each window into a 3D event tensor; generating each 3D event tensor and using an accumulator to generate a new image reconstruction and a new tensor, and combining them into a set; after accumulating for a certain period of time, reconstructing the final grayscale image from the set of all tensors and image reconstructions in that set.

4. The binocular event camera calibration method based on a scintillation calibration board according to claim 3, characterized in that, The time for the accumulator to accumulate events should be no less than the reciprocal of the video flicker frequency created in step 2.

5. The binocular event camera calibration method based on a scintillation calibration board according to claim 1, characterized in that, Step 5 specifically involves: detecting and sorting feature points in the reconstructed grayscale image, performing initial calibration using a traditional calibration algorithm, and obtaining initial values ​​for camera intrinsic and extrinsic parameters and distortion coefficients. Step 5.1: Denoise the reconstructed grayscale image using a Gaussian smoothing template and median filtering to obtain a noise-free grayscale image. Step 5.2: Apply an edge detection algorithm to the denoised grayscale image to find the brightness gradient of the grayscale image, and perform non-maximum suppression based on the gradient to obtain a binarized image; Step 5.3: For the binarized image, according to the 8-neighborhood connectivity criterion, replace the arc with multiple polylines, connect the polylines, and obtain the image of continuous boundaries; Step 5.4: For images with continuous boundaries, use the least squares method to fit elliptical features and obtain the center position and parameter information of the ellipse; Step 5.5: Calculate the ellipticity based on the ellipse center position and parameter information, select the feature points that meet the requirements for conic section reconstruction, extract the coordinate information from the reconstructed feature points, and obtain the coordinate values ​​of the feature point center. Step 5.6: Starting from the top left feature point, traverse the columns until the bottom right feature point, sort the coordinates of the feature point centers to obtain the feature point sorting result; Step 5.7: Using the obtained coordinate information of the center of the feature points on the calibration plate and the sorting results, the traditional calibration algorithm is used to solve the image coordinates and three-dimensional coordinates of the feature points on the calibration plate according to the pinhole imaging model, so as to obtain the camera intrinsic parameter matrix, rotation matrix, translation vector and distortion coefficient.

6. A binocular event camera calibration device based on a scintillation calibration plate, used to implement the binocular event camera calibration method according to any one of claims 1 to 5, characterized in that, The binocular event camera calibration device includes: The modeling module is used to build a binocular event camera calibration system and generate virtual calibration board images with specific physical dimensions, and to create a video by combining the images with a black background. The preprocessing module is used to play videos, stimulate the generation of binocular event information, adjust the calibration pose, and collect calibration board event data; the accumulator reconstructs the calibration board event data into a grayscale image. The initial calibration module is used to extract features from the reconstructed grayscale image and use traditional calibration algorithms to obtain the initial values ​​of the intrinsic parameters of the left and right event cameras, the extrinsic parameters between the two cameras, and the distortion coefficients. The parameter optimization module is used to construct a global optimization function and a cost function. By substituting the camera's intrinsic and extrinsic parameters and initial values ​​of distortion coefficients under different poses, the global optimization function is iteratively solved to make the cost function value less than a set threshold, thus obtaining the final camera parameters.

Images