An event camera calibration method based on event stream features

Abstract

The application discloses an event camera calibration method based on event stream features, comprising the following steps: using an event camera and an LCD display screen, the event camera acquires one horizontal and one vertical stripe event stream file. The event stream files are respectively subjected to phase unwrapping processing to obtain horizontal and vertical wrapped phase images. The wrapped phase images are subjected to edge detection and linear fitting to obtain edge line binary images of the horizontal and vertical wrapped phase images. The edge line binary images of the horizontal and vertical wrapped phase images are superimposed to obtain intersection points of horizontal and vertical stripe wrapped phase image edge lines, so as to determine pixel coordinates of calibration points. Through detection of feature points by using inherent characteristics of event streams, the application significantly improves calibration accuracy and effectiveness.

Description

Technical Field

[0001] This invention belongs to the field of camera calibration technology. Specifically, this invention relates to an event camera calibration method based on event flow characteristics. Background Technology

[0002] Event cameras, as an emerging sensor technology, are currently widely used in vision tasks such as stereo matching, simultaneous localization and mapping (SLAM), and structured light measurement. Event cameras offer advantages such as low latency, high dynamic range, and insensitivity to motion blur. Therefore, they can be used to measure fast-moving targets, address the problem of uneven lighting in measurement scenes, and improve measurement efficiency.

[0003] Currently, event camera calibration methods attempt to trigger events through various means, while fundamentally relying on spatial intensity distribution for feature point detection. Event cameras asynchronously measure intensity changes as they occur, but they are inherently susceptible to various forms of noise, including event jitter and quantization noise, and are insensitive to the spatial distribution of features. Therefore, the accuracy of existing event camera calibration methods remains limited, posing a continued challenge to achieving flexible and accurate event camera calibration. Summary of the Invention

[0004] This invention provides an event camera calibration method based on event flow features to solve the problems existing in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: an event camera calibration method based on event flow characteristics, specifically including the following steps:

[0006] Step S1: Set up an event camera calibration system; the system includes an event camera and an LCD display screen; place the LCD display screen within the imaging range of the event camera, and display a moving stripe pattern on the LCD display screen to trigger an event;

[0007] Step S2: Acquire event stream; The LCD screen is located directly in front of the event camera, and alternately displays the horizontal moving sine stripe image and the vertical moving sine stripe image once each. The event camera simultaneously captures images to acquire one horizontal stripe event stream file and one vertical stripe event stream file respectively.

[0008] Step S3: Perform phase unwrapping processing on each pixel of the horizontal stripe event stream and the vertical stripe event stream files using the time offset to obtain the horizontal wrapped phase image and the vertical wrapped phase image.

[0009] Step S4: Perform edge detection on the horizontal and vertical wrapped phase images respectively, and use the least squares method to fit the detected edge parts with straight lines to obtain the binary image of the edge line of the horizontal wrapped phase image and the binary image of the edge line of the vertical wrapped phase image.

[0010] Step S5: Superimpose the binary images of the horizontal and vertical edges of the phase image to obtain the intersection of the horizontal and vertical stripes of the phase image edge lines, which is the pixel coordinate of the calibration point.

[0011] Step S6: The LCD display screen performs translational and rotational rigid body motion N times, where N is an integer and N≥3; at each new position, repeat steps S2-S5 to obtain the pixel coordinates of the calibration point at different positions;

[0012] Step S7: Calculate the intrinsic parameters of the event camera using the ordinary calibration algorithm based on the pixel coordinates of the calibration points; firstly, solve for the parameters of the event camera using the direct linear transformation method, then use the obtained parameters as initial values, and then optimize the event camera parameter values ​​according to the LM algorithm to obtain the optimized event camera parameter values.

[0013] Furthermore, in step S2, the event camera triggers an event and generates an event stream whenever it detects a change in the logarithmic intensity of a pixel.

[0014] J(x,y,t) represents pixels. and time Logarithmic strength, event e = (e x ,e y ,e t ,e p It will report the spatial location (x, y) = (e x ,e y ), timestamp t = e t This indicates that J(x,y,t) exceeds the threshold, and the polarity e p ∈{+1,-1} indicates whether the change in logarithmic strength is an increase or a decrease;

[0015] Ideally, an event is triggered when the logarithmic intensity exceeds a pre-defined threshold in the event camera hardware δ.

[0016] The i-th event The definition is as follows:

[0017]

[0018] in, R(x,y) is the previous event of the sensor, and R(x,y) is the reference value of pixel (x,y) based on the logarithmic intensity value of the most recent event; Equation (1) outlines the event occurrence model of an ideal sensor.

[0019] Furthermore, the event stream files in step S3 are all periodic, because the target stripe pattern itself is periodic;

[0020] Assuming there is no noise during the operation, consider the noise originating from position s(x) respectively. ref ,y ref Two event streams s(x) and (x,y) of pixels (x,y) ref ,y ref ,t) and s(x,y,t); like s(x,t) ref ,y ref This corresponds to a point in the background, which is used as a reference point; the event flow is periodic, and the period is...

[0021] In order to calibrate the parameters of the event camera, the wrap phase in step S3 must be estimated from the event stream;

[0022] First, the lag point at which the cross-correlation between event streams reaches its maximum value is calculated as the maximum value:

[0023]

[0024] Among them, <·> t Indicates temporal cross-correlation; since the event stream is The period is limited to one period, therefore the cross-correlation calculation is limited to one period. Equation (2) gives the cross-correlation with respect to a fixed reference position s(x). ref ,y ref The time delay of ); the corresponding phase is:

[0025] φ(x,y)=T max (x,y)w0 (3)

[0026] because Therefore, the periodicity of the event flow only gives the wrapping phase, i.e., φ(x,y)∈[0,2π].

[0027] Furthermore, the world coordinate system O w -X w Y w The plane is established on the plane where the LCD screen is located, with the upper left corner of the LCD screen as the origin O. w Z w The axis is perpendicular to the plane of the LCD display screen and points upwards.

[0028] Based on the perspective projection model, any point on the two-dimensional plane is matched with its corresponding point in three-dimensional space. The upper left corner of the LCD screen is the origin O. w Z w The axis is perpendicular to the plane of the LCD display and points upwards. Based on the perspective projection model, any point on the two-dimensional plane is matched with its corresponding point in three-dimensional space. The projection relationship between the camera coordinates and world coordinates of the feature points is as follows:

[0029]

[0030] In this context, the transformation from the world coordinate system to the camera coordinate system involves R as the rotation matrix, T as the translation vector, and the camera intrinsic parameter matrix. Including focal length f x and f y Principal points u0 and v0, γ is the tilt coefficient, representing the non-perpendicularity factor between the u-axis and v-axis; γ for a standard camera is typically 0. [X w ,Y w Z w ] T Represents the world coordinates of the feature point, [u,v] T This represents the coordinates of the feature point in the two-dimensional image.

[0031] Furthermore, the initial values ​​of the event camera parameters in step S7 need to be optimized using a nonlinear optimization algorithm. When optimizing camera parameters, a common strategy is to minimize the camera reprojection error through iteration. The event camera calibration reprojection error is the difference between the pixel coordinates of the feature points located on the calibration image and the pixel coordinates calculated by combining the event camera parameters with the world coordinates of the feature points. The camera parameter optimization model is shown in Equation (5). It is assumed that there are m calibration target images used for camera calibration, and each image contains n feature points.

[0032]

[0033] In the formula, P i,j Let p represent the world coordinates of the j-th feature point in the i-th calibration target image. i,j It is point P i,j The image's pixel coordinates; point This represents the feature point P calculated based on camera parameters. i,j The pixel coordinates, M1 is the camera's intrinsic parameter, M i Let k1, k2, k3, p1, p2 represent the camera extrinsic parameters of the i-th calibration target image, where k1, k2, k3, p1, p2 are the lens distortion coefficients of the camera.

[0034] The beneficial effects of adopting the above technical solutions are:

[0035] 1. The present invention provides an event camera calibration method based on event flow features, which has a simple structure, low manufacturing cost and fast calibration speed.

[0036] 2. The event camera calibration method based on event stream features of the present invention only requires acquiring one event stream file each for the horizontal and vertical stripes at each calibration target pose, without converting them into frame images, to extract the pixel coordinates of the calibration points. This avoids information loss that may be caused by frame image conversion and ensures the reliability of the calibration results.

[0037] 3. The event camera calibration method based on event flow features of this invention simplifies the complexity of calibration calculations. By utilizing the inherent characteristics of event flow to detect feature points, the accuracy and effectiveness of calibration are significantly improved. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the event camera calibration based on event flow characteristics of the present invention;

[0039] Figure 2 It is a phase image encompassing a horizontally oriented striped image;

[0040] Figure 3 It is a phase image encompassing a vertically oriented stripe image;

[0041] Figure 4 It is the horizontal edge line that wraps around the phase image;

[0042] Figure 5 It is the vertical edge line that wraps around the phase image;

[0043] Figure 6 This is a schematic diagram of the calibration points;

[0044] The diagram is labeled as follows: 1-Event camera, 2-LCD display screen, 3-Horizontal moving stripe image, 4-Vertical moving stripe image. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, in order to help those skilled in the art to have a more complete, accurate and in-depth understanding of the concept and technical solutions of the present invention, and to facilitate its implementation.

[0046] like Figures 1 to 6 As shown, this invention is an event camera calibration method based on event stream features, specifically including the following steps:

[0047] Step S1: Set up an event camera 1 calibration system; the system includes an event camera 1 and an LCD display screen 2; place the LCD display screen 2 within the imaging range of the event camera 1, and display a moving stripe pattern on the LCD display screen 2 to trigger an event.

[0048] A high-resolution LCD display screen 2 is preferably used. In a specific embodiment, the event camera 1 is a DVXplorer Lite DXB00127 with a lens focal length of 4-12mm and a resolution of 320×240 pixels. The LCD display screen 2 is an iPad MM9D3CH / A with a resolution of 2360×1640 pixels. During calibration, the LCD display screen 2 is used as a carrier to display a horizontally moving sinusoidal fringe image 3 and a vertically moving sinusoidal fringe image 4 in a time-division manner. The LCD display screen 2 is placed within the imaging range of the event camera 1, and the moving fringe pattern displayed on the LCD display screen 2 triggers the event.

[0049] Step S2: Acquire event stream; LCD display 2 is located directly in front of event camera 1, and alternately displays horizontal moving sine stripe image 3 and vertical moving sine stripe image 4 once each; event camera 1 performs synchronous shooting to acquire one horizontal stripe event stream file and one vertical stripe event stream file respectively.

[0050] Step S21: Whenever the event camera 1 detects a change in the logarithmic intensity of a pixel, it triggers an event and generates an event stream.

[0051] J(x,y,t) represents pixels. and time Logarithmic strength, event e = (e x ,e y ,e t ,e p It will report the spatial location (x, y) = (e x ,e y ), timestamp t = e t This indicates that J(x,y,t) exceeds the threshold, and the polarity e p ∈{+1,-1} indicates whether the change in logarithmic strength is an increase or a decrease.

[0052] Ideally, an event will be triggered when the logarithmic intensity exceeds a pre-set threshold in the hardware δ of event camera 1.

[0053] The i-th event The definition is as follows:

[0054]

[0055] in, R(x,y) is the previous event of the sensor, and R(x,y) is the reference value of pixel (x,y) based on the logarithmic intensity value of the most recent event; Equation (1) outlines the event occurrence model of an ideal sensor.

[0056] Step S3: Perform phase unwrapping processing on each pixel of the horizontal stripe event stream file and the vertical stripe event stream using a time offset. In the implementation scheme, for example... Figure 2 and Figure 3 As shown, the horizontal and vertical wrapped phase images are obtained.

[0057] Step S31: The event stream files in step S3 are all periodic, because the target stripe pattern itself is periodic.

[0058] Assuming there is no noise during the operation, consider the noise originating from position s(x) respectively. ref ,y ref Two event streams s(x) and (x,y) of pixels (x,y) ref ,y ref ,t) and s(x,y,t). Like s(x,t) ref ,y ref This corresponds to a point in the background, which is used as a reference point. The event flow is periodic, with a period of [time period missing].

[0059] In order to calibrate the parameters of event camera 1, the wrap phase in step S3 must be estimated from the event stream.

[0060] First, the lag point at which the cross-correlation between event streams reaches its maximum value is calculated as the maximum value:

[0061]

[0062] Among them, <·> t Indicates temporal cross-correlation; since the event stream is The period is limited to one period, therefore the cross-correlation calculation is limited to one period; Equation (2) gives the cross-correlation with respect to a fixed reference position s(x). ref ,y ref The time delay of ); the corresponding phase is:

[0063] φ(x,y)=T max (x,y)w0 (3)

[0064] because Therefore, the periodicity of the event flow only gives the wrapping phase, i.e., φ(x,y)∈[0,2π].

[0065] Step S4: Perform edge detection on the horizontal and vertical bounding phase images respectively, and fit the detected edges with lines using the least squares method. In the implementation scheme, for example... Figure 4 and Figure 5 As shown, the binary images of the edge lines wrapping the phase image in the horizontal direction and the binary images of the edge lines wrapping the phase image in the vertical direction are obtained.

[0066] Step S5: Superimpose the binary images of the horizontally bounding edge lines and the vertically bounding edge lines of the phase image. In the implementation scheme, for example... Figure 6 As shown, the intersection of the horizontal stripe wrapping the edge line of the phase map and the vertical stripe wrapping the edge line of the phase map is obtained, which is the pixel coordinate of the calibration point.

[0067] Step S6: In the implementation scheme, the LCD display screen 2 performs translational and rotational rigid body movements 3 times. At each new position, steps S2-S5 are repeated to obtain the pixel coordinates of the calibration point at different positions.

[0068] Step S7: Calculate the intrinsic parameters of event camera 1 using the ordinary calibration algorithm based on the pixel coordinates of the calibration point; firstly, solve for the parameters of event camera 1 using the direct linear transformation method, then use the obtained parameters as initial values, take into account the distortion factors of event camera 1 itself, and then optimize the event camera parameter values ​​according to the LM algorithm to obtain the optimized event camera parameter values.

[0069] Step S71: World Coordinate System O w -X w Y w The plane is established on the plane where LCD display 2 is located, with the upper left corner of LCD display 2 as the origin O. w Z w The axis is perpendicular to the plane of the LCD display screen 2 and points upwards.

[0070] Based on the perspective projection model, any point on the two-dimensional plane is matched with its corresponding point in three-dimensional space. The projection relationship between the camera coordinates and world coordinates of the target point is as follows:

[0071]

[0072] In this context, the transformation from the world coordinate system to the camera coordinate system involves R as the rotation matrix, T as the translation vector, and the camera intrinsic parameter matrix. Including focal length f x and f y Principal points u0 and v0, γ is the tilt coefficient, representing the non-perpendicularity factor between the u-axis and v-axis; γ for a standard camera is typically 0. [X w ,Y w Z w ]T Represents the world coordinates of the feature point, [u,v] T This represents the coordinates of the feature point in the two-dimensional image.

[0073] Step S72: In a specific implementation, the influence of lens distortion of the event camera 1 is considered. The most common type is radial distortion, which is typically distributed along the lens radius. It occurs because light rays bend more far from the lens center than near the center, and mainly includes barrel distortion and pincushion distortion. Therefore, the following transformation relationship can be obtained for radial distortion:

[0074] x0=x(1+k1r 2 +k2r 4 +k3r 6 )

[0075] y0=y(1+k1r 2 +k2r 4 +k3r 6 );

[0076] In the formula, r 2 =x 2 +y 2 x0 and y0 are the camera coordinates after distortion correction, while x and y are the camera coordinates before distortion correction. k1, k2, and k3 are the three radial distortion parameters, which are the other three intrinsic parameters of the camera.

[0077] Step S73: The above obtains the initial values ​​of the parameters of event camera 1. The parameters need to be optimized using the LM algorithm. A common strategy for camera parameter optimization is to minimize the camera reprojection error through iteration. The event camera calibration reprojection error is the difference between the pixel coordinates of the feature points located on the calibration image and the pixel coordinates calculated by combining the event camera parameters with the world coordinates of the feature points. The camera parameter optimization model is shown in Equation (5). It is assumed that there are m calibration target images used for camera calibration, and each image contains n feature points.

[0078]

[0079] In the formula, P i,j Let p represent the world coordinates of the j-th feature point in the i-th calibration target image. i,j It is point P i,j The pixel coordinates of the image on the image. Point This represents the feature point P calculated based on camera parameters. i,j The pixel coordinates, M1 is the camera's intrinsic parameter, M i Let k1, k2, k3, p1, p2 represent the camera extrinsic parameters of the i-th calibration target image, where k1, k2, k3, p1, p2 are the lens distortion coefficients of the camera.

[0080] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without modification, are all within the protection scope of the present invention.

Claims

1. An event camera calibration method based on event flow features, characterized in that: Specifically, it includes the following steps: Step S1: Set up an event camera (1) calibration system; the system includes an event camera (1) and an LCD display screen (2); place the LCD display screen (2) within the imaging range of the event camera (1), and display a moving stripe pattern on the LCD display screen (2) to trigger the event; Step S2: Acquire event stream; The LCD screen (2) is located in front of the event camera (1), and alternately displays the horizontal moving sine stripe image (3) and the vertical moving sine stripe image (4) once each. The event camera (1) performs synchronous shooting to acquire one horizontal stripe event stream file and one vertical stripe event stream file respectively. Step S3: Perform phase unwrapping processing on each pixel of the horizontal stripe event stream and the vertical stripe event stream files using the time offset to obtain the horizontal wrapped phase image and the vertical wrapped phase image. Step S4: Perform edge detection on the horizontal and vertical wrapped phase images respectively, and use the least squares method to fit the detected edge parts with straight lines to obtain the binary image of the edge line of the horizontal wrapped phase image and the binary image of the edge line of the vertical wrapped phase image. Step S5: Superimpose the binary images of the horizontal and vertical edges of the phase image to obtain the intersection of the horizontal and vertical stripes of the phase image edge lines, which is the pixel coordinate of the calibration point. Step S6: The LCD display screen (2) performs translational and rotational rigid body motion N times, where N is an integer and N≥3; at each new position, repeat steps S2-S5 to obtain the pixel coordinates of the calibration point at different positions; Step S7: Calculate the intrinsic parameters of the event camera (1) using the ordinary calibration algorithm for the pixel coordinates of the calibration point; firstly, solve the parameters of the event camera (1) using the direct linear transformation method, then use the obtained parameters as initial values, and then optimize the event camera parameter values ​​according to the LM algorithm to obtain the optimized event camera (1) parameter values. In step S2, the event camera (1) will trigger an event and generate an event stream whenever it detects a change in the logarithmic intensity of a pixel. Representing pixels and time Logarithmic strength, event It will report spatial location timestamp show Exceeding the threshold, polarity Indicates whether the change in logarithmic strength is increasing or decreasing; Ideally, when the logarithmic intensity exceeds the event camera (1) hardware When the preset threshold is reached, the event will be triggered; The i-th event The definition is as follows: (1) in, It is the previous event from the sensor. It is a pixel The reference value is based on the logarithmic intensity value of the most recent event; Formula (1) outlines the event occurrence model of the ideal sensor.

2. The event camera calibration method based on event flow features according to claim 1, characterized in that: The event stream files in step S3 are all periodic because the target stripe pattern itself is periodic. Assuming there is no noise during the operation, consider the noise originating from the location. and Two event streams of a pixel and ;picture This corresponds to a point in the background, which is used as a reference point; The event stream is periodic, and the period is [time period missing]. ; In order to calibrate the parameters of the event camera (1), the wrap phase in step S3 must be estimated from the event stream; First, the lag point at which the cross-correlation between event streams reaches its maximum value is calculated as the maximum value: (2) in, Indicates temporal cross-correlation; since the event stream is The period is limited to one period, therefore the cross-correlation calculation is limited to one period; Formula (2) gives the period relative to a fixed reference position. The time delay; the corresponding phase is: (3) because Therefore, the periodicity of the event flow only gives the package phase, i.e. .

3. The event camera calibration method based on event flow features according to claim 1, characterized in that: The world coordinate system The plane is established on the plane where the LCD display (2) is located, with the upper left corner of the LCD display (2) as the origin of the coordinate system. , The axis is perpendicular to the plane of the LCD display (2) and pointing upwards; Based on the perspective projection model, information matching is performed between any point on the two-dimensional plane and its corresponding point in three-dimensional space. The projection relationship between the camera coordinates and world coordinates of the feature points is as follows: (4) Among them, the transformation from the world coordinate system to the camera coordinate system, For rotation matrix, The translation vector is the camera intrinsic parameter matrix. Including focal length and Main point and , The slope coefficient represents... shaft and The non-perpendicularity factor of the axis, for standard cameras It is usually 0. Represents the world coordinates of the feature points. This represents the coordinates of the feature point in the two-dimensional image.

4. The event camera calibration method based on event flow features according to claim 1, characterized in that: In step S7, the initial values ​​of the event camera parameters are optimized using the LM algorithm. When optimizing the camera parameters, the reprojection error of the camera is minimized through iteration. The reprojection error of the event camera (1) calibration is the difference between the pixel coordinates of the feature point located on the calibration image and the pixel coordinates calculated by combining the event camera (1) parameters with the world coordinates of the feature point. The camera parameter optimization model is shown in Equation (5). It is assumed that there are m calibration target images used for camera calibration, and each image contains n feature points. (5) In the formula, This represents the world coordinates of the j-th feature point in the i-th calibration target image. It is a point The image's pixel coordinates; point Indicates feature points calculated based on camera parameters. pixel coordinates, These are camera-in-camera parameters. This represents the camera extrinsic parameters of the i-th calibration target image. This is the lens distortion coefficient of the camera.

Images