Event camera high-precision calibration method and system for space situation awareness

Abstract

The invention discloses an event camera high-precision calibration method for space situation awareness. The event camera high-precision calibration method comprises the following steps: establishing a combined calibration frame of a fixed star simulator, a high-precision turntable and an event camera; controlling the high-precision turntable to rotate, collecting event flow information generated by the relative motion point light source, and obtaining a gray value of the latticed track image; extracting a connected domain, counting crossed overlapped pixels and pixel overlapped areas in the horizontal direction and the vertical direction, and calculating a coarse centroid position; determining a space window and a time window, determining a centroid position based on a time interpolation gray centroid positioning algorithm, and performing linear fitting on the centroid position to determine a trajectory equation; calculating coarse centroid positions of all cross points and a trajectory equation of each space window, and calculating a sub-pixel centroid position based on the vertical trajectory equation; and the internal and external parameters of the event camera are optimally solved based on the corresponding relation between the sub-pixel centroid position and the rotary table angle. The invention further discloses a corresponding system, electronic equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of space situational awareness and high-precision calibration methods for event cameras, and in particular to a high-precision calibration method and system for event cameras used for space situational awareness. Background Art

[0002] Space situational awareness is a comprehensive process covering the acquisition and processing of space target information, situational awareness and comprehensive monitoring of the space environment. It is the basis for ensuring space security and the smooth execution of various space missions. With the rapid development and widespread application of space technology, the space environment is becoming increasingly crowded, and competition and confrontation have significantly increased. The rise of large-scale low-orbit constellations, the complexity of space system functions, and the frequent occurrence of space events have made the space security environment more complex, changeable and full of uncertainty. As a result, higher requirements are placed on tasks such as rapid and accurate monitoring and identification of dynamic targets in space.

[0003] Event cameras, as a sensor inspired by biological vision mechanisms, abandon the traditional camera's way of capturing images at a fixed frame rate. Its core output is a series of event streams. The event information includes the timestamp, spatial position and direction of change (increase or decrease) of the brightness change. Compared with traditional cameras, event cameras show significant advantages, including high temporal resolution, high dynamic range, low latency and low energy consumption. These features effectively make up for the shortcomings of traditional cameras in specific application scenarios (such as space situational awareness SSA). In order to achieve accurate recognition, rapid positioning and continuous monitoring of space targets, a core challenge is the precise calibration of camera parameters.

[0004] There are two main methods for event camera calibration. One is to use an LED screen of known size to dynamically generate a flashing checkerboard pattern or a synchronously flashing LED calibration board. These dynamic visual stimuli can be effectively captured by the event camera and identified by it as feature points similar to those in traditional camera calibration, thereby achieving the calibration of the event camera. Another strategy is to keep the calibration target stationary and collect the event stream generated by the target pattern by moving the event camera. On this basis, the image grayscale reconstruction technology based on the event stream is used, or features are efficiently extracted directly from the event stream data. Once the feature information related to the target is successfully extracted from the event stream, the traditional camera calibration method can be used to complete the calibration of the event camera.

[0005] There is a certain gap between the calibration accuracy of the above methods and the space surveillance performance requirements of traditional space-based visible light sensors. Therefore, in order to ensure that the event camera can achieve the same measurement accuracy as traditional cameras in space situational awareness applications, a new calibration method needs to be proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a high-precision calibration method and system for event cameras for space situation awareness. Different from the calibration method of existing event cameras using a checkerboard or similar target, the present invention uses a calibration framework of a star simulator and a high-precision turntable to meet the task of accurately and quickly monitoring space targets in space situation awareness by event cameras. First, according to the characteristics of asynchronous imaging of event cameras, a dynamic calibration trajectory acquisition strategy is adopted to obtain a large number of calibration points in a simple and efficient manner. Subsequently, the sub-pixel centroid position of the calibration point is determined in three steps. In the first step, the rough centroid of the calibration point is determined according to the overlapping pixels intersecting the grid-like trajectory; in the second step, the spatial window and the time window are determined by the rough centroid position, and the centroid position of each time step is determined in the time window using the time interpolation grayscale centroid positioning algorithm, and the trajectory equation is determined in the spatial window using the linear fitting technology; in the third step, the sub-pixel position of the intersection is obtained according to the mutually perpendicular trajectory equations as the sub-pixel position of the calibration point. Finally, the internal and external parameters are optimized and solved according to the one-to-one correspondence between the calibration point and the turntable angle. Comprehensive experimental results show that the calibration accuracy of the proposed method is comparable to that of traditional cameras, and is significantly improved over the current event camera calibration method, and can meet the requirements of space target detection accuracy.

[0007] A first aspect of the present invention is to provide a high-precision calibration method for an event camera for space situational awareness, comprising:

[0008] S1, establish a combined calibration framework of the star simulator, high-precision turntable and event camera;

[0009] S2, controls the rotation of the high-precision turntable, collects the event stream information generated by the relative motion point light source, performs time aggregation on the event stream, and obtains the grayscale value of the grid-shaped trajectory image;

[0010] S3, extracting the connected domain where the grid-shaped trajectory is located, counting the intersecting overlapping pixels and the pixel overlapping areas in the horizontal and vertical directions in the grid-shaped trajectory, and calculating the rough centroid position of the calibration point based on the pixel overlapping area;

[0011] S4, determining a spatial window according to the position of the rough centroid, determining a time window within the spatial window, dividing time into multiple time steps with equal steps within the time window, and determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and performing linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates;

[0012] S5, calculating the rough centroid positions of all intersections in the grid-shaped trajectory and the trajectory equation in each spatial window, and then calculating the positions of the intersections based on the mutually perpendicular trajectory equations, and using them as the sub-pixel centroid positions of the calibration points;

[0013] S6, optimizing the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

[0014] Preferably, the combined calibration framework of S1 includes a hardware system and an imaging system, wherein:

[0015] The core components of the hardware system include a star simulator, a high-precision turntable and an event camera; wherein the star simulator is fixed on a marble platform and emits a collimated point light source, and the event camera is installed on the inner frame of the high-precision turntable to sense the light source; the middle frame and the outer frame of the high-precision turntable rotate at a constant speed to adjust the angle respectively, changing the position of the point light source on the image plane of the event camera, and there is a unique corresponding relationship between the centroid position of the point light source and the angle of the middle frame and the outer frame of the high-precision turntable;

[0016] The imaging system comprises an imaging mechanism and an imaging model, wherein:

[0017] (1) The imaging mechanism includes: In the event camera, each pixel continuously monitors the logarithm of the light intensity Once the change exceeds the preset threshold C, the corresponding pixel An event will be triggered immediately Specifically, as shown in formula (1):

[0018]

[0019] Among them, Δt k is pixel x k The time since the last event, polarity p k ∈{+1, -1} is the sign of brightness change;

[0020] (2) The imaging model includes: the propagation process of the parallel light vector emitted by the star simulator through the event camera lens on the image plane, and the vector V is the coordinate system X of the turntable. r -Y r -Z r The parallel light vector emitted by the star simulator under is expressed as shown in equation (2), where α and β are the pitch angle and yaw angle of vector V;

[0021]

[0022] Rotation matrix R rot It is obtained by synthesizing according to the different rotation angles of the turntable; R re is the mounting matrix between the event camera and the turntable, the event camera coordinate system X e -Y e -Z eThe expression of the parallel light vector V′ under φ is shown in formula (3), where: and are the rotation angles from the turntable coordinate system to the event camera coordinate system, θ x ,θ y They are the angles at which the turntable rotates the middle frame and the outer frame;

[0023]

[0024] According to the pinhole imaging model, the projection position (u′, v′) of the parallel light vector on the pixel coordinate system uv is shown in equation (4), where s is the scale factor, F is the lens focal length, d is the pixel size, and u0 and v0 are the principal point positions of the pixel coordinate system respectively;

[0025]

[0026] On this basis, lens distortion is added, where q1 and q2 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients; du and dv are the distortion magnitudes of the lens along the u and v directions of the pixel coordinate system, respectively. The final projection position (u, v) is shown in formula (5):

[0027]

[0028] By integrating formulas (2)-(5), we can establish the pixel coordinate position (u, v) and the rotation angle (θ x ,θ y ) have a one-to-one correspondence between them; based on this clear correspondence, a unified calibration algorithm for internal and external parameters is used to calibrate 13 key parameters F, d, u0, v0, q1, q1, p1, p2, α, β, Perform optimization solution; the logic of the unified calibration algorithm for internal and external parameters is shown in formula (6):

[0029]

[0030] Preferably, S2 includes:

[0031] S21, driving the high-precision turntable to rotate the outer frame and the middle frame respectively;

[0032] S22, the event camera obtains the event flow information generated by the relative motion point light source based on the asynchronous imaging principle, counts the number of positive events generated by each pixel, uses this as the intensity value of the pixel, and obtains the grayscale value of the grid trajectory image.

[0033] Preferably, S3 includes:

[0034] S31, performing noise filtering on the grid-shaped trajectory image, and counting the maximum connected domain in the image plane based on the nearest connected domain algorithm to extract the connected domain where the grid-shaped trajectory is located;

[0035] S32, acquiring pixels related to the motion trajectory of the point light source, counting the overlapping pixels intersecting in the grid-shaped trajectory, and counting the pixel overlapping areas in the horizontal and vertical motion trajectories;

[0036] S33, calculating the centroid position of the pixel overlap area based on the geometric center method, and using it as the rough centroid position of the calibration point.

[0037] Preferably, S4 includes:

[0038] S41, selecting a fixed spatial window according to the neighboring area of ​​the pixel where the rough centroid is located;

[0039] S42, sorting the positive events generated by all pixels in the window according to the time axis, determining the start time and the end time as the time window, and dividing the time period into multiple time steps;

[0040] S43, determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm;

[0041] S44, within the spatial window, linear fitting is performed on the center of mass position to determine the trajectory equation of the group of center of mass coordinates.

[0042] Preferably, S5 includes:

[0043] S51, calculating the rough centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window;

[0044] S52, solving a two-variable linear equation based on the mutually perpendicular trajectory equations, and the calculated coordinates are the positions of the intersection points, which are used as the sub-pixel centroid positions of the calibration points corresponding to each set of mutually perpendicular trajectories.

[0045] Preferably, S6 includes:

[0046] S61, making one-to-one correspondence between the sub-pixel centroid position of the calibration point and the position of the turntable outer frame and the middle frame;

[0047] S62, solving the internal and external parameter coefficients of the event camera based on a unified internal and external parameter calibration method, wherein the internal and external parameters of the event camera include: internal parameters including focal length, principal point and distortion coefficient of the event camera, and external parameters including the installation matrix of the event camera coordinate system and the turntable coordinate system and the stellar simulator transmitting star vector in the turntable coordinate system.

[0048] The second aspect of the present invention further provides a high-precision calibration system for event cameras for space situational awareness, which is used to implement the method of the first aspect, including:

[0049] Combined calibration framework building module, used to build a combined calibration framework of the star simulator, high-precision turntable and event camera;

[0050] The grid trajectory acquisition module is used to control the rotation of the high-precision turntable, collect the event stream information generated by the relative motion point light source, perform time aggregation on the event stream, and obtain the grayscale value of the grid trajectory image;

[0051] A coarse centroid position extraction module is used to extract the connected domain where the grid-shaped trajectory is located, count the intersecting coincident pixels in the grid-shaped trajectory and the pixel coincidence areas in the horizontal and vertical directions, and calculate the coarse centroid position of the calibration point based on the pixel coincidence areas;

[0052] A trajectory equation determination module is used to determine a spatial window according to the position of the rough centroid, determine a time window within the spatial window, divide the time into multiple time steps with equal steps within the time window, determine the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and perform linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates;

[0053] A sub-pixel centroid position extraction module, used to calculate the coarse centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window, and then calculate the positions of the intersections based on the mutually perpendicular trajectory equations, and use them as the sub-pixel centroid positions of the calibration points;

[0054] The parameter optimization solution and high-precision calibration module is used to optimize the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

[0055] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method described in the first aspect.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and execute the method described in the first aspect.

[0057] Beneficial effects of the method and system of the present invention:

[0058] (1) A calibration framework based on a star simulator and a high-precision turntable is used to meet the task of accurately and quickly monitoring space targets in space situational awareness by event cameras.

[0059] (2) Based on the characteristics of asynchronous imaging of event cameras, a dynamic calibration trajectory acquisition strategy is adopted to obtain a large number of calibration points in a simple and efficient manner.

[0060] (3) The calibration accuracy of this method is comparable to that of traditional cameras and is significantly improved over the current event camera calibration method, thus meeting the requirements for space target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 (a) is a schematic diagram of a process flow of a high-precision calibration method for an event camera for space situational awareness according to an embodiment of the present invention;

[0063] Figure 1 (b) is a schematic diagram of the software process of the high-precision calibration method for event cameras for space situational awareness provided according to an embodiment of the present invention, excluding the combined calibration framework building part;

[0064] Figure 2 A hardware system structure diagram of a calibration framework used according to an embodiment of the present invention;

[0065] Figure 3 The imaging model of an event camera in an imaging system provided according to an embodiment of the present invention represents the imaging process of an event camera receiving a parallel light vector in a world coordinate system to a pixel coordinate system;

[0066] Figure 4 A schematic diagram of a calibration trajectory acquisition strategy provided according to an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of a rough extraction process of calibration points provided according to an embodiment of the present invention;

[0068] Figure 6 A schematic diagram of the principle of an event-based time linear interpolation grayscale centroid positioning algorithm provided according to an embodiment of the present invention;

[0069] Figure 7Schematic diagram of sub-pixel coordinate extraction according to an embodiment of the present invention. The shaded area is the pixel where the rough centroid is located, the gray area is the effective pixel, the horizontal and vertical dots are the sub-pixel centroid positions at different times in the horizontal and vertical space windows, respectively, and the straight line is the linear fitting trajectory;

[0070] Figure 8 A schematic diagram of a calibration grid trajectory provided according to an embodiment of the present invention;

[0071] Fig. 9 A schematic diagram of the centroid position of the calibration points provided according to an embodiment of the present invention;

[0072] Fig.10 A schematic diagram of a calibration point residual result provided according to an embodiment of the present invention;

[0073] Fig.11 A schematic diagram of residual results of test points provided according to an embodiment of the present invention;

[0074] Fig.12 A structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0077] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0078] Embodiment 1

[0079] like Figure 1 As shown, this embodiment provides a high-precision calibration method for an event camera for space situational awareness, including:

[0080] S1, establish a combined calibration framework of the star simulator, high-precision turntable and event camera;

[0081] S2, controls the rotation of the high-precision turntable, collects the event stream information generated by the relative motion point light source, performs time aggregation on the event stream, and obtains the grayscale value of the grid-shaped trajectory image;

[0082] S3, extracting the connected domain where the grid-shaped trajectory is located, counting the intersecting overlapping pixels and the pixel overlapping areas in the horizontal and vertical directions in the grid-shaped trajectory, and calculating the rough centroid position of the calibration point based on the pixel overlapping area;

[0083] S4, determining a spatial window according to the position of the rough centroid, determining a time window within the spatial window, dividing time into multiple time steps with equal steps within the time window, and determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and performing linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates;

[0084] S5, calculating the rough centroid positions of all intersections in the grid-shaped trajectory and the trajectory equation in each spatial window, and then calculating the positions of the intersections based on the mutually perpendicular trajectory equations, and using them as the sub-pixel centroid positions of the calibration points;

[0085] S6, optimizing the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

[0086] Implementation principle of steps S1-S6:

[0087] Due to the unique imaging mechanism of the event camera, the core of its high-precision calibration method lies in two parts: calibration data acquisition and sub-pixel calibration point centroid positioning. This embodiment uses a high-precision turntable with a mature framework of a star simulator to collect calibration data. In the traditional camera calibration method, the camera usually performs timed exposure sampling on the light source while the turntable and the star simulator remain relatively still. However, without taking other measures to change the brightness of the light source, the event camera will not be able to output event information for the light source. Therefore, in order to integrate the event camera calibration into this framework, the present invention proposes for the first time a simple and efficient high-precision turntable dynamic calibration trajectory acquisition strategy, which extracts the centroid position of the calibration point through coarse extraction and local linear sub-pixel extraction. Among them, the original time interpolation grayscale centroid positioning algorithm can calculate the centroid position of point light sources at different times by processing the event time information.

[0088] As a preferred implementation, the combined calibration framework of S1 includes a hardware system and an imaging system, wherein the imaging system includes an imaging mechanism and an imaging model.

[0089] In this embodiment, Figure 2 Shown is the hardware system in the calibration framework adopted by the present invention. The core components of the hardware system include a star simulator, a high-precision turntable and an event camera. Among them, the star simulator is fixed on a marble platform and emits a collimated point light source. The event camera is installed on the inner frame of the high-precision turntable to sense the light source. The design accuracy of the high-precision turntable reaches 0.35″(3σ), where the frame and the outer frame can be rotated at a uniform speed to adjust the angle respectively, changing the position of the point light source on the image plane of the event camera. There is a unique corresponding relationship between the center of mass position of the point light source and the angle of the middle frame and the outer frame of the turntable, which provides a reference benchmark for the accuracy of subsequent calibration experiments.

[0090] In this embodiment, the imaging system includes an imaging mechanism and an imaging model, wherein:

[0091] (1) Imaging mechanism

[0092] An event camera is a bio-inspired visual sensor that works very differently from a standard camera. In an event camera, each pixel acts as an independent photodetector, continuously monitoring the logarithm of light intensity. Once this change exceeds the preset threshold C, the corresponding pixel An event will be triggered immediately The details are shown in formula (1).

[0093]

[0094] Among them, Δt k is pixel x k The time since the last event, polarity p k ∈{+1, -1} is the sign of brightness change.

[0095] It is particularly worth pointing out that, given that the research background of the present invention is set under the extremely dark light conditions of the space environment, the detection delay of negative events (i.e., brightness reduction events) in this special environment reaches a level that is difficult to ignore. According to experimental observations, for the same pixel, in the event stream generated by the motion of a 2Mv star, the time interval between two negative events is more than a hundred times that between two positive events. There is currently no relevant literature to explain this phenomenon. Therefore, in order to ensure the accuracy of the experimental data and the effectiveness of the analysis, the research scope of the present invention focuses on the observation and analysis of positive events (i.e., brightness increase events), and temporarily ignores the impact of changes in negative events.

[0096] (2) Imaging model

[0097] like Figure 3 The figure shows the imaging model of the event camera in the imaging system. The imaging model of the event camera describes the imaging process of the parallel light vector received by the event camera in the world coordinate system to the pixel coordinate system.

[0098] This embodiment constructs the propagation process of the coordinate position of the parallel light vector emitted by the star simulator through the event camera lens on the image plane. Among them, the conversion relationship between the coordinate systems is given by Figure 3 Explanation. Vector V is the turntable coordinate system X r -Y r -Z r The parallel light vector emitted by the star simulator under is expressed as shown in formula (2), where α and β are the pitch angle and yaw angle of vector V.

[0099]

[0100] Rotation matrix R rot It can be obtained by synthesizing the different rotation angles of the turntable. re is the mounting matrix between the event camera and the turntable, the event camera coordinate system X e -Y e -Z e The expression of the parallel light vector V′ under the condition is shown in equation (3). and are the rotation angles from the turntable coordinate system to the event camera coordinate system. x ,θ y They are the angles at which the turntable rotates the middle frame and outer frame respectively.

[0101]

[0102] According to the pinhole imaging model, the projection position (u′, v′) of the parallel light vector on the pixel coordinate system uv is shown in equation (4). Where s is the scale factor, F is the lens focal length, d is the pixel size, and u0 and v0 are the principal point positions of the pixel coordinate system.

[0103]

[0104] In addition, lens distortion needs to be added on this basis, where q1 and q2 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients. du and dv are the distortion magnitudes of the lens along the u and v directions of the pixel coordinate system, respectively. The final projection position (u, v) is shown in formula (5).

[0105]

[0106] By integrating formulas (2)-(5), we can establish the pixel coordinate position (u, v) and the rotation angle (θ x ,θ y ) have a one-to-one correspondence. Then, based on this clear correspondence, a unified calibration algorithm for internal and external parameters is used to calibrate the 13 key parameters F, d, u0, v0, q1, q1, p1, p2, α, β, The optimization solution is performed. The logic of the unified calibration algorithm for internal and external parameters is shown in formula (6).

[0107]

[0108] As a preferred embodiment, S2 includes:

[0109] S21, driving the high-precision turntable to rotate the outer frame and the middle frame respectively;

[0110] S22, the event camera obtains the event flow information generated by the relative motion point light source based on the asynchronous imaging principle, counts the number of positive events generated by each pixel, uses this as the intensity value of the pixel, and obtains the grayscale value of the grid trajectory image.

[0111] In this embodiment, the middle frame and the inner frame of the turntable are kept stationary, and the outer frame of the turntable is started and controlled to rotate at a constant speed at a preset fixed angular velocity. The parallel light source emitted by the star simulator forms a continuous and stable movement on the imaging plane of the event camera, thereby drawing a trajectory that is approximately a straight line. Similarly, in order to obtain the calibration trajectory in another direction (vertical direction), the middle frame of the turntable needs to be set as the rotation object, while keeping the outer frame and the inner frame stationary. The middle frame also rotates at a preset fixed angular velocity to generate a new trajectory that is perpendicular to the previous trajectory, as shown in the schematic diagram. Figure 4 As shown in the figure. In order to construct a dense calibration grid, the above calibration trajectory acquisition process is repeated. By adjusting the rotation starting position or angle, M horizontal trajectories and N vertical trajectories with equal spacing are generated in the imaging area of ​​the event camera. These trajectories intersect with each other to form M×N precise calibration intersections. Each calibration point corresponds to a unique angle value of the turntable, which provides rich data support for subsequent camera parameter calibration. This acquisition method is simple and efficient. As the number of trajectories in different directions increases, the number of calibration points collected increases in O(MN), which greatly improves the calibration efficiency.

[0112] As a preferred implementation, S3 includes:

[0113] S31, performing noise filtering on the grid-shaped trajectory image, and counting the maximum connected domain in the image plane based on the nearest connected domain algorithm to extract the connected domain where the grid-shaped trajectory is located;

[0114] S32, acquiring pixels related to the motion trajectory of the point light source, counting the overlapping pixels intersecting in the grid-shaped trajectory, and counting the pixel overlapping areas in the horizontal and vertical motion trajectories;

[0115] S33, calculating the centroid position of the pixel overlap area based on the geometric center method, and using it as the rough centroid position of the calibration point.

[0116] In this embodiment, step S3 involves a rough extraction step in the extraction of calibration points, such as Figure 5 The rough extraction process flow chart of the calibration point shown in the figure first performs time aggregation on the event stream generated by the moving point light source, that is, the number of positive events generated by each pixel is counted, which is used as the intensity value of the pixel. In actual tests, there is often a large amount of background noise. In the embodiment of the present invention, the pixels with a number of positive events not exceeding 2 are regarded as events caused by background noise, and the event stream stimulated by the movement of the point light source is retained. Subsequently, in order to further determine the pixels passed by the point light source, the nearest connected domain algorithm is used to extract the maximum connected domain of the image, and the pixels closely related to the motion trajectory of the point light source are obtained. Finally, the pixel overlap area in the horizontal and vertical motion trajectories is counted, and the center of mass position (u, v) of the area is calculated using the geometric center method, and it is used as the preliminary center of mass position of the calibration point.

[0117] As a preferred implementation, the S4 includes:

[0118] S41, selecting a fixed spatial window according to the neighboring area of ​​the pixel where the rough centroid is located;

[0119] S42, sorting the positive events generated by all pixels in the window according to the time axis, determining the start time and the end time as the time window, and dividing the time period into multiple time steps;

[0120] S43, determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm;

[0121] S44, within the spatial window, linear fitting is performed on the center of mass position to determine the trajectory equation of the group of center of mass coordinates.

[0122] In this embodiment, Figure 6 The figure shows the principle diagram of the time interpolation grayscale centroid positioning algorithm. From theoretical analysis and experimental tests, it is known that when a point light source enters and leaves a pixel, it will first generate a series of positive events and then a series of negative events. The moment when the last positive event and the first negative event are generated is the moment when the pixel is closest to the center plane of the point light source. According to this process, an event-based time interpolation grayscale centroid positioning algorithm is designed. Under extremely dark light conditions, negative events have extremely high delays, resulting in the unavailability of negative events, and it is difficult to determine the moment t when the energy center of the light spot coincides with the center of the pixel.mid Therefore, an assumption is made in the algorithm that for a positive event time series {t i ,i=1,2,...,k}, it is assumed that the last positive event occurs at time t k is the moment when the center plane of the point light source coincides with the center of the pixel, and it is assumed that the time of the last negative event is t k+1 =2t k -t1, for a certain time t id , when t k <t id <t k+1 When t id =2t k -t id . Statistical time t id In the event sequence range of the pixels in the selected spatial window, determine whether the straight-line distance between these pixels is less than d. If the condition is met, record the pixel position {(u i ,v i ),i=1,2,...,l}.

[0123] Judgment time t id The position in the event sequence, if t i <t id ≤t i+1 , then id = i; if t id = t1, then id = 0. Here, it is assumed that the background noise intensity of all pixels is the same and is set to a value of 1. According to the linear interpolation method, the time t id The logarithmic value of the pixel grayscale at id (i), as shown in formula (7):

[0124]

[0125] Use formula (7) to calculate the grayscale values ​​of l pixels that meet the conditions, and calculate the time t according to the grayscale centroid algorithm id The centroid position of the point light source at time (u id ,v id ), as shown in formula (8):

[0126]

[0127] As a preferred implementation, S5 includes:

[0128] S51, calculating the rough centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window;

[0129] S52, solving a two-variable linear equation based on the mutually perpendicular trajectory equations, and the calculated coordinates are the positions of the intersection points, which are used as the sub-pixel centroid positions of the calibration points corresponding to each set of mutually perpendicular trajectories.

[0130] In this embodiment, step S5 involves local linear sub-pixel extraction in calibration point extraction, and step S3 completes the preliminary extraction of the centroid of the calibration point. Based on this, the sub-pixel centroid position of the calibration point needs to be further calculated in step S5. To this end, step S4 of this embodiment uses a time interpolation grayscale centroid positioning algorithm to calculate the centroid position of the point light source motion trajectory at different times. Subsequently, a series of centroid positions are linearly fitted to solve the trajectory equation, and the sub-pixel centroid position of the calibration point is obtained by calculating the intersection of mutually perpendicular trajectory equations. However, lens distortion has a nonlinear effect on the motion trajectory of the point light source. Therefore, this embodiment selects a fixed spatial window in the vicinity of the pixel where the coarse centroid is located, and considers that the motion trajectory of the point light source in the window is approximately a straight line. Step S3 determines whether the pixel is a noise signal based on the positive event generated by the pixel, without considering whether the noise signal is contained in a series of events generated by the pixel. Next, the event information generated by each pixel in the selected spatial window is analyzed, and judged one by one according to the time before and after the event occurs. A time threshold th is set as the judgment criterion. If the time interval between two adjacent positive events exceeds the threshold, the first event is considered to be inconsistent with the continuity requirement of the calibration trajectory and is removed. This process is iterated until a pair of adjacent events is found, the time interval between them is less than the threshold th, and these two positive events are considered to be the first two valid events of the pixel excited by the motion of the point light source.

[0131] Then, all positive events generated by all pixels after filtering noise in the spatial window are sorted according to the time axis, and the time t of the first event is recorded. s and the last event occurred at time t e , and divide the time period into equal time periods to obtain the time series {t n}, the division method is shown in formula (9). Using the time interpolation grayscale centroid positioning algorithm, the time series {t n}, forming a set of centroid coordinates {(u n ,v n )}, and obtain the trajectory equation v=ku+b of the group of centroid coordinates according to the linear fitting method.

[0132]

[0133] like Figure 7 As shown in the figure, a region with a length of m and a width of n pixels is selected in the horizontal and vertical directions as a spatial window, and the time series {t h}、{tv}, and then calculate the trajectory equations v=k in the horizontal and vertical directions h u+b h 、v=k v u+b v . This method is used to calculate the horizontal trajectory equation v = k in the M×N intersection neighboring spatial window. h,i u+b h,i (i=1,2,...M×N) and vertical trajectory equation v=k v,i u+b v,i (i=1,2,...M×N), the horizontal and vertical trajectory equations of the spatial window related to each intersection constitute a set of two-variable linear equations, with a total of M×N equations. The sub-pixel centroid positions of all calibration points are obtained by solving the equations, as shown in equation (10).

[0134]

[0135] As a preferred implementation, S6 includes:

[0136] S61, making one-to-one correspondence between the sub-pixel centroid position of the calibration point and the position of the turntable outer frame and the middle frame;

[0137] S62, solving the internal and external parameter coefficients of the event camera based on a unified internal and external parameter calibration method, wherein the internal and external parameters of the event camera include: internal parameters including focal length, principal point and distortion coefficient of the event camera, and external parameters including the installation matrix of the event camera coordinate system and the turntable coordinate system and the stellar simulator transmitting star vector in the turntable coordinate system.

[0138] In this embodiment, based on the sub-pixel coordinates (u i ,v i ) and the corresponding turntable angle The information is used to optimize the internal and external parameters of formula (6) in a unified manner, and the internal parameters such as the focal length, principal point, distortion coefficient of the event camera, the installation matrix between the event camera coordinate system and the turntable coordinate system, and the external parameters such as the stellar simulator launch star vector in the turntable coordinate system are solved.

[0139] Application Examples

[0140] In the calibration trajectory collection, the outer frame angle range is set to (5.5°, -4.5°), the span is 2°, and a total of 6 trajectories are collected; the turntable middle frame angle range is (-8.5°, 9.5°), the span is 3°, and a total of 7 trajectories are collected. Aggregate 13 trajectories into one image as shown in Figure 8 As shown, a total of 42 intersection points are obtained. The position of the centroid calculated by the method of this embodiment on the image plane is as follows: Fig. 9 As shown in Table 1, the internal and external parameters of the event camera are calibrated using a unified internal and external parameter calibration method.

[0141] Table 1 Calibration results

[0142]

[0143] Fig.10 The magnitude and direction of the residual error in the focal plane show a significant random distribution characteristic. This observation directly points to an important conclusion: the system error has been effectively suppressed to a minimum through the calibration process. In order to quantify and evaluate the accuracy of the calibration process, the reprojection error is introduced as a core evaluation indicator. The calculated reprojection error values ​​are 0.0253 pixels in the x-axis direction and 0.0292 pixels in the y-axis direction, which are converted into equivalent angular errors of 1.55 arc seconds and 1.79 arc seconds respectively.

[0144] In order to verify the calibration results, another set of mesh trajectories was collected for testing. The outer frame angle was set to (5°, -4°), the span was 1.5°, and there were 7 trajectories in total; the turntable middle frame angle was set to (-6.5°, 7.5°), the span was 2°, and there were 8 trajectories in total, resulting in a total of 56 test points. Using the turntable position and the calibration parameters in Table 1, the center of mass position of the test point is calculated as a reference value. The test value of the center of mass position of the test point is obtained by the method of the first aspect. The accuracy of the calibration parameters is evaluated by the residual values ​​of the two sets of center of mass positions. The calculation results are as follows: the residual in the x-axis direction is 0.0250 pixels, and the residual in the y-axis direction is 0.0301 pixels. The equivalent angle errors of the corresponding pitch and yaw are 1.53 arc seconds and 1.84 arc seconds, respectively. The directions and sizes of the residuals of the specific test points are as follows. Fig.11 As shown in Figure 2, the residual amplitude and direction are randomly distributed. The above analysis not only verifies the reliability of the calibration parameters, but also strongly proves the accuracy and effectiveness of this method in the event camera calibration task.

[0145] Embodiment 2

[0146] This embodiment provides a high-precision event camera calibration system for space situational awareness, which is used to implement the method of embodiment 1, including:

[0147] Combined calibration framework building module, used to build a combined calibration framework of the star simulator, high-precision turntable and event camera;

[0148] The grid trajectory acquisition module is used to control the rotation of the high-precision turntable, collect the event stream information generated by the relative motion point light source, perform time aggregation on the event stream, and obtain the grayscale value of the grid trajectory image;

[0149] A coarse centroid position extraction module is used to extract the connected domain where the grid-shaped trajectory is located, count the intersecting coincident pixels in the grid-shaped trajectory and the pixel coincidence areas in the horizontal and vertical directions, and calculate the coarse centroid position of the calibration point based on the pixel coincidence areas;

[0150] A trajectory equation determination module is used to determine a spatial window according to the position of the rough centroid, determine a time window within the spatial window, divide the time into multiple time steps with equal steps within the time window, determine the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and perform linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates;

[0151] A sub-pixel centroid position extraction module, used to calculate the coarse centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window, and then calculate the positions of the intersections based on the mutually perpendicular trajectory equations, and use them as the sub-pixel centroid positions of the calibration points;

[0152] The parameter optimization solution and high-precision calibration module is used to optimize the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

[0153] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method as in the first embodiment.

[0154] like Fig.12 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, the memory 302 stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method as in the first embodiment.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-precision calibration method for event cameras for space situational awareness, characterized in that: include: S1, establish a combined calibration framework of the star simulator, high-precision turntable and event camera; S2, controls the rotation of the high-precision turntable, collects the event stream information generated by the relative motion point light source, performs time aggregation on the event stream, and obtains the grayscale value of the grid-shaped trajectory image; S3, extracting the connected domain where the grid-shaped trajectory is located, counting the intersecting overlapping pixels and the pixel overlapping areas in the horizontal and vertical directions in the grid-shaped trajectory, and calculating the rough centroid position of the calibration point based on the pixel overlapping area; S4, determining a spatial window according to the position of the rough centroid, determining a time window within the spatial window, dividing time into multiple time steps with equal steps within the time window, and determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and performing linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates; S5, calculating the rough centroid positions of all intersections in the grid-shaped trajectory and the trajectory equation in each spatial window, and then calculating the positions of the intersections based on the mutually perpendicular trajectory equations, and using them as the sub-pixel centroid positions of the calibration points; S6, optimizing the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

2. The high-precision calibration method for event cameras for space situational awareness according to claim 1, characterized in that: The combined calibration framework of S1 includes a hardware system and an imaging system, wherein: The core components of the hardware system include a star simulator, a high-precision turntable and an event camera; wherein the star simulator is fixed on a marble platform and emits a collimated point light source, and the event camera is installed on the inner frame of the high-precision turntable to sense the light source; the middle frame and the outer frame of the high-precision turntable rotate at a constant speed to adjust the angle respectively, changing the position of the point light source on the image plane of the event camera, and there is a unique corresponding relationship between the centroid position of the point light source and the angle of the middle frame and the outer frame of the high-precision turntable; The imaging system comprises an imaging mechanism and an imaging model, wherein: (1) The imaging mechanism includes: In the event camera, each pixel continuously monitors the logarithm of the light intensity Once the change exceeds the preset threshold C, the corresponding pixel An event will be triggered immediately The specific formula is as shown in formula (1); Among them, Δt k is pixel x k The time since the last event, polarity p k ∈{+1, -1} is the sign of brightness change; (2) The imaging model includes: the propagation process of the parallel light vector emitted by the star simulator through the event camera lens on the image plane, and the vector V is the coordinate system X of the turntable. r -Y r -Z r The parallel light vector emitted by the star simulator under is expressed as shown in equation (2), where α and β are the pitch angle and yaw angle of vector V; Rotation matrix R rot It is obtained by synthesizing according to the different rotation angles of the turntable; R re is the mounting matrix between the event camera and the turntable, the event camera coordinate system X e -Y e -Z e The expression of the parallel light vector V′ under φ is shown in formula (3), where: and are the rotation angles from the turntable coordinate system to the event camera coordinate system, θ x ,θ y They are the angles at which the turntable rotates the middle frame and the outer frame; According to the pinhole imaging model, the projection position (u′, v′) of the parallel light vector on the pixel coordinate system uv is shown in equation (4), where s is the scale factor, F is the lens focal length, d is the pixel size, and u0 and v0 are the principal point positions of the pixel coordinate system respectively; On this basis, lens distortion is added, where q1 and q2 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients; du and dv are the distortion magnitudes of the lens along the u and v directions of the pixel coordinate system, respectively. The final projection position (u, v) is shown in formula (5): By integrating formulas (2)-(5), we can establish the pixel coordinate position (u, v) and the rotation angle (θ x ,θ y ) have a one-to-one correspondence between them; based on this clear correspondence, a unified calibration algorithm for internal and external parameters is used to calibrate 13 key parameters F, d, u0, v0, q1, q1, p1, p2, α, β, Perform optimization solution; the logic of the unified calibration algorithm for internal and external parameters is shown in formula (6):

3. The high-precision calibration method for event cameras for space situational awareness according to claim 2, characterized in that: The S2 includes: S21, driving the high-precision turntable to rotate the outer frame and the middle frame respectively; S22, the event camera obtains the event flow information generated by the relative motion point light source based on the asynchronous imaging principle, counts the number of positive events generated by each pixel, uses this as the intensity value of the pixel, and obtains the grayscale value of the grid trajectory image.

4. The high-precision calibration method for event cameras for space situational awareness according to claim 3, characterized in that: The S3 includes: S31, performing noise filtering on the grid-shaped trajectory image, and counting the maximum connected domain in the image plane based on the nearest connected domain algorithm to extract the connected domain where the grid-shaped trajectory is located; S32, acquiring pixels related to the motion trajectory of the point light source, counting the overlapping pixels intersecting in the grid-shaped trajectory, and counting the pixel overlapping areas in the horizontal and vertical motion trajectories; S33, calculating the centroid position of the pixel overlap area based on the geometric center method, and using it as the rough centroid position of the calibration point.

5. The high-precision calibration method for event cameras for space situational awareness according to claim 4, characterized in that: The S4 includes: S41, selecting a fixed spatial window according to the neighboring area of ​​the pixel where the rough centroid is located; S42, sorting the positive events generated by all pixels in the window according to the time axis, determining the start time and the end time as the time window, and dividing the time period into multiple time steps; S43, determining the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm; S44, within the spatial window, linear fitting is performed on the center of mass position to determine the trajectory equation of the group of center of mass coordinates.

6. The high-precision calibration method for event cameras for space situational awareness according to claim 5, characterized in that: The S5 includes: S51, calculating the rough centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window; S52, solving a two-variable linear equation based on the mutually perpendicular trajectory equations, and the calculated coordinates are the positions of the intersection points, which are used as the sub-pixel centroid positions of the calibration points corresponding to each set of mutually perpendicular trajectories.

7. The high-precision calibration method for event cameras for space situational awareness according to claim 6, characterized in that: The S6 includes: S61, one-to-one correspondence between the sub-pixel centroid position of the calibration point and the position of the turntable outer frame and the middle frame; S62, solving the internal and external parameter coefficients of the event camera based on a unified internal and external parameter calibration method, wherein the internal and external parameters of the event camera include: internal parameters including focal length, principal point and distortion coefficient of the event camera, and external parameters including the installation matrix of the event camera coordinate system and the turntable coordinate system and the stellar simulator transmitting star vector in the turntable coordinate system.

8. A high-precision calibration system for event cameras for space situational awareness, used to implement any of the methods of claims 1-7, characterized in that: include: Combined calibration framework building module, used to build a combined calibration framework of the star simulator, high-precision turntable and event camera; The grid trajectory acquisition module is used to control the rotation of the high-precision turntable, collect the event stream information generated by the relative motion point light source, perform time aggregation on the event stream, and obtain the grayscale value of the grid trajectory image; A coarse centroid position extraction module is used to extract the connected domain where the grid-shaped trajectory is located, count the intersecting coincident pixels in the grid-shaped trajectory and the pixel coincidence areas in the horizontal and vertical directions, and calculate the coarse centroid position of the calibration point based on the pixel coincidence areas; A trajectory equation determination module is used to determine a spatial window according to the position of the rough centroid, determine a time window within the spatial window, divide the time into multiple time steps with equal steps within the time window, determine the centroid position of each time step based on a time interpolation grayscale centroid positioning algorithm, and perform linear fitting on the centroid position within the spatial window to determine the trajectory equation of the group of centroid coordinates; A sub-pixel centroid position extraction module, used to calculate the coarse centroid positions of all intersections in the grid-like trajectory and the trajectory equation in each spatial window, and then calculate the positions of the intersections based on the mutually perpendicular trajectory equations, and use them as the sub-pixel centroid positions of the calibration points; The parameter optimization solution and high-precision calibration module is used to optimize the internal and external parameters of the event camera based on the one-to-one correspondence between the sub-pixel centroid position of the calibration point and the turntable angle.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method according to any one of claims 1 to 7.

Images