Imaging distortion correction method of strapdown event camera seeker

Abstract

The invention relates to an imaging distortion correction method for a seeker of a strapdown event camera. Comprising the following steps: collecting original event stream data in a guidance process by using an event camera, and generating event frames through time window integration; feature points are extracted, cross-frame matching is realized, and a feature track is constructed; based on external attitude reference measurement information provided by a carrier, constructing a rotation compensation model to carry out preliminary distortion correction on an event frame; mapping the attitude measurement error and the uncertainty to a camera internal reference error, and designing an internal reference error optimization model; and if the deviation of the feature points after preliminary compensation still exceeds a set threshold value, iteratively updating the internal parameter according to the optimization model so as to weaken residual distortion caused by the attitude compensation error, and re-executing rotation compensation by using the updated internal parameter. According to the invention, the distortion influence caused by attitude coupling can be effectively reduced in a complex rotation and high-dynamic guidance environment, and the imaging precision and image stabilization performance of the event camera seeker are improved.

Description

Technical Field

[0001] This invention belongs to the field of imaging, specifically relating to an imaging distortion correction method for a strapdown event camera seeker. Background Technology

[0002] In the terminal guidance phase, missile-borne seekers play a crucial role in target imaging stability and pointing accuracy. Under conditions of high-speed spin, strong illumination, and high dynamic range, traditional frame-based visual seekers are prone to motion blur and overexposure, making it difficult to meet real-time imaging requirements. Although event cameras possess advantages such as microsecond-level temporal resolution and high dynamic range, enabling them to maintain stable sensing capabilities under high-speed motion and extreme lighting conditions, existing event camera stabilization methods typically rely directly on external attitude references for rotation compensation, failing to consider the propagation effect of attitude measurement errors during the imaging process. These errors can easily accumulate in high-speed spin environments, leading to residual distortion. Summary of the Invention

[0003] The purpose of this invention is to provide an imaging distortion correction method for a strapdown event camera seeker, which is used to solve the problem of imaging distortion accumulation and attitude compensation error caused by high-speed spin and aerodynamic disturbance of the airborne strapdown event camera, reduce the target pointing deviation caused by it, and improve the image stabilization performance and tracking accuracy of the seeker.

[0004] The technical solution for achieving the objective of this invention is: a method for correcting imaging distortion of a strapdown event camera seeker, characterized by comprising the following steps:

[0005] Step (1): Use the event camera to collect raw event stream data of the dynamic scene;

[0006] Step (2): The event stream data obtained in step (1) is processed into time slices according to the preset observation window to form event frames;

[0007] Step (3): Perform feature extraction and cross-frame matching on the event frames obtained in step (2) to form feature trajectories;

[0008] Step (4): Construct an attitude compensation matrix using inertial measurement unit data, perform preliminary distortion correction and reprojection on event feature points to obtain residuals, construct an intrinsic parameter compensation optimization model and solve it;

[0009] Step (5): Output the target information after image stabilization.

[0010] Compared with the prior art, the significant advantages of this invention are:

[0011] (1) Based on prior attitude information, a composite compensation model is constructed, which can simultaneously handle multi-axis coupled motions such as spin, nutation and precession. Compared with the traditional single rotation compensation method, it can better suppress registration error and distortion accumulation under complex ballistic conditions.

[0012] (2) Based on the initial attitude compensation, this invention introduces a dynamic optimization mechanism for camera intrinsic parameter error, which maps the attitude measurement error to intrinsic parameter disturbance and performs iterative correction. Compared with the existing image stabilization method that treats the intrinsic parameter as a fixed value, it can more effectively offset the residual distortion caused by attitude measurement deviation.

[0013] (3) The method has a simple structure and low computational cost. It can be efficiently combined with the sparse event flow of the event camera and is suitable for real-time deployment on embedded platforms such as missile-borne seekers. Compared with traditional methods, it has better engineering applicability and image stabilization performance in high spin and high dynamic application scenarios. Attached Figure Description

[0014] Figure 1 This is a flowchart of the imaging distortion correction method of the present invention.

[0015] Figure 2 The diagram shows a 3D scene model and its rendering result constructed using Blender software as an example. By setting the target objects, lighting conditions and camera viewpoints in the virtual environment, a controllable image sequence for event data simulation is obtained.

[0016] Figure 3 Based on Figure 2 The diagram shows the event data obtained by transforming the rendered image sequence through the event generator v2e (video-to-events); the event data is output in the form of (x, y, p, t) quadruples, which represent the coordinates of the event on the image plane, the polarity of the brightness change, and the timestamp of the event, respectively, to simulate the actual output characteristics of the event camera.

[0017] Figure 4 To apply the event distortion correction and attitude compensation algorithm proposed in this invention to... Figure 3 A schematic diagram showing the effect of processing event data; through the method of this invention, distortion and attitude error in event data are corrected, and target features are clear. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings.

[0019] This invention proposes an imaging distortion correction method for a strapdown event camera seeker. It fully integrates the high temporal resolution of the event camera with the multi-axis attitude variation characteristics of the guidance carrier, constructing a composite rotation compensation model suitable for strapdown installation. Based on this, an intrinsic parameter error optimization model is further designed. By mapping attitude errors to camera intrinsic parameter offsets and using an annealing optimization strategy for iterative updates, adaptive compensation for residual distortion under high-speed spin and complex maneuvering conditions is achieved. This significantly improves the geometric consistency and image stabilization quality of event data, providing a reliable visual information foundation for target recognition, pointing control, and stable tracking of the seeker.

[0020] This application provides a method for correcting imaging distortion of a strapdown event camera seeker, the method flow of which is as follows:

[0021] Step (1): Use the event camera to collect raw event stream data of the dynamic scene. The event stream data is represented in the form of a quadruple:

[0022] ,

[0023] in, For pixel coordinates, For event timestamps, The polarity is given by , and Q represents the number of event data points.

[0024] Step (2): The event stream data obtained in step (1) is processed by time slicing according to a preset observation window. It is divided into M consecutive time windows from 1 to M, and each window contains the event frames within its corresponding time interval. .

[0025] M

[0026] in, Indicates the first A collection of events in a time slice. For the integration time window, M = , t k This is the start time of the k-th time window.

[0027] Step (3): Perform feature extraction and cross-frame matching on the event frames obtained in step (2) to form feature trajectories.

[0028] Step (4): Construct an attitude compensation matrix using inertial measurement unit data, perform preliminary distortion correction and reprojection on event feature points to obtain residuals, construct an intrinsic parameter compensation optimization model and solve it.

[0029] Step (5): Output the target information after image stabilization.

[0030] Step (2) specifically includes the following sub-steps:

[0031] Set points time From the initial moment Start accumulating events until the accumulated time exceeds [a certain threshold]. End the current time slice.

[0032] Step (3) specifically includes the following sub-steps:

[0033] Step (31): For each event frame Clustering is performed to extract event clusters with spatial coherence. To distinguish between point targets and area targets, an event cluster size parameter is introduced. This represents the number of valid pixels contained in an event cluster. It sets the target discrimination threshold. , here will Set to 25, when satisfied When, classify it as a point target; when When that happens, it is classified as a surface target.

[0034] cluster centroid The mean of the pixel coordinates of all events within the cluster:

[0035]

[0036] Where N is the number of events within the cluster, These are the pixel coordinates of the event.

[0037] For point targets, their characteristics are determined solely by the centroids of the event clusters. Characterization.

[0038] For surface targets, their features are determined by the cluster centroid. It is characterized together with the outer bounding box B.

[0039] Step (32): Perform cross-frame matching on the feature information in adjacent event frames.

[0040] For point targets, a distance threshold is used. To determine centroid feature consistency, if the distance between two feature points in adjacent frames (either Euclidean or Manhattan distance can be selected) is less than 1 / 3. At that time, they are considered to be the same target.

[0041] For area targets, let the p-th area target in the k-th frame be... The q-th surface target in the (k+1)-th adjacent frame is Their cluster centroids on the pixel plane are respectively and The corresponding minimum bounding boxes are respectively and The distance between the centroids of the clusters is ,in The bounding box overlap is determined by the Euclidean norm, and the overlap ratio is calculated between the intersection-union ratios of the bounding boxes of objects in two adjacent frames. An evaluation was conducted, including This represents the area on the pixel plane. The aspect ratios of the bounding box are respectively... , ,in and Let represent the width and height of the bounding box of the p-th face target in the k-th frame, respectively. Then, the shape difference is denoted as . When both conditions are met At that time, the judgment and For the same goal, among which , , These are the distance threshold, bounding box overlap threshold, and aspect ratio difference threshold, respectively.

[0042] Step (33): Based on M consecutive observation time windows, statistical analysis is performed on the feature points matched across frames to construct a feature trajectory set. When the number of occurrences of a certain feature point in the M windows is not less than... When the feature point forms a valid feature trajectory T, it is considered that the feature point forms a valid feature trajectory T; the trajectory is represented as:

[0043] .

[0044] Step (4) specifically includes the following sub-steps:

[0045] Step (41): Construct the attitude compensation matrix using inertial measurement unit (IMU) data. The spin angular velocity is obtained by projecting the three-axis angular velocities provided by the IMU onto the projectile axis. The remaining lateral angular velocities are then extracted using a frequency separation algorithm to obtain the nutation angular velocity. With precession angular velocity These three factors collectively characterize the attitude change of the carrier at time t, i.e. Based on the Rodriguez formula, an attitude compensation matrix generated by three parts of angular momentum is constructed. ( , This is to facilitate preliminary distortion correction for event imaging.

[0046] Step (42): Based on the overall attitude compensation matrix obtained in step (41) Preliminary distortion correction is performed on the feature points of the event frame, and then the target features are reprojected to calculate the distortion residual.

[0047] ① Preliminary distortion correction: For feature points within the k-th time window The camera intrinsic parameter matrix K is used to transform it to a normalized coordinate system:

[0048]

[0049] in, The static calibration yields the camera principal point coordinates as (u0, v0) and the camera focal length as (f). x f y ).

[0050] By rotation matrix Compensate the feature point coordinates back to the reference frame:

[0051]

[0052] ② Feature reprojection: The compensated points are obtained by normalization and reprojection:

[0053]

[0054] This allows us to obtain the compensated event feature point locations for each time window. This represents the pixel coordinates of the i-th feature point within the k-th time window after pose compensation and reprojection by the camera model. Since the camera intrinsic parameter matrix K contains the coordinates of the camera principal point (u0, v0), therefore... There is a functional dependency on (u0, v0);

[0055] ③ Calculate the reprojection residual:

[0056] ,

[0057] in, This represents the pixel coordinates of the i-th feature point in the reference frame (time index k=0).

[0058] Step (43): After obtaining the reprojection residual, an optimization model is established with the minimum reprojection residual as the optimization objective and the camera intrinsic center point as the optimization parameter. By solving this optimization model, the camera compensation parameters that minimize the reprojection error are obtained, and these compensation parameters are updated in the camera intrinsic parameter matrix. Based on the updated intrinsic parameters, distortion correction and attitude compensation are re-performed on the entire event stream data to eliminate the pixel deviation mapped from attitude measurement error to imaging distortion correction. Specifically:

[0059] (1) Establish an intrinsic parameter compensation optimization model based on minimizing the reprojection residual:

[0060] ,

[0061] in, Event frame collection; : The kth event frame; Total number of event frames; : Feature trajectory; |T|: Number of feature points; It is the Euclidean norm.

[0062] (2) The simulated annealing algorithm is used to solve the intrinsic parameter compensation optimization model. The algorithm achieves global search through random perturbation and step size reduction mechanism to avoid local minima, thereby obtaining the solution that makes the objective function... Minimum compensation parameter .

[0063] (3) Update the compensation parameters obtained by solving to the camera intrinsic parameter matrix, and perform distortion correction and attitude compensation on the event flow again accordingly to obtain the distortion-corrected event flow data.

Claims

1. A method for correcting imaging distortion in a strapdown event camera seeker, characterized in that, Includes the following steps: Step (1): Use the event camera to collect raw event stream data of the dynamic scene; Step (2): The event stream data obtained in step (1) is processed into time slices according to the preset observation window to form event frames; Step (3): Perform feature extraction and cross-frame matching on the event frames obtained in step (2) to form feature trajectories; Step (4): Construct an attitude compensation matrix using inertial measurement unit data, perform preliminary distortion correction and reprojection on event feature points to obtain residuals, construct an intrinsic parameter compensation optimization model and solve it; Step (5): Output the target information after image stabilization.

2. The method according to claim 1, characterized in that, In step (1), the event stream data is represented in the form of a quadruple as follows: ， in, For pixel coordinates, For event timestamps, The polarity is given by , and Q represents the number of event data points.

3. The method according to claim 2, characterized in that, Step (2) specifically involves dividing the time into M consecutive time windows from 1 to M, with each window containing event frames within the corresponding time interval. : M， in, Indicates the first A collection of events in a time slice. For the integration time window, M = , t k This is the start time of the k-th time window.

4. The method according to claim 3, characterized in that, Step (3) specifically includes the following steps: Step (31): For each event frame Clustering is performed to extract event clusters with spatial coherence; Introducing event cluster size parameter Set target discrimination threshold When satisfied When, it is classified as a point target; when When this happens, it is classified as a surface object; cluster centroid The mean of the pixel coordinates of all events within the cluster: ， Where N is the number of events within the cluster; For point targets, their characteristics are determined by the centroid of the event cluster. Characterization; For surface targets, their features are determined by the cluster centroid. It is characterized together with the outer bounding box B; Step (32): Perform cross-frame matching on the feature information in adjacent event frames; For point targets, a distance threshold is used. To determine centroid feature consistency, when the distance between two feature points in adjacent frames is less than 1 / 3... At that time, they are identified as the same target; For area targets, let the p-th area target in the k-th frame be... The q-th surface target in the (k+1)-th adjacent frame is Their cluster centroids on the pixel plane are respectively and The corresponding minimum bounding boxes are respectively and The distance between the centroids of the clusters is ,in The bounding box overlap is determined by the Euclidean norm, and the overlap ratio is calculated between the intersection-union ratios of the bounding boxes of objects in two adjacent frames. An evaluation was conducted, including Represents the area on the pixel plane; the aspect ratios of the bounding box are respectively , ,in and Let represent the width and height of the bounding box of the p-th face target in the k-th frame, respectively. Then, the shape difference is denoted as . When both conditions are met At that time, the judgment and For the same goal, among which , , These are the distance threshold, bounding box overlap threshold, and aspect ratio difference threshold, respectively. Step (33): Based on M consecutive observation time windows, statistical analysis is performed on the feature points that have been matched across frames to construct a feature trajectory set; When a certain feature point appears at least once in M ​​windows When the feature point forms a valid feature trajectory T, it is considered that the feature point forms a valid feature trajectory T; the trajectory is represented as: 。 5. The method according to claim 1, characterized in that, Step (4) specifically includes the following steps: Step (41): Construct an attitude compensation matrix using inertial measurement unit data; The spin angular velocity is obtained by projecting the triaxial angular velocity provided by the inertial measurement unit onto the projectile axis. The remaining lateral angular velocities are then extracted using a frequency separation algorithm to obtain the nutation angular velocity. With precession angular velocity These three factors collectively characterize the attitude change of the carrier at time t, i.e. Based on the Rodriguez formula, construct the overall attitude compensation matrix generated by the three parts of angular momentum: ， ， ； Step (42): Based on the overall attitude compensation matrix obtained in step (41) Preliminary distortion correction is performed on the feature points of the event frame, the target features are reprojected, and the distortion residuals are calculated. Step (43): After obtaining the reprojection residual, take the minimum reprojection residual as the optimization objective and the camera intrinsic center point as the optimization parameter to establish an optimization model. By solving the optimization model, obtain the camera compensation parameters that minimize the reprojection error and update the compensation parameters to the camera intrinsic matrix. Based on the updated intrinsic parameters, re-perform distortion correction and attitude compensation on the entire event stream data to eliminate the pixel deviation mapped from attitude measurement error to imaging distortion correction.

6. The method according to claim 5, characterized in that, Step (42) is as follows: Preliminary distortion correction: for feature points within the k-th time window The camera intrinsic parameter matrix K is used to transform it to a normalized coordinate system: ， in, The static calibration yields the camera principal point coordinates as (u0, v0) and the camera focal length as (f). x f y ); By rotation matrix Compensate the feature point coordinates back to the reference frame: ， Feature reprojection: The compensated points are obtained by normalization and reprojection. ， This allows us to obtain the compensated event feature point locations for each time window, where... This represents the pixel coordinates of the i-th feature point within the k-th time window after pose compensation and reprojection by the camera model. Since the camera intrinsic parameter matrix K contains the coordinates of the camera principal point (u0, v0), therefore... There is a functional dependency on (u0, v0); Calculate the reprojection residual: ， in, This represents the pixel coordinates of the i-th feature point in the reference frame (time index k=0).

7. The method according to claim 6, characterized in that, Step (43) is as follows: Establish an intrinsic parameter compensation optimization model based on minimizing the reprojection residual: ， in, Event frame collection; : The kth event frame; Total number of event frames; : Feature trajectory; |T|: Number of feature points; It is the Euclidean norm; Solve the optimization model to obtain the objective function. Minimum compensation parameter ; The obtained compensation parameters are updated to the camera intrinsic parameter matrix, and the event stream is re-distorted and attitude-compensated to obtain the distortion-corrected event stream data.

Images