Method for measuring position and attitude of spatial non-cooperative target based on binocular event stream
By acquiring event streams and performing 3D point cloud processing using a binocular event camera, accurate pose measurement of non-cooperative targets in space was achieved. This solves the problem of insufficient measurement accuracy of traditional optical sensors in the space environment and is applicable to non-cooperative targets in space and targets with rich general linear features.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-10-25
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional optical sensors struggle to effectively measure the position and attitude of non-cooperative targets in complex space environments, especially under extreme lighting conditions where their accuracy is insufficient.
A binocular event flow-based approach is adopted. By setting up a binocular event camera, the target event flow in motion is collected, converted into a 3D point cloud, straight line features are extracted, event-line matching is performed, and the target pose is solved by a nonlinear optimization algorithm.
It enables precise pose measurement of non-cooperative targets in space in complex space environments, can cope with changes in lighting and dynamic scenes, and provides an effective solution for reconstruction, tracking and localization.
Smart Images
Figure CN117437280B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method for measuring the position and attitude of a non-cooperative spatial target based on binocular event streams. Background Technology
[0002] In recent years, with the increasing frequency of space exploration activities and the growing number of space launch missions, the number of non-cooperative targets in orbit has increased rapidly. Non-cooperative targets refer to objects in space that do not have negotiation or cooperation with other spacecraft or space systems, and whose three-dimensional structural information is unknown, such as damaged and abandoned satellites, depleted spacecraft, or unauthorized aircraft. Measuring the position and attitude of non-cooperative targets is a crucial prerequisite for subsequent operations such as rendezvous, capture, and repair. Traditional optical sensors have been attempted to be applied to the attitude measurement of non-cooperative targets.
[0003] However, the extremely complex observation environment and rapidly changing lighting conditions in space pose many challenges to the pose measurement of traditional optical sensors, necessitating the development of a method that can achieve pose measurement of non-cooperative targets in space within complex space environments. Summary of the Invention
[0004] Therefore, it is necessary to provide a method for measuring the position and attitude of non-cooperative targets in space based on binocular event flow, which can adapt to the complex space environment and address the above-mentioned technical problems.
[0005] A method for measuring the position and attitude of a non-cooperative target in space based on binocular event flow, the method comprising:
[0006] Step 1: Install and set up a binocular event camera system by fixing the two event cameras together. The internal and external parameters of the binocular camera have been calibrated in advance.
[0007] Step 2: Use a binocular event camera to capture images of a moving target and collect left and right event streams at fixed time intervals; the left and right event streams represent the event streams collected by the left and right event cameras.
[0008] Step 3: Project the left and right event streams at the initial moment into three-dimensional space to obtain a three-dimensional point cloud; extract straight lines from the three-dimensional point cloud and reconstruct the target three-dimensional straight line based on the straight lines;
[0009] Step 4: Project the target 3D straight line onto the image plane of the binocular event camera according to the pre-set projection relationship to obtain the projected straight line; match the events in the left and right event streams with the projected straight line according to the nearest neighbor principle to obtain the event-line matching result;
[0010] Step 5: Calculate the distances from the left and right event flows of the binocular event camera to the corresponding projected lines based on the event-line matching results. Use a nonlinear optimization algorithm to solve for the position and attitude of the non-cooperative target in space by minimizing the distances from the left and right event flows to the corresponding projected lines.
[0011] Step 6: During the target's movement, collect event stream data for the next time interval, use the pose of the previous time interval as the initial pose value for the next time interval, and repeat steps 4 and 5 to solve for the spatial non-cooperative target position and attitude for the next time interval.
[0012] In one embodiment, extracting straight lines from a 3D point cloud and reconstructing a target 3D straight line from the straight lines includes:
[0013] Let a certain three-dimensional straight line of the target be... In Plück coordinates, it is represented as n is the normal vector of the plane passing through the three-dimensional line and the origin, and v is the direction vector of the line;
[0014] Extract a pair of corresponding lines and set up The plane formed by the left event camera center is represented as π. l , The plane formed by the right event camera center is represented as π. r Target three-dimensional straight line Images are captured in the fields of view of the left and right event cameras, respectively, corresponding to a pair of lines extracted from the left and right event streams. and set up The plane formed by the left event camera center is represented as π. l , The plane formed by the right event camera center is represented as π. r Construct a dual Pruk matrix based on the two planar representations, and obtain the three-dimensional straight line from the dual Pruk matrix. Plück coordinates.
[0015] In one embodiment, constructing the dual Pruk matrix based on the two planar representations includes:
[0016] Construct the dual Pruk matrix based on the two planar representations as follows:
[0017]
[0018] Among them, superscript () T This indicates the transpose operation.
[0019] In one embodiment, the pre-set projection relationships include the projection relationship of the left event camera and the projection relationship of the right event camera; the projection relationship of the left event camera is as follows:
[0020]
[0021] Among them, K l R is the intrinsic parameter of the left event camera. l It is a rotation matrix, T l It is a translation vector, [·] × It is an antisymmetric form.
[0022] In one embodiment, the left and right event streams include a left event stream captured by the left event camera and a right event stream captured by the right event camera; calculating the distances from the left and right event streams of the binocular event cameras to the corresponding projected lines based on the event-line matching results includes:
[0023] Based on the event-line matching results, the distance from the left event stream of the stereo event camera to the corresponding projected line is calculated as follows:
[0024]
[0025] Among them, the event-line matching result is event e l Belongs to line I l Line I l The coefficient is, the straight line I l The coefficient is I l1 ,I l2 ,I l3 .
[0026] In one embodiment, a nonlinear optimization algorithm is used to obtain the position and attitude of the non-cooperative target in space by minimizing the distance from the left and right event flows to the corresponding projected lines, including:
[0027] The position and orientation of the non-cooperative target in space are obtained by using a nonlinear optimization algorithm to minimize the distance from the left and right event flows to the corresponding projected lines.
[0028]
[0029] in, Let R denote the observation covariance matrix, ρ(·) denote the robust cost function, and the rotation matrix R l Translation vector T l This indicates the position and orientation of a non-cooperative target in space relative to the camera.
[0030] The aforementioned method for measuring the position and attitude of non-cooperative targets in space based on binocular event streams involves: acquiring motion event stream data of non-cooperative targets using binocular event cameras, dividing the data into fixed time intervals; converting the event stream data into 3D point clouds, extracting straight line features, and reconstructing the target's 3D straight line through straight line matching between the left and right event cameras; reprojecting the target's straight line model onto the planes of the left and right event cameras to establish an event-line matching relationship; solving for the target's pose by minimizing the event-line distance; and continuing to process the event stream of the next time interval to continuously solve for the target's pose. Through binocular event camera configuration, multi-view information can be acquired, and the target's 3D straight line can be reconstructed, thereby achieving accurate target pose estimation. This pose estimation method provides a significantly advantageous and feasible solution for the reconstruction, tracking, and localization of non-cooperative targets in the complex environment of space. It can effectively address challenges such as changes in illumination and dynamic scenes in the space environment, providing important support for space exploration and perception. Furthermore, this method is not only applicable to non-cooperative targets in space but also suitable for general targets with rich straight line features to achieve position and attitude measurement. Attached Figure Description
[0031] Figure 1 This is a flowchart illustrating a method for measuring the position and attitude of a non-cooperative target in space based on binocular event flow in one embodiment. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0033] In one embodiment, such as Figure 1 As shown, a method for measuring the position and attitude of a non-cooperative target in space based on binocular event flow is provided, including the following steps:
[0034] Step 1: Install and set up a binocular event camera system by fixing the two event cameras together. The internal and external parameters of the binocular camera have been calibrated in advance.
[0035] Unlike traditional cameras, event cameras do not output image information at a fixed rate. Instead, they sense brightness changes in each pixel based on asynchronous events and asynchronously and sparsely output the pixel coordinates, timestamps, and event polarity information of the event at microsecond-level resolution. This output method not only provides high-precision time information but also effectively compresses the data volume. As an advanced visual sensor, event cameras exhibit significant advantages in the high-brightness, high-contrast, and high-dynamic-range space environment. Compared to traditional cameras, event cameras can overcome the shortcomings of low frame rates, high latency, and overexposure in extreme lighting conditions, thus playing an important role in non-cooperative target pose measurement. A binocular event camera system is constructed, consisting of two event cameras fixedly mounted left and right, referred to as the left and right event cameras. The relative positional relationship between the two cameras is known, determined by the rotation matrix R. r2l Translation vector T r2l describe.
[0036] Step 2: Use a binocular event camera to capture images of a moving target and collect left and right event streams at fixed time intervals; the left and right event streams represent the event streams collected by the left and right event cameras.
[0037] A fixed stereo event camera is used to capture images of a moving, non-cooperative spatial target (hereinafter referred to as the target). The event streams acquired by the left and right event cameras within a fixed time interval are denoted as {e}. l},{e r Each event e in the event stream contains location information (x, y), time information t, and polarity p.
[0038] Step 3: Project the left and right event streams at the initial moment into three-dimensional space to obtain a three-dimensional point cloud; extract straight lines from the three-dimensional point cloud and reconstruct the target three-dimensional straight line based on the straight lines;
[0039] The event stream {e l},{e r Projecting these points onto the xyt 3D space yields a 3D point cloud. Lines are then extracted from the 3D point cloud, and based on the nearest neighbor principle, lines are drawn from the event flow {e... l},{e r The extracted lines are matched one-to-one, and then the extracted lines are matched to reconstruct the spatial lines.
[0040] Step 4: Project the target 3D straight line onto the image plane of the binocular event camera according to the pre-set projection relationship to obtain the projected straight line; match the events in the left and right event streams with the projected straight line according to the nearest neighbor principle to obtain the event-line matching result;
[0041] Step 5: Calculate the distances from the left and right event flows of the binocular event camera to the corresponding projected lines based on the event-line matching results. Use a nonlinear optimization algorithm to solve for the position and attitude of the non-cooperative target in space by minimizing the distances from the left and right event flows to the corresponding projected lines.
[0042] Solve the events {e} of the left and right event cameras based on the event-line matching results. l},{e r Distance to the corresponding projected line A nonlinear optimization algorithm is used to solve the pose of the non-cooperative target by minimizing the event-line distance.
[0043] Step 6: During the target's movement, collect event stream data for the next time interval, use the pose of the previous time interval as the initial pose value for the next time interval, and repeat steps 4 and 5 to solve for the spatial non-cooperative target position and attitude for the next time interval.
[0044] As the target moves, event stream data for the next time interval is collected. The pose of the previous time interval is used as the initial pose value for the next time interval. Steps 4 and 5 are repeated to solve for the pose of the next time interval until the target moves out of the field of view. The target pose is solved by minimizing the event-to-line distance. The event stream for the next time interval is processed again to continuously solve for the target pose. With the configuration of a binocular event camera, information from multiple perspectives can be obtained and the three-dimensional line of the target can be reconstructed, thereby achieving an accurate solution for the target pose.
[0045] In the aforementioned method for measuring the position and attitude of non-cooperative targets in space based on binocular event streams, this application acquires motion event stream data of non-cooperative targets by constructing binocular event cameras and dividing the data into fixed time intervals. The event stream data is then converted into a 3D point cloud, and straight-line features are extracted. The left and right event cameras are used to reconstruct the target's 3D straight line through straight-line matching. The target's straight-line model is reprojected onto the planes of the left and right event cameras to establish an event-line matching relationship. The target pose is solved by minimizing the event-line distance. The event stream of the next time interval is then processed, continuously solving for the target pose. Through the binocular event camera configuration, multi-view information can be obtained, and the target's 3D straight line can be reconstructed, thereby achieving an accurate solution for the target pose. This pose estimation method provides a significantly advantageous and feasible solution for the reconstruction, tracking, and localization of non-cooperative targets in the complex environment of space. It can effectively address challenges such as changes in illumination and dynamic scenes in the space environment, providing important support for space exploration and perception. Furthermore, this application is not only applicable to space targets but also suitable for position measurement of ordinary targets with rich straight-line features.
[0046] In one embodiment, extracting straight lines from a 3D point cloud and reconstructing a target 3D straight line from the straight lines includes:
[0047] Let a certain three-dimensional straight line of the target be... In Plück coordinates, it is represented as n is the normal vector of the plane passing through the three-dimensional line and the origin, and v is the direction vector of the line;
[0048] Extract a pair of corresponding lines and set up The plane formed by the left event camera center is represented as π. l , The plane formed by the right event camera center is represented as π. r Target three-dimensional straight line Images are captured in the fields of view of the left and right event cameras, respectively, corresponding to a pair of lines extracted from the left and right event streams. and set up The plane formed by the left event camera center is represented as π. l , The plane formed by the right event camera center is represented as π. r Construct a dual Pruk matrix based on the two planar representations, and obtain the three-dimensional straight line from the dual Pruk matrix. Using the Plück coordinates, repeat the above steps to reconstruct the target's three-dimensional straight line.
[0049] In one embodiment, constructing the dual Pruk matrix based on the two planar representations includes:
[0050] Construct the dual Pruk matrix based on the two planar representations as follows:
[0051]
[0052] Among them, superscript () T This indicates the transpose operation.
[0053] In one embodiment, the pre-set projection relationships include the projection relationship of the left event camera and the projection relationship of the right event camera; the projection relationship of the left event camera is as follows:
[0054]
[0055] Among them, K l R is the intrinsic parameter of the left event camera. l It is a rotation matrix, T l It is a translation vector, [·] × It is an antisymmetric form.
[0056] In a specific embodiment, the calculation process of projecting the reconstructed target 3D straight line onto the image plane of the right event camera to obtain the projected straight line is similar to that on the left. The projection relationship of the right event camera is similar to that of the left event camera, and will not be elaborated further in this application.
[0057] In one embodiment, the left and right event streams include a left event stream captured by the left event camera and a right event stream captured by the right event camera; calculating the distances from the left and right event streams of the binocular event cameras to the corresponding projected lines based on the event-line matching results includes:
[0058] Based on the event-line matching results, the distance from the left event stream of the stereo event camera to the corresponding projected line is calculated as follows:
[0059]
[0060] Among them, the event-line matching result is event e l Belongs to line I l Line I l The coefficient is, the straight line I l The coefficient is I l1 ,I l2 ,I l3 .
[0061] In a specific embodiment, the distance from the right event stream of the binocular event camera to the corresponding projected line is calculated based on the event-line matching result in a similar manner to that of the left event stream, and will not be elaborated further in this application.
[0062] In one embodiment, a nonlinear optimization algorithm is used to obtain the position and attitude of the non-cooperative target in space by minimizing the distance from the left and right event flows to the corresponding projected lines, including:
[0063] The position and orientation of the non-cooperative target in space are obtained by using a nonlinear optimization algorithm to minimize the distance from the left and right event flows to the corresponding projected lines.
[0064]
[0065] in, Let R denote the observation covariance matrix, ρ(·) denote the robust cost function, and the rotation matrix R l Translation vector T l This indicates the position and orientation of a non-cooperative target in space relative to the camera.
[0066] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0068] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for measuring the position and attitude of a non-cooperative spatial target based on binocular event flow, characterized in that, The method includes: Step 1: Install and set up a binocular event camera system by fixing the two event cameras together on the left and right sides. The internal and external parameters of the binocular event camera have been calibrated in advance. Step 2: Use the binocular event camera to capture images of a moving target and collect left and right event streams at fixed time intervals; the left and right event streams refer to the event streams collected by the left and right event cameras. Step 3: Project the left and right event streams at the initial moment into three-dimensional space to obtain a three-dimensional point cloud; extract straight lines from the three-dimensional point cloud and reconstruct the target three-dimensional straight line based on the straight lines; Step 4: Project the target 3D straight line onto the image plane of the binocular event camera according to the preset projection relationship to obtain the projected straight line; match the events in the left and right event streams with the projected straight line according to the nearest neighbor principle to obtain the event-straight line matching result; Step 5: Calculate the distances from the left and right event flows of the binocular event camera to the corresponding projected lines based on the event-line matching results. Use a nonlinear optimization algorithm to solve for the position and attitude of the non-cooperative target in space by minimizing the distances from the left and right event flows to the corresponding projected lines. Step 6: During the target's movement, collect event stream data for the next time interval, use the pose of the previous time interval as the initial pose value for the next time interval, and repeat steps 4 and 5 to solve for the spatial non-cooperative target position and attitude for the next time interval. Extracting straight lines from the 3D point cloud and reconstructing the target 3D straight line based on the straight lines includes: Let a certain three-dimensional straight line of the target be... In Plücker coordinates, it is represented as , It is the normal vector of the plane that passes through a three-dimensional line and passes through the origin. It is a vector representing the direction of a straight line; Extract a pair of corresponding lines and ,set up The plane formed by the center of the left event camera is represented as , The plane formed by the right event camera center is represented as Target three-dimensional straight line Images are captured in the fields of view of the left and right event cameras, respectively, corresponding to a pair of lines extracted from the left and right event streams. and ,set up The plane formed by the center of the left event camera is represented as , The plane formed by the right event camera center is represented as A dual Pluke matrix is constructed based on the two planar representations, and a three-dimensional straight line is obtained from the dual Pluke matrix. Plück coordinates.
2. The method according to claim 1, characterized in that, Construct the dual Pruk matrix based on the two planar representations, including: Construct the dual Pruk matrix based on the two planar representations as follows: Among them, superscript This indicates the transpose operation.
3. The method according to claim 1, characterized in that, The pre-set projection relationships include the projection relationships of the left event camera and the right event camera; the projection relationship of the left event camera is as follows: in, These are the intrinsic parameters of the left event camera. It is a rotation matrix. It is a translation vector. It is an antisymmetric form. Represents a straight line The coefficient.
4. The method according to claim 3, characterized in that, The left and right event streams include the left event stream acquired by the left event camera and the right event stream acquired by the right event camera; the distances from the left and right event streams of the binocular event cameras to the corresponding projected lines are calculated based on the event-line matching results, including: Based on the event-line matching results, the distance from the left event stream of the stereo event camera to the corresponding projected line is calculated as follows: Among them, the event-line matching result is the event. Belongs to a straight line ,straight line The coefficient is .
5. The method according to claim 4, characterized in that, A nonlinear optimization algorithm is used to obtain the position and attitude of the non-cooperative target in space by minimizing the distance from the left and right event flows to the corresponding projected lines, including: The position and orientation of the non-cooperative target in space are obtained by using a nonlinear optimization algorithm to minimize the distance from the left and right event flows to the corresponding projected lines. in, , Represents the observed covariance matrix. Represents the robust cost function, rotation matrix Translation vector This indicates the position and orientation of a non-cooperative target in space relative to the camera.