Space target reconstruction and pose estimation method and system based on multi-view vision

Abstract

The application provides a space target reconstruction and pose estimation method and system based on multi-vision, and belongs to the technical field of space non-cooperative target pose estimation. Firstly, an ideal camera model and a distortion model are established, a multi-vision camera visual system is established, and a simulation camera internal parameter is calibrated. Then, a deep convolutional neural network is used to realize feature point extraction and matching, multi-vision space target three-dimensional point cloud reconstruction of the convolutional neural network is carried out, and sparse point cloud generation is completed. Finally, non-cooperative target pose estimation based on a probabilistic nearest point iteration algorithm (PICP) is carried out. The application improves the problems of feature point mismatching, too small point cloud quantity, and too low point cloud registration robustness in the binocular vision process, overcomes the problems of traditional methods, such as being easily disturbed by external environment, being easily disturbed to cause mismatching to occur when facing complex environment, and affecting subsequent pose measurement, and the like.

Description

Technical Field

[0001] This invention belongs to the field of spatial non-cooperative target pose estimation technology, specifically, it relates to a method and system for spatial target reconstruction and pose estimation based on multi-view vision. Background Technology

[0002] In space missions, continuous and accurate state observation of mission targets is an essential component. Commonly used attitude measurement sensors include GPS navigators, inertial navigation elements, laser rangefinders, and visual sensors. Based on whether the mission target contains an attitude measurement sensor, methods can be categorized into internal and external methods. For targets with internally installed GPS navigators or inertial sensors, the position and attitude of the space target can be directly determined using data provided by the measurement tools, thus obtaining the target's motion state. For targets that do not contain internal attitude measurement equipment and cannot communicate, laser rangefinders or visual sensors are typically used to achieve non-contact attitude measurement; this process is called the external method. Due to their simple structure and lightweight nature, visual sensors are frequently used in space target attitude measurement missions.

[0003] In spacecraft missions, mission targets can be categorized into cooperative targets and non-cooperative targets based on whether there is prior knowledge about the target, such as characteristic targets and target geometry parameters. Therefore, vision-based space target measurement technology can be divided into cooperative target vision measurement technology and non-cooperative target vision measurement technology.

[0004] Visual measurements based on cooperative targets can achieve high-precision state estimation of close-range targets, but cannot be applied to non-cooperative targets. In addition, when cooperative targets are outside the observation window, the target position will be lost.

[0005] Visual measurement based on non-cooperative targets can be divided into five types according to technical principles: based on strong geometric features, based on model matching, based on laser projection, based on point cloud registration, and based on multi-sensor fusion. However, each of these methods has its own shortcomings. Summary of the Invention

[0006] To address the issue that existing, relatively mature binocular vision-based point cloud registration schemes are easily affected by interference and prone to mismatches in complex scenes, this invention proposes a spatial target reconstruction and pose estimation method and system based on multi-view vision. The feature point matching and point cloud registration processes are optimized. A monocular vision model is analyzed, a multi-view vision measurement system is derived, and the intrinsic parameters of a simulated camera are solved.

[0007] This invention is achieved through the following technical solution:

[0008] A method for spatial target reconstruction and pose estimation based on multi-view vision: The method specifically includes the following steps:

[0009] S1. Establish an ideal camera model and distortion model, establish a multi-view camera vision system, and calibrate the intrinsic parameters of the simulation camera;

[0010] S2, based on the system established in S1, uses a deep convolutional neural network to extract and match feature points, performs multi-view visual spatial target 3D point cloud reconstruction using a convolutional neural network, and completes sparse point cloud generation;

[0011] S3, Non-cooperative target pose estimation based on the Probabilistic Nearest Point Iterative Algorithm (PICP).

[0012] Furthermore, in S1, S1.1, the monocular camera vision system is modeled, establishing an ideal camera model and a camera distortion model;

[0013] Optical cameras typically use a pinhole camera model, which employs the pinhole imaging principle to translate a two-dimensional inverted image of a three-dimensional object onto an image plane P in front of the camera lens, forming an upright image. The pinhole camera imaging model involves the transformation of four coordinate systems: the world coordinate system, the camera coordinate system, the physical imaging coordinate system, and the image pixel coordinate system.

[0014] The transformation process between the world coordinate system and the camera coordinate system involves the pose R and t of the camera in the world coordinate system, which are usually referred to as the camera's extrinsic parameters. In monocular vision experiments, the world coordinate system is often coincident with the camera coordinate system to avoid the calculation of extrinsic parameters. In binocular vision experiments, the world coordinate system is coincident with the left camera coordinate system to reduce computation.

[0015] The transformation process between the camera coordinate system and the physical imaging coordinate system can be described as a scaling process of similar triangles, assuming that the physical imaging plane is parallel to the object-image plane and perpendicular to the camera's optical axis.

[0016] The coordinates of a three-dimensional point P(X,Y,Z) in space on the physical imaging plane can be represented as:

[0017]

[0018] The transformation relationship between the physical imaging plane coordinate system and the image coordinate system is translation and scaling. Since the origin of the physical plane coordinate system is at the camera's optical center, and the origin of the image coordinate system is at the top left corner of the image, with the u-axis pointing in the column direction and the v-axis pointing in the row direction, let's assume the translation distance between the camera's optical center and the image coordinate system is (p...). u ,p v (Distance m), according to equation (1), the coordinates of point P in the image coordinate system are:

[0019]

[0020] At the same time, the physical imaging coordinate system and the image coordinate system have a scaling relationship, as shown in equation (2). The coordinate unit is m, while the digital image is a matrix with the unit pixel; assuming the pixel coordinate system is scaled by a factor of α on the u-axis and by a factor of β on the v-axis, where α and β are the lengths of a single pixel on the x-axis and y-axis of the image, respectively, according to equation (2):

[0021]

[0022]

[0023] In summary, the transformation relationship between the world coordinate system and the image pixel coordinate system can be obtained as follows:

[0024]

[0025] Where s is the scaling factor, f x ,f y ,c x ,c y The internal parameters of a camera, which can be abbreviated as:

[0026]

[0027] Formula (5) is the transformation relationship between the image pixel coordinate system and the world coordinate system of the distortion-free camera in the ideal model; distortion parameters are introduced to eliminate errors caused by real-world processes;

[0028] Camera distortion can be divided into tangential distortion and radial distortion. Tangential distortion has little effect on the image imaging process, while radial distortion has a significant impact on the final image. Therefore, only radial distortion needs to be corrected to obtain the distortion-free image required for subsequent experiments.

[0029] Considering radial distortion, a radial distortion parameter is introduced into the ideal pinhole imaging model to eliminate the effect of this distortion. Since the distortion occurs during imaging, assuming the transformation between the world coordinate system and the camera coordinate system is as follows, for any point P(X,Y,Z) in the world coordinate system, we have:

[0030]

[0031] Where P c Let P be the coordinates of point P in the camera coordinate system; let P c If the distance to the origin of the coordinate system is r, then r 2 ==(p cx / p cz ) 2 +(p cy / p cz ) 2 Using polynomials for this P c Distortion removal can be achieved through:

[0032]

[0033] In equation (8), k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters;

[0034] Although the tangential distortion of the camera can be replaced with 0 in actual experiments, it is also calculated during camera calibration. The distortion parameters of the camera are:

[0035] distCoeffs=[k1 k2 p1 p2 k3] (9)

[0036] From equations (5) and (9), it can be seen that, considering camera distortion, the total number of transformation matrices between the real-world coordinate system and the image pixel coordinate system is f. x ,f y ,c x ,c y There are nine parameters: k1, k2, k3, p1, and p2. The first four parameters are collectively referred to as camera intrinsic parameters, and the last five parameters are collectively referred to as distortion parameters.

[0037] S1.2, Multi-view camera vision system modeling;

[0038] In an ideal binocular vision measurement model, the imaging planes of the two cameras are coplanar, their optical axes are parallel, and they have the same intrinsic parameters, while their extrinsic parameters only have X-axis. LR Translation on the axis, no rotation; the coordinate systems of the left and right cameras are O and O respectively. L -X L Y L Z L and O R -X R Y R Z R The line connecting the origins of the two camera coordinate systems is called the baseline, usually denoted by b; in an ideal case, the optical axis of the camera passes through the center of the image coordinate system;

[0039] Let the coordinates of an arbitrary point P in space be (X, Y, Z), and let point P lie in the pixel coordinate system of the left and right cameras. L -u L v L and o R -u R v R The projection points on are p L (u L ,v L ) and pR (u R ,v R The corresponding coordinates in the physical imaging coordinate system are (x, y). L ,y L ) and (x R ,y R According to the principle of similar triangles, we have:

[0040]

[0041] Where f is the focal length of the camera; substituting the transformation relationship between the physical imaging coordinate system and the pixel coordinate system, we have:

[0042]

[0043] Therefore, the coordinates of point P are:

[0044]

[0045] Where d is defined as the difference in the horizontal coordinates of the left and right cameras, commonly referred to as disparity; by introducing several cameras of the same specifications into the binocular stereo vision measurement model, a multi-camera observation model can be formed; any two cameras in the multi-camera observation model can be regarded as a pair of binocular stereo vision measurement models. Therefore, for a multi-camera observation model with n cameras, there are a total of A combination of binocular stereo measurement models.

[0046] Further, in S1.3, the intrinsic parameters of the simulated camera are calibrated; the intrinsic parameters of the camera in the simulation environment are solved; the simulated camera model is selected as an OpenGL perspective camera model, and the parameter relationship with the ideal pinhole camera model is as follows:

[0047]

[0048] Where θ is the field of view angle in the pitch direction; Aspect is the aspect ratio; n is the near-plane distance in meters; f is the far-plane distance in meters; f x ,f y ,c x ,c y For camera internal parameters;

[0049] The intrinsic parameters of the simulated camera under the current parameters can be calculated using equation (16). Since this environment is a simulation environment, it can be regarded as an ideal camera model. Therefore, there is no distortion, and both radial and tangential distortion are 0.

[0050] In S2, S2.1, image acquisition and image preprocessing are performed, and the SuperPoint algorithm is used to extract the pixel positions of feature points and their corresponding descriptors.

[0051] SuperPoint employs a semi-self-supervised training approach, with the training process consisting of three steps: generation of the basic extractor, self-labeling of feature points, and final network training and error evaluation.

[0052] The basic extractor uses virtual 3D objects as a training set to extract such highly characteristic corner points, preparing for subsequent self-calibration.

[0053] Feature point self-annotation: Based on the obtained basic feature point extractor, self-annotation is performed on unlabeled general images; homography transformation is performed on unlabeled images to generate new images, and feature points are re-annotated using the basic extractor. After extraction, inverse transformation is used to generate the final self-annotated image.

[0054] Final network training: By performing known transformations on the self-labeled images, the pose relationship between the two images can be obtained. Then, the two images are fed into the SuperPoint network for training. Since the relative pose relationship between the two images is known, error analysis can be performed on the finally extracted feature points and descriptors.

[0055] S2.2, Feature point detection and matching: The SuperGlue algorithm is used to match the descriptors obtained in S2.1 to obtain the final feature point matching pairs and matching confidence.

[0056] S2.3, False Match Removal: The RANSAC algorithm is used to remove false match pairs from S2.2.

[0057] Let the size of the dataset be K, and the minimum number of parameters required to solve the model be S. Then the steps of the RANSAC algorithm are as follows:

[0058] S2.3.1 Randomly sample S points from dataset K as the initial model to solve the dataset;

[0059] S2.3.2 Fit a possible model for the dataset based on the S points sampled in the previous step;

[0060] S2.3.3 Substitute other points in the dataset into the model to calculate the error. If the error is less than the threshold P, the point is taken as an interior point.

[0061] S2.3.4 Repeat N times, select the set with the most interior points and refit a new possible model;

[0062] S2.3.5 Repeat the above steps until the specified number of iterations is reached, then end the algorithm;

[0063] To achieve the desired accuracy, different numbers of iterations are required depending on the outlier count. For any dataset, the number of iterations can be expressed as:

[0064]

[0065] Where w is the probability that a point randomly selected from the dataset is an interior point, and p is the probability that the sampling is successful.

[0066] Further, in S2.4, the camera relative pose is solved and spatial points are determined. The camera relative pose is solved using epipolar constraints. Based on the obtained camera relative pose, the three-dimensional coordinates of the feature points are obtained using triangulation. Then, a three-dimensional point cloud reconstruction based on triangulation is performed.

[0067] Camera calibration can obtain the relative pose relationships of a multi-camera group. Based on this, the principle of triangulation is used to generate sparse point clouds.

[0068] Consider images I1 and I2 taken by two parallel cameras in different poses. Taking the coordinate system of the left camera as the reference coordinate system, the transformation matrix of the right camera is T. Let the optical centers of the cameras be O1 and O2, and let the projected pixel of any 3D point P in space be p1 in I1 and p2 in I2. Theoretically, the lines O1p1 and O2p2 will intersect at the spatial point P in the scene. However, due to factors such as noise and lens distortion, the two lines emanating from the optical centers often do not intersect in space. Therefore, a least squares function can be constructed to solve the above process. Let x1 and x2 be the normalized coordinates of two feature points, and we can obtain:

[0069] s1x1=s2Rx2+t. (18)

[0070] In equation (18), the camera pose R and position t are known. What needs to be solved is the depths s1 and s2 of the two feature points in their respective images. Multiply both sides of the above equation by a factor. We can obtain:

[0071]

[0072] s2, i.e., the three-dimensional coordinates of the spatial point, can be obtained by solving equation (19);

[0073] S2.5, point cloud optimization, utilizes the Bundle Adjustment algorithm to simultaneously optimize camera pose and 3D coordinates of spatial points;

[0074] To minimize reprojection error, both the 3D point coordinates and the camera pose need to be optimized simultaneously, ensuring that light rays emitted from any feature point converge to the optical center, i.e., BA (Bundle Adjustment). Considering the camera observation model, the least squares method is used to calculate the error generated in a single step of the above process:

[0075] e = h(x,y) - h(ξ,p) (20)

[0076] Where ξ represents the Lie group corresponding to the camera's true pose R and t at the current moment, p represents the true 3D coordinates of the landmark at the current moment, x represents the Lie group corresponding to the camera's pose R and t at this moment, and y represents the 3D coordinates of the landmark p observed in state x.

[0077] Considering the overall observation process, the cost function can be expressed as:

[0078]

[0079] Where z = h(x,y), z ij For the camera in pose ξ i Observation marker p j The data obtained;

[0080] Since equation (21) is a nonlinear function, to solve this equation, we add an increment Δx to the overall independent variable of the equation, and we have:

[0081]

[0082] Where E ij The partial derivative of the cost function with respect to the coordinates of the landmark's three-dimensional point in this state is:

[0083]

[0084] F ij Let the partial derivative of the cost function with respect to the camera pose in this state be:

[0085]

[0086] Solving the above equations using the LM (Levenberg-Marquardt) algorithm yields:

[0087] f(x+δx)≈f(x)+Jδx (25)

[0088] Where J is the Jacobian matrix obtained by taking the first-order partial derivative of the cost function, from the above equation we have J = [FE]; minimizing the reprojection error and taking f(x) = ε, we have:

[0089] (J T J+λI)δx=-J T ε (26)

[0090] Solving the above equations allows for simultaneous optimization of both camera pose and 3D point coordinates.

[0091] Further, in step 3, S3.1: Probability point cloud generation, the point cloud generated in S2.5 of the sparse point cloud generation module is fused with the matching confidence obtained in S2.2 to obtain the probability point cloud;

[0092] S3.2: Distant interference point elimination based on outlier removal: Outliers are removed to improve the subsequent registration accuracy; at the same time, in order to improve the operation speed of the registration algorithm, the point cloud is downsampled to reduce the size of the point cloud;

[0093] To perform preliminary processing of the source and target point clouds, an outlier elimination algorithm based on statistical filtering was adopted; the point cloud set in the three-dimensional coordinate system can be represented as:

[0094] P = {p i ∈R 3} i＝1,2,3,... (27)

[0095] Where p i =(x i ,y i ,z i The coordinates of a point are set in a Cartesian coordinate system. For each 3D point, two steps are taken: first, search for the nearest set of points in its neighborhood and calculate the Eulerian distance; then, calculate the standard deviation of the distance between that point and the closed point. In this way, it can be determined whether a point is an outlier.

[0096]

[0097] Where d i The statistical distance between points is represented by k; the number of selected neighboring points is represented by m; and d represents the threshold for controlling the number of outliers. std p i Standard deviation between the standard deviation and the nearest neighbor;

[0098] For p i If p j With p i The Euler distance between them is greater than d i Then p j It is considered an outlier.

[0099] Furthermore, S3 also includes: S3.3: Point cloud registration, using the Probabilistic Nearest Iterative Point (PICP) algorithm to register the point clouds between frames, obtaining the relative poses of non-cooperative targets, and then comparing them with the ground truth poses to measure the robustness and accuracy of the algorithm; given two point cloud models, namely the reference point cloud {p i} and target point cloud {p' iThe pose estimation problem for a non-cooperative target is transformed into finding the optimal transformation parameters that minimize the distance between the reference point cloud and the target point cloud. This allows us to obtain the rotation matrix R and translation vector t of the non-cooperative target.

[0100]

[0101] The distance error of a single point can be expressed as:

[0102] e i =p i -(Rp′ i +t) (30)

[0103] As can be seen from the above equation, the registration problem can be transformed into minimizing the sum of the Euclidean distances between the reference point cloud and the target point cloud errors, i.e., equation (31):

[0104]

[0105] Considering the impact of near interference points on the point cloud registration process, a distance weight is added to Equation (31) to reduce the impact of near interference points on the registration process. Therefore, the new loss function can be expressed as:

[0106]

[0107] Where w i Represents distance weights and w i ∈[0,1]; if w i If the value of approaches 0, the error generated by the loss function at that point will decrease simultaneously.

[0108] Near-interference points are usually located at the edge of the target, so the weights can be constructed using the distance to the centroid of the point cloud; for point pairs, the distances from the points in the source and target point clouds to the centroids are shown below:

[0109]

[0110] Where q μ and q′ μ Let d represent the centroids of the reference point cloud and the target point cloud, respectively; considering d pi and d qi As an independent variable, a two-dimensional Gaussian distribution function is chosen to evaluate the weight of each point, as shown in Equation (34); the shapes of non-cooperative targets such as satellites are roughly symmetrical and the structure is uniform; at the same time, the two-dimensional Gaussian function has rotational symmetry, so the weight will not be biased to any side and cause registration failure; in addition, the Gaussian function is a single-valued function, and the weight of each point increases or decreases monotonically with the distance to the centroid.

[0111]

[0112] Because of w i ∈[0,1], the value of σ can be set as Therefore w i It can be expressed as the following formula:

[0113]

[0114] Combining equation (32), the loss function considering the near-interference point can be obtained as follows:

[0115]

[0116] To address the issue of decreased accuracy in feature recognition algorithms due to feature degradation under extreme lighting conditions, the matching confidence score obtained from the SuperGlue algorithm is used as the confidence score of the point cloud. This confidence score is then incorporated as a weight in the point cloud registration process to reduce the impact of mismatches on the final pose estimation. This yields a new loss function:

[0117]

[0118] Where P i For probability weights and P i ∈[0,1]; Considering the matching confidence obtained by the SuperGlue algorithm, a higher matching confidence indicates that the pair of feature points has strong features in the image recognition process, and the 3D points obtained from the pair of feature points have higher reliability; conversely, a lower matching confidence indicates that the pair of feature points is more likely to be mismatched, therefore the probability weight P i It can be represented as:

[0119]

[0120] Where P(p i ) and P(p' i These represent the reference point probability and the target point probability, respectively. Therefore, considering the influence of complex backgrounds and extreme lighting conditions, the loss function of the Probabilistic Nearest Iteration Point (PICP) algorithm can be obtained as follows:

[0121]

[0122] According to the above formula, the point cloud registration problem can be reformulated as a least squares problem; to simplify the calculation, the transformation of the point cloud can be cleverly converted into two steps: translation and rotation; the motion of the rigid body should be regarded as the rotation of the source point cloud around its centroid and the translation of the centroid; therefore, the centroids of two sets of point clouds are listed:

[0123]

[0124] Combining the above formula, we can obtain:

[0125]

[0126] Due to the cross term part (p i -pR(p i Since the summation of '-p' is zero, the optimization function can be expressed as:

[0127]

[0128] The centroid-free coordinates of the reference point cloud and the target point cloud are calculated as follows:

[0129] q i =p i -p,q′ i =p′ i -p′ (43)

[0130] The least squares problem can be optimized as follows:

[0131]

[0132] Expanding the error term with respect to R, we have:

[0133]

[0134] Remove terms irrelevant to R:

[0135]

[0136] Singular value decomposition of the above equation yields:

[0137]

[0138] have:

[0139] Therefore:

[0140] The above formula yields the rotation matrix and translation vector for point cloud registration.

[0141] A multi-view vision-based spatial target reconstruction and pose estimation system includes a camera vision module, a sparse point cloud generation module, and a point cloud registration module. The camera vision module establishes an ideal camera model and distortion model, builds a multi-view camera vision system, and calibrates the intrinsic parameters of the simulated camera. The sparse point cloud generation module, based on the system established in S1, uses a deep convolutional neural network to extract and match feature points, performs multi-view vision spatial target 3D point cloud reconstruction using convolutional neural networks, and completes sparse point cloud generation. The point cloud registration module uses the Probabilistic Nearest Point Iterative Algorithm (PICP) for non-cooperative target pose estimation.

[0142] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0143] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0144] Beneficial effects of the invention

[0145] This invention optimizes the feature point matching and point cloud registration processes, addressing issues such as feature point mismatch, insufficient point cloud quantity, and low robustness of point cloud registration in binocular vision. It overcomes the problem that traditional methods such as SIFT, ORB, and SURF, while performing well in stable indoor environments, are highly susceptible to external interference. In complex environments, such as low light or overexposure, they are prone to mismatches, affecting subsequent pose measurements.

[0146] To address the issue that traditional feature point extraction and matching algorithms are easily affected by interference and prone to mismatches in complex environments, this paper utilizes the SuperPoint and SuperGlue algorithms to extract and match feature points from non-cooperative satellites. The RANSAC algorithm is used to remove mismatched pairs, and triangulation principles are employed to reconstruct the 3D point cloud of the target. Finally, the BA algorithm is used to simultaneously optimize both camera pose and 3D point coordinates. Simulation experiments were conducted to reconstruct the 3D point cloud of non-cooperative satellites, ultimately generating a sparse point cloud with a scale error within the range of 1% to 2%, demonstrating high accuracy.

[0147] To address the problem of distant interference points, an outlier removal method is proposed to eliminate the influence of distant interference points on the pose measurement process. To address the problems of near interference points and unavoidable mismatches, distance weights and probability weights are added to the traditional point cloud registration algorithm. The proposed Probabilistic Nearest Point Iterative Algorithm (PICP) adds distance weights and probability weights to each point in the point cloud to solve the problems of complex backgrounds and strong light that non-cooperative target pose estimation needs to face in real-world environments. Attached Figure Description

[0148] Figure 1 This is the framework of the pose measurement system of the present invention; Figure 2 For camera distortion model; Figure 3 This is a schematic diagram of radial distortion; Figure 4 For an ideal binocular vision model;

[0149] Figure 5 It is a trinocular vision model; Figure 6 This is an OpenGL perspective camera model; Figure 7 Generate bounding boxes for sparse point clouds; Figure 8The SuperPoint algorithm architecture;

[0150] Figure 9 Framework for SuperPoint training steps; Figure 10 The architecture is based on the SuperGlue algorithm.

[0151] Figure 11 This is a schematic diagram of RANSAC algorithm mismatch removal, where (a) is the original SIFT recognition result and (b) is the result after RANSAC mismatch removal.

[0152] Figure 12 This is a schematic diagram of triangulation.

[0153] Figure 13 The diagram shows the results of the SuperGlue algorithm, where (a) is frame 1, (b) is frame 88, (c) is frame 183, and (d) is frame 268.

[0154] Figure 14 A schematic diagram of sparse point clouds from non-cooperative satellites; Figure 15 Here is the flowchart for the PICP algorithm; Figure 16 This is a schematic diagram of outliers. Detailed Implementation

[0155] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0156] Combination Figures 1 to 16 A method for spatial target reconstruction and pose estimation based on multi-view vision: The method specifically includes the following steps:

[0157] S1. Establish an ideal camera model and distortion model, establish a multi-view camera vision system, and calibrate the intrinsic parameters of the simulation camera;

[0158] S2, based on the system established in S1, uses a deep convolutional neural network to extract and match feature points, performs multi-view visual spatial target 3D point cloud reconstruction using a convolutional neural network, and completes sparse point cloud generation;

[0159] S3, Non-cooperative target pose estimation based on the Probabilistic Nearest Point Iterative Algorithm (PICP).

[0160] In S1, S1.1, the monocular camera vision system is modeled, establishing an ideal camera model and a camera distortion model. Using the pinhole imaging principle, light passes through the camera's optical center and falls on the photosensitive plane, forming a two-dimensional inverted image of the three-dimensional object. For ease of modeling, this two-dimensional inverted image of the three-dimensional object is usually translated to the artifact plane P in front of the camera lens to form an upright image. The pinhole camera imaging model mainly involves the transformation of four coordinate systems: the world coordinate system (inertial coordinate system), the camera coordinate system, the physical imaging coordinate system, and the image pixel coordinate system.

[0161] The transformation process between the world coordinate system and the camera coordinate system mainly involves the camera's pose R and t in the world coordinate system, which are usually referred to as the camera's extrinsic parameters. In monocular vision experiments, the world coordinate system is often coincident with the camera coordinate system to avoid calculating extrinsic parameters. In binocular vision experiments, the world coordinate system is often coincident with the left camera coordinate system to reduce computation. The transformation process between the camera coordinate system and the physical imaging coordinate system can be simply described as a scaling process of similar triangles. To clearly describe the transformation process, it is assumed that the physical imaging plane is parallel to the object plane and perpendicular to the camera's optical axis.

[0162] Based on the similarity relationship of triangles, the coordinates of a three-dimensional point P(X,Y,Z) in space in the physical imaging plane can be expressed as:

[0163]

[0164] The transformation relationship between the physical imaging plane coordinate system and the image coordinate system is translation and scaling. Since the origin of the physical plane coordinate system is at the camera's optical center, and the origin of the image coordinate system is at the top left corner of the image, with the u-axis pointing in the column direction and the v-axis pointing in the row direction, let's assume the translation distance between the camera's optical center and the image coordinate system is (p...). u ,p v (Distance m), according to equation (1), the coordinates of point P in the image coordinate system are:

[0165]

[0166] At the same time, the physical imaging coordinate system and the image coordinate system have a scaling relationship, as shown in equation (2). The coordinate unit is m, while the digital image is a matrix with the unit pixel; assuming the pixel coordinate system is scaled by a factor of α on the u-axis and by a factor of β on the v-axis, where α and β are the lengths of a single pixel on the x-axis and y-axis of the image, respectively, according to equation (2):

[0167]

[0168]

[0169] In summary, the transformation relationship between the world coordinate system and the image pixel coordinate system can be obtained as follows:

[0170]

[0171] Where s is the scaling factor, f x ,f y ,c x ,c y The internal parameters of a camera, which can be abbreviated as:

[0172]

[0173] Formula (5) is the transformation relationship between the image pixel coordinate system and the world coordinate system of the distortion-free camera in the ideal model. Since the manufacturing process of real camera lenses inevitably causes unevenness in the camera lens surface, the path of light propagation will be curved to a certain extent. The ideal pinhole imaging model will have an error with the real model. This error will cause distortion in the image captured by the camera. Therefore, it is necessary to introduce distortion parameters to eliminate the error caused by the real process.

[0174] Camera distortion can be divided into tangential distortion and radial distortion. Tangential distortion is caused by manufacturing defects that prevent the lens from being perfectly parallel to the imaging plane, while radial distortion is caused by light rays that bend more inwards the further away from the optical axis, resulting in a square image becoming barrel-shaped. Figure 3 As shown, this is also called barrel distortion; if light rays farther from the optical axis bend even further outward, it is called pincushion distortion. In actual camera use, it can be found that the tangential distortion of the camera is very small, usually within 10. -4 The magnitude is small, having little impact on the image imaging process, while radial distortion is typically in the range of 10. -2 The magnitude of the radial distortion has a significant impact on the final image formation. Therefore, in some image correction experiments, it is often only necessary to correct the radial distortion to obtain the distortion-free image required for subsequent experiments.

[0175] Considering radial distortion, from Figure 3 It can be seen that pixels farther from the image center exhibit greater distortion, with noticeable curvature near the edges. This will introduce significant errors into subsequent image processing and feature matching. A radial distortion parameter is introduced into the ideal pinhole imaging model to eliminate the impact of this distortion. Since the distortion occurs during imaging, assuming the transformation between the world coordinate system and the camera coordinate system is as follows, for any point P(X,Y,Z) in the world coordinate system, we have:

[0176]

[0177] Where P c Let P be the coordinates of point P in the camera coordinate system. cIf the distance to the origin of the coordinate system is r, then r 2 ==(p cx / p cz ) 2 +(p cy / p cz ) 2 Using polynomials for this P c Distortion removal can be achieved through:

[0178]

[0179] In equation (8), k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters. Although the tangential distortion of the camera can be replaced by 0 in actual experiments, it is also solved during camera calibration. The distortion parameters of the camera are:

[0180] distCoeffs=[k1 k2 p1 p2 k3] (9)

[0181] From equations (5) and (9), it can be seen that, considering camera distortion, the total number of transformation matrices between the real-world coordinate system and the image pixel coordinate system is f. x ,f y ,c x ,c y There are nine parameters: k1, k2, k3, p1, and p2. The first four parameters are collectively referred to as camera intrinsic parameters, and the last five parameters are collectively referred to as distortion parameters.

[0182] S1.2, Multi-view camera vision system modeling; Before establishing the multi-view camera vision system model, it is necessary to study the most basic binocular stereo vision system. Inspired by the human visual system, the binocular stereo vision system consists of two optical cameras located at different positions in space. Utilizing the principle of binocular parallax, it performs matching and calculation of corresponding feature points on images captured by the two cameras at different positions to obtain the three-dimensional geometric information and motion state of the target object in space.

[0183] like Figure 4 In an ideal binocular vision measurement model, the imaging planes of the two cameras are coplanar, their optical axes are parallel, and they have the same intrinsic parameters. Their extrinsic parameters only have X-axis... LR Translation on the axis, no rotation. The coordinate systems of the left and right cameras are O and O respectively. L -X L Y L Z L and O R -X R Y R Z RThe line connecting the origins of the two camera coordinate systems is called the baseline, usually denoted by b; in an ideal case, the optical axis of the camera passes through the center of the image coordinate system.

[0184] Let the coordinates of an arbitrary point P in space be (X, Y, Z), and let point P lie in the pixel coordinate system of the left and right cameras. L -u L v L and o R -u R v R The projection points on are p L (u L ,v L ) and p R (u R ,v R The corresponding coordinates in the physical imaging coordinate system are (x, y). L ,y L ) and (x R ,y R According to the principle of similar triangles, we have:

[0185]

[0186] Where f is the focal length of the camera. Substituting the transformation relationship between the physical imaging coordinate system and the pixel coordinate system, we have:

[0187]

[0188] Therefore, the coordinates of point P are:

[0189]

[0190] Where d is defined as the difference between the horizontal coordinates of the left and right cameras, usually referred to as disparity. Since the binocular stereo vision measurement process uses the principle of triangulation, combined with equation (10), it can be seen that the depth that the binocular camera can measure is inversely proportional to the disparity and directly proportional to the baseline. When the left and right cameras are at X LR If the baselines are too close together, the binocular camera array will degenerate into a monocular camera, making triangulation impossible and thus target depth estimation impossible. Therefore, in actual experiments, the choice of baseline often affects the overall identifiable range and accuracy of the measurement system. In engineering experiments, it is generally considered that a baseline distance to working distance ratio between 0.5 and 2.2 can achieve better measurement results.

[0191] By introducing several cameras of the same specifications onto a binocular stereo vision measurement model, a multi-camera observation model can be formed. Any two cameras in the multi-camera observation model can be considered as a pair of binocular stereo vision measurement models. Therefore, for a multi-camera observation model with n cameras, there are a total of... A combination of binocular stereo measurement models.

[0192] Taking a trinocular camera as an example, the process of establishing a multi-view camera observation model is illustrated. Figure 5 As shown. Assume the coordinates of any point P in the world coordinate system are (X, Y, Z), and the coordinates of its projection points on the pixel planes of the three cameras are p, p, and p, respectively. L (u L ,v L ), p H (u H ,v H ) and p R (u R ,v R Considering that the three cameras have a total of Therefore, the three binocular stereo vision systems formed by cameras L, H, and R intersect at points P0(X0,Y0,Z0), P1(X1,Y1,Z1), and P2(X2,Y2,Z2) in space. Ideally, these three points would simultaneously intersect at point P; however, due to errors such as camera calibration, the results obtained by the multi-camera measurement system in actual experiments often differ. A spatial point, meaning that there exists one spatial point for every pair of stereo vision systems, such as... Figure 5 As shown in P0, P1, and P2. Based on the aforementioned binocular stereo vision measurement principle, assume that the projection matrices of camera L, camera H, and camera R are M, respectively. L M H and M R The coordinates of spatial points P0, P1, and P2 can be calculated using the following formula:

[0193]

[0194]

[0195]

[0196] The three-dimensional coordinates of spatial points P0, P1 and P2 can be calculated from equations (13), (14) and (15).

[0197] S1.3, Simulation camera intrinsic parameter calibration; In order to facilitate subsequent simulation experiments, the intrinsic parameters of the camera in the simulation environment are solved.

[0198] The selected simulation camera model is an OpenGL perspective camera model, such as... Figure 6 As shown. According to Figure 6 The perspective camera model shown has the following calibration parameter relationship with the ideal pinhole camera model:

[0199]

[0200] Where θ is the field of view angle in the pitch direction; Aspect is the aspect ratio; n is the near-plane distance in meters; f is the far-plane distance in meters; f x ,f y ,c x ,c y These are the camera's intrinsic parameters. The simulated camera parameter settings are shown in Table 1.

[0201] Table 1 OpenGL Camera Parameter Settings

[0202]

[0203] The intrinsic parameters of the simulated camera under the current parameters can be calculated using Equation (16). Since this environment is a simulation environment, it can be regarded as an ideal camera model. Therefore, there is no distortion caused by lens manufacturing or other issues. Thus, both radial and tangential distortion are 0. The obtained camera intrinsic parameters are shown in Table 2.

[0204] Table 2 Ideal Camera Internal Parameters

[0205]

[0206]

[0207] S1 first proposes an improved framework for non-cooperative target pose measurement based on multi-view vision, building upon the traditional framework for non-cooperative target pose measurement based on binocular vision. This framework addresses issues such as feature point mismatch, insufficient point cloud quantity, and low robustness of point cloud registration during binocular vision. Secondly, it models an ideal monocular camera vision system, including the transformation relationships between the world coordinate system, camera coordinate system, physical imaging coordinate system, and image pixel coordinate system during camera capture, and derives camera intrinsic parameters. Real-world error influences are incorporated into the ideal monocular camera vision system, leading to a monocular camera distortion model and camera distortion parameters. Next, to establish a multi-view camera vision system model, the binocular vision system model is derived, including disparity and solving for the three-dimensional coordinates of spatial points. Adding several cameras to the binocular vision system model constitutes a multi-view vision system. This multi-view vision system model is derived by decomposing it into combinations of multiple pairs of binocular vision systems.

[0208] In S2, because non-cooperative target pose measurement lacks prior information about the target, such as geometry, key features, and cooperative targets, it is impossible to obtain the target's relative pose through identifying cooperative markers, strong geometric features, and model matching. Traditional non-cooperative target pose estimation algorithms typically generate a 3D point cloud of the target using binocular vision and feature point matching, transforming the non-cooperative target into a cooperative one. However, this approach is highly susceptible to interference from the external environment, affecting subsequent pose measurement. To address this issue, based on the principle of feature point extraction and matching in deep convolutional neural network algorithms, the SuperPoint algorithm is used to extract the pixel positions of feature points and their corresponding descriptors. The SuperGlue algorithm is then used to match the descriptors obtained in the previous step to obtain the final feature point matching pairs and matching confidence scores. The RANSAC algorithm, combined with triangulation principles, is used to generate a sparse point cloud of the non-cooperative target. Finally, the BundleAdjustment algorithm is used to simultaneously optimize the camera pose and 3D point coordinates, obtaining a 3D point cloud with minimized reprojection error to support subsequent experiments. The sparse point cloud generation framework structure is as follows: Figure 7 As shown.

[0209] S2.1 involves image acquisition and preprocessing, primarily including image distortion correction, using the SuperPoint algorithm to extract feature point pixel locations and corresponding descriptors; the architecture of the SuperPoint algorithm is as follows. Figure 8 As shown, the main framework of the algorithm consists of a shared encoder and decoders corresponding to feature point extraction and descriptor generation, respectively.

[0210] In its shared encoder section, SuperPoint follows the approach of the VGG network, with the encoder consisting of convolutional layers, pooling spatial downsampling, and a non-linear activation function. SuperPoint's encoder uses three max-pooling layers, defining an image of size H×W as H... c =H / 8 and W c =W / 8, three 2×2 non-overlapping max pooling operations produce 8×8 pixel units. The encoder takes the input image... Mapped to intermediate tensors with smaller spatial dimensions and greater channel depth superior.

[0211] The feature point partial decoder will calculate And output a value of size The tensor. 65 channels correspond to local, non-overlapping 8×8 pixel grid regions plus an additional "no-interest point". After channel-wise softmax, the additional "no-interest point" is removed and...

[0212] Descriptor Part Decoder Computation And output a value of size The tensor. To output a dense mapping of L2-normalized fixed-length descriptors, SuperPoint uses a UCN-like model that first outputs a semi-dense grid of descriptors (e.g., one per 8 pixels). Learning descriptors semi-randomly rather than densely reduces the memory required for training and maintains runtime tractability. The decoder then performs cubic interpolation on the descriptors and finally normalizes them to unit length.

[0213] Because feature points cannot be described using rigorous mathematical expressions, it is impossible to manually annotate camera-captured images to form a feature point dataset. If traditional feature point detection algorithms, such as SIFT, are used to extract and annotate feature points to form a training dataset, the recognition accuracy of these algorithms is inherently limited, making it difficult for models trained on such datasets to achieve good recognition results. SuperPoint employs a semi-self-supervised training method, such as... Figure 9 As shown, the training process of SuperPoint mainly consists of three steps: generation of the basic extractor, self-labeling of feature points, and final network training and error evaluation.

[0214] Basic Extractor Generation: Although feature points lack a rigorous mathematical definition, some points are still widely recognized due to their strong characteristics, such as the corner points of squares and triangles. Based on this principle, virtual 3D objects are used as the training set to extract these highly characteristic corner points, preparing for subsequent self-calibration.

[0215] Feature point self-annotation: Based on the obtained basic feature point extractor, unlabeled general images are self-annotated. Since the basic feature point extractor can only extract strong feature points, in order to obtain more feature point annotations, homography transformation is performed on the unlabeled image to generate a new image, and the basic extractor is used again for feature point annotation. After extraction, an inverse transformation is used to generate the final self-annotated image.

[0216] Final network training: A known transformation is performed on the self-labeled images to obtain the pose relationship between the two images. Then, the two images are fed into the SuperPoint network for training. Since the relative pose relationship between the two images is known, error analysis can be performed on the finally extracted feature points and descriptors.

[0217] S2.2, Feature point detection and matching: The SuperGlue algorithm is used to match the descriptors obtained in S2.1 to obtain the final feature point matching pairs and matching confidence.

[0218] The SuperGlue matching algorithm is used in conjunction with the SuperPoint algorithm to achieve excellent matching results. The architecture of the SuperGlue algorithm is as follows: Figure 10 As shown, it can be mainly divided into an attention map neural network part and an optimal matching layer part. The first part uses a keypoint encoder to map keypoint locations p and their visual descriptors d into a single vector. Inspired by the Transfoemer network, SuperGlue uses self-attention layers and cross-attention layers (repeated L times) to create a more powerful representation f. The optimal matching layer creates an M×N score matrix and then uses the Sinkhorn algorithm (for T iterations) to find the optimal part assignment.

[0219] S2.3, Mismatch Removal: The RANSAC algorithm is used to remove mismatched pairs from S2.2. Although the SuperGlue algorithm can achieve accurate and robust feature point extraction and matching, it can still produce a small number of mismatches under harsh environments. To further improve the accuracy of subsequent camera pose estimation and 3D point cloud reconstruction, the RANSAC algorithm is used to remove mismatched pairs from the feature point matching pairs obtained above. A common approach to mismatch removal is to filter and remove matching pairs whose matching distance is greater than a multiple of the minimum matching distance, and then use the RANSAC algorithm to remove mismatched pairs from the filtered dataset.

[0220] Let the size of the dataset be K, and the minimum number of parameters required to solve the model be S. Then the steps of the RANSAC algorithm are as follows:

[0221] S2.3.1 Randomly sample S points from dataset K as the initial model to solve the dataset;

[0222] S2.3.2 Fit a possible model for the dataset based on the S points sampled in the previous step;

[0223] S2.3.3 Substitute other points in the dataset into the model to calculate the error. If the error is less than the threshold P, the point is taken as an interior point.

[0224] S2.3.4 Repeat N times, select the set with the most interior points and refit a new possible model;

[0225] S2.3.5 Repeat the above steps until the specified number of iterations is reached, then end the algorithm.

[0226] To achieve the desired accuracy, different numbers of iterations are required depending on the outlier count. For any dataset, the number of iterations can be expressed as:

[0227]

[0228] Where w is the probability that a randomly selected point in the dataset is an interior point, and p is the probability of successful sampling. When p = 99%, the corresponding number of iterations is shown in Table 3.

[0229] Table 3 RANSAC iteration count

[0230]

[0231] To demonstrate the algorithm's effectiveness, the original matching pairs obtained by the SIFT algorithm were subjected to RANSAC algorithm removal to eliminate mismatches. The removal results are as follows: Figure 11 As shown in (b).

[0232] Figure 11 (a) shows the original results of SIFT feature recognition, with a total of 138 feature matching pairs. It can be seen that although there are many matching pairs, there are a large number of mismatched pairs, which will seriously affect the accuracy of camera pose estimation and 3D point cloud generation in actual use. Figure 11 (b) shows the SIFT feature point pairs after RANSAC mismatch removal, totaling 65 pairs. Although the number of feature matching pairs in the image is significantly reduced after RANSAC processing, the remaining matching pairs are all correct matches. This shows that the RANSAC algorithm can effectively remove mismatch pairs.

[0233] S2.4, Camera relative pose calculation and spatial point determination: The camera relative pose is solved using epipolar constraints. Based on the obtained camera relative pose, the three-dimensional coordinates of feature points are obtained using triangulation. Three-dimensional point cloud reconstruction based on triangulation is then performed.

[0234] Camera calibration can obtain the relative pose relationships of a multi-camera group. Based on this, sparse point clouds can be generated using the principle of triangulation. Figure 12 Consider images I1 and I2 taken by two parallel cameras in different poses. Using the coordinate system of the left camera as the reference coordinate system, the transformation matrix of the right camera is T. Let the optical centers of the cameras be O1 and O2. Let p1 be the projection pixel of any 3D point P in space onto I1, and p2 be the projection pixel onto I2. Theoretically, lines O1p1 and O2p2 should intersect at point P in space. However, due to noise, lens distortion, and other factors, the two lines emanating from the optical centers often do not intersect in space. Therefore, a least-squares function can be constructed to solve the above process. Let x1 and x2 be the normalized coordinates of two feature points, then:

[0235] s1x1=s2Rx2+t. (18)

[0236] In equation (18), the camera pose R and position t are known. What needs to be solved is the depths s1 and s2 of the two feature points in their respective images. Multiplying both sides of the above equation by a factor... We can obtain:

[0237]

[0238] s2, i.e., the three-dimensional coordinates of the spatial point, can be obtained by equation (19).

[0239] S2.5, point cloud optimization, utilizes the Bundle Adjustment algorithm to simultaneously optimize camera pose and 3D coordinates of spatial points;

[0240] Due to factors such as camera parameter calibration errors and mismatches, the lines connecting the 3D coordinates of the same feature point to the optical centers of different cameras do not intersect at a single point in space. To minimize reprojection errors, it is necessary to simultaneously optimize the 3D point coordinates and camera pose so that the light rays emitted from any feature point eventually converge to the optical center, i.e., BA (Bundle Adjustment). Considering the camera observation model, the least squares method is used to establish the error generated in a single step of the above process:

[0241] e = h(x,y) - h(ξ,p) (20)

[0242] Where ξ represents the Lie group corresponding to the camera's true pose R and t at the current moment, p represents the true 3D coordinates of the landmark at the current moment, x represents the Lie group corresponding to the camera's current pose R and t, and y represents the 3D coordinates of the landmark p observed in state x. Considering the overall observation process, the cost function can be expressed as:

[0243]

[0244] Where z = h(x,y), z ij For the camera in pose ξ i Observation marker p j The data obtained.

[0245] Since equation (21) is a nonlinear function, to solve this equation, we add an increment Δx to the overall independent variable of the equation, and we have:

[0246]

[0247] Where E ij The partial derivative of the cost function with respect to the coordinates of the landmark's three-dimensional point in this state is:

[0248]

[0249] F ij Let the partial derivative of the cost function with respect to the camera pose in this state be:

[0250]

[0251] Solving the above equations using the LM (Levenberg-Marquardt) algorithm yields:

[0252] f(x+δx)≈f(x)+Jδx (25)

[0253] Where J is the Jacobian matrix obtained by taking the first-order partial derivative of the cost function, from the above equation we have J = [FE]. Minimizing the reprojection error and taking f(x) = ε, we have:

[0254] (J T J+λI)δx=-J T ε (26)

[0255] Solving the above equations allows for the simultaneous optimization of camera pose and 3D point coordinates.

[0256] Comparison and Analysis of Target 3D Point Cloud Reconstruction Algorithm Simulation Experiment: A multi-camera group consisting of four cameras was built in a simulation environment to observe and photograph a non-cooperative satellite, with background starlight added as interference. Camera parameters are shown in the calibration results above, and the camera attitudes in the world coordinate system are as follows: position data are shown in Table 4.

[0257]

[0258] Table 4 Spatial Location of Multi-Camera

[0259] The non-cooperative satellite model used was the Aura satellite model provided on NASA's official website. Multi-camera arrays were used to photograph the non-cooperative satellite, and SuperPoint feature extraction and SuperGlue feature point matching were performed on the images obtained from the multi-camera arrays. Figure 13 The confidence levels of the matching pairs are represented by blue, yellow, and red from low to high, indicating that the feature point pairs obtained by SuperPoint combined with the SuperGlue algorithm have high confidence levels.

[0260] After obtaining the feature point pairs, the RANSAC algorithm is used to remove mismatches. Based on the camera relative pose data in Table 4, triangulation is used for each feature point pair to obtain a sparse point cloud for non-cooperative satellites. Finally, BA is used to simultaneously optimize the camera pose and 3D point cloud coordinates to obtain the final sparse point cloud. Figure 14 The length and width of the sails in the obtained sparse point cloud have an error of 1% to 2% compared with the true values, indicating high reconstruction accuracy.

[0261] In S3, considering the complex environment of non-cooperative targets in space, the factors affecting the accuracy of pose recognition of non-cooperative targets in space mainly come from two aspects: edge interference points caused by complex backgrounds; and mismatch and point cloud generation problems caused by feature destruction under extreme lighting conditions.

[0262] To analyze the impact of cluttered backgrounds on non-cooperative target pose recognition, interference points are categorized into two types to construct the pose estimation model. The first type consists of distant interference points, which are far from the actual target and significantly affect the centroid of the target point cloud. Therefore, distant interference points need to be screened and removed before point cloud registration. The second type consists of near-point interference points that are close to or even in contact with the target. These interference points have a smaller impact on the centroid of the target point cloud, but because near-point interference points often maintain a fixed or even repulsive motion pattern compared to the target point cloud, they significantly affect the overall registration accuracy during point cloud registration.

[0263] Extreme lighting conditions, primarily characterized by both dim and bright environments, pose the most significant challenge to non-cooperative target pose recognition by disrupting the target's features. Since optical cameras rely heavily on target features for matching and point cloud generation, severe feature corruption leads to numerous mismatched points in the point cloud generated by multi-camera arrays. This significantly impacts the accuracy of target point cloud registration, necessitating further refinement of traditional point cloud registration algorithms.

[0264] S3.1: Probabilistic Point Cloud Generation. The point cloud generated in S2.5 of the sparse point cloud generation module is fused with the matching confidence obtained in S2.2 to obtain a probabilistic point cloud; point cloud registration is then performed using distance weights and confidence probabilities. The former eliminates the influence of nearby interference points on the point cloud registration process, while the latter reduces the influence of illumination interference. The overall algorithm flow framework is as follows: Figure 15 As shown.

[0265] S3.2: Outlier removal-based distant interference point elimination improves subsequent registration accuracy by removing outliers. Simultaneously, to increase the computational speed of the registration algorithm, the point cloud is downsampled to reduce its size. To perform preliminary processing of the source and target point clouds, a statistical filtering-based outlier elimination algorithm is employed. The point cloud set in the three-dimensional coordinate system can be represented as:

[0266] P = {p i ∈R 3} i＝1,2,3,... (27)

[0267] Where p i =(x i ,y i ,z i The coordinates of a point are set in a Cartesian coordinate system. For each 3D point, two steps are taken: first, the nearest set of points in its neighborhood is searched and the Eulerian distance is calculated; then, the standard deviation of the distance between that point and the closure point is calculated. In this way, outliers can be identified.

[0268]

[0269] Where d i The statistical distance between points is represented by k; k represents the number of selected neighboring points, which can be 20; m is the threshold for controlling the number of outliers; d std p i The standard deviation between p and its nearest neighbors. i If p j With p i The Euler distance between them is greater than d i Then p j Considered an outlier; such as Figure 16 As shown.

[0270] S3.3: Point cloud registration. The Probabilistic Nearest Iterative Point (PICP) algorithm is used to register point clouds between frames, obtaining the relative poses of non-cooperative targets. These poses are then compared with the ground truth poses to evaluate the algorithm's robustness and accuracy. Two point cloud models are given, namely the reference point cloud {p...} i} and target point cloud {p' i The pose estimation problem for a non-cooperative target can be transformed into finding the optimal transformation parameters that minimize the distance between the reference point cloud and the target point cloud. This allows us to obtain the rotation matrix R and translation vector t of the non-cooperative target.

[0271]

[0272] The distance error of a single point can be expressed as:

[0273] e i =p i -(Rp′ i +t) (30)

[0274] As can be seen from the above equation, the registration problem can be transformed into minimizing the sum of the Euclidean distances between the reference point cloud and the target point cloud errors, i.e., equation (31):

[0275]

[0276] Considering the impact of near interference points on the point cloud registration process, a distance weight is added to Equation (31) to reduce the impact of near interference points on the registration process. Therefore, the new loss function can be expressed as:

[0277]

[0278] Where w i Represents distance weights and w i ∈[0,1]. From the perspective of the algorithm itself, w i This indicates the degree to which the corresponding points should be given attention. Such point pairs qualify for highlighting during iteration. In other words, the loss function (32) reaches its minimum only when such point pairs achieve ideal registration. When wi When the maximum value is reached, the solution for this point pair is the same as the corresponding value in the original algorithm, which means that the point has been fully considered; conversely, if w i As the value of approaches 0, the error generated by the loss function for that point pair will decrease simultaneously. Therefore, such point pairs are more easily weakened and ignored. When it approaches zero (which is impossible to reach), the solution result for that corresponding point is close to zero, indicating that the point pair will not be considered at all and will not participate in the calculation.

[0279] To weaken near-interference points and reasonably control the weight w i The location of nearby interference points needs to be considered. For the model above, nearby interference points are usually located at the edges of the target, so the weights can be constructed using the distance to the centroid of the point cloud. For point pairs, the distances from the points in the source and target point clouds to the centroids are shown below:

[0280]

[0281] Where q μ and q′ μ Let represent the centroids of the reference point cloud and the target point cloud, respectively. Considering d pi and d qi As an independent variable, a two-dimensional Gaussian distribution function is chosen to evaluate the weight of each point, as shown in Equation (34). Non-cooperative targets such as satellites have roughly symmetrical shapes and uniform structures. Furthermore, the two-dimensional Gaussian function possesses rotational symmetry, meaning it is similar to analyzing smoothness in all directions. Therefore, the weights will not be biased to any one side, preventing registration failure. In addition, the Gaussian function is a single-valued function, and the weight of each point monotonically increases or decreases with distance from the centroid.

[0282]

[0283] Because of w i ∈[0,1], the value of σ can be set as Therefore w i It can be expressed as the following formula:

[0284]

[0285] Combining equation (32), the loss function considering the near-interference point can be obtained as follows:

[0286]

[0287] To address the issue of decreased accuracy in feature recognition algorithms due to feature degradation under extreme lighting conditions, the matching confidence score obtained from the SuperGlue algorithm is used as the confidence score of the point cloud. This confidence score is then incorporated as a weight in the point cloud registration process to reduce the impact of mismatches on the final pose estimation. This yields a new loss function:

[0288]

[0289] Where P i For probability weights and P i ∈[0,1]. Considering the matching confidence obtained by the SuperGlue algorithm, a higher matching confidence indicates that the pair of feature points has strong features in the image recognition process, and the 3D points obtained from the pair of feature points have higher reliability; conversely, a lower matching confidence indicates that the pair of feature points is more likely to be mismatched, and we hope to eliminate the influence of these points as much as possible during the registration process. Therefore, the probability weight P i It can be represented as:

[0290]

[0291] Where P(p i ) and P(p' i Let be the probability of the reference point and the probability of the target point, respectively. Therefore, considering the effects of complex backgrounds and extreme lighting, the loss function of the Probabilistic Nearest Iteration Point (PICP) algorithm can be obtained as follows:

[0292]

[0293] According to the above formula, the point cloud registration problem can be reformulated as a least squares problem. To simplify the calculation, the transformation of the point cloud can be cleverly converted into two steps: translation and rotation. In this sense, the motion of the rigid body should be regarded as the rotation of the source point cloud around its center of mass and the translation of the center of mass. Therefore, the centers of mass of two sets of point clouds are listed:

[0294]

[0295] Combining the above formula, we can obtain:

[0296]

[0297] Due to the cross term part (p i -pR(p i Since the summation of '-p' is zero, the optimization function can be expressed as:

[0298]

[0299] The centroid-free coordinates of the reference point cloud and the target point cloud are calculated as follows:

[0300] q i =p i -p,q′ i =p′ i -p′ (43)

[0301] The least squares problem can be optimized as follows:

[0302]

[0303] Expanding the error term with respect to R, we have:

[0304]

[0305] Remove terms irrelevant to R:

[0306]

[0307] Singular value decomposition of the above equation yields:

[0308]

[0309] have:

[0310]

[0311] Therefore:

[0312]

[0313] The above equation yields the rotation matrix and translation vector for point cloud registration. Compared to the traditional Inter-Closest Point Iterative Algorithm (ICP), the Probabilistic Inter-Closest Point Iterative Algorithm (PICP) proposed in this invention adds distance and probability weights to each point in the point cloud to address the complex backgrounds and strong lighting issues faced by non-cooperative target pose estimation in real-world environments.

[0314] The sources of error were identified as distant and near interference points caused by the starry background, and mismatch issues caused by extreme lighting conditions. To address the distant interference point problem, an outlier removal method was proposed to eliminate its impact on the pose measurement process. For near interference points and unavoidable mismatches, distance and probability weights were added to the traditional point cloud registration algorithm. In a simulation environment, a multi-camera group was built to photograph non-cooperative satellites and generate sparse point clouds. The pose measurement accuracy of the traditional ICP algorithm and the PICP algorithm was compared under three conditions: translation, rotation, and translation + rotation. Experiments showed that, under the non-cooperative satellite translation condition, although the observation error and fluctuation of the traditional ICP algorithm were far greater than those of the PICP algorithm; under the rotation condition, the traditional ICP algorithm completely failed, while the PICP algorithm could still perform accurate and stable pose measurement of non-cooperative targets; under the translation + rotation condition, the traditional ICP algorithm completely failed, and the error of the PICP algorithm increased significantly compared to the translation and rotation conditions, but the recognition results were still far superior to those of the ICP algorithm.

[0315] A multi-view vision-based spatial target reconstruction and pose estimation system includes a camera vision module, a sparse point cloud generation module, and a point cloud registration module. The camera vision module establishes an ideal camera model and distortion model, builds a multi-view camera vision system, and calibrates the intrinsic parameters of the simulated camera. The sparse point cloud generation module, based on the system established in S1, uses a deep convolutional neural network to extract and match feature points, performs multi-view vision spatial target 3D point cloud reconstruction using convolutional neural networks, and completes sparse point cloud generation. The point cloud registration module uses a non-cooperative target pose estimation based on the Probabilistic Nearest Point Iterative Algorithm (PICP).

[0316] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0317] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0318] The above provides a detailed description of the spatial target reconstruction and pose estimation method and system based on multi-view vision proposed in this invention, and elucidates the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A spatial target reconstruction and pose estimation method based on multi-view vision, characterized in that: The method specifically includes the following steps: S1. Establish an ideal camera model and distortion model, establish a multi-view camera vision system, and calibrate the intrinsic parameters of the simulation camera; S2, based on the system established in S1, uses a deep convolutional neural network to extract and match feature points, performs multi-view visual spatial target 3D point cloud reconstruction using a convolutional neural network, and completes sparse point cloud generation; S2.1, Image acquisition and preprocessing are performed, and the SuperPoint algorithm is used to extract the pixel positions of feature points and their corresponding descriptors; SuperPoint employs a semi-self-supervised training approach, with the training process consisting of three steps: generation of the basic extractor, self-labeling of feature points, and final network training and error evaluation. Generation of the base extractor: Using virtual 3D objects as the training set, the strong feature corner points of the virtual 3D objects are extracted, which prepares for subsequent self-calibration; Feature point self-annotation: Based on the obtained basic feature point extractor, self-annotation is performed on unlabeled general images; homography transformation is performed on unlabeled images to generate new images, and feature points are re-annotated using the basic extractor. After extraction, inverse transformation is used to generate the final self-annotated image. Final network training: By performing known transformations on the self-labeled images, the pose relationship between the two images can be obtained. Then, the two images are fed into the SuperPoint network for training. Since the relative pose relationship between the two images is known, error analysis can be performed on the finally extracted feature points and descriptors. S2.2, Feature point detection and matching: The SuperGlue algorithm is used to match the descriptors obtained in S2.1 to obtain the final feature point matching pairs and matching confidence. S2.3, False Match Removal: The RANSAC algorithm is used to remove false match pairs from S2.

2. Given a dataset of size K', find the minimum number of parameters required to solve the model. The steps of the RANSAC algorithm are as follows: S2.3.1 Random sampling from the dataset Use 1 point as the initial model solution dataset; S2.3.2 Based on the sampling in the previous step A possible model for the dataset can be fitted using points; S2.3.3 Substitute other points from the dataset into the model to calculate the error. When the error is less than the threshold... Py When this point is in the interior, then this point is considered an interior point. S2.3.4 Repeat Next, select the set with the most interior points and refit a new possible model; S2.3.5 Repeat the above steps until the specified number of iterations is reached, then end the algorithm; To achieve the desired accuracy, different numbers of iterations are required depending on the outlier count. For any dataset, the number of iterations can be expressed as: （17） in w 1 represents the probability that a point randomly selected from the dataset is an interior point. This represents the probability of a successful sampling. S2.4, Camera relative pose calculation and spatial point determination: The camera relative pose is solved using epipolar constraints. Based on the obtained camera relative pose, the three-dimensional coordinates of feature points are obtained using triangulation. Three-dimensional point cloud reconstruction based on triangulation is then performed. S2.5, point cloud optimization, utilizes the Bundle Adjustment algorithm to simultaneously optimize camera pose and 3D coordinates of spatial points; S3, Non-cooperative target pose estimation based on the Probabilistic Nearest Point Iterative Algorithm (PICP); S3.1: Probabilistic point cloud generation. The point cloud generated in S2.5 is fused with the matching confidence obtained in S2.2 to obtain the probabilistic point cloud. S3.2: Distant interference point elimination based on outlier removal: Outliers are removed to improve the subsequent registration accuracy; at the same time, in order to improve the operation speed of the registration algorithm, the point cloud is downsampled to reduce the size of the point cloud; To perform preliminary processing of the reference and target point clouds, an outlier elimination algorithm based on statistical filtering was adopted; the point cloud set in the three-dimensional coordinate system can be represented as: （27） in These are the three-dimensional coordinates of a point set in the Cartesian coordinate system; For each 3D point, two steps are taken: first, search for the nearest set of points in its neighborhood and calculate the Eulerian distance; then, calculate the standard deviation of the distance between that point and the closed point. This method can be used to determine whether a point is an outlier. （28） in Represents the statistical distance between points; Indicates the number of selected adjacent points; This refers to the threshold for controlling the number of outliers; express Standard deviation between the standard deviation and the nearest neighbor; for ,if and The Euler distance between them is greater than ,but It is considered an outlier.

2. The method according to claim 1, characterized in that: In S1, S1.1, Modeling of a monocular camera vision system, establishing an ideal camera model and a camera distortion model; The optical camera uses a pinhole camera model and employs the pinhole imaging principle to translate a two-dimensional inverted image of a three-dimensional object onto an artificial plane in front of the camera lens. P x Above, a positive image is formed; in the pinhole camera imaging model, the transformation of four coordinate systems is involved, namely the world coordinate system, the camera coordinate system, the physical imaging coordinate system, and the image pixel coordinate system; The transformation process between the world coordinate system and the camera coordinate system involves the pose of the camera in the world coordinate system. These are also known as the camera's extrinsic parameters. In monocular vision experiments, the world coordinate system is aligned with the camera coordinate system to avoid calculating the extrinsic parameters. In binocular vision experiments, the world coordinate system is aligned with the left camera's coordinate system to reduce computation. The transformation process between the camera coordinate system and the physical imaging coordinate system can be described as a scaling process of similar triangles, assuming that the physical imaging plane is parallel to the object-image plane and perpendicular to the camera's optical axis. Three-dimensional points in space The coordinates in the physical imaging plane can be represented as: （1） The transformation relationship between the physical imaging plane coordinate system and the image pixel coordinate system is translation and scaling. Since the origin of the physical plane coordinate system is at the camera's optical center, and the origin of the image pixel coordinate system is at the top left corner of the image, The axis points in the column direction of the image. The axis points in the row direction of the image. Assume the distance the camera optical center has been translated from the image pixel coordinate system is... The distance unit is m. According to equation (1), the point The coordinates in the image pixel coordinate system are: （2） At the same time, the physical imaging coordinate system and the image pixel coordinate system have a scaling relationship, as shown in equation (2). The coordinate system uses m as the unit of measurement, while a digital image is a matrix with the unit of pixel; assuming the image pixel coordinate system is in... Scaled on axis times, in Scaled on axis times, of which and The images are respectively in shaft and The length of a single pixel on the axis can be obtained according to equation (2): （3） （4） In summary, the transformation relationship between the world coordinate system and the image pixel coordinate system can be obtained as follows: （5） in This is the scaling factor. The internal parameters of a camera, which can be abbreviated as: （6） Formula (5) is the transformation relationship between the image pixel coordinate system and the world coordinate system of the distortion-free camera in the ideal model; distortion parameters are introduced to eliminate errors caused by real-world processes; Camera distortion can be divided into tangential distortion and radial distortion. Tangential distortion has little effect on the image imaging process, while radial distortion has a significant impact on the final image. Therefore, only radial distortion needs to be corrected to obtain the distortion-free image required for subsequent experiments. Considering radial distortion, a radial distortion parameter is introduced into the ideal pinhole imaging model to eliminate the effect of this distortion. Since the distortion occurs during imaging, it is assumed that the transformation between the world coordinate system and the camera coordinate system is as follows: for any point in the world coordinate system... ,have: （7） in For point In the camera coordinate system, let the coordinates be... The distance to the origin of the coordinate system is ,but Using polynomials for this Distortion removal can be achieved through: （8） In equation (8) , , For radial distortion parameters; Although the tangential distortion of the camera can be replaced with 0 in actual experiments, it is also calculated during camera calibration. The camera's distortion parameters are: （9） , These are tangential distortion parameters; From equations (5) and (9), it can be seen that, considering camera distortion, the transformation matrix between the real-world coordinate system and the image pixel coordinate system is calculated in total. There are nine parameters. The first four parameters are collectively referred to as camera intrinsic parameters, and the last five parameters are collectively referred to as distortion parameters. S1.2, Multi-view camera vision system modeling; In an ideal binocular vision measurement model, the imaging planes of the two cameras are coplanar, their optical axes are parallel, and they have the same intrinsic parameters. Their extrinsic parameters only include translation along the X-axis, with no rotation. The coordinate systems of the left and right cameras are respectively... and The line connecting the origins of the two camera coordinate systems is called the baseline. This indicates that, under ideal conditions, the camera's optical axis passes through the center of the image pixel coordinate system. Let an arbitrary point in space be given. The coordinates are spatial point Image pixel coordinate system of left and right cameras and The projection points on are respectively and The corresponding coordinates in the physical imaging coordinate system are as follows: and According to the principle of similar triangles, we have: （10） in, The focal length of the camera; Substituting the transformation relationship between the physical imaging coordinate system and the image pixel coordinate system, we have: （11） From this we can obtain The coordinates of the point are: （12） in The difference in the horizontal coordinates of the left and right cameras is called parallax. By introducing several cameras of the same specifications into the binocular stereo vision measurement model, a multi-camera observation model can be formed. Any two cameras in the multi-camera observation model can be considered as a pair of binocular stereo vision measurement models. Therefore, for... For a multi-camera observation model with multiple cameras, there are a total of A combination of binocular stereo measurement models.

3. The method according to claim 2, characterized in that: S1 also includes: S1.3, Simulation camera intrinsic parameter calibration; Solving the intrinsic parameters of the camera in the simulation environment; The selected simulation camera model is an OpenGL perspective camera model, and its calibration parameters are related to those of an ideal pinhole camera model as follows: （16） in The field of view is in the pitch direction; aspect The aspect ratio is 4x4; This is the near-plane distance, in meters (m). This is the distance to the far plane, in meters. For camera internal parameters; The intrinsic parameters of the simulated camera under the current parameters can be calculated using Equation (16). Since this environment is a simulation environment, it can be regarded as an ideal camera model, so there is no distortion, and both radial distortion and tangential distortion are 0.

4. A spatial target reconstruction and pose estimation system based on multi-view vision, characterized in that: The system is used to execute the spatial target reconstruction and pose estimation method based on multi-view vision as described in any one of claims 1 to 3; The system includes a camera vision module, a sparse point cloud generation module, and a point cloud registration module. The camera vision module is used to establish an ideal camera model and distortion model, establish a multi-view camera vision system, and calibrate the intrinsic parameters of the simulation camera. Image acquisition and preprocessing are performed, and the SuperPoint algorithm is used to extract the pixel positions of feature points and their corresponding descriptors. SuperPoint employs a semi-self-supervised training approach, with the training process consisting of three steps: generation of the basic extractor, self-labeling of feature points, and final network training and error evaluation. Generation of the base extractor: Using virtual 3D objects as the training set, the strong feature corner points of the virtual 3D objects are extracted, which prepares for subsequent self-calibration; Feature point self-annotation: Based on the obtained basic feature point extractor, self-annotation is performed on unannotated general images; The unlabeled image is subjected to homography transformation to generate a new image. The basic extractor is used to label the feature points. After extraction, the inverse transformation is used to generate the final self-labeled image. Final network training: By performing known transformations on the self-labeled images, the pose relationship between the two images can be obtained. Then, the two images are fed into the SuperPoint network for training. Since the relative pose relationship between the two images is known, error analysis can be performed on the finally extracted feature points and descriptors. Feature point detection and matching: The SuperGlue algorithm is used to match the preprocessed descriptors to obtain the final feature point matching pairs and matching confidence scores. False match removal: The RANSAC algorithm is used to remove false match pairs in feature point detection and matching. Given a dataset of size K', find the minimum number of parameters required to solve the model. The steps of the RANSAC algorithm are as follows: Random sampling from the dataset The initial dataset is obtained by using 10 points as the initial model solution; based on the samples from the previous step... A possible model is fitted to each point in the dataset; the error is calculated by substituting other points from the dataset into the model, and when it is less than a threshold... Py When this point is in the interior, then this point is considered an interior point. repeat Next, select the set with the most interior points and refit a new possible model; Repeat the above steps until the specified number of iterations is reached, then terminate the algorithm. To achieve the desired accuracy, different numbers of iterations are required depending on the outlier count. For any dataset, the number of iterations can be expressed as: （17） in w 1 represents the probability that a point randomly selected from the dataset is an interior point. This represents the probability of a successful sampling. Camera relative pose calculation and spatial point determination: The camera relative pose is solved using epipolar constraints. Based on the obtained camera relative pose, the 3D coordinates of feature points are obtained using triangulation. 3D point cloud reconstruction based on triangulation is then performed. Point cloud optimization utilizes the Bundle Adjustment algorithm to simultaneously optimize camera pose and 3D coordinates of spatial points. The sparse point cloud generation module, based on the system established by the camera vision module, uses a deep convolutional neural network to extract and match feature points, performs multi-view vision spatial target 3D point cloud reconstruction using a convolutional neural network, and completes sparse point cloud generation. The point cloud registration module performs non-cooperative target pose estimation based on the probabilistic nearest point iterative algorithm. Probabilistic point cloud generation involves fusing the generated point cloud with the obtained matching confidence score to obtain a probabilistic point cloud. The method of eliminating distant interference points based on outlier removal removes outliers to improve the accuracy of subsequent registration. Meanwhile, in order to improve the computation speed of the registration algorithm, the point cloud is downsampled to reduce the size of the point cloud; To perform preliminary processing of the reference and target point clouds, an outlier elimination algorithm based on statistical filtering was adopted; the point cloud set in the three-dimensional coordinate system can be represented as: （27） in These are the three-dimensional coordinates of a point set in the Cartesian coordinate system; For each 3D point, two steps are taken: first, search for the nearest set of points in its neighborhood and calculate the Eulerian distance; then, calculate the standard deviation of the distance between that point and the closed point. This method can be used to determine whether a point is an outlier. （28） in Represents the statistical distance between points; Indicates the number of selected adjacent points; This refers to the threshold for controlling the number of outliers; express Standard deviation between the standard deviation and the nearest neighbor; for ,if and The Euler distance between them is greater than ,but It is considered an outlier.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 3.

Images