Method for refining pose of moving body by adaptive sliding window incremental adjustment with variable scale

Abstract

The application discloses a variable-scale moving body pose refinement method based on adaptive sliding window incremental adjustment, which comprises the following steps: acquiring high-frame-frequency sequence images of a variable-scale moving body; performing incremental Schur complement edge-based bundle adjustment on the sequence images in an optimal sliding window; optimizing the continuous pose based on a weighted pseudo-observation value constraint model; and outputting the refined pose parameters. The application has high flexibility, relies on an airborne moving platform to realize large field observation, has high efficiency and strong adaptability, balances the calculation efficiency and precision through an adaptive sliding window, processes data quickly, has high precision, improves the variable-scale moving body pose refinement precision with the aid of an optimal window, Schur complement edge-based bundle adjustment and weighted constraint, and is suitable for deep space exploration and other scenes.

Description

Technical Field

[0001] This invention relates to the field of motion body pose refinement technology, and in particular to an adaptive sliding window incremental adjustment method for variable-scale motion body pose refinement. Background Technology

[0002] When variable-scale moving bodies, such as landers and flyby vehicles, conduct deep space exploration missions to the Moon and Mars, high-precision, high-dynamic attitude testing technology is required to accurately acquire the position and attitude parameters of the spacecraft during critical processes such as descent, landing, flyby, and recovery, providing a basis for performance optimization and safety assessment. Therefore, this invention employs high-speed video measurement theory and methods from an airborne moving platform to conduct research, pioneering a new model for high-dynamic, high-precision testing of high-speed moving bodies. The research results are of great significance for ensuring the safe landing of my country's lunar and Mars exploration missions and the development of next-generation missile weapons and equipment.

[0003] Accurate continuous pose estimation for high-speed video measurement of moving bodies at varying scales is a complex and challenging task, especially under conditions of high-speed movement, large field of view, and complex backgrounds. Bundle adjustment is an advanced technique for globally optimizing the intrinsic and extrinsic parameters of sequential images and the 3D coordinates of target points. It improves the accuracy of moving body pose estimation through globally consistent parameter calculation. However, it suffers from drawbacks such as high computational complexity in large-scale scenes and dependence on initial values. To address this issue, scholars at home and abroad have proposed a series of improved methods, such as incremental bundle adjustment, sparse bundle adjustment, and sliding window bundle adjustment. Incremental bundle adjustment optimizes only the newly added camera pose and 3D point coordinates each time, resulting in high computational efficiency, but it suffers from problems such as accumulated errors, poor global consistency, and sensitivity to initial values. Sparse bundle adjustment can optimize all image poses and 3D point coordinates simultaneously, offering advantages in global consistency and high accuracy, but the implementation and optimization of sparse matrix techniques are relatively complex. Sliding window bundle adjustment optimizes only the data within the image window, reducing error accumulation and balancing computational complexity and accuracy. However, a window that is too small leads to insufficient optimization, while a window that is too large increases computational complexity, and data outside the window does not participate in the optimization, resulting in limited global consistency. Although the above methods can improve the computational efficiency of massive sequence images, the high-speed video measurement of the moving platform of the deep space probe and other variable-scale landing objects studied in this invention involves drastic changes in the imaging scale and rotation of the moving objects, making it difficult for the above methods to accurately calculate the continuous pose of variable-scale moving objects. Therefore, an adaptive sliding window incremental adjustment method for variable-scale moving object pose refinement is needed. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body, so as to solve the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0006] On one hand, the present invention includes the following steps:

[0007] Step 1: Acquire high frame rate sequence images of a variable-scale moving object.

[0008] The optimal sliding window is determined based on an adaptive weighted sliding window calculation model. The model includes: determining a basic sliding window based on the average values ​​of motion speed, scene complexity, and accuracy requirements in the sequence images; calculating adaptive weight coefficients based on the skewness of motion speed, scene complexity, and accuracy requirements in the sequence images and the basic window; and incrementally optimizing the basic sliding window into the optimal window.

[0009] Step 2: Perform incremental Shure complement marginal bundle adjustment on the sequence of images within the optimal sliding window, including:

[0010] A dual constraint criterion of variable contribution and redundancy is constructed. The variable contribution of the object point is calculated using the Frobenius norm of the Jacobian matrix, and the extrinsic parameters of the discard window camera are determined using the trace of the information matrix. The redundancy of the object point is calculated using the rank of the Jacobian matrix, and the extrinsic parameters of the discard window camera are calculated using the condition number of the information matrix.

[0011] Marginalization variables are dynamically selected based on the aforementioned constraint criteria. The constraint information outside the window is preserved through Schur complement marginalization, and the continuous pose of the moving body is calculated in combination with the PnP method.

[0012] The continuous pose is optimized based on a weighted pseudo-observation constraint model, and the refined pose parameters are output. The specific technical solution is as follows:

[0013] First, construct the adaptive weighted sliding window calculation model and determine the base window:

[0014]

[0015] Where K is the basic sliding window, V max ,S max ,P max These represent the maximum values ​​of V, S, and P in the image sequence, respectively. α represents the mean values ​​of V, S, and P within the base window K, and α, β, and γ are adaptive weighting coefficients for adjusting V, S, and P. These represent the mean values ​​of V, S, and P in the image sequence, respectively. This represents assigning equal weights to the three influencing factors and normalizing them, V i ,S i ,P i The motion speed, scene complexity, and accuracy requirements of the i-th frame in the sequence image are determined respectively.

[0016] Furthermore, using the basic sliding window K as the initial window, an optimal sliding window W is adjusted through adaptive weighting, and bundle adjustment is performed within the optimal sliding window W. After the current window optimization is completed, the optimal sliding window W is updated based on the basic sliding window K, and this process is repeated incrementally until the optimization of the entire image sequence is completed. The formulas for the adaptive adjustment weight coefficients of V, P, and S skewness in the image sequence are as follows:

[0017]

[0018]

[0019] Where M represents the total number of image frames within window K, N is the sum of the sequence images, and ò is a given small smoothing factor to prevent the denominator from being 0; V j ,S j ,P j These represent the motion speed, scene complexity, and accuracy requirement of the j-th frame within the basic sliding window K, respectively.

[0020] Furthermore, in the Schur complement marginal adjustment, the error equation is transformed to obtain a refined model of the moving body pose parameters, as follows:

[0021]

[0022] in, This represents the object-side coordinates and camera extrinsic parameters after refinement by the sliding window W. This represents the discarded camera extrinsic parameters and object point 3D coordinates within the window;

[0023] By using Gaussian elimination, we can finally obtain... for:

[0024]

[0025] The obtained camera extrinsic parameters and target object 3D coordinates are used to calculate the continuous pose of the variable-scale moving body using the PnP (Perspective-n-Point) method.

[0026] Furthermore, the formula for constructing a pose optimization model with weighted pseudo-observation constraints is as follows:

[0027]

[0028] Where C is the observation matrix, determined according to the parameter weights; V ps These are pseudo-observations, which can be determined using prior information; I is the identity matrix, P... ps This represents the weighting coefficient, which is assigned different weights based on the influence of the parameters during weighting. Since the moving body is descending in the Z direction, it is assigned the weight P, which is the smallest in Z. z=0.00000000001, the rest are P x =P y =P φ =P ω =P κ =0.1, l ps It is observation noise.

[0029] The beneficial effects of this invention are:

[0030] This invention is an adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body. Compared with the prior art, this invention has the following technical advantages:

[0031] This invention offers high flexibility, enabling large field-of-view observations via an airborne moving platform; it is highly efficient and adaptable, balancing computational efficiency and accuracy through an adaptive sliding window for rapid data processing; and it boasts high accuracy, enhancing the precision of pose refinement for variable-scale moving bodies through optimal windows, Shur complement marginalization, and weighted constraints, making it suitable for scenarios such as deep space exploration. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the adjustment system framework for an adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body according to the present invention. Detailed Implementation

[0033] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0034] like Figure 1 As shown, an adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body includes the following steps:

[0035] Step one involves using an adaptive weighted sliding window to calculate the optimal sliding window size, ensuring that the adjustment is performed within a local time window, thus guaranteeing the accuracy and efficiency of the adjustment.

[0036] Step 2: By using the incremental Schur complement marginalization bundle adjustment technique, a dual constraint standard of variable contribution and redundancy is constructed. Variables that need to be marginalized are dynamically selected, and Schur complement marginalization is performed within the optimal time window. Adjustment optimization is then performed to solve the problem of limited global consistency of continuous pose caused by discarding variable information of image frames leaving the window when the window slides.

[0037] Step 3: Use weighted pseudo-observation constraints to obtain the optimal pose of the deep space probe's variable-scale moving body. (1) Adaptive weighted sliding window calculation

[0038] During the lander's descent, adjustment is needed to optimize camera pose and object coordinates. However, batch optimization methods like adjustment become increasingly computationally inefficient. Therefore, it's crucial to control the adjustment scale to maintain real-time computation and ensure both efficiency and accuracy by performing the adjustment optimization within a local adjustment window. This paper employs a sliding window incremental adjustment strategy, fixing the adjustment within a single time window and retaining only the most recent landing image frame while discarding earlier frames. A key challenge is fixing the adjustment within a specific time window—the selection of the sliding window size. Therefore, the choice of sliding window size is critical for accurate estimation of the continuous pose of a variable-scale moving object. To address this challenge, an adaptive weighted sliding window computation model is proposed, comprehensively considering multiple influencing factors such as motion velocity V, scene complexity S, and accuracy requirements P. The specific steps are as follows:

[0039] 1) Adaptive weighted sliding window calculation model: An adaptive weighted sliding window calculation model is constructed from a series of high frame rate image sequences as input.

[0040]

[0041] Where K is the basic sliding window, V max ,S max ,P max These represent the maximum values ​​of V, S, and P in the image sequence, respectively. Let V, S, and P represent the mean values ​​within the base window K, and α, β, and γ be the adaptive weighting coefficients that adjust V, S, and P. Therefore, the key to this model lies in how to accurately calculate the base sliding window K and the adaptive weighting coefficients α, β, and γ.

[0042] 2) Basic sliding window K: When sliding on a sequence of images, a minimum value W of the sliding window needs to be given. min To ensure smooth window sliding, the three influencing factors—motion speed V, scene complexity S, and accuracy requirement P—are assigned equal weights, and their average is used to determine the calculation model for the basic sliding window K:

[0043]

[0044] in, These represent the mean values ​​of V, S, and P in the image sequence, respectively. This represents assigning equal weights to the three influencing factors and normalizing them, V i ,S i ,P i The motion speed, scene complexity, and accuracy requirements of the i-th frame in the sequence image are determined respectively.

[0045] 3) Adaptive weighting coefficients α, β, γ: Calculating the adaptive weighting coefficients is a crucial step in ensuring that the basic sliding window K is optimized and updated to the optimal sliding window W. Skewness has the advantage of intuitively revealing the distribution of the three influencing factors V, P, and S around the mean. Therefore, the weighting coefficients are adaptively adjusted according to the skewness of V, P, and S in the image sequence.

[0046]

[0047] Where M represents the total number of image frames within window K, N is the sum of the image sequences, and ò is a given small smoothing factor to prevent the denominator from being 0. V j ,S j ,P j These represent the motion speed, scene complexity, and accuracy requirement of the j-th frame within the basic sliding window K, respectively.

[0048] Finally, using the basic sliding window K as the initial window, an optimal sliding window W is adjusted through adaptive weighting, and bundle adjustment is performed within W. After optimizing the current window, W is updated based on K, incrementally increasing until the optimization of the entire image sequence is completed, thereby ensuring the accuracy and efficiency of the calculation.

[0049] When calculating the size of the adaptive weighted sliding window, the computational efficiency of batch optimization methods like adjustment continuously decreases when using controlled adjustment to optimize camera pose and object coordinates. However, the adaptive weighted sliding window calculation only needs to comprehensively consider influencing factors such as motion speed V, scene complexity S, and accuracy requirements P, controlling the adjustment calculation within the optimal window for the current time period. It then incrementally updates the base window to the optimal window, sequentially completing the optimization of the entire image sequence while maintaining efficiency during adjustment optimization. By comprehensively processing multiple frames of images using the sliding window technique, the complexity of independent calculations for each frame is reduced, enabling rapid processing of large amounts of image data. It is highly adaptable to various environments, improving the real-time performance and reliability of monitoring.

[0050] (2) Incremental Schur complement marginalized bundle adjustment

[0051] When deleting an old image frame from the window, directly discarding the variables related to the image frame leaving the window would result in information loss. These discarded variables might contain constraints more important to adjacent frames, and direct discarding would obviously break these constraints. Furthermore, since the old image frame to be deleted is not isolated and observes the same object points as other frames, discarding some variables would make the entire problem no longer sparsity. Therefore, this invention uses Schur complement elimination to marginalize variables and modifies the marginalization process. When marginalizing an old image frame, the observed object points are simultaneously marginalized. The remaining object points are then converted into co-observational information among the remaining image frames, thus maintaining the sparse structure of the matrix.

[0052] Therefore, an adaptive Shur complement marginalization adjustment strategy with dual constraints of variable contribution and redundancy is constructed to dynamically select the variables that need to be marginalized, thereby solving the problem of limited global consistency of continuous pose caused by discarding variable information of image frames leaving the window during window sliding.

[0053] 1) Dual Constraint Criterion of Variable Contribution and Redundancy: Construct a Shure complement marginalization variable selection criterion that discards camera extrinsic parameters T and object point 3D coordinates X within the window based on dual constraints of variable contribution and redundancy. The key to this criterion lies in how to accurately calculate the variable contribution Q and redundancy H of variables T and X:

[0054]

[0055] Wherein, G1 and G2 are the selection criteria for X and T, respectively, and γ1 and γ2 are the thresholds corresponding to G1 and G2, respectively. If they are less than a certain threshold, they are discarded.

[0056] Variable contribution Q primarily addresses the relationship between marginalized variable information and overall information. The Jacobian matrix Frobenius norm is used to represent the variable contribution of the object point X in the discard window. The variable contributions of the camera extrinsic parameter T in the drop-window are determined by the diagonal elements of the information matrix.

[0057]

[0058] in, It is the object point X j The Jacobian matrix observed in the i-th frame of the camera. The Jacobian matrix is ​​the sum of all camera frames that observed the object point, ||.|| F Denotes the Frobenius norm of the Jacobian matrix. It is the camera pose T i The matrix block corresponding to the complete information matrix, trace(.) represents the trace of the matrix, and the information matrix is ​​calculated using the Jacobian matrix.

[0059] Variable redundancy H: This mainly addresses whether marginalized variables contain repetitive or low-value information during pose refinement. The rank of the Jacobian matrix is ​​used to represent the redundancy of the object point X in the discard window. The condition number of the information matrix represents the redundancy of the discard window camera extrinsic parameters.

[0060]

[0061] Where rank(.) denotes the rank of the Jacobian matrix, and λ max ,λ minThese represent the maximum and minimum eigenvalues ​​of the information matrix. It represents the combined Jacobian matrix of the object point under all camera views.

[0062] In summary, the above method can dynamically select high-quality marginalization variables, providing constraints for the marginalization adjustment within the sliding window W using the Schul complement.

[0063] 2) Shure complement marginalization adjustment: Definition To solve the collinearity error equation, the coefficient matrix B is divided into four parts, Λ N Store the object coordinates and camera extrinsic parameters at the edge of the discard window N. W-N The camera extrinsic parameters within the sliding window W are stored, and Λ stores the constraint information of the discarded window and the sliding window. The error equation is transformed to obtain a refined model of the moving body pose parameters, as follows:

[0064]

[0065] in, This represents the object-side coordinates and camera extrinsic parameters after refinement by the sliding window W. This represents the discarded camera extrinsic parameters and the 3D coordinates of the object point within the window.

[0066] By using Gaussian elimination, we can finally obtain... for:

[0067]

[0068] Finally, using the camera extrinsic parameters and the 3D coordinates of the target object obtained by the above method, the continuous pose of the variable-scale moving body is calculated by the PnP (Perspective-n-Point) method.

[0069] (3) Weighted pseudo-observation constrained pose optimization: Since the extrinsic parameters change little within each frame of the sliding window W, there is a significant correlation, which may lead to ill-conditioned normal equations in the adjustment. Therefore, a weighted pseudo-observation constrained pose optimization model is constructed to reduce its correlation:

[0070]

[0071] Where C is the observation matrix, determined according to the parameter weights; V ps These are pseudo-observations, which can be determined by setting prior information or weak (strong) constraint parameters to relatively small (large) values; I is the identity matrix, P... ps This represents the weighting coefficient, which is assigned different weights based on the influence of the parameters during weighting. Since the moving body is descending in the Z direction, it is assigned the weight P, which is the smallest in Z. z =0.00000000001, the rest are P x =P y =Pφ =P ω =P κ =0.1, l ps It is observation noise.

[0072] Techniques such as optimized sliding window, incremental Shur complement marginalized bundle adjustment, and weighted pseudo-observation constrained pose optimization control the adjustment within the optimized sliding window, ensuring that the obtained camera pose and object coordinates are optimal, rather than globally optimized. Subsequently, a dual constraint criterion of variable contribution and redundancy is used to select marginalized variables, dynamically choosing high-quality marginalized variables to provide constraints for Shur complement marginalized adjustment within the sliding window. A weighted pseudo-observation constrained pose optimization is constructed, assigning different weights based on parameter influence; specifically, the Z-image is largest during the detector descent and is assigned a smaller weight. In summary, this ensures high accuracy in detector pose measurement, and the obtained detector pose is the optimized optimal pose, enabling continuous pose robustness calculation for the landing, flyby, and recovery processes of the airborne moving platform's large field-of-view observation detector.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for refining the pose of a variable-scale moving body using adaptive sliding window incremental adjustment, characterized in that, Includes the following steps: Step 1: Acquire high frame rate sequence images of a variable-scale moving object. The optimal sliding window is determined based on an adaptive weighted sliding window calculation model. The model includes: determining a basic sliding window based on the average values ​​of motion speed, scene complexity, and accuracy requirements in the sequence images; calculating adaptive weight coefficients based on the skewness of motion speed, scene complexity, and accuracy requirements in the sequence images and the basic window; and incrementally optimizing the basic sliding window into the optimal window. Step 2: Perform incremental Shure complement marginal bundle adjustment on the sequence of images within the optimal sliding window, including: A dual constraint criterion of variable contribution and redundancy is constructed. The variable contribution of the object point is calculated by the Frobenius norm of the Jacobian matrix, and the extrinsic parameters of the discard window camera are determined by the trace of the information matrix. The redundancy of the object point is calculated by the rank of the Jacobian matrix, and the extrinsic parameters of the discard window camera are calculated by the condition number of the information matrix. Marginalization variables are dynamically selected based on the aforementioned constraint criteria. The constraint information outside the window is preserved through Schur complement marginalization, and the continuous pose of the moving body is calculated in combination with the PnP method. The continuous pose is optimized based on a weighted pseudo-observation constraint model, and the refined pose parameters are output. Construct an adaptive weighted sliding window computation model: in, It is a basic sliding window. Representing sequential images The maximum value, Represents the basic window Inside The mean, It is regulation Adaptive weighting coefficients, Representing sequential images mean This means assigning equal weights to the three influencing factors and normalizing them. The first in the sequence of images Frame motion speed, scene complexity, and accuracy requirements; With basic sliding window The initial window is adjusted to the optimal sliding window using adaptive weights. and in the optimal sliding window Perform bundle adjustment within the inner window; after optimizing the current window, then adjust based on the basic sliding window. Update the optimal sliding window The updates are incrementally added until the optimization of the entire image sequence is completed. The formula for the skewness adaptive adjustment weight coefficient is: in, Representative window Total number of frames in the video, It is the sum of the sequence images. It is a given small smoothing factor to prevent the denominator from being 0.

2. The adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body according to claim 1, characterized in that, In the Schur complement marginal adjustment, the error equation is transformed to obtain a refined model of the pose parameters of the moving body, as follows: in, Represents a sliding window Refined object point coordinates and camera extrinsic parameters This represents the discarded camera extrinsic parameters and object point 3D coordinates within the window; By using Gaussian elimination, we can finally obtain... for: The obtained camera extrinsic parameters and target object 3D coordinates are used to calculate the continuous pose of the variable-scale moving body using the PnP method.

3. The adaptive sliding window incremental adjustment method for refining the pose of a variable-scale moving body according to claim 1, characterized in that, Formula for constructing a pose optimization model with weighted pseudo-observation constraints: in, It is the observation matrix, determined based on the parameter weights; These are spurious observations and can be determined using prior information; It is the identity matrix. This represents the weighting coefficient, which is assigned different weights based on the influence of the parameters during weighting. This is because the moving body has different directions... Landing, giving Minimum weight The rest are , It is observation noise.

Images