Method and system for automatically calibrating internal and external parameters of satellite-borne binocular camera vibration test system

Abstract

The invention provides an automatic calibration method and system for internal and external parameters of a satellite-borne binocular camera vibration test system. The method comprises the following steps: extracting and matching feature points of image sequences of a left camera and a right camera; calculating basic matrixes of the left camera and the right camera and a basic matrix between the left camera and the right camera based on the matched feature point pairs; according to the basic matrixes of the left camera and the right camera, an objective function is constructed based on the mapping relation between the essential matrix and the camera internal reference, and an IAO algorithm is adopted to solve the optimal solution of the camera internal reference; calculating a camera extrinsic parameter candidate solution and a projection matrix candidate solution according to the basic matrix between the left camera and the right camera and the camera internal parameters, screening a correct projection matrix through triangularization, and taking the corresponding extrinsic parameters as initial values of the camera extrinsic parameters; and solving the optimal extrinsic parameter of the camera by minimizing the re-projection error and the translation vector modulus length error. The method does not depend on known coordinate characteristics of the earth surface or structural characteristics of a satellite body, does not need a calibration board and manual operation, and can realize automatic real-time high-precision calibration of internal and external parameters of a satellite-borne binocular camera system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of binocular camera calibration, and particularly relates to a method for automatically calibrating internal and external parameters of a spaceborne binocular camera vibration test system and a system for automatically calibrating internal and external parameters of a spaceborne binocular camera vibration test system. BACKGROUND

[0002] With the development of space technology, spaceborne cameras have become the core sensors of the structural health monitoring system of on-orbit spacecraft, providing basic data support for the vibration monitoring of key structures such as spacecraft flexible antennas, flexible solar sails, thin-walled shells, and the precise control of satellite mechanical arms. The accuracy of the vision-based vibration measurement system is highly dependent on the accuracy of the camera internal and external parameter calibration. However, factors such as the intense vibration during the launch of the spacecraft, the extreme temperature difference in space, and the long-term on-orbit operation may cause the camera optical components to shift and the structural support to deform, thereby causing the misalignment of the camera internal and external parameters, and seriously affecting the reliability of the vibration measurement results. Therefore, there is an urgent need for a camera parameter automatic calibration technology suitable for space environment without human intervention.

[0003] Currently, the mainstream camera calibration methods are as follows:

[0004] Traditional calibration object-dependent method (such as Chinese patent application CN202411572461.6): a specific pattern such as a circular marker point calibration board is needed, and the parameters are solved by manually adjusting the position of the calibration board and cooperating with the optimization algorithm. This method is completely ineffective in space scenarios: first, it is impossible to deploy a calibration board in space and there is a lack of manual operation conditions; second, the calibration board is prone to deformation under extreme thermal cycling, resulting in distortion of the calibration reference; third, it cannot meet the timeliness requirements of on-orbit real-time calibration;

[0005] Active vision motion control method (such as Chinese patent application CN202410973709.3): it relies on a precise guide rail to control the camera to move according to a preset trajectory, and the positions of the two cameras are associated through the marker points of the calibration object. This scheme has fundamental obstacles for implementation on a spacecraft: the spaceborne camera is usually fixed to the surface of the spacecraft and cannot be actively moved; at the same time, the motion control mechanism increases the complexity and failure risk of the system, and still cannot get rid of the dependence on the calibration object;

[0006] Calibration object-free self-calibration method: the constraint method based on Kruppa equation is extremely sensitive to image noise, and its robustness significantly decreases when the imaging quality fluctuates in the high radiation environment in space; the layered calibration method needs to rely on strong geometric constraints (such as parallel lines and orthogonal planes) in the scene, but the deep space background photographed by the spacecraft lacks such features, and the algorithm complexity is high, making it difficult to meet the real-time requirements.

[0007] In addition, existing self-calibration methods generally use traditional optimizers such as genetic algorithms, particle swarm algorithms, and grey wolf optimization algorithms, which have problems such as slow convergence and easy falling into local optimality, making it difficult to adapt to the resource constraints of on-orbit computing.

[0008] It is particularly noteworthy that the current on-orbit calibration of spaceborne cameras usually relies on known coordinate features on the Earth's surface or the structural features of the satellite itself (such as the size of the solar wing). However, such methods are completely ineffective when there is no prior geographic information of deep space missions or target celestial bodies, or when the satellite structure deforms in the extreme thermal cycle environment of space. In addition, many spaceborne cameras are only calibrated in the laboratory before launch, without fully considering the camera structure deformation and optical parameter drift caused by the space environment, resulting in systematic deviations between the calibration results and the actual physical parameters. Summary of the Invention

[0009] In response to the above-mentioned technical gaps, the present invention proposes a method for automatically calibrating the internal and external parameters of a space-borne binocular camera vibration test system and a system for automatically calibrating the internal and external parameters of a space-borne binocular camera vibration test system. By extracting natural features from images taken by the space-borne camera, real-time, high-precision calibration of internal and external parameters is achieved without calibration objects or human intervention, and the system can adapt to complex space environments.

[0010] To achieve the above objectives, the technical solutions provided by the present invention are:

[0011] On the one hand, a method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system is provided, comprising the following steps:

[0012] Step 1: Obtain the left and right camera image sequences and the initial value of the left and right camera distance taken by the onboard binocular camera;

[0013] Step 2: Extract and match feature points from the acquired left and right camera image sequences to obtain sub-pixel coordinates of matching feature point pairs;

[0014] Step 3: Based on the matched feature point pairs, calculate the left camera basic matrix, the right camera basic matrix, and the basic matrix between the left and right cameras;

[0015] Step 4: Based on the calculated left camera fundamental matrix and right camera fundamental matrix, the objective function is constructed based on the mapping relationship between the essential matrix and the camera intrinsic parameters, and the IAO (Information Acquisition Optimizer) algorithm is used to solve the optimal solution of the camera intrinsic parameters;

[0016] Step 5: Based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters, the candidate solutions of the camera extrinsic parameters and the projection matrix are calculated. Combined with the matched feature point pairs, the correct camera projection matrix is ​​selected by triangulation, and its corresponding extrinsic parameters are used as the initial values ​​of the camera extrinsic parameters.

[0017] Step 6: Combine the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances to construct an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as the joint optimization targets. The selected initial values ​​of the camera extrinsic parameters are used as initial parameters, and a nonlinear least squares estimation algorithm is used to solve the optimal camera extrinsic parameters.

[0018] Step 7: Output the calibrated camera intrinsic and extrinsic parameters.

[0019] Furthermore, in step 2, an improved SuperPoint+SuperGlue feature detection and matching algorithm is used, which adds the Keypt2Subpx deep learning algorithm to the traditional SuperPoint+SuperGlue detector.

[0020] Furthermore, in step 3, the RANSAC (Random Sample Consensus) method is used to calculate the basic matrix corresponding to every two images based on the sub-pixel coordinates of the matching feature point pairs.

[0021] Furthermore, in step 4, the objective function for solving the internal parameter is:

[0022]

[0023] Where K is the camera’s internal parameter matrix to be solved, F i is the calculated left camera fundamental matrix or right camera fundamental matrix, n is the number of fundamental matrices used for monocular intrinsic parameter estimation, and tr represents the matrix trace operation.

[0024] Furthermore, step 5 specifically includes the following steps:

[0025] Step 5.1: Calculate the essential matrix based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters:

[0026] E=K l T F l,r K r

[0027] Where E is the essential matrix, K l To solve the left camera internal parameters, K r To solve the internal parameters of the right camera, F l,r is the calculated fundamental matrix between the left and right cameras;

[0028] Step 5.2, perform SVD decomposition E=UDV on the essential matrix T , calculate the candidate solutions for camera extrinsic parameters and projection matrix:

[0029] Other The rotation matrix is ​​expressed as: R1 = UW T V T Or R2=UWV T ; The translation vector is expressed as: Where u3 is the last column vector of matrix U;

[0030] Get four possible right camera projection matrices P r For: P r1 =K r [R1 t1]、P r2 =K r [R1 t2]、P r3 =K r [R2t1] and P r4 =K r [R2 t2], where the left camera coordinate system is the reference coordinate system and the left camera projection matrix is ​​P l =K l [I 0], I represents the identity matrix;

[0031] Step 5.3: Establish a triangulation equation system to obtain the three-dimensional point coordinates corresponding to the matching feature point pairs:

[0032]

[0033] Where, Represents the left camera projection matrix P l The i-th row of Represents the right camera projection matrix P r The i-th row, (x l ,y l ) and (x r ,y r ) are the pixel coordinates of the matching points corresponding to the left and right camera images, Q w is the coordinate of the three-dimensional point to be solved;

[0034] Step 5.4, filter the camera projection matrix and obtain the initial value of the camera external parameter: substitute the pixel coordinates of the matching points on the left and right camera images and the candidate solution of the projection matrix into the established equations for triangulation to obtain the three-dimensional point coordinates Q w The projection matrix located in front of the left and right cameras is used to obtain the rotation matrix and translation vector corresponding to the projection matrix as the initial value of the camera external parameters.

[0035] Furthermore, in step 6, the LM (Levenberg-Marquardt) method is used to perform nonlinear least squares estimation to obtain the optimal external parameters of the camera with the smallest corresponding error.

[0036] Further, the joint optimization objective function for nonlinear least squares estimation is:

[0037]

[0038] where n is the number of matching points, P l and P r are the left camera projection matrix and the right camera projection matrix, q li and q ri are the feature point pixel coordinates of the left and right camera images, Q wi is the real world coordinate obtained by triangulation, and are the pixel coordinates obtained by re-projecting the real world coordinate to the camera imaging plane using the left camera projection matrix and the right camera projection matrix, respectively, L is the initial value of the left and right camera distance, and t is the initial translation vector obtained.

[0039] Further, in step 7, the optimal rotation matrix obtained in step 6 is converted into rotation angles around the x, y, and z coordinate axes, the optimal translation vector obtained in step 6, and the left and right camera internal parameters obtained in step 4 are output as the final calibration result.

[0040] Further, the left and right camera image sequences obtained in step 1 include:

[0041] Different pose images of the target object at different moments of motion, or different view images of the target object during the rotation of the spaceborne camera, for calculating the left camera fundamental matrix and the right camera fundamental matrix;

[0042] A sequence of target images captured by the spaceborne binocular camera after the pose of the spaceborne binocular camera is fixed, for calculating the left and right camera fundamental matrix.

[0043] On the other hand, an internal and external parameter automatic calibration system for a spaceborne binocular camera vibration test system is provided, comprising:

[0044] An image and parameter input module for receiving a left and right camera image sequence captured by a spaceborne binocular camera and an initial value of the left and right camera distance;

[0045] A feature extraction and matching module for extracting and matching feature points from the obtained left and right camera image sequence to obtain sub-pixel coordinates of matching feature point pairs;

[0046] A fundamental matrix calculation module for calculating a left camera fundamental matrix, a right camera fundamental matrix, and a left and right camera fundamental matrix based on the matching feature point pairs;

[0047] The intrinsic parameter calibration module is used to construct the objective function based on the mapping relationship between the essential matrix and the camera intrinsic parameters according to the calculated left camera fundamental matrix and the right camera fundamental matrix, and use the IAO algorithm to solve the optimal solution of the camera intrinsic parameters;

[0048] The module for determining the initial solution of extrinsic parameters is used to calculate candidate solutions of camera extrinsic parameters and projection matrix based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters. It also selects the correct camera projection matrix through triangulation based on the matched feature point pairs, and uses the corresponding extrinsic parameters as the initial value of the camera extrinsic parameters.

[0049] The extrinsic parameter optimization module is used to combine the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances to construct an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as the joint optimization targets. The selected initial values ​​of the camera extrinsic parameters are used as the initial parameters, and the nonlinear least squares estimation algorithm is used to solve the optimal camera extrinsic parameters.

[0050] The parameter output module is used to output the calibrated camera intrinsic and extrinsic parameters.

[0051] The advantages of the present invention are:

[0052] 1. The proposed method and system for automatic calibration of the internal and external parameters of a spaceborne binocular camera vibration test system is based on a sequence of target images captured by the spaceborne camera. It does not rely on known coordinate features on the Earth's surface or the structural features of the satellite itself, and does not require a calibration plate or manual operation. It can achieve automatic real-time calibration of the internal and external parameters of the spaceborne binocular camera system.

[0053] 2. The proposed method and system for automatic calibration of internal and external parameters of a spaceborne binocular camera vibration test system uses an essential matrix (E_based) method to construct the objective function in solving the internal parameters. Compared with the traditional self-calibration method based on the Kruppa equation, it has stronger robustness and calibration accuracy.

[0054] 3. The automatic calibration method and system for the internal and external parameters of the spaceborne binocular camera vibration test system proposed in the present invention use the IAO optimization algorithm in nonlinear optimization, and the calibration speed and accuracy are superior to the automatic calibration methods using other optimization algorithms.

[0055] 4. The proposed method and system for automatic calibration of internal and external parameters of a spaceborne binocular camera vibration test system proposes a joint optimization objective function for the comprehensive reprojection error and camera center distance error in external parameter calibration. Compared with traditional external parameter self-calibration, it does not require the three-dimensional features of the captured structure (parallel relationship, distance, etc.) and has a wider applicability.

[0056] 5. The proposed method and system for automatic calibration of internal and external parameters of a spaceborne binocular camera vibration test system uses an improved SuperPoint+SuperGlue method for feature detection and matching. The Keypt2Subpx deep learning algorithm is added to the traditional SuperPoint+SuperGlue detector, improving the accuracy of feature point detection and matching, thereby significantly enhancing calibration accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and / or other features and advantages of the present invention will become more readily understood through the following description with reference to the accompanying drawings, in which:

[0058] Figure 1 This is a flow chart of the method for automatically calibrating the internal and external parameters of the spaceborne binocular camera vibration test system of the present invention;

[0059] Figure 2 This is a diagram showing the composition of the automatic calibration system for internal and external parameters of the spaceborne binocular camera vibration test system of the present invention;

[0060] Figure 3 This is a flow chart of feature extraction, matching, and basic matrix calculation in the present invention;

[0061] Figure 4 This is a flow chart of the camera internal parameter calibration in the present invention;

[0062] Figure 5 This is a flow chart of camera extrinsic parameter calibration in the present invention;

[0063] Figure 6 A comparison diagram of the convergence curves of the automatic calibration method based on the essential matrix of the present invention and the traditional automatic calibration method based on the Kruppa equation;

[0064] Figure 7 A comparison chart of optimization results between the automatic calibration method based on the essential matrix of the present invention and the traditional automatic calibration method based on the Kruppa equation;

[0065] Figure 8 This is a comparison chart of the convergence curves of the automatic calibration method using the IAO optimization algorithm and the automatic calibration method using the GWO (Grey Wolf Optimizer) optimization algorithm.

[0066] Figure 9 This is a comparison chart of the optimization results of the automatic calibration method combined with the IAO optimization algorithm of the present invention and the automatic calibration method using the GWO optimization algorithm. DETAILED DESCRIPTION

[0067] The present invention will be described in detail below with reference to the accompanying drawings by means of exemplary embodiments of the present invention. It should be noted that the following detailed description of the present invention is only for the purpose of illustration and is not intended to limit the present invention.

[0068] The present invention provides a method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system and a system for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system. The method can be used to recalibrate and correct the parameters of visual monitoring cameras on on-orbit spacecraft and has high versatility for visual measurement systems in other environments. It not only has high calibration accuracy but also can perform real-time and efficient calibration.

[0069] First refer to Figure 1 The method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system provided by the present invention comprises the following steps:

[0070] Step S1, obtaining a left and right camera image sequence and an initial value of the left and right camera distance taken by a spaceborne binocular camera;

[0071] Step S2: extracting and matching feature points from the acquired left and right camera image sequences to obtain sub-pixel coordinates of matching feature point pairs;

[0072] Step S3, based on the matched feature point pairs, calculating the left camera basic matrix, the right camera basic matrix and the basic matrix between the left and right cameras;

[0073] Step S4: constructing an objective function based on the calculated left camera fundamental matrix and right camera fundamental matrix and the mapping relationship between the essential matrix and the camera intrinsic parameters, and using the IAO algorithm to solve the optimal solution of the camera intrinsic parameters;

[0074] Step S5: Calculate candidate camera extrinsic parameter solutions and projection matrix solutions based on the calculated left and right camera fundamental matrices and the solved camera intrinsic parameters. Combined with the matched feature point pairs, the correct camera projection matrix is ​​selected through triangulation, and the corresponding extrinsic parameters are used as the initial values ​​of the camera extrinsic parameters.

[0075] Step S6, combining the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances, constructing an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as joint optimization targets, using the selected initial values ​​of the camera extrinsic parameters as initial parameters, and applying a nonlinear least squares estimation algorithm to solve the optimal camera extrinsic parameters;

[0076] Step S7: output the calibrated camera intrinsic and extrinsic parameters.

[0077] In some embodiments of the present invention, in step S1, the acquired left and right camera image sequences may include:

[0078] Different pose images of the target at the moment of motion, or different view images of the target by the satellite-borne camera during rotation, are used to calculate the left camera fundamental matrix and the right camera fundamental matrix;

[0079] The target image sequence captured by the satellite-borne binocular camera after the pose is fixed is synchronously collected, and is used to calculate the fundamental matrix between the left and right cameras.

[0080] The initial value input is the distance between the left and right cameras of the binocular measurement system, which is used as a reference for restoring the calibration scale factor.

[0081] In step S2, the feature point matching is performed between two images, the matching result of the left camera image is used for left camera internal parameter estimation, the matching result of the right camera image is used for right camera internal parameter estimation, and the matching result of the two images collected at the same moment by the left and right cameras is used for external parameter estimation.

[0082] In the preferred embodiment, the feature point extraction and matching of the components to be detected of the on-orbit spacecraft use the improved SuperPoint+SuperGlue feature detection and matching algorithm, specifically, the Keypt2Subpx deep learning algorithm is added to the traditional SuperPoint+SuperGlue detector for sub-pixel refinement, further improving the feature detection accuracy and matching accuracy, thereby greatly improving the calibration accuracy and calibration efficiency.

[0083] Specifically, for the two images to be matched, the SuperPoint deep learning algorithm is first used to extract the structure or starry sky feature points on the captured image and generate feature descriptors. SuperPoint is an unsupervised algorithm used to detect feature points and their descriptors in images. It trains the network in a self-supervised manner and can learn how to detect feature points and describe these features without labeled data. It adopts an Encoder-Decoder structure to extract feature points and their descriptors through a shared encoding network. The feature point detector and the descriptor network share a forward encoder, and then learn different network parameters according to different tasks, and finally obtain feature points and descriptors. Furthermore, for the obtained feature point coordinates and corresponding feature descriptors, the SuperGlue deep learning algorithm is used to match all feature points in the two array images. SuperGlue is an end-to-end feature matching algorithm based on deep learning, which achieves highly robust key point matching through graph neural networks (GNN) and attention mechanisms. By inputting the key points of two images and their descriptors (generated by SuperPoint), a graph node containing position encoding is constructed; then the internal context information of a single image is enhanced through the self-attention layer, and the global correlation across images is captured using the cross-attention layer, so that the key point features are integrated with spatial and semantic information; the descriptor similarity matrix is ​​calculated and the geometric prior is integrated, and the optimal transmission problem is solved by the differentiable Sinkhorn algorithm to obtain a soft assignment matrix to balance the matching probability and bidirectional constraints, and finally the final matching pair is output after thresholding and bidirectional testing. If the number of matching point pairs is less than 8, the image pair is discarded. For the integer pixel matching point pair coordinates initially obtained using the SuperPoint and SuperGlue feature detection and matching algorithms, the Keypt2Subpx module learns to perform multi-viewing on any key point given the expected key point correspondence between the two images. Figure 1 The sub-pixel matching point pair coordinates are obtained through consistent sub-pixel adjustment, thereby improving the accuracy of feature point detection and matching.

[0084] In step S3 , the basic matrix between the two images can be calculated by using the RANSAC (Random Sample Consensus) method, which calculates the basic matrix corresponding to each two images according to the sub-pixel coordinates of the matching point pairs, and this method can eliminate incorrect matching points.

[0085] In a specific embodiment of the present invention, the objective function for solving the internal parameter in step S4 may be:

[0086]

[0087] Where K is the camera’s internal parameter matrix to be solved, F iis the calculated left camera fundamental matrix or right camera fundamental matrix, n is the number of fundamental matrices used for monocular intrinsic parameter estimation, and tr represents the matrix trace operation.

[0088] The objective function is constructed based on:

[0089] For the camera's essential matrix E, let S = EE T According to the properties of the basic matrix, two of its three singular values ​​are equal and the other is 0, so the relationship between the two eigenvalues ​​of the matrix S and the two singular values ​​of the matrix E is: λ1=σ1 2 ,λ2=σ2 2 . From this we can get:

[0090] tr(S)=λ1+λ2=σ1 2 +σ2 2

[0091] tr(S) 2 =(σ1 2 +σ2 2 ) 2

[0092] According to the constraint condition that the two singular values ​​of the essential matrix E are equal σ1=σ2=σ, the constraint condition of the matrix S can be obtained:

[0093] tr(S) 2 =2tr(S 2 )

[0094] According to the conversion relationship between the essential matrix E and the basic matrix F, E=K T FK, get the constraints of the internal parameter K:

[0095] tr(K T FKK T F T K) 2 -2tr((K T FKK T F T K) 2 )=0

[0096] It is further converted into the objective function for solving the intrinsic parameters of the spaceborne camera. The objective function is constructed using the method based on the essential matrix in the intrinsic parameter solution. Compared with the traditional self-calibration method based on the Kruppa equation, it has stronger robustness and calibration accuracy.

[0097] The nonlinear optimization of the above objective function adopts the IAO optimization algorithm, which consists of three key strategies: information collection, filtering and evaluation, and analysis and organization. Its convergence speed and convergence accuracy are better than traditional intelligent optimization algorithms (such as genetic algorithm, particle swarm algorithm and gray wolf algorithm).

[0098] According to the application, the method for solving the initial value of the external parameter in step S5 is as follows: the essential matrix E is calculated through the fundamental matrix between the left and right cameras and the camera intrinsic parameter, and then the essential matrix is decomposed by SVD, and four possible solutions of the projection matrix are further calculated, and the triangularization verification is performed, that is, the three-dimensional point coordinates are recovered through the matching point coordinates of two frames of images and the internal and external parameters, the projection matrix that meets the actual situation is selected, and the corresponding external parameter is the initial external parameter.

[0099] Specifically, the method for calculating the essential matrix is as follows:

[0100] E = K l T F l,r K r

[0101] In the formula, K l is the internal parameter of the left camera solved, K r is the internal parameter of the right camera solved, F l,r is the fundamental matrix calculated between the left and right cameras;

[0102] The method for solving the projection matrix is as follows:

[0103] The essential matrix is decomposed by SVD to obtain: E = UDV T

[0104] Let The rotation matrix R can be expressed as:

[0105] R1 = UW T V T

[0106] Or

[0107] R2 = UWV T

[0108] The translation vector can be expressed as:

[0109]

[0110] In the formula, u3 is the last column vector of the matrix U;

[0111] Four possible right camera projection matrices P r are obtained as follows: P r1 = K r [R1 t1], P r2 = K r [R1 t2], P r3 = K r [R2t1], and P r4 = K r[R2 t2], wherein, with the left camera coordinate system as a reference coordinate system, the left camera projection matrix is P l = K l [I 0], I represents a unit matrix;

[0112] The triangulation process is: the three-dimensional point coordinates corresponding to the matching point pairs are obtained by solving the equation set:

[0113]

[0114] In the formula, represents the i-th row of the left camera projection matrix P l , represents the i-th row of the right camera projection matrix P r , (x l , y l ) and (x r , y r ) are the pixel coordinates of the matching points corresponding to the left and right camera images respectively, and Q w is the three-dimensional point coordinates to be solved.

[0115] The above initial external parameters are: the matching point pixel coordinates (x l , y l ) and (x r , y r ) on the left and right camera images and the candidate solution of the projection matrix are substituted into the above equation for triangulation, the projection matrix meeting the condition that the three-dimensional point coordinates Q w are located in front of the left and right cameras, and then the rotation matrix and the translation vector corresponding to the correct projection matrix are obtained.

[0116] In step S6 of the application, the method for solving the external parameters of the spaceborne binocular camera system is to use the external parameters solved in step S5 as initial parameters, use the L-M method for nonlinear least square estimation, and obtain the optimal external parameters corresponding to the minimum error.

[0117] Particularly, the joint optimization objective function for nonlinear least square estimation is:

[0118]

[0119] In the formula, n is the number of matching points, P l and P r are the left camera projection matrix and the right camera projection matrix respectively, q li and q ri are the pixel coordinates of the feature points of the left and right camera images, Q wi is the real world coordinates obtained by triangulation solution, and To reproject the real-world coordinates to the pixel coordinates of the camera imaging plane using the left camera projection matrix and the right camera projection matrix, L is the initial value of the left and right camera distance, and t is the obtained initial translation vector.

[0120] In the external parameter calibration, the present invention proposes a joint optimization objective function of the comprehensive reprojection error and the camera center distance error. Compared with the traditional external parameter self-calibration, it does not require the three-dimensional features of the captured structure (parallel relationship, distance, etc.) and has a wider applicability.

[0121] Step S7 specifically converts the optimal rotation matrix obtained in step S6 into rotation angles around the x, y, and z coordinate axes, and outputs the optimal translation vector obtained in step S6 and the internal parameters of the left and right cameras obtained in step S4 as the final calibration result.

[0122] Next refer to Figure 2 The automatic calibration system for internal and external parameters of a spaceborne binocular camera vibration test system provided by the present invention includes: an image and parameter input module, a feature extraction and matching module, a basic matrix calculation module, an internal parameter calibration module, an external parameter initial solution determination module, an external parameter optimization module, and a parameter output module.

[0123] The image and parameter input module receives the left and right camera image sequences and the initial distance between the left and right cameras captured by the onboard binocular camera. The feature extraction and matching module extracts and matches feature points from the acquired left and right camera image sequences to obtain the sub-pixel coordinates of the matching feature point pairs. The fundamental matrix calculation module calculates the left camera fundamental matrix, the right camera fundamental matrix, and the left and right camera inter-camera fundamental matrix based on the matched feature point pairs. The intrinsic parameter calibration module constructs an objective function based on the mapping relationship between the essential matrix and the camera intrinsic parameters based on the calculated left and right camera fundamental matrices, and uses the IAO algorithm to solve for the optimal solution of the camera intrinsic parameters. The extrinsic parameter initial solution determination module calculates candidate camera extrinsic parameter solutions and candidate projection matrix solutions based on the calculated left and right camera inter-camera fundamental matrix and the solved camera intrinsic parameters. Combined with the matched feature point pairs, the correct camera projection matrix is ​​selected through triangulation, and its corresponding extrinsic parameters are used as the initial values ​​of the camera extrinsic parameters. The extrinsic parameter optimization module combines the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances to construct an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as the joint optimization objectives. Using the selected initial values ​​of the camera extrinsic parameters as the initial parameters, a nonlinear least squares estimation algorithm is used to solve for the optimal camera extrinsic parameters. The parameter output module outputs the calibrated camera intrinsic and extrinsic parameters.

[0124] The method and system for automatically calibrating the internal and external parameters of the spaceborne binocular camera vibration test system proposed in the present invention can realize automatic real-time calibration of the internal and external parameters of the spaceborne binocular camera system based on the target image sequence captured by the spaceborne camera, without relying on known coordinate features on the earth's surface or the structural features of the satellite body, and without the need for calibration plates or manual operation.

[0125] The automatic calibration method and system for internal and external parameters of a spaceborne binocular camera vibration test system provided by the present invention will now be further described with reference to examples.

[0126] In this example, a binocular camera system for satellite solar array vibration monitoring is calibrated. The binocular camera's internal and external parameters are automatically calibrated based on the following conditions:

[0127] 1) The left and right camera image sequences img1 and img2 taken by the onboard camera are known;

[0128] 2) The installation distance between the left and right cameras is known.

[0129] Reference Figure 3 , describes the feature point extraction and matching and basic matrix calculation process.

[0130] Assuming n images are input from the left and right cameras, three types of fundamental matrices can be calculated. The first type is the left camera fundamental matrix. One fundamental matrix is ​​calculated for every two images, theoretically resulting in n(n-1) / 2 left camera fundamental matrices. The second type is the right camera fundamental matrix, the same number as for the left camera. The third type is the fundamental matrix between the left and right cameras. One fundamental matrix is ​​calculated for two images captured simultaneously by the left and right cameras, theoretically resulting in n fundamental matrices. The calculation methods for all three types of fundamental matrices are the same, and the following uses the calculation of the left camera fundamental matrix as an example to illustrate.

[0131] For the two images img1 and img2 to be matched, the SuperPoint deep learning algorithm is used to extract the structural or starry sky feature points on the captured solar cell array image and generate feature descriptors.

[0132] Furthermore, the obtained feature point coordinates and corresponding feature descriptors are used to match all feature points in the two solar array images img1 and img2 using the SuperGlue deep learning algorithm. If the number of matching point pairs is less than 8, the image pair is discarded.

[0133] For the initial integer pixel matching point pair coordinates q obtained using the SuperPoint and SuperGlue feature detection and matching algorithms l (x l ,y l ) and q r (x r ,yr ) for sub-pixel optimization. By attaching a Keypt2Subpx module to the superpoint+SuperGlue feature matching detector, the Keypt2Subpx module is fed with matching solar cell array image pairs and the corresponding matching point pair coordinates, feature descriptors, and dense score maps. The Keypt2Subpx module can obtain the sub-pixel matching point pair coordinates q. lsubpx (x lsubpx ,y lsubpx ) and q rsubpx (x rsubpx ,y rsubpx ), thereby improving the accuracy of feature point detection and matching.

[0134] Furthermore, the RANSAC algorithm is used to calculate the basic matrix for the obtained sub-pixel matching point pair coordinates. First, the matching point pairs are randomly sampled, and 8 point pairs are randomly selected in each iteration (the minimum sample set of the eight-point method); then the model calculation is performed, and the candidate basic matrix F is calculated using the selected 8 point pairs through the eight-point method; then the inlier judgment is performed, and the remaining data points are tested with the current model. If the error is less than the threshold, it is determined to be an inlier, and the number of inliers is counted; iterative optimization is performed, and the above steps are repeated to retain the model with the most inliers; after reaching the preset number of iterations, the final model is refitted with the maximum inlier set to obtain the basic matrix.

[0135] According to the above process, the three basic matrices are calculated (one for each category):

[0136]

[0137] Where, F l is the basic matrix of the left camera onboard, F is the basic matrix of the right camera onboard, and F l,r is the basic matrix between the left and right cameras on board.

[0138] Reference Figure 4 ,Taking the intrinsic parameter calibration of the left camera of the spaceborne binocular camera system as an example, the specific process of intrinsic parameter calibration is described.

[0139] for Figure 3 The n(n-1) / 2 left camera basic matrices F are obtained from the process shown. First, the left camera intrinsic parameter matrix K is set l The optimization range is:

[0140]

[0141] Where, f x and f yare the effective focal lengths in the x and y directions respectively, (u0, v0) are the coordinates of the camera principal point. Currently, most spaceborne cameras have square pixels, so set f x =f y =f. Currently, the f parameter of most spaceborne cameras is in the range of [1,5000]. It can also be set according to the camera's factory parameters. The camera's principal point is generally near the center of the image. The range of setting the principal point is:

[0142] u0∈[0.8·C x ,1.2·C x ],v0∈[0.8·C y ,1.2·C y ]

[0143] Where C x 、C y is the coordinate of the center point of the image. Set the number of individuals in each generation to N and the maximum number of iterations to M.

[0144] Specifically, the initial optimization parameter values ​​are shown in Table 1.

[0145] Table 1 Initial optimization parameter settings

[0146]

[0147] Furthermore, the N initial left camera intrinsic parameter matrices generated according to the given parameter range and Figure 3 The n(n-1) / 2 left camera fundamental matrices F obtained by the process shown are substituted into the initial value of the objective function C, where:

[0148]

[0149] In the formula, m = n (n-1) / 2, record the best fitness C best The corresponding optimal internal parameter K lbest .

[0150] Furthermore, it is determined whether the given number of iterations M has been reached. If the maximum number of iterations has not been reached, the internal parameter matrix is ​​updated and the fitness is recalculated according to the evaluation and organization strategy of the IAO optimization algorithm, and the optimal internal parameters and fitness are updated. Until the maximum number of iterations M is reached, the optimal left camera internal parameter matrix K is output. lbest The intrinsic parameter calibration of the left camera of the spaceborne binocular camera system was performed five times, and the results were compared with the actual internal parameters, as shown in Table 2.

[0151] Table 2 Optimization results

[0152]

[0153] like Figure 6As shown in the figure, the automatic calibration method based on the basic matrix (E_based) and the automatic calibration method based on the Kruppa equation are used to perform an internal parameter calibration respectively, and the convergence curves of the calibration process are obtained. According to the convergence curves of the two methods, the convergence speed and convergence accuracy of the automatic calibration method based on the basic matrix (E_based) are better than those of the automatic calibration method based on the Kruppa equation.

[0154] like Figure 7 As shown in the figure, the two methods mentioned above are used to calibrate the intrinsic parameters of the left camera of the spaceborne binocular camera system five times respectively. The calibration results of the two methods and the distribution diagram of the true intrinsic parameter values ​​are plotted. According to the calibration diagram of the distribution results, the robustness and accuracy of the automatic calibration method based on the fundamental matrix (E_based) are better than the automatic calibration method based on the Kruppa equation.

[0155] Table 3 Optimization results of the automatic calibration method based on the Kruppa equation

[0156]

[0157] like Figure 8 As shown in the figure, an internal parameter calibration is performed using the automatic calibration method combined with the IAO optimization algorithm, and a convergence curve of the calibration process is obtained. Compared with the convergence curve of the automatic calibration method combined with the Grey Wolf Optimizer (GWO), the convergence speed and convergence accuracy of the automatic calibration method combined with the IAO optimization algorithm of the present invention are better than those of the automatic calibration method combined with the GWO optimization algorithm.

[0158] like Figure 9 As shown in the figure, the two methods were used to calibrate the intrinsic parameters of the left camera of the spaceborne binocular camera system five times respectively. The calibration results of the two methods and the distribution diagram of the true intrinsic parameter values ​​were plotted. According to the calibration diagram of the distribution results, the robustness of the automatic calibration method combined with the IAO optimization algorithm of the present invention is better than that of the automatic calibration method combined with the GWO optimization algorithm.

[0159] Reference Figure 5 ,The specific process of extrinsic parameter calibration of spaceborne binocular camera system is described.

[0160] Combine Figure 3 and Figure 4 The process shown above obtains the left and right camera basic matrices and the left and right camera intrinsic calibration results of the spaceborne binocular camera system, and solves the essential matrix E, where E = K l T F l,r K r .

[0161] Further, the essential matrix SVD decomposition is obtained: E = UDV T .

[0162] make Solve the rotation matrix R as: R1=UW T V T Or R2=UWV T .

[0163] The translation vector is solved as:

[0164] Furthermore, assuming that the coordinate system of the left camera onboard is the reference coordinate system, the projection matrix of the left camera is P l =K l [I 0]; get four possible right camera projection matrices P r P r1 =K r [R1 t1]、P r2 =K r [R1 t2]、P r3 =K r [R2 t1] and P r4 =K r [R2 t2].

[0165] Further, triangulate (Triangulate), for Figure 3 The image feature matching point pair sub-pixel coordinates q obtained by the process shown lsubpx (x lsubpx ,y lsubpx ) and q rsubpx (x rsubpx ,y rsubpx ), let the projection matrix P be expressed as:

[0166]

[0167] Construct an overdetermined system of equations:

[0168]

[0169] Substitute the four possible projection matrices and solve the least squares solution through SVD decomposition to verify the solution Q w The projection matrix corresponding to the left and right cameras is the correct matrix. The corresponding rotation matrix and translation vector of the onboard binocular camera system are used as external parameters to calibrate the initial parameters as follows:

[0170]

[0171] Furthermore, we set the optimization parameters: the number of individuals in each generation N and the maximum number of iterations M, perform least squares optimization on the external parameters of the camera system, and use the coordinates of the matching point pairs and the initial input camera distance parameters to calculate the reprojection error and translation vector error. The objective function is:

[0172]

[0173] Furthermore, the iterative optimization is performed until the maximum number of iteration steps M is met, and the best individual corresponding to the best fitness is output as the external parameter, and the external parameter calibration is completed.

[0174] Among them, the comparison between the best optimization parameters and the actual external parameters is shown in Table 4.

[0175] Table 4 External parameter calibration results

[0176]

[0177] As can be seen from the table, the maximum relative error between the calibration results of the spaceborne camera external parameters and the true parameters is 0.75%, which proves that this method has good calibration accuracy.

[0178] Finally, the external parameters of the spaceborne binocular camera system and the internal parameters of the left and right cameras obtained in the above calibration process are output, and the calibration of the spaceborne binocular camera system for vibration monitoring of the solar cell array of an on-orbit spacecraft is completed, verifying the effectiveness of the automatic calibration method and system for the internal and external parameters of the spaceborne binocular camera vibration test system proposed in the present invention.

[0179] Finally, it should be noted that the features mentioned and / or illustrated in the above description of the exemplary embodiments of the present invention may be incorporated into one or more other embodiments in the same or similar manner, combined with features in other embodiments, or substituted for corresponding features in other implementations. The technical solutions obtained by such combination or substitution shall also be deemed to be included in the scope of protection of the present invention.

Claims

1. A method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system, characterized in that: The following steps are involved: Step 1: Obtain the left and right camera image sequences and the initial value of the left and right camera distance taken by the onboard binocular camera; Step 2: Extract and match feature points from the acquired left and right camera image sequences to obtain sub-pixel coordinates of matching feature point pairs; Step 3: Based on the matched feature point pairs, calculate the left camera basic matrix, the right camera basic matrix, and the basic matrix between the left and right cameras; Step 4: Based on the calculated left camera fundamental matrix and right camera fundamental matrix, the objective function is constructed based on the mapping relationship between the essential matrix and the camera intrinsic parameters, and the IAO algorithm is used to solve the optimal solution of the camera intrinsic parameters; Step 5: Based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters, the candidate solutions of the camera extrinsic parameters and the projection matrix are calculated. Combined with the matched feature point pairs, the correct camera projection matrix is ​​selected by triangulation, and its corresponding extrinsic parameters are used as the initial values ​​of the camera extrinsic parameters. Step 6: Combine the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances to construct an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as the joint optimization targets. The selected initial values ​​of the camera extrinsic parameters are used as initial parameters, and a nonlinear least squares estimation algorithm is used to solve the optimal camera extrinsic parameters. Step 7: Output the calibrated camera intrinsic and extrinsic parameters.

2. The method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1, characterized in that: In step 2, an improved SuperPoint+SuperGlue feature detection and matching algorithm is used, which adds the Keypt2Subpx deep learning algorithm to the traditional SuperPoint+SuperGlue detector.

3. The automatic calibration method for internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: In step 3, the RANSAC method is used to calculate the basic matrix corresponding to each two images based on the sub-pixel coordinates of the matching feature point pairs.

4. The automatic calibration method for internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: In step 4, the objective function used to solve the internal parameter is: Where K is the camera’s internal parameter matrix to be solved, F i is the calculated left camera fundamental matrix or right camera fundamental matrix, n is the number of fundamental matrices used for monocular intrinsic parameter estimation, and tr represents the matrix trace operation.

5. The automatic calibration method for internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: Step 5 specifically includes the following steps: Step 5.1: Calculate the essential matrix based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters: E=K l T F l,r K r Where E is the essential matrix, K l To solve the left camera internal parameters, K r To solve the internal parameters of the right camera, F l,r is the calculated fundamental matrix between the left and right cameras; Step 5.2, perform SVD decomposition E=UDV on the essential matrix T , calculate the candidate solutions for camera extrinsic parameters and projection matrix: Other The rotation matrix is ​​expressed as: R1 = UW T V T Or R2=UWV T ; The translation vector is expressed as: Where u3 is the last column vector of matrix U; Get four possible right camera projection matrices P r For: P r1 =K r [R1 t1]、P r2 =K r [R1 t2]、P r3 =K r [R2 t1] and P r4 =K r [R2 t2], where the left camera coordinate system is the reference coordinate system and the left camera projection matrix is ​​P l =K l [I 0], I represents the identity matrix; Step 5.3: Establish a triangulation equation system to obtain the three-dimensional point coordinates corresponding to the matching feature point pairs: Where, P l i Represents the left camera projection matrix P l The i-th row of Represents the right camera projection matrix P r The i-th row, (x l ,y l ) and (x r ,y r ) are the pixel coordinates of the matching points corresponding to the left and right camera images, Q w is the coordinate of the three-dimensional point to be solved; Step 5.4, filter the camera projection matrix and obtain the initial value of the camera external parameter: substitute the pixel coordinates of the matching points on the left and right camera images and the candidate solution of the projection matrix into the established equations for triangulation to obtain the three-dimensional point coordinates Q w The projection matrix located in front of the left and right cameras is used to obtain the rotation matrix and translation vector corresponding to the projection matrix as the initial value of the camera external parameters.

6. The method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: In step 6, the LM method is used to perform nonlinear least squares estimation to obtain the optimal external parameters of the camera with the smallest corresponding error.

7. The method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system according to claim 6, characterized in that: The joint optimization objective function for nonlinear least squares estimation is: Where n is the number of matching points, P l and P r are the left camera projection matrix and the right camera projection matrix, q li and q ri is the pixel coordinate of the feature points in the left and right camera images, Q wi is the real-world coordinate obtained by triangulation, and To reproject the real-world coordinates to the pixel coordinates of the camera imaging plane using the left camera projection matrix and the right camera projection matrix, L is the initial value of the left and right camera distance, and t is the obtained initial translation vector.

8. The method for automatically calibrating the internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: In step 7, the optimal rotation matrix obtained in step 6 is converted into the rotation angle around the x, y, and z coordinate axes, and the optimal translation vector obtained in step 6 and the left and right camera internal parameters obtained in step 4 are output as the final calibration result.

9. The automatic calibration method for internal and external parameters of a spaceborne binocular camera vibration test system according to claim 1 or 2, characterized in that: The left and right camera image sequences obtained in step 1 include: Images of the target object at different positions during its motion, or images of the target object at different perspectives during the rotation of the onboard camera, are used to calculate the left camera fundamental matrix and the right camera fundamental matrix; The target image sequence captured synchronously by the spaceborne binocular camera after the position is fixed is used to calculate the basic matrix between the left and right cameras.

10. An automatic calibration system for internal and external parameters of a spaceborne binocular camera vibration test system, characterized in that: include: The image and parameter input module is used to receive the left and right camera image sequences taken by the spaceborne binocular camera and the initial value of the left and right camera distance; The feature extraction and matching module is used to extract and match feature points from the acquired left and right camera image sequences to obtain the sub-pixel coordinates of the matching feature point pairs; The basic matrix calculation module is used to calculate the left camera basic matrix, the right camera basic matrix and the basic matrix between the left and right cameras based on the matched feature point pairs; The intrinsic parameter calibration module is used to construct the objective function based on the mapping relationship between the essential matrix and the camera intrinsic parameters according to the calculated left camera fundamental matrix and the right camera fundamental matrix, and use the IAO algorithm to solve the optimal solution of the camera intrinsic parameters; The module for determining the initial solution of extrinsic parameters is used to calculate candidate solutions of camera extrinsic parameters and projection matrix based on the calculated basic matrix between the left and right cameras and the solved camera intrinsic parameters. It also selects the correct camera projection matrix through triangulation based on the matched feature point pairs, and uses the corresponding extrinsic parameters as the initial value of the camera extrinsic parameters. The extrinsic parameter optimization module is used to combine the coordinates of the matched feature point pairs, the camera projection matrix, and the initial values ​​of the left and right camera distances to construct an extrinsic parameter optimization model with the reprojection error and the translation vector modulus error as the joint optimization targets. The selected initial values ​​of the camera extrinsic parameters are used as the initial parameters, and the nonlinear least squares estimation algorithm is used to solve the optimal camera extrinsic parameters. The parameter output module is used to output the calibrated camera intrinsic and extrinsic parameters.

Images