On-orbit camera calibration method

Abstract

The invention discloses an in-orbit camera calibration method, which comprises the following steps of: acquiring a plurality of groups of complete satellite images in a space illumination environment by utilizing an in-orbit camera, and each satellite image comprises a solar panel; extracting an edge feature line segment of the satellite image, and obtaining an optimal image distortion center position and a preliminary distortion parameter; extracting all feature line segments of the satellite image, and constructing a feature line segment set; solving an optimal vanishing point according to the feature line segment set; according to the optimal vanishing point, final internal and external parameters and distortion parameters are obtained through iterative optimization, and the final internal and external parameters and distortion parameters are calibrated camera parameters. According to the method, the optimal internal and external parameters and distortion parameters of the camera are obtained through rich linear characteristics of the solar panel under the conditions that a specific target is not needed and the calculation capability of a satellite-borne calculation platform is met, so that final calibration is completed, and a solid foundation is laid for the accuracy and stability of subsequent space tasks executed by the in-orbit camera.

Description

Technical Field

[0001] The present invention belongs to the technical field of space camera calibration, and in particular relates to an on-orbit camera calibration method. Background Art

[0002] With the development of space technology, space missions such as space debris cleaning, spacecraft rendezvous and docking, spacecraft on-orbit repair, maintenance, assembly, and reconstruction require accurate and real-time acquisition of the relative position and posture of the target spacecraft and the service spacecraft. The advantage of the measurement system composed of stereo cameras is that it can obtain depth information and three-dimensional information of the target without additional light sources by using the principle of geometric view, which makes up for the limitation of depth information of a single monocular camera. At the same time, compared with various active measurement methods, its compact size, passive operation mode, simple hardware and low power consumption are very suitable for space pose estimation. In order to ensure stable and accurate acquisition of the position and posture of non-cooperative targets, the internal and external parameters of the camera, distortion parameters, and the relative rotation matrix and translation vector between the binocular cameras must be sufficiently accurate. However, the on-orbit camera needs to withstand long-term harsh mechanical and space thermal environments in the space environment. Changes in the above parameters greatly affect the accuracy of subsequent pose data, so accurate calibration of the on-orbit camera is very necessary.

[0003] It is well known (e.g., Huang Wenwen et al., Review of Camera Calibration Methods and Progress), that the imaging model of the on-orbit camera is as follows: in is the unknown scale factor, is the camera internal parameter matrix, including information such as the focal length of the principal point, and is represented as ,in It contains the above-mentioned information such as the focal length of the principal point, that is, the internal parameters of the on-orbit camera.

[0004] M is the camera extrinsic parameter matrix, which is expressed as , the camera center is generally selected as the center point of the world coordinate system, and the external parameter matrix M is the unit matrix.

[0005] in: is the radial distortion parameter, is the tangential distortion parameter. is the horizontal coordinate of the image on the imaging device, is the ordinate of the image on the imaging device, is the floating change of the horizontal axis, is the floating change of the vertical axis, , , is the coordinate of the imaging point in the world coordinate system, R, T are the external parameters of the on-orbit camera, r is the distance from the coordinate on the current imaging device to the imaging center point, Represents a zero matrix, in order to make the external parameter matrix zero-filled to be complete.

[0006] At present, a lot of work has been carried out in this field at home and abroad, among which the representative is: Zhang Zhengyou uses multi-view images to obtain the internal and external parameters of the camera, which is widely used in various camera calibration tasks and has become a standard method for camera calibration. At present, a lot of improvements have been made to Zhang Zhengyou's method for different scenarios, such as improving the accuracy of corner detection, redesigning the calibration plate, etc., and combining point cloud data and images for calibration to improve accuracy. However, in a space environment, high-precision targets cannot be provided. Deep learning methods can learn the internal and external parameters and distortion parameters of the camera, and optimize the accuracy and robustness of camera calibration. However, the computational complexity of deep learning methods is unacceptable for on-orbit computing platforms. Therefore, the existing technology cannot meet the requirements of space missions for camera calibration stability, accuracy and real-time performance under the harsh mechanical jitter and space thermal conditions of space on-orbit. Summary of the invention

[0007] In order to overcome the above-mentioned shortcomings of the prior art, an object of the present invention is to provide an on-orbit camera calibration method to solve the problem in the prior art that it is difficult to accurately calibrate an on-orbit camera in real time.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: An on-orbit camera calibration method comprises the following steps: Step 1: using an on-orbit camera to obtain several sets of complete satellite images in a space lighting environment, and each satellite image contains a solar panel; Step 2: extract edge feature segments of the satellite image to obtain the optimal image distortion center position and preliminary radial distortion parameters; Step 3, extract all feature line segments of the satellite image and construct a feature line segment set; Step 4, finding the optimal vanishing point according to the characteristic line segment set; Step 5: Iteratively optimize and obtain final internal and external parameters and distortion parameters according to the optimal vanishing point. The final internal and external parameters and distortion parameters are the calibrated camera parameters.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. Most of the current camera calibration methods require specific 2D or 3D high-precision targets. However, it is difficult to obtain high-precision targets in the space working environment of the on-orbit camera. The present invention adopts edge feature segments and constructs a self-calibration method based on straight line structure features, thereby avoiding the above problems.

[0010] 2. Solar panels are common and unique components on satellites. They have large opening areas and rich linear features, making them easy to capture. The present invention uses solar panels to provide linear structural features, which is convenient and fast.

[0011] 3. Due to the limitation of computing power of the computing platform of the on-orbit camera and the requirement of high precision of camera parameters in space missions, the calibration method based on deep learning cannot be fully applied to space mission scenarios. This paper adopts traditional algorithms and constructs a calibration method suitable for space environment based on derivation and optimization, which meets the accuracy requirements of current space missions.

[0012] In summary, the present invention provides an on-orbit binocular camera calibration method based on vanishing points. Through the rich linear characteristics of solar panels, the final on-orbit camera self-calibration is completed without the need for specific targets and under the condition of satisfying the computing power of the onboard computing platform, thus meeting the requirements of increasingly complex space missions for the accuracy and stability of on-orbit camera parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the on-orbit camera calibration method of the present invention.

[0014] Figure 2 It is a schematic diagram of the principle of fitting a quadratic curve to obtain the distortion center point of the present invention.

[0015] Figure 3 It is a schematic diagram of the principle of obtaining a characteristic line segment set to obtain an optimal vanishing point in the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0017] After the on-orbit camera in space is subjected to the harsh mechanical vibration and thermal conditions in space, it needs to be calibrated on-orbit to prevent the camera parameters from failing to meet the requirements of subsequent space missions. However, obtaining accurate parameters of the on-orbit camera in space still faces the following problems: 1. The on-orbit camera needs to withstand the harsh mechanical and thermal environment in the space environment. At this time, the relevant parameters of the on-orbit camera cannot meet the high-precision requirements of subsequent space missions.

[0018] 2. Due to the limitations of the working environment of the on-orbit camera, it is impossible to provide 2D or 3D targets such as a chessboard. It can only use the structural features of existing related targets. At the same time, the lighting conditions in the space environment are limited, and the target structural features are not clear enough.

[0019] 3. Due to the limitation of computing power of the computing platform of the on-orbit camera, there are certain requirements for the computational complexity of the calibration method. For example, in theory, traditional algorithms and deep learning algorithms can be used to calibrate the on-orbit aircraft camera, but the existing algorithm effects cannot fully meet the hardware and environmental limitations of space missions.

[0020] To this end, the present invention provides an on-orbit camera calibration method, which performs on-orbit binocular camera calibration based on vanishing points. By utilizing the rich linear features of solar panels, the final calibration is completed without the need for specific targets and within the computing capacity of the onboard computing platform, so as to meet the requirements of increasingly complex space missions for the accuracy and stability of on-orbit camera parameters.

[0021] like Figure 1 As shown, the on-orbit camera calibration method of the present invention comprises the following steps: Step 1: Use an on-orbit camera to obtain several sets of complete satellite images in a space lighting environment.

[0022] Spacecraft generally adopts a rigid frame structure, usually made of aluminum alloy, titanium alloy or composite material, and has a metallic luster such as silver, gray or aluminum on the surface. At the same time, in order to enhance corrosion resistance, radiation resistance and thermal protection performance, the surface of the spacecraft will be coated with special coatings, which will appear black, white or reflective silver. At the same time, there are obvious ribs and reinforcements on its surface, which are also displayed to a certain extent on the image. The solar panels of the spacecraft are rectangular, square or polygonal in shape, with a large surface area. The surface of the panels is covered with multiple solar cell units, which are usually arranged in regular rectangles and present a black, dark blue or green appearance. There are slender partitions between the battery cells, which are usually made of metal or plastic and play a role of isolation and support. The edges of the panels usually have metal frames, which are used to support and protect the battery cells, and the color is usually silver or aluminum. The above structural features will be reflected in the image with rich linear features. The present invention uses these structural features to complete the self-calibration of the on-orbit camera.

[0023] In the present invention, the number of satellite image groups can be any group. Obviously, the more groups there are, the more accurate the calibration result is. In each group of satellite images, a number of images are included, each of which contains the linear structure features required for the calibration of the present invention, and the more images there are, the more accurate the calibration result is. For example, in order to balance the computing resources and accuracy, the present invention selects three images.

[0024] Due to the position of the solar panels in the satellite, each image of the present invention contains the solar panels. Therefore, the straight line structure features of the present invention are provided by the solar panels, which overcomes the problem of lack of calibration targets in the traditional method.

[0025] Step 2: extract edge feature segments of the satellite image to obtain the optimal image distortion center position and preliminary radial distortion parameters.

[0026] In most cases, the center of image distortion is at the center of the image, but the center of distortion will often be displaced. Since a straight line will bend in a distorted image, the center of image distortion can be inferred from the bending of the straight line. This step can be specifically described as follows: Step 21, using the Canny edge detection algorithm, LSD algorithm and other algorithms, extract edge feature line segments in the satellite image and obtain their description equations. Since the solar panels provide straight line structural features in the present invention, the edge feature line segments here are the edge segments of the solar panels, and ideally, four edges.

[0027] Step 22: Ideally, the extracted edge feature line segment is not curved but a straight line segment. Therefore, if the lower left corner of the image is the Cartesian coordinate origin, the horizontal axis is the x-axis, and the vertical axis is the y-axis, then the linear equation is satisfied: in and is the linear equation coefficient. After distortion, the straight line will bend. Considering the distortion, a cubic polynomial is used to fit the equation to obtain a quadratic curve, which is expressed as: in , , , The curve parameters for fitting edge feature segments.

[0028] Step 23, assume that the image distortion center At the center of the image , that is, the initial value is the center point of the image , then according to the structural characteristics of satellite images, the above fitting curve should actually be a straight line, which is closer to the linear equation .

[0029] Define a reprojection error function: in, , is the width and height of the satellite image. The two end points of each edge feature line segment constitute a point set. The number of points in the set is n. is the coordinate of the i-th point in the ideal case, are the coordinates of the i-th point after correction considering distortion.

[0030] Step 24, calculate the gradient as follows, and perform nonlinear optimization on the reprojection error function: in, , , For point The distance to the center of distortion, is the coordinate of the point on the currently acquired distorted image, To ensure that the on-orbit camera is free of distortion The coordinates of the corresponding point.

[0031] The gradient is solved by the chain rule, and the Levenberg-Marquardt algorithm is used to iteratively optimize the final function. The final function of the present invention is expressed as: in Represents radial distortion parameters, obtains the quadratic curve of the edge feature line segment, optimizes the final function to obtain the distortion center point projected by each quadratic curve, obtains the average value of each distortion center point, which is the optimal image distortion center position, and optimizes the preliminary radial distortion parameters at the same time.

[0032] like Figure 2 As shown, in this embodiment, when the edge feature line segments extracted from the satellite image are four edge line segments of the solar panel, in step 24, the quadratic curves of all four edge line segments are obtained, and the final function is optimized to finally obtain the center points mapped by the four quadratic curves. , and find the average value of the four center points, expressed as: Obtain the optimal image distortion center position through optimization ,Right now Figure 2 Point A in .

[0033] Step 3: Extract all feature line segments of the satellite image and construct a feature line segment set. The specific implementation process can be described as follows: Step 31, using existing algorithms such as the Canny edge detection algorithm and the LSD algorithm, extract all feature line segments in the satellite image, that is, all extractable feature line segments of the solar panels in this embodiment, and ensure that the length of the extracted feature line segments is greater than the set value .

[0034] Step 32: Due to the unique structural characteristics of satellite images, the characteristic line segments have only two direction clusters. In the present invention, the optimal image distortion center position in step 2 is used. As reference point The feature line segments extracted in step 31 are marked with numbers , t is the number of feature line segments, select the feature line segment set close to the reference point , close to the reference point here means that the distance between the feature line segment and the reference point is less than pixels.

[0035] Step 33, taking the left endpoint of the characteristic line segment as the reference, detect the characteristic line segments one by one and set: in, and Respectively represent the left endpoint coordinates of the two feature line segments, It is the maximum angle between two feature line segments, the default value is ±3°.

[0036] If the two characteristic line segments satisfy the above formula, continue to calculate: in Represents the x-axis coordinates of the left endpoints of the two feature segments 1l With x 2l The distance between the left endpoint of a feature line segment and the x-axis coordinate x 1l The y-coordinate y of the left endpoint of another feature line segment 2l The distance between the left endpoint of a feature line segment and the y-axis coordinate y 1l The x-axis coordinate x of the left endpoint of another feature line segment 2l The distance between the two feature line segments and the y-axis coordinates of the left endpoints of the two feature line segments 1l With y 2l The distance between is the maximum distance between the four endpoints of the two feature line segments. If the above formula is satisfied, the two feature line segments are merged into one line segment, and each feature line segment is gradually traversed to obtain the line segment close to the reference point. The long line segment set .

[0037] Step 34, obtain the reference point close to the reference point through step 33 The long line segment set On this basis, continue to select the optimal image distortion center position As reference point For the feature line segments extracted in step 31, select the feature line segment set far away from the reference point , far away from the reference point here means that the distance between the feature line segment and the reference point is greater than pixels, repeat step 33, and use the same method to obtain the distance from the reference point The long line segment set , and The characteristic line segment set to be constructed in this step is formed. In this embodiment, the value of H can be 1024.

[0038] Step 4: Find the optimal vanishing point based on the feature line segment set.

[0039] In the derivation of projective geometry theory, the vanishing point is the point where parallel straight lines in space intersect on a two-dimensional image after passing through the camera. Ideally, the vanishing points of parallel lines in the same direction should be the same point. In reality, due to the existence of camera parameters and line feature extraction detection errors, the vanishing point is offset, so it is necessary to calculate the optimal vanishing point. This step can be described as follows: Step 41: Theoretically, line segments with the same direction intersect at one point , the present invention uses the actual intersection Define a set of long line segments close to the reference point Line segment , The endpoints are: , and , .

[0040] By solving the system of equations: Get the intersection for: , , , are the coefficients of the linear equation.

[0041] Step 42, setting the error equation to: Set the objective function as: Further deduction can be obtained: Where N is the number of intersection points, Represents the algorithm weight parameter. The weight is updated iteratively through the weighted least squares method: in A small positive number to prevent it from being 0; express The t+1th iteration result of express The t-th iteration result of .

[0042] Finally, the optimal intersection point can be obtained as: The optimal intersection point is the optimal vanishing point defined in the present invention.

[0043] Step 43, due to the special image features of satellite images, the long line segments close to the reference point are A cluster of line segments with two directions will obtain at least two vanishing points, namely Figure 3 In , , the set of long line segments far away from the reference point There will also be at least two vanishing points. So, 4-8 optimal vanishing points will be obtained.

[0044] Step 5: According to the optimal vanishing point, iterative optimization is performed to obtain the final internal parameters, external parameters and distortion parameters, which are the calibrated camera parameters.

[0045] Due to errors in the actual derivation and calculation process, it is impossible for all vanishing points to completely satisfy the ideal projection relationship. Therefore, it is necessary to iteratively solve the internal, external parameters and distortion parameters of the on-orbit camera through nonlinear optimization to minimize the reprojection error. This step can be described as follows: Define the above 4-8 optimal vanishing points as , since the set of parameters to be determined for the on-orbit camera is: in, represents the mth optimal vanishing point, M is the number of optimal vanishing points, That is, the internal parameters of the on-orbit camera. Since the center of the camera is selected as the center point of the world coordinate system, the external parameters are the unit matrix, and no additional calibration calculation is required. is the tangential distortion parameter, Together they constitute the distortion parameters.

[0046] The optimization objective function is: The objective function is a function that contains nine parameters. The nonlinear problem is Represents the current camera parameter set The derived vanishing point position coordinates, in the actual derivation, through the pre-acquired initial value, first calculate and derive a vanishing point position coordinates, and use it to calculate the loss, and then gradually iterate and optimize to obtain the final parameter set The present invention can specifically use the Levenberg-Marquardt algorithm to minimize the above objective function , get the final parameter set , expressed as: in, To optimize the final internal parameters of the on-orbit camera, To optimize the final tangential distortion parameters of the on-orbit camera, To optimize the final radial distortion parameters of the on-orbit camera, This is the final calibration result of the present invention.

Claims

1. An on-orbit camera calibration method, characterized in that: The steps include: Step 1: using an on-orbit camera to obtain several sets of complete satellite images in a space lighting environment, and each satellite image contains a solar panel; Step 2: extract edge feature segments of the satellite image to obtain the optimal image distortion center position and preliminary radial distortion parameters; Step 3, extract all feature line segments of the satellite image and construct a feature line segment set; Step 4, finding the optimal vanishing point according to the characteristic line segment set; Step 5: Iteratively optimize and obtain final internal and external parameters and distortion parameters according to the optimal vanishing point. The final internal and external parameters and distortion parameters are the calibrated camera parameters.

2. The on-orbit camera calibration method according to claim 1, characterized in that: In the step 1, each group of satellite images has a plurality of images.

3. The on-orbit camera calibration method according to claim 1, characterized in that: The step 2 is to extract edge feature segments of the satellite image to obtain the optimal image distortion center position and preliminary radial distortion parameters, and the implementation method is as follows: Step 21, extracting edge feature line segments of the satellite image and obtaining their description equations; Step 22: Ideally, the extracted edge feature line segment is a straight line segment, with the lower left corner of the image as the Cartesian coordinate origin, the horizontal as the x-axis, and the vertical as the y-axis, satisfying the linear equation: in and is the linear equation coefficient, considering the distortion, a cubic polynomial is used to fit the described equation to obtain a quadratic curve, which is expressed as: in , , , Curve parameters for fitting the edge feature line segment; Step 23, assume that the image distortion center The initial value is the center of the image , define a reprojection error function: in, , is the width and height of the satellite image. The two end points of each edge feature line segment constitute a point set. The number of points in the set is n. is the coordinate of the i-th point in the ideal case, is the coordinate of the i-th point after correction considering distortion; Step 24, calculate the gradient, perform nonlinear optimization on the reprojection error function, and the final function is expressed as: in Represents radial distortion parameters, obtains the quadratic curve of the edge feature line segment, optimizes the final function to obtain the distortion center point projected by each quadratic curve, obtains the average value of each distortion center point, which is the optimal image distortion center position, and optimizes the preliminary radial distortion parameters at the same time.

4. The on-orbit camera calibration method according to claim 3, characterized in that: In step 21, the edge feature line segments of the satellite image are extracted as four edge line segments of the solar panel; in step 24, the quadratic curves of all four edge line segments are obtained, the final function is optimized, and finally the center points mapped by the four quadratic curves are obtained. , and find the average value of the four center points, expressed as: The optimal image distortion center position is .

5. The on-orbit camera calibration method according to claim 3, characterized in that: In step 24, the gradient is defined as follows: in, , , For point The distance to the center of distortion, is the coordinate of the point on the currently acquired distorted image, To ensure that the on-orbit camera is free of distortion The coordinates of the corresponding point.

6. The on-orbit camera calibration method according to claim 1, characterized in that: The step 3 extracts all the characteristic line segments of the satellite image and constructs a characteristic line segment set. The implementation method is as follows: Step 31: extract all feature line segments of the satellite image and ensure that the length of the feature line segments is greater than the set value. ; Step 32, taking the optimal image distortion center position as the reference point, for the extracted feature line segments, select the line segments whose distance to the reference point is less than The feature line segment set of pixels ; Step 33, taking the left endpoint of the characteristic line segment as the reference, detect the characteristic line segments one by one and set: in, and Respectively represent the left endpoint coordinates of the two feature line segments, is the maximum angle between two characteristic line segments; If the two characteristic line segments satisfy the above formula, continue to calculate: in Represents the x-axis coordinates of the left endpoints of the two feature segments 1l With x 2l The distance between the left endpoint of a feature line segment and the x-axis coordinate x 1l The y-coordinate y of the left endpoint of another feature line segment 2l The distance between the left endpoint of a feature line segment and the y-axis coordinate y 1l The x-axis coordinate x of the left endpoint of another feature line segment 2l The distance between the two feature line segments and the y-axis coordinates of the left endpoints of the two feature line segments 1l With y 2l The distance between is the maximum distance between the four endpoints of the two feature line segments. If the above formula is satisfied, the two feature line segments are merged into one line segment, and each feature line segment is gradually traversed to obtain a set of long line segments close to the reference point. ; Step 34, select the optimal image distortion center position as the reference point, and for the extracted feature line segments, select the line segments that are greater than the distance from the reference point. The feature line segment set of pixels Repeat step 33 to obtain the set of long line segments far away from the reference point , and Composed of feature line segment sets.

7. The on-orbit camera calibration method according to claim 6, characterized in that: The step 4, according to the characteristic line segment set, finds the optimal vanishing point, and the implementation method is as follows: Step 41, through the actual intersection Theoretical intersection points of line segments with the same fitting direction , define the set of long line segments close to the reference point The line segment in , The coordinates of the left and right endpoints are: , and , ; By solving the system of equations: Get the intersection for: , , , is the coefficient of the linear equation; Step 42, setting the error equation to: Set the objective function as: but: N is the number of intersection points, Represents the algorithm weight parameter, and the weight is iteratively updated by weighted least squares method: , A small positive number to prevent it from being 0; express The t+1th iteration result of express The tth round iteration result of ; Finally, the optimal intersection point is obtained: The optimal intersection point is the optimal vanishing point; Step 43: Set of long line segments close to the reference point There are two-directional line segment clusters, which obtain at least two vanishing points and a long line segment set far away from the reference point. There will also be at least two vanishing points, and thus 4-8 optimal vanishing points are obtained.

8. The on-orbit camera calibration method according to claim 7, characterized in that: In step 5, according to the optimal vanishing point, iterative optimization is performed to obtain the final internal and external parameters and distortion parameters. The implementation method is as follows: The 4-8 optimal vanishing points are defined as , the set of parameters to be determined for the on-orbit camera is: in, represents the mth optimal vanishing point, M is the number of optimal vanishing points, That is, the internal parameters of the on-orbit camera, is the tangential distortion parameter; the external parameters of the on-orbit camera are the unit matrix; The optimization objective function is: in, Represents the current camera parameter set The derived vanishing point position coordinates are used to minimize the objective function using the Levenberg-Marquardt algorithm , and get the final parameter set.

Images