On-orbit pointing resolution method based on small field of view array camera constrained by ground surface buildings

Abstract

The present application relates to remote sensing image geometric inversion technical field, specifically for small field of view array camera pointing on-orbit solution method based on surface building constraint, obtain single image shot by small field of view array camera, extract regular building outer contour direction in image, based on geometric relation that building facade is perpendicular to ground and adjacent facade is perpendicular to each other, establish direction constraint between building own coordinate system and world coordinate system;Then, according to small field of view imaging approximate relation, utilize the extraction direction of building three-axis direction in image coordinate system and theoretical projection direction to form equation group, and solve camera rotation matrix and building azimuth angle parameter through nonlinear optimization;The method does not depend on camera internal parameter, external posture sensor and multiple images, and can realize self-on-orbit solution of array camera pointing only through single image.

Description

Technical Field

[0001] This invention relates to the field of geometric inversion technology of remote sensing images, specifically to an on-orbit solution method for pointing small-view array cameras based on surface building constraints. Background Technology

[0002] With the development of remote sensing mapping, space monitoring, and long-range target observation technologies, long-focal-length imaging equipment has been widely used in high-resolution surface observation, target area identification, and spatial information acquisition. These imaging tasks typically require the imaging equipment to maintain a stable pointing relationship during operation to ensure a usable correspondence between image data and geospatial information. Due to the complex operating environment of the imaging platform, the equipment may be affected by factors such as vibration, temperature changes, structural assembly misalignment, and load attitude changes during long-term use, leading to deviations between the actual imaging pointing and the preset pointing, thus affecting the reliability of subsequent positioning, mapping, and target analysis results.

[0003] In existing technologies, pointing calculations for imaging devices typically rely on external attitude measurement devices, pre-calibrated parameters, or motion relationships between multiple frames of images. In practical applications, external measurement devices are susceptible to changes in installation relationships, environmental occlusion, or attitude transfer errors. Calculation methods relying on preset intrinsic and extrinsic parameters are ill-suited for on-orbit use after parameter drift in imaging devices. Processing methods based on multiple frames place high demands on image sequences, feature continuity, and computational resources, making them unsuitable for applications requiring only single-image acquisition or rapid calculation. Therefore, in situations lacking stable prior parameters or with limited external measurement conditions, extracting constraint relationships from the observable information within the image itself for pointing calculation has become a crucial technological direction for improving the autonomous calculation capabilities of imaging systems.

[0004] Regular ground features exhibit strong structural stability in remote sensing imagery and long-range observation images. Their contour orientation, boundary relationships, and spatial geometric relationships can provide usable geometric constraints for attitude calculation of imaging equipment. However, existing methods often rely on relatively complete camera parameters, obvious vanishing point locations, or known spatial control points when utilizing such geometric information. When the imaging field of view is small, the vanishing point is located outside the image range, or there are few available features in the image, problems such as insufficient constraints, computational instability, or difficulty in converging the solution results are prone to occur. Therefore, there is a need for an on-orbit solution technique that can establish a constraint model using the geometric relationships of regular ground features under limited image information conditions, in order to improve the autonomous pointing solution capability of imaging equipment under complex operating conditions.

[0005] In view of this, the present invention proposes an on-orbit solution method for pointing of small-view array cameras based on surface building constraints. Summary of the Invention

[0006] The purpose of this invention is to provide an on-orbit solution method for pointing of a small-view field array camera based on surface building constraints. Under the condition that the intrinsic parameters of the field array camera are unknown, the shooting pose of the small-view field array camera is estimated by using the outer contour of regular buildings in a single image, so as to realize the autonomous on-orbit solution of camera pointing.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides an on-orbit solution method for pointing of a small-view field-array camera based on surface building constraints, comprising the following steps:

[0009] S101, acquire the image to be solved by the tilted capture of the small field-of-view array camera, the image to be solved is a remote sensing image that has been radiometrically corrected but not geometrically corrected, and extract the outer contour line segments of regular buildings from the image to be solved, the regular buildings being buildings that can form two adjacent lateral and vertical contours in the image to be solved.

[0010] S102, based on the outer contour line segments, determine the first lateral direction, second lateral direction, and vertical direction corresponding to each regular building in the image coordinate system, wherein the regular building has an exterior facade perpendicular to the ground, and adjacent exterior facades satisfy a geometric relationship of mutual perpendicularity in space, the vertical direction is perpendicular to the ground and upwards, and the first lateral direction, second lateral direction, and vertical direction conform to the right-hand coordinate system;

[0011] S103, Select a regular building as the reference building, establish a world coordinate system, and set the azimuth parameters of the rotation of the other regular buildings relative to the reference building around the vertical axis. Construct the three-axis orientation of each regular building in the world coordinate system based on the azimuth parameters.

[0012] S104. Based on the small field of view imaging approximation relationship, the three-axis direction of the building is mapped to the camera coordinate system through the rotation matrix to be solved, and the theoretical image direction is formed according to the horizontal and vertical components of the transformed direction vector in the camera coordinate system. The theoretical image direction is then used to establish a set of direction consistency constraint equations with the first lateral direction, the second lateral direction, and the vertical direction in the image coordinate system.

[0013] S105, perform nonlinear least squares optimization on the set of direction consistency constraint equations to obtain the rotation matrix from the world coordinate system to the camera coordinate system, and output the pointing solution result of the small field-of-view array camera based on the rotation matrix.

[0014] As a preferred technical solution of the present invention, extracting the outer contour line segments of regular buildings from the image to be solved includes:

[0015] Edge segments are extracted from the building boundaries in the image to be solved to obtain a set of candidate segments; based on the intersection, extension direction and closed contour relationships between the candidate segments, the contour segments of the corresponding building facade are selected.

[0016] Outline segments that can form two adjacent lateral directions and vertical edge directions are used as the outer outline segments of regular buildings; building outlines that cannot correspond to the vertical facade or cannot form a three-axis relationship are eliminated.

[0017] As a preferred embodiment of the present invention, determining the first lateral direction, the second lateral direction, and the vertical direction corresponding to each regular building in the image coordinate system includes:

[0018] The two sets of horizontal outer contour line segments corresponding to adjacent facades of the same building are respectively merged into the first lateral line segment group and the second lateral line segment group;

[0019] The outer contour lines corresponding to the building height direction are grouped into vertical line segments; based on the image coordinate changes of the endpoints of each line segment in the first lateral line segment group, the second lateral line segment group, and the vertical line segment group, the first lateral direction vector, the second lateral direction vector, and the vertical direction vector are generated respectively.

[0020] Line segments that are opposite in direction but collinear are made to be in the same direction, so that the three axes of the same building have a consistent coordinate representation basis, and the first lateral direction, the second lateral direction and the vertical direction conform to the right-hand coordinate system.

[0021] As a preferred technical solution of the present invention, establishing a world coordinate system includes:

[0022] Regular buildings in the image whose contour line segments meet the construction conditions in the three-axis direction are selected as the reference buildings;

[0023] The first lateral direction of the reference building corresponds to the x-axis of the world coordinate system, the second lateral direction of the reference building corresponds to the y-axis of the world coordinate system, and the direction perpendicular to the ground corresponds to the z-axis of the world coordinate system.

[0024] For the nth regular building, set its azimuth parameter θ relative to the reference building about the z-axis. n Based on this azimuth parameter, the first lateral axis, second lateral axis, and vertical axis of the nth regular building in the world coordinate system are generated.

[0025] As a preferred embodiment of the present invention, the first lateral axis and the second lateral axis of the nth regular building in the world coordinate system are determined by its azimuth parameters, and the vertical axis of the nth regular building is in the same direction as the vertical axis of the reference building; wherein, the first lateral axis and the second lateral axis satisfy a perpendicular relationship, and the directions of the first lateral axis, the second lateral axis and the vertical axis conform to the right-hand coordinate system.

[0026] As a preferred embodiment of the present invention, a set of directional consistency constraint equations is constructed, including:

[0027] The three-axis orientation vectors of buildings in the world coordinate system are transformed to the camera coordinate system through rotation matrices. Based on the small field of view approximation in the imaging of a small field-of-view array camera, the theoretical orientations of the three-axis orientations of each building in the image coordinate system are obtained according to the transformed orientation components.

[0028] Collinear constraints are established between the theoretical direction and the first lateral direction, the second lateral direction, and the vertical direction in the image coordinate system obtained in S102, respectively, and the collinear constraints corresponding to each regular building are combined into a set of direction consistency constraint equations.

[0029] As a preferred embodiment of the present invention, the collinearity constraint is established through the two-dimensional cross product relationship between the image direction vector and the theoretical direction vector;

[0030] Using the two-dimensional cross product result as the collinearity constraint value, ideally the actual direction vector of the image is collinear with the theoretical direction vector, that is, the two-dimensional cross product result is 0. This is used as the condition for determining whether the actual direction vector of the image and the theoretical direction vector satisfy the direction consistency constraint.

[0031] The two-dimensional cross product relationships corresponding to the first lateral direction, the second lateral direction, and the vertical direction are written into the constraint expression of the same building, and the constraint expressions of multiple regular buildings are solved jointly.

[0032] As a preferred embodiment of the present invention, solving the set of directional consistency constraint equations includes:

[0033] The attitude parameters of the rotation matrix to be solved and the azimuth parameters of each regular building except the reference building are taken as variables to be solved; the constraint difference of the direction consistency constraint equation set is taken as the optimization object, and the variables to be solved are solved jointly by nonlinear optimization method.

[0034] During the solution process, the orthogonality constraint of the rotation matrix is ​​maintained so that the obtained rotation matrix meets the requirements of camera pose representation.

[0035] As a preferred embodiment of the present invention, it further includes an abnormal building constraint removal step:

[0036] Detect the angular relationship between the vertical direction and the first or second lateral direction of the same regular building in the image coordinate system;

[0037] When the vertical direction is collinear with the first lateral direction or the second lateral direction, the building is marked as a degenerate constraint building.

[0038] When only degenerate-constrained buildings exist, regular buildings with different azimuth parameters are introduced to participate in the solution; when non-degenerate-constrained buildings exist, the directional consistency constraints corresponding to the degenerate-constrained buildings are removed from the equation set or their constraint weights are reduced.

[0039] As a preferred technical solution of the present invention, the pointing solution result of the small field-of-view array camera includes the rotation matrix from the world coordinate system to the camera coordinate system, the attitude angle converted by the rotation matrix, and the orientation consistency residual; the orientation consistency residual is used to characterize the deviation between the coordinate axis direction of the extracted building in the image and the theoretical direction obtained by projection through the rotation matrix, and serves as the basis for the autonomous on-orbit pointing solution of the camera.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This invention extracts the outer contour direction of regular buildings from a single small field-of-view image and constructs pointing constraints using the geometric relationship between the building facade and the ground and the mutual perpendicularity of adjacent facades. This enables the camera to solve the shooting posture based on the image's own information even when the intrinsic parameters are unknown or drift occurs. Since this process does not rely on external attitude sensors, preset control points, or multiple consecutive images, it can reduce the impact of installation angle changes, intrinsic parameter drift, and missing image sequences on the pointing calculation results. This enables autonomous on-orbit pointing calculation of small field-of-view array cameras, improving the reliability of the data foundation for subsequent positioning, mapping, and target analysis of remote sensing images. Attached Figure Description

[0042] Figure 1 This is a flowchart of the autonomous on-orbit solution method for pointing small-view field array cameras according to the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the extraction of the outer contour and three-axis direction of the regular building according to the present invention;

[0044] Figure 3 This is a schematic diagram of the angle error distribution for solving a single simulated building according to the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In the description of this invention, it should be noted that the terms "vertical," "upper," "lower," "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0047] Example 1

[0048] Please see Figure 1 This invention provides a technical solution for on-orbit resolution of pointing parameters using a small-view field-of-sight camera based on surface building constraints. It is applicable to high-resolution remote sensing, long-distance target observation, UAV telephoto imaging, and other scenarios captured by small-view field-of-sight cameras. The small-view field-of-sight camera is a telephoto imaging device with both horizontal and vertical field of view angles less than 4°. The image to be resolved is a single remote sensing image formed by tilted capture by the small-view field-of-sight camera. The image to be resolved has undergone radiometric correction but not geometric correction. A regular building is defined as a building from which at least two adjacent facades can be extracted in terms of direction and height, and where adjacent facades in actual space satisfy a vertical relationship and are perpendicular to the ground. The method includes the following steps:

[0049] S101. Obtain the image to be solved and extract the outer contour of regular buildings;

[0050] An image to be solved is acquired by a small-view field-array camera taking tilted images. The image to be solved is a remote sensing image that has undergone radiometric correction but not geometric correction. The image to be solved contains multiple building outlines, road outlines, and terrain feature boundaries.

[0051] like Figure 2 As shown, buildings with regular facade structures are selected from the image to be solved as candidate regular buildings, and their outer contours are represented by lines. Candidate regular buildings are represented by rectangles, polygons, or combined contours, and the outer contour segments of the buildings participating in the three-axis extraction are represented by thick solid lines. , ,..., This represents the three-axis reference coordinate system for the corresponding regular building. , and These represent the projection directions of the first lateral direction, the second lateral direction, and the vertical direction of the nth regular building in the image to be solved.

[0052] Building boundaries are extracted from the image to be solved to obtain a set of candidate line segments. Based on the intersection, extension direction, and closed contour relationships between the candidate line segments, contour lines corresponding to the facades of regular buildings are selected from the candidate line segment set. Buildings that can form two adjacent lateral directions and a vertical edge direction are treated as regular buildings and participate in subsequent processing. Building contours that cannot form two lateral directions or a vertical direction are not included in the subsequent directional consistency constraint equations.

[0053] S102. Determine the three-axis direction vectors of the building based on the outer contour line segments.

[0054] For the nth regular building, the outer contour lines belonging to the horizontal extension direction of the first facade are grouped into the first lateral line group, the outer contour lines belonging to the horizontal extension direction of the second facade are grouped into the second lateral line group, and the outer contour lines corresponding to the building height direction are grouped into the vertical line group.

[0055] Based on the image coordinate changes of the endpoints of each line segment in the first lateral line segment group, the second lateral line segment group, and the vertical line segment group, the first lateral direction vector, the second lateral direction vector, and the vertical direction vector of the nth regular building in the image coordinate system are generated respectively, denoted as […]. , and ;

[0056] in, , These represent the components of the first lateral direction vector in the horizontal and vertical directions of the image coordinate system, respectively; , These represent the components of the second lateral direction vector in the horizontal and vertical directions of the image coordinate system, respectively; , These represent the components of the vertical direction vector in the horizontal and vertical directions of the image coordinate system, respectively.

[0057] It should be noted that: Figure 2 The marked , and The direction does not directly represent the actual spatial coordinate axis in the world coordinate system, but rather the directional observation result of the two-dimensional image extracted from the outer contour line segment of the building. The first lateral direction and the second lateral direction correspond to the horizontal extension direction of two adjacent facades of the building, respectively, and the vertical direction corresponds to the edge direction of the building perpendicular to the ground. For the same axial line segment of the same building, line segments with opposite directions but collinear are unified into the same direction expression. The vertical direction refers to the vertically upward direction. The first lateral direction, the second lateral direction, and the vertical direction conform to the right-hand coordinate system. For buildings that cannot form two sets of lateral directions and one set of vertical directions, they are not included in the subsequent constraint equations.

[0058] S103. Establish a reference coordinate system and set the azimuth parameters;

[0059] A regular building in the image whose contour line segment integrity meets the three-axis direction construction condition is selected as the reference building. The first lateral direction of the reference building corresponds to the x-axis of the world coordinate system, the second lateral direction of the reference building corresponds to the y-axis of the world coordinate system, and the direction perpendicular to the ground upward corresponds to the z-axis of the world coordinate system, thus establishing a world coordinate system.

[0060] For the nth regular building other than the reference building, an azimuth parameter is set to represent the rotation of the nth regular building's own coordinate system relative to the reference building's own coordinate system. The first lateral axis and the second lateral axis of the nth regular building in the world coordinate system are determined by the azimuth parameter, and the vertical axis of the nth regular building is in the same direction as the vertical axis of the reference building.

[0061] S104. Establish the approximate relationship for small field-of-view imaging and construct the set of directional consistency constraint equations.

[0062] When using a small field-of-view area scan camera to photograph a regular building at an angle, the edge contour of the regular building is projected onto the image coordinate system after the camera pose is transformed. In order to establish the correspondence between the spatial orientation of the building and the orientation of the image, this step physically models the process of projecting the edge contour of the building onto the area scan camera.

[0063] Let the rotation and translation matrices from the world coordinate system to the camera coordinate system satisfy:

[0064] ; (1)

[0065] Where: R represents the rotation matrix from the world coordinate system to the camera coordinate system; T represents the translation matrix from the world coordinate system to the camera coordinate system; to Represents the matrix elements in the rotation matrix R; , , These represent the components of the translation matrix T along the x, y, and z axes, respectively.

[0066] Consider a straight line in the world coordinate system The line passes through the point The direction is along the unit vector: ;but Any point on can be represented as:

[0067] ; (2)

[0068] in: Represents a straight line in the world coordinate system; , , Represents a straight line The three-dimensional coordinates of the known point A traversed in the world coordinate system; , , Represents a straight line The three-dimensional coordinates of any point on the map in the world coordinate system; The parameters represent the corresponding points on line l; , , Represents a straight line The components of the unit direction vector in the x-axis, y-axis, and z-axis directions of the world coordinate system.

[0069] A point in the world coordinate system In a straight line Above, the corresponding parameters are In the camera coordinate system, Coordinates are Then the following relationship is satisfied:

[0070] ; (3)

[0071] in: Indicates a line in the world coordinate system The point on; Point The corresponding point after transformation to the camera coordinate system; , , Point Three-dimensional coordinates in the world coordinate system; , , Point The 3D coordinates in the camera coordinate system; R represents the rotation matrix from the world coordinate system to the camera coordinate system; T represents the translation matrix from the world coordinate system to the camera coordinate system. to Represents the matrix elements in the rotation matrix R; , , These represent the three translation components of the translation matrix T.

[0072] The above equation can be simplified to:

[0073] ; (4)

[0074] in: Point In a straight line The parameter values ​​on; to Represents the matrix elements in the rotation matrix R; , , Represents a straight line The components of the unit direction vector along the three coordinate axes of the world coordinate system; , , Represents a straight line The three-dimensional coordinates of the known point A traversed in the world coordinate system; , , These represent the three translation components of the translation matrix T;

[0075] For ease of explanation, the following variable substitutions are made, let:

[0076] ; (5)

[0077] Where: A, B, and C can be regarded as the components of the unit direction vector of the line l in the world coordinate system after being transformed by the rotation matrix R in the x-axis, y-axis, and z-axis directions of the camera coordinate system; D, E, and F are the substitution variables used for convenience.

[0078] Equation (4) can then be rewritten as:

[0079] ; (6)

[0080] Now consider the straight line The point that is L away from point P Then point Q corresponds to the parameter in equation (2). satisfy The coordinates of point Q in the camera coordinate system are: ,satisfy:

[0081] ; (7)

[0082] According to equation (7), the vector in the camera coordinate system It can be written as:

[0083] ; (8)

[0084] in: Represents a straight line The point that is L away from point P; Point The corresponding point after transformation to the camera coordinate system; , , Point Three-dimensional coordinates in the camera coordinate system; , , Point P C Three-dimensional coordinates in the camera coordinate system; This represents the parameter value of point Q on line l; This indicates that point P lies on the line. The parameter value is L; L represents the distance parameter between point P and point Q along the direction of line l.

[0085] From equation (8), we can see that: in the world coordinate system, the length is L, and the direction is along... The building vector corresponding to the vector in the camera coordinate system is (AL, BL, CL), which represents the direction of the building edge in the camera coordinate system after the camera pose transformation.

[0086] The building outline vector is then transformed from the camera coordinate system to the image coordinate system. Based on the small field-of-view imaging condition, an approximate directional relationship is obtained, and the transformation relationship between the three-axis orientation of the building and the rotation matrix is ​​constructed accordingly. The intrinsic parameter matrix of the area scan camera is denoted as:

[0087] ; (9)

[0088] Where K is the intrinsic parameter matrix, f x and f y c represents the equivalent pixel value of the focal length of the area array camera along the x and y directions, respectively. x and c y This represents the optical center coordinates of the area array camera. (The point is...) and points Projected onto the image coordinate system of the area scan camera, the corresponding point is denoted as... and We can obtain:

[0089] ; (10)

[0090] Correspondingly, vector It can be represented as:

[0091] ; (11)

[0092] The following is about and analyze:

[0093] ;(12)

[0094] According to equation (5), A, B, and C can be written as:

[0095] ; (13)

[0096] The geometric meaning of equation (13) is: to transform the unit vector in the world coordinate system Transforming to the camera coordinate system, the resulting vector is... A, B, and C are the components of the unit vector on the x, y, and z axes of the camera coordinate system. Generally, for a vector that has a certain angle with the camera's optical axis, its corresponding A, B, and C should belong to the same order of magnitude and all of them should not be 0.

[0097] According to equation (3). , and In camera coordinate system The coordinates of the point, assuming the horizontal and vertical field of view of the area array camera are respectively... , ,but , and The following conditions must be met:

[0098] ;(14)

[0099] For small field-of-view area array cameras, typically , Less than 4°, therefore we can obtain:

[0100] ; (15)

[0101] Based on equations (13) and (15), equation (12) is rewritten as follows:

[0102] ; (16)

[0103] As mentioned above, since A, B, and C belong to the same order of magnitude, , Therefore and Compared to A and B, which are higher-order minor quantities, they can be ignored. Therefore... and It can be approximated as:

[0104] ; (17)

[0105] Furthermore, according to equation (11), the vector It can be approximated as:

[0106] ; (18)

[0107] In summary, according to equation (18), the length in the world coordinate system is L, and the direction is along... The vector corresponding to the vector in the camera's image coordinate system can be approximately represented as:

[0108] ;

[0109] Typically, for telephoto area array cameras with a small field of view, and Almost the same, the absolute value of C is less than 1, and the order of magnitude of CL is usually in the tens of meters range, z p The magnitude of zp' is typically on the order of hundreds of thousands of meters, so the size of CL is negligible compared to zp'. Here, we take two approximations: 1) fx = fy; 2) zp' + CL = zp', then the vector can be further approximated as:

[0110] ; (19)

[0111] If we only focus on the direction of the vector here, it can be further simplified as follows:

[0112] ; (20)

[0113] Equation (20) shows that, under the condition of a small-view area array camera, after the direction vector in the world coordinate system is transformed to the camera coordinate system by a rotation matrix, its image direction can be characterized by components A and B in the camera coordinate system. Equation (20) is the basis for solving the camera pointing problem under the condition of unknown area array camera intrinsic parameters.

[0114] Based on the above small field-of-view imaging approximation relationship, the transformation relationship between the three-axis directions of the building and the rotation matrix is ​​established.

[0115] Extract images directly captured by a satellite (camera parameters unknown). It is assumed that the sides of each building are perpendicular to the ground, and the angle between adjacent sides is 90°. Establish a coordinate system O for each building. n -xyz, where the x and y axes are along the side direction and the z axis is perpendicular to the ground and upward, the three axes satisfy the right-hand rule. Taking the self-coordinate system O1-xyz of the first building as the reference, a world coordinate system is established, and the rotation matrix R from the world coordinate system to the camera coordinate system is set, satisfying:

[0116] ; (twenty one)

[0117] Let the first The building's own coordinate system has rotated about the z-axis relative to the reference building. Then the coordinate system of the building itself , and The axis can be represented in the world coordinate system as:

[0118] ; (twenty two)

[0119] According to equations (20) and (5), , , The corresponding vector on the image , , The direction can be written as:

[0120] ; (twenty three)

[0121] On the other hand, the outer contour of a building can be extracted from the captured image, that is, the directions of the x, y, and z coordinate axes of each building in the image, denoted as , and The direction of the extracted image vector should be parallel to the direction of the theoretical derivation. Based on this, the following system of equations can be obtained:

[0122] ; (twenty four)

[0123] Therefore, for each building, we can obtain a set of equations in the form of equation (24). Considering that theoretically the z-axis direction of each building is the same, each additional building will add three equations, of which two are effective new equations, and an unknown azimuth parameter is added. Since the first building is used as the reference, the azimuth parameters corresponding to the first building are... The value is 0. The following is a system of equations for buildings 1 to n:

[0124] ; (25)

[0125] According to equation (25), given multiple building outlines, the rotation matrix R can be solved by substituting it into the system of equations.

[0126] The discussion regarding the full-rank problem of the system of equations is as follows. The unknown parameters of the system of equations are: the rotation matrix R (with 3 degrees of freedom), and the azimuth parameters of each building. The number of parameters is n-1, and the total number of unknown parameters is n+2. Each building can provide 3 constraint equations. Since the z-axis of each building coincides, the remaining buildings (excluding the first building) can provide 2 effective constraint equations, for a total of 2n+1 constraint equations. When the number of buildings is greater than or equal to 1, the number of constraint equations is greater than or equal to the number of unknown parameters, and the system of equations can be solved.

[0127] S105. Solve the rotation matrix using nonlinear optimization and output the solution results.

[0128] Equation (25) is used as a set of direction consistency constraint equations, and the attitude parameters of the rotation matrix R and the azimuth parameters of each building are considered. As variables to be solved, the rotation matrix R and the azimuth parameters of each building are obtained through nonlinear optimization.

[0129] To ensure that the rotation matrix R represents an effective camera pose, the orthogonality of the rotation matrix is ​​maintained during the solution process, or the rotation matrix is ​​parameterized using Euler angles, quaternions, or Lie algebras. After the solution is completed, the optimization result is converted into a rotation matrix R from the world coordinate system to the camera coordinate system, and the pointing solution result of the small-view array camera is output based on this rotation matrix. The pointing solution result includes the rotation matrix R and the attitude angles converted from the rotation matrix.

[0130] Before solving equation (25), a degradation judgment is performed on the three-axis orientation of each building. There is an extreme case: the z-axis of the building in the image is collinear with the y-axis or the two axes. In this case, the camera pose cannot be determined based on a single building, and it is necessary to introduce other buildings with different azimuth parameters to solve the pose.

[0131] Specifically, when the vertical direction extracted from the image With the first lateral direction Or the second lateral direction When buildings are collinear, they are marked as degenerate constraint buildings. Degenerate constraint buildings are not used alone for camera attitude solving; when there are buildings with different azimuth parameters, they are all included in equation (25) for joint solution.

[0132] To verify the accuracy of the attitude angle calculation, a shooting scene of a small field-of-view camera was simulated using 3D software. Seventeen regular cubes with different azimuth parameters were added to the scene to simulate buildings. The size of the camera image was set to 1920x1080, with horizontal and vertical field of view angles of 3° and 1.69°, respectively, and the camera side tilt angle was 30°.

[0133] During the simulation, the three-axis directions extracted from a single regular cube are used as constraint inputs. The corresponding direction consistency constraint equations are established according to equation (24), and the corresponding rotation matrices are obtained by solving them. The obtained rotation matrices are compared with the actual rotation matrices in the simulation scene, and the sum of the absolute values ​​of the differences between the corresponding three-axis Euler angles is used as the angle error.

[0134] like Figure 3 As shown, the angular errors obtained when different simulated buildings participate in the solution individually vary. These differences are related to the projection direction of the corresponding building in the image, the building's own azimuth parameters, and the distinguishability between the three-axis directions. When the three-axis directions of a single building can form effective directional constraints in the image, its corresponding angular error is within an acceptable range, all less than 2°, with an average error of 0.907° for all buildings. If all buildings are used to calculate the rotation matrix together, the angular error can be significantly reduced to 0.16°.

[0135] Therefore, in the actual solution process, degradation judgment is performed on the three-axis direction of each regular building; when the vertical direction extracted from the image is collinear with the first lateral direction or the second lateral direction, the building is marked as a degenerate constraint building and is not used alone for camera attitude solution. When there are multiple regular buildings with different azimuth parameters in the image to be solved, the non-degenerate regular buildings are jointly included in the direction consistency constraint equation set shown in equation (25) for joint solution, so as to reduce the influence of single building constraint degradation on the rotation matrix solution result.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An on-orbit solution method for pointing of small-view field-array cameras based on surface building constraints, characterized in that: Includes the following steps: S101, acquire the image to be solved by the tilted capture of the small field-of-view array camera, the image to be solved is a remote sensing image that has been radiometrically corrected but not geometrically corrected, and extract the outer contour line segments of regular buildings from the image to be solved, the regular buildings being buildings that can form two adjacent lateral and vertical contours in the image to be solved. S102, based on the outer contour line segments, determine the first lateral direction, second lateral direction, and vertical direction corresponding to each regular building in the image coordinate system, wherein the regular building has an exterior facade perpendicular to the ground, and adjacent exterior facades satisfy a geometric relationship of mutual perpendicularity in space, the vertical direction is perpendicular to the ground and upwards, and the first lateral direction, second lateral direction, and vertical direction conform to the right-hand coordinate system; S103, Select a regular building as the reference building, establish a world coordinate system, and set the azimuth parameters of the rotation of the other regular buildings relative to the reference building around the vertical axis. Construct the three-axis orientation of each regular building in the world coordinate system based on the azimuth parameters. S104. Based on the small field of view imaging approximation relationship, the three-axis direction of the building is mapped to the camera coordinate system through the rotation matrix to be solved, and the theoretical image direction is formed according to the horizontal and vertical components of the transformed direction vector in the camera coordinate system. The theoretical image direction is then used to establish a set of direction consistency constraint equations with the first lateral direction, the second lateral direction, and the vertical direction in the image coordinate system. S105, perform nonlinear least squares optimization on the set of direction consistency constraint equations to obtain the rotation matrix from the world coordinate system to the camera coordinate system, and output the pointing solution result of the small field-of-view array camera based on the rotation matrix.

2. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, Extracting the outer contour segments of regular buildings from the image to be solved includes: Edge segments are extracted from the building boundaries in the image to be solved to obtain a set of candidate segments; based on the intersection, extension direction and closed contour relationships between the candidate segments, the contour segments of the corresponding building facade are selected. Outline segments that can form two adjacent lateral directions and vertical edge directions are used as the outer outline segments of regular buildings; building outlines that cannot correspond to the vertical facade or cannot form a three-axis relationship are eliminated.

3. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, Determine the first lateral direction, second lateral direction, and vertical direction of each regular building in the image coordinate system, including: The two sets of horizontal outer contour line segments corresponding to adjacent facades of the same building are respectively merged into the first lateral line segment group and the second lateral line segment group; The outer contour lines corresponding to the building height direction are grouped into vertical line segments; based on the image coordinate changes of the endpoints of each line segment in the first lateral line segment group, the second lateral line segment group, and the vertical line segment group, the first lateral direction vector, the second lateral direction vector, and the vertical direction vector are generated respectively. Line segments that are opposite in direction but collinear are made to be in the same direction, so that the three axes of the same building have a consistent coordinate representation basis, and the first lateral direction, the second lateral direction and the vertical direction conform to the right-hand coordinate system.

4. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, Establishing a world coordinate system includes: Regular buildings in the image whose contour line segments meet the construction conditions in the three-axis direction are selected as the reference buildings; The first lateral direction of the reference building corresponds to the x-axis of the world coordinate system, the second lateral direction of the reference building corresponds to the y-axis of the world coordinate system, and the direction perpendicular to the ground corresponds to the z-axis of the world coordinate system. For the nth regular building, set its azimuth parameter θ relative to the reference building about the z-axis. n Based on this azimuth parameter, the first lateral axis, second lateral axis, and vertical axis of the nth regular building in the world coordinate system are generated.

5. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 4, characterized in that, The first and second lateral axes of the nth regular building in the world coordinate system are determined by its azimuth parameters. The vertical axis of the nth regular building is in the same direction as the vertical axis of the reference building. The first and second lateral axes are perpendicular to each other, and the directions of the first, second, and vertical axes conform to the right-hand coordinate system.

6. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, Construct a system of directional consistency constraint equations, including: The three-axis orientation vectors of buildings in the world coordinate system are transformed to the camera coordinate system through rotation matrices. Based on the small field of view approximation in the imaging of a small field-of-view array camera, the theoretical orientations of the three-axis orientations of each building in the image coordinate system are obtained according to the transformed orientation components. Collinear constraints are established between the theoretical direction and the first lateral direction, the second lateral direction, and the vertical direction in the image coordinate system obtained in S102, respectively, and the collinear constraints corresponding to each regular building are combined into a set of direction consistency constraint equations.

7. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 6, characterized in that, The collinearity constraint is established through the two-dimensional cross product relationship between the image direction vector and the theoretical direction vector; Using the two-dimensional cross product result as the collinearity constraint value, ideally the actual direction vector of the image is collinear with the theoretical direction vector, that is, the two-dimensional cross product result is 0. This is used as the condition for determining whether the actual direction vector of the image and the theoretical direction vector satisfy the direction consistency constraint. The two-dimensional cross product relationships corresponding to the first lateral direction, the second lateral direction, and the vertical direction are written into the constraint expression of the same building, and the constraint expressions of multiple regular buildings are solved jointly.

8. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, Solving the aforementioned system of directional consistency constraint equations includes: The attitude parameters of the rotation matrix to be solved and the azimuth parameters of each regular building except the reference building are taken as variables to be solved; the constraint difference of the direction consistency constraint equation set is taken as the optimization object, and the variables to be solved are solved jointly by nonlinear optimization method. During the solution process, the orthogonality constraint of the rotation matrix is ​​maintained so that the obtained rotation matrix meets the requirements of camera pose representation.

9. The method for on-orbit calculation of pointing of small-view field array cameras based on surface building constraints according to claim 1, characterized in that, It also includes the abnormal building constraint removal step: Detect the angular relationship between the vertical direction and the first or second lateral direction of the same regular building in the image coordinate system; When the vertical direction is collinear with the first lateral direction or the second lateral direction, the building is marked as a degenerate constraint building. When only degenerate-constrained buildings exist, regular buildings with different azimuth parameters are introduced to participate in the solution; when non-degenerate-constrained buildings exist, the directional consistency constraints corresponding to the degenerate-constrained buildings are removed from the equation set or their constraint weights are reduced.

10. The method for on-orbit calculation of pointing of a small-view field-array camera based on surface building constraints according to claim 1, characterized in that, The pointing solution results of the small-view field array camera include the rotation matrix from the world coordinate system to the camera coordinate system, the attitude angle converted from the rotation matrix, and the orientation consistency residual. The orientation consistency residual is used to characterize the deviation between the coordinate axis direction of the extracted building in the image and the theoretical direction obtained by projection through the rotation matrix, and serves as the basis for the autonomous on-orbit pointing solution of the camera.

Images