Space target attitude estimation method and device based on multi-type component structure association

Abstract

The invention relates to a space target attitude estimation method and device based on multi-type component structure association, and the method comprises the steps: carrying out the modeling of a space target, obtaining a target structure model, constructing a point-line combined attitude solving function and an optimization target, extracting key points of the space target in a current optical image, and image linear features, then, a two-dimensional projection image is obtained based on attitude iterative optimization and a projection target structure model, projection straight line features are extracted and are matched and associated with image straight line features, the straight line features are endowed with association weights, and a linear and point feature association relationship and an optimization target are utilized to obtain an optimal target attitude; and solving the point-line combined attitude solving function through a convex relaxation optimization algorithm to obtain an intermediate target attitude. And updating the linear feature correlation related to the cylinder part to finish one iteration. And repeating the process until the attitude estimation result of the current space target is obtained. By adopting the method, generalization and practicability can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for estimating the posture of a space target based on the structural association of multiple types of components. Background Art

[0002] As the number of spacecraft in orbit increases, space activities such as on-orbit servicing and debris removal based on satellite platforms are becoming more frequent. These tasks require real-time acquisition of target attitude. For satellite-borne sensors, compared with binocular cameras and lidar, monocular cameras have advantages such as long range and low cost. Attitude estimation based on monocular vision is a current research hotspot.

[0003] In recent years, deep learning technology has developed rapidly and has been widely used in the field of space target attitude estimation. Current research on attitude estimation based on deep learning can be divided into one-stage and two-stage methods. The first stage is an end-to-end method that relies heavily on training samples and is sensitive to complex environments. Currently, the two-stage method is the most studied, that is, the semantic key points of the space target are first extracted through a convolutional neural network, and then the attitude is estimated using the PnP method through the relationship between the key points and their corresponding three-dimensional coordinates. With the emergence of public satellite image data sets such as SPEED and SPEED+, the research on intelligent methods for spacecraft attitude estimation has been promoted. Among them, the attitude estimation method based on key point network extraction and PnP has achieved good results on specific data sets.

[0004] However, most existing methods are customized to train key point extraction networks for specific types of space targets. In practical applications, the number of spacecraft is increasing day by day, and the shapes and structures of different space targets are different. Therefore, it is necessary to propose a posture estimation method that can be applied to targets of different structural types. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for space target posture estimation based on the structural association of multiple types of components, which can effectively improve generalization and practicality in order to address the above technical problems.

[0006] A method for estimating a space target attitude based on multi-type component structure association, the method comprising: Modeling is performed according to the space target to be subjected to real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and a point-line joint attitude solution function and an optimization target are constructed based on the target structure model; Acquire a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; Iterative optimization is performed based on the initial target posture, under the initial target posture, the target structure model is projected on the image domain to obtain a two-dimensional projection image, and projection straight line features in the two-dimensional projection image are extracted; Matching and associating the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assigning a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; According to the line feature association relationship, the point feature association relationship and the optimization target, the point-line joint posture solution function is solved by using a convex relaxation optimization algorithm to obtain an intermediate target posture; According to the intermediate target posture, the straight line feature association related to the cylindrical component in the line feature association relationship is updated to obtain an updated line feature association relationship to complete an iterative optimization; After applying the intermediate target posture and the updated line feature association relationship to the target structure model, a new projected straight line feature is obtained, and a new iterative optimization is performed until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

[0007] In one embodiment, the target structure model is expressed as: ; In the above formula, represents the three-dimensional key points in the body coordinate system, Represents the two endpoints of the straight line, Represents the two points of the center of the upper and lower bases of the cylinder, and the radius is .

[0008] In one embodiment, the point-line joint posture solution function is expressed as: ; In the above formula, A and B are and The matrix of Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

[0009] In one embodiment, when extracting key points and image straight line features of the space target in the current optical image: Use deep neural network to extract key points of spatial targets in the current optical image; A manually designed straight line extraction method is used to extract the image straight line features of the space target in the current optical image.

[0010] In one embodiment, when obtaining the initial target posture based on the key points: In the target structure model, extracting three-dimensional key points that are semantically consistent with the key points, forming a point matching set, and obtaining the point feature association relationship; Based on the point matching set and the point feature association relationship, a convex relaxation optimization algorithm is used to perform posture estimation to obtain the initial target posture.

[0011] In one embodiment, when assigning a weight for measuring the degree of straight line matching to each straight line feature association in the line feature association relationship: A weight is assigned according to the distance between two straight lines in each of the straight line feature associations.

[0012] In one embodiment, while assigning weights to the line feature association relationship, weights are also assigned to the point feature association relationship, and a weight matrix is ​​used to assign weights to both the line feature association relationship and the point feature association relationship, and the weight matrix is ​​a diagonal matrix.

[0013] In one embodiment, the optimization objective is expressed as: ; In the above formula, represents the weight matrix, Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

[0014] The present application also provides a space target posture estimation device based on multi-type component structure association, the device comprising: An offline preparation module is used to model a space target for real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and to construct a point-line joint attitude solution function and an optimization target based on the target structure model; An initial target posture generation module is used to obtain a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; A projection line feature extraction module, used for iterative optimization based on the initial target posture, projecting the target structure model on the image domain to obtain a two-dimensional projection image under the initial target posture, and extracting projection line features in the two-dimensional projection image; A line feature association relationship generation module is used to match and associate the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and to assign a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; An intermediate target posture estimation module is used to solve the point-line joint posture solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship and the optimization target to obtain the intermediate target posture; A line feature association relationship updating module, used for updating the straight line feature association related to the cylindrical component in the line feature association relationship according to the intermediate target posture, to obtain an updated line feature association relationship, so as to complete an iterative optimization; The module for estimating the current posture after iterative optimization is used to apply the intermediate target posture and the updated line feature association relationship to the target structure model to obtain new projected straight line features and perform a new iterative optimization until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

[0015] The above-mentioned method and device for estimating the attitude of a space target based on the association of multi-type component structures, by modeling the space target to obtain a target structure model, that is, a set of point coordinates containing multi-type component structures, and constructing a point-line joint attitude solution function and optimization target, extracting the key points of the space target in the current optical image, the image straight line features and the point feature association relationship, and performing attitude estimation to obtain the initial target attitude, and then based on this attitude iterative optimization, projecting the target structure model to obtain a two-dimensional projection image, extracting the projection straight line features, matching and associating with the image straight line features, assigning straight line feature association weights, using the line and point feature association relationship and the optimization target, solving the point-line joint attitude solution function through a convex relaxation optimization algorithm, and obtaining the intermediate target attitude. Update the straight line feature association related to the cylindrical component to complete one iteration. Repeat this process until the intermediate target attitude meets the preset requirements, that is, the attitude estimation result of the current space target is obtained. The use of this method can effectively improve generalization and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flow chart of a method for estimating a space target posture based on structural association of multiple types of components in one embodiment; Figure 2 Schematic diagram of the overall framework of a method for estimating a space target posture in one embodiment; Figure 3 A schematic diagram of the line structure projection relationship of a point, a straight line and a cylinder in one embodiment; Figure 4 A schematic diagram of a dynamic update of a feature association relationship of a cylindrical structure in one embodiment; Figure 5 A schematic diagram of a test data set in an experiment, where Figure 5 (a) represents the simulated image of target 1, Figure 5 (b) represents the simulated image of target 2; Figure 6 The following is a schematic diagram of the target wireframe model of two targets in an experiment, where: Figure 6 (a) Schematic diagram of the target wireframe model of target 1. Figure 6 (b) Schematic diagram of target wireframe model showing target 2; Figure 7 This is a schematic diagram of the key point annotation of two targets in an experiment, where: Figure 7 (a) and Figure 7 (b) is a schematic diagram of the key point annotation of target 1. Figure 7 (c) and Figure 7 (d) Schematic diagram of key point marking of target 2; Figure 8 This is a schematic diagram of the impact of key point extraction deviation on accuracy using different methods for two targets in an experiment, where: Figure 8 (a) Schematic diagram showing the impact of key point extraction deviation on accuracy of target 1. Figure 8 (b) Schematic diagram showing the impact of key point extraction deviation on accuracy of target 2; Fig. 9 This is a schematic diagram of the qualitative results of posture estimation using this method in an experiment, where: Fig. 9 (a) Schematic diagram showing the results of image key point and line extraction. Fig. 9 (b) shows the corresponding model projection diagram. Fig. 9 (c) is a schematic diagram showing the association result of matching and associating with the extracted straight line. Fig. 9 (d) is a schematic diagram showing the projection of the target model corresponding to the estimated pose result of point-line combination. Fig. 9 (e) is a schematic diagram of the target model projection obtained in the second iteration. Fig. 9 (f) shows the projection diagram of the target model obtained after the fifth iteration. Fig. 9 (g) is a schematic diagram of the fifth iteration wireframe projection. Fig. 9 (h) is a schematic diagram showing the matching association results of the fifth iteration of straight line extraction. Fig. 9 (i) Schematic diagram showing the pose estimation results of the fifth iteration; Fig.10 This is a schematic diagram of the results of an experiment using this method to test an unknown target, where: Fig.10 (a) Fig.10 (b) and Fig.10 (c) represents the three objectives used for training, Fig.10 (d) shows a new target captured in a dark room. Fig.10 (e) represents the extracted features, Fig.10 (f) is a schematic diagram of the projection of the posture estimation model after being processed by the method in this paper; Fig.11 It is a structural block diagram of a space target posture estimation device based on multi-type component structure association in one embodiment; DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In the existing field of spacecraft attitude estimation based on optical images, the most widely used method is to extract the target key points to establish the association between the 2D image and the 3D model, and then use the PnP (Perspective-n-Point) algorithm to obtain the target's three-dimensional attitude. However, in practice, it is difficult to extract enough and stable key points from some target images, such as weak texture images or cylindrical structure target images, which limits the generalizability of the above attitude estimation method. In addition, due to the obvious differences in the structures of different targets, the key point definitions are usually different. Therefore, different targets need to customize their own key point extraction networks, resulting in poor generalization of the method. With the surge in the number of spacecraft, the actual application cost is high.

[0019] Furthermore, there are currently two main problems in extending the attitude estimation method to targets of different structural types. One problem is that for weakly textured space target images, such as some cylindrical spacecraft, the area suitable for defining key points is significantly smaller than that of spacecraft images with rich texture details; at the same time, factors such as illumination changes and degradation often lead to deviations in key point extraction, which significantly affects the accuracy of attitude solution. Another problem is that the structures of different types of spacecraft usually have obvious differences, resulting in different positions and numbers of key points; the current practice is to train key point extraction networks for different targets, resulting in poor generalization of the method and high application costs. Overall, although key points can easily establish semantic associations, there are many application limitations for attitude estimation based solely on key points, and both promotion and generalization need to be improved.

[0020] In response to the above problems, the first idea proposed is to use a small number of semantic key points that are commonly present in spacecraft as a guide and combine them with manually designed straight line features for attitude estimation. On the one hand, semantic key points that are commonly present in spacecraft, such as the corners of solar panels, have the potential to generalize to different targets or even unknown targets; on the other hand, manually designed straight line features can effectively make up for the lack of features to improve the accuracy of target attitude estimation under different structures without the need for additional network training. However, the above ideas still face some urgent problems: (1) Compared with key points, the projection relationship of straight line features corresponding to different types of geometric structures is more complicated to construct. For example, the straight line extracted from the edge of the cylindrical region in the image has a corresponding position coordinate in three-dimensional space coupled with the observation angle, which is difficult to accurately determine when the attitude is unknown; (2) Although manually designed straight line features do not require training, they are prone to mismatching; (3) Point and line features from different target structures have different projection correlation relationships. How to deal with them in a unified solution framework?

[0021] Further, in view of the above problem, in one embodiment, Figure 1 As shown, a method for estimating the posture of a space target based on the structural association of multiple types of components is provided, comprising the following steps: Step S100, modeling is performed according to the space target to be subjected to real-time posture monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and a point-line joint posture solution function and an optimization target are constructed based on the target structure model.

[0022] Step S110, obtaining the current optical image of the space target, extracting the key points and image straight line features of the space target in the current optical image, and obtaining the point feature association relationship, performing posture estimation based on the key points and the point feature association relationship, and obtaining the initial target posture.

[0023] Step S120, performing iterative optimization based on the initial target posture, projecting the target structure model on the image domain under the initial target posture to obtain a two-dimensional projection image, and extracting projection straight line features in the two-dimensional projection image.

[0024] Step S130, matching and associating the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assigning a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching.

[0025] Step S140, using a convex relaxation optimization algorithm to solve the point-line joint posture solution function according to the line feature association relationship, the point feature association relationship and the optimization target, to obtain an intermediate target posture.

[0026] Step S150, updating the straight line feature association related to the cylindrical component in the line feature association relationship according to the intermediate target posture, obtaining an updated line feature association relationship, so as to complete an iterative optimization.

[0027] Step S160, after applying the intermediate target posture and the updated line feature association relationship to the target structure model, a new projected straight line feature is obtained, and a new iterative optimization is performed until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

[0028] In this method, an iterative optimization framework of pose estimation of "common semantic point guidance + traditional line feature matching" is proposed. The pose solution is performed based on the two-dimensional-three-dimensional association relationship of multiple structural types including key points, lines and cylinders, aiming to enhance the generalizability of target types and the generalization of new targets.

[0029] like Figure 2 As shown, in this embodiment, it includes two stages, the first stage is an offline stage, that is, step S100, followed by the second stage of iterative optimization stage, that is, step S110 to step S160, which first constructs the initial target posture based on the key points of the space target, updates the weight of the line feature association relationship based on the initial target posture, iteratively optimizes the target posture, and then optimizes the line feature association relationship in the cylindrical component structure in the space target, so as to achieve the final optimization of the current target posture. Among them, step S120 to step S160 is an iterative optimization process, by continuously repeating several steps, the current posture optimization of the space target is finally achieved, that is, the accurate current posture is obtained, which provides a good foundation for subsequent work.

[0030] In step S100, it is first necessary to construct a corresponding target structure model, i.e., a wireframe model, according to the structure of the space target to be monitored. Here, the space target includes artificial satellites and other space targets with point, line, and cylindrical structures. Considering that the space target posture estimation is essentially to find the imaging coordinate system O C - X C Y C Z C To the body coordinate system O O - X O Y O Z O The rotation matrix and translation vector ,like Figure 3 shown.

[0031] In this embodiment, considering the cylindrical structure in the space target, the target structure model is expressed as: (1) In formula (1), represents the three-dimensional key points in the body coordinate system, Represents the two endpoints of the straight line, Represents the two points of the center of the upper and lower bases of the cylinder, and the radius is .

[0032] Furthermore, a posture optimization framework based on multi-type geometric structure association is constructed based on the target structure model shown in formula (1). Figure 2 ,set up is the corresponding point in the image, then the optical center of the imaging coordinate system and The line passes through .but The normalized coordinates in the camera coordinate system are: (2) In formula (2), is the camera intrinsic parameter matrix. Yes The coordinates in the imaging coordinate system are There are collinear constraints: (3) Furthermore, in order to unify the expression, R is expanded into a one-dimensional vector r, and we can get: (4) In formula (4), A and B are and The matrix of and Jointly decided.

[0033] In fact, what is extracted from the image is not necessarily a complete straight line, that is, the projection point , and There is not necessarily a strict projection relationship as shown in (3); however, the normal and straight line of the surface formed by the projection point and the optical center have a perpendicular constraint relationship. Let and is the normalized coordinate in the camera coordinate system, then these two points and the camera optical center can form the normal of the plane as follows: (5) but and The constraints are: (6) Similarly, formula (6) can be expressed as: (7) For cylindrical structures, the straight lines in the image usually correspond to the edges of the cylindrical regions. and are the normalized coordinates of the two endpoints of the cylinder edge line in the image, and the three-dimensional line corresponding to the line in the image is , The connected lines should satisfy: (8) (9) It should be noted that and The connecting line and the central axis of the cylindrical structure and The lines are parallel and the distance is It is important to emphasize that and The coordinates of are coupled with the target posture and cannot be directly determined when the target posture is unknown. Therefore, an iterative update strategy is designed in this method to deal with this problem, which is described in detail in step S150. Expand R into r and write it in the following form: (10) At this point, the projection relationship between cylinders, lines, and points can be stacked and written into a unified posture estimation form with different structures: (11) Next, taking t as the independent variable, the least squares solution of this linear equation system about t can be expressed as: (12) Substituting formula (12) into formula (11), the point-line joint posture solution function can be expressed as: (13) In formula (13), A and B are and The matrix of Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

[0034] Furthermore, the point-line joint posture solution function shown in formula (13) can be solved using the convex relaxation posture optimization algorithm, and the optimization objective is given at the same time It should be noted that the projection relationship of the cylinder in the formula is related to the target posture to be solved, that is, it is an implicit expression, which is quite different from the existing research.

[0035] In step S110, after the offline preparation work of step S100, an optical image of the current space target is acquired, and key points and straight line features of the space target are first extracted.

[0036] In this embodiment, a deep neural network is used to extract key points of the space target in the current optical image, and a manually designed straight line extraction method is used to extract image straight line features of the space target in the current optical image.

[0037] Specifically, the network for extracting key points can be typical such as HRnet, HourglassNet, MobileNet, etc. At the same time, manually design straight line extraction methods such as EDlines, LSD, Hough transform, etc. to obtain image straight line features.

[0038] In this embodiment, when the initial target posture is obtained based on key points: in the target structure model, three-dimensional key points that are semantically consistent with the key points are extracted to form a point matching set, and point feature association relationships are obtained. Based on the point matching set and the point feature association relationship, a convex relaxation optimization algorithm is used to perform posture estimation to obtain the initial target posture.

[0039] Next, the iterative optimization step is entered. In view of the actual situation of the coupling between the 2D-3D projection relationship of the target structure and the target posture to be solved, as well as the possible mismatching problem of 2D-3D feature association, this method constructs an iterative optimization process for posture estimation, and embeds feature weight update steps and feature association relationship update steps to deal with the above problems.

[0040] In step S120, the target structure model is adjusted according to the initial target posture, and the target structure model under the initial target posture is projected into the image domain to obtain a two-dimensional projection image under the posture. Further, straight lines in the two-dimensional projection image are extracted to obtain projection straight line features.

[0041] In step S130, the projected straight line features and the image straight line features are matched to obtain an initial line feature association relationship, that is, a projected straight line and an image straight line with the same semantic features are found, and the two straight lines are associated to obtain a line feature association relationship.

[0042] In this method, considering that the 2D-3D correspondence of the straight line is determined by the matching metric between the straight line in the projection area of ​​the target model and the straight line detected in the image, the straight line may be mismatched in this process. In step S140, a weight is assigned to each feature association pair in the line feature association relationship, and the matching degree between the two straight lines is measured by the weight.

[0043] In this embodiment, when assigning a weight for measuring the degree of straight line matching to each straight line feature association in the line feature association relationship, the weight is assigned according to the distance between two straight lines in each straight line feature association.

[0044] Furthermore, while assigning weights to the line feature association relationship, the point feature association relationship is also assigned weights, and a weight matrix is ​​used to assign weights to both the line feature association relationship and the point feature association relationship, wherein the weight matrix is ​​a diagonal matrix. It should be noted here that when updating the weights of the feature association relationship, the weights of the line association relationship are actually updated with emphasis, because the point feature association relationship is easy to determine accurately, so it does not need to be adjusted too much.

[0045] Specifically, in the matrix of formula (13) Medium, top 3 n The row is determined by the key point, and the last 2 m The row is determined by the straight line characteristics, and the optimization goal is . Define the weight matrix diagonal matrix as follows: (14) Furthermore, is a size (3 n +2 m )×(3 n +2 m ). is the weight of the key point. It is k The weight of the straight line, the upper right corner is the number of rows in the matrix. Therefore, the optimization target becomes: (15) In formula (15), represents the weight matrix, Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

[0046] Furthermore, the weights are continuously updated during the iteration process, and the weights of feature pairs with incorrect matches are expected to gradually decrease.

[0047] Next, in step 140, based on the line feature association relationship, the point feature association relationship and the optimization goal expressed by formula (15), the point-line joint posture solution function shown in formula (13) is solved using a convex relaxation optimization algorithm to obtain an intermediate target posture, that is, the target posture of the current iterative optimization.

[0048] Considering that the image features and the 2D-3D feature association relationship of the target model are coupled with the unknown posture, and the posture is calculated by the feature association relationship. Therefore, in the iterative process, the feature association relationship is updated according to the posture estimation value in this iteration, and the feature is the line feature association relationship. Because the straight line extracted according to the cylindrical structure is considered in this method, but its straight line association relationship is difficult to correspond accurately, so the line feature association relationship corresponding to the cylindrical structure is updated by the dynamic change of the cylinder. The line feature association relationship is important information for solving the posture value, especially in weak texture images. However, the three-dimensional coordinates of the line features at the edge of the cylindrical structure image area in this system are coupled with the unknown posture value; therefore, calculating the posture according to the feature association relationship containing the cylindrical structure is an implicit problem that needs to be solved by iteration. Specifically, the two steps of solving the posture according to the feature association and updating the association relationship according to the posture are iterative.

[0049] Specifically, Figure 4 As shown in k In the iteration, it is known that R(k) and T (k) , which can be used to determine the imaging coordinate system and .Depend on and The perpendicular relationship between the two planes and the radius of the cylinder can determine and So far, through the three-dimensional straight line The projection and the straight lines extracted from the image The matching relationship can form a 2D-3D association of straight line features.

[0050] In this embodiment, after the line feature association relationship is updated, the posture of the target structure model is adjusted according to the intermediate target posture obtained in the current iteration and the updated line feature association relationship, and projected into the image domain to enter the next optimization iteration process. After multiple iterations, if the difference between the intermediate target posture obtained in the current iteration and the intermediate target posture obtained in the previous iteration is less than a preset threshold, the iteration is stopped, and the current intermediate target posture is the final posture after optimization.

[0051] Specifically, in each iteration process, after obtaining the intermediate target posture, it can be compared with the intermediate target posture obtained in the previous iteration to determine whether the difference is less than the preset threshold. If it is less than, the iteration is stopped; if not, the line feature association relationship continues to be optimized and updated.

[0052] In one embodiment, the algorithm steps for implementing the method for estimating the posture of a space target can be expressed as follows: Step 1: Extract semantic key points of the input image (i.e., the optical image of the current spatial target) and straight line .

[0053] Step 2: Make the wireframe model 3D points Extract key points Semantically consistent points are represented as . Form a matching set , the convex relaxation optimization algorithm is used for initial attitude estimation, and the initial attitude estimation result is obtained .

[0054] Step 3: Next, the set of lines in the target model Back-projection, we get Model projection lines taking into account hidden effects , the straight line detected in the image In the example, according to the distance between the straight lines, matching is performed under the set threshold to form a matching set .

[0055] Step 4: Under this condition, the cylinder can be calculated The 3D straight line corresponding to the edge of the cylinder in the image , its projection and After matching and associating, you can get the matching set .

[0056] Step 5: Combine the three matches The points and lines in the figure are uniformly solved using the convex relaxation method adopted in this paper, and we get Repeat Step 3 and Step 4. k Step Utilization get .

[0057] Step 6: When and If the change of is less than the threshold, the iteration ends.

[0058] Furthermore, in this paper, experiments are conducted to prove the effectiveness of the method proposed in this paper.

[0059] First, in the experiment, computer graphics methods are used to generate two simulated image datasets of typical spacecraft, such as Figure 5 As shown, the size of each image is 256 × 256. In the image simulation, motion blur and illumination direction are determined by simulating the orbital positions of the observation satellite and the target satellite.

[0060] Furthermore, target wireframe models of two typical spacecraft are established, namely, target structure models, such as Figure 6 As shown in the figure. Gray points are nodes, red points are trained points, and blue points are cylinders. The target model consists of points, polyhedrons, and cylinders. Gray points are nodes of the target wireframe model, red points are key points for network training, and red lines are used to match the extracted line features.

[0061] In the experiment, the evaluation criteria used is to express the estimated value as and , the true value is and , pose estimation and position estimation error They are defined as: (16) (17) The quantitative results of target 1 are given below. The proposed method only extracts 4 sailboard corner points as key points, and combines key points and line features to solve in the pose estimation stage. Methods 1 and 3 are key point extraction plus Epnp models, which are widely used by researchers. Method 1 only trains 4 sailboard points, and method 3 extracts all annotated key points. The detailed annotation is shown in Tables 1 and Figure 7 Method 2 also extracts only 4 points, but integrates straight line features in the pose estimation stage, and the pose solution method uses CvxPnPL. For ease of comparison, all methods use the HRnet network.

[0062] Specifically, Figure 7 Indicates the key point annotation of the two targets. Green is the invisible point. Figure 7 (a) and Figure 7 (c) 7 and 13 points are marked respectively. Figure 7 (b) and Figure 7 (d) Four key points are marked, corresponding to the corners of solar panels, which are widely present in different types of spacecraft.

[0063] Table 1

[0064] Furthermore, the pose estimation results of target 1 and target 2 are shown in Table 2 and Table 3 respectively: Table 2. Posture estimation results of target 1

[0065] Table 3. Posture estimation results of target 2

[0066] From the verification results, it can be seen that both this method and method 1 use the four key points extracted by the network. In comparison, the pose estimation accuracy of this method is significantly improved. This is because the line features are introduced in this method, and the manually designed line features extracted in this method do not increase the additional training cost of the network. Although line features are also introduced in method 2, this method designs an iterative optimization process for the potential challenges brought by complex line projection relationships (especially cylinders) and possible line mismatches. Therefore, this method obtains better results than method 2. In method 3, all marked key points are used to estimate the pose, and its accuracy is quite close to that of this method. In contrast, the number of key points required for this method is greatly reduced, which has significant advantages for weakly textured space target images such as cylindrical structure spacecraft.

[0067] Furthermore, we consider the influence of key point extraction errors. Key point extraction errors are inevitable in practice. The following uses the true value of the key point as a benchmark, adds disturbances on it, and compares the tolerance of different methods to key point extraction errors. Figure 8 As shown in Figure 2, it is the influence of key point extraction deviation on accuracy, where: Figure 8 (a) is dataset 1, Figure 8 (b) is dataset 2.

[0068] The results show that the proposed method has a more obvious advantage when there is a deviation in key point extraction. This is because in the pose optimization iterative framework established in this paper, the weights of feature associations are constantly adjusted to reduce the weights of features with lower matching degrees.

[0069] like Fig. 9 As shown, the intuitive process of the iteration of this method is given, where Fig. 9 (a) is the result of image key point and line extraction. Based on the key point extraction result, an initial value of posture estimation can be obtained. The corresponding model projection is shown in Fig. 9 (b) Match and associate with the extracted straight line. The association results are shown in Fig. 9 (c), lines of the same color are straight lines of the same name. The projection of the target model corresponding to the estimated pose result of the point-line joint is shown in 9 (d), which is the result of the first iteration. Fig. 9 (e) is the result of the second iteration, Fig. 9 (f) is the result of the fifth iteration. Fig. 9 (g) and Fig. 9 (h) is the matching association result of wireframe projection and line extraction of the fifth iteration. Fig. 9 (i) is the pose estimation result of the fifth iteration.

[0070] Furthermore, the generalization of zero-sample space targets. By using the proposed method to estimate the attitude of space targets without training samples, three targets are used for training. The four corner points of the target sailboard, a total of 1000 images, such as Fig.10 (a) Fig.10 (b) and Fig.10 (c) shows 100 unknown targets. 100 semi-physical simulation images were taken in an optical darkroom for testing. Fig.10 (d) shown.

[0071] Experimental results show that the average error of zero-sample target pose estimation in the constructed dataset is 3.5°. Based on the fact that sailboard structural features are common in typical spacecraft, this method combines sailboard key points with manual line features for pose estimation, which has good generalization performance.

[0072] In the above-mentioned space target attitude estimation method based on the structural association of multiple types of components, in order to improve the adaptability and generalization ability of the existing monocular spacecraft attitude estimation method for different types of targets, a unified attitude iteration framework based on the association of two-dimensional and three-dimensional features of key points, polyhedrons and cylinders is established by combining key points and manual line features. In the attitude iteration optimization framework, feature association update and feature weight update modules are proposed. Experimental results show that compared with existing methods, this method has better robustness and accuracy in the case of weak texture and key point deviation. In addition, the generalization of this method to zero-sample targets is verified by semi-physical simulation. The research on this method is of great significance for improving the universality and accuracy of spacecraft perception algorithms.

[0073] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0074] In one embodiment, Fig.11 As shown, a space target posture estimation device based on multi-type component structure association is provided, comprising: an offline preparation module 200, an initial target posture generation module 210, a projection line feature extraction module 220, a line feature association relationship generation module 230, an intermediate target posture estimation module 240, a line feature association relationship update module 250 and a current posture estimation module 260 after iterative optimization, wherein: The offline preparation module 200 is used to model the space target to be monitored in real time, obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and construct a point-line joint attitude solution function and an optimization target based on the target structure model; The initial target posture generation module 210 is used to obtain the current optical image of the space target, extract the key points and image straight line features of the space target in the current optical image, obtain the point feature association relationship, perform posture estimation based on the key points and point feature association relationship, and obtain the initial target posture; A projection line feature extraction module 220 is used to perform iterative optimization based on the initial target posture, project the target structure model on the image domain to obtain a two-dimensional projection image under the initial target posture, and extract the projection line features in the two-dimensional projection image; A line feature association relationship generation module 230 is used to match and associate the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and to assign a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; An intermediate target posture estimation module 240 is used to solve the point-line joint posture solution function using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship and the optimization target to obtain the intermediate target posture; A line feature association relationship updating module 250 is used to update the straight line feature association related to the cylindrical component in the line feature association relationship according to the intermediate target posture, to obtain an updated line feature association relationship, so as to complete an iterative optimization; The module 260 for estimating the current posture after iterative optimization is used to apply the intermediate target posture and the updated line feature association relationship to the target structure model to obtain new projected straight line features and perform a new iterative optimization until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

[0075] For the specific definition of the space target attitude estimation device based on the association of multi-type component structures, please refer to the definition of the space target attitude estimation method based on the association of multi-type component structures in the above text, which will not be repeated here. Each module in the above-mentioned space target attitude estimation device based on the association of multi-type component structures can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0076] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for estimating the attitude of a space target based on the structural association of multiple types of components, characterized in that: The method comprises: Modeling is performed according to the space target to be subjected to real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and a point-line joint attitude solution function and an optimization target are constructed based on the target structure model; Acquire a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; Iterative optimization is performed based on the initial target posture, under the initial target posture, the target structure model is projected on the image domain to obtain a two-dimensional projection image, and projection straight line features in the two-dimensional projection image are extracted; Matching and associating the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assigning a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; According to the line feature association relationship, the point feature association relationship and the optimization target, the point-line joint posture solution function is solved by using a convex relaxation optimization algorithm to obtain an intermediate target posture; According to the intermediate target posture, the straight line feature association related to the cylindrical component in the line feature association relationship is updated to obtain an updated line feature association relationship to complete an iterative optimization; After applying the intermediate target posture and the updated line feature association relationship to the target structure model, a new projected straight line feature is obtained, and a new iterative optimization is performed until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

2. The method for estimating a space target posture based on multi-type component structure association according to claim 1, characterized in that: The target structure model is expressed as: In the above formula, represents the three-dimensional key points in the body coordinate system, Represents the two endpoints of the straight line, Represents the two points of the center of the upper and lower bases of the cylinder, and the radius is .

3. The method for estimating a space target posture based on multi-type component structure association according to claim 2, characterized in that: The point-line joint posture solution function is expressed as: In the above formula, A and B are and The matrix of Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

4. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 3, characterized in that: When extracting key points and image straight line features of the space target in the current optical image: Use deep neural network to extract key points of spatial targets in the current optical image; A manually designed straight line extraction method is used to extract the image straight line features of the space target in the current optical image.

5. The method for estimating the posture of a space target based on the association of multi-type component structures according to claim 4, characterized in that: When the initial target posture is obtained based on the key points: In the target structure model, extracting three-dimensional key points that are semantically consistent with the key points, forming a point matching set, and obtaining the point feature association relationship; Based on the point matching set and the point feature association relationship, a convex relaxation optimization algorithm is used to perform posture estimation to obtain the initial target posture.

6. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 5, characterized in that: When assigning a weight for measuring the degree of straight line matching to each straight line feature association in the line feature association relationship: A weight is assigned according to the distance between two straight lines in each of the straight line feature associations.

7. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 6, characterized in that: While assigning weights to the line feature association relationship, weights are also assigned to the point feature association relationship. A weight matrix is ​​used to assign weights to the line feature association relationship and the point feature association relationship at the same time. The weight matrix is ​​a diagonal matrix.

8. The method for estimating the attitude of a space target based on the structural association of multiple types of components according to claim 7, characterized in that: The optimization objective is expressed as: In the above formula, represents the weight matrix, Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

9. A space target attitude estimation device based on multi-type component structure association, characterized in that: The device comprises: An offline preparation module is used to model a space target for real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and to construct a point-line joint attitude solution function and an optimization target based on the target structure model; An initial target posture generation module is used to obtain a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; A projection line feature extraction module, used for iterative optimization based on the initial target posture, projecting the target structure model on the image domain to obtain a two-dimensional projection image under the initial target posture, and extracting projection line features in the two-dimensional projection image; A line feature association relationship generation module is used to match and associate the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and to assign a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; An intermediate target posture estimation module is used to solve the point-line joint posture solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship and the optimization target to obtain the intermediate target posture; A line feature association relationship updating module, used for updating the straight line feature association related to the cylindrical component in the line feature association relationship according to the intermediate target posture, to obtain an updated line feature association relationship, so as to complete an iterative optimization; The module for estimating the current posture after iterative optimization is used to apply the intermediate target posture and the updated line feature association relationship to the target structure model to obtain new projected straight line features and perform a new iterative optimization until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

Images