Spatial target attitude estimation method and device based on multi-type component structure association

Abstract

The present application relates to a method and device for estimating the attitude of a space target based on the structural association of multiple types of components. By modeling the space target, a target structure model is obtained, and a point-line joint attitude solution function and an optimization objective are constructed. Key points and image line features of the space target in the current optical image are extracted, and based on this, the initial target attitude is estimated. Subsequently, based on this attitude, iterative optimization is performed. The target structure model is projected to obtain a two-dimensional projection image, and the projection line features are extracted and matched and associated with the image line features. The line feature association weights are assigned. Using the line-point feature association relationship and the optimization objective, the point-line joint attitude solution function is solved by a convex relaxation optimization algorithm to obtain an intermediate target attitude. The line feature association related to the cylinder component is updated to complete one iteration. This process is repeated until the attitude estimation result of the current space target is obtained. Using this method can effectively improve the generalization and practicality.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a method and device for spatial target attitude estimation based on the association of multiple types of component structures. Background Art

[0002] With the increasing number of on-orbit spacecraft, space activities such as on-orbit servicing and debris removal based on spaceborne platforms are becoming more and more frequent. These tasks all require real-time acquisition of the target attitude. For spaceborne sensors, compared with binocular cameras, lidar, etc., monocular cameras have advantages such as long distance and low cost. Attitude estimation based on monocular vision is a current research hotspot.

[0003] In recent years, deep learning technology has developed rapidly and has been widely applied in the field of spatial target attitude estimation. Current research on attitude estimation based on deep learning can be divided into one-stage and two-stage methods. The one-stage method is an end-to-end method, which has a large dependence on training samples and is more sensitive to complex environments. Currently, more research is on the two-stage method, that is, first extracting the semantic key points of the spatial target through a convolutional neural network, and then using the PnP method for attitude estimation based on the relationship between the key points and their corresponding three-dimensional coordinates. With the emergence of satellite image public datasets such as SPEED and SPEED+, it has promoted the research on intelligent methods for spacecraft attitude estimation. Among them, the attitude estimation method based on key point network extraction and PnP has achieved good results on specific datasets.

[0004] However, most of the existing methods are customized to train the key point extraction network for specific types of spatial targets. In practical applications, the number of spacecraft is increasing day by day, and the shape structures of different spatial targets are different. Therefore, there is a need to propose an attitude estimation method that can be applied to targets of different structural types. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method and device for spatial target attitude estimation based on the association of multiple types of component structures that can effectively improve generalization and practicability.

[0006] A method for spatial target attitude estimation based on the association of multiple types of component structures, the method includes:

[0007] Modeling according to the spatial target to be monitored in real time to obtain a target structure model, the target structure model being a set of point coordinates including multiple types of component structures, and constructing a point-line joint attitude solution function and an optimization target based on the target structure model;

[0008] Obtain the current optical image of the spatial target, extract the key points and image line features of the spatial target in the current optical image, and obtain the point feature association relationship. Based on the key points and the point feature association relationship, perform attitude estimation to obtain the initial target attitude;

[0009] Based on the initial target attitude, perform iterative optimization. At the initial target attitude, project the target structure model onto the image domain to obtain a two-dimensional projection image, and extract the projection line features in the two-dimensional projection image;

[0010] Match and associate according to the image line features and the projection line features to obtain the two-dimensional to three-dimensional line feature association relationship, and assign a weight for measuring the line matching degree to each line feature association in the line feature association relationship;

[0011] According to the line feature association relationship, the point feature association relationship, and the optimization objective, use the convex relaxation optimization algorithm to solve the point-line joint attitude solution function to obtain the intermediate target attitude;

[0012] According to the intermediate target attitude, update the line feature associations related to the cylinder components in the line feature association relationship to obtain the updated line feature association relationship, so as to complete one iteration of optimization;

[0013] After applying the intermediate target attitude and the updated line feature association relationship to the target structure model, obtain new projection line features and perform a new iteration of optimization until the intermediate target attitude obtained in the current iteration meets the preset requirements. Then, the intermediate target attitude obtained in the current iteration is the attitude estimation result of the current spatial target.

[0014] In one embodiment, the target structure model is expressed as:

[0015] ;

[0016] In the above formula, represents the three-dimensional key points in the body coordinate system, represents the two end points of the line, represents the two points of the centers of the upper and lower bottoms of the cylinder, and the radius is .

[0017] In one embodiment, the point-line joint attitude solution function is expressed as:

[0018] ;

[0019] In the above formula, A and B are respectively and matrices, represents Expand it into a one-dimensional vector form, represents a rotation matrix used to describe the target pose, is a matrix of and respectively represent the number of points and lines.

[0020] In one embodiment, when extracting the key points and image line features of the spatial target in the current optical image:

[0021] Use a deep neural network to extract the key points of the spatial target in the current optical image;

[0022] Use a manually designed line extraction method to extract the image line features of the spatial target in the current optical image.

[0023] In one embodiment, when obtaining the initial target pose based on the key points:

[0024] In the target structure model, extract the three-dimensional key points that are semantically consistent with the key points, form a point matching set, and obtain the point feature association relationship;

[0025] Based on the point matching set and the point feature association relationship, use a convex relaxation optimization algorithm for pose estimation to obtain the initial target pose.

[0026] In one embodiment, when assigning weights for measuring the line matching degree to each line feature association in the line feature association relationship:

[0027] Assign weights according to the distance between two lines in each line feature association.

[0028] In one embodiment, while assigning weights to the line feature association relationship, weights are also assigned to the point feature association relationship, and a weight matrix is used to assign weights to both the line feature association relationship and the point feature association relationship, and the weight matrix uses a diagonal matrix.

[0029] In one embodiment, the optimization objective is expressed as:

[0030] ;

[0031] In the above formula, represents the weight matrix, represents expanding it into a one-dimensional vector form, represents a rotation matrix used to describe the target pose, is a matrix of and respectively represent the number of points and lines.

[0032] In this application, a spatial target attitude estimation device based on the structural association of multiple types of components is also provided. The device includes:

[0033] An offline preparation module, configured to model according to a spatial target to be monitored in real time to obtain a target structure model. The target structure model is a set of point coordinates including multiple types of component structures, and based on the target structure model, a point-line joint attitude solution function and an optimization target are constructed;

[0034] An initial target attitude generation module, configured to obtain a current optical image of the spatial target, extract key points and image line features of the spatial target in the current optical image, and obtain a point feature association relationship. Based on the key points and the point feature association relationship, attitude estimation is performed to obtain an initial target attitude;

[0035] A projected line feature extraction module, configured to perform iterative optimization based on the initial target attitude. Under the initial target attitude, project the target structure model onto the image domain to obtain a two-dimensional projected image, and extract the projected line features in the two-dimensional projected image;

[0036] A line feature association relationship generation module, configured to perform matching association according to the image line features and the projected line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assign a weight for measuring the line matching degree to each line feature association in the line feature association relationship;

[0037] An intermediate target attitude estimation module, configured to solve the point-line joint attitude solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship, and the optimization target to obtain an intermediate target attitude;

[0038] A line feature association relationship update module, configured to update the line feature associations related to the cylinder components in the line feature association relationship according to the intermediate target attitude to obtain an updated line feature association relationship, so as to complete one iteration of optimization;

[0039] An estimated current attitude module after iterative optimization, configured to obtain new projected line features after applying the intermediate target attitude and the updated line feature association relationship to the target structure model, and perform a new iteration of optimization until the intermediate target attitude obtained in the current iteration meets the preset requirements. Then, the intermediate target attitude obtained in the current iteration is the attitude estimation result of the current spatial target.

[0040] The above method and device for spatial target attitude estimation based on the association of multi-type component structures model the spatial target to obtain a target structure model, that is, a set of point coordinates containing multi-type component structures, and construct a point-line joint attitude solution function and an optimization objective. Key points, image line features, and point feature association relationships of the spatial target in the current optical image are extracted, and based on this, the initial target attitude is obtained through attitude estimation. Subsequently, based on this attitude, iterative optimization is performed, and the target structure model is projected to obtain a two-dimensional projection image. The projected line features are extracted, matched and associated with the image line features, and weights are assigned to the line feature associations. Using the line and point feature association relationships and the optimization objective, the point-line joint attitude solution function is solved through a convex relaxation optimization algorithm to obtain an intermediate target attitude. The line feature association related to the cylinder component is updated to complete one iteration. This process is repeated until the intermediate target attitude meets the preset requirements, that is, the attitude estimation result of the current spatial target is obtained. Using this method can effectively improve the generalization and practicability. Description of the Drawings

[0041] Figure 1 It is a schematic flowchart of a method for spatial target attitude estimation based on the association of multi-type component structures in an embodiment;

[0042] Figure 2 It is a schematic overall framework diagram of a method for spatial target attitude estimation in an embodiment;

[0043] Figure 3 It is a schematic diagram of the line structure projection relationship of points, lines, and cylinders in an embodiment;

[0044] Figure 4 It is a schematic diagram of the dynamic update of the feature association relationship of the cylinder structure in an embodiment;

[0045] Figure 5 It is a schematic diagram of a test data set in an experiment, where Figure 5 (a) represents the simulation image of target 1, Figure 5 (b) represents the simulation image of target 2;

[0046] Figure 6 It is a schematic diagram of the target wireframe models of two targets in an experiment, where Figure 6 (a) represents the schematic diagram of the target wireframe model of target 1, Figure 6 (b) represents the schematic diagram of the target wireframe model of target 2;

[0047] Figure 7 It is a schematic diagram of the key point annotation situation of two targets in an experiment, where Figure 7 (a) and Figure 7 (b) are the schematic diagrams of the key point annotation situation of target 1, Figure 7 (c) and Figure 7(d) Schematic diagram of key point annotation for Target 2;

[0048] Figure 8 Schematic diagram of the influence of key point extraction deviation on accuracy for two targets in an experiment using different methods. Among them, Figure 8 (a) Schematic diagram of the influence of key point extraction deviation of Target 1 on accuracy, Figure 8 (b) Schematic diagram of the influence of key point extraction deviation of Target 2 on accuracy;

[0049] Figure 9 Schematic diagram of the qualitative results of pose estimation using the proposed method in an experiment. Among them, Figure 9 (a) Schematic diagram of the results of image key point and line extraction, Figure 9 (b) Schematic diagram of the corresponding model projection, Figure 9 (c) Schematic diagram of the association results of matching and associating with the extracted lines, Figure 9 (d) Schematic diagram of the target model projection corresponding to the estimated pose result of point-line combination, Figure 9 (e) Schematic diagram of the target model projection obtained in the second iteration, Figure 9 (f) Schematic diagram of the target model projection obtained in the fifth iteration, Figure 9 (g) Schematic diagram of the wireframe projection in the fifth iteration, Figure 9 (h) Schematic diagram of the matching and associating results of line extraction in the fifth iteration, Figure 9 (i) Schematic diagram of the pose estimation result in the fifth iteration;

[0050] Figure 10 Schematic diagram of the results of testing unknown targets using the proposed method in an experiment. Among them, Figure 10 (a), Figure 10 (b) and Figure 10 (c) represent 3 targets for training, Figure 10 (d) represents a new target collected in a dark room, Figure 10 (e) represents the extracted features, Figure 10 (f) represents the schematic diagram of the pose estimation model projection after processing using the method of this paper;

[0051] Figure 11 Structural block diagram of a spatial target pose estimation device based on multi-type component structure association in an embodiment; Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of this application clearer, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0053] In the existing field of spacecraft attitude estimation based on optical images, the method of establishing the correlation between 2D images and 3D models by extracting target key points and then using the PnP (Perspective-n-Point) algorithm to obtain the target three-dimensional attitude is the most widely used method at present. However, in practice, it is difficult to extract sufficient and stable key points from some target images, such as weak texture images or cylindrical structure target images, which limits the generalization of the above attitude estimation method. In addition, due to the obvious differences in the structures of different targets, the definitions of key points are usually different. Therefore, key point extraction networks need to be customized for different targets, resulting in poor generalization of the method and high actual application costs as the number of spacecraft increases sharply.

[0054] Furthermore, when generalizing the attitude estimation method to targets of different structural types, there are currently two main problems. One problem is that for weak texture space target images, such as some cylindrical main body spacecraft, the areas suitable for defining key points are significantly fewer than those of spacecraft images with rich texture details. At the same time, factors such as illumination changes and degradation often lead to deviations in key point extraction, which have a significant impact on the accuracy of attitude solution. Another problem is that the structures of different types of spacecraft usually have obvious differences, resulting in different positions and numbers of key points. Currently, the practice is to train key point extraction networks for different targets separately, resulting in poor generalization of the method and too high application costs. Generally speaking, although key points can conveniently establish semantic associations, there are many application limitations in attitude estimation relying solely on key points, and both generalization and extensibility need to be improved.

[0055] In response to the above problems, the first proposed idea is to use a small number of semantic key points that are common in spacecraft as a guide and combine manually designed line features for attitude estimation. On the one hand, semantic key points that are common in spacecraft, such as the corners of solar panels, have the potential to generalize to different targets or even unknown targets. On the other hand, manually designed line features can effectively make up for the lack of features to improve the accuracy of target attitude estimation under different structures and do not require additional network training. However, the above idea still faces some problems that need to be solved urgently: (1) Compared with key points, the construction of the projection relationship of line features corresponding to different types of geometric structures is more complex. For example, the straight lines extracted from the edges of the cylindrical region in the image are coupled with the position coordinates in the three-dimensional space and the observation perspective, and it is difficult to accurately determine when the attitude is unknown; (2) Although manually designed line features do not require training, they are prone to false matching; (3) The point and line features from different target structures have different projection correlation relationships, and how to process them under a unified solution framework.

[0056] Furthermore, in response to the above problems, in one embodiment, as Figure 1As shown in the figure, a spatial target attitude estimation method based on the structural association of multiple types of components is provided, including the following steps:

[0057] Step S100: Model the spatial target to be monitored for real-time attitude to obtain a target structure model, which is a set of point coordinates including multiple types of component structures, and construct a point-line joint attitude solution function and an optimization target based on the target structure model.

[0058] Step S110: Obtain the current optical image of the spatial target, extract the key points and image line features of the spatial target in the current optical image, and obtain the point feature association relationship. Based on the key points and the point feature association relationship, perform attitude estimation to obtain the initial target attitude.

[0059] Step S120: Perform iterative optimization based on the initial target attitude. Under the initial target attitude, project the target structure model onto the image domain to obtain a two-dimensional projection image, and extract the projection line features in the two-dimensional projection image.

[0060] Step S130: Perform matching association according to the image line features and the projection line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assign a weight for measuring the line matching degree to each line feature association in the line feature association relationship.

[0061] Step S140: Solve the point-line joint attitude solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship, and the optimization target to obtain an intermediate target attitude.

[0062] Step S150: Update the line feature associations related to the cylinder components in the line feature association relationship according to the intermediate target attitude to obtain an updated line feature association relationship, so as to complete one iteration of optimization.

[0063] Step S160: After applying the intermediate target attitude and the updated line feature association relationship to the target structure model, obtain new projection line features, and perform a new iteration of optimization until the intermediate target attitude obtained in the current iteration meets the preset requirements. Then, the intermediate target attitude obtained in the current iteration is the attitude estimation result of the current spatial target.

[0064] In this method, an attitude estimation iterative optimization framework of "common semantic point guidance + traditional line feature matching" is proposed, and attitude calculation is performed based on the two-dimensional to three-dimensional association relationships of multiple structural types including key points, lines, and cylinders, aiming to enhance the generalization of target types and the generalization ability for new targets.

[0065] Such as Figure 2As shown, in this embodiment, it includes two stages. The first stage is the offline stage, which is step S100. Then the second stage is the iterative optimization stage, which is steps S110 to S160. This step first constructs an initial target pose based on the key points of the spatial target, and based on the initial target pose, updates the weights of the line feature association relationship to iteratively optimize the target pose, and then optimizes the line feature association relationship in the cylindrical component structure of the spatial target, thereby realizing the final optimization of the current target pose. Among them, steps S120 to S160 are an iterative optimization process. By continuously repeating these steps, the current pose of the spatial target is finally optimized, that is, an accurate current pose is obtained, providing a good foundation for subsequent work.

[0066] In step S100, first, a corresponding target structure model, that is, a wireframe model, needs to be constructed according to the structure of the spatial target to be monitored. Here, the spatial target includes artificial satellites and other spatial targets with point, straight line, and cylindrical structures. Considering that the spatial target pose estimation is essentially to find the rotation matrix O C - X C Y C Z C from the imaging coordinate system O O - X O Y O Z O to the body coordinate system and the translation vector , as Figure 3 shown.

[0067] In this embodiment, considering the cylindrical structure in the spatial target, the target structure model is expressed as:

[0068] (1)

[0069] In formula (1), represents the three-dimensional key points in the body coordinate system, represents the two endpoints of the straight line, represents the two center points of the upper and lower bottoms of the cylinder, and the radius is .

[0070] Furthermore, based on the target structure model shown in formula (1), a pose optimization framework based on the association of multiple types of geometric structures is constructed. Referring to Figure 2 , let be the corresponding points in the image, then the optical center of the imaging coordinate system The connection line with passes through . Then The normalized coordinates in the camera coordinate system are:

[0071] (2)

[0072] In formula (2), is the camera internal parameter matrix. is the coordinate of point in the imaging coordinate system, and it has a collinear constraint relationship with :

[0073] (3)

[0074] Furthermore, for unified expression, expand R into a one-dimensional vector r, and we can get:

[0075] (4)

[0076] In formula (4), A and B are respectively and matrices, which are jointly determined by and .

[0077] Actually, the straight line extracted in the image is not necessarily complete, that is, the projection points , and do not necessarily have the strict projection relationship shown in (3); but there is a perpendicular constraint relationship between the normal of the plane formed by the projection point and the optical center and the straight line. Let and be the normalized coordinates in the camera coordinate system, then the normal of the plane formed by these two points and the camera optical center is as follows:

[0078] (5)

[0079] Then The constraint relationship with is:

[0080] (6)

[0081] Similarly, formula (6) can be expressed as:

[0082] (7)

[0083] For the cylindrical structure, the straight line reflected in the image usually corresponds to the edge of the cylindrical area. Let and are the normalized coordinates of the two endpoints of the straight line on the edge of the cylinder in the image. Let the three-dimensional straight line corresponding to the straight line in the image be , The connected line should satisfy:

[0084] (8)

[0085] (9)

[0086] It should be noted that and The connecting line of is parallel to the central axis of the cylinder structure and The connecting line, and the distance is . It should be emphasized that and The coordinates and the target pose are coupled and cannot be directly determined when the target pose is unknown. Therefore, an iterative update strategy is designed in the method of this article to address this problem, which will be described in detail in step S150. Expand R into r and write it in the following form:

[0087] (10)

[0088] So far, the projection relationships of the cylinder, straight line, and point can be stacked and written in a unified pose estimation form with different structures:

[0089] (11)

[0090] Next, taking t as the independent variable, the least squares solution of this linear equation system about t can be expressed as:

[0091] (12)

[0092] Substitute formula (12) into formula (11), and the point-line joint pose solution function can be obtained as:

[0093] (13)

[0094] In formula (13), A and B are the matrices of and respectively, means expanding into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target pose, is The matrix of, where and represent the number of points and lines respectively.

[0095] Furthermore, the point-line joint attitude solution function shown in formula (13) can be solved by using a convex relaxation attitude optimization algorithm, and the optimization objective is also given. It should be particularly noted that the projection relationship of the cylinder in the formula is related to the target attitude to be solved, which is an implicit expression, and this is a relatively large difference from existing research.

[0096] In step S110, after the offline preparation work in step S100, the optical image of the current space target is obtained, and the key points and line features of the space target are extracted first.

[0097] In this embodiment, a deep neural network is used to extract the key points of the space target in the current optical image, and a manually designed line extraction method is used to extract the image line features of the space target in the current optical image.

[0098] Specifically, typical networks for extracting key points can be HRnet, HourglassNet, MobileNet, etc. At the same time, manually designed line extraction methods such as EDlines, LSD, and Hough transform are used to obtain image line features.

[0099] In this embodiment, when obtaining the initial target attitude based on the key points: in the target structure model, three-dimensional key points with the same semantics as the key points are extracted to form a point matching set, and the point feature association relationship is obtained. Based on the point matching set and the point feature association relationship, a convex relaxation optimization algorithm is used for attitude estimation to obtain the initial target attitude.

[0100] Then, enter the iterative optimization step. In view of the actual situation that the 2D-3D projection relationship of the target structure is coupled with the target attitude to be solved, and the problem of possible mismatches in 2D-3D feature association, in this method, an iterative optimization process for attitude estimation is constructed, and a feature weight update step and a feature association relationship update step are embedded to address the above problems.

[0101] In step S120, the target structure model is adjusted according to the initial target attitude, and the target structure model in the initial target attitude is projected into the image domain to obtain a two-dimensional projection image in this attitude. Further, the lines in the two-dimensional projection image are extracted to obtain the projected line features.

[0102] In step S130, the projected line features and the image line features are matched to obtain an initial line feature association relationship, that is, the projected line and the image line with the same semantic features are found, and these two lines are associated to obtain the line feature association relationship.

[0103] In this method, considering that the 2D-3D correspondence of lines is determined by the matching metric between the lines in the projection area of the target model and the lines detected in the image, there may be cases of incorrect line matching during this process. In step S140, a weight is assigned to each feature association pair in the line feature association relationship, and this weight is used to measure the matching degree between two lines.

[0104] In this embodiment, when assigning a weight for measuring the line matching degree to each line feature association in the line feature association relationship, the weight is assigned according to the distance between the two lines in each line feature association.

[0105] Furthermore, while assigning weights to the line feature association relationship, weights are also assigned to the point feature association relationship. A weight matrix is used to assign weights to both the line feature association relationship and the point feature association relationship simultaneously, where the weight matrix adopts a diagonal matrix. It should be noted here that when updating the weights of the feature association relationship, in fact, the focus is on updating the weights of the line association relationship because it is easy to accurately determine the point feature association relationship, so not much adjustment is required for it.

[0106] Specifically, in the matrix of formula (13) the first 3 n rows are determined by key points, and the last 2 m rows are determined by line features. The optimization objective is . Define the weight matrix diagonal matrix as follows:

[0107] (14)

[0108] Furthermore, is a diagonal matrix of size (3 n +2 m )×(3 n +2 m ). Where is the weight of the key point. is the weight of the k th line, and the upper right corner is the row number in the matrix. Thus, after adding weights to the optimization objective, it becomes:

[0109] (15)

[0110] In formula (15), represents the weight matrix, represents expanding into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target pose, is matrix, where and represent the number of points and lines respectively.

[0111] Furthermore, during the iteration process, the weights will be continuously updated, and the weights of the feature pairs with expected incorrect matches will gradually decrease.

[0112] Next, in step 140, according to the line feature association relationship, the point feature association relationship, and the optimization objective represented by formula (15), the point-line joint pose solving function shown in formula (13) is solved using a convex relaxation optimization algorithm to obtain an intermediate target pose, that is, the target pose optimized in the current iteration.

[0113] Considering that since there is a coupling relationship between the image features and the 2D-3D feature association relationship of the target model, and the pose is calculated from the feature association relationship. Therefore, during the iteration process, according to the pose estimation value in this iteration, the feature association relationship is updated, and the feature is the line feature association relationship. Because in this method, straight lines extracted according to the cylinder structure are considered, but their straight line association relationships are difficult to correspond accurately. So in this method, the line feature association relationship corresponding to the cylinder structure is updated by the dynamic change of the cylinder. The line feature association relationship corresponding to the cylinder structure is important information for pose value calculation, especially in weakly textured images. However, the three-dimensional coordinates of the line features at the edge of the image area of the cylinder structure in this system are coupled with the unknown pose value. Therefore, calculating the pose according to the feature association relationship containing the cylinder structure is an implicit problem and needs to be solved iteratively. Specifically, the two steps of solving the pose according to the feature association and updating the association relationship according to the pose are iterated with each other.

[0114] Specifically, as Figure 4 shown, in the k th iteration, given R (k) and T (k) , the and in the imaging coordinate system can be determined therefrom. From the perpendicular relationship between the and two planes and the cylinder radius, the and can be determined. Thus far, through the projection of the three-dimensional straight line and the matching relationship with the straight line extracted from the image, a 2D-3D association of the line features can be formed.

[0115] In this embodiment, after updating the line feature association relationship, the pose of the target structure model is adjusted according to the intermediate target pose obtained in the current iteration and the updated line feature association relationship, and projected into the image domain to enter the next optimization iteration process. After multiple iterations, if the difference between the intermediate target pose obtained in the current iteration and the intermediate target pose obtained in the previous iteration is less than the preset threshold, the iteration is stopped, and the intermediate target pose obtained currently is the optimized final pose.

[0116] Specifically, in each iteration process, after obtaining the intermediate target pose, it can be compared with the intermediate target pose obtained in the previous iteration to determine whether the difference is less than the preset threshold. If it is less, the iteration is stopped; if not, the line feature association relationship is continuously optimized and updated.

[0117] In one embodiment, the algorithm steps for implementing the spatial target pose estimation method can be expressed as:

[0118] Step 1: Extract the semantic key points and straight lines .

[0119] Step 2: Let the three-dimensional points of the wireframe model corresponding to the extracted key points with the same semantics be represented as . Form a matching set , and use the convex relaxation optimization algorithm to perform initial pose estimation to obtain the initial pose estimation result .

[0120] Step 3: Back-project the set of straight lines in the target model under to obtain model projection lines considering the occlusion effect . Among the straight lines detected in the image, match them according to the distance between the straight lines under the set threshold to form a matching set .

[0121] Step 4: Under , the three-dimensional straight line corresponding to the cylinder edge in the image of the cylinder can be calculated. After its projection is matched and associated with , a matching set can be obtained.

[0122] Step 5: Use the convex relaxation method adopted in this article to solve the pose for the points and lines in the three matching sets to obtain Repeat Step3 and Step4, i.e., the k step uses to obtain .

[0123] Step 6: When the change between and is less than the threshold, the iteration ends.

[0124] Furthermore, in this paper, the effectiveness of the proposed method is also verified through experiments.

[0125] First, in the experiment, a simulation image dataset of two typical spacecrafts is generated using computer graphics methods, as Figure 5 shown. The size of each image is 256×256. In the image simulation, the motion blur and the illumination direction are determined by simulating the orbital positions of the observation satellite and the target satellite.

[0126] Furthermore, a target wireframe model of two typical spacecrafts, i.e., the target structure model, is established, as Figure 6 shown. Among them, the gray points are nodes, the red points are the trained points, and the blue color represents cylinders. The target model consists of points, polyhedrons, and cylinders. The gray points are the nodes of the target wireframe model, the red points are the key points trained by the network, and the red lines are used to match the extracted line features.

[0127] In the experiment, the evaluation criteria adopted are that the estimated values are denoted as and , the true values are and , and the attitude estimation and the position estimation error are respectively defined as:

[0128] (16)

[0129] (17)

[0130] The following gives the quantitative results of Target 1. The method in this paper only extracts 4 corner points of the sailboard as key points and solves them by combining key points and line features in the attitude estimation stage. Method 1 and Method 3 are the modes of key point extraction plus Epnp, which are widely adopted by researchers. Method 1 only trains 4 sailboard points, and Method 3 extracts all the labeled key points. The detailed labeling situation is shown in Table 1 and Figure 7 . Method 2 also only extracts 4 points, but fuses the line features in the attitude estimation stage, and the attitude solution method used is CvxPnPL. For the convenience of comparison, all methods use the HRnet network uniformly.

[0131] Specifically, Figure 7Indicates the key point annotation situation of two targets. The green points are invisible points. Figure 7 (a) and Figure 7 (c) respectively annotate 7 points and 13 points. Figure 7 (b) and Figure 7 (d) mark 4 key points, corresponding to the corner points of the solar panel, which are widely present in different types of spacecraft.

[0132] Table 1

[0133]

[0134] Furthermore, the attitude estimation results of Target 1 and Target 2 are respectively represented by Table 2 and Table 3:

[0135] Table 2 Attitude Estimation Results of Target 1

[0136]

[0137] Table 3 Attitude Estimation Results of Target 2

[0138]

[0139] From the verification results, it can be seen that both this method and Method 1 use 4 key points extracted by the network. In comparison, the pose estimation accuracy of this method has been significantly improved. This is because line features are introduced in this method, and the manually designed line features extracted in this method do not increase the additional training cost of the network. Although line features are also introduced in Method 2, this method designs an iterative optimization process for the potential challenges brought by complex line projection relationships (especially for cylinders) and possible line mismatches. Therefore, this method obtains better results than Method 2. In Method 3, all marked key points are used to estimate the attitude, and its accuracy is quite close to that of this method. In comparison, the number of key points required by this method is greatly reduced, which has significant advantages for images of weak-texture space targets such as cylindrical-structured spacecraft.

[0140] Furthermore, consider the influence of key point extraction errors. Key point extraction errors are inevitable in practice. Below, taking the true value of the key points as the benchmark, perturbations are added to it to compare the tolerance of different methods to key point extraction errors. As Figure 8 shown, it is the influence of key point extraction deviation on accuracy, where Figure 8 (a) is Dataset 1, Figure 8 (b) is Dataset 2.

[0141] The results show that the method in this paper has more obvious advantages when there are deviations in key point extraction. This is because in the pose optimization iteration framework established in this paper, the weights of feature associations will be continuously adjusted to make the weights of features with lower matching degrees smaller.

[0142] As Figure 9 shown, the intuitive process of the iteration of the method in this paper is given. Among them, Figure 9 (a) is the result of image key point and line extraction. According to the key point extraction result, an initial value of pose estimation can be obtained, and the corresponding model projection is shown in Figure 9 (b), which is matched and associated with the extracted lines. The association result is shown in Figure 9 (c). The lines of the same color are homologous lines. The target model projection corresponding to the pose estimation result of the combined point and line is shown in 9(d), that is, the result of the first iteration. Figure 9 (e) is the result of the second iteration, Figure 9 (f) is the result of the fifth iteration. Figure 9 (g) and Figure 9 (h) are the matching and association results of the wireframe projection and line extraction of the fifth iteration. Figure 9 (i) is the pose estimation result of the fifth iteration.

[0143] Furthermore, for the generalization of zero-sample space targets. By using the method in this paper for the pose estimation performance of space targets without training samples, 4 corner points of the training target sailboard are used for three targets, with a total of 1000 images, as Figure 10 (a), Figure 10 (b) and Figure 10 (c) shown. There are 100 unknown targets. 100 semi-physical simulation images are taken in an optical darkroom for testing, Figure 10 (d) shown.

[0144] The experimental results show that the average error of the method in this paper for zero-sample target pose estimation in the constructed dataset is 3.5°. Based on the fact that the sailboard structure features are generally present in typical spacecrafts, using this method to combine the key points of the sailboard with manual line features for pose estimation has good generalization.

[0145] In the above-mentioned spatial target attitude estimation method based on the structural association of multiple types of components, in order to improve the adaptability and generalization ability of the existing monocular spacecraft attitude estimation method to different types of targets, a unified attitude iteration framework based on the two-dimensional and three-dimensional feature association of key points, polyhedrons and cylinders is established by combining key points and manual line features. In the attitude iteration optimization framework, a feature association update and a feature weight update module are proposed. Experimental results show that, compared with the existing methods, the proposed method has better robustness and accuracy in the case of weak texture and key point deviation. In addition, the generalization of the proposed method to zero-sample targets is verified by means of semi-physical simulation. The research of the proposed method is of great significance for improving the universality and accuracy of spacecraft perception algorithms.

[0146] It should be understood that although Figure 1 the steps in the flowchart of Figure 1 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover,

[0147] In one embodiment, as Figure 11 shown, a spatial target attitude estimation device based on the structural association of multiple types of components is provided, including: an offline preparation module 200, an initial target attitude generation module 210, a projected line feature extraction module 220, a line feature association relationship generation module 230, an intermediate target attitude estimation module 240, a line feature association relationship update module 250, and a module 260 for obtaining the current attitude estimation after iterative optimization, where:

[0148] The offline preparation module 200 is configured to model according to the spatial target to be monitored in real time to obtain a target structure model, where the target structure model is a set of point coordinates including multiple types of component structures, and construct a point-line joint attitude solution function and an optimization target based on the target structure model;

[0149] The initial target attitude generation module 210 is configured to obtain the current optical image of the spatial target, extract the key points and image line features of the spatial target in the current optical image, and obtain a point feature association relationship, and perform attitude estimation based on the key points and the point feature association relationship to obtain an initial target attitude;

[0150] The projected straight-line feature extraction module 220 is configured to perform iterative optimization based on the initial target pose. At the initial target pose, project the target structure model onto the image domain to obtain a two-dimensional projected image, and extract the projected straight-line features in the two-dimensional projected image;

[0151] The line feature association relationship generation module 230 is configured to perform matching association based on the image straight-line features and the projected straight-line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assign a weight for measuring the straight-line matching degree to each straight-line feature association in the line feature association relationship;

[0152] The intermediate target pose estimation module 240 is configured to solve the point-line joint pose solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship, and the optimization target to obtain an intermediate target pose;

[0153] The line feature association relationship update module 250 is configured to update the straight-line feature associations related to the cylinder component in the line feature association relationship according to the intermediate target pose to obtain an updated line feature association relationship, so as to complete one iteration of optimization;

[0154] The current pose estimation module 260 after iterative optimization is configured to, after applying the intermediate target pose and the updated line feature association relationship to the target structure model, obtain new projected straight-line features and perform a new iteration of optimization until the intermediate target pose obtained in the current iteration meets the preset requirements, and then the intermediate target pose obtained in the current iteration is the pose estimation result of the current spatial target.

[0155] For the specific limitations of the spatial target pose estimation device based on the multi-type component structure association, reference may be made to the limitations of the spatial target pose estimation method based on the multi-type component structure association in the foregoing text, which will not be elaborated here. Each module in the above-mentioned spatial target pose estimation device based on the multi-type component structure association can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0156] The technical features of the above embodiments can be combined arbitrarily. For the sake of brief description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0157] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for estimating the attitude of a space target based on the structural association of multiple types of components, characterized in that: The method comprises: Modeling is performed according to the space target to be subjected to real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and a point-line joint attitude solution function and an optimization target are constructed based on the target structure model; Acquire a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; Iterative optimization is performed based on the initial target posture, under the initial target posture, the target structure model is projected on the image domain to obtain a two-dimensional projection image, and projection straight line features in the two-dimensional projection image are extracted; Matching and associating the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and assigning a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; According to the line feature association relationship, the point feature association relationship and the optimization target, the point-line joint posture solution function is solved by using a convex relaxation optimization algorithm to obtain an intermediate target posture; According to the intermediate target posture, the straight line feature association related to the cylindrical component in the line feature association relationship is updated to obtain an updated line feature association relationship to complete an iterative optimization; After applying the intermediate target posture and the updated line feature association relationship to the target structure model, a new projected straight line feature is obtained, and a new iterative optimization is performed until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

2. The method for estimating a space target posture based on multi-type component structure association according to claim 1, characterized in that: The target structure model is expressed as: In the above formula, represents the three-dimensional key points in the body coordinate system, Represents the two endpoints of the line, Represents the two points of the center of the upper and lower bases of the cylinder, and the radius is .

3. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 2, characterized in that: The point-line joint posture solution function is expressed as: In the above formula, A and B are and The matrix of Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

4. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 3, characterized in that: When extracting key points and image straight line features of the space target in the current optical image: Use deep neural network to extract key points of spatial targets in the current optical image; A manually designed straight line extraction method is used to extract the image straight line features of the space target in the current optical image.

5. The method for estimating the posture of a space target based on the association of multi-type component structures according to claim 4, characterized in that: When the initial target posture is obtained based on the key points: In the target structure model, extracting three-dimensional key points that are semantically consistent with the key points, forming a point matching set, and obtaining the point feature association relationship; Based on the point matching set and the point feature association relationship, a convex relaxation optimization algorithm is used to perform posture estimation to obtain the initial target posture.

6. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 5, characterized in that: When assigning a weight for measuring the degree of straight line matching to each straight line feature association in the line feature association relationship: A weight is assigned according to the distance between two straight lines in each of the straight line feature associations.

7. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 6, characterized in that: While assigning weights to the line feature association relationship, weights are also assigned to the point feature association relationship. A weight matrix is ​​used to assign weights to the line feature association relationship and the point feature association relationship at the same time. The weight matrix is ​​a diagonal matrix.

8. The method for estimating the attitude of a space target based on the association of multi-type component structures according to claim 7, characterized in that: The optimization objective is expressed as: In the above formula, represents the weight matrix, Indicates that Expanded into a one-dimensional vector form, represents the rotation matrix, which is used to describe the target posture. yes The matrix of and Represents the number of points and lines respectively.

9. A space target attitude estimation device based on multi-type component structure association, characterized in that: The device comprises: An offline preparation module is used to model a space target for real-time attitude monitoring to obtain a target structure model, wherein the target structure model is a point coordinate set including multiple types of component structures, and to construct a point-line joint attitude solution function and an optimization target based on the target structure model; An initial target posture generation module is used to obtain a current optical image of the space target, extract key points and image straight line features of the space target in the current optical image, obtain point feature association relationships, perform posture estimation based on the key points and point feature association relationships, and obtain an initial target posture; A projection line feature extraction module, used for iterative optimization based on the initial target posture, projecting the target structure model on the image domain to obtain a two-dimensional projection image under the initial target posture, and extracting projection line features in the two-dimensional projection image; A line feature association relationship generation module is used to match and associate the image straight line features and the projected straight line features to obtain a two-dimensional to three-dimensional line feature association relationship, and to assign a weight to each straight line feature association in the line feature association relationship for measuring the degree of straight line matching; An intermediate target posture estimation module is used to solve the point-line joint posture solution function by using a convex relaxation optimization algorithm according to the line feature association relationship, the point feature association relationship and the optimization target to obtain the intermediate target posture; A line feature association relationship updating module, used for updating the straight line feature association related to the cylindrical component in the line feature association relationship according to the intermediate target posture, to obtain an updated line feature association relationship, so as to complete an iterative optimization; The module for estimating the current posture after iterative optimization is used to apply the intermediate target posture and the updated line feature association relationship to the target structure model to obtain new projected straight line features and perform a new iterative optimization until the intermediate target posture obtained in the current iteration meets the preset requirements. The intermediate target posture obtained in the current iteration is the posture estimation result of the current space target.

Images