Precise estimation method for position and attitude of space target

Abstract

The invention provides an accurate estimation method for the position and attitude of a space target. Comprising the steps of image data acquisition, visual target detection, spatial constraint inspection, position and attitude calculation and the like, and precise pose estimation is carried out cooperatively. According to the method, the position constraint features are constructed and the constraint check model is established by using a machine learning method, so that the real-time target detection result can be quickly and accurately checked and judged, wrong detection targets are eliminated, and missing detection targets are complemented; the detection module and the pose solving module are combined, and the position of an expected target is limited and constrained by using a constraint check result; according to different application scenes and relative distances, the pose measurement mode switching state machine is used to flexibly select different pose measurement algorithms, and high-precision and low-delay pose measurement of the target can be realized. The precise estimation method for the position and the attitude of the space target is high in precision, high in real-time performance and high in robustness.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, target detection, and relative positioning technology, and specifically relates to a method for accurately estimating the position and attitude of a spatial target. Background Technology

[0002] In visual pose measurement, one of the core tasks is target detection and recognition. This involves capturing targets in images using a drone camera, then classifying and locating these targets. Traditional target detection algorithms primarily use a sliding window approach to generate candidate regions. Because the number, size, and position of targets in an image vary, rectangular windows of different sizes and aspect ratios are slid across the image, with all sliding results used as candidate regions. Feature extraction and classification are then performed on each candidate region. Given so many candidate regions, the designed features and the classifier used cannot be too complex, otherwise the detection algorithm will be too time-consuming. Operators such as SIFT (Scale Invariant Feature Transform), SURF (Accelerated Robust Features), FAST (Accelerated Segmentation Test Features), and ORB (Feature Extraction Combining FAST Detection and BRIEF Description) are widely used to extract feature points from images, which are then combined with a classifier for classification and recognition. However, using feature points to represent objects is difficult to adapt to object deformation and changes in viewpoint.

[0003] With the continuous development of deep learning, deep learning-based object detection technology has gradually surpassed traditional object detection techniques. Deep learning-based object detectors are mainly divided into two categories: single-stage object detectors and two-stage object detectors. Two-stage object detectors focus on finding the location of the target object in the first stage, obtaining bounding boxes to ensure sufficient accuracy and recall. The second stage focuses on classifying the bounding boxes to find more precise locations. Single-stage object detectors do not require a bounding box stage; they directly generate the object's class probability and location coordinates, obtaining the final detection result in a single detection. Typical single-stage object detectors include SSD (Single-Step Multi-Box Detection) and the YOLO (You Only Look Once) series. Compared to two-stage object detectors, single-stage object detectors have a faster detection speed, but their detection accuracy is somewhat reduced. The emergence of deep learning-based single-stage object detection algorithms, such as YOLO and SSD, has significantly accelerated the detection speed while achieving accuracy comparable to other deep learning algorithms.

[0004] Lightweight object detection refers to methods for object detection in the field of computer vision that utilize fewer computational resources. This is particularly important for unmanned aerial vehicle (UAV) systems, which typically have limited computing power and storage space, while also needing to maintain a small size and weight. However, lightweighting algorithms inevitably leads to a decrease in accuracy and real-time performance. Therefore, providing a systematic approach to improve the detection accuracy of lightweight object detection presents a significant challenge.

[0005] After obtaining the image features of the target, visual pose estimation technology solves the mapping problem between the two-dimensional position of the image features and the three-dimensional position in space. Visual relative positioning technology has advantages such as high accuracy at close range, high frequency of pose generation, simple structure, and low cost. Many researchers have already conducted research on visual pose measurement schemes for UAV aerial docking. However, monocular vision suffers from problems such as inability to acquire depth information, poor anti-interference capability, and poor versatility, while stereo vision has advantages such as high accuracy and strong system anti-interference capability. Therefore, under the premise of meeting accuracy requirements, the stereo vision scheme with stronger anti-interference capability is the best choice. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a method for accurately estimating the position and attitude of spatial targets. It includes steps such as image data acquisition, visual target detection, spatial constraint checking, and position and attitude calculation, all working together to achieve accurate pose estimation. This invention constructs position constraint features and uses machine learning methods to establish a constraint checking model, enabling rapid and accurate checking and judgment of real-time target detection results, eliminating erroneous targets and supplementing missing targets. By combining the detection module with the pose calculation module, the position of the desired target is restricted and constrained using the constraint checking results. A pose measurement mode switching state machine is used to flexibly select between different pose measurement algorithms based on different application scenarios and relative distances, achieving high-precision, low-latency pose measurement of targets. This invention is a high-precision, high-real-time, and highly robust method for accurately estimating the position and attitude of spatial targets.

[0007] A method for accurately estimating the position and attitude of a space target, characterized by the following steps:

[0008] Step 1, Image Data Acquisition: Use a visual sensor to image the target and obtain real-time image data;

[0009] Step 2, Visual target detection: The real-time image data acquired in Step 1 is processed using a target detection model to output the real-time detection results of the target, including the position and size of the target detection box, the target category, and the confidence score.

[0010] Step 3, Spatial Constraint Check: The visual relationship check model is used to judge and reason about the real-time target detection results obtained in Step 2. This includes screening the visual target detection results by using the spatial constraints between various targets in the same frame image to remove false and duplicate targets, and predicting missed targets based on prior information, including the target's category, location, and size.

[0011] Step 4, Position and Pose Calculation: Feature extraction is performed on the image of the target region. The extracted features are matched with the pre-constructed key target feature template, and then the pose calculation method is used to estimate the target position and pose.

[0012] Specifically, the object detection model described in step 2 uses a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a Transformer-based object detection network.

[0013] Specifically, the visual relationship inspection model described in step 3 is constructed according to the following process:

[0014] Step 1), Feature Engineering Construction: Various visual relationship features, including target spatial relationship features, target scale features, target category features, and target color features, are used as feature engineering to describe the model; among them, target spatial relationship features include Euclidean distance, Mahalanobis distance, Chebyshev distance, absolute distance, and normalized distance; target scale features include the target's width ratio, height ratio, and area ratio;

[0015] Step 2), Dataset construction: Collect q images containing p targets of interest to form an image dataset. Randomly select k targets from the p targets, k∈(2,p), and extract the scale, category, color features of the k targets and the spatial relationship features between the k targets to form positive samples. Randomly generate feature values ​​based on the number of features of the positive samples to form negative samples.

[0016] Step 3), Model Training: Construct a model pool consisting of logistic regression, random forest, support vector machine, naive Bayes and decision tree algorithms. Use the dataset constructed in Step 2) to train the different algorithms in the model pool several times. Select the model with the best performance in terms of accuracy, recall and computation speed as the visual relationship inspection model.

[0017] The model uses features from feature engineering to describe and judge each target in the target set, and obtains the probability that the target set meets the constraint check model. If the probability is greater than the set threshold, the set is determined to be a fixed combination of targets that meet the requirements; otherwise, it is determined that there are targets in the set that do not meet the conditions.

[0018] Specifically, the pose calculation methods described in step 4 include satellite-based DGPS relative positioning, lidar relative positioning, and deep learning-based visual pose measurement.

[0019] Specifically, step 4 is implemented using a vision-based triangulation method, and the specific process is as follows:

[0020] First, the grayscale image data of the left and right eyes containing N target scenes acquired by the vision sensor are preprocessed, including grayscale conversion and Gaussian filtering. Then, the feature point extraction algorithm is used to detect feature points in the preprocessed left and right eye images respectively. After obtaining key points and corresponding descriptors, they are numbered in order and saved in the form of a matrix.

[0021] Then, using the target detection results after the constraint check in step 3, the key target area is outlined in the image. The feature point matching algorithm is used to match the feature points detected in the left and right eyes within this range. The RANSAC iterative algorithm is used to filter and eliminate mismatched point pairs. The best matching point pair is taken as the key target point.

[0022] Next, based on the principle of binocular vision imaging, the coordinates of the target point on the image planes of the two cameras are obtained. The three-dimensional spatial coordinates of the target point are then obtained by solving the 2D-3D problem using the parallax method. The calculation formula is as follows:

[0023]

[0024] Where P(X,Y,Z) represents the three-dimensional coordinates of the target point in the left eye camera coordinate system, P l (u l ,v l ) and P r (u r ,v r ) represent the coordinates of the target point in the left and right image coordinate systems, respectively; b is the distance between the optical centers of the two cameras; and f is the camera focal length.

[0025] Finally, using the physical positional relationship between the camera and other targets, a coordinate system transformation method is employed to obtain the relative positional relationship between the camera and other targets.

[0026] Specifically, step 4 is implemented using the Aruco QR code detection algorithm, and the specific process is as follows:

[0027] Step (1), Initial image processing: For the first frame of input image, crop it into 4 images of size w. i ×h i The image, where i is the image number and w i h represents the width of the i-th image. iLet represent the height of the i-th image; then, perform QR code detection and recognition on each image. If the detection fails, perform the same operation on the next frame image until a complete QR code is detected. Record the position and size of the QR code, including its upper left corner coordinates (u,v), width w, and height h.

[0028] Step (2), EKF prediction: Receive the position and size information of several frames of QR codes recorded as input state vectors, and then use the EKF algorithm to construct a motion model. Use the motion model to predict the position (u,v) and size (w,h) of the QR code at the next moment. At the next moment, select the region with the upper left corner coordinates (u-3w,v-3h), width 7w and height 7h as the region of interest based on the predicted value.

[0029] Step (3), QR code detection and recognition: After preprocessing the region of interest obtained by EKF prediction, contour detection and screening are performed to obtain a closed contour. Polygon approximation and corner point extraction are performed on the contour to find the convex quadrilateral and four corner points. In the quadrilateral area, the QR code is first transformed by perspective to obtain the image under the frontal view. Then, the image encoding extraction and decoding are performed. The ID obtained by decoding is compared with the dictionary, the encoding is verified and the corner points are sorted. Finally, the position and size information of the QR code are obtained according to the corner point coordinates, and the EKF prediction value is optimally corrected by using the measurement values ​​of the position and size of the QR code in consecutive frames.

[0030] Step (4), pose calculation: First, based on the position and size information of the QR code, obtain the coordinates of its four corner points in the world frame. Based on the pixel coordinates and camera intrinsic parameters, obtain the coordinates of the four corner points in the normalized camera frame. Then, use the PnP algorithm to calculate the transformation matrix between the world frame and the normalized camera frame. Finally, use the transformation matrix to calculate the pose of the world frame with the center point of the QR code as the origin in the camera frame.

[0031] Step (5), coordinate system transformation: using the physical positional relationship between the QR code and other targets, the relative positional relationship between the camera and other targets is obtained through coordinate system transformation.

[0032] Specifically, the process of screening visual target detection results using spatial constraints between various targets in the same frame image described in step 3 is as follows:

[0033] Step ①: For the target set S = {T1, T2, ..., Tn} output by the target detection model, each target Ti contains its category Ci, position Pi, and confidence Conf_i information, 1 ≤ i ≤ n, 1 ≤ Ci ≤ m, where m is the total number of categories. Let the maximum number of targets of each category in a fixed combination be p[e], 1 ≤ e ≤ m, and the minimum number of targets expected to be output be k.

[0034] Step ②, Preprocessing and Initialization: Group S into G[1], G[2],..., G[m] by category. The number of targets in the e-th group is |G[e]|, and sort the targets in each group in descending order of confidence. Initialize the set S_opt as an empty set, max_size = 0, and proceed to Step ③;

[0035] Step ③, Construct Initial Subset: First, let the subset S_cur be an empty set. Then, for each e from 1 to m, if G[e] is not empty, take the target with the highest confidence in G[e] and add it to the subset S_cur until proceed to Step ④;

[0036] Step ④, Check Initial Subset: If CheckModel(S_cur) is true, update the set S_opt according to S_opt = S_cur, update max_size according to max_size = |S_cur|, and proceed to Step ⑤; otherwise, remove the target with the highest confidence in the first group G[1] and return to Step ③. If all targets in G[1] have been removed, it means that all targets of this category are misdetected. Then, let G[2] be the starting group for traversal, take the target with the highest confidence from the groups that have not been selected yet and add it to the subset S_cur, and then perform the check; where CheckModel(S_cur) represents using the visual relationship check model to judge the set S_cur. If it is greater than the set threshold, it is considered that CheckModel(S_cur) is true, indicating that the current set S_cur is a fixed combination of targets that meet the requirements;

[0037] Step ⑤, Expand Subset: Let e range from 1 to m, and repeat the following operations. The finally obtained set S_opt is the target detection result after screening:

[0038] If the number of targets corresponding to the category Ce in the current S_cur < min{p[e], |G[e]|}, traverse the remaining targets Hx in G[e] in descending order of confidence, 1 ≤ x ≤ ||p[e]| - |G[e]||. Let the set S_temp = S_cur ∪ {Hx}. If CheckModel(S_temp) is true, update the set S_opt and S_cur according to S_opt = S_cur = S_temp, and update max_size according to l = max_size = |S_cur|; otherwise, continue to try the next target in this category; where the remaining targets refer to the targets in G[e] except those added to the subset S_cur;

[0039] Perform the same operations on all category targets, and finally complete the screening of all targets.

[0040] Specifically, the process of predicting the missed detection target based on prior information in step 3 is as follows:

[0041] Step a, Missing target location estimation: Suppose the screened target set S_opt contains l targets Z, and the total number of targets in the fixed combination is... If Zl≠0, it indicates that there are missed detections in the current set, and the number of missing targets is Zl. Select a type of missing target, use the spatial relationship features between the missing target and the detected targets to estimate the set C of all possible center point coordinates of the target. Combine each center point in C with the set S_opt and input it into the visual relationship inspection model to obtain the probability values ​​of different combinations. Select the center point corresponding to the maximum probability value as the center point of the missing target.

[0042] Step b, Optimization of missing target position: Move the estimated missing target center point in its vertical, horizontal and two diagonal directions by step size s. Combine the moved point with S_opt and input it into the visual relationship inspection model. Calculate the combination probability value. Select the center point movement direction corresponding to the maximum probability value as the optimal movement direction. At the same time, update the center point position corresponding to the maximum probability value. Continue to repeat the above process r times from this point to obtain the final optimized center point position.

[0043] Step c, missing target size completion: estimate the width and height of the missing target using the statistical values ​​of the target bounding box scale features, and combine them with the center point position obtained in step b to complete the target bounding box for this type;

[0044] Step d, setting the confidence level of missing targets: Based on the optimal combination probability value in step b, set the confidence level of the detection box for missing targets;

[0045] Step e, Iterative prediction: Let l = l + 1, repeat the above steps until...

[0046] Specifically, the target feature point extraction algorithms include the SIFT algorithm, ORB algorithm, SuperPoint algorithm, and Shi-Tomasi detection algorithm; the feature point matching algorithms include the KNN matching algorithm, FLANN matching algorithm, and brute-force matching BF algorithm.

[0047] Specifically, the vision-based triangulation method and the Aruco-based QR code detection algorithm can be flexibly selected according to different application scenarios and relative distances. The selection is implemented using a pose measurement mode switching state machine. This state machine includes five basic states: initial state S_INIT, normal state for binocular pose solving S1, abnormal state for binocular pose solving S2, normal state for QR code pose solving S3, and abnormal state for QR code pose solving S4. Its workflow is as follows:

[0048] Step a), Initial state S_INIT: In this state, all counters are reset, and it is determined whether the binocular pose solving mode and the QR code pose solving mode are working normally. If neither of them is working, the current state is maintained. If only the binocular pose solving mode is working normally, the process switches to S2. If only the QR code pose solving mode is working normally, the process switches to S4. If both are working normally, the average depth measurement value Lm of the two modes and the distance threshold Ld for switching from binocular pose solving to QR code pose solving are taken. If Lm > Ld, the process switches to S2; otherwise, the process switches to S4. The binocular pose solving mode refers to the mode using a vision-based triangulation method, and the QR code pose solving mode refers to the mode using an Aruco QR code detection algorithm.

[0049] Step b), normal state S1 for binocular pose solving: In this state, first reset the binocular pose abnormality counter Err_T1 = 0. If the depth measurement value P1 is normal, maintain the current state. If P1 ≤ Ld, switch to S3; if P1 is abnormal, switch to S2.

[0050] Step c), Binocular pose solving abnormal state S2: In this state, first increment the binocular pose abnormal counter Err_T1 by 1. If the depth measurement value P1 is abnormal, maintain the current state and reset the binocular pose normal counter Nor_T1 = 0; if the abnormal counter Err_T1 exceeds the threshold M1, switch to S_INIT; if the depth measurement value P1 is normal, increment the normal counter Nor_T1 by 1 and maintain the current state; when the normal counter Nor_T1 exceeds the threshold N1, switch to S1.

[0051] Step d), QR code pose solving normal state S3: In this state, first reset the QR code pose abnormal counter Err_T2 = 0. If the QR code depth measurement value P2 is normal, maintain the current state. If P2 > Lq, switch to S1; if P2 is abnormal, switch to S4.

[0052] Step e), QR code pose solving abnormal state S4: In this state, first increment the QR code pose abnormal counter Err_T2 by 1. If the QR code depth measurement value P2 is abnormal, maintain the current state and reset the QR code pose normal counter Nor_T2 = 0; if the abnormal counter Err_T2 exceeds the threshold M2, switch to S_INIT; if the QR code depth measurement value P2 is normal, increment the normal counter Nor_T2 by 1 and maintain the current state; when the normal counter Nor_T2 exceeds the threshold N2, switch to S3.

[0053] The beneficial effects of this invention are as follows: By constructing position constraint features, the positional relationships between targets can be well characterized, exhibiting strong regularity and high sensitivity. Furthermore, it can be flexibly selected according to different scenarios and is easily expandable. By using machine learning methods to establish a constraint checking model, the real-time detection results obtained from target detection can be quickly and accurately checked and judged, and errors can be eliminated, missing data filled in, and coordinates refined. By combining the detection module with the pose solving module, and utilizing the constraint checking results to restrict and constrain the position of the desired target, high-precision, low-latency pose measurement of the target can be achieved. By using a pose measurement mode switching state machine to flexibly select and switch between different pose measurement algorithms, high-precision, low-latency pose measurement of the target can be achieved. Using the method of this invention, target detection models can be deployed on edge devices. The datasets are rich, the models are lightweight, and the detection results are good, enabling a high-precision, high-real-time, and highly robust pose measurement system, providing a novel technical solution for tasks such as UAV aerial refueling and docking. Attached Figure Description

[0054] Figure 1 This is a flowchart of a method for accurately estimating the position and attitude of a space target according to the present invention;

[0055] Figure 2 This is a flowchart of the visual relationship inspection model construction process;

[0056] Figure 3 This is a flowchart of the process for screening test results using spatial constraints;

[0057] Figure 4 This is a flowchart for predicting missed detection targets;

[0058] Figure 5 This is a flowchart of position and attitude calculation;

[0059] Figure 6 This is a schematic diagram of the state machine workflow for switching pose measurement modes. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0061] This invention provides a method for accurately estimating the position and attitude of a space target, such as... Figure 1 As shown, the specific implementation process is as follows:

[0062] 1. Image Data Acquisition

[0063] A visual sensor is used to image the target, obtaining real-time image data.

[0064] 2. Visual target detection

[0065] The real-time image data acquired in step 1 is processed using an object detection model to output real-time detection results of the target, including the position and size of the target detection box, the target category, and the confidence score.

[0066] Object detection models use convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or Transformer-based object detection networks (DETRs).

[0067] 3. Spatial constraint check

[0068] The visual relationship inspection model is used to judge and infer the real-time target detection results obtained in step 2. This includes screening the visual target detection results by using the spatial constraints between various targets in the same frame image, eliminating false detections and duplicate detections, and predicting missed targets based on prior information, including the target's category, location, and size.

[0069] Among them, such as Figure 2 As shown, the visual relationship inspection model is constructed according to the following process:

[0070] (1) Feature Engineering Construction

[0071] Feature engineering incorporates various visual relationship features, including target spatial relationship features, target scale features, target category features, and target color features, as features used to describe the model. Figure 2 In this context, Mi represents the number of predicted bounding boxes output by the object detection algorithm, where i = 1, 2, ..., n.

[0072] Among them, the target spatial relationship features can include spatial target distances under various definitions, such as Euclidean distance, Mahalanobis distance, Chebyshev distance, etc., or the absolute distance between spatial targets, normalized distance, etc. can be selected; the target scale features can be the width ratio, height ratio, area ratio, etc. of the target. In practical applications, stable features can be selected according to the task scenario.

[0073] (2) Dataset Construction

[0074] Collect q images containing p targets of interest to form an image dataset. Randomly select k targets from the p targets, k∈(2,p), and extract the scale, category, color features of the k targets and the spatial relationship features between the k targets to form positive samples. Based on the number of features of the positive samples, randomly generate feature values ​​to form negative samples.

[0075] (3) Model Training

[0076] A model pool consisting of logistic regression, random forest, support vector machine, Naive Bayes, and decision tree algorithms is constructed. The dataset constructed in step (2) is used to train the different algorithms in the model pool several times. The model with the best performance across metrics including accuracy, recall, and computational speed is selected as the visual relationship checking model. This model uses features from feature engineering to describe and judge each target in the target set. If the target set meets the constraint checking model, it is determined that the set is a fixed combination of targets that meet the requirements; otherwise, it is determined that there are targets in the set that do not meet the conditions.

[0077] Visual relation checking models can be used to determine the reliability of visual target detection results and to eliminate false positives and duplicate detections. This involves using spatial constraints between various targets within the same image frame to screen visual target detection results. Figure 3 As shown, the specific process is as follows:

[0078] (1) For the target set S = {T1, T2, ..., Tn} output by the target detection model, each target Ti contains its category Ci, position Pi, and confidence Conf_i information, 1 ≤ i ≤ n, 1 ≤ Ci ≤ m, where m is the total number of categories. Let the maximum number of targets of each category in a fixed combination be p[e], 1 ≤ e ≤ m, and the minimum number of targets expected to be output be k.

[0079] (2) Preprocessing and initialization: Group S into G[1], G[2], ..., G[m] according to categories. The number of targets in the e-th group is |G[e]|. Sort the targets in each group according to confidence from high to low. Initialize the set S_opt to be an empty set, max_size = 0, and proceed to step (3).

[0080] (3) Constructing the initial subset: First, let the subset S_cur be an empty set. Then, for each e from 1 to m, if G[e] is not empty, take the target with the highest confidence in G[e] and add it to the subset S_cur, until... Proceed to step (4);

[0081] (4) Step ④, check the initial subset: If CheckModel(S_cur) is true, update the set S_opt according to S_opt = S_cur, update max_size according to max_size = |S_cur|, and proceed to step (5); otherwise, return to step (3), traverse G[e] from the highest confidence to the lowest, replace the target with the highest confidence and then continue to check the initial subset; where CheckModel(S_cur) means using the visual relationship checking model to judge the set S_cur. If it is greater than the set threshold, it is considered that CheckModel(S_cur) is true, indicating that the current set S_cur is a fixed combination of targets that meet the requirements, but not necessarily the largest combination.

[0082] (5) Expand the subset: Let e range from 1 to m, and repeat the following operations. The finally obtained set S_opt is the object detection result after screening:

[0083] If the number of objects corresponding to category e in the current S_cur < min{p[e], |G[e]|}, traverse the remaining object set H in G[e] from the highest confidence to the lowest, that is, the objects Hx in G[e] except the objects added to the S_cur subset, 1 ≤ x ≤ ||p[e]| - |G[e]||. Let the set S_temp = S_cur ∪ {Hx}. If CheckModel(S_temp) is true, update the set S_opt and S_cur according to S_opt = S_cur = S_temp, and update max_size according to l = max_size = |S_cur|; otherwise, continue to try the next object in this category until the remaining object set H of the current category is traversed. Then perform the same operations on the next category of objects, and finally complete the expansion of all objects.

[0084] The visual relationship checking model can also infer the category and location of the missed key objects, and complete the missed key objects, that is, predict the missed objects according to the prior information, as Figure 4 shown. The specific implementation process is as follows:

[0085] (1) Completing the object location: Suppose the set S_opt after screening contains l objects, and the total number of objects in the fixed combination is If it indicates that there may be missed detections in the current set, and the number of missing objects is First, a certain type of missing target is selected. Using the spatial relationship features between the missing target and the detected target, the coordinates of all possible center points of the target are estimated. Each center point is combined with the set S_opt and input into the visual relationship inspection model to obtain the probability values ​​of different combinations. The center point corresponding to the highest probability value is selected as the center point of the missing target.

[0086] (2) Optimization of missing target position: The estimated missing target center point is moved in eight directions around it, namely up and down, left and right and two diagonal directions, with a step size s. The moved point is combined with S_opt and input into the visual relationship inspection model. The combined probability value is calculated respectively. The center point movement direction corresponding to the maximum probability value is selected as the optimal movement direction. At the same time, the center point position corresponding to the maximum probability value is updated. The above process is repeated from this point. The process is iterated for a total of r times to obtain the final optimized center point position estimate.

[0087] (3) Missing target size completion: The width and height of the missing target are estimated by using the statistical value of the target box scale feature. Combined with the optimized center point position obtained in step (2), the target box of this type can be successfully completed.

[0088] (4) Setting the confidence level of key targets: Based on the optimal combination probability value in step b, set the confidence level of the detection box for missing targets. The confidence level of missing targets should be less than the confidence threshold for target detection, so as to facilitate the distinction between the completed targets and the actual detected targets.

[0089] (5) Iterative prediction: Let l = l + 1, repeat the above steps until...

[0090] 4. Position and attitude calculation

[0091] Feature extraction is performed on the image of the target region. The extracted features are matched with a pre-constructed key target feature template. Then, a pose estimation method is used to estimate the target's position and orientation. The pose estimation method includes satellite-based DGPS relative positioning, lidar relative positioning, and deep learning-based visual pose measurement.

[0092] like Figure 5 As shown, step 4 can be implemented using a vision-based triangulation method or an Aruco QR code detection algorithm. The vision-based triangulation method, also known as a binocular depth calculation method, is implemented as follows:

[0093] First, the grayscale image data of the left and right eyes, containing N target scenes, acquired by the vision sensor, are preprocessed, including grayscale conversion and Gaussian filtering. Then, a feature point extraction algorithm is used to detect feature points in the preprocessed left and right eye images respectively, obtaining key points and their corresponding descriptors. These key points are then sequentially labeled and stored in matrix form. The feature point extraction algorithm can employ SIFT, ORB, SuperPoint, or Shi-Tomasi detection algorithms.

[0094] Then, using the target detection results after the constraint check in step 3, the key target region is outlined in the image. A feature point matching algorithm is used to match the feature points detected in the left and right eye images within this region. The RANSAC iterative algorithm is then used to filter and eliminate mismatched point pairs, and the best matching point pair is taken as the key target point. The feature point matching algorithm can be KNN matching algorithm, FLANN matching algorithm, or brute-force matching BF algorithm.

[0095] Next, based on the principle of binocular vision imaging, the coordinates of the target point on the image planes of the two cameras are obtained. The three-dimensional spatial coordinates of the target point are then obtained by solving the 2D-3D problem using the parallax method. The calculation formula is as follows:

[0096]

[0097] Where (X, Y, Z) represents the three-dimensional coordinates of the target point in the left eye camera coordinate system, (u l v l ) and (u r v r ) represent the coordinates of the target point in the left and right image coordinate systems, respectively; b is the distance between the optical centers of the two cameras; and f is the camera focal length.

[0098] Finally, using the physical positional relationship between the camera and other targets, a coordinate system transformation method is employed to obtain the relative positional relationship between the camera and other targets.

[0099] The specific implementation process based on the Aruco QR code detection algorithm is as follows:

[0100] Step (1), Initial image processing: For the first frame of input image, crop it into 4 images of size w. i ×h i The image, where i is the image number and w i h represents the width of the i-th image. i Let represent the height of the i-th image; then, perform QR code detection and recognition on each image. If the detection fails, perform the same operation on the next frame image until a complete QR code is detected. Record the position and size of the QR code, including its upper left corner coordinates (u,v), width w, and height h.

[0101] Step (2), EKF prediction: Receive the position and size information of several frames of QR codes recorded as input state vectors, and then use the EKF algorithm to construct a motion model. Use the motion model to predict the position (u,v) and size (w,h) of the QR code at the next moment. At the next moment, expand the QR code appropriately according to the predicted value, that is, select the image with the upper left corner coordinates (u-3w,v-3h), width 7w and height 7h as the region of interest;

[0102] Step (3), QR code detection and recognition: After preprocessing the image of interest obtained by EKF prediction, contour detection and filtering are performed to obtain closed contours; polygon approximation and corner point extraction are performed on the filtered contours to find the convex quadrilaterals and obtain the four corner points; then, the QR code is first transformed by perspective within the quadrilateral area to obtain the image from the frontal view, and then the image is encoded, extracted and decoded; the decoded ID is compared with the dictionary and the encoding is verified and the corner points are sorted; finally, the position and size information of the QR code are obtained according to the corner point coordinates, and the EKF prediction value is optimally corrected by using the measurement values ​​of the position and size of the QR code in consecutive frames;

[0103] Step (4), pose calculation: First, based on the position and size information of the QR code, obtain the coordinates of its four corner points in the world frame. Based on the pixel coordinates and camera intrinsic parameters, obtain the coordinates of the four corner points in the normalized camera frame. Then, use the PnP algorithm to calculate the transformation matrix between the world frame and the normalized camera frame. Finally, use the transformation matrix to calculate the pose of the world frame with the center point of the QR code as the origin in the camera frame.

[0104] Step (5), coordinate system transformation: using the physical positional relationship between the QR code and other targets, the relative positional relationship between the camera and other targets is obtained through coordinate system transformation.

[0105] The aforementioned vision-based triangulation method and Aruco-based QR code detection algorithm can be flexibly selected according to different application scenarios and relative distances. The selection is implemented using a pose measurement mode switching state machine, such as... Figure 6 As shown, the state machine includes five basic states: initial state S_INIT, normal state S1 for binocular pose solving, abnormal state S2 for binocular pose solving, normal state S3 for QR code pose solving, and abnormal state S4 for QR code pose solving. Its workflow is as follows:

[0106] (1) Initial state S_INIT: In this state, all counters are reset and it is determined whether the binocular pose solving mode and the QR code pose solving mode are working properly. If neither of them is working properly, the current state is maintained. If only the binocular pose solving mode is working properly, the switch is made to S2. If only the QR code pose solving mode is working properly, the switch is made to S4. If both of them are working properly, the average value of the depth measurement values ​​of the two modes Lm and the distance threshold Ld for switching from binocular pose solving to QR code pose solving are taken. If Lm>Ld, the switch is made to S2; otherwise, the switch is made to S4.

[0107] (2) Normal state S1 for binocular pose solving: In this state, first reset the binocular pose abnormal counter Err_T1 = 0. If the depth measurement value P1 is normal, maintain the current state. If P1 ≤ Ld, switch to S3; if P1 is abnormal, switch to S2.

[0108] (3) Abnormal state S2 for binocular pose solving: In this state, first increment the binocular pose abnormal counter Err_T1 by 1. If the depth measurement value P1 is abnormal, maintain the current state and reset the binocular pose normal counter Nor_T1 = 0. If the abnormal counter Err_T1 exceeds the threshold M1, switch to S_INIT. If the depth measurement value P1 is normal, increment the normal counter Nor_T1 by 1 and maintain the current state. When the normal counter Nor_T1 exceeds the threshold N1, switch to S1.

[0109] (4) Normal state S3 for QR code pose solving: In this state, first reset the QR code pose abnormal counter Err_T2 = 0. If the QR code depth measurement value P2 is normal, maintain the current state. If P2 > Lq, switch to S1; if P2 is abnormal, switch to S4.

[0110] (5) QR code pose solving abnormal state S4: In this state, first increment the QR code pose abnormal counter Err_T2 by 1. If the QR code depth measurement value P2 is abnormal, maintain the current state and reset the QR code pose normal counter Nor_T2 = 0; if the abnormal counter Err_T2 exceeds the threshold M2, switch to S_INIT; if the QR code depth measurement value P2 is normal, increment the normal counter Nor_T2 by 1 and maintain the current state; when the normal counter Nor_T2 exceeds the threshold N2, switch to S3.

Claims

1. A method for accurately estimating the position and attitude of a space target, characterized in that... The steps are as follows: Step 1, Image Data Acquisition: Use a visual sensor to image the target and obtain real-time image data; Step 2, Visual target detection: The real-time image data acquired in Step 1 is processed using a target detection model to output the real-time detection results of the target, including the position and size of the target detection box, the target category, and the confidence score. Step 3, Spatial Constraint Check: The visual relationship check model is used to judge and reason about the real-time target detection results obtained in Step 2. This includes screening the visual target detection results by using the spatial constraints between various targets in the same frame image to remove false and duplicate targets, and predicting missed targets based on prior information, including the target's category, location, and size. Step 4, Position and Pose Calculation: Feature extraction is performed on the image of the target region. The extracted features are matched with the pre-constructed key target feature template, and then the pose calculation method is used to estimate the target position and pose.

2. The method for accurately estimating the position and attitude of a space target as described in claim 1, characterized in that: The object detection model described in step 2 uses convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Transformer-based object detection networks.

3. The method for accurately estimating the position and attitude of a space target as described in claim 1, characterized in that: The visual relationship inspection model described in step 3 is constructed according to the following process: Step 1), Feature Engineering Construction: Various visual relationship features, including target spatial relationship features, target scale features, target category features, and target color features, are used as feature engineering to describe the model; among them, target spatial relationship features include Euclidean distance, Mahalanobis distance, Chebyshev distance, absolute distance, and normalized distance; target scale features include the target's width ratio, height ratio, and area ratio; Step 2), Dataset construction: Collect q images containing p targets of interest to form an image dataset. Randomly select k targets from the p targets, k∈(2,p), and extract the scale, category, color features of the k targets and the spatial relationship features between the k targets to form positive samples. Randomly generate feature values ​​based on the number of features of the positive samples to form negative samples. Step 3), Model Training: Construct a model pool consisting of logistic regression, random forest, support vector machine, naive Bayes and decision tree algorithms. Use the dataset constructed in Step 2) to train the different algorithms in the model pool several times. Select the model with the best performance in terms of accuracy, recall and computation speed as the visual relationship inspection model. The model uses features from feature engineering to describe and judge each target in the target set, and obtains the probability that the target set meets the constraint check model. If the probability is greater than the set threshold, the set is determined to be a fixed combination of targets that meet the requirements; otherwise, it is determined that there are targets in the set that do not meet the conditions.

4. The method for accurately estimating the position and attitude of a space target as described in claim 1, characterized in that: The pose determination methods described in step 4 include satellite-based DGPS relative positioning, lidar relative positioning, and deep learning-based visual pose measurement.

5. The method for accurately estimating the position and attitude of a space target as described in claim 1, characterized in that: Step 4 is implemented using a vision-based triangulation method, and the specific process is as follows: First, the grayscale image data of the left and right eyes containing N target scenes acquired by the vision sensor are preprocessed, including grayscale conversion and Gaussian filtering. Then, the feature point extraction algorithm is used to detect feature points in the preprocessed left and right eye images respectively. After obtaining key points and corresponding descriptors, they are numbered in order and saved in the form of a matrix. Then, using the target detection results after the constraint check in step 3, the key target area is outlined in the image. The feature point matching algorithm is used to match the feature points detected in the left and right eyes within this range. The RANSAC iterative algorithm is used to filter and eliminate mismatched point pairs. The best matching point pair is taken as the key target point. Next, based on the principle of binocular vision imaging, the coordinates of the target point on the image planes of the two cameras are obtained. The three-dimensional spatial coordinates of the target point are then obtained by solving the 2D-3D problem using the parallax method. The calculation formula is as follows: Where (X, Y, Z) represents the three-dimensional coordinates of the target point in the left eye camera coordinate system, (u l v l ) and (u r v r ) represent the coordinates of the target point in the left and right image coordinate systems, respectively; b is the distance between the optical centers of the two cameras; and f is the camera focal length. Finally, using the physical positional relationship between the camera and other targets, a coordinate system transformation method is employed to obtain the relative positional relationship between the camera and other targets.

6. The method for accurately estimating the position and attitude of a space target as described in claim 1, characterized in that: Step 4 is implemented using the Aruco QR code detection algorithm, and the specific process is as follows: Step (1), Initial image processing: For the first frame of input image, crop it into 4 images of size w. i ×h i The image, where i is the image number and w i h represents the width of the i-th image. i Let represent the height of the i-th image; then, perform QR code detection and recognition on each image. If the detection fails, perform the same operation on the next frame image until a complete QR code is detected. Record the position and size of the QR code, including its upper left corner coordinates (u,v), width w, and height h. Step (2), EKF prediction: Receive the position and size information of several frames of QR codes recorded as input state vectors, and then use the EKF algorithm to construct a motion model. Use the motion model to predict the position (u,v) and size (w,h) of the QR code at the next moment. At the next moment, select the region with the upper left corner coordinates (u-3w,v-3h), width 7w and height 7h as the region of interest based on the predicted value. Step (3), QR code detection and recognition: After preprocessing the region of interest obtained by EKF prediction, contour detection and screening are performed to obtain a closed contour. Polygon approximation and corner point extraction are performed on the contour to find the convex quadrilateral and four corner points. In the quadrilateral area, the QR code is first transformed by perspective to obtain the image under the frontal view. Then, the image encoding extraction and decoding are performed. The ID obtained by decoding is compared with the dictionary, the encoding is verified and the corner points are sorted. Finally, the position and size information of the QR code are obtained according to the corner point coordinates, and the EKF prediction value is optimally corrected by using the measurement values ​​of the position and size of the QR code in consecutive frames. Step (4), pose calculation: First, based on the position and size information of the QR code, obtain the coordinates of its four corner points in the world frame. Based on the pixel coordinates and camera intrinsic parameters, obtain the coordinates of the four corner points in the normalized camera frame. Then, use the PnP algorithm to calculate the transformation matrix between the world frame and the normalized camera frame. Finally, use the transformation matrix to calculate the pose of the world frame with the center point of the QR code as the origin in the camera frame. Step (5), coordinate system transformation: using the physical positional relationship between the QR code and other targets, the relative positional relationship between the camera and other targets is obtained through coordinate system transformation.

7. The method for accurately estimating the position and attitude of a space target as described in claim 3, characterized in that: The specific process of screening visual target detection results using spatial constraints between various targets in the same frame image, as described in step 3, is as follows: Step ①: For the target set S = {T1, T2,..., Tn} output by the object detection model, where each target Ti contains its category Ci, location Pi, and confidence Conf_i information, 1 ≤ i ≤ n, 1 ≤ Ci ≤ m, m is the total number of categories. Let the maximum number of each type of target in the fixed combination be p[e], 1 ≤ e ≤ m, and the minimum number of expected output targets be k; Step ②, Preprocessing and initialization: Group S by category into G[1], G[2],..., G[m]. The number of targets in the e-th group is |G[e]|, and sort the targets in each group from high to low according to the confidence. Initialize the set S_opt as an empty set, max_size = 0, and enter Step ③; Step 3, construct the initial subset: First, let the subset S_cur be an empty set. Then, for each e from 1 to m, if G[e] is not empty, take the target with the highest confidence in G[e] and add it to the subset S_cur, until |S_cur|≥ Proceed to step ④; Step ④, Check the initial subset: If CheckModel(S_cur) is true, update the set S_opt according to S_opt = S_cur, update max_size according to max_size = |S_cur|, and enter Step ⑤; otherwise, remove the target with the highest confidence in the first group G[1], and return to Step ③. If all targets in G[1] are removed, it means that all targets of this category are misdetected. Then let G[2] be the starting group for traversal, take the target with the highest confidence from the groups that have not been taken yet and add it to the subset S_cur, and then check; where CheckModel(S_cur) means using the visual relationship check model to judge the set S_cur. If it is greater than the set threshold, it is considered that CheckModel(S_cur) is true, indicating that the current set S_cur is a fixed combination of targets that meet the requirements; Step ⑤, Expand the subset: Let e range from 1 to m, repeat the following operations, and the finally obtained set S_opt is the object detection result after screening: If the number of targets corresponding to category Ce in the current S_cur < min{p[e], |G[e]|}, traverse the remaining targets Hx in G[e] from high to low according to the confidence, 1 ≤ x ≤ ||p[e]| - |G[e]||. Let the set S_temp = S_cur ∪ {Hx}. If CheckModel(S_temp) is true, update the set S_opt and S_cur according to S_opt = S_cur = S_temp, and update max_size according to l = max_size = |S_cur|; otherwise, continue to try the next target in this category; where the remaining targets refer to the targets in G[e] except those added to the subset S_cur; Perform the same operations for all types of targets, and finally complete the screening of all targets.

8. The method for accurately estimating the position and attitude of a space target as described in claims 3 to 7, characterized in that: The specific process of predicting missed detection targets according to the prior information described in Step 3 is as follows: Step a, Missing target location estimation: Suppose the screened target set S_opt contains l targets Z, and the total number of targets in the fixed combination is . If Zl≠0, it indicates that there are missed detections in the current set, and the number of missing targets is Zl. Select a type of missing target, use the spatial relationship features between the missing target and the detected targets to estimate the set C of all possible center point coordinates of the target. Combine each center point in C with the set S_opt and input it into the visual relationship inspection model to obtain the probability values ​​of different combinations. Select the center point corresponding to the maximum probability value as the center point of the missing target. Step b, Optimization of missing target position: Move the estimated missing target center point in its vertical, horizontal and two diagonal directions by step size s. Combine the moved point with S_opt and input it into the visual relationship inspection model. Calculate the combination probability value. Select the center point movement direction corresponding to the maximum probability value as the optimal movement direction. At the same time, update the center point position corresponding to the maximum probability value. Continue to repeat the above process r times from this point to obtain the final optimized center point position. Step c, missing target size completion: estimate the width and height of the missing target using the statistical values ​​of the target bounding box scale features, and combine them with the center point position obtained in step b to complete the target bounding box completion for this type of target. Step d, setting the confidence level of missing targets: Based on the optimal combination probability value in step b, set the confidence level of the detection box for missing targets; Step e, iterative prediction: Let l = l + 1, repeat the above steps until...

9. The method for accurately estimating the position and attitude of a space target as described in claim 5, characterized in that: The target feature point extraction algorithms include SIFT algorithm, ORB algorithm, SuperPoint algorithm, and Shi-Tomasi detection algorithm; the feature point matching algorithms include KNN matching algorithm, FLANN matching algorithm, and brute-force matching BF algorithm.

10. A method for accurately estimating the position and attitude of a space target as described in claim 5 or 6, characterized in that: The vision-based triangulation method and the Aruco-based QR code detection algorithm can be flexibly selected according to different application scenarios and relative distances. The selection is implemented using a pose measurement mode switching state machine. This state machine includes five basic states: initial state S_INIT, normal state for binocular pose solving S1, abnormal state for binocular pose solving S2, normal state for QR code pose solving S3, and abnormal state for QR code pose solving S4. Its workflow is as follows: Step a), Initial state S_INIT: In this state, all counters are reset, and it is determined whether the binocular pose solving mode and the QR code pose solving mode are working normally. If neither of them is working, the current state is maintained. If only the binocular pose solving mode is working normally, the process switches to S2. If only the QR code pose solving mode is working normally, the process switches to S4. If both are working normally, the average depth measurement value Lm of the two modes and the distance threshold Ld for switching from binocular pose solving to QR code pose solving are taken. If Lm > Ld, the process switches to S2; otherwise, the process switches to S4. The binocular pose solving mode refers to the mode using a vision-based triangulation method, and the QR code pose solving mode refers to the mode using an Aruco QR code detection algorithm. Step b), normal state S1 for binocular pose solving: In this state, first reset the binocular pose abnormality counter Err_T1 = 0. If the depth measurement value P1 is normal, maintain the current state. If P1 ≤ Ld, switch to S3; if P1 is abnormal, switch to S2. Step c), Binocular pose solving abnormal state S2: In this state, first increment the binocular pose abnormal counter Err_T1 by 1. If the depth measurement value P1 is abnormal, maintain the current state and reset the binocular pose normal counter Nor_T1 = 0; if the abnormal counter Err_T1 exceeds the threshold M1, switch to S_INIT; if the depth measurement value P1 is normal, increment the normal counter Nor_T1 by 1 and maintain the current state; when the normal counter Nor_T1 exceeds the threshold N1, switch to S1. Step d), QR code pose solving normal state S3: In this state, first reset the QR code pose abnormal counter Err_T2 = 0. If the QR code depth measurement value P2 is normal, maintain the current state. If P2 > Lq, switch to S1; if P2 is abnormal, switch to S4. Step e), QR code pose solving abnormal state S4: In this state, first increment the QR code pose abnormal counter Err_T2 by 1. If the QR code depth measurement value P2 is abnormal, maintain the current state and reset the QR code pose normal counter Nor_T2 = 0; if the abnormal counter Err_T2 exceeds the threshold M2, switch to S_INIT; if the QR code depth measurement value P2 is normal, increment the normal counter Nor_T2 by 1 and maintain the current state; when the normal counter Nor_T2 exceeds the threshold N2, switch to S3.

Images