A method and system for collaborative positioning of astronauts and scientific targets on the lunar surface

Abstract

This application relates to the fields of positioning, navigation, semantic segmentation, and target detection technologies, and particularly to a method and system for collaborative positioning of astronauts and scientific targets on the lunar surface. The method includes: constructing a feature point map by collecting visual and inertial data, and using VIO sliding window optimization to estimate the astronaut's pose, generating a 3D environmental point cloud; further performing 3D target detection on the point cloud, distinguishing between scientific targets and lunar base station objects, and using prior information of scientific targets and a random sampling consensus algorithm for screening and constraint, constructing geometric constraints by fitting a geometric model of the targets and feeding them back into the pose optimization process, introducing iterative optimization to achieve pose correction; simultaneously calculating the relative pose of the astronaut and each target, and updating the environmental map. This method improves positioning accuracy and robustness in the absence of GPS and is suitable for autonomous navigation and mission execution in the complex lunar environment.

Description

Technical Field

[0001] This application relates to the fields of positioning, navigation, semantic segmentation, and target detection technologies, and in particular to a method and system for collaborative positioning of astronauts and scientific targets on the lunar surface. Background Technology

[0002] Achieving high-precision autonomous positioning and 3D semantic perception in GPS-free environments is a key technical challenge for lunar surface operations. The core objective is to accurately segment key targets such as lunar base stations and scientific objectives in the unique environment of the moon, characterized by weak texture, strong radiation, extreme lighting, and the absence of GPS signals. Simultaneously, it aims to achieve high-precision autonomous positioning of both targets and personnel, providing reliable technical support for path planning, mission decision-making, and safety avoidance. In existing technologies, Visual Inertial Odometry (VIO), leveraging the fusion of visual sensors and Inertial Measurement Units (IMUs), can estimate the six-position of a vehicle even without satellite positioning. In 2017, Tong Qin et al. proposed VINS-Mono, which employs a tightly coupled sliding window nonlinear optimization framework to integrate IMU pre-integration constraints and visual feature reprojection errors into the optimization, thereby achieving high-precision monocular visual inertial state estimation. In 2020, Carlos Campos et al. proposed ORB-SLAM3, further integrating multi-map and loop closure mechanisms into the visual / visual-inertial SLAM framework, demonstrating strong robustness and real-time performance in typical indoor and outdoor scenarios. In 2018, Y. Zhang et al. proposed PL-VIO, which introduces joint constraints of point features and line features in tightly coupled optimization to improve stability in certain weak texture scenes.

[0003] On the other hand, in the area of ​​3D point cloud semantic understanding, Qi et al. proposed the PointNet network in 2017 to achieve end-to-end feature learning for unordered point clouds, which can be used for point cloud classification and semantic segmentation. Subsequently, Qi et al. further proposed PointNet++, which enhances the ability to express local geometric structures through hierarchical sampling and local feature aggregation mechanisms. In the direction of localization and semantic fusion, McCormac et al. proposed the SemanticFusion method in 2017 to fuse semantic information into the SLAM graph building process; Salas-Moreno et al. proposed the SLAM++ method in 2013, which enhances localization accuracy by introducing object geometric models as constraints. These methods attempt to introduce semantic or geometric structures as factor constraints into graph optimization to improve the localization stability of the system in structured scenes.

[0004] While the aforementioned technologies provide a foundation for GPS-free positioning and 3D point cloud semantic understanding, they still have the following shortcomings for the lunar environment and mission requirements addressed in this invention:

[0005] (1) The VIO positioning is not robust enough due to the poor adaptability to the extreme lunar environment. Most existing VIO / SLAM methods are designed and verified under conventional ground lighting and sensor noise models (such as VINS-Mono, ORB-SLAM3, etc.). Their front end usually relies on stable visual feature tracking. Under conditions of large area weak texture, strong lighting contrast and significant changes in sensor noise, feature matching degradation and pose drift accumulation are likely to occur. Moreover, there is a lack of dedicated constraint utilization and correction mechanisms for lunar environmental disturbances.

[0006] (2) Although point cloud semantic segmentation can identify targets, it is often decoupled from the localization module, making it difficult to form stable constraints that can be used for pose correction. PointNet / PointNet++ methods mainly solve the problem of point cloud semantic understanding, but do not directly provide a closed-loop correction mechanism for VIO pose estimation. Even if structural factors such as planes are introduced in some semantic SLAM studies, they usually rely on specific sensor morphology (such as RGB-D) or structured indoor scene assumptions, making it difficult to directly transfer to the construction and robust selection of target-level constraints under the conditions of complex lunar terrain and sparse point clouds.

[0007] (3) There is a lack of "prior + semantic" constraint construction and feedback loop for mission-critical objectives (such as lunar base stations, craters, etc.).

[0008] Existing technologies often use localization and perception in series: VIO outputs point clouds for segmentation and detection, but the semantic results obtained from segmentation (and their target geometric priors) are not systematically transformed into optimizable geometric factors and fed back into the sliding window / graph optimization; therefore, it is difficult to actively suppress localization drift and error accumulation by utilizing "stable target structure", and it is also difficult to output target relative pose information for navigation decision-making at the same time. Summary of the Invention

[0009] The purpose of this application is to overcome the defects and deficiencies of the existing technology, thereby providing a method and system for collaborative positioning of astronauts and scientific targets on the lunar surface.

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for real-time collaborative positioning of astronauts and scientific targets in the lunar environment. This aims to achieve continuous pose estimation of astronauts using visual inertial odometry in the lunar environment with weak texture, extreme lighting, and strong radiation, thereby improving positioning accuracy, robustness, and adaptability. It also provides a method for identifying key lunar targets based on point cloud semantic segmentation, enabling 3D target segmentation and positioning of key objects such as lunar base stations and craters. Furthermore, it constructs a pose feedback optimization mechanism based on semantic targets and prior information, transforming target geometric information into pose optimization constraints to suppress pose drift and error accumulation caused by weak texture environment and noise factors. Simultaneously with positioning and target recognition, it calculates the relative positional relationship between the astronaut and key targets in real time, providing spatial location basis for path planning and navigation decisions.

[0011] To address the aforementioned technical problems, the present application provides a method for collaborative positioning of astronauts and scientific targets on the lunar surface, comprising:

[0012] Collect visual data and inertial data, extract visual feature points from the visual data, assemble the visual feature points into a feature point map, and update the environment map;

[0013] The feature point map and the inertial data are input into the VIO module. The sliding window optimization of the VIO module is used to jointly optimize the visual data and the inertial data to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map.

[0014] Three-dimensional target detection is performed on the three-dimensional environment point cloud map to obtain a point cloud set of multiple targets, including: point cloud of scientific target objects and point cloud of lunar base station objects;

[0015] The point cloud of the scientific target object is a set of point clouds of scientific targets. Combining its prior information, the point cloud of the scientific target object is robustly screened using a random sampling consensus algorithm to obtain a target fitting geometric model. Then, the target fitting geometric model is reintroduced into the sliding window optimization as a geometric constraint to correct the pose estimation result and calculate its pose correction amount. If the pose correction amount is less than or equal to a preset threshold, the sliding window optimization iteration process based on the target fitting geometric model is stopped; if the pose correction amount is greater than or equal to the preset threshold, the sliding window optimization is reintroduced based on the target fitting geometric model as a geometric constraint until the pose correction amount is less than or equal to the preset threshold.

[0016] The positions of the center points of the scientific target object point cloud and the lunar base station object point cloud in the world coordinate system are calculated respectively, and the relative pose information between the astronaut and the target is calculated based on the astronaut pose, the position of the center point of the scientific target object, and the position of the center point of the lunar base station object.

[0017] Finally, the environmental map is updated based on the feature point map, the point cloud of the scientific target object, and the point cloud of the lunar base station object.

[0018] According to the astronaut-scientific target collaborative positioning method of the present invention, the visual feature point extraction of visual data is performed by extracting corner points from the image and generating descriptors through the ORB feature extraction algorithm to obtain visual feature points.

[0019] According to the astronaut-scientific target cooperative positioning method of the present invention, the VIO module performs joint optimization of visual observation constraints and inertial constraints through the VIO tightly coupled algorithm to estimate the astronaut's pose in three-dimensional space.

[0020] According to the astronaut-scientific target collaborative positioning method of the present invention, the joint optimization is to jointly optimize visual data and inertial data within the sliding window.

[0021] According to the astronaut-scientific target collaborative positioning method of the present invention, the semantic segmentation of the three-dimensional environmental point cloud map is performed point-by-point classification based on the PointNet++ structure point cloud semantic segmentation network to distinguish different categories of point clouds.

[0022] According to the astronaut-scientific target collaborative localization method of the present invention, the three-dimensional target detection includes: semantic segmentation of the three-dimensional environmental point cloud map and target category clustering, wherein the target category clustering is spatial clustering of point cloud regions belonging to the same target category.

[0023] According to the astronaut-scientific target cooperative localization method of the present invention, the correction of pose estimation results is based on the RANSAC algorithm and prior information, robustly screening the three-dimensional target detection and semantic segmentation results, constructing reliable geometric constraints, and feeding the geometric constraints as factors into the visual-inertial sliding window optimization to correct the pose estimation results.

[0024] According to the astronaut-scientific target cooperative positioning method of the present invention, the relative pose information between the astronaut and the target includes: the relative distance and azimuth angle and pitch angle between the astronaut and the target.

[0025] The present invention also provides a collaborative positioning system for astronauts and scientific targets applicable to the lunar surface, for performing any of the methods described above, comprising:

[0026] Data collection module: used to collect visual and inertial data;

[0027] Feature point extraction module: used to extract visual feature points from visual data, and to form a feature point map from the visual feature points. The feature point map and the inertial data are then input into the VIO module.

[0028] VIO module: Used to jointly optimize visual and inertial data through sliding window optimization to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map;

[0029] Semantic segmentation module: used to perform semantic segmentation on the 3D environment point cloud map and spatial clustering of point clouds of the same category to obtain multiple target point cloud sets, including: scientific target object point cloud and lunar base station object point cloud;

[0030] Constraint filtering module: For the point cloud of the scientific target object, the module performs constraint filtering on the point cloud of the scientific target object in combination with its prior information, and filters the point cloud of the scientific target object through a random sampling consensus algorithm, and then fits to obtain the target fitting geometric model;

[0031] Sliding window optimization and correction module: used to reintroduce the target fitted geometric model as a geometric constraint into the sliding window optimization to correct the pose estimation results;

[0032] Pose calculation module: calculates the position of the center point of the scientific target object point cloud and the lunar base station object point cloud in the world coordinate system, and calculates the relative pose information between the astronaut and the target based on the astronaut pose, the position of the center point of the scientific target object and the position of the center point of the lunar base station object;

[0033] Map Management and Update Module: Updates and manages the environmental map based on the feature point map, the point cloud of the scientific target object, and the point cloud of the lunar base station object.

[0034] According to the astronaut-scientific target cooperative positioning system of the present invention, the data collection module includes: a monocular camera for collecting visual data and an inertial measurement unit for collecting inertial data.

[0035] The advantages of this application are:

[0036] This invention utilizes visual sensors and feature extraction methods to construct 3D point cloud data of the environment. Without GPS positioning, it achieves continuous estimation of astronaut pose in the lunar environment with weak texture, extreme lighting, and strong radiation through tightly coupled visual-inertial odometry, thereby improving positioning accuracy, robustness, adaptability, and autonomous positioning capability in the lunar environment.

[0037] Meanwhile, this invention combines prior information about lunar scientific targets and constructs a feedback mechanism through semantic segmentation and geometric constraints. This introduces stable target geometric information into the pose optimization process, effectively suppressing pose drift and error accumulation caused by factors such as weak texture environment and radiation noise, and improving the long-term stability and robustness of positioning results.

[0038] Furthermore, while acquiring the astronaut's pose, this invention estimates the positions of key targets such as lunar base stations and scientific objectives in the world coordinate system, and calculates the relative pose information between the astronaut and the targets in real time, thereby providing a reliable spatial position basis for path planning, target navigation, and operational decision-making. Attached Figure Description

[0039] Figure 1 The diagram shown is a flowchart of the method proposed in this invention.

[0040] Figure 2 The diagram shown is a flowchart of the pose feedback mechanism proposed in this invention.

[0041] Figure 3 The diagram shown is a flowchart of the PointNet proposed in this invention.

[0042] Figure 4 The flowchart of PointNet++ proposed in this invention is shown below;

[0043] Figure 5 This is a flowchart of the real-time calculation of relative position proposed in this invention. Detailed Implementation

[0044] The technical solutions provided in this application are further illustrated below with reference to the embodiments.

[0045] Example 1

[0046] A method for collaborative localization of astronauts and scientific targets on the lunar surface, comprising:

[0047] Collect visual and inertial data, extract visual feature points from the visual data, assemble the visual feature points into a feature point map, and update the environment map.

[0048] The feature point map and inertial data are input into the VIO module. The sliding window optimization of the VIO module is used to jointly optimize the visual data and inertial data to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map.

[0049] 3D target detection is performed on the 3D environment point cloud map to obtain a point cloud set of multiple targets, including: point cloud of scientific target objects and point cloud of lunar base station objects;

[0050] The point cloud of the scientific target object is a set of point clouds of scientific targets. Combining its prior information, the point cloud of the scientific target object is robustly screened using a random sampling consensus algorithm to obtain a target fitting geometric model. Then, the target fitting geometric model is reintroduced as a geometric constraint into the sliding window optimization to correct the pose estimation results and calculate its pose correction amount. If the pose correction amount is less than or equal to a preset threshold, the sliding window optimization iteration process based on the target fitting geometric model is stopped; if the pose correction amount is greater than or equal to the preset threshold, the sliding window optimization is reintroduced as a geometric constraint based on the target fitting geometric model until the pose correction amount is less than or equal to the preset threshold.

[0051] Calculate the positions of the center points of the scientific target point cloud and the lunar base station point cloud in the world coordinate system, respectively, and calculate the relative pose information between the astronaut and the target based on the astronaut's pose, the position of the center point of the scientific target, and the position of the center point of the lunar base station.

[0052] Finally, the environmental map is updated based on the feature point map, the point cloud of scientific target objects, and the point cloud of lunar base station objects.

[0053] According to the astronaut-scientific target collaborative positioning method of the present invention, visual feature point extraction of visual data is performed by extracting corner points from the image and generating descriptors through the ORB feature extraction algorithm to obtain visual feature points.

[0054] According to the astronaut-scientific target cooperative positioning method of the present invention, the VIO module performs joint optimization of visual observation constraints and inertial constraints through the VIO tightly coupled algorithm to estimate the astronaut's pose in three-dimensional space.

[0055] According to the astronaut-scientific target collaborative positioning method of the present invention, joint optimization is performed on visual data and inertial data within the sliding window.

[0056] According to the astronaut-scientific target collaborative localization method of the present invention, semantic segmentation of the three-dimensional environmental point cloud map is performed point-by-point classification based on the PointNet++ structure of the point cloud semantic segmentation network to distinguish different categories of point clouds.

[0057] According to the astronaut-scientific target collaborative localization method of the present invention, the three-dimensional target detection includes: semantic segmentation of the three-dimensional environmental point cloud map and target category clustering, wherein target category clustering is spatial clustering of point cloud regions belonging to the same target category.

[0058] According to the astronaut-scientific target cooperative localization method of the present invention, the pose estimation results are corrected based on the RANSAC algorithm and prior information. The results of three-dimensional target detection and semantic segmentation are robustly screened to construct reliable geometric constraints. The geometric constraints are then fed back as factors to the visual-inertial sliding window optimization to correct the pose estimation results.

[0059] According to the astronaut-scientific target cooperative positioning method of the present invention, the relative pose information between the astronaut and the target includes: the relative distance and azimuth angle and pitch angle between the astronaut and the target.

[0060] Example 2

[0061] The present invention also provides a collaborative positioning system for astronauts and scientific targets applicable to the lunar surface, for performing any of the methods described above, comprising:

[0062] Data collection module: used to collect visual and inertial data;

[0063] Feature point extraction module: used to extract visual feature points from visual data, and to assemble the visual feature points into a feature point map. The feature point map and inertial data are then input into the VIO module.

[0064] VIO module: Used to jointly optimize visual and inertial data through sliding window optimization to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map;

[0065] Semantic segmentation module: used to perform semantic segmentation on 3D environment point cloud maps and spatial clustering of point clouds of the same category to obtain multiple target point cloud sets, including: point clouds of scientific target objects and point clouds of lunar base station objects;

[0066] Constraint Filtering Module: For the point cloud of the scientific target object, the module performs constraint filtering on the point cloud of the scientific target object based on its prior information, and then filters the point cloud of the scientific target object through the random sampling consensus algorithm, and finally fits the target fitting geometric model.

[0067] Sliding window optimization and correction module: This module is used to reintroduce the target fitted geometric model as a geometric constraint into the sliding window optimization to correct the pose estimation results.

[0068] Pose calculation module: Calculates the position of the center point of the scientific target object point cloud and the lunar base station object point cloud in the world coordinate system, and calculates the relative pose information between the astronaut and the target based on the astronaut pose, the position of the center point of the scientific target object and the position of the center point of the lunar base station object;

[0069] Map Management and Update Module: Updates and manages the environmental map based on the feature point map, the point cloud of scientific target objects, and the point cloud of lunar base station objects.

[0070] According to the astronaut-scientific target cooperative positioning system of the present invention, the data collection module includes: a monocular camera for collecting visual data and an inertial measurement unit for collecting inertial data.

[0071] Example 3

[0072] like Figure 1 The diagram shows the method flowchart of this invention. The top layer of the system consists of hardware sensors carried by the astronaut, mainly including a monocular camera and an inertial measurement unit (IMU). The monocular camera is responsible for acquiring visual data containing visual information about the environment, while the IMU provides inertial data containing motion information such as the astronaut's acceleration and angular velocity. These two types of sensors provide raw data input for subsequent positioning, mapping, and target recognition. The process involves acquiring visual information (visual data), measuring astronaut motion information (inertial data), and providing raw input for subsequent positioning, mapping, and target recognition.

[0073] In the front-end processing unit, visual data first enters the unit, where the system extracts stable visual feature points from the image and organizes these feature points into a feature point map. This map serves as an intermediate representation to support subsequent pose estimation and optimization processes. The feature point map and inertial data are input together into the VIO module.

[0074] In this embodiment, visual feature points are extracted using the ORB feature extraction algorithm. ORB is a fast binary feature point detection and description algorithm with rotation and scale invariance, and is an efficient alternative to SIFT and SUFT. ORB combines improved FAST corner detection points and BRIEF descriptors, and makes the feature points rotation-invariant by calculating the principal orientation of the feature points, and achieves scale invariance by constructing an image pyramid.

[0075] 1. FAST corner detection

[0076] Compare the grayscale values ​​of a given pixel with those of the 16 pixels surrounding it on a circle. If N consecutive pixels (usually 12) have grayscale values ​​that are simultaneously greater than or simultaneously less than the grayscale value of the center pixel (plus a threshold), then the comparison is successful. If the point is a corner point, then that point is considered a corner point.

[0077]

[0078] in, This represents the grayscale value of the center pixel. This represents the grayscale value of the pixel on the circumference. This represents the grayscale difference threshold.

[0079] 2. BRIEF descriptor

[0080] Within a square neighborhood around the feature point, randomly select n pairs of pixels (e.g., n=256) and compare the grayscale values ​​of each pair. If the first pixel is greater than the second, the corresponding bit is set to 1; otherwise, it is set to 0. This results in an n-bit binary string as a descriptor.

[0081]

[0082] in Point grayscale value, Point grayscale value, This represents a binary function.

[0083] Rotation correction: based on the direction calculated in the first step. Rotate the random points of BRIEF relative to the template Angle, to ensure rotational invariance.

[0084]

[0085] in, Represents the set of original sample point pairs. Represents the rotation matrix. This represents the set of sampling points after rotation.

[0086] 3. Hamming Feature Matching

[0087] Hamming distance is used to compare two binary descriptors, which involves calculating the number of different bits in two binary strings. The smaller the distance, the more similar the features.

[0088]

[0089] in, A binary vector representing two feature descriptors. Descriptor 1 Bit, Descriptor 2 Bit, Indicates the length of the descriptor. This represents the XOR operation. This represents the Hamming distance.

[0090] The VIO module uses a sliding window optimization method to jointly optimize observation data over a period of time, thereby estimating the astronaut's pose in space.

[0091] In this embodiment, the VIO module employs visual inertial odometry, with the tightly coupled VIO fusion algorithm serving as its core. It fuses and optimizes image feature information and raw IMU measurements within a unified state estimation framework. Its goal is to estimate the lunar rover's pose more accurately and robustly using both image and IMU data. Compared to loose coupling, it utilizes more raw information, is more robust to rapid motion and weakly textured environments, and is better suited to the lunar environment.

[0092] 1. State Vector

[0093] The state vector contains all the variables to be optimized, typically within a sliding window:

[0094]

[0095] in, Represented as the first System status at all times Represents the feature point parameters. (System number...) Frame state is generally defined as:

[0096]

[0097] in, This refers to the position of the IMU unit in the world coordinate system. The attitude of the IMU unit in the world coordinate system. The velocity of the IMU unit in the world coordinate system. Bias for the IMU accelerometer / gyroscope.

[0098] 2. IMU pre-integration constraints

[0099] To avoid repeated integration during state changes, for adjacent frames and Pre-integrate all IMU measurements between them to obtain a state-independent incremental measurement. .

[0100] The IMU residual term is defined as the difference between the pre-integrated measurement and the predicted value estimated based on the state:

[0101]

[0102] in, It is rotational error. It is position and velocity error. For acceleration bias error, This refers to the gyroscope bias error.

[0103] 3. Visual reprojection constraints

[0104] For a in the first Map points observed in a frame In its first The reprojection error on the frame image is:

[0105]

[0106] in, Image pixel coordinates, and These are the camera projection and back projection functions, respectively. It is the pose transformation matrix. For camera-to-IMU conversion, Indicates the first Transformation from IMU to world coordinate system at frame rate. This indicates a transformation from the world coordinate system to the first coordinate system. The IMU coordinate system of the frame.

[0107] Joint optimization objective function:

[0108]

[0109] in, It is the set of IMU pre-integration constraints. Represents a set of visual observations. , It is an information matrix of each measurement, used for weighting. It is a robust kernel function used to suppress the effects of mismatches.

[0110] 3D target detection involves first performing 3D semantic segmentation, followed by target category clustering. Detected 3D targets are classified into different categories based on semantics and attributes, such as lunar base station objects and scientific target objects. Different types of targets have different priorities and roles in subsequent processing and decision-making.

[0111] For point clouds of scientific targets, the system combines prior scientific information with random sampling consistency to remove noise points and filter interior points, improving the accuracy of the target's geometric description. A fitted geometric model of the target is then obtained. This fitted geometric model is reintroduced as a geometric constraint into the sliding window optimization to calibrate the visual-inertial odometry. This process forms a closed-loop feedback, improving the stability and long-term consistency of the overall pose estimation.

[0112] In pose estimation based on vision-inertial fusion, outliers can be included in the geometric observations used for pose optimization due to visual feature mismatches, environmental noise, and target recognition errors. This reduces pose estimation accuracy and affects system stability and reliability. This invention introduces a semantic target-based pose feedback mechanism in the sliding window optimization process of visual-inertial odometry. This mechanism robustly filters the results of 3D target detection and semantic segmentation to construct reliable geometric constraints, which are then fed back as factors to the visual-inertial sliding window optimization to correct the pose estimation results. Figure 2 As shown.

[0113] 1. RANSAC algorithm incorporating prior information

[0114] RANSAC is a robust model parameter estimation method that can estimate the parameters of a mathematical model with high probability from observation datasets containing a large number of "outliers" (error data, noise). Addressing the issues of noisy points, missegmented points, and missing points in point clouds of scientific targets, this invention, based on the RANSAC algorithm, incorporates prior geometric information of the scientific target to generate a target fitting model.

[0115] Suppose that the point cloud set of a certain scientific target object obtained after semantic segmentation and clustering is:

[0116]

[0117] in, For the first Point cloud points.

[0118] For different categories of scientific objectives, corresponding geometric prior information is given in advance, including at least one of the following: objective shape type, size range, principal axis direction range, center position range, and curvature range.

[0119] Here, assuming the target, such as a meteorite crater, is modeled using a quadratic surface, its geometric model is represented as follows:

[0120]

[0121] in, It is a symmetric matrix. For vectors, As a scalar, This is the transpose. When the parameters satisfy different constraints, the model can be represented as a spherical model, an ellipsoidal model, or other curved surface models.

[0122] (1) Random sampling

[0123] Randomly select the smallest subset from the entire dataset that can determine the model parameters.

[0124] (2) Model building

[0125] A model parameter is calculated using this minimal subset. After solving for the candidate model parameters, determine whether they satisfy the prior constraints:

[0126]

[0127] in, This represents the feasible parameter domain comprised of prior information about the target. If the candidate model parameters do not satisfy the parameter domain of the prior information, the candidate model is discarded and resampling is performed; otherwise, the following process is initiated.

[0128] (3) Classification of interior points

[0129] Using the calculated model Test with all other data points in the dataset. If the error between a point and the model's prediction is less than a set threshold... If a point is found to be an interior point of the model, then that point is considered an "interior point".

[0130] Count the number of interior points:

[0131]

[0132] in, Indicates the first Error per point, Represents data points, This represents the distance function from the point to the model. Indicates the model parameters A defined geometric model.

[0133] like

[0134]

[0135] but Let be an interior point. To determine the threshold.

[0136] This yields the set of interior points for the candidate model:

[0137]

[0138] (4) Evaluation of the prior consistency model

[0139] This invention constructs a joint scoring function for candidate models:

[0140]

[0141] in The number of interior points, These are the weighting coefficients. This represents the prior parameters corresponding to the target. This represents the parameters of the currently fitted model.

[0142] Finally, the candidate model with the highest score is selected as the optimal target candidate model:

[0143]

[0144] (5) Final estimate

[0145] Least squares optimization with prior constraints based on the optimal set of interior points:

[0146]

[0147] And obtain the fitted target geometric model ;

[0148] in, These are the weighting coefficients.

[0149] 2. Residual Optimization

[0150] (1) Geometric consistency residuals

[0151] The target geometric model, after being filtered by RANSAC and incorporating prior information, is constructed into geometric factors.

[0152]

[0153] in, Represents geometrically constrained residuals. This represents the geometric constraint function under the current pose. For the current pose, This is a geometric consistency metric function used to calculate the geometric consistency at the current pose. Below observation data and target geometric model The deviation between them.

[0154] (2) Prior consistency residuals

[0155] This invention incorporates prior knowledge into residual construction:

[0156]

[0157] in, This represents the prior parameters corresponding to the target. This represents the parameters of the currently fitted model.

[0158] (3) Jointly optimize the objective function

[0159] The residual is introduced into the sliding window optimization process of the visual inertial odometry as a constraint term for pose calculation.

[0160]

[0161] in, This represents the overall optimization objective. , Represents the geometric factor weights. This represents the objective function for optimizing the original VIO sliding window.

[0162] Based on the introduced geometric and prior factors, the pose state within the sliding window is jointly optimized to obtain the corrected astronaut pose estimation result, which is then used for real-time relative position calculation. Minimization The corrected pose is obtained.

[0163]

[0164] in, This is the optimized pose. It is a minimization function.

[0165] For point cloud objects of lunar base stations / scientific targets, the system calculates the pose of its center point in the world coordinate system, inputs the pose information of the astronauts, and calculates the relative pose information in real time.

[0166] In this embodiment, a visual-inertial positioning module is first used to generate 3D point cloud data of the environment. The point cloud is then downsampled and noise filtered to reduce computational complexity and improve the stability of subsequent processing. Subsequently, the processed point cloud data is input into a point cloud semantic segmentation network based on the PointNet++ architecture to classify different categories of points in the scene point by point, distinguishing lunar base station structures, crater areas, and other surface background areas.

[0167] After obtaining the semantic segmentation results, spatial clustering is performed on point cloud regions belonging to the same target category to form multiple independent target point cloud sets. The stability and integrity of the target regions are improved by outlier removal and other methods, thereby obtaining target candidate regions.

[0168] 1. PointNet

[0169] PointNet is an end-to-end deep neural network architecture primarily designed to address the processing difficulties arising from the disordered distribution and irregular structure of 3D point cloud data. Its key innovation lies in proposing a network structure capable of learning features directly from the raw point cloud, allowing the model to perform feature extraction and learning without requiring meshing or constructing other structured representations of the point cloud. To address the issue of the non-fixed order of points in point cloud data, PointNet introduces a symmetric function, ensuring that the network output is unaffected by changes in the order of input points, thus guaranteeing permutation invariance. For data containing... Data set of input points The process of calculating its characteristics can be represented by the following function.

[0170]

[0171] in, This indicates any permutation or arrangement of positions.

[0172] PointNet selects max pooling as the symmetric function.

[0173]

[0174] PointNet uses a network with shared parameters to extract features from each point in the point cloud. Then, it uses a symmetric function to aggregate the features of all points, forming a global feature representation of the entire point cloud. This structural design ensures that the network output is unaffected by changes in the order of the input points, thus guaranteeing the model's adaptability to the unordered nature of point cloud data. For the input 3D point coordinates, the network first maps low-dimensional coordinate information to a high-dimensional feature space using a multilayer perceptron, and then integrates the features of each point through max pooling to obtain a global feature description of the entire point cloud.

[0175] In the PointNet architecture, symmetric operations preserve key information from the point cloud data. This information can then be further processed by another network to extract features from the point cloud. This process can be viewed as a function. and The combination of , the formula is as follows:

[0176]

[0177] in, and Represents a multilayer perceptron. It is a symmetric function.

[0178] like Figure 3The diagram shows the PointNet flowchart. The network input is 3D point cloud data, where each point contains only spatial coordinate information. All points are unordered at input, and the network itself does not depend on the point arrangement. First, the point cloud enters an input transformation network. This module automatically learns a spatial alignment method to transform the original point cloud as a whole, making the point cloud have a more consistent representation under different poses and orientations, thereby improving the network's robustness to rotation and translation. The aligned point cloud is then fed into a shared multilayer perceptron. Here, features are extracted for each point individually, using the same network parameters for all points. This stage maps each point from simple coordinate information to a higher-dimensional local feature representation. Next, the network introduces a second transformation network to align the intermediate features of the points. The goal of this module is to normalize the feature space, making high-dimensional features stable and consistent under different input conditions, which helps to learn a more robust global description later. After feature alignment, the point features are mapped again through the shared multilayer perceptron. This stage gradually increases the feature dimension of each point, making it contain richer geometric and semantic information. The network uses max pooling to aggregate the features of all points. This operation selects the most significant response from each feature dimension, generating a fixed-length global feature vector. Since max pooling is independent of the order of points, the network naturally satisfies the requirement of point cloud unorderedness. The pooled result is a global feature that describes the overall shape and structure of the entire point cloud. This feature integrates the most crucial information from all points, making it suitable for overall judgment or regression tasks. Finally, the global feature is input into a fully connected network, which outputs results based on the specific task. In classification tasks, the network outputs scores for each category; in regression tasks, the network outputs continuous values, such as target attributes or geometric parameters.

[0179] 2. PointNet++

[0180] PointNet++ is an extended version of PointNet designed to improve PointNet's performance when handling complex, non-uniform point cloud data. By introducing local feature learning, PointNet++ addresses PointNet's inability to capture local details when processing large-scale point cloud data. Through recursive hierarchical sampling and neighborhood feature aggregation, it can extract local features of point clouds at different scales, thereby enhancing the network's performance on complex point cloud datasets.

[0181] like Figure 4The diagram shows the PointNet++ flowchart. The network first downsamples the input point cloud, selecting a representative subset of points as center points. Based on these center points, surrounding points are clustered within their neighborhoods, forming multiple local point sets. Each local point set contains not only spatial location information but also feature information passed down from the previous layer. For each local point set, the network uses a shared PointNet unit for feature learning. This PointNet unit extracts features from each point in the local point set and generates a feature representation of the local region through pooling operations. Thus, each center point corresponds to a feature vector describing its local geometry. After sampling, grouping, and PointNet processing, the number of points decreases, but the features carried by each point become richer. The network repeats this process multiple times, progressively expanding the receptive field, allowing point features to gradually transition from local geometry to higher-level, more abstract semantic information. This process forms a hierarchical feature representation similar to a "pyramid structure." After completing hierarchical feature learning, PointNet++ can be divided into different processing branches according to the specific task.

[0182] Classification task: The deepest point set features are considered as a high-level abstract representation of the entire point cloud. These features are integrated through a PointNet module to form global features. The global features are input into a fully connected network. The network outputs a score for each class to complete the overall point cloud classification task.

[0183] Segmentation Task: As the number of points gradually decreases during the preceding hierarchical abstraction process, the segmentation task requires remapping high-level features back to the original points. The network propagates high-level semantic features layer by layer to the low-level point set through feature interpolation. During feature propagation, the network concatenates high-level semantic features with low-level local geometric features. This skip connection method ensures that each point retains fine geometric details while incorporating global semantic information. The fused point features are then used again for point-by-point feature learning through PointNet units. This process enhances the ability of each point to discriminate its semantic category. Finally, the network outputs a category score for each input point, achieving point-level semantic segmentation or instance segmentation tasks.

[0184] After obtaining the astronaut pose estimation results and target semantic segmentation results, this invention further proposes a real-time relative position calculation method based on semantic target localization. This method is used to achieve stable calculation of the spatial relationship between the astronaut and key targets (lunar base station / scientific targets), providing direct geometric basis for navigation decisions and mission planning. Figure 5 As shown.

[0185] Input the astronaut's pose data and the three-dimensional center coordinates of the key target in the world coordinate system, and output the relative position vector between the astronaut and the target. It can also output the relative distance, azimuth angle and pitch angle between the astronaut and the key target.

[0186] If the point cloud of the key target is

[0187]

[0188] Center point:

[0189]

[0190] in, Indicates the number of point clouds, Center point.

[0191] Relative position:

[0192]

[0193] in, Indicates the astronaut's position and orientation. Indicates the key objective center. This indicates the relative position of the astronauts and the key target.

[0194] distance:

[0195]

[0196] Azimuth:

[0197]

[0198] in, This represents the two-parameter arctangent function. This indicates that the astronauts and key targets are in The difference in coordinates of direction This indicates that the astronauts and key targets are in The difference in coordinates of direction.

[0199] Pitch angle:

[0200]

[0201] This indicates that the astronauts and key targets are in The difference in coordinates of direction.

[0202] Finally, on the backend, the point cloud map, composed of feature point maps, lunar base station objects, and scientific target objects, is managed and updated.

[0203] In summary, through the above embodiments, this invention constructs a method and system for real-time collaborative positioning of astronauts and scientific targets suitable for the lunar environment. Compared with traditional visual-inertial positioning or simple target detection methods, this invention achieves continuous pose estimation of astronauts using visual-inertial odometry in the lunar environment with weak texture, extreme lighting, and strong radiation, improving positioning accuracy, robustness, and adaptability. Specifically, this invention mainly has the following technical features and effects:

[0204] 1. Adaptive optimization mechanism for visual-inertial parameters for extreme lunar environments

[0205] To address the problem of increased measurement errors of visual and inertial sensors under weak texture, high radiation, and extreme lighting conditions on the moon, this invention introduces an environment-adaptive parameter estimation mechanism into the visual-inertial tightly coupled optimization framework. During the sliding window optimization process, the bias error of the inertial measurement unit and the visual observation distortion error are dynamically estimated and compensated, thereby reducing the impact of environmental factors on pose estimation and improving the robustness and stability of the system under weak texture and high noise conditions.

[0206] 2. A semantic segmentation method for lunar scene targets based on PointNet++

[0207] To address the problem that general point cloud segmentation models are difficult to adapt to the morphological features of targets in the lunar environment, this invention is based on the PointNet++ hierarchical feature learning structure. It combines the geometric characteristics of lunar base stations, craters, and scientific targets to construct a point cloud semantic segmentation network optimized for lunar scenes, achieving high-precision segmentation of targets in complex terrain, thereby providing a reliable target point cloud set for subsequent geometric constraint construction.

[0208] 3. Semantic target-based pose feedback closed-loop optimization mechanism

[0209] This invention transforms the results of 3D target detection and semantic segmentation into geometric constraint factors that can be used for pose optimization. Furthermore, it combines prior information about the target to construct stable constraints, achieving feedback optimization of the visual-inertial pose estimation results. Specifically, the system first extracts the target region from the target point cloud obtained from semantic segmentation and, combined with the target's prior geometric information or structural parameters, performs constraint screening and model initialization on the target point cloud. Then, robust estimation algorithms such as RANSAC are used to fit the geometric model of the target point cloud, obtaining a stable 3D geometric model. The selected and fitted geometric model is constructed into geometric constraint factors and introduced into the sliding window optimization process of visual-inertial odometry in factor form to provide feedback correction to the pose estimation results. This utilizes the structural stability of the semantic target to suppress pose drift caused by mismatches and noise, forming a closed-loop optimization mechanism of "localization—perception—re-optimization localization".

[0210] 4. Real-time calculation method for target center point and relative position

[0211] After obtaining the target semantic segmentation results, this invention calculates the target center point position by clustering the target point cloud (lunar base station / scientific target), and combines it with the astronaut's real-time pose information to calculate the three-dimensional relative position vector, distance and orientation angle information between the astronaut and the target, so as to realize the real-time output of the key target position and provide direct geometric basis for path planning and navigation decision.

[0212] 5. Integrated processing architecture for localization and target perception

[0213] This invention constructs a collaborative processing architecture for localization and target perception, enabling point cloud generation, target segmentation, geometric constraint construction, and pose optimization to operate collaboratively in the same processing flow. This avoids the problem of one-way propagation of localization errors in traditional systems and improves the overall environmental perception and localization accuracy from the system structure level.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for collaborative positioning of astronauts and scientific targets on the lunar surface, comprising: Collect visual data and inertial data, extract visual feature points from the visual data, assemble the visual feature points into a feature point map, and update the environment map; The feature point map and the inertial data are input into the VIO module. The sliding window optimization of the VIO module is used to jointly optimize the visual data and the inertial data to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map. Three-dimensional target detection is performed on the three-dimensional environment point cloud map to obtain a point cloud set of multiple targets, including: point cloud of scientific target objects and point cloud of lunar base station objects; The point cloud of the scientific target object is a set of point clouds of scientific targets. Combining its prior information and using a random sampling consensus algorithm, the point cloud of the scientific target object is filtered to obtain a target fitting geometric model. Then, the target fitting geometric model is reintroduced into the sliding window optimization as a geometric constraint to correct the pose estimation result and calculate its pose correction amount. If the pose correction amount is less than or equal to a preset threshold, the sliding window optimization iteration process based on the target fitting geometric model is stopped; if the pose correction amount is greater than the preset threshold, the sliding window optimization is reintroduced based on the target fitting geometric model as a geometric constraint until the pose correction amount is less than or equal to the preset threshold. The positions of the center points of the scientific target object point cloud and the lunar base station object point cloud in the world coordinate system are calculated respectively, and the relative pose information between the astronaut and the target is calculated based on the astronaut pose, the position of the center point of the scientific target object, and the position of the center point of the lunar base station object. The environmental map is updated based on the feature point map, the point cloud of the scientific target object, and the point cloud of the lunar base station object.

2. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The visual feature point extraction of visual data is performed by using the ORB feature extraction algorithm to extract corner points from the image and generate descriptors to obtain visual feature points.

3. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The VIO module uses a tightly coupled VIO algorithm to jointly optimize visual observation constraints and inertial constraints in order to estimate the astronaut's pose in three-dimensional space.

4. The method for collaborative positioning of astronauts and scientific targets according to claim 3, characterized in that, The joint optimization involves jointly optimizing the visual data and inertial data within the sliding window.

5. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The 3D target detection of the 3D environment point cloud map is performed point-by-point classification based on a point cloud semantic segmentation network with a PointNet++ structure, which is used to distinguish different categories of point clouds.

6. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The three-dimensional target detection includes: semantic segmentation of the three-dimensional environment point cloud map and target category clustering, wherein target category clustering is spatial clustering of point cloud regions belonging to the same target category.

7. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The correction of the pose estimation results is as follows: based on the RANSAC algorithm and prior information, robust screening of the 3D target detection and semantic segmentation results is performed to construct reliable geometric constraints, and the geometric constraints are fed back as factors to the visual inertial sliding window optimization to correct the pose estimation results.

8. The method for collaborative positioning of astronauts and scientific targets according to claim 1, characterized in that, The relative pose information between the astronaut and the target includes: the relative distance, azimuth angle, and pitch angle between the astronaut and the target.

9. A collaborative positioning system for astronauts and scientific targets applicable to the lunar surface, used to perform the method according to any one of claims 1-8, characterized in that, include: Data collection module: used to collect visual and inertial data; Feature point extraction module: used to extract visual feature points from visual data, and to form a feature point map from the visual feature points. The feature point map and the inertial data are then input into the VIO module. VIO module: Used to jointly optimize visual and inertial data through sliding window optimization to estimate the astronaut's pose in three-dimensional space and generate a three-dimensional environmental point cloud map; Semantic segmentation module: used to perform semantic segmentation on the 3D environment point cloud map and spatial clustering of point clouds of the same category to obtain multiple target point cloud sets, including: scientific target object point cloud and lunar base station object point cloud; Constraint filtering module: For the point cloud of the scientific target object, constrain filtering is performed on the point cloud of the scientific target object in combination with its prior information, and the point cloud of the scientific target object is filtered through random sampling consensus algorithm; Sliding window optimization and correction module: used to reintroduce the target fitted geometric model as a geometric constraint into the sliding window optimization to correct the pose estimation results; Pose calculation module: Calculates the positions of the center points of the scientific target point cloud and the lunar base station point cloud in the world coordinate system, respectively; and calculates the relative pose information between the astronaut and the target based on the astronaut's pose, the positions of the scientific target center points, and the lunar base station center point positions; and Map Update Module: Updates the environment map.

10. The astronaut-scientific target cooperative positioning system according to claim 9, characterized in that, The data collection module includes a monocular camera for collecting visual data and an inertial measurement unit for collecting inertial data.

Images