A new lunar surface wide-range environment fusion perception method

Abstract

The application discloses a novel lunar surface large-range environment fusion perception method and belongs to the field of image processing. The method comprises the following steps: performing feature matching on a new lunar surface image and a historical lunar surface image, and performing geometric filtering processing on a matching result to obtain a globally effective feature matching dataset; performing local triangulation and pose solving on a subset of landmark points associated with the new lunar surface image according to historical camera poses, historical three-dimensional landmark data and the globally effective feature matching dataset to obtain a camera pose corresponding to the new lunar surface image and an effective landmark point subset; performing iterative optimization and abnormal elimination on the camera pose and the effective landmark point subset to obtain an optimized pose and a high-precision local map; and performing online collaborative optimization and offline global fusion processing on all single-camera output high-precision local maps and the optimized pose to obtain a global lunar surface environment map meeting task requirements. The application can simultaneously meet the real-time requirement, mapping precision and large-range collaborative detection requirement of lunar surface detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a novel method for large-scale environmental fusion perception on the lunar surface. Background Technology

[0002] The world is currently experiencing a new wave of lunar exploration. Lunar exploration is inherently characterized by long exploration distances, long mission cycles, and a complex and extreme lunar environment. Traditional single-robot exploration systems suffer from inherent weaknesses such as limited onboard computing resources and insufficient mission resilience. Therefore, highly efficient and robust multi-robot collaborative exploration systems have become a key technological direction for future lunar exploration. The implementation of multi-robot collaborative lunar exploration places dual core rigid requirements on environmental perception technology. On the one hand, it requires stable online operation capabilities for positioning and mapping to meet the basic needs of autonomous real-time navigation for lunar rovers. On the other hand, it requires efficient data collaboration and map fusion within the formation to overcome the exploration range limitations of single robots and achieve comprehensive perception of a large area of ​​the lunar environment.

[0003] Existing unmanned mobile platforms (such as lunar rovers) typically employ two types of technologies: Simultaneous Localization and Mapping (SLAM) and Structure of Motion (SfM). These technologies cannot meet the requirements for large-scale, high-precision mapping of the lunar surface and cannot be implemented online on unmanned autonomous mobile platforms on the lunar surface, which are limited by computing power and energy.

[0004] Therefore, there is an urgent need for a new method for large-scale lunar surface environment fusion perception to solve the above-mentioned technical problems. Summary of the Invention

[0005] This invention provides a novel method for large-scale lunar surface environmental fusion sensing, which can simultaneously meet the requirements of real-time lunar exploration, mapping accuracy, and large-scale collaborative exploration. The technical solution is as follows: On the one hand, a novel method for large-scale lunar surface environment fusion perception is provided, the method comprising: The new lunar surface image after feature extraction is matched with historical lunar surface images, and the matching results are geometrically filtered to obtain a globally effective feature matching dataset that has completed incremental updates. Based on historical camera poses, historical 3D landmark data, and the global effective feature matching dataset, local triangulation and pose solving are performed on the subset of landmark points associated with the new lunar surface image to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image. The camera pose and effective punctuation subset corresponding to the lunar surface image are iteratively optimized and anomalies are removed to obtain an optimized pose and a high-precision local map. Based on the image feature data of the multi-machine interaction area, the high-precision local maps and optimized poses output by all individual machines are subjected to online collaborative optimization and offline global fusion processing to obtain a global lunar environment map that meets the mission requirements.

[0006] On the other hand, a novel lunar surface large-scale environment fusion sensing device is provided, the device comprising: The matching module is used to perform feature matching between the new lunar surface image after feature extraction and the historical lunar surface image, and to perform geometric filtering on the matching results to obtain a globally effective feature matching dataset that has completed incremental updates. The calculation module is used to perform local triangulation and pose solving on the subset of landmark points associated with the new lunar surface image based on historical camera poses, historical 3D landmark data and the global effective feature matching dataset, so as to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image. The optimization module is used to iteratively optimize and remove anomalies from the camera pose and effective punctuation subset corresponding to the lunar surface image to obtain optimized pose and high-precision local map; The fusion module is used to perform online collaborative optimization and offline global fusion processing on the high-precision local maps and optimized poses output by all individual machines based on the image feature data of the multi-machine interaction area, so as to obtain a global lunar environment map that meets the mission requirements.

[0007] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing computer programs, and the processor for executing the computer programs stored in the memory to implement the steps of the novel lunar surface large-scale environment fusion sensing method described above.

[0008] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the novel lunar surface large-scale environment fusion perception method described above.

[0009] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the novel lunar surface large-scale environment fusion perception method described above.

[0010] The technical solution provided by this invention can bring at least the following beneficial effects: First, feature extraction, feature matching, and geometric filtering are performed on the new image to obtain stable feature matching data; then, incremental pose estimation and landmark reconstruction are completed based on disjoint-set data structure and inverse depth-weighted triangulation; subsequently, local bundle adjustment and iterative outlier removal are used to significantly reduce computational overhead while ensuring accuracy; finally, unified storage with hash index, lightweight feature interaction, online collaborative optimization, and offline global fusion are used to achieve efficient integration of multi-robot maps. This solution transforms traditional offline mapping into an incremental framework that can run online, with controllable time overhead and map accuracy no less than batch processing methods; multi-robot collaboration only transmits feature data, saving lunar surface communication bandwidth; dual-mode operation balances real-time tracking and global mapping, effectively suppressing cumulative drift, and improving the robustness, efficiency, and accuracy of large-scale lunar surface exploration, making it suitable for multi-robot lunar rover exploration missions. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a novel lunar surface large-scale environment fusion perception method provided by an embodiment of the present invention; Figure 2 This is a schematic diagram showing the reprojection error distribution of the online incremental method and the offline method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram showing the reprojection error distribution of the local triangulation, BA method, and global method provided in an embodiment of the present invention; Figure 4 This is a time cost comparison diagram provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of a data fusion data framework provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0014] Please refer to Figure 1This invention provides a novel method for large-scale lunar surface environment fusion sensing, which includes: Step 100: The new lunar surface image after feature extraction is matched with the historical lunar surface image, and the matching result is geometrically filtered to obtain a globally effective feature matching dataset that has completed incremental updates. Step 102: Based on the historical camera pose, historical 3D landmark data, and the global effective feature matching dataset, perform local triangulation and pose solving on the subset of landmark points associated with the new lunar surface image to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image. Step 104: Iteratively optimize and remove anomalies from the camera pose and effective punctuation subset corresponding to the new lunar surface image to obtain the optimized pose and high-precision local map; Step 106: Based on the image feature data of the multi-machine interaction area, perform online collaborative optimization and offline global fusion processing on the high-precision local maps and optimized poses output by all single machines to obtain a global lunar environment map that meets the mission requirements.

[0015] In this embodiment of the invention, feature extraction, feature matching, and geometric filtering are first performed on the new image to obtain stable feature matching data. Then, incremental pose estimation and landmark reconstruction are completed based on disjoint-set data structure and inverse depth-weighted triangulation. Subsequently, local bundle adjustment and iterative outlier removal significantly reduce computational overhead while maintaining accuracy. Finally, unified storage using hash indexes, lightweight feature interaction, online collaborative optimization, and offline global fusion achieve efficient integration of multi-robot maps. This scheme transforms traditional offline mapping into an online incremental framework with controllable time overhead and map accuracy no less than batch processing methods. Multi-robot collaboration only transmits feature data, saving lunar surface communication bandwidth. Dual-mode operation balances real-time tracking and global mapping, effectively suppressing cumulative drift and comprehensively improving the robustness, efficiency, and accuracy of large-scale lunar surface exploration, making it suitable for multi-robot lunar rover exploration missions.

[0016] The following description Figure 1 The execution method of each step is shown.

[0017] First, for step 100, the new lunar surface image after feature extraction is matched with the historical lunar surface image, and the matching result is geometrically filtered to obtain a globally effective feature matching dataset that has completed incremental updates.

[0018] In this embodiment of the invention, the globally valid feature matching dataset is obtained in the following way: Scale-invariant feature transformation is performed on the pixel information of the newly input single-frame lunar image to obtain the image's feature points and corresponding descriptors.

[0019] Specifically, for a newly input original lunar surface image, it is first necessary to perform image distortion correction based on camera intrinsic parameters, and simultaneously convert it into a single-channel grayscale image to eliminate the interference of lens distortion on feature extraction and adapt to the input requirements of the SIFT algorithm.

[0020] Next, the SIFT algorithm is used to construct a Gaussian pyramid and a Differential Gaussian (DoG) pyramid in sequence to detect extreme points in the scale space. Low-contrast points and edge response points are removed by thresholding, and finally stable key points (feature points) with scale invariance are retained. The basic attributes of the feature points, such as pixel coordinates, scale, and orientation, are output.

[0021] Finally, a principal direction is assigned to each feature point to ensure the rotation invariance of the feature. Based on the pixel gradient information of the key point neighborhood, multidimensional high-discrimination SIFT feature descriptors are generated, ultimately forming a pair of feature point pixel coordinates-feature descriptors with one-to-one correspondence.

[0022] Based on the feature points and descriptor pairs of the image, a fast cascaded hash matching is performed on the image combination consisting of new image frames and historical image frames to obtain an initial feature matching pair data set containing all possible matching feature point pairs between the new image frame and each historical image frame.

[0023] Specifically, with new images Using this as the core, the system iterates through all historical frames in the historical frame library, generating pairwise matching pairs between new frames and each historical frame, forming a complete set of matching pairs for all frames. This ensures that new frames are matched and associated with all historical frames, without overlooking key inter-frame constraints.

[0024] Frame matching pair set

[0025] Based on matching pairs Using Fast Cascade Hashing (FCH), feature matching pairs are obtained. in, express The One component; This represents the feature pair that matches between the new frame and the kth old frame.

[0026] The initial feature matching pair dataset is subjected to iterative geometric filtering using the AC-RANSAC algorithm to obtain a new subset of feature matching pairs containing only correct matches. This new subset of feature matching pairs is then incorporated into the historical feature pair set, resulting in an updated global effective feature matching dataset.

[0027] Specifically, feature matching pairs are based on AC-RANSAC (A Contrario RANSAC). Perform geometric filtering to eliminate mismatches and obtain feature matching pairs. The feature matching pairs from the new frame are merged with the past feature matching data to update the matching data. Then, for step 102, based on the historical camera pose, historical 3D landmark data and the global effective feature matching dataset, the local triangulation and pose solving are performed on the subset of landmark points associated with the new lunar surface image to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image.

[0028] In this embodiment of the invention, the camera pose and effective markers corresponding to the crescent image are obtained in the following manner: Based on the global effective feature matching dataset, taking a single feature point of a single image as the smallest unit, all observed image frame information and image observation data of the same feature point in the global effective feature matching dataset are grouped into the same set, and a three-dimensional landmark map is built based on the set; wherein, each set corresponds to a unique three-dimensional spatial landmark point in the map.

[0029] Specifically, first, the full set of feature matching data is loaded to determine the corresponding image frame and pixel coordinates for each feature matching pair; then, the existing landmark-feature associations in the historical landmark map are loaded simultaneously to avoid redundant construction.

[0030] Next, a disjoint-set data structure is used, with a single feature point of a single image as the smallest unit. Matching feature points are grouped into the same set, and each set is defined as a landmark. It contains all the image frames observed at that landmark. and corresponding feature points Each independent set of feature points corresponds to a unique three-dimensional spatial landmark, enabling the unified aggregation of features of the same scene point across multiple frames of images.

[0031] A specific landmark can be represented as follows: in, This represents the i-th image frame. This represents the observation data of the l-th feature point in the i-th image frame.

[0032] The entire map can theoretically be represented as: .

[0033] Based on historical 3D landmark data, all landmark points in the 3D landmark map are filtered to obtain an initial subset of landmark points that are associated with the crescent image.

[0034] Specifically, based on the 3D-2D correspondence of landmarks in the map (i.e., which image features each landmark is associated with), landmarks that are associated with the new input image frame (i.e., the feature points of the new frame can match these landmarks) are selected from the entire landmark set. In other words, landmarks containing the new image are selected from the entire landmark set. The landmarks form the initial subset of landmark points associated with the new image. It clearly distinguishes between known landmarks with existing historical coordinates and newly added landmark candidates to be reconstructed, and only performs subsequent processing on this subset to achieve incremental calculation and avoid full recalculation.

[0035] Based on the historical camera poses, the landmarks in the initial landmark set that do not have 3D coordinates are triangulated to obtain the 3D spatial coordinates of the landmarks. Then, the camera pose of the crescent image is calculated based on the coordinates of each landmark in the initial landmark set. .

[0036] Specifically, based on the camera pose of each frame, the virtual camera center of the corresponding landmark point observed in that frame is calculated. Camera ray unit vector Depth along the ray

[0037] Next, triangulation is performed using the inverse depth-weighted midpoint method. This method utilizes multiple camera poses observing the same landmark point, using inverse depth as the weight, to perform a weighted summation of the ray midpoints from multiple frames of observation, thus calculating the 3D coordinates X of the landmark point. The formula is as follows: Subsequently, based on the local landmark map that already possessed three-dimensional coordinates... The problem is modeled as a P3P (Perspective-3-Point) problem by combining the new image with the corresponding two-dimensional feature points of these landmarks, and the pose of the new frame camera relative to these known three-dimensional landmarks is calculated by solving cubic equations.

[0038] For example, from the calculated landmarks, four sets of 3D-2D matching pairs with uniform spatial distribution and sufficient observation angles are selected as the core samples for P3P pose determination. This avoids pose degradation caused by coplanar or close-range samples. Based on the perspective projection model, a cubic equation system for P3P pose determination is constructed, and candidate solutions for the new frame camera pose are obtained. Through additional matching pairs for verification, the unique optimal solution that conforms to physical meaning is selected, and the rotation matrix R and translation vector t of the new frame virtual camera are obtained. These are combined to form the pose corresponding to the new image. Based on the calculated pose, 3D landmarks are projected onto the new frame image, and the reprojection error is calculated. If the overall reprojection error exceeds a preset threshold, the triangulation result is re-optimized until the pose accuracy meets the requirements, and the high-precision camera pose of the new image is output. .

[0039] Based on the historical camera pose and the camera pose of the crescent image, the coordinates of the landmark points in the initial landmark point subset are triangulated again to obtain a valid landmark point subset after removing landmarks with unstable solutions.

[0040] Specifically, the above triangulation process is repeated, and the initial subset of landmark points is subjected to secondary local triangulation by combining the new image pose with the historical image pose. Based on the results, candidate landmarks whose stable 3D coordinates cannot be calculated are removed, resulting in the final effective subset of landmark points. .

[0041] For step 104, the camera pose and effective punctuation subset corresponding to the new lunar surface image are iteratively optimized and anomaly removal is performed to obtain the optimized pose and high-precision local map.

[0042] In this embodiment of the invention, the optimized pose and high-precision local map are obtained in the following way: S31. Perform local bundle optimization on the landmark subset and the camera poses associated with the landmarks in the subset to obtain the optimized landmark subset and optimized pose after the reprojection error is reduced.

[0043] Specifically, Bundle Adjustment (BA) is a nonlinear least-squares optimization process that iteratively adjusts the coordinates of 3D landmarks and the camera pose parameters to minimize the error (i.e., reprojection error) between the reprojected locations of the landmarks on the image (predicted 2D points) and the actual observed feature point locations. The overall reprojection error of the landmark coordinates and camera pose after initial optimization is lower than before optimization.

[0044] S32. Perform abnormal observation removal processing on the optimized landmark point subset according to the preset reprojection error threshold to obtain the first landmark point subset after removal; wherein, the abnormal observation removal processing includes removing the observation data of the corresponding observation image frame if the reprojection error of the optimized landmark point in each corresponding observation image frame is greater than the error threshold. Specifically, the removed observations can be expressed as the following formula: in, For projection function, , Landmarks The camera pose and intrinsic parameters of the j-th observation. This is the set reprojection error threshold.

[0045] S33. Perform abnormal landmark removal processing on the first landmark subset according to the preset quantity threshold and the included angle threshold to obtain the second landmark subset; wherein, the abnormal landmark removal processing includes removing the first landmark if the number of remaining observation data of the first landmark is less than the quantity threshold, and removing the first landmark if the largest included angle of the first landmark is less than the included angle threshold.

[0046] Specifically, the removed landmarks can be expressed by the following formula: in, This represents the minimum number of remaining observations that need to be retained after landmark observations are removed. Used to calculate the angle between two observations of the same landmark. This represents the minimum value that the maximum observation angle within a landmark must satisfy.

[0047] S34. Determine the total number of landmark points removed from the second landmark point subset relative to the optimized landmark point subset. If the total number of removed landmark points is higher than a preset removal threshold, repeat steps S31-S34 for the second landmark point subset. If the total number of removed landmark points is lower than the removal threshold, output the optimized pose and generate a high-precision local map based on the second landmark point subset.

[0048] The final optimized and refined local map and pose are stable solutions that converge after multiple "optimization-verification-elimination" cycles, with higher accuracy and robustness, providing a high-quality foundation for subsequent steps.

[0049] For step 106, based on the image feature data of the multi-machine interaction area, online collaborative optimization and offline global fusion processing are performed on the high-precision local maps and optimized poses output by all individual machines to obtain a global lunar environment map that meets the mission requirements.

[0050] In this embodiment of the invention, the global lunar environment map is obtained in the following way: Based on hierarchical hash key values, standardized indexing and unified archiving are performed on multi-source high-precision local map data from multiple machines to obtain an interoperable multi-machine map standardized index database.

[0051] Specifically, a hash key-value pair storage format is designed for multi-source map data, and a globally unique index identifier is established for each category and each data element. Meanwhile, to reduce communication bandwidth consumption, it is stipulated that individuals can only exchange image features collected within the "interaction area". and description subfile: This results in a standardized, unambiguous data management framework, ensuring that multi-source data does not become confused during fusion.

[0052] Based on the index database, the single machine performs online mapping on the map data within the interaction area and the external feature data received from other individuals, resulting in a high-precision single-machine map optimized through multi-machine collaboration.

[0053] Specifically, for the received external feature data and the data generated by the individual within the interaction area, the external feature data is treated as a "new frame" and incorporated into a standalone online incremental SfM process (i.e., starting from feature matching, performing local triangulation, pose estimation, local BA, etc.), thereby providing more observations for shared landmarks within the interaction area and optimizing the accuracy of the local map. During this process, the data generated by the individual within the interaction area is protected to prevent it from being mistakenly removed during optimization. The final result is a standalone map with optimized and enhanced accuracy.

[0054] In an offline centralized processing scenario, all stand-alone high-precision maps are integrated based on a unified index value. The integrated data is then subjected to global 3D reconstruction and loop closure detection using the multi-trajectory correlation constraints provided by the interactive regional data, resulting in a complete and high-precision global lunar environment map.

[0055] Specifically, in offline centralized processing scenarios, map data from all individual robots is integrated based on a unified index. Then, based on a global 3D reconstruction process (such as global triangulation and global bundle adjustment (BA)), global optimization and loop closure detection are performed using the inter-trajectory correlation constraints provided by the interactive region data. This binds multiple independent trajectories into a consistent whole, resulting in a complete, consistent, and high-precision global lunar environment map. This map integrates observation information from all robots, eliminating the cumulative errors and perspective limitations of individual robot mapping, and achieving unified perception of a large-scale environment.

[0056] The feasibility of the above method is verified by a specific embodiment below: The offline batch processing SfM algorithm is broken down as follows: In each round of frame interpolation reconstruction, the data related to the new frame in the image feature extraction, feature matching, and filtering steps are separated and processed. Each processing step is an incremental process based on the previous data. Before incremental reconstruction, a 3D reconstruction is performed based on a subset of frames to obtain the basic results. The offline batch processing algorithm and the online incremental algorithm are compared below, as shown in Table 1: Table 1 Comparison of Results Indicators between Online Incremental Methods and Offline Methods We used the OpenMVG single-machine "Church" image set; for the time being, we did not introduce local triangulation and local BA, but simply compared the impact of the online incremental frame on map accuracy. The first 3 frames of the image set were set as the base frames for 3D reconstruction, and the following 8 frames were used for incremental reconstruction.

[0057] The reprojection error distributions of the two sets of experiments are as follows: Figure 2 As shown.

[0058] Based on incremental SfM, this method restricts each frame processing to reconstructing only the landmarks corresponding to the new frame. It then performs local BA optimization by combining local landmarks and known view poses, maintaining tracking mode while more frames are being inserted, and generating a global map after the new frame insertion is complete. A comparative experiment is conducted between the original global triangulation and global BA algorithms and the local triangulation and local BA algorithms.

[0059] Comparative experiment setup: Using the OpenMVG single-machine "Church" image set, the reprojection error distribution of the two sets of experiments is as follows. Figure 3 As shown.

[0060] The time cost of the two algorithms will be analyzed next, such as... Figure 4 As shown.

[0061] It can be observed that as the number of frames increases, the time difference between local triangulation, BA algorithms, and global methods generally widens. Although the global algorithm still reuses the view's pose data in each reconstruction round under the incremental framework, it is foreseeable that its reconstruction and optimization time will continue to increase as the map expands. On the other hand, the local algorithm only processes landmarks related to the new frame each time. Although the time cost may fluctuate due to the differences in the new frames, it still only processes incremental data overall, and its time advantage is very obvious.

[0062] Supported by an online incremental framework, local triangulation, and business intelligence (BA), a data framework based on hash key-value pair indexes is designed, suitable for both single-machine and multi-machine data fusion. Figure 5 As shown: In collaborative mapping scenarios, data exchange between individuals is limited to image feature data, namely key points and descriptors. This technique is based on SIFT features. Data compression experiments were conducted using a high-resolution image set (255 images) of a rock scene.

[0063] To conduct a collaborative map fusion experiment, a set of images of round-trip trajectories were captured. The turning points of the trajectories were defined as the intersection points, and the two ends of the trajectories before and after the turning points were used as independent mapping data. Differentiated designs were made in terms of trajectory length, number of sequences, and intersection points, resulting in two sets of images, r1 (208 images) and r2 (42 images). The intersection point is located at the end of r1 and the beginning of r2.

[0064] The first step was to verify whether collaborative map fusion brought improvements in online tracking mode. The program process of the collaborative mapping experimental group was stopped when all external images at intersections were fused, while ensuring that the image sequence input by the individual user in the independent mapping experimental group was consistent with that of the experimental group. The experimental results are as follows: Table 2 Results of online tracking experiment for r1 atlas collaborative mapping Number of views 208 213 Number of effective poses 208 213 Landmark Points 1538 1563 Average reprojection error / pix 0.331 0.327 Table 3. Results of online tracking experiment for r2 atlas collaborative mapping. Number of views 20 25 Effective pose number 20 25 Landmark Points 1794 1769 Average reprojection error / pix 0.212 0.206 In online tracking mode, the cooperative map fusion algorithm improved the accuracy of trajectories r1 and r2 by 1.208% and 2.830%, respectively. This indicates that cooperative map fusion can, to some extent, improve the localization and pose determination accuracy of subsequent new frames and their associated landmarks in tracking mode.

[0065] Next, we verified whether collaborative map fusion under the mapping mode brought about improvements. Based on the online tracking mode experimental group, we completed the reconstruction of all sequences, performed global mapping and optimization. The experimental results are shown in the table below: Table 4. Experimental Results of Global Mapping in R1 Atlas Collaborative Mapping Number of views 208 213 Number of effective poses 208 213 Landmark Points 16208 15247 Average reprojection error / pix 0.280 0.284 Table 5. Experimental Results of Global Mapping in R2 Atlas Collaborative Mapping Number of views 36 41 Number of effective poses 35 40 Landmark Points 2856 4219 Average reprojection error / pix 0.227 0.253 In the mapping mode, the collaborative map fusion algorithm caused a 1.429% and 11.434% reduction in accuracy for trajectories r1 and r2, respectively. For r1, this accuracy loss combined with the accuracy improvement in the tracking mode will not have a significant impact; for r2, although the accuracy loss is larger, the number of images involved is smaller than that of r1, which can still be improved by incrementally accumulating images.

[0066] Next, map fusion experiments were conducted using the reconstructed data from r1 and r2. Based on key-value pair data indexing and an interactive frame retention mechanism, image matching associations were bound between the two sets of data, and map fusion was performed based on a global 3D reconstruction method. The experimental results are as follows: Table 6. Map Fusion Experiment Results Number of views 255 Effective pose number 254 Landmark Points 17297 Average reprojection error / pix 0.277 This invention provides a novel lunar surface large-scale environmental fusion sensing device, which includes: The matching module is used to perform feature matching between the new lunar surface image after feature extraction and the historical lunar surface image, and to perform geometric filtering on the matching results to obtain a globally effective feature matching dataset that has completed incremental updates. The calculation module is used to perform local triangulation and pose solving on the subset of landmark points associated with the new lunar surface image based on historical camera poses, historical 3D landmark data and the global effective feature matching dataset, so as to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image. The optimization module is used to iteratively optimize and remove anomalies from the camera pose and effective punctuation subset corresponding to the lunar surface image to obtain optimized pose and high-precision local map; The fusion module is used to perform online collaborative optimization and offline global fusion processing on the high-precision local maps and optimized poses output by all individual machines based on the image feature data of the multi-machine interaction area, so as to obtain a global lunar environment map that meets the mission requirements.

[0067] In this embodiment of the invention, the step of performing feature matching between the feature-extracted new lunar surface image and historical lunar surface images, and performing geometric filtering on the matching results to obtain a globally effective feature matching dataset with incremental updates, includes: Scale-invariant feature transformation is performed on the pixel information of the newly input single-frame lunar surface image to obtain the image's feature points and corresponding descriptors; Based on the feature points and descriptor pairs of the image, a fast concatenated hash matching is performed on the image combination consisting of new image frames and historical image frames to obtain an initial feature matching pair data set containing all possible matching feature point pairs between the new image frame and each historical image frame; The initial feature matching pair dataset is subjected to iterative geometric filtering using the AC-RANSAC algorithm to obtain a new subset of feature matching pairs containing only correct matches. This new subset of feature matching pairs is then incorporated into the historical feature pair set, resulting in an updated global effective feature matching dataset.

[0068] In this embodiment of the invention, the step of performing local triangulation and pose solving on the subset of landmarks associated with the crescent image to obtain the camera pose and effective subset of landmarks corresponding to the crescent image includes: Based on the global effective feature matching dataset, taking a single feature point of a single image as the smallest unit, all observed image frame information and image observation data of the same feature point in the global effective feature matching dataset are grouped into the same set, and a three-dimensional landmark map is built based on the set; wherein, each set corresponds to a unique three-dimensional spatial landmark point in the map; Based on historical 3D landmark data, all landmark points in the 3D landmark map are filtered to obtain an initial subset of landmark points that are associated with the crescent image; Based on the historical camera pose, the landmarks in the initial landmark point subset that do not have three-dimensional coordinates are triangulated to obtain the three-dimensional spatial coordinates of the landmarks, and the camera pose of the crescent image is calculated based on the coordinates of each landmark in the initial landmark point subset. Based on the historical camera pose and the camera pose of the crescent image, the coordinates of the landmark points in the initial landmark point subset are triangulated again to obtain a valid landmark point subset after removing landmarks with unstable solutions.

[0069] In this embodiment of the invention, the three-dimensional spatial coordinates of the landmark are calculated using the following formula: In the formula, Represents the virtual camera center of the i-th frame image; The unit vector representing the virtual camera ray of the i-th frame image; Indicates the depth along the ray.

[0070] In this embodiment of the invention, the iterative optimization and anomaly removal of the camera pose and effective marker subset corresponding to the crescent image to obtain an optimized pose and a high-precision local map includes: S31. Perform local bundle optimization on the landmark point subset and the camera poses related to the landmark points in the subset to obtain the optimized landmark point subset and optimized pose after the reprojection error is reduced. S32. Perform abnormal observation removal processing on the optimized landmark point subset according to the preset reprojection error threshold to obtain the first landmark point subset after removal; wherein, the abnormal observation removal processing includes removing the observation data of the corresponding observation image frame if the reprojection error of the optimized landmark point in each corresponding observation image frame is greater than the error threshold. S33. Perform abnormal landmark removal processing on the first landmark point subset according to the preset quantity threshold and the included angle threshold to obtain the second landmark point subset; wherein, the abnormal landmark removal processing includes removing the first landmark point if the number of remaining observation data of the first landmark point is less than the quantity threshold, and removing the first landmark point if the largest included angle of the first landmark point is less than the included angle threshold. S34. Determine the total number of landmark points removed from the second landmark point subset relative to the optimized landmark point subset. If the total number of removed landmark points is higher than a preset removal threshold, repeat steps S31-S34 for the second landmark point subset. If the total number of removed landmark points is lower than the removal threshold, output the optimized pose and generate a high-precision local map based on the second landmark point subset.

[0071] In this embodiment of the invention, the step of performing online collaborative optimization and offline global fusion processing on all single-machine output high-precision local maps and optimized poses based on image feature data of the multi-machine interaction area to obtain a global lunar environment map that meets the mission requirements includes: Based on hierarchical hash key values, standardized indexing and unified archiving are performed on multi-source high-precision local map data from multiple machines to obtain an interoperable multi-machine map standardized index database. Based on the index database, the single machine performs online mapping on the map data within the interaction area and the external feature data received from other individuals, resulting in a high-precision single-machine map optimized through multi-machine collaboration. In an offline centralized processing scenario, all stand-alone high-precision maps are integrated based on a unified index value. The integrated data is then subjected to global 3D reconstruction and loop closure detection using the multi-trajectory correlation constraints provided by the interactive regional data, resulting in a complete and high-precision global lunar environment map.

[0072] It should be noted that the novel lunar surface large-scale environment fusion sensing device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the novel lunar surface large-scale environment fusion sensing device and the novel lunar surface large-scale environment fusion sensing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0073] Embodiments of this application also provide a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the novel lunar surface large-scale environment fusion perception method provided in the above-described method embodiments.

[0074] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the novel lunar surface large-scale environment fusion perception method provided in the above-described method embodiments.

[0075] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform any of the novel lunar surface large-scale environment fusion sensing methods described in the above embodiments.

[0076] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0077] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0078] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A novel method for large-scale lunar surface environmental fusion perception, characterized in that, The method includes: The new lunar surface image after feature extraction is matched with historical lunar surface images, and the matching results are geometrically filtered to obtain a globally effective feature matching dataset that has completed incremental updates. Based on historical camera poses, historical 3D landmark data, and the global effective feature matching dataset, local triangulation and pose solving are performed on the subset of landmark points associated with the new lunar surface image to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image. The camera pose and effective punctuation subset corresponding to the lunar surface image are iteratively optimized and anomalies are removed to obtain an optimized pose and a high-precision local map. Based on the image feature data of the multi-machine interaction area, the high-precision local maps and optimized poses output by all individual machines are subjected to online collaborative optimization and offline global fusion processing to obtain a global lunar environment map that meets the mission requirements.

2. The method as described in claim 1, characterized in that, The process involves matching the feature-extracted new lunar surface image with historical lunar surface images, and then performing geometric filtering on the matching results to obtain a globally effective feature matching dataset that has undergone incremental updates. This dataset includes: Scale-invariant feature transformation is performed on the pixel information of the newly input single-frame lunar surface image to obtain the image's feature points and corresponding descriptors; Based on the feature points and descriptor pairs of the image, a fast concatenated hash matching is performed on the image combination consisting of new image frames and historical image frames to obtain an initial feature matching pair data set containing all possible matching feature point pairs between the new image frame and each historical image frame; The initial feature matching pair dataset is subjected to iterative geometric filtering using the AC-RANSAC algorithm to obtain a new subset of feature matching pairs containing only correct matches. This new subset of feature matching pairs is then incorporated into the historical feature pair set, resulting in an updated global effective feature matching dataset.

3. The method as described in claim 1, characterized in that, The step of performing local triangulation and pose solving on the subset of landmarks associated with the new lunar surface image to obtain the camera pose and effective subset of landmarks corresponding to the new lunar surface image includes: Based on the global effective feature matching dataset, taking a single feature point of a single image as the smallest unit, all observed image frame information and image observation data of the same feature point in the global effective feature matching dataset are grouped into the same set, and a three-dimensional landmark map is built based on the set; wherein, each set corresponds to a unique three-dimensional spatial landmark point in the map; Based on historical 3D landmark data, all landmark points in the 3D landmark map are filtered to obtain an initial subset of landmark points that are associated with the crescent image; Based on the historical camera pose, the landmarks in the initial landmark point subset that do not have three-dimensional coordinates are triangulated to obtain the three-dimensional spatial coordinates of the landmarks, and the camera pose of the crescent image is calculated based on the coordinates of each landmark in the initial landmark point subset. Based on the historical camera pose and the camera pose of the crescent image, the coordinates of the landmark points in the initial landmark point subset are triangulated again to obtain a valid landmark point subset after removing landmarks with unstable solutions.

4. The method as described in claim 3, characterized in that, The three-dimensional spatial coordinates of the landmark are calculated using the following formula: In the formula, Represents the virtual camera center of the i-th frame image; The unit vector representing the virtual camera ray of the i-th frame image; Indicates the depth along the ray.

5. The method as described in claim 1, characterized in that, The iterative optimization and anomaly removal of the camera pose and effective punctuation subset corresponding to the lunar surface image to obtain the optimized pose and high-precision local map includes: S31. Perform local bundle optimization on the landmark point subset and the camera poses related to the landmark points in the subset to obtain the optimized landmark point subset and optimized pose after the reprojection error is reduced. S32. Perform abnormal observation removal processing on the optimized landmark point subset according to the preset reprojection error threshold to obtain the first landmark point subset after removal; wherein, the abnormal observation removal processing includes removing the observation data of the corresponding observation image frame if the reprojection error of the optimized landmark point in each corresponding observation image frame is greater than the error threshold. S33. Perform abnormal landmark removal processing on the first landmark point subset according to the preset quantity threshold and the included angle threshold to obtain the second landmark point subset; wherein, the abnormal landmark removal processing includes removing the first landmark point if the number of remaining observation data of the first landmark point is less than the quantity threshold, and removing the first landmark point if the largest included angle of the first landmark point is less than the included angle threshold. S34. Determine the total number of landmark points removed from the second landmark point subset relative to the optimized landmark point subset. If the total number of removed landmark points is higher than a preset removal threshold, repeat steps S31-S34 for the second landmark point subset. If the total number of removed landmark points is lower than the removal threshold, output the optimized pose and generate a high-precision local map based on the second landmark point subset.

6. The method as described in claim 2, characterized in that, The process involves online collaborative optimization and offline global fusion of high-precision local maps and optimized poses output by all individual machines, based on image feature data from the multi-machine interaction area, to obtain a global lunar environment map that meets the mission requirements, including: Based on hierarchical hash key values, standardized indexing and unified archiving are performed on multi-source high-precision local map data from multiple machines to obtain an interoperable multi-machine map standardized index database. Based on the index database, the single machine performs online mapping on the map data within the interaction area and the external feature data received from other individuals, resulting in a high-precision single-machine map optimized through multi-machine collaboration. In an offline centralized processing scenario, all stand-alone high-precision maps are integrated based on a unified index value. The integrated data is then subjected to global 3D reconstruction and loop closure detection using the multi-trajectory correlation constraints provided by the interactive regional data, resulting in a complete and high-precision global lunar environment map.

7. A novel large-scale lunar surface environment fusion sensing device, characterized in that, The device includes: The matching module is used to perform feature matching between the new lunar surface image after feature extraction and the historical lunar surface image, and to perform geometric filtering on the matching results to obtain a globally effective feature matching dataset that has completed incremental updates. The calculation module is used to perform local triangulation and pose solving on the subset of landmark points associated with the new lunar surface image based on historical camera poses, historical 3D landmark data and the global effective feature matching dataset, so as to obtain the camera pose and effective landmark subset corresponding to the new lunar surface image; The optimization module is used to iteratively optimize and remove anomalies from the camera pose and effective punctuation subset corresponding to the lunar surface image to obtain optimized pose and high-precision local map; The fusion module is used to perform online collaborative optimization and offline global fusion processing on the high-precision local maps and optimized poses output by all individual machines based on the image feature data of the multi-machine interaction area, so as to obtain a global lunar environment map that meets the mission requirements.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images