Hierarchical queriable three-dimensional scene graph construction method for star catalogue large-range detection

Abstract

The invention discloses a layered queriable three-dimensional scene graph construction method for star catalogue large-range detection, and belongs to the field of environment perception and mapping of planet exploration robots. The method aims to solve the problem that a navigation map and a scientific target perception map are mutually independent in an existing environment characterization method. The method comprises the following steps: generating an initial three-dimensional point cloud of rock based on a rock mask, downsampling the initial three-dimensional point cloud into sparse voxel representation, and constructing an elevation map of a local area around a robot in real time according to a depth point cloud; generating a binary trafficability grid map based on the real-time sensing data, and in the moving process of the robot, trying to connect with existing nodes by combining rocks and impact craters as new nodes so as to construct an incremental topological map; and then constructing an instance layer for storing instance information, a topology layer for passable road network information, a clustering layer for clustering adjacent scientific targets, a target center layer for storing the scientific targets or the center position of a target gathering area, and a planet exploration robot path layer for providing context for path planning and analysis.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception and mapping for planetary exploration robots, and relates to a three-dimensional scene mapping system and method applicable to wheeled, legged and other planetary exploration robots. Background Technology

[0002] Large-scale autonomous scientific exploration and resource discovery on planetary surfaces (such as preliminary surveys for lunar base construction) require robotic systems to possess the ability to perform long-term, accurate environmental understanding and modeling of unstructured, semantically sparse, but instance-dense planetary environments. This understanding includes not only the identification of individual scientific instances but also the formation of a comprehensive cognition and record of spatial structures, target attributes, and accessibility to support autonomous navigation and scientific mission decision-making. Existing technologies revolve around the core task of planetary robot environmental understanding and mapping, and their development reflects an evolutionary trend from "geometric modeling" to "semantic perception" and then to "task fusion."

[0003] Early work on understanding the planetary environment focused on precise geometric modeling. To ensure safe robot movement, Mars exploration missions such as Spirit and Opportunity, as well as China's Yutu-2 lunar rover, all constructed high-precision digital terrain models (DEMs) and orthophoto maps. These maps provide precise geometric information about the environment, forming the basis for path planning and localization. However, they lack semantic understanding of objects within the environment, failing to distinguish between different semantic units such as rocks and sand, and unable to interpret their scientific value.

[0004] To enhance the semantic level of environmental understanding, research has begun to integrate object recognition and terrain classification techniques. Vision-based rock and crater detection is key. Early methods relied on handcrafted features, while deep learning models subsequently achieved significant improvements in detection and segmentation accuracy. However, these methods typically output independent bounding boxes or masks, limiting information to the level of "what kind of object it is," without deep integration with map structure. Furthermore, impact crater detection primarily relies on offline analysis of orbital remote sensing data, making it difficult to apply to localized and incomplete real-time observations from a rover's perspective. In recent years, while vision-language models have demonstrated the potential for open-vocabulary description, their accuracy in specialized descriptions within geological verticals remains insufficient, making it difficult to generate structured attribute information suitable for scientific analysis.

[0005] As task complexity increases, environmental understanding is evolving towards task-driven, multimodal, and structured representations. Researchers are attempting to combine semantic perception results with maps to construct map formats that serve specific objectives. For example, NASA's risk perception map distinguishes between "sharp rocks" and "rounded rocks" through online terrain classification and integrates this information into a geometric map to plan safe routes. Furthermore, scientific hypothesis maps incorporate probabilistic models, combining scientists' prior beliefs with robot observations to optimize exploration paths for acquiring scientific information. These works mark a shift in environmental understanding from "perceiving objects" to "serving the task."

[0006] However, existing technologies have the following significant drawbacks, making it difficult to meet the needs of large-scale, long-term scientific exploration missions on the star surface:

[0007] The semantic understanding is superficial. Existing object recognition methods (such as rock segmentation) lack deep integration with semantic maps (such as risk maps). The recognition results are mostly isolated labels, failing to form an instance-level knowledge base that includes fine morphological attributes, multi-view descriptions, and spatial relationships. This causes environmental understanding to remain at the level of "there are rocks," unable to answer scientific questions such as "where are the rocks with what geological characteristics?"

[0008] The representation structure is fragmented. Geometric maps, semantic maps, and scientific hypothesis maps are usually built and maintained as independent modules, lacking a unified data structure. The topological information required for navigation planning is separated from the target attribute information that scientific tasks are concerned with, resulting in a disconnect between target retrieval and path planning, which cannot efficiently support closed-loop tasks such as "going to and re-observing a specific scientific target".

[0009] Insufficient real-time performance and adaptability. Existing unified representation methods for complex environments (such as some 3D scene maps) are mostly designed for indoor or urban environments, relying on rich prior semantic relationships and category definitions. The semantically sparse and instance-dense characteristics of the planetary environment (a large number of rocks and impact craters of various shapes) make them difficult to apply directly. In addition, indoor environments are structurally simple, and this representation method cannot preserve large-scale access structures, often only preserving path structures. This simple environmental representation loses a large number of potential accessible paths, making it unsuitable for the open environment of the planetary surface. At the same time, existing methods often have high computational and storage costs, making it difficult to achieve online, incremental construction and updates with the limited onboard resources of planetary rovers, and unable to adapt to the dynamic changes in the environment during long-distance exploration.

[0010] In summary, there is an urgent need for a new environmental understanding and representation framework that can be constructed online, hierarchically organized, deeply semantically fused, and uniformly support scientific goal understanding and autonomous navigation decision-making, in order to break through the current bottleneck of the planetary rover's cognitive and operational capabilities in complex planetary environments. Summary of the Invention

[0011] This invention addresses the problem that existing environmental characterization methods have independent navigation maps and scientific target perception maps.

[0012] A method for constructing a hierarchical queryable 3D scene graph for large-scale star catalog exploration includes the following steps:

[0013] S1. Rock depth semantic representation based on a vision-language model:

[0014] Obtain the 2D rock mask of the rock in the color image, namely the 2D bounding box and pixel-level mask of the rock; back-project the 2D pixel-level mask of the rock into 3D space to generate the initial 3D point cloud of the rock and obtain the center position of the rock, and store the center position as an abstract information of the rock instance; downsample the 3D point cloud of the rock into sparse voxel representation, and store it as the key geometric feature in the instance layer for subsequent cross-view rock ID discrimination;

[0015] Cross-frame association and fusion are performed on the same rock observed from different perspectives, and the geometric and semantic attributes of the rock are updated. For newly detected rocks, rock IDs are assigned and their geometric and semantic attributes are registered. For each rock instance, multiple close-up images from different perspectives are selected as image snapshots and input into the visual-language large model. Based on predefined geological attribute dimensions, a structured natural language description is generated, and the geological attribute information of each aspect is stored in a structured manner according to the specified response paradigm to obtain structured geological attributes. Then, the color image and natural language description are stored as semantic information in the scene graph instance layer.

[0016] S2. Real-time detection and parameter extraction of impact craters on stellar surfaces based on geometric features:

[0017] The system acquires a depth point cloud and constructs a local elevation map of the area surrounding the robot in real time. A set of contour lines is extracted from the elevation map. Impact crater outlines are determined based on the contour lines, and candidate impact crater outlines are selected using geometric criteria. Circle fitting is performed on each candidate outline to estimate its center coordinates and radius. Nested outlines from the same impact crater are clustered and merged. The identified rock location information mentioned in S1 is used to determine whether rocks are embedded within the impact crater, serving as a semantic attribute of the impact crater. For newly detected impact craters, association and fusion are performed with existing impact crater instances based on center distance and radius. Newly detected impact craters that cannot be associated and fused are registered.

[0018] S3. Incremental topology map construction for exploration:

[0019] Based on real-time sensing data, the passage cost of local areas is calculated, and a binary accessibility grid map is generated. During the robot's movement, new nodes are continuously sampled in the nearby accessible areas. After successful sampling, new nodes are attempted to connect with existing nodes. An initial graph structure is generated based on Delaunay triangulation, and the edges in the graph are organized using R-tree spatial index to support efficient querying. Each edge is stored in the R-tree with its spatial bounding box as an index item.

[0020] S4. Organization and Construction of Hierarchical Scene Diagrams:

[0021] Instance layer: As the core layer, it stores instance information corresponding to all scientific targets, including rocks and impact craters, including their unique ID, 3D location, geometric parameters, sparse voxel representation, and structured semantic description.

[0022] Topology layer: As the core layer, it stores the road network information that can be accessed within the environment, that is, it stores the global undirected and non-intersecting topology map constructed by step S3, which is represented by a graph structure and contains only points and edges;

[0023] Clustering layer: Based on the spatial distribution of instances, nearby scientific targets are clustered to form target cluster areas, thereby realizing the division and recording of the overall rock distribution;

[0024] Target Center Layer: Abstractly stores the central location of each scientific target or target cluster area, facilitating rapid retrieval and task scheduling;

[0025] Planetary exploration robot path layer: records the robot's historical paths, providing context for path planning and analysis.

[0026] Furthermore, in the process of obtaining the 2D rock mask of the rock in the image described in step S1, the 2D rock mask of the rock is obtained through a detection and segmentation model for the RGB image.

[0027] Furthermore, in step S1, during the process of back-projecting the 2D pixel-level mask of the rock into three-dimensional space, the depth information source obtained synchronously with the RGB image is combined to back-project the 2D pixel-level mask of the rock into three-dimensional space.

[0028] Furthermore, the process of cross-frame correlation of the same rock observed from different perspectives, as described in step S1, includes:

[0029] The central position of Rock A in the current frame and Rock B in the historical frame is less than the first distance threshold. Calculate the point cloud overlap between Rock A and Rock B. If there is overlap, determine that they belong to different azimuth surfaces of the same instance to achieve rock association. If there is no point cloud overlap, further calculate the convex hull property of the point cloud, that is, the convex hull volume corresponding to the point cloud of Rock A recognized in the current frame is V_A, and the convex hull volume corresponding to the point cloud of Rock B recognized in the historical frame is V_B. The convex hull volume corresponding to the point cloud set of the union of the point clouds of AB is V_{AB}. Calculate the convex hull ratio t = (V_A + V_B) / V_{AB}. If 0 < t < 1, associate AB, otherwise determine that the rock recognized in the current frame is a new rock instance.

[0030] Further, the depth point cloud obtained based on the accumulated dense depth image or the depth point cloud obtained from the accumulated lidar in step S2 is used to construct an elevation map of the local area around the robot in real time.

[0031] Further, the process of determining the impact crater contour based on the contour line and screening the candidate impact crater contours through geometric criteria in step S2 includes:

[0032] First, judge whether the impact crater contour presents a central depression feature; then calculate its roundness, rectangularity, aspect ratio and other common measurement measures for measuring the shape of a closed curve; finally, retain the candidate contours that meet all shape thresholds.

[0033] Further, when continuously sampling new nodes in the passable area near the robot during the movement of the robot in step S3, the new nodes have the following characteristics: First, the position where the node is located is unobstructed. Second, the distance between the node and the node closest to it needs to be greater than the second distance threshold to avoid dense sampling. Third, within the local area centered on the robot, the total number of nodes cannot exceed the saturation number. If the saturation number is reached, stop sampling.

[0034] Further, during the process of attempting to connect a new node to an existing node in step S3, the connection needs to meet the distance constraint, the entire path is passable and does not intersect with the existing edges.

[0035] Further, during the process of constructing the incremental topological map for exploration in step S3, it is necessary to adjust the topological map according to the instance layer and the passable area, including:

[0036] Delete the nodes that fall into the obstacle area and all the edges connected to them, and delete the edges that intersect with the area; then resample nodes in the updated passable area to maintain the graph connectivity.

[0037] Further, the method also includes the construction process of the query module, specifically including:

[0038] The CLIP model is used to encode and store close-up images of each rock in S1, thereby providing a query service. The query is achieved through "matching": the requirement to find a rock is encoded, and the CLIP model is used to align and match the features of the identified rocks and the rocks to be found in a high-dimensional vector space based on the encoding of the close-up images and the encoding of the rock search requirement. The two are matched in the vector space, and the similarity is judged based on the encoded vectors. If the dot product of the two direction vectors is close to 1, the similarity between the two is considered to be higher, so as to determine the target to be searched.

[0039] To address the query question, a large language model is introduced for comprehensive reasoning, and the corresponding geological attributes are obtained. Based on the clues left by the visual-language model for the geological attributes of the rocks, matching is performed to achieve the corresponding search.

[0040] The clustering layer is used to form instance clusters based on neighboring scientific objectives. When querying, you can directly enter a description of the instance cluster to find the corresponding instance cluster.

[0041] Beneficial effects:

[0042] This invention designs a unified hierarchical scene graph framework that organically integrates multi-dimensional information such as instance-level semantics, global topological road network, and target distribution clustering. Combined with the "exploration-oriented incremental topological map construction" process, it achieves online incremental updates and collaboration, realizing real-time linkage between perception and navigation. The detection results of scientific targets are immediately transformed into dynamic obstacle constraints in the topological map, triggering real-time updates and replanning of nodes and edges. This closes the loop of the "discovery-recording-return" autonomous exploration task flow at the data structure level, fundamentally overcoming the drawback of the disconnect between target retrieval and path planning in traditional systems. Furthermore, this invention innovatively proposes a "matching-inference" two-stage query mechanism, supporting three different levels of scientific intent understanding: distributed query, ontology query, and inference query.

[0043] This invention, through the deep integration of semantic perception, hierarchical scene representation, and lightweight navigation planning, forms a complete, efficient, and autonomous exploration and cognition system for complex planetary environments. It not only significantly improves the depth and accuracy of scientific target perception but also achieves synergy between perception, planning, and decision-making through a unified knowledge representation framework. This effectively overcomes key bottlenecks in existing technologies regarding semantic depth, structural uniformity, and real-time lightweighting, providing reliable technical support for future unmanned and intelligent planetary scientific exploration missions. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating the process of parsing multi-source sensors into multi-layer scene image layers;

[0045] Figure 2 This is a schematic diagram of the scene graph layer structure and inter-layer update methods. Detailed Implementation

[0046] This invention aims to provide a queryable, hierarchical, 3D scene graph online construction method for large-scale scientific exploration of the star surface. This method collaboratively characterizes complex star surface environments from multiple dimensions, including geometry, semantics, instances, and topology, providing a real-time, multi-dimensional environmental cognitive foundation for autonomous scientific decision-making and mission planning. Through multimodal perception fusion and structured knowledge organization, this method achieves deep semantic understanding of star surface scientific targets (such as rocks and impact craters) and topological graph representation of environmental accessibility information. The system supports semantic-based target retrieval and global path planning for autonomous return navigation tasks. This invention has the following characteristics:

[0047] 1. Layer content design: This invention includes an instance layer, an instance clustering layer, a topology network layer, and a path layer. It can be extended to more layers, such as a throughput layer, to make it richer, but its form is still a layered scene diagram.

[0048] 2. Rock instance identification method: The present invention uses a pre-trained identification model for rock identification and segmentation, which can be extended to manual identification, self-built network identification, and other methods for rock identification.

[0049] 3. Semantic extraction and retrieval method for rock instances: This invention samples a pre-trained multimodal model to extract the geological morphology of rocks, which can be extended to a vertically fine-tuned model for semantic extraction of rocks; This invention uses feature similarity judgment + large language model reasoning to screen rocks, which can be replaced by, for example, a multimodal large model for traversal screening, or other methods for screening by calculating semantic similarity.

[0050] 4. In-situ identification method for impact craters: This invention uses contour line extraction and clustering to detect small impact craters in situ, which can be extended to image recognition for impact crater detection.

[0051] 5. Incremental construction of topological maps: This invention constructs undirected and unweighted topological graphs without intersections, and can assign weights to edges according to detection preferences to make them directed or weighted topological graphs; it can also construct intersecting topological graphs to represent the environment.

[0052] The present invention will now be described in detail with reference to specific practical methods.

[0053] Specific implementation method one: Combining Figure 1 This implementation method is described below.

[0054] This implementation method is a hierarchical queryable 3D scene map construction method for large-scale star catalog exploration, specifically including the following steps:

[0055] S1. Rock depth semantic representation based on a vision-language model:

[0056] S101, Rock Detection and Segmentation:

[0057] The Grounding DINO object detection model and the Segment Anything segmentation model are concatenated to process the RGB image stream input from the camera in real time. In the star-like environment, rocks are the most prominent visible objects. The 2D rock masks of the rocks in the image are obtained, namely the 2D bounding boxes and pixel-level masks of the rocks.

[0058] S102, Three-dimensional position projection:

[0059] By combining depth information sources acquired synchronously with RGB images, such as depth information sources provided by depth images or LiDAR, the 2D pixel-level mask of the rock is back-projected into 3D space to generate an initial 3D point cloud of the rock, and ground interference is removed using a ground filtering algorithm. Then, the center position of the rock is calculated using the 3D point cloud.

[0060] The specific process of calculating the center location of a rock using 3D point cloud computing includes: generating a 3D bounding box from the 3D point cloud; determining the XYZ coordinate boundaries in the world coordinate system; and then calculating the center of the bounding box as the center location of the rock. This location is stored as an abstract type of information for the rock instance. Since dense point clouds consume significant storage resources, and this system employs sparse representation to balance efficiency and discriminability, the 3D point cloud of the rock is downsampled into a sparse voxel representation and stored as a key geometric feature in the instance layer for subsequent cross-view rock ID discrimination.

[0061] S103, Multi-view attribute fusion and cross-frame correlation:

[0062] Cross-frame association is performed on the same rock observed from different perspectives. That is, the N rocks detected in the current frame are matched with the M rocks detected in the past. If a rock in the current frame has been detected in a past frame, it is fused and registered as the same rock, and the geometric and semantic attributes of the rock are updated, that is, the rock's center position, size and other information are updated. If a rock in the current frame is determined to be a newly detected rock, its rock ID should be assigned, and its geometric (including center position, size) and semantic attributes should be registered.

[0063] The detection and association steps specifically adopt a cascaded verification method. First, a rough screening of the rocks is carried out based on the spatial distance, that is, it is quickly determined that the point clouds of two parts of the rocks with a distance exceeding 5m belong to different rocks. According to the research on the rocks on the planet's surface, in this embodiment, it is assumed that the length, width, and height of the rocks are all less than 5m. During the movement of the vehicle body, this rough screening can quickly complete the identity determination of the rocks without consuming extra computing power. However, during the vehicle's stay detection process or when the vehicle revisits a certain area, the same rock may be detected multiple times. At this time, the rocks obtained by the rough screening are finely matched and associated through voxel overlap inspection. At this time, the central positions of rock A in the current frame and rock B in the historical frame are less than 5m, and there is a certain probability that they belong to the same rock. At this time, calculate the point cloud overlap situation between rock A and rock B. If there is partial overlap between the point clouds of the two rocks, it means that they belong to the different azimuth surfaces of the same instance and are associated; if there is no overlap in the point clouds, further calculate the convex hull properties of the point clouds, that is, the convex hull volume corresponding to the point cloud of rock A identified in the current frame is V_A, and the convex hull volume corresponding to the point cloud of rock B identified in the historical frame is V_B. The convex hull volume corresponding to the point cloud set of the union of the point clouds of AB is V_{AB}, and calculate the convex hull ratio t=(V_A+V_B) / V_{AB}. If the ratio t exceeds a certain threshold, 0<t<1, then AB is associated, and it is considered that these two clusters of rock point clouds have very similar spatial characteristics. In other cases, it is determined that rock A identified in the current frame is a new rock instance.

[0064] S104. Generation of semantic description of geological features:

[0065] Select multiple perspective image close-ups, that is, image snapshots, for each rock instance. The close-ups are from the cropped results of the regions containing the close-up rocks corresponding to the RGB images obtained by the camera, and are input into the vision-language large model to be parsed into natural language descriptions. For the image close-ups, the vision-language large model generates structured natural language descriptions based on predefined geological attribute dimensions (such as rock type, grain size, color, bedding, nodules, fractures, etc.), and stores the information of various geological attributes (such as surface roughness, edge roundness, whether there is bedding or nodules, etc.) in a structured manner according to the specified answer paradigm to obtain structured geological attributes; then store the RGB image obtained by the color camera and the above natural language description (the manifestation form of the structured geological attributes) as semantic information in the instance layer of the scene graph.

[0066] To provide targeted queries for the star catalog environment, a fine-grained evidence preservation mechanism for highly similar instances on the star catalog is specifically designed: During rock identification, the system not only records single-frame detection results but also saves multiple image snapshots from various angles and their corresponding structured geological attributes (such as surface roughness, edge roundness, presence of stratification or nodules, etc.) for each rock instance through multi-view observation fusion. These structured geological attributes form a reliable basis for subsequent rock queries. For example, in this mapping phase, rock A is saved as having "rough surface, smooth edges, and no obvious nodules." When searching for "rocks that may have undergone weathering," these three geological attributes will provide effective guidance, and rock A will become a highly probable recommended instance.

[0067] S2. Real-time detection and parameter extraction of impact craters on stellar surfaces based on geometric features:

[0068] S201, Construction of local elevation map:

[0069] Due to the limited number of laser beams in some LiDAR models, the point cloud detected by a single frame is very sparse and cannot provide a dense representation of the environment. Therefore, when using LiDAR, a multi-frame accumulation method (i.e., taking the union of several frames of radar point clouds) is generally used to obtain a relatively dense point cloud. If the depth information is obtained from a depth image, there is a problem that although it is dense, the range is small. Due to the limitations of the camera's field of view, the range of depth information that the camera can provide is much smaller than that that the radar can provide. Therefore, the point cloud obtained by the depth camera also needs to be accumulated for use.

[0070] Based on the depth point cloud obtained from accumulated dense depth images or the depth point cloud obtained from accumulated LiDAR, an elevation map of the local area around the robot is constructed in real time.

[0071] S202. Contour line extraction and filtering:

[0072] A set of contour lines is extracted from the elevation map. The impact crater outline is determined based on these contour lines, and candidate impact crater outlines are screened using geometric criteria: First, it is determined whether the impact crater outline exhibits a central depression feature (the elevation of the center point is lower than the edge); then, its roundness, rectangularity, aspect ratio, and other common metrics used to measure the shape of closed curves are calculated; finally, high-confidence candidate outlines that meet all shape thresholds are retained. Specifically, considering that the contour lines inside the wall of a single impact crater should exhibit circular or elliptical shape characteristics close to circular, i.e., high roundness, low rectangularity, and an aspect ratio close to 1.0, thresholds for these three attributes are set for contour line screening.

[0073] S203, Parameter Estimation and Instance Association:

[0074] For each candidate contour, least-squares circle fitting is performed to estimate its center coordinates and radius. Nested contours from the same impact crater are clustered and merged. Using the identified rock location information mentioned in S1, it is determined whether rocks are embedded within the impact crater, which is used as a semantic attribute of the impact crater. Newly detected impact craters are associated and fused with existing impact crater instances by comparing their center distance and radius to avoid duplicate registration. Specifically, if N impact craters are detected in the current frame, and the historical detection results contain M impact crater results, it should be determined whether the N craters contain previously identified craters. If the center distance is less than a certain threshold and the radii are similar, they are considered to be the same impact crater and will not be registered again. Otherwise, they are considered new impact craters, and their attributes, such as center position, radius, and whether rocks are embedded, are initialized.

[0075] S3. Incremental topology map construction for exploration:

[0076] S301. Generation of local accessibility map:

[0077] Based on real-time sensing data (such as point clouds and images), the passage cost of local areas is calculated, and a binarized accessibility raster map is generated.

[0078] S302, Incremental Construction and Update of Probabilistic Road Maps:

[0079] In the initialization process, nodes are randomly sampled in the passable area, an initial graph structure is generated based on Delaunay triangulation, and the edges in the graph are organized using R-tree spatial index to support efficient querying. Each edge is stored in the R-tree with its spatial bounding box as the index item.

[0080] It is important to note that, unlike structured environments such as indoor spaces or urban roads where sparse but safe path frameworks like Voronoi diagrams or paths are available, the space surface environment consists of large areas of unstructured open space, with scientific targets (such as rocks and impact craters) widely distributed in seemingly "barrier-free" flat areas. If traditional topology generation methods (such as Voronoi diagrams) are used, which prioritize maximizing obstacle avoidance, they naturally tend to deviate from the center of all obstacles, leading to paths that are significantly off-center from areas with high concentrations of potential scientific targets, thus losing valuable in-situ observation opportunities. Therefore, this invention employs a topology construction strategy based on accessibility and guided by scientific coverage:

[0081] During robot movement, new nodes are continuously sampled in the nearby passable area. These new nodes possess the following characteristics: first, their location is unobstructed; second, the distance between the node and its nearest neighbor must be greater than a set distance threshold to avoid dense sampling; and third, within a certain local area centered on the robot, such as a 10m x 10m square, the total number of nodes cannot exceed a certain saturation point. If the saturation point is reached, sampling stops. After successful sampling, new nodes are attempted to connect with existing nodes to form new topological edges.

[0082] Connections must satisfy distance constraints (i.e., the distance cannot exceed a certain range to ensure uniform mapping and avoid sharp, long edges), and the path must be traversable throughout without intersecting existing edges. Since the instance layer's identification results can also provide accessibility information (i.e., the location of an instance is impassable), during the update process of the instance layer and the topology layer, rocks and impact crater boundaries detected from the instance layer are marked as additional obstacles on the traversability map. Nodes falling into obstacle areas and their associated edges are removed. Specifically, nodes falling into obstacle areas and all their connected edges are deleted. Additionally, edges intersecting the area are deleted (this occurs when a node is not within the area, but its connected edge passes through the obstacle area; these edges also need to be deleted, indicating that the edge is no longer traversable). Subsequently, nodes are resampled in the updated traversable area to maintain graph connectivity. This process not only ensures navigation safety but also achieves real-time mapping from scientific perception results to navigation constraints by preserving reachable path nodes around the scientific target, effectively supporting closed-loop scientific tasks such as "approach-observation-return."

[0083] S4. Organization and Construction of Hierarchical Scene Diagrams:

[0084] The system constructs a scene graph containing multiple functional layers, with each layer working collaboratively; a diagram illustrating the scene graph layer structure and inter-layer update methods is shown below. Figure 2 As shown, the plurality of functional layers include:

[0085] Instance layer: As the core layer, it stores instance information for all scientific targets (rocks, impact craters), including their unique ID, 3D location, geometric parameters (such as size and shape), sparse voxel representation, and structured semantic description.

[0086] When identifying objects, the stored information is concise and abstract. That is, the way instances are recorded adopts an abstract model, only recording the information needed for subsequent decision-making, including rock / impact crater locations, basic dimensions, concise semantic information, and necessary sparse voxels for determining instance IDs. This storage method is lighter than the commonly used method of storing 3D reconstruction results of the environment for indoor mapping on the ground, but it sacrifices the realistic scene restoration capabilities of 3D reconstruction. In the process of star surface exploration, research on target objects revealed that star surface exploration is still in its early stages. Identifying and understanding a large number of targets is the priority, while realistic environmental restoration is less important. Therefore, this invention's ability to retain concise information to maintain the system's operation in a wide range of star surface environments is crucial and necessary for the early exploration of star surfaces.

[0087] Topology layer: As the core layer, it stores the road network information that can be accessed in the environment, that is, it stores the global undirected and non-intersecting topology map constructed in step S3, which is represented by a graph structure; the environmental accessibility map recorded by the topology layer is a topology map, which only contains points and edges. It is relatively lightweight compared to other maps. The node set represents reachable locations, and the edge set represents the accessible paths between nodes.

[0088] Clustering layer: Based on the spatial distribution of instances, nearby scientific targets are clustered to form instance clusters. That is, the Hungarian algorithm is used to group rocks that are less than a certain distance into a cluster, thereby realizing the division and recording of the overall rock distribution.

[0089] Target Center Layer: Abstractly stores the central location of each scientific target or target cluster area, facilitating rapid retrieval and task scheduling.

[0090] Planetary exploration robot path layer: records the robot's historical paths, providing context for path planning and analysis.

[0091] S5. Query module construction:

[0092] To ensure that the 3D scene map constructed in this invention can effectively meet the needs of star catalog exploration, this invention designs a query module tailored to specific requirements. Considering that multimodal models have the ability to project data from different modalities onto the same feature space for alignment and matching, such as the CLIP model, which can align visual modality data (i.e., images) and linguistic modality data (i.e., text) in the feature space and measure similarity, the CLIP model is used to encode and store close-up images of each rock in S1. This provides a query service, which is achieved through "matching": the CLIP model is used to align and match the features of identified rocks and the rocks to be searched in a high-dimensional vector space. Based on the image encoding of snapshots of the rocks and the encoding of the desired rock, the two are matched in the vector space. For example, the vector encoded by "round rock" and the vector encoded by an image of a relatively round rock will have similar directions, i.e., the dot product of the direction vectors is close to 1, indicating a high similarity, meaning this rock is likely the target.

[0093] However, due to the limitations of the CLIP model, it cannot analyze more complex scientific requirements that require decomposition and reasoning, such as "rocks that may reflect water activity." Therefore, for the query question, it is necessary to introduce a large language model (such as GPT4) for comprehensive reasoning to decompose the corresponding geological attributes. Since the visual-language model in step S1 has already left rich clues (i.e., geological morphological clues) for the rocks during the previous rock identification process, the large language model can perform corresponding searches based on these geological attribute clues. For example, the large language model decomposes the reasoning of "water activity" into appearance information such as "rough surface" or "rounded edges," and performs small-scale matching based on the geological clues left for the rocks by the visual-language model in step S1. This method utilizes both the efficiency of CLIP and the rich prior geological knowledge and powerful reasoning ability of the large language model, effectively improving the system's ability to retrieve and recommend targets.

[0094] Furthermore, the clustering layer clusters nearby scientific targets based on the spatial distribution of instances, forming instance clusters. This clustering preserves the center and number of instances within each cluster. Queries directly input a description of the instance cluster, which then corresponds to the instance cluster. This query perspective is commonly used to understand an unfamiliar environment, i.e., macroscopic distribution queries, such as, "Where are the most rocks clustered within 100 square meters in this area?" These questions help to quickly form a basic concept of the area, which corresponds to the instance clustering layer preserved in this system. The clustering layer in S4 preserves the rock cluster distribution information within the environment.

[0095] Therefore, this approach supports three types of scientific intent understanding with varying degrees of complexity: ontology queries, distribution queries, and inference queries. Ontology queries refer to queries about the properties of instances within the instance layer of S1, such as "finding a rock with a volume greater than 5 m³." In this case, the size information stored in S103 can be used for comparison to achieve a successful retrieval. The second type is inference queries, which involve retrieval of implicit information, such as "rocks containing iron ore." This requires the large model to first resolve geological morphological clues such as "the rock is brownish-yellow" and "it exhibits a special crystal structure," and then compare these with the geological descriptions stored in S104 to select the most matching instance.

[0096] Compared to existing technologies, it has achieved significant improvements and breakthroughs in many aspects:

[0097] To address the limitations of existing rock detection methods, which suffer from coarse semantic output and inability to support deep geological attribute queries, this invention achieves refined semantic representation of scientific targets, particularly rocks, by integrating multi-view observations with a geologically fine-tuned visual-language model. The system can generate structured natural language descriptions for each rock instance, including dimensions such as rock type, grain size, color, and weathering state, overcoming the limitations of traditional methods that only output categories or masks. In dense scene testing within the high-fidelity MarsSim simulation environment, the system achieved an 88% recognition rate for 167 rocks, while the stored semantic information supports fine-grained natural language queries such as "rocks with rough surfaces, angular shapes, and surrounding debris." Regarding impact crater detection, traditional methods often rely on offline analysis of orbital images, making them ill-suited for localized and occluded observations from a rover's perspective. The online detection method proposed in this invention, based on local elevation maps and contour line geometric analysis, successfully detected 9 out of 10 impact craters in the simulation, with a center positioning error of less than 0.5 meters. It can also determine whether the craters contain rocks, an important geological attribute, thus achieving a leap from "detecting existence" to "understanding attributes".

[0098] To address the shortcomings of existing environmental representation methods, which often present fragmented approaches and independent navigation maps from scientific target perception maps, this invention designs a unified hierarchical scene graph framework. This framework organically integrates multi-dimensional information such as instance-level semantics, global topological road network, and target distribution clustering, enabling online incremental updates and collaboration. Its core innovation lies in achieving real-time linkage between perception and navigation: the detection results of scientific targets are immediately transformed into dynamic obstacle constraints in the topological map, triggering real-time updates and replanning of nodes and edges. This closes the loop of the autonomous exploration task flow of "discovery-recording-return" at the data structure level, fundamentally overcoming the drawback of the disconnect between target retrieval and path planning in traditional systems.

[0099] To address the problems of existing 3D scene mapping methods being primarily geared towards indoor or urban environments, incurring high computational and storage costs, and being difficult to run online under spaceborne resource constraints, this invention ensures the system's real-time performance and practicality through a series of lightweight and efficient designs. For example, this invention adopts an abstract approach to recording instances, recording only the information needed for subsequent decision-making, including rock / impact crater locations, basic dimensions, concise semantic information, and necessary sparse voxel point clouds for determining instance IDs, rather than the dense point cloud approach commonly used in indoor mapping. This saves significant storage overhead at the instance level while retaining crucial information, without affecting subsequent decisions and selections. Furthermore, the hierarchical approach to the scene graph ensures the system has a topology layer, keeping the map relatively lightweight. The system uses an incremental probabilistic roadmap method to construct the topology layer. During the process of adding, deleting, searching, and modifying edges and nodes, it is necessary to quickly locate the involved points and edges. If maintenance is done through traversal, the complexity will reach the level of O(|E|), where |E| is the number of edges. As the map size increases, this linear complexity will significantly slow down the mapping speed, making it difficult to meet real-time requirements. This invention introduces an R-tree spatial index structure to organize and manage the edge set in the topological graph. Specifically, each edge is stored in the R-tree using its spatial bounding box as an index. Leveraging the hierarchical spatial partitioning characteristics of the R-tree, querying, inserting, or deleting edges involving specific regions can be completed in logarithmic time. Thus, the average time complexity of graph update operations is significantly reduced from O(|E|) to O(log|E|), effectively ensuring graph building efficiency and real-time performance in large-scale environments. At the storage level, the system adopts a hierarchical sparse strategy, maintaining only a high-resolution cost map locally, while compressing the global map into a lightweight topological graph structure. In Mars simulation tests, the system can complete global path search from any starting point to the scientific target point within milliseconds, demonstrating excellent real-time performance; planning a 70m path takes only 5.1ms. Furthermore, a contour-based impact crater detection method designed specifically for the planetary environment avoids the use of computationally intensive models and exhibits good robustness to point cloud defects and occlusion, further reducing the platform's computing power requirements.

[0100] In terms of semantic querying and task support, traditional systems typically only support retrieval based on fixed categories. It's worth noting that indoor or urban environments contain a rich variety of object categories with significant appearance differences (e.g., chairs, cars, trees), and general-purpose visual-language models (such as CLIP) can already meet most query needs in these scenarios through coarse-grained semantic matching. However, the astronomical environment is drastically different: its visible objects are mainly rocks, weathered debris, and impact craters with highly similar morphologies, exhibiting a single category but diverse geological significance. For example, two rocks might be judged as "similar" in the CLIP embedding space due to their dark color and fine texture, but geologically they might represent volcanic eruption products and water-eroded sediments, respectively. Such subtle differences are crucial for scientific exploration but cannot be captured by general-purpose models. Therefore, this invention specifically designs a fine-grained evidence preservation mechanism for highly similar instances on the astronomical surface: during rock identification, the system not only records single-frame detection results but also saves multiple image snapshots from various angles and their corresponding structured geological attributes (such as surface roughness, edge roundness, presence of stratification or nodules, etc.) for each rock instance through multi-view observation fusion. These refined visual-semantic evidences form a reliable foundation for subsequent queries. This invention innovatively proposes a two-stage "matching-reasoning" query mechanism, supporting the understanding of scientific intent at three different levels of complexity: distributed queries, ontology queries, and reasoning queries. The system can handle both macroscopic distributed queries such as "finding the area with the highest rock density" and complex tasks requiring reasoning based on geological knowledge, such as "finding rocks that may have been formed by water flow." In an evaluation containing 35 random queries, the system's retrieval accuracy significantly outperformed baseline methods using only general embedding matching or image description, demonstrating its practical ability to directly serve scientific task decision-making.

[0101] In summary, this invention, through the deep integration of semantic perception, hierarchical scene representation, and lightweight navigation planning, forms a complete, efficient, and autonomous exploration and cognition system for complex planetary environments. It not only significantly improves the depth and accuracy of scientific target perception but also achieves synergy between perception, planning, and decision-making through a unified knowledge representation framework. This effectively overcomes key bottlenecks in existing technologies regarding semantic depth, structural uniformity, and real-time lightweighting, providing reliable technical support for future unmanned and intelligent planetary scientific exploration missions.

[0102] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for constructing a layered, queryable 3D scene map for large-scale exploration of star catalogs, characterized in that, Includes the following steps: S1. Rock depth semantic representation based on a vision-language model: Obtain the 2D rock mask of the rock in the color image, namely the 2D bounding box and pixel-level mask of the rock; back-project the 2D pixel-level mask of the rock into 3D space to generate the initial 3D point cloud of the rock and obtain the center position of the rock, and store the center position as an abstract information of the rock instance; downsample the 3D point cloud of the rock into sparse voxel representation, and store it as the key geometric feature in the instance layer for subsequent cross-view rock ID discrimination; Cross-frame association and fusion are performed on the same rock observed from different perspectives, and the geometric and semantic attributes of the rock are updated. For newly detected rocks, rock IDs are assigned and their geometric and semantic attributes are registered. For each rock instance, multiple close-up images from different perspectives are selected as image snapshots and input into the visual-language large model. Based on predefined geological attribute dimensions, a structured natural language description is generated, and the geological attribute information of each aspect is stored in a structured manner according to the specified response paradigm to obtain structured geological attributes. Then, the color image and natural language description are stored as semantic information in the scene graph instance layer. S2. Real-time detection and parameter extraction of impact craters on stellar surfaces based on geometric features: Obtain a depth point cloud and construct an elevation map of the local area around the robot in real time; extract a set of contour lines from the elevation map; determine the impact crater outline based on the contour lines and filter candidate impact crater outlines through geometric criteria; perform circle fitting on each candidate outline and estimate its center coordinates and radius. Nested contours from the same impact crater are clustered and merged; the rock location information identified in S1 is used to determine whether rocks are embedded in the impact crater, which is used as the semantic attribute of the impact crater; for newly detected impact craters, the center distance and radius are used to perform association and fusion with existing impact crater instances, and newly detected impact craters that cannot be associated and fused are registered. S3. Incremental topology map construction for exploration: Based on real-time sensing data, the passage cost of local areas is calculated, and a binary accessibility grid map is generated. During the robot's movement, new nodes are continuously sampled in the nearby accessible areas. After successful sampling, new nodes are attempted to connect with existing nodes. An initial graph structure is generated based on Delaunay triangulation, and the edges in the graph are organized using R-tree spatial index to support efficient querying. Each edge is stored in the R-tree with its spatial bounding box as an index item. S4. Organization and Construction of Hierarchical Scene Diagrams: Instance layer: As the core layer, it stores instance information corresponding to all scientific targets, including rocks and impact craters, including their unique ID, 3D location, geometric parameters, sparse voxel representation, and structured semantic description. Topology layer: As the core layer, it stores the road network information that can be accessed within the environment, that is, it stores the global undirected and non-intersecting topology map constructed by step S3, which is represented by a graph structure and contains only points and edges; Clustering layer: Based on the spatial distribution of instances, nearby scientific targets are clustered to form target cluster areas, thereby realizing the division and recording of the overall rock distribution; Target center layer: Abstractly stores the central positions of various scientific targets or target aggregation areas for fast retrieval and task scheduling; Planet exploration robot path layer: Records the historical paths of the robot to provide context for path planning and analysis.

2. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 1, characterized in that, In the process of obtaining the 2D rock mask of the rock in the image described in step S1, for the RGB image, the 2D rock mask of the rock is obtained through a detection and segmentation model.

3. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 2, characterized in that, In the process of back-projecting the rock 2D pixel-level mask into the three-dimensional space described in step S1, in combination with the depth information source obtained synchronously with the RGB image, the rock 2D pixel-level mask is back-projected into the three-dimensional space.

4. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 1, characterized in that, The process of cross-frame association of the same rock observed from different perspectives described in step S1 includes: The central positions of rock A in the current frame and rock B in the historical frame are less than the first distance threshold. Calculate the point cloud overlap of rock A and rock B. If there is overlap, judge that they belong to different azimuth surfaces of the same instance to achieve rock association; if there is no overlap in the point cloud, further calculate the convex hull property of the point cloud, that is, the convex hull volume corresponding to the point cloud of rock A identified in the current frame is V_A, the convex hull volume corresponding to the point cloud of rock B identified in the historical frame is V_B, and the convex hull volume corresponding to the union point cloud set of the point clouds of AB is V_{AB}. Calculate the convex hull ratio t=(V_A + V_B) / V_{AB}. If 0 < t < 1, then associate AB, otherwise determine that the rock identified in the current frame is a new rock instance.

5. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 1, characterized in that, Based on the depth point cloud obtained from the accumulated dense depth image or the depth point cloud obtained from the accumulated lidar described in step S2, a local elevation map around the robot is constructed in real time.

6. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 1, characterized in that, The process of determining the impact crater contour based on the contour line and screening candidate impact crater contours through geometric criteria described in step S2 includes: First, judge whether the impact crater contour presents a central depression feature; then calculate its roundness, rectangularity, aspect ratio and other common measure metrics for measuring the shape of a closed curve; finally, retain the candidate contours that meet all shape thresholds.

7. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 1, characterized in that, In step S3, when continuously sampling new nodes in the passable area near the robot during its movement, the new nodes have the following characteristics: First, the position where the node is located is unobstructed; second, the distance between the node and the node closest to it needs to be greater than the second distance threshold to avoid dense sampling; third, within the local area centered on the robot, the total number of nodes cannot exceed the saturation number. If the saturation number is reached, stop sampling.

8. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 7, characterized in that, In the process of attempting to connect a new node to an existing node described in step S3, the connection needs to meet the distance constraint, the entire path is passable and does not intersect with the existing edges.

9. The method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to claim 8, characterized in that, In the process of constructing an incremental topological map for exploration described in step S3, it is necessary to adjust the topological map according to the instance layer and the passable area, including: Delete the nodes that fall into the obstacle area and all the edges connected to them, and delete the edges that intersect with the area; then resample nodes in the updated passable area to maintain graph connectivity.

10. A method for constructing a layered queryable 3D scene map for large-scale star catalog exploration according to any one of claims 1 to 9, characterized in that, The method also includes the construction process of a query module, specifically including: The CLIP model is used to encode and store close-up images of each rock in S1, thereby providing a query service. The query is achieved through "matching": the requirement to find a rock is encoded, and the CLIP model is used to align and match the features of the identified rocks and the rocks to be found in a high-dimensional vector space based on the encoding of the close-up images and the encoding of the rock search requirement. The two are matched in the vector space, and the similarity is judged based on the encoded vectors. If the dot product of the two direction vectors is close to 1, the similarity between the two is considered to be higher, so as to determine the target to be searched. To address the query question, a large language model is introduced for comprehensive reasoning, and the corresponding geological attributes are obtained. Based on the clues left by the visual-language model for the geological attributes of the rocks, matching is performed to achieve the corresponding search. The clustering layer is used to form instance clusters based on neighboring scientific objectives. When querying, you can directly enter a description of the instance cluster to find the corresponding instance cluster.

Images