Target navigation method and device based on active 3DGS and visual language model reasoning

Abstract

The invention discloses a target navigation method and device based on active three-dimensional Gaussian spatter and visual language model reasoning, and the method comprises the steps: obtaining an RGB-D image and a pose in an unknown environment, and constructing an incremental three-dimensional Gaussian spatter map as persistent memory through active perception; generating an exploration map based on the constructed 3DGS map, extracting a leading edge point and carrying out space structure adaptive clustering; generating a guidance track, carrying out free viewpoint optimization based on the track, and rendering a leading edge point first-person view angle image containing rich information; constructing a structured visual prompt, combining with a thinking chain prompt, inputting a visual language model to carry out reasoning planning, and selecting an optimal navigation target; in the navigation process, a real-time target detector is used for screening potential targets, and a new view angle is rendered in a 3DGS space through an action decision VLM for target re-verification. The navigation success rate and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous robot navigation, specifically to a target navigation method and apparatus based on active 3D Gaussian Splatting (3DGS) and visual language model reasoning. Background Technology

[0002] Target navigation is a key capability in the field of embodied intelligence, requiring robots or intelligent agents to autonomously locate targets and plan paths based on perceptual information in unknown or partially known environments. This process not only relies on the agent's proactive perception capabilities but also places high demands on its spatial structure understanding and semantic reasoning abilities.

[0003] In recent years, with the rapid development of Visual Language Models (VLMs) in cross-modal understanding and general reasoning, related research has begun to explore their application in target navigation tasks to enhance agents' understanding of complex environments and high-level instructions. Existing methods typically utilize VLMs to perform semantic parsing of environmental observations, abstracting target objects or regions in the scene into textual descriptions, or generating semantic scoring maps based on current observations to assist navigation decisions. However, these methods generally represent the continuous physical environment as discrete semantic or textual units, making it difficult to fully preserve the spatial relationships and geometric information between objects in the environment, thus weakening the stability and robustness of spatial association modeling.

[0004] To address the aforementioned issues, existing research and related patents have proposed various improvement schemes. For example, patent number CN121297851A proposes an autonomous search and navigation method for mobile robots based on a large visual language model. This method mainly focuses on improving target recognition performance in the perception stage and has not fully utilized the capabilities of the large visual language model in general reasoning and high-level decision-making and planning. On the other hand, patent number CN121230734A proposes a robot navigation method and system based on a large visual language model and experience memory. By constructing a semantic map, parsing instructions to generate structured task representations, and combining historical experience for path planning and confidence assessment, it achieves autonomous navigation decisions with a certain degree of interpretability. However, in this scheme, environmental information is mainly represented in semantic hierarchy or text form. Continuous spatial structure and fine-grained environmental information are difficult to fully retain during the modeling process, making the high-level decision-making process highly dependent on the underlying semantic modules and textual prompts. This, in turn, limits navigation efficiency and the upper limit of system performance in complex scenarios. Summary of the Invention

[0005] To address the aforementioned shortcomings in the prior art, this invention provides a target navigation method and apparatus based on active 3DGS and visual language model reasoning.

[0006] To achieve the above objectives, the first aspect of the present invention provides the following technical solution: The target navigation method based on active 3DGS and visual language model inference includes the following steps: S1: Acquire RGB-D images and poses in an unknown environment, and construct an incremental 3D Gaussian splash (3DGS) map as a persistent memory through active perception; S2: Generate an exploration map based on the constructed 3DGS map, extract frontier points and perform adaptive spatial structure clustering; S3: Generate a guidance trajectory and perform free viewpoint optimization based on the trajectory to render a first-person perspective (FPV) image of the leading edge point containing rich information. S4: Construct structured visual cues, combine them with Chain-of-Thought (CoT) cues, input them into the visual language model for reasoning and planning, and select the best navigation target; S5: During navigation, a real-time target detector is used to filter potential targets, and a new perspective is rendered in 3DGS space through motion decision VLM to re-verify the target.

[0007] Further, step S1 includes the following steps: S11: Obtain the agent's current RGB image, depth map, and world-to-camera pose transformation. ; S12: Constructing a 3D map composed of anisotropic Gaussian primitives: S121: Each Gaussian element is defined by position, opacity, color, and covariance matrix. (1) Each Gaussian is determined by its position in the world coordinate system. 3D covariance matrix ,transparency And the first-order spherical harmonic function representing the color of each Gaussian sphere. Composition. Each Gaussian first-order spherical harmonic function has a total of 12 coefficients. By rotation matrix and scale matrix composition: (2) It should be noted that directly optimizing the covariance matrix does not guarantee that it is a positive definite matrix, so quaternions are used to replace the rotation matrix. , Represented by a 3D scale vector.

[0008] S122: Obtain camera data via front-end visual odometry 3D Gaussian coordinates in the world coordinate system Transformed to the corresponding two-dimensional plane using the projection matrix. (3) Where the covariance matrix Jacobian matrix approximated by affine Projected onto a plane: (4) in It is a projection function. yes The rotation matrix in the matrix.

[0009] S123: After obtaining the projected 3D Gaussian points, sort them according to their distance from the front and back of the canvas, and then efficiently obtain the pixel color values ​​through the alpha blending method: (5) S124: Set color value Replace with depth value This allows you to obtain the rendered depth value: (6) To pass the spherical harmonic coefficient The color obtained from the Gaussian function, Two-dimensional Gaussian equation and transparency Density values ​​of the product: (7) Three-dimensional Gaussian center Splash to planar pixels , Given the camera intrinsic parameter matrix, Let be the transformation matrix from world coordinates to camera coordinates in the k-th frame. This represents the z-axis distance of the corresponding 3D point, i.e., the depth.

[0010] S125: Calculate the opacity field : (8) S13: Execute an active perception strategy to render a panoramic opacity field in a virtual camera to estimate observation integrity; S131: Utilizing the flexibility of virtual cameras in adjusting intrinsic parameters, adaptive configuration of the observation angle is achieved by setting the horizontal field of view of the camera to hFoV and the vertical field of view to vFoV.

[0011] S132: To achieve panoramic coverage of the environment, centered on the robot, according to... The number of images is uniformly rendered across multiple viewpoints in the horizontal direction. The resolution of each rendered viewpoint is set to W×H pixels, and the corresponding camera intrinsic parameter matrix is ​​calculated based on the aforementioned field of view parameters. (9) Panoramic Opacity Field This is achieved by stitching together the images from different viewpoints rendered in the above steps in the horizontal direction; S133: Using the DBSCAN algorithm Clustering of low-opacity regions in the panoramic opacity field This allows for the identification of representative discrete observation target perspectives within a continuous space. S134: Since the roll angle does not affect the observation process, the target viewpoint is based on the region with the largest area of ​​low opacity. Calculations were performed, and the pitch angle was used. With yaw angle Perform parameterization: (10) in To measure the threshold of low opacity. Then, using the optimal pitch angle. With yaw angle The system guides the real camera to adjust its posture, enabling it to actively cover areas lacking visual information and gradually build up complete information about the surrounding environment.

[0012] Further, step S2 includes the following steps: S21: Based on 3DGS rendering and agent height information, construct an exploration map from a top view, dividing the area into explored, unexplored, and obstacle areas; S22: Extract frontier points from the exploration map and refine their locations based on Chebyshev distance; S23: Construct a distance field based on the exploration map and detect local maxima to identify core skeleton points. Use the skeleton points as seeds to cluster the frontier points using the watershed algorithm, and select the centroids of each category as representative frontier points.

[0013] Further, step S3 includes the following steps: S31: Constructing the guide trajectory cost function The optimal guidance trajectory to each frontier point is generated based on the improved Dijkstra algorithm. : (11) (12) in, Step size, Distance to the obstacle, The relevant nonlinear penalty term, As a safety threshold, It's the weight.

[0014] S32: Curvature-based and distance The maximization strategy initializes the virtual viewpoint position on the guided trajectory. : (13) in, It is a weighted threshold that measures curvature and distance. Based on the position of the virtual viewpoint and combined with its directional information toward the leading edge point, the initial pose is finally determined.

[0015] S33: Constructing a composite loss function By optimizing the camera rotation quaternion Translation vector Determine the optimal virtual viewpoint: (14) in, This is used to balance the observed and unobserved areas due to opacity loss. To shield against radiation damage and ensure a direct line of sight to the front line; To compensate for the loss in viewpoint alignment, ensure the camera is facing the leading edge point; To mitigate trajectory loss, camera motion is constrained, causing the camera to move near and maintain close proximity to the guide trajectory. , , , This is a hyperparameter.

[0016] S331: The specific optimization loss is: (15) in, This is for averaging operations.

[0017] S332: The specific optimization loss is: (16) For the sigmoid function, To render the sampling depth on the depth map, This represents the depth value corresponding to the leading edge point.

[0018] S333: The specific optimization loss is: (17) The angle between the camera's forward vector and the ray from the camera's optical center to the leading edge is denoted as .

[0019] S334: The specific optimization loss is: (18) in, For the trajectory point, This is a smoothing factor. The weighting factor is adaptively adjusted based on the opacity loss, and the specific calculation method is as follows: (19) in, Need to be from It was separated from gradient calculation.

[0020] S34: Based on steps S32 and S33, the optimal virtual viewpoint pose can be obtained. The first-view (FPV) image of each observable frontier point is rendered using the optimal virtual view.

[0021] Further, step S4 includes the following steps: S41: Render the FPV of the leading edge point in the 3DGS map using the optimized viewpoint in step S3, and generate the corresponding bird's-eye view (BEV). S42: Overlay structured visual annotations on FPV and BEV, including: marking unobserved areas in FPV with a specific color (purple) and marking gaze points with red dots; drawing historical trajectories (blue), spatial distribution of leading edge points corresponding to FPV (red dots), guiding trajectories connecting leading edge points (green), and current position (purple) on BEV. S43: Input the annotated FPV, BEV, and Chain-of-Thought (CoT) text prompts into the VLM. The VLM analyzes the spatial layout and semantic information and outputs the next navigation target point.

[0022] Further, step S5 includes the following steps: S51: Use a real-time target detector to detect target objects in the environment; S52: When the detection result is blurry, trigger VLM to make action decisions and select an action from the action space (forward, backward, left turn, right turn, etc.); S53: Project the selected action into the 3D Gaussian Splash (3DGS) space, render a new perspective image of the corresponding target, and input the image back into the visual language model for target confirmation; if the target object required in the instruction is confirmed to exist, guide the robot to approach the target object and end the navigation process; otherwise, continue to perform exploration navigation.

[0023] A second aspect of the present invention relates to a target navigation system based on active 3DGS and visual language model inference, the system comprising the following modules: Perception and mapping module: used to acquire multimodal perception data of the environment and construct a three-dimensional Gaussian splash (3DGS) environmental representation based on an active perception strategy; Frontier processing module: used to generate exploration maps, extract and cluster the frontier regions at the boundary between known and unknown spaces to obtain representative frontier points; Structured visual cueing module: used to generate guidance trajectories based on leading edge points, calculate free viewpoints through multi-constraint optimization, and then render information-enhanced observation images; Reasoning module: used to drive the visual language model to perform path planning and decision-making based on the structured visual cues and the chain-thinking CoT mechanism; Re-verification module: Used to combine real-time target detection results with 3DGS new perspective rendering to perform secondary confirmation of potential targets from different angles.

[0024] A third aspect of the invention relates to a target navigation device based on active three-dimensional Gaussian splashing and visual language model reasoning, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method described in the invention.

[0025] A fourth aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the method described in the invention.

[0026] This invention utilizes an active perception strategy to construct a continuous 3DGS environment representation and proposes a trajectory-guided free-view rendering method to provide the Virtual Machine (VLM) with a first-person view image that is both visually reasonable and information-dense. Combined with Chain-of-Thought (CoT) cues, it enables high-level decision-making and reasoning. During the navigation execution phase, this invention introduces a high real-time target detector to quickly filter potential targets and reconfirms candidate targets through active viewpoint switching driven by the VLM, thereby achieving efficient and reliable target recognition. System evaluations were conducted on multiple standard benchmarks and a real quadruped robot platform. Experimental results show that this invention improves the success rate by at least 13.2% and the navigation efficiency by 31.9% compared to the existing best methods, validating its effectiveness and robustness in both simulated and real-world environments.

[0027] The innovation of this invention is: 1. Enhanced spatial reasoning capabilities: By using 3DGS as the persistent memory of VLM, replacing the traditional semantic abstract map, the continuous geometric structure and texture details of the environment are preserved, enabling VLM to perform deeper spatial reasoning.

[0028] 2. Active perception and viewpoint optimization: The introduction of opacity-based active perception and trajectory-guided free viewpoint optimization solves the problems of line-of-sight occlusion and information loss caused by passive observation, providing VLM with high-quality and information-rich first-person observation.

[0029] 3. Efficient re-verification mechanism: Utilizing the new perspective synthesis capabilities of 3DGS, recall-based target re-verification can be performed without moving the physical robot, significantly reducing exploration costs and improving recognition accuracy.

[0030] 4. Structured visual cues: By explicitly labeling historical trajectories, fixation points, and unexplored areas in the observed images, VLM is effectively guided to focus on key spatial information, improving the robustness of long-term planning.

[0031] Compared with existing technologies, the advantages of this invention are: strong spatial reasoning ability, preservation of continuous geometric structure and texture details of the environment, enabling VLM to perform deeper spatial reasoning; providing VLM with high-quality, information-rich first-person observations; reducing exploration costs and improving recognition accuracy; effectively guiding VLM to focus on key spatial information, and improving the robustness of long-term planning. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is the experimental platform used in this invention; Figure 3This is a diagram of the navigation process of the present invention in a real environment, which includes example diagrams of structured visual cues (FPV and BEV annotations) and VLM reasoning self-explanation; Figure 4 It is a flowchart illustrating the reasoning process for re-verification. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the described embodiments are only for explaining the invention and are not intended to limit the invention.

[0034] Example 1

[0035] This invention relates to a target navigation method based on active 3DGS and visual language model inference, comprising the following steps: S1: Acquire RGB-D images and poses in an unknown environment, and construct an incremental 3D Gaussian splash (3DGS) map as a persistent memory through active perception; S2: Generate an exploration map based on the constructed 3DGS map, extract frontier points and perform adaptive spatial structure clustering; S3: Generate a guidance trajectory and perform free viewpoint optimization based on the trajectory to render a first-person perspective (FPV) image of the leading edge point containing rich information. S4: Construct structured visual cues, combine them with Chain-of-Thought (CoT) cues, input them into the visual language model for reasoning and planning, and select the best navigation target; S5: During navigation, a real-time target detector is used to filter potential targets, and a new perspective is rendered in 3DGS space through motion decision VLM to re-verify the target.

[0036] Further, step S1 includes the following steps: S11: Obtain the agent's current RGB image, depth map, and world-to-camera pose transformation. ; S12: Constructing a 3D map composed of anisotropic Gaussian primitives: S121: Each Gaussian element is defined by its position, opacity, color, and covariance matrix. (1) Each Gaussian is determined by its position in the world coordinate system. 3D covariance matrix ,transparency And the first-order spherical harmonic function representing the color of each Gaussian sphere. Composition. Each Gaussian first-order spherical harmonic function has a total of 12 coefficients. By rotation matrix and scale matrix composition: (2) It should be noted that directly optimizing the covariance matrix does not guarantee that it is a positive definite matrix, so quaternions are used to replace the rotation matrix. , Represented by a 3D scale vector.

[0037] S122: Obtain camera data via front-end visual odometry 3D Gaussian coordinates in the world coordinate system Transformed to the corresponding two-dimensional plane using the projection matrix. (3) Where the covariance matrix Jacobian matrix approximated by affine Projected onto a plane: (4) in It is a projection function. yes The rotation matrix in the matrix.

[0038] S123: After obtaining the projected 3D Gaussian points, sort them according to their distance from the front and back of the canvas, and then efficiently obtain the pixel color values ​​through the alpha blending method: (5) S124: Set color value Replace with depth value This allows you to obtain the rendered depth value: (6) To pass the spherical harmonic coefficient The color obtained from the Gaussian function, Two-dimensional Gaussian equation and transparency Density values ​​of the product: (7) Three-dimensional Gaussian center Splash to planar pixels , Given the camera intrinsic parameter matrix, Let be the transformation matrix from world coordinates to camera coordinates in the k-th frame. This represents the z-axis distance of the corresponding 3D point, i.e., the depth.

[0039] S125: Calculate the opacity field : (8) S13: Execute an active perception strategy to render a panoramic opacity field in a virtual camera to estimate observation integrity; S131: By utilizing the flexibility of the virtual camera in adjusting intrinsic parameters, the camera's horizontal field of view is set to hFoV=120° and its vertical field of view is set to vFoV=150°, thus achieving adaptive configuration of the observation angle.

[0040] S132: To achieve panoramic coverage of the environment, centered on the robot, according to... The number of images is uniformly rendered across multiple viewpoints in the horizontal direction. The resolution of each rendered viewpoint is set to 120×150 pixels, and the corresponding camera intrinsic parameter matrix is ​​calculated based on the aforementioned field-of-view parameters. (9) Panoramic Opacity Field This is achieved by stitching together the images from different viewpoints rendered in the above steps in the horizontal direction; S133: Using the DBSCAN algorithm Clustering of low-opacity regions in the panoramic opacity field This allows for the identification of representative discrete observation target perspectives within a continuous space. S134: Since the roll angle does not affect the observation process, the target viewpoint is based on the region with the largest area of ​​low opacity. Calculations were performed, and the pitch angle was used. With yaw angle Perform parameterization: (10) in =0.3 is the threshold for measuring low opacity. Then, the optimal pitch angle is used... With yaw angle The system guides the real camera to adjust its posture, enabling it to actively cover areas lacking visual information and gradually build up complete information about the surrounding environment.

[0041] Further, step S2 includes the following steps: S21: Based on 3DGS rendering and agent height information, construct an exploration map from a top view, dividing the area into explored, unexplored, and obstacle areas; S22: Extract frontier points from the exploration map and refine their locations based on Chebyshev distance; S23: Construct a distance field based on the exploration map and detect local maxima to identify core skeleton points. Use the skeleton points as seeds to cluster the frontier points using the watershed algorithm, and select the centroids of each category as representative frontier points.

[0042] Further, step S3 includes the following steps: S31: Constructing the guide trajectory cost function The optimal guidance trajectory to each frontier point is generated based on the improved Dijkstra algorithm. : (11) (12) in, =4 is the step size. Distance to the obstacle, The relevant nonlinear penalty term, =10 is the safety threshold. =5 is the weight.

[0043] S32: Curvature-based and distance The maximization strategy initializes the virtual viewpoint position on the guided trajectory. : (13) in, =0.7 is the weighted threshold for measuring curvature and distance. Based on the position of the virtual viewpoint and combined with its directional information toward the leading edge point, the initial pose is finally determined.

[0044] S33: Constructing a composite loss function By optimizing the camera rotation quaternion Translation vector Determine the optimal virtual viewpoint: (14) in, This is used to balance the observed and unobserved areas due to opacity loss. To shield against radiation damage and ensure a direct line of sight to the front line; To compensate for the loss in viewpoint alignment, ensure the camera is facing the leading edge point; To mitigate trajectory loss, camera motion is constrained, causing the camera to move near and maintain close proximity to the guide trajectory. , , , As a hyperparameter, this invention sets it to... =0.01, =1.0, =0.01, =0.1.

[0045] S331: The specific optimization loss is: (15) in, This is for averaging operations.

[0046] S332: The specific optimization loss is: (16) For the sigmoid function, To render the sampling depth on the depth map, This represents the depth value corresponding to the leading edge point.

[0047] S333: The specific optimization loss is: (17) The angle between the camera's forward vector and the ray from the camera's optical center to the leading edge is denoted as .

[0048] S334: The specific optimization loss is: (18) in, For the trajectory point, =5 is the smoothing factor. The weighting factor is adaptively adjusted based on the opacity loss, and the specific calculation method is as follows: (19) in, Need to be from It was separated from gradient calculation.

[0049] S34: Based on steps S32 and S33, the optimal virtual viewpoint pose can be obtained. The first-view (FPV) image of each observable frontier point is rendered using the optimal virtual view.

[0050] Further, step S4 includes the following steps: S41: Render the FPV of the leading edge point in the 3DGS map using the optimized viewpoint in step S3, and generate the corresponding bird's-eye view (BEV). S42: Overlay structured visual annotations on FPV and BEV, including: marking unobserved areas in FPV with a specific color (purple) and marking gaze points with red dots; drawing historical trajectories (blue), spatial distribution of leading edge points corresponding to FPV (red dots), guiding trajectories connecting leading edge points (green), and current position (purple) on BEV. S43: Input the annotated FPV, BEV and Chain-of-Thought (CoT) text prompts into the VLM. The VLM used in this invention is Gemini3. The VLM analyzes the spatial layout and semantic information and outputs the next navigation target point.

[0051] S431: Furthermore, the specific CoT text prompts are as follows: You are a 3D indoor exploration assistant. The sensor used is a single camera. The current map is incomplete, only showing explored areas. The target object may exist in unexplored areas.

[0052] The input is a stitched image: The BEV map on the left (approximately 640×640) is a top-down projection showing the actual environmental colors of the explored areas. The map expands as the robot moves. Unexplored areas are blank.

[0053] The FPV image on the right: the first-person view of each leading edge point in the BEV.

[0054] Your task is to use BEV and FPV to rationally plan an efficient exploration sequence based on spatial layout, visual cues, and common sense. Gradually select the most reasonable exploration frontier point for the next step, expand the map, and gradually approach the target object to ultimately find the target object via the shortest path.

[0055] Mandatory rules

[0056] All rules must be subordinate to the efficiency of path planning and the maximum probability of target discovery.

[0057] 0. (Required) When selecting the furthest point, it is essential to consider whether the loss of surrounding information along the way is reasonable. Due to camera perspective limitations, information about the surrounding environment can only be obtained by reaching the corresponding point. Therefore, heading directly to a distant point will skip nearby leading points along the way, making it impossible to obtain the actual environment of the purple area in the FPV corresponding to the leading point, resulting in information loss. The priority of leading points should be based on a comprehensive evaluation of visible cues, including: whether the area of ​​the purple area is appropriate (too large an area indicates high uncertainty, and detailed information about distant points can be obtained by exploring nearby points; too small an area or a closed structure has limited potential), whether it is a local corner, and the overall structural characteristics.

[0058] 1. Each time, all frontier points must be analyzed, and finally, one frontier point must be selected for exploration. Only the single-point analysis and the final result need to be output; the previous steps do not output anything. Note: Internal analysis is only performed when the number of frontier points exceeds 6; only the score and the final selection are output.

[0059] 2. Room type must be based on at least one identifiable and clear object or structural cue (such as bed, refrigerator, cabinet, sink, television, sofa, doorway orientation). Inferences based solely on material, color, shadow, or local texture are not acceptable. A room type match score of 0 will be given if evidence is insufficient, and the result will be marked "Uncertain." Visual identification criteria must be described objectively.

[0060] 3. Output summary information (final order, next exploration suggestions, possible areas where the target object may exist) is only output once after all frontline point analyses are completed.

[0061] 4. The corridor is structurally extended by default and cannot be regarded as a dead end. The leading edge of the corridor connects to other rooms.

[0062] 5. The sequential path between the leading edges planned based on BEV analysis must be the shortest. It is crucial to carefully distinguish between the blue trajectories (only one connecting the purple triangle) and the green trajectories (multiple trajectories starting from the purple triangle and ending at the red leading edge point) in the BEV analysis; mixing them is strictly prohibited. Backtracking routes should be avoided as much as possible.

[0063] 6. The possible location of the target object needs to be inferred based on the analysis results of the room types and front point that have been observed, rather than guessing out of thin air, unless there are no obvious clues.

[0064] 7. When clues are similar, prioritize the closer leading point to obtain more environmental information before making the next decision.

[0065] 8. Red Dot Guiding Intent Focusing Principles and Exploration Priorities in FPV: When an FPV displays multiple doors or channels, the analysis must strictly focus on the environment near the red dot. If the red dot is located at or inside a doorway, the analysis should absolutely and exclusively focus on the interior space led to by that doorway, especially the purple area (unknown). Visual information from outside the doorway, already explored areas (such as views of adjacent rooms), should not be used as the primary semantic or exploration intent basis for that leading point.

[0066] 9. Areas not covered by the blue trajectory in BEV cannot be considered dead ends or areas that cannot be expanded.

[0067] Input instructions

[0068] BEV Map: Red dot = leading edge point, the top right corner of the red dot is the ID; Purple triangle = robot position (connecting the blue trajectory is the back); Green line = robot direction guidance to the leading edge point (unvisited, indicating the path that can be taken to the corresponding leading edge point); Blue trajectory = explored (already passed).

[0069] FPV image: Top left red number = leading edge point ID; top right green number = distance; red dot = robot's final location (may be plotted on the wall and analyzed in conjunction with BEV); purple area = unknown (more purple areas indicate greater uncertainty, fewer purple areas indicate less exploration potential).

[0070] Semantic information of the target object: The target object is usually associated with a specific room type (refrigerator → kitchen, sofa → living room, bed → bedroom).

[0071] The target object exists only on this current floor. It does not exist on floors above or below.

[0072] Reasoning process

[0073] Step 1 — FPV Local Spatial Analysis

[0074] Observe the FPV environment: structural cues, object cues, and spatial morphology.

[0075] In the BEV (Build-Effective Vehicle) assessment of FPV (Front-Field Measurement), red dot locations are used: at the end of a corridor, at a doorway, beside a bed, next to a sofa, and in unknown areas. A red dot at a doorway indicates that a new space can be entered from that point.

[0076] Red Dot Intent Analysis: Analyze which room or area the red dot leads to based on multiple doorways. If the red dot is located at or inside a doorway, its exploration intent is to enter the space behind that door. In this case, infer the room type solely based on visible clues inside the door (or behind the red dot), and do not be distracted by other information outside the door. For example: if a bed is visible through one door on the left, and a refrigerator through another, but the red dot is in the door with the refrigerator, then the red dot leads to the kitchen, not the bedroom. Purple areas represent unknown areas that may have high exploration value and can be inferred appropriately.

[0077] Visual cues must be clearly defined (color, shape, material, function, spatial context). Vague terms such as light and shadow should not be used to describe visual cues; descriptions must be objective and precise.

[0078] Do not fabricate the existence of target objects. Observe the size relationships between objects. For example, if you want to find a chair, the chair should be about the same height as the sofa, so you don't need to look for the target in a corner.

[0079] Note that determining whether a region is closed in FPV must be done in conjunction with BEV analysis. If the floor color and shape at the leading edge point abruptly break (i.e., the edge is an unexplored blank area), then there is a new channel that is not a closed region. If the color and shape are intact, then it is a closed region.

[0080] Step 2 — BEV Spatial Layout Analysis

[0081] Based on BEV analysis, the spatial location of the frontier point corresponding to FPV and the explorable space are determined.

[0082] The green trajectory analysis reveals the relationship between the leading edge point and the robot's position, as well as the path. It analyzes the path efficiency between the trajectory reaching the leading edge point and the sequentially planned leading edge points, avoiding continuous backtracking.

[0083] Analyze the trend of the blue trajectory line to infer where the robot came from and whether it passed near the leading edge point.

[0084] Step 3 — Matching Room Type with Target Semantics

[0085] 1. Determine which functional area or space the red dot predicted in steps 1–2 is located in.

[0086] 2. Match between room type and target object: Matching is performed by combining the semantic information of functional areas with target objects. For example, bed: bedroom, television: living room, etc.

[0087] When room type is not definitively established, one should remain conservative in their assumptions and avoid over-speculation.

[0088] Step 4 — Frontier Planning

[0089] Points that have already been reached or passed through are set to the lowest priority.

[0090] The order of exploration is planned based on a combination of factors, including distance, room type compatibility, and exploration potential score.

[0091] The room type matches the area where the target object is likely to be located with a high degree of similarity, and the robot is not far from the front point, which increases the exploration potential score.

[0092] Corridor / Waypoint Assessment: Leading points in corridors or passageways, especially those located at the entrance to new rooms, are intended for exploration into new areas. If their FPV shows a door, opening, or corner (exploration potential score should be 2 points or higher), or if not, exploration potential score should be 1 point or higher, their priority should be increased and should not be reduced simply because they are located on a path leading to the end of a corridor.

[0093] Prioritize rooms based on common sense considerations regarding their connectivity (e.g., kitchen and dining room connected, bedroom and walk-in closet connected).

[0094] Scoring Rules

[0095] Indicator range description

[0096] A. Room type matching score: -2 to +4. The degree of matching between room type and target object (0 if there is insufficient information or weak inference; 2 or more if it matches the area where the inferred target object may exist (kitchen is not the same as the corridor near the kitchen). The more black lines, the higher the uncertainty and the lower the accuracy of the room type inference.

[0097] B. Exploration Potential Score: 0-4. The probability that a red dot leads to an unexplored area. This is based on: new openings / corners, passageways, corridors visible in the FPV, the extent of the purple area in the FPV, and whether the point is located at the edge of known space. The score should not be reduced simply because the point is a "path midpoint."

[0098] C. Blue trajectory overlap -2~0: Has the leading edge point been reached or passed through (-2 for reached)?

[0099] Output template

[0100] When the number of leading edge points exceeds 6: Frontier Point ID: [ID] Final Score: [Calculation Process] When the number of leading edge points is less than 6: Single frontier point analysis (output point by point; when the number of points exceeds 6, only the score is output). Frontier Point ID: [ID] Predicted room type: [Right dot indicates room type (only one possible) / Uncertain] Analysis: [Red dot location description: [Local spatial location]; Red dot intent: [Intent analysis]; Has it been reached: [Yes / No]; FPV environment information, BEV layout, and distance between the robot point and the leading edge, exploration potential description. The use of "coverage" or "transition node" to lower the score is strictly prohibited. Explain the basis for visual recognition (within 20 characters)] Final Score: [Calculation process (scores for each item must be output)] Conclusion: [A Brief Overview of Information Sufficiency and Exploration Priorities] Summary output (after all frontier point analyses are completed) Final order: [ID sequence, sorted by exploration priority and shortest path] Next step exploration suggestion: [Highest scoring ID], reason: [Greatest exploration potential, highest probability of finding the target, unexplored frontier, leading to unknown territory] Inferred areas where the target object may exist: [at least 3 functional zones] (inferred based on the analysis of observed room types and frontal points, not guesswork, unless there are no obvious clues) After evaluating all frontier points, choose exactly one explorationpoint that is most likely to lead toward the target object, based on your reasoning above (including both BEV and FPV analyses). Output one strict JSON line for the final choice, following thisformat exactly: json { Final Choice:<ID of the exploration point most likely to lead tothe target object> } ``` Further, step S5 includes the following steps: S51: Use a real-time target detector to detect target objects in the environment; S52: When the detection result is blurry, trigger the VLM to make an action decision (the action decision VLM used in this invention is GLM-4.5v), and select an action from the action space (forward, backward, left turn, right turn, etc.); S521: Furthermore, the cue words for the action decision VLM are: This is an image of the robot detecting a target. I want to change the perspective to further check if this is what I'm looking for. What should my next move be? Action options are: 1. Move forward (zooms in on the object in the center of the image), 2. Move backward (zooms out on the object in the center of the image to see more of the field of view), 3. Turn right, 4. Turn left, 5. Leave. If the target object is detected, select it. If you believe the target object does not exist in the current environment, select "Leave".

[0101] Note: Please determine whether the image is the target based on the existing image. If not, please select option 5. Leave. If it is, please output the coordinates [xmin, ymin, xmax, ymax].

[0102] Mandatory rules: A chair is a chair, a sofa is a sofa, a gym chair is not a chair, and a toy plant is not a plant; they should not be confused.

[0103] Do not immediately leave if there are similar objects; choose your actions based on the objects' locations.

[0104] Tell me your reasons, and finally output your choice. It must be in the following format:

[0105] {

[0106] "Ans": Action space numbers(1-5) or [xmin,ymin,xmax,ymax] if istarget detected.

[0107] }

[0108] S53: Project the selected action into the 3DGS space, render a new perspective image of the corresponding target, and input the image back into the visual language model for target confirmation; if the target object required in the instruction is confirmed to exist, guide the robot to approach the target object and end the navigation process; otherwise, continue to perform exploration navigation.

[0109] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is the experimental platform used in this invention; Figure 3 This is an example diagram demonstrating the navigation process of the present invention in a real environment, including structured visual cues (FPV and BEV annotations) and VLM reasoning self-explanation; Figure 4 It demonstrates the reasoning process for re-verification; As attached Figure 2 As shown, the experimental platform used in this invention further utilizes the Go2 robotic dog as a mobile platform, equipped with an Intel RealSense D455 depth camera, a Jetson AGX Orin computing device, and a Piper robotic arm. (See attached image) Figure 3This paper demonstrates the navigation process of the present invention in a real-world environment, including example diagrams of structured visual cues (FPV and BEV annotations) and VLM inference self-explanation. Figure 4 illustrates the inference flow during the re-validation phase, where green dashed circles represent potential targets detected through real-time target detection, and red dashed boxes represent targets successfully detected by the action-decision-based visual language model through field of view expansion. Appendix Table 1 presents the quantitative results of the present invention on the HM3Dv1, HM3Dv2, and MP3D datasets, comparing it with various existing methods in terms of target finding success rate (SR) and navigation efficiency (SPL). The results show that the present method significantly improves both success rate and navigation efficiency. The VLFM algorithm is described in "Vlfm: Vision-language frontier maps for zero-shotsemantic navigation," published in the proceedings of the 2024 IEEE Conference on Robotics and Automation (ICRA), pp. 42-48; the SG-NAV algorithm is described in "Sg-nav: Online 3d scene graph prompting for LLM-based zero-shot object navigation," published in the proceedings of the 37th Neural Information Processing Systems Conference (NeurIPS 2024), pp. 5285-5307; the BeliefMapsNav algorithm is described in "BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation," published in the arXiv preprint, 2025; and the ApexNav algorithm is described in "ApexNav: An Adaptive Exploration Strategy for Zero-Shot Object Navigation with Target-centric Semantic Fusion," published in the journal *IEEE Robotics and Automation*. Letters, 2025; Table 1. Target navigation evaluation on the HM3Dv1, HM3Dv2, and MP3D datasets. Bold text represents the optimal results.

[0110]

[0111] Compared to existing methods based on scene abstraction, this invention further enhances the decision-making ability and performance ceiling of visual language models in target navigation tasks while preserving continuous spatial and visual detail information.

[0112] Example 2

[0113] This embodiment relates to a target navigation device based on active 3DGS and visual language model inference, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the method described in Embodiment 1. Specifically, the device is deployed on a mobile robot platform (such as a quadruped robot) and equipped with an RGB-D camera and an onboard computing unit (such as an NVIDIA Jetson). The processor runs a SLAM thread for 3DGS mapping and a parallel planning thread for viewpoint optimization and VLM communication.

[0114] Example 3

[0115] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the target navigation method based on active 3DGS and visual language model reasoning as described in Embodiment 1.

[0116] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation schemes. Any other similar implementations derived by those skilled in the art based on the technical solutions of this invention also fall within the protection scope of this invention.

Claims

1. A target navigation method based on active 3DGS and visual language model inference, comprising the following steps: S1: Acquire RGB-D images and poses in an unknown environment, and construct an incremental 3D Gaussian splash (3DGS) map as a persistent memory through active perception; S2: Generate an exploration map based on the constructed 3DGS map, extract frontier points and perform adaptive spatial structure clustering; S3: Generate a guidance trajectory and perform free viewpoint optimization based on the trajectory to render a first-person perspective (FPV) image of the leading edge point containing rich information. S4: Construct structured visual cues, combine them with Chain-of-Thought (CoT) cues, input them into the visual language model for reasoning and planning, and select the best navigation target; S5: During navigation, a real-time target detector is used to filter potential targets, and a new perspective is rendered in 3DGS space through motion decision VLM to re-verify the target.

2. The method of claim 1, wherein, Step S1 includes the following steps: S11: Obtain the agent's current RGB image, depth map, and world-to-camera pose transformation. ; S12: Construct a 3D map composed of anisotropic Gaussian primitives; S13: Execute an active perception strategy to render a panoramic opacity field in a virtual camera to estimate observation integrity.

3. The method as described in claim 2, characterized in that, Step S12 specifically includes: S121: Each Gaussian element is defined by position, opacity, color, and covariance matrix. (1) Each Gaussian is determined by its position in the world coordinate system. 3D covariance matrix ,transparency And the first-order spherical harmonic function representing the color of each Gaussian sphere. Composition; wherein, each Gaussian first-order spherical harmonic function has a total of 12 coefficients, By rotation matrix and scale matrix composition: (2) Replace rotation matrix with quaternion , Represented by a 3D scale vector; S122: Obtain camera data via front-end visual odometry 3D Gaussian coordinates in the world coordinate system Transformed to the corresponding two-dimensional plane using the projection matrix. (3) Where the covariance matrix Jacobian matrix approximated by affine Projected onto a plane: (4) in It is a projection function. yes The rotation matrix in; S123: After obtaining the projected 3D Gaussian points, sort them according to their distance from the front and back of the canvas, and then efficiently obtain the pixel color values ​​through the alpha blending method: (5) S124: Set color value Replace with depth value This allows you to obtain the rendered depth value: (6) To pass the spherical harmonic coefficient The color obtained from the Gaussian function, Two-dimensional Gaussian equation and transparency Density values ​​of the product: (7) Three-dimensional Gaussian center Splash to planar pixels , Given the camera intrinsic parameter matrix, Let be the transformation matrix from world coordinates to camera coordinates in the k-th frame. This represents the z-axis distance of the corresponding 3D point, i.e., the depth. S125: Calculate the opacity field : (8) 4. The method as described in claim 3, characterized in that, Step S13 specifically includes: S131: Utilizing the flexibility of virtual cameras in adjusting intrinsic parameters, adaptive configuration of the observation angle can be achieved by setting the horizontal field of view of the camera to hFoV and the vertical field of view to vFoV. S132: To achieve panoramic coverage of the environment, centered on the robot, according to... The number of images is uniformly rendered across multiple viewpoints in the horizontal direction; the resolution of each rendered viewpoint is set to W×H pixels, and the corresponding camera intrinsic parameter matrix is ​​calculated based on the aforementioned field of view parameters. (9) Panoramic Opacity Field This is achieved by stitching together the images from different viewpoints rendered in the above steps in the horizontal direction; S133: Using the DBSCAN algorithm Clustering of low-opacity regions in the panoramic opacity field This allows for the identification of representative discrete observation target perspectives within a continuous space. S134: Since the roll angle does not affect the observation process, the target viewpoint is based on the region with the largest area of ​​low opacity. Calculations were performed, and the pitch angle was used. With yaw angle Perform parameterization: (10) in To measure the threshold of low opacity; then, using the optimal pitch angle. With yaw angle The system guides the real camera to adjust its posture, enabling it to actively cover areas lacking visual information and gradually build up complete information about the surrounding environment.

5. The method as described in claim 1, characterized in that, Step S2 includes the following steps: S21: Based on 3DGS rendering and agent height information, construct an exploration map from a top view, dividing the area into explored, unexplored, and obstacle areas; S22: Extract frontier points from the exploration map and refine their locations based on Chebyshev distance; S23: Construct a distance field based on the exploration map and detect local maxima to identify core skeleton points. Use the skeleton points as seeds to cluster the frontier points using the watershed algorithm, and select the centroids of each category as representative frontier points.

6. The method as described in claim 1, characterized in that, Step S3 includes the following steps: S31: Constructing the guide trajectory cost function The optimal guidance trajectory to each frontier point is generated based on the improved Dijkstra algorithm. : (11) (12) in, Step size, Distance to the obstacle, The relevant nonlinear penalty term, As a safety threshold, It is weight; S32: Curvature-based and distance The maximization strategy initializes the virtual viewpoint position on the guided trajectory. : (13) in, It is a weighted threshold that measures curvature and distance; based on the position of the virtual viewpoint and combined with its directional information toward the leading edge point, the initial pose is finally determined. S33: Constructing a composite loss function By optimizing the camera rotation quaternion Translation vector Determine the optimal virtual viewpoint: (14) in, This is used to balance the observed and unobserved areas due to opacity loss. To shield against radiation damage and ensure a direct line of sight to the front line; To compensate for the loss in viewpoint alignment, ensure the camera is facing the leading edge point; To mitigate trajectory loss, camera motion is constrained, causing the camera to move along and remain close to the guide trajectory. , , , For hyperparameters; S331: The specific optimization loss is: (15) in, To perform an averaging operation; S332: The specific optimization loss is: (16) For the sigmoid function, To render the sampling depth on the depth map, This represents the depth value corresponding to the leading edge point; S333: The specific optimization loss is: (17) The angle between the camera's forward vector and the ray from the camera's optical center to the leading edge point; S334: The specific optimization loss is: (18) in, For the trajectory point, It is a smoothing factor; The weighting factor is adaptively adjusted based on the opacity loss, and the specific calculation method is as follows: (19) in, Need to be from Separated from gradient calculation; S34: Based on steps S32 and S33, the optimal virtual viewpoint pose can be obtained. The first-view (FPV) image of each observable frontier point is rendered using the optimal virtual view.

7. The method as described in claim 1, characterized in that, Step S4 includes the following steps: S41: Render the FPV of the leading edge point in the 3DGS map using the optimized viewpoint in step S3, and generate the corresponding bird's-eye view (BEV). S42: Overlay structured visual annotations on FPV and BEV, including: marking unobserved areas in FPV with a specific color (purple) and marking gaze points with red dots; drawing historical trajectories (blue), spatial distribution of leading edge points corresponding to FPV (red dots), guiding trajectories connecting leading edge points (green), and current position (purple) on BEV. S43: Input the annotated FPV, BEV, and Chain-of-Thought (CoT) text prompts into the VLM. The VLM analyzes the spatial layout and semantic information and outputs the next navigation target point.

8. The method as described in claim 1, characterized in that, Step S5 includes the following steps: S51: Use a real-time target detector to detect target objects in the environment; S52: When the detection result is blurry, trigger VLM to make action decisions and select an action from the action space (forward, backward, left turn, right turn, etc.); S53: Project the selected action into the 3D Gaussian Splash (3DGS) space, render a new perspective image of the corresponding target, and input the image back into the visual language model for target confirmation; if the target object required in the instruction is confirmed to exist, guide the robot to approach the target object and end the navigation process; otherwise, continue to perform exploration navigation.

9. A target navigation device based on active 3DGS and visual language model reasoning, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the target navigation method based on active three-dimensional Gaussian splashing and visual language model reasoning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the target navigation method based on active 3DGS and visual language model reasoning as described in any one of claims 1-8.

Images