Plateau complex terrain oriented robot enhanced visual language navigation method and system

Abstract

The application discloses a robot enhanced visual language navigation method and system for complex plateau terrain, which comprises collecting multi-modal perception data and constructing a four-layer hierarchical scene graph; receiving instructions, converting them into executable target sequences, and performing semantic alignment to form a path planning framework; introducing a retrieval enhancement generation mechanism and a thinking chain reasoning mechanism, retrieving a navigation strategy similar to current perception information in a historical experience memory bank, combining the thinking chain for step-by-step reasoning and reflection-type updating, and generating a path decision; based on the path decision and the four-layer hierarchical scene graph, performing three-layer dynamic planning of high-level task decomposition, middle-layer path generation, and bottom-layer action and obstacle avoidance control to form a full-link closed loop from task understanding to safe execution. The application considers long-distance navigation success rate and dynamic obstacle avoidance real-time performance by fusing an elevation constraint scene graph, experience reuse reasoning and three-layer closed loop execution.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to an enhanced visual-language navigation method and system for robots oriented towards complex plateau terrain. Background Technology

[0002] With the development of intelligent logistics and long-distance material transportation, the demand for robots in complex field environments such as plateaus is increasing. Plateau environments are characterized by rugged terrain, unstructured landforms, and variable weather conditions (such as snow accumulation, dust storms, and sudden changes in lighting), which places extremely high demands on the autonomous navigation capabilities of robots.

[0003] Traditional robot navigation methods primarily focus on geometric obstacle avoidance and point-to-point path planning, struggling to understand complex human natural language commands and lacking the ability to deeply integrate linguistic constraints with 3D spatial information in complex tasks. Existing visual-language navigation (VLN) technologies are typically applied indoors or on structured urban roads, where robots act according to human commands. However, they suffer from the following serious shortcomings in high-altitude environments: (1) Single representation dimension and ignores key physical constraints: Existing topological maps usually only contain nodes and edges, do not integrate elevation data, and do not consider the drastic elevation changes unique to plateaus, which may result in the planned path being physically infeasible, such as the slope being too large to allow the robot to pass.

[0004] (2) Decision-making is black box, lacking interpretability and experience reuse: Existing visual-language navigation mostly adopts end-to-end reinforcement learning, lacks step-by-step reasoning mechanism. In long-distance transportation tasks, it is difficult to trace and correct errors once they occur. Furthermore, it is impossible to transfer and reuse past navigation experience across tasks and terrains. When facing new tasks, it is often necessary to rebuild maps and train again, resulting in low efficiency of long-distance navigation.

[0005] (3) Semantic planning and motion control are disconnected, and there is a lack of real-time safety closed loop: The high-level semantic planning and low-level real-time motion control of existing systems are often disconnected, making it difficult to coordinate long-term task objectives and sudden dynamic obstacles (such as landslides, falling rocks or moving obstacles), and unable to form a safety closed loop from perception to execution. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology. To achieve the above objective, an enhanced visual language navigation method and system for robots in complex plateau terrain is adopted to solve the problems mentioned in the background technology.

[0007] This invention provides an enhanced visual-language navigation method for robots in complex plateau terrain, comprising the following steps: Step S1: Collect multimodal perception data and construct a four-layer hierarchical scene map based on the multimodal perception data, which includes an elevation layer, a region layer, a view layer, and an object layer; wherein, the elevation layer records terrain undulation and slope information to constrain the feasibility of passage, the region layer divides sub-regions with macro-semantic attributes, the view layer records visible topological nodes to form a navigation map, and the object layer marks ground features, obstacles and their semantic attributes. Step S2: Receive natural language transport instructions, convert the natural language transport instructions into an executable target sequence, and semantically align the target sequence with the four-layer hierarchical scene graph to form a path planning framework under semantic constraints. Step S3: Introduce a retrieval-enhanced generation mechanism and a thought chain reasoning mechanism. By retrieving navigation strategies similar to the current perceived information from the historical experience memory bank, and combining thought chain to perform step-by-step reasoning and reflective updates, a path decision is generated. Step S4: Based on the path decision and the four-layer hierarchical scene graph, perform three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control, forming a closed-loop chain from task understanding to safe execution, enabling the robot to operate stably in complex plateau environments. The low-level control introduces real-time risk assessment to respond to dynamic obstacles.

[0008] Furthermore, the construction of a four-layer hierarchical scene graph in step S1 specifically includes: extracting terrain topology and feature labels from multimodal perception data; mapping elevation data to the elevation layer, mapping regional divisions to the region layer, mapping topological nodes to the view layer, and mapping semantic labels to the object layer; establishing spatial and semantic associations between the four-layer hierarchical scene graphs, wherein the four-layer hierarchical scene graphs are associated through a unified spatial coordinate system and cross-layer semantic mapping relationships; Among them, the elevation layer uses raster or point cloud units to represent the terrain elevation, slope and undulation features, and provides the underlying geometric attributes and access constraints for each semantic sub-region in the regional layer. The region layer divides the environment based on spatial adjacency and scene semantics. Each sub-region is associated with elevation units, view nodes and object nodes within its coverage area. Each view node in the view layer corresponds to the robot's observable position in the environment. The nodes are connected topologically based on spatial reachability and are associated with their respective regions and the objects that can be observed at that position. The object layer records semantic attributes such as feature identification, obstacles and their categories, locations and states. It is associated with corresponding areas through inclusion relationships and with one or more viewpoint nodes through visibility relationships. At the same time, its spatial occupancy and obstacle attributes can be fed back to the elevation layer and viewpoint layer to update local accessibility and node connection relationships, thus forming a multi-scale spatial semantic association structure composed of bottom-level terrain geometry, macro-regional semantics, accessible viewpoint topology and fine-grained object semantics.

[0009] Furthermore, the acquisition of multimodal sensing data in step S1 specifically includes: fusing RGB-D images, lidar point clouds, digital elevation model data, and inertial navigation information; performing denoising and anti-interference processing on the multimodal sensing data, the processing including dynamically adjusting the fusion weights of different sensor data according to the current meteorological conditions, and performing joint robust modeling of visual and geometric information.

[0010] Further, step S2 specifically includes: converting transportation task instructions into an executable target sequence through natural language processing; semantically aligning the spatial constraints, object constraints, and region constraints in the target sequence with the corresponding layers of the four-layer hierarchical scene graph to form a path planning framework under the semantic constraints.

[0011] Further, step S3 specifically includes: storing historical trajectories, landmark information, and task execution results in the historical experience memory bank; when facing a new task, retrieving similar historical navigation strategy fragments from the experience memory bank based on the current scene graph information; using the retrieved strategy fragments as context, generating a sequential decision-making process containing multiple navigation sub-steps through a thought chain reasoning model.

[0012] Furthermore, the three-layer dynamic programming in step S4 specifically includes: the high-level task decomposition, which is responsible for decomposing the macro-level task objective into regional sub-objectives; the mid-level path generation, which is responsible for generating specific paths and local navigation strategies in the four-layer hierarchical scene graph, in combination with the regional sub-objectives; and the low-level action and obstacle avoidance control, which is responsible for introducing a real-time risk assessment mechanism and responding to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

[0013] This invention also provides an enhanced vision-language navigation system for robots designed for complex plateau terrain, comprising: The multimodal sensing module is used to collect multimodal sensing data; The scene graph construction module is used to construct a four-layer hierarchical scene graph based on the multimodal perception data, which includes an elevation layer, a region layer, a view layer, and an object layer. The elevation layer records terrain undulation and slope information to constrain the feasibility of passage; the region layer divides sub-regions with macro-semantic attributes; the view layer records visible topological nodes to form a navigation map; and the object layer marks ground features, obstacles, and their semantic attributes. The language instruction understanding module is used to receive natural language transport instructions, convert them into executable target sequences, and semantically align them with the four-layer hierarchical scene graph. The decision-making and planning module includes a memory bank unit, a retrieval enhancement generation unit, a thought chain reasoning unit, and a hierarchical planning unit. The retrieval enhancement generation unit is used to retrieve navigation strategies similar to the current perceived information from the memory bank unit. The thought chain reasoning unit is used to generate path decisions by performing step-by-step reasoning and reflective updates based on the retrieval results. The hierarchical planning unit is used to perform three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control. A motion control module is used to perform the path decision and introduce real-time risk assessment in response to dynamic obstacles.

[0014] Furthermore, the multimodal perception module is used to fuse RGB-D images, lidar point clouds, digital elevation model data, and inertial navigation information, and dynamically adjust the weights of different sensor data for anti-interference processing.

[0015] Furthermore, the language instruction understanding module is used to convert transportation task instructions into an executable target sequence through natural language processing, and to semantically align the spatial constraints, object constraints, and region constraints in the target sequence with the corresponding layers of the four-layer hierarchical scene graph.

[0016] Furthermore, the hierarchical planning unit adopts a three-layer execution framework consisting of high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control. The low-level action and obstacle avoidance control incorporates a real-time risk assessment mechanism, which responds to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

[0017] Compared with the prior art, the present invention has the following technical advantages: By employing the aforementioned technical solutions, and constructing a four-layer hierarchical scene map that integrates elevation physical constraints, the system effectively addresses the problems of physical infeasibility and dead ends in complex, undulating terrain encountered by traditional navigation methods, thereby improving the success rate and accuracy of long-distance navigation. The introduction of a retrieval-enhanced generation mechanism and a thought chain reasoning mechanism enables the system to dynamically invoke historical experience and perform step-by-step reasoning, achieving cross-terrain and cross-task experience transfer and online self-evolution, significantly improving navigation efficiency and generalization capabilities. Furthermore, by executing three-layer dynamic programming and introducing real-time risk assessment at the bottom layer, a closed-loop system from task understanding to safe execution is formed, ensuring the robot's stable operation under sudden dynamic obstacles. In addition, anti-interference processing of multimodal perception data enhances the system's environmental adaptability to extreme weather conditions on plateaus (sudden changes in light intensity, snow accumulation, dust storms, etc.). Attached Figure Description

[0018] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings: Figure 1 This is a schematic diagram illustrating the steps of the visual language navigation method according to an embodiment of this application; Figure 2 A flowchart illustrating the construction of a four-layer hierarchical scene graph according to an embodiment of this application; Figure 3 This is a flowchart illustrating the semantic alignment and target sequence generation process of an embodiment disclosed in this application. Figure 4 This is a flowchart of the search enhancement generation and thought chain reasoning process according to an embodiment of this application; Figure 5 This is a block diagram of a visual language navigation system according to an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1 Please refer to Figure 1 This invention provides an enhanced visual-language navigation method for robots in complex plateau terrain, applicable to robot transportation missions in such environments. Specific steps include: Step S1: Collect multimodal perception data and construct a four-layer hierarchical scene map based on the multimodal perception data, which includes an elevation layer, a region layer, a viewpoint layer, and an object layer.

[0021] like Figure 2 As shown, the diagram illustrates the process of constructing a four-layer hierarchical scene diagram. The robot's sensor system collects multimodal perception data, including RGB-D images, LiDAR point clouds, digital elevation model data, and inertial navigation information.

[0022] Based on the collected multimodal sensing data, plateau environment modeling and multimodal sensing processing are performed. To address interference from factors such as light variations, snow accumulation, and dust in the plateau environment, multimodal data fusion and denoising algorithms are introduced to perform joint visual-geometric information modeling. Simultaneously, static information such as terrain topology, road signs, and key feature labels is extracted and used in conjunction with real-time sensing results.

[0023] Based on this, a four-layer hierarchical scene graph is constructed, consisting of an elevation layer, a region layer, a viewpoint layer, and an object layer, decomposing complex scene information into four layers: The elevation layer records information on terrain undulations, slopes, and the feasibility of vehicle passage. The regional layer divides a large spatial area into sub-regions with macroscopic semantic attributes; The view layer records the visible topological nodes of the robot at a specific location, forming a navigation map; Fine-grained labeling of ground features, obstacles, and their semantic attributes at the object layer.

[0024] Step S2: Receive natural language transport instructions, convert the natural language transport instructions into an executable target sequence, and semantically align the target sequence with the four-layer hierarchical scene graph to form a path planning framework under semantic constraints.

[0025] The system receives human transportation task instructions from operators through a natural language processing module and transforms these instructions into a structured, executable sequence of objectives.

[0026] Subsequently, instruction correction and contextual reasoning are performed by combining environmental semantic information, and the linguistic constraints in the target sequence are spatially aligned with the four-layer scene graph. Specifically, the spatial constraints, object constraints, and region constraints in the instructions are semantically aligned with the elevation layer, object layer, and region layer of the scene graph, respectively, forming a path planning framework under semantic constraints to ensure that the navigation strategy is highly consistent with the task objective.

[0027] Step S3: Introduce a retrieval-enhanced generation mechanism and a thought chain reasoning mechanism. By retrieving navigation strategies similar to the current perceived information from the historical experience memory bank, and combining thought chains for step-by-step reasoning and reflective updates, a path decision is generated.

[0028] The system has a historical experience memory bank, which records historical trajectories, landmark information, and task execution results.

[0029] When faced with new tasks or terrain, the system retrieves similar navigation strategies from its historical experience memory based on current perception information. This is the application of the retrieval enhancement generation mechanism, which enables the system to dynamically call upon knowledge from the long-term experience base.

[0030] After retrieving similar strategies, the system combines retrieved experience with current multimodal information to perform multi-step planning and reflective updates through a thought chain reasoning mechanism. The system can quickly transfer existing strategies to similar terrains or tasks, achieving self-evolving online strategy adjustments.

[0031] Step S4: Based on the path decision and the four-layer hierarchical scene graph, perform three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control, forming a closed-loop chain from task understanding to safe execution, enabling the robot to operate stably in complex plateau environments. The low-level control introduces real-time risk assessment to respond to dynamic obstacles.

[0032] The high-level task decomposition is responsible for breaking down macro-level task objectives into regional-level sub-objectives.

[0033] Mid-level path generation generates specific paths and local navigation strategies in the scene graph, and optimizes path selection through dynamic programming and reinforcement learning.

[0034] The underlying motion and obstacle avoidance control incorporates a real-time risk assessment and safety planning mechanism. Through obstacle prediction, speed regulation, and path correction, the robot is able to cope with sudden landslides, rockfalls, or moving obstacles on the plateau and operate stably in the complex plateau environment.

[0035] Through the above steps S1 to S4, multimodal perception, language command understanding, semantic map construction and dynamic path planning are organically combined, solving the problems of long-distance navigation, obstacle avoidance and command understanding for robots in plateau terrain and complex tasks.

[0036] Based on the above embodiments, the specific process of constructing a four-layer hierarchical scene graph in step S1 will be further explained.

[0037] First, terrain topology and feature labels are extracted from multimodal sensing data.

[0038] Then, the elevation data is mapped to the elevation layer, the region division results are mapped to the region layer, the topology nodes are mapped to the view layer, and the semantic tags are mapped to the object layer. The data in each layer are independent yet interconnected.

[0039] Finally, spatial and semantic relationships are established among the four hierarchical scene graphs, enabling information from each layer to work together to provide multi-dimensional support for subsequent navigation decisions.

[0040] The four-layer hierarchical scene graph is associated through a unified spatial coordinate system and cross-layer semantic mapping relationships.

[0041] Among them, the elevation layer uses raster or point cloud units to represent the terrain elevation, slope and undulation features, and provides the underlying geometric attributes and access constraints for each semantic sub-region in the regional layer.

[0042] The region layer divides the environment based on spatial adjacency and scene semantics. Each sub-region is associated with elevation units, view nodes, and object nodes within its coverage area.

[0043] Each view node in the view layer corresponds to an observable location of the robot in the environment. Topological connections are established between nodes based on spatial reachability, and each node is associated with its own region and the observable objects at that location.

[0044] The object layer records semantic attributes such as feature identification, obstacles and their categories, locations and states. It is associated with corresponding areas through inclusion relationships and with one or more viewpoint nodes through visibility relationships. At the same time, its spatial occupancy and obstacle attributes can be fed back to the elevation layer and viewpoint layer to update local accessibility and node connection relationships, thus forming a multi-scale spatial semantic association structure composed of bottom-level terrain geometry, macro-regional semantics, accessible viewpoint topology and fine-grained object semantics.

[0045] Based on the above embodiments, the processing steps for collecting multimodal sensing data in step S1 are further explained.

[0046] The collected multimodal sensing data integrates RGB-D images, lidar point clouds, digital elevation model data, and inertial navigation information.

[0047] To address interference factors such as light variations, snow accumulation, or dust in high-altitude environments, denoising and anti-interference processing is performed on multimodal sensing data. This processing includes dynamically adjusting the fusion weights of different sensor data based on current meteorological conditions, performing joint robust modeling of visual and geometric information, and improving the robustness of perception under extreme weather conditions.

[0048] Specifically, the steps for determining current weather conditions are as follows: Current weather conditions are determined by the combined data quality indicators collected by the robot's RGB-D camera, lidar, and inertial sensors, without relying on external weather monitoring equipment.

[0049] Specifically, visual quality indicators such as average brightness, brightness variance, image contrast, Laplacian variance (sharpness), haze index, and color saturation are extracted from RGB images; geometric quality indicators such as effective point cloud density, mean echo intensity, point cloud integrity, and distance attenuation rate are extracted from LiDAR point cloud computing; and attitude stability and vibration amplitude from inertial navigation data are combined to form a current environment perception quality feature vector. Based on the feature vector, threshold discrimination or a lightweight classification model is used to identify whether the current environment belongs to meteorological conditions such as normal lighting, strong light, weak light, snow cover, dust interference, or haze.

[0050] The dynamic weight adjustment strategy is as follows: The reliability score is calculated based on the data quality of each sensor, where the visual reliability is denoted as Rv and the LiDAR reliability is denoted as Rg. The fusion weights are dynamically allocated according to the reliability normalization principle, that is, the weight of the visual sensor is Wv = Rv / (Rv + Rg) and the weight of the LiDAR is Wl = Rg / (Rv + Rg).

[0051] When snow accumulation or sudden changes in lighting cause a decrease in visual quality, the weight of lidar data is increased; when dust or haze causes a decrease in point cloud quality, the weight of visual data is increased; when both types of sensors are interfered with, digital elevation model data and inertial navigation information are combined for joint compensation to achieve adaptive fusion of visual and geometric information and improve the robustness of environmental perception in complex plateau environments.

[0052] Based on the above embodiments, step S2 will be further explained.

[0053] First, natural language processing is used to transform transportation task instructions into an executable sequence of targets.

[0054] Then, the spatial constraints, object constraints, and region constraints in the target sequence are semantically aligned with the corresponding layers of the four-layer hierarchical scene graph to form a path planning framework under semantic constraints, ensuring that the navigation strategy is highly consistent with the task objective.

[0055] like Figure 3 The diagram shows the flowchart for semantic alignment and target sequence generation. Specifically, the executable target sequence is semantically aligned with a four-layer hierarchical scene graph. This includes: segmenting natural language transportation task instructions, performing entity recognition, referential resolution, and semantic role labeling; extracting action types, target objects, target regions, spatial relationships, sequence relationships, and passage conditions from the instructions; and organizing these into a structured instruction representation composed of action nodes and constraint attributes. The category, name, and attributes of the target object are matched with object nodes in the object layer; the target region and scene type are matched with region nodes in the region layer; spatial relationships such as "nearby," "ahead," "through," "along," and "arrive" are matched with node positions, topological edges, and visibility relationships in the viewpoint layer; and slope, elevation difference, terrain type, and passage conditions are matched. The system matches capability constraints with terrain attributes in the elevation layer. For each language target, candidate nodes are retrieved from the four-layer hierarchical scene graph, and a comprehensive matching score is calculated based on semantic similarity, spatial relationship conformity, topological accessibility, observation visibility, and terrain traversal feasibility. Candidate nodes whose comprehensive matching scores meet preset conditions are selected as alignment results. When multiple candidate nodes exist, ambiguity is resolved by combining the robot's current position, the order in which targets appear in the instructions, the topological continuity between adjacent targets, and the historical dialogue context. Finally, the aligned region nodes, viewpoint nodes, object nodes, and elevation constraints are combined into an executable target sequence according to the task execution order for subsequent path generation and motion control calls.

[0056] Set language sub-goals With candidate scene graph nodes The overall matching score is:

[0057] in: This refers to the semantic similarity between the language description and the node category, name, and attributes. For the degree of conformity of spatial orientation, distance and inclusion relationship; This refers to the topological reachability between the candidate node and the robot's current position and the previous target node. The visibility matching degree between the target object and the candidate viewpoint nodes; The terrain accessibility determined by the elevation layer; The weights are preset or dynamically determined according to the instruction type, and the sum of all weights is 1.

[0058] Based on the above embodiments, step S3 will be further explained.

[0059] The historical experience memory bank stores historical trajectories, landmark information, and task execution results.

[0060] When faced with a new task, the experience memory base is retrieved based on the current scene graph information to obtain similar historical navigation strategy fragments.

[0061] Using the retrieved strategy fragments as context, a sequential decision-making process containing multiple navigation sub-steps is generated through the thought chain reasoning model.

[0062] like Figure 4 As shown, the diagram illustrates the retrieval enhancement generation and thought chain reasoning flowchart; Specifically, the historical experience memory bank stores multiple historical navigation cases. Each historical navigation case includes at least the task instructions, a four-layer hierarchical scene diagram, the robot's historical trajectory, key landmark information, environmental status, dynamic obstacle events, path execution results, and task evaluation information.

[0063] The system first generates a task feature vector based on the current task. The task feature vector includes current scene graph features, target objects, target areas, language intent, terrain features, and environmental state. Then, it calculates the semantic similarity, terrain similarity, scene topology similarity, and task type similarity between the current task and each historical navigation case. Based on the weighted comprehensive similarity, all historical navigation cases are sorted, and a preset number of historical navigation cases with the highest comprehensive similarity are selected as the search results.

[0064] Subsequently, the retrieved navigation cases are used as the planning context and input into the inference module along with the current multimodal perception information. Multiple consecutive navigation sub-steps are generated in the order of "task understanding - target localization - area planning - path generation - risk analysis - path verification". After each navigation sub-step is completed, the current path is judged to meet the target constraints, terrain constraints, and safety constraints based on the real-time perception results. When dynamic obstacles, environmental changes, or planning failures are detected, the reasons for the failure are analyzed, new navigation cases are retrieved from the historical experience memory, and subsequent navigation sub-steps are corrected. After the task is completed, the current navigation trajectory, environmental status, planning adjustment process, and execution results are written into the historical experience memory as new historical navigation cases for reuse and continuous updating of experience in subsequent navigation tasks.

[0065] Based on the above embodiments, the three-level dynamic programming in step S4 will be further explained.

[0066] The high-level task decomposition is responsible for breaking down macro-level task objectives into regional-level sub-objectives.

[0067] The intermediate path generation is responsible for generating specific paths and local navigation strategies in the four-layer hierarchical scene graph, in conjunction with the region-level sub-targets.

[0068] The underlying motion and obstacle avoidance control is responsible for introducing a real-time risk assessment mechanism, which responds to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

[0069] Specifically, the real-time risk assessment mechanism includes obstacle status acquisition, obstacle motion prediction, collision risk calculation, risk level determination, and graded obstacle avoidance response.

[0070] The robot detects static and dynamic obstacles in the environment based on continuously acquired RGB-D images and LiDAR point clouds. By associating the obstacle positions at adjacent moments, it obtains the obstacle's position, size, speed, and direction of motion. Based on the obstacle's historical motion state, it predicts the obstacle's trajectory within a preset time range using a constant velocity motion model, a constant acceleration motion model, or a state estimation model. Based on the robot's current pose, speed, braking distance, planned path, and predicted obstacle trajectory, it calculates the current distance between the robot and the obstacle, the estimated minimum distance, the estimated collision time, and the degree of trajectory overlap. Combining this with the slope, elevation difference, terrain roughness, and passable width in the elevation layer, it determines the comprehensive risk value corresponding to the current obstacle.

[0071] The comprehensive risk value is obtained by weighting distance risk, collision time risk, trajectory intersection risk, terrain environment risk, and robot motion state risk.

[0072] Based on the comprehensive risk value, obstacle states are classified into low-risk, medium-risk, and high-risk. In a low-risk state, the robot maintains its current path and limits its maximum speed according to the distance to the obstacle. In a medium-risk state, the robot reduces its linear and angular velocities and adjusts the local path control points or switches to adjacent reachable viewpoint nodes based on the obstacle's direction of movement and the available local space to bypass the obstacle. In a high-risk state, the robot performs emergency deceleration or stops and marks the elevation units, viewpoint nodes, or topological connections occupied by the obstacle as temporarily impassable, triggering the mid-level path generation module to recalculate the detour path. When no detour path meets the safety conditions, the robot is controlled to retreat to the nearest safe viewpoint node or remain stopped until the risk is reduced.

[0073] During the obstacle avoidance process, the system continuously collects the obstacle status and robot motion status, and recalculates the comprehensive risk value. When the obstacle leaves the current passage area or the comprehensive risk value drops below the preset threshold, the system removes the impassable marker of the corresponding node or topological edge, and restores the original path or generates a new subsequent path based on the updated four-layer hierarchical scene graph, thus forming a closed-loop obstacle avoidance process of risk prediction, action response, status feedback and path update.

[0074] Example 2 This embodiment provides an enhanced visual-language navigation system for robots operating in complex plateau terrain. This system can be used to implement the navigation method described in any one of Embodiments 1 above. The system includes a multimodal perception module, a scene graph construction module, a language command understanding module, a decision-making and planning module, and a motion control module.

[0075] like Figure 5 The diagram shown is a block diagram of a visual language navigation system. The multimodal sensing module is used to collect multimodal sensing data, including RGB-D images, LiDAR point clouds, digital elevation model data, and inertial navigation information.

[0076] The scene graph construction module, connected to the multimodal perception module, is used to construct a four-layer hierarchical scene graph based on the multimodal perception data. This graph includes an elevation layer, a region layer, a view layer, and an object layer. Specifically, the elevation layer records terrain undulations and slope information to constrain travel feasibility; the region layer divides areas into sub-regions with macroscopic semantic attributes; the view layer records visible topological nodes to form a navigation map; and the object layer marks features, obstacles, and their semantic attributes.

[0077] The language instruction understanding module is used to receive natural language transportation instructions issued by the operator, convert them into executable target sequences, and semantically align them with the four-layer scene graph.

[0078] The decision-making and planning module includes a memory bank unit, a retrieval enhancement generation unit, a thought chain reasoning unit, and a hierarchical planning unit. The memory bank unit stores historical trajectories, landmark information, and task execution results. The retrieval enhancement generation unit retrieves navigation strategies similar to the currently perceived information from the memory bank unit. The thought chain reasoning unit performs step-by-step reasoning and reflective updates based on the retrieval results to generate path decisions. The hierarchical planning unit performs three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control.

[0079] The motion control module, connected to the decision and planning module, is used to execute the path decision and introduce real-time risk assessment in response to dynamic obstacles.

[0080] Based on the above system embodiments, the multimodal perception module is further described. This multimodal perception module is used to fuse RGB-D images, lidar point clouds, digital elevation model data, and inertial navigation information. It can dynamically adjust the weights of different sensor data according to current weather conditions, performing robust joint modeling of visual and geometric information to improve perception robustness under extreme weather conditions.

[0081] Based on the above system embodiment, the language instruction understanding module is further described. This module is used to convert transportation task instructions into an executable target sequence through natural language processing, and to semantically align the spatial constraints, object constraints, and region constraints in the target sequence with the corresponding layers of the four-layer hierarchical scene graph, forming a path planning framework under semantic constraints.

[0082] Based on the above system implementation, the hierarchical planning unit is further described. The hierarchical planning unit adopts a three-layer execution framework: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control. The low-level action and obstacle avoidance control incorporates a real-time risk assessment mechanism, responding to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. A robot-enhanced visual-language navigation method for complex plateau terrain, characterized in that, include: Step S1: Collect multimodal perception data and construct a four-layer hierarchical scene map based on the multimodal perception data, which includes an elevation layer, a region layer, a view layer, and an object layer; wherein, the elevation layer records terrain undulation and slope information to constrain the feasibility of passage, the region layer divides sub-regions with macro-semantic attributes, the view layer records visible topological nodes to form a navigation map, and the object layer marks ground features, obstacles and their semantic attributes. Step S2: Receive natural language transport instructions, convert the natural language transport instructions into an executable target sequence, and semantically align the target sequence with the four-layer hierarchical scene graph to form a path planning framework under semantic constraints. Step S3: Introduce a retrieval-enhanced generation mechanism and a thought chain reasoning mechanism. By retrieving navigation strategies similar to the current perceived information from the historical experience memory bank, and combining thought chain to perform step-by-step reasoning and reflective updates, a path decision is generated. Step S4: Based on the path decision and the four-layer hierarchical scene graph, perform three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control, forming a closed-loop chain from task understanding to safe execution, enabling the robot to operate stably in complex plateau environments. The low-level control introduces real-time risk assessment to respond to dynamic obstacles.

2. The enhanced visual-language navigation method for robots in complex plateau terrain according to claim 1, characterized in that, The construction of a four-layer hierarchical scene graph in step S1 specifically includes: Extracting terrain topology and feature labels from multimodal sensing data; Elevation data is mapped to the elevation layer, region division is mapped to the region layer, topology nodes are mapped to the view layer, and semantic tags are mapped to the object layer; Establish spatial and semantic relationships among four hierarchical scene graphs, which are associated through a unified spatial coordinate system and cross-layer semantic mapping relationships; Among them, the elevation layer uses raster or point cloud units to represent the terrain elevation, slope and undulation features, and provides the underlying geometric attributes and access constraints for each semantic sub-region in the regional layer. The region layer divides the environment based on spatial adjacency and scene semantics. Each sub-region is associated with elevation units, view nodes and object nodes within its coverage area. Each view node in the view layer corresponds to the robot's observable position in the environment. The nodes are connected topologically based on spatial reachability and are associated with their respective regions and the objects that can be observed at that position. The object layer records semantic attributes such as feature identification, obstacles and their categories, locations and states. It is associated with corresponding areas through inclusion relationships and with one or more viewpoint nodes through visibility relationships. At the same time, its spatial occupancy and obstacle attributes can be fed back to the elevation layer and viewpoint layer to update local accessibility and node connection relationships, thus forming a multi-scale spatial semantic association structure composed of bottom-level terrain geometry, macro-regional semantics, accessible viewpoint topology and fine-grained object semantics.

3. The enhanced visual-language navigation method for robots in complex plateau terrain according to claim 1, characterized in that, The step S1, which involves collecting multimodal sensing data, specifically includes: It integrates RGB-D images, LiDAR point clouds, digital elevation model data, and inertial navigation information; The multimodal sensing data is subjected to denoising and anti-interference processing, which includes dynamically adjusting the fusion weights of different sensor data according to the current meteorological conditions and performing joint robust modeling of visual and geometric information.

4. The enhanced visual-language navigation method for robots in complex plateau terrain according to claim 1, characterized in that, Step S2 specifically includes: Transform transportation task instructions into an executable sequence of targets using natural language processing; The spatial constraints, object constraints, and region constraints in the target sequence are semantically aligned with the corresponding layers of the four-layer hierarchical scene graph to form a path planning framework under the semantic constraints.

5. The enhanced visual-language navigation method for robots in complex plateau terrain according to claim 1, characterized in that, Step S3 specifically includes: The historical experience memory bank stores historical trajectories, landmark information, and task execution results; When faced with a new task, the experience memory bank is retrieved based on the current scene graph information to obtain similar historical navigation strategy fragments; Using the retrieved strategy fragments as context, a sequential decision-making process containing multiple navigation sub-steps is generated through the thought chain reasoning model.

6. The enhanced visual-language navigation method for robots in complex plateau terrain according to claim 1, characterized in that, The three-layer dynamic programming in step S4 specifically includes: The high-level task decomposition is responsible for breaking down macro-level task objectives into regional-level sub-objectives. The intermediate path generation is responsible for generating specific paths and local navigation strategies in the four-layer hierarchical scene graph, in conjunction with the region-level sub-targets. The underlying motion and obstacle avoidance control is responsible for introducing a real-time risk assessment mechanism, which responds to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

7. A robot-enhanced visual-language navigation system for complex plateau terrain as described in any one of claims 1 to 6, characterized in that, include: The multimodal sensing module is used to collect multimodal sensing data; The scene graph construction module is used to construct a four-layer hierarchical scene graph based on the multimodal perception data, which includes an elevation layer, a region layer, a view layer, and an object layer. The elevation layer records terrain undulation and slope information to constrain the feasibility of passage; the region layer divides sub-regions with macro-semantic attributes; the view layer records visible topological nodes to form a navigation map; and the object layer marks ground features, obstacles, and their semantic attributes. The language instruction understanding module is used to receive natural language transport instructions, convert them into executable target sequences, and semantically align them with the four-layer hierarchical scene graph. The decision-making and planning module includes a memory bank unit, a retrieval enhancement generation unit, a thought chain reasoning unit, and a hierarchical planning unit. The retrieval enhancement generation unit is used to retrieve navigation strategies similar to the current perceived information from the memory bank unit. The thought chain reasoning unit is used to generate path decisions by performing step-by-step reasoning and reflective updates based on the retrieval results. The hierarchical planning unit is used to perform three-layer dynamic planning: high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control. A motion control module is used to perform the path decision and introduce real-time risk assessment in response to dynamic obstacles.

8. The robot enhanced visual-language navigation system for complex plateau terrain according to claim 7, characterized in that, The multimodal perception module is used to fuse RGB-D images, lidar point clouds, digital elevation model data, and inertial navigation information, and dynamically adjust the weights of different sensor data for anti-interference processing.

9. The enhanced visual-language navigation system for robots oriented towards complex plateau terrain according to claim 7, characterized in that, The language instruction understanding module is used to convert transportation task instructions into an executable target sequence through natural language processing, and to semantically align the spatial constraints, object constraints, and region constraints in the target sequence with the corresponding layers of the four-layer hierarchical scene graph.

10. The robot enhanced visual-language navigation system for complex plateau terrain according to claim 7, characterized in that, The hierarchical planning unit adopts a three-layer execution framework consisting of high-level task decomposition, mid-level path generation, and low-level action and obstacle avoidance control. The low-level action and obstacle avoidance control introduces a real-time risk assessment mechanism, which responds to dynamic obstacles through obstacle prediction, speed regulation, and path correction.

Images