Zero-shot visual language navigation method based on graph constraint optimization

Abstract

The application discloses a zero-shot visual language navigation method based on graph constraint optimization, and the method comprises the following steps: performing multi-stage decomposition, constructing a directed acyclic graph comprising path point nodes and object nodes, wherein each path point node corresponds to one straight displacement and one or less rotation movement, and determining a navigation sequence through topological sorting; converting semantic constraint relationships in the directed acyclic graph into mathematical constraint conditions based on a predefined constraint library, and the constraint conditions of each path point node are constructed based on the known coordinates of its parent node; solving the coordinates of the path point nodes by using a nonlinear optimization method, verifying whether the coordinates satisfy the drivable area constraint, and generating a navigation tree structure comprising multiple possible solutions; when the current path is not feasible, backtracking to an unexplored path with the deepest common parent node through a multi-solution path management mechanism of the navigation tree, and selecting an alternative path to continue navigation until the target position is reached.

Description

Technical Field

[0001] This invention relates to the field of zero-shot visual language navigation technology, and more particularly to a zero-shot visual language navigation method based on graph constraint optimization. Background Technology

[0002] Visual Language Navigation (VLN), as a core technology of multimodal perception and decision-making systems for intelligent robots, is widely used in scenarios such as home services, industrial inspection, and autonomous navigation in dynamic environments. Among related technologies, traditional methods construct a navigation framework based on labeled data through the collaborative operation of supervised learning and discrete path point planning. Specifically, this technology system covers the entire process from natural language command parsing to path point matching, including key steps such as visual feature extraction, semantic alignment, and path planning. With the development of Large Language Models (LLM) and computer vision technology, zero-shot navigation methods are gradually becoming a research direction for bridging the "simulation to reality" gap. Their core lies in utilizing the cross-environment generalization ability of pre-trained models to achieve a navigation system that does not require training in a specific environment.

[0003] However, existing zero-shot VLN methods directly model discrete path points, failing to effectively handle spatial constraints in continuous environments. This can lead to accumulated path planning errors or the loss of crucial spatial information during complex command parsing, thus impacting navigation success rates. Specifically, existing technologies typically employ end-to-end learning frameworks, but these have two significant limitations: First, research on zero-shot methods for continuous environments is limited, with most solutions still confined to the discrete space of pre-defined path points, resulting in navigation failure rates exceeding 40% in real 3D environments. Second, the spatial relationships such as orientation and distance implicit in natural language commands are not explicitly modeled, making it difficult to satisfy constraints when solving for path point coordinates. For example, when processing compound commands such as "turn left and go straight to the sofa," existing methods often fail to convert semantic relationships into computable constraint forms, leading to path conflicts during BEV constraint verification. This technical deficiency not only limits the system's adaptability in dynamic environments but also results in a task interruption rate exceeding 25%, severely hindering the practical application of zero-shot navigation technology. Summary of the Invention

[0004] The main objective of this invention is to provide a zero-shot visual language navigation method based on graph constraint optimization.

[0005] Another objective of this invention is to propose a zero-sample visual language navigation device based on graph constraint optimization.

[0006] The third objective of this invention is to provide a computer device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a zero-shot visual language navigation method based on graph constraint optimization, comprising:

[0009] S1 utilizes a large language model to decompose natural language instructions in multiple stages, constructs a directed acyclic graph containing path point nodes and object nodes, where each path point node corresponds to one linear displacement and one or less rotational motion, and determines the navigation sequence through topological sorting. S2, based on a predefined constraint library, the semantic constraint relationships in the directed acyclic graph are transformed into mathematical constraint conditions. The mathematical constraint conditions include angle constraints and distance constraints, and the constraint conditions of each path node are constructed based on the known coordinates of its parent node. S3. The coordinates of the path point nodes are solved by nonlinear optimization method. The coordinates are verified by combining the real-time acquired RGB-D environmental observation data to see if they meet the drivable area constraints, and a navigation tree structure containing multiple possible solutions is generated. S4 uses the multi-solution path management mechanism of the navigation tree to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and selects an alternative path to continue navigation until the target location is reached.

[0010] In one embodiment of the present invention, the method of using a large language model to perform multi-stage decomposition of natural language instructions, constructing a directed acyclic graph containing path node nodes and object nodes, wherein each path node node corresponds to one linear displacement and one or fewer rotational movements, and determining the navigation sequence through topological sorting, includes: S11 requires that each object node mentioned in the instruction must be uniquely assigned to a navigation phase, and that the path point nodes and object nodes of each navigation phase establish explicit spatial constraint relationships through edges. S12, when converting a multi-stage directed acyclic graph into a linear navigation sequence through topological sorting, the navigation direction and object matching order are determined sequentially according to the topological order of the path point nodes.

[0011] In one embodiment of the present invention, the semantic constraint relationships in the directed acyclic graph are transformed into mathematical constraints based on a predefined constraint library. These mathematical constraints include angle constraints and distance constraints, and the constraints for each path node are constructed based on the known coordinates of its parent node, including: S21, transform semantic constraints into angular constraints. and distance constraints ,in and These represent the tolerance thresholds for angle and distance, respectively. S22 dynamically generates corresponding mathematical constraints based on six types of spatial relationships in the constraint library, and combines all constraints into a computable optimization objective function.

[0012] In one embodiment of the present invention, the step of solving for the coordinates of path point nodes using a nonlinear optimization method, verifying whether the coordinates satisfy the drivable area constraints by combining real-time acquired RGB-D environmental observation data, and generating a navigation tree structure containing multiple possible solutions includes: S31, the problem of finding the coordinates of path points is modeled as maximizing the constraint function. The nonlinear optimization problem, in which The summation function represents the constraint conditions; S32 generates a bird's-eye view using real-time acquired RGB-D environmental observation data, verifies whether the obtained coordinates are within the drivable area, and excludes solutions that are in contact with obstacles.

[0013] In one embodiment of the present invention, the multi-solution path management mechanism through the navigation tree, which backtracks to the unexplored path with the deepest common parent node when the current path is infeasible, selects an alternative path to continue navigation until the target location is reached, includes: S41, when the current path branch has no solution in the constraint solver, backtrack to the nearest unexplored path branch with the deepest common parent node, and resolve the coordinates of subsequent path nodes based on that branch.

[0014] In one embodiment of the present invention, it further includes: S5 dynamically adjusts the tolerance thresholds in the constraint library based on real-time acquired RGB-D environmental observation data. and To adapt to the spatial characteristics of the current environment; S6, select the path that satisfies the multiple solutions from the navigation tree. The path with the smallest function value is selected as the priority path for exploration. The function representing the minimum value of the constraint condition.

[0015] To achieve the above objectives, a second aspect of the present invention provides a zero-shot visual language navigation device based on graph constraint optimization, comprising: The instruction decomposition and graph construction module is used to decompose natural language instructions in multiple stages using a large language model, construct a directed acyclic graph containing path point nodes and object nodes, where each path point node corresponds to one linear displacement and one or less rotational motion, and determines the navigation sequence through topological sorting. The semantic constraint conversion module is used to convert the semantic constraint relationships in the directed acyclic graph into mathematical constraint conditions based on a predefined constraint library. The mathematical constraint conditions include angle constraints and distance constraints, and the constraint conditions of each path node are constructed based on the known coordinates of its parent node. The coordinate optimization and verification module is used to solve the coordinates of path point nodes using nonlinear optimization methods, and to verify whether the coordinates meet the drivable area constraints by combining real-time acquired RGB-D environmental observation data, and to generate a navigation tree structure containing multiple possible solutions. The multi-solution path management module is used to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and select an alternative path to continue navigation until the target location is reached.

[0016] To achieve the above objectives, a third aspect of this application provides a computer device comprising a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, for implementing a zero-shot visual language navigation method based on graph constraint optimization as described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a zero-shot visual language navigation method based on graph constraint optimization as described in the first aspect embodiment.

[0018] The embodiments of the present invention have the following beneficial effects: The embodiments of the present invention can achieve high-precision visual language navigation without specific training in continuous unknown environments, effectively parsing and utilizing the implicit spatial constraints in natural language instructions, thereby improving the accuracy of path planning and the robustness of the system. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a zero-shot visual language navigation method based on graph constraint optimization, provided as an embodiment of the present invention; Figure 2 An architecture diagram of a zero-shot visual language navigation method based on graph constraint optimization provided in an embodiment of the present invention; Figure 3 A path diagram illustrating a zero-shot visual language navigation method based on graph constraint optimization, provided in an embodiment of the present invention; Figure 4This is a structural diagram of a zero-sample visual language navigation device based on graph constraint optimization, provided in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following description, with reference to the accompanying drawings, describes a zero-shot visual language navigation method and apparatus based on graph constraint optimization according to embodiments of the present invention.

[0023] Example 1 This embodiment provides a zero-shot visual language navigation method based on graph constraint optimization. For example... Figure 1 As shown, the method includes the following steps: S1 utilizes a large language model to decompose natural language instructions into multiple stages, constructing a directed acyclic graph containing path point nodes and object nodes. Each path point node corresponds to one linear displacement and one or fewer rotational movements, and the navigation sequence is determined through topological sorting.

[0024] Specifically, the core of this step lies in using a Large Language Model (LLM) to decompose natural language instructions into multiple stages, constructing a directed acyclic graph (DAG) containing path point nodes and object nodes, and determining the navigation sequence based on the topological sorting of this graph. This process is a key step in achieving zero-shot navigation in the GC-VLN method, and its technical implementation is based on the deep integration of natural language understanding and graph structure modeling.

[0025] In this embodiment of the invention, the LLM guides the reader with preset prompts, delivering natural language instructions. Parsing into a structured graph representation ,in It contains two types of nodes: path point nodes. and object nodes Path point nodes represent stages in the navigation process. Each stage involves only one linear displacement and one or fewer rotational movements, ensuring the executability and uniqueness of the navigation actions. Object nodes represent environmental landmarks mentioned in the instructions, and are connected to path points via edges. Establish spatial constraints, including information such as relative orientation and distance.

[0026] Furthermore, path point nodes The decomposition granularity is determined by the number of navigation stages mentioned in the instruction. The decision is that the displacement distance and rotation angle at each stage must satisfy the actual robot's kinematic constraints, such as the maximum displacement step size. and maximum rotation angle Edge set Edges divided into connecting path points and the edges connecting path points and object nodes Its form can be expressed as:

[0027] This step is applicable to visual language navigation tasks in continuous 3D environments, such as indoor service robots and autonomous driving assisted navigation. By converting instructions into a graph structure, the system can achieve structured understanding and path planning of complex instructions without environment-specific training.

[0028] This step leverages the semantic parsing capabilities of LLM to explicitly model implicit spatial relationships in natural language as graph constraints, providing structured input for subsequent nonlinear optimization and navigation tree backtracking mechanisms. Its value lies in improving the interpretability and executability of the navigation path, while enhancing the system's accuracy in matching object nodes in instructions, thus providing a solid foundation for zero-shot navigation.

[0029] Furthermore, S1 includes: S11 requires that each object node mentioned in the instruction must be uniquely assigned to a navigation phase, and that the path nodes and object nodes in each navigation phase establish explicit spatial constraints through edges.

[0030] Specifically, the step in this invention that "requires each object node mentioned in the instruction to be uniquely assigned to a navigation stage, and that the path node nodes and object nodes in each navigation stage establish explicit spatial constraint relationships through edges" is a key step in constructing the Graph Constrained Navigation Framework (GC-VLN). Its technical implementation is based on the structured decomposition of natural language instructions and graph theory modeling methods.

[0031] This step uses a Large Language Model (LLM) to semantically parse natural language instructions, transforming them into a Directed Acyclic Graph (DAG) structure. Nodes in the DAG are divided into two categories: path nodes and object nodes. Path nodes represent key positions of the agent during navigation, while object nodes represent environmental landmarks or target objects mentioned in the instructions. To ensure the executability and uniqueness of the navigation process, this invention stipulates that each object node must be uniquely assigned to a single navigation stage; that is, each object establishes a semantic association with only one path node. This unique attribution mechanism avoids the reuse or ambiguous matching of object nodes across multiple stages, thereby improving the accuracy of path planning.

[0032] Furthermore, explicit spatial constraints are established between each pathpoint node and its associated object node through edges. These edges not only represent connections between nodes but also encapsulate specific spatial semantic information such as orientation and distance. For example, if the instruction describes "turn left and walk to the table," the LLM will treat "table" as an object node and establish an edge with the current pathpoint node, containing constraints on direction (turn left) and distance (to the table). This process can be formalized as follows:

[0033] in, Represents the decomposed graph structure. For the set of path point nodes, For a set of object nodes, This represents the constraint edges between path points and object nodes.

[0034] In this embodiment of the invention, the constraint information carried by each edge includes an angle deviation tolerance threshold. and distance deviation tolerance threshold These parameters are used in subsequent constraint optimization solutions. For example, angle constraints. and distance constraints They are defined as follows:

[0035]

[0036] in, Indicates the target direction angle. Indicates the distance to the target. and The maximum allowable deviation is typically set to [value]. and Meters, to accommodate perception errors in actual navigation.

[0037] This step is applicable to agents performing zero-shot navigation tasks in unknown, continuous environments. Through explicit constraint modeling, the system can transform spatial relationships in natural language instructions into computable mathematical expressions, thereby achieving high-precision path planning without environment-specific training. This step provides structured input for subsequent constraint optimization and navigation tree construction, and is the foundation for the entire system to achieve robust navigation.

[0038] This step ensures the semantic integrity and constraint consistency of the navigation graph structure, avoiding semantic conflicts and ambiguities between waypoints and object nodes, and significantly improving the executability and stability of the navigation path. Simultaneously, through explicit modeling, the system can more effectively handle spatial relationships in complex instructions, providing a clear constraint basis for subsequent nonlinear optimization and path backtracking mechanisms.

[0039] S12, when converting a multi-stage directed acyclic graph into a linear navigation sequence through topological sorting, the navigation direction and object matching order are determined sequentially according to the topological order of the path point nodes.

[0040] Specifically, in this invention, converting a multi-stage directed acyclic graph (DAG) into a linear navigation sequence through topological sorting is one of the key steps in realizing the zero-shot visual language navigation (VLN) task. The core of this step lies in transforming the navigation path and spatial relationships implicit in natural language instructions into a structured graph representation, and ensuring the sequentiality and logical consistency of the navigation process through topological sorting.

[0041] This step first relies on the Large Language Model (LLM)'s ability to parse natural language instructions. Guided by a preset prompt, the LLM decomposes the instruction into multiple navigation stages, each containing one linear displacement and one or fewer rotational movements. The decomposition results in a Directed Acyclic Graph (DAG) structure, where nodes are divided into path node nodes (…). ) and object nodes ( Edges represent spatial constraints between nodes. and A topological sorting algorithm (such as Kahn's algorithm or DFS postorder retrieval) is used to arrange the nodes in a DAG into a linear sequence according to their dependencies. This provides the intelligent agent with a clear navigation direction and object matching order.

[0042] In this embodiment of the invention, the execution of topological sorting depends on the structural characteristics of the graph. Each path node... Compared with the previous path point There are clear sequential dependencies between them, ensuring that the navigation order does not result in cycles or conflicts. Furthermore, the edges between each waypoint and object node... It will carry semantic constraint information, such as azimuth angle. and distance These parameters will be used to construct mathematical constraints in subsequent constraint solving. ,in The parent node with known coordinates.

[0043] This step is applicable to agents performing zero-sample navigation tasks in unknown continuous environments. For example, in indoor service robots or autonomous mobile devices, when receiving instructions such as "turn left, walk to the table, then turn right, and go through the door," this step can decompose the instructions into multiple stages and determine the execution order of each stage through topological sorting, thereby avoiding path confusion caused by instruction ambiguity or environmental changes.

[0044] The technical advantage of this step lies in transforming the originally ambiguous natural language instructions into executable navigation sequences through structured modeling and topological sorting, providing a clear input framework for subsequent constraint solving and path optimization. Simultaneously, this method ensures the logical consistency of the navigation path, improving the system's robustness and interpretability under complex instructions.

[0045] S2, based on a predefined constraint library, the semantic constraint relationships in the directed acyclic graph are transformed into mathematical constraints. The mathematical constraints include angle constraints and distance constraints, and the constraints of each path node are constructed based on the known coordinates of its parent node.

[0046] Specifically, the core of this step lies in transforming the semantic constraints in the directed acyclic graph (DAG) parsed from natural language instructions into computable mathematical constraints, including angle constraints and distance constraints. This process is one of the key steps in achieving zero-shot navigation in the GC-VLN method, and its technical implementation is based on a predefined constraint library. By mapping semantic information into mathematical expressions, input is provided for subsequent nonlinear optimization solutions.

[0047] In this embodiment of the invention, each path point node The constraints are based on its parent node. It is constructed using known coordinates. The parent node can be a previous path point. Or object nodes that have a spatial relationship with the current path point. Constraint library The graph defines six typical spatial constraint relationships, including orientation, distance, and relative position. These relationships are expressed through the edges in the graph. The system represents the semantic relationships. When converting semantic relations into mathematical constraints, it queries the constraint library based on the edge type to obtain the corresponding constraint function. And apply it to the coordinate optimization of the current path point.

[0048] Specifically, angle constraints Modeled using the cosine function, its form is:

[0049] in Indicates the angle tolerance threshold. This refers to the direction angle described in the instruction. Distance constraint. Then, modeling is done using the difference of squares form, and its expression is:

[0050] in This is the distance tolerance threshold. This is the expected distance described in the instruction. These constraints are combined into a total constraint. The system is able to accurately model the coordinates of path points.

[0051] This step is applicable to path planning tasks based on natural language instructions in continuous 3D environments. For example, in indoor navigation scenarios, a robot needs to construct a graph structure containing angle and distance constraints based on semantic information such as instructions like "turn 90 degrees to the left, move forward 5 meters, and then approach the table." By converting these semantic constraints into mathematical expressions, the system can achieve accurate solution for path points without environment-specific training.

[0052] The technical advantage of this step lies in transforming the originally vague natural language description into computable mathematical constraints, thereby significantly improving the accuracy and feasibility of path planning. This is achieved by introducing tolerance thresholds for angles and distances. and The system can still generate reasonable pathpoint coordinates even in the presence of perceptual errors or semantic ambiguity, enhancing the robustness of navigation. Furthermore, this step provides structured input for subsequent nonlinear optimization solutions, forming the core foundation for achieving zero-shot navigation.

[0053] Furthermore, S2 includes: S21, transform semantic constraints into angular constraints. and distance constraints ,in and These represent the tolerance thresholds for angle and distance, respectively.

[0054] Specifically, this step involves transforming the semantic constraints extracted from natural language instructions into computable mathematical constraints, including angular constraints. and distance constraints ,in and These represent the tolerance thresholds for angle and distance, respectively. This step is one of the key steps in achieving zero-shot navigation in the GC-VLN method, and its technical implementation is based on accurate modeling and constraint optimization of spatial semantics.

[0055] In this embodiment of the invention, angle constraint The design aims to quantify two nodes. and The relative directional deviation between them. Specifically, Indicates from node arrive The expected direction, and This represents the system's tolerance range for direction errors. It is calculated by the vector... With unit direction vector The dot product of these two vectors yields the projected length of the vector in the desired direction. If the actual displacement... The difference between the projection length and the tolerance threshold exceeds the threshold. The corresponding cosine value multiplied by the displacement length results in a negative constraint value, indicating that the direction does not meet the semantic requirements. This constraint form can effectively capture directional semantics such as "turn left" and "go right" in instructions and transform them into an optimizable mathematical expression.

[0056] Distance constraints This is used to measure the actual distance and the expected distance between two nodes. The deviation. This represents the maximum allowable distance error for the system. This constraint is calculated by... (The sentence is incomplete and requires more context to translate accurately.) Distance from Expected The square of the difference, and the tolerance threshold. A comparison is made to assess whether the distance is within an acceptable range. If the square of the difference is greater than... If the constraint value is negative, it means that the distance relationship has not been satisfied.

[0057] Furthermore, The value is usually taken as radians (approximately) ), Values The specific values ​​can be adjusted according to the accuracy requirements of the actual navigation environment. These parameters define the tolerance range of the navigation path in terms of direction and distance, and are an important guarantee of the system's robustness.

[0058] This step is mainly used in practical applications to construct graph constraints. Its nodes include path points and object nodes, while edges represent the spatial constraints between nodes. By transforming semantic constraints into mathematical forms, the system can perform precise path point coordinate optimization in continuous environments, thereby achieving high-precision navigation without training. This method overcomes the limitations of traditional rule-based or end-to-end learning, providing an interpretable and optimizable modeling approach for zero-shot visual-language navigation, significantly improving the accuracy and adaptability of path planning.

[0059] S22 dynamically generates corresponding mathematical constraints based on six types of spatial relationships in the constraint library, and combines all constraints into a computable optimization objective function.

[0060] Specifically, in this invention, the key step of dynamically generating corresponding mathematical constraints based on six types of spatial relationships (including orientation and distance) in the constraint library, and combining all constraints into a computable optimization objective function, is to achieve zero-shot visual language navigation. The core of this step lies in transforming the spatial semantics implicit in natural language instructions into executable mathematical expressions, thereby providing structured input for subsequent path optimization.

[0061] This step first relies on a predefined constraint library. It contains six types of standardized spatial relationship descriptions, such as orientation, distance, relative position, direction, and path order. These relationships are defined in graph constraints. The values ​​are mapped to specific mathematical constraint functions. ,in This indicates the coordinates of the waypoints that need to be optimized. Indicates its parent node (coordinates are known). Each edge For a corresponding constraint type, through The constraint library is queried to generate the corresponding mathematical expression. For example, an azimuth constraint can be converted into a cosine function of the angle difference, as shown in the equation. The distance constraint is expressed by the formula It means that, among them and These are the tolerance thresholds for angle and distance, respectively, and are typically set to... , Meters, to adapt to perception errors in real-world environments.

[0062] In this embodiment of the invention, the output value of each constraint function reflects the degree of matching between the current solution and the semantic description. and The function aggregates multiple constraints to form a comprehensive optimization objective. The optimization problem (P1) is defined as maximizing... At the same time satisfy ,in The minimum safe distance is usually set to Meters, to ensure that waypoints are far from obstacles.

[0063] This step is applicable to path planning based on natural language commands in continuous 3D environments, especially in unseen scenes, achieving high-precision navigation without specific training. By transforming semantic constraints into mathematical forms, the system can solve for the coordinates of path points that meet the command requirements with the aid of real-time perceived data (such as RGB-D images).

[0064] The technical advantage of this step lies in achieving a high-fidelity mapping between natural language commands and spatial geometric constraints, providing a computable input foundation for subsequent nonlinear optimization and navigation tree backtracking mechanisms. Through the combination and optimization of structured constraints, the system can effectively handle complex spatial relationships in commands, improving the accuracy and robustness of path planning. This is the core supporting module for achieving zero-shot navigation in this invention.

[0065] S3. The coordinates of the path point nodes are solved using a nonlinear optimization method. The coordinates are then verified by combining the real-time acquired RGB-D environmental observation data to determine whether they meet the drivable area constraints, and a navigation tree structure containing multiple possible solutions is generated.

[0066] Specifically, this step employs a nonlinear optimization method to solve for the coordinates of path nodes, and combines real-time acquired RGB-D environmental observation data to verify whether the coordinates satisfy the drivable area constraints, ultimately generating a navigation tree structure containing multiple possible solutions. This step is the core component of the GC-VLN method for implementing path planning and fault tolerance mechanisms, possessing significant technical value and implementation depth.

[0067] In this embodiment of the invention, this step is first based on graph constraints. Nodes in and its associated set of constraints Construct a nonlinear optimization problem (P1) with the objective function of maximizing the sum of all constraints, i.e.:

[0068] in, This indicates that for each constraint The angle and distance components are weighted and summed, specifically in the following form:

[0069] angle constraints and distance constraints The expressions are as follows:

[0070]

[0071] in, and These represent the tolerance thresholds for angle and distance, respectively, used to measure the degree of matching between the current solution and the constraints. During the optimization process, the parent node... Given the coordinates, the current waypoint is determined step by step through iterative solutions. The optimal coordinates.

[0072] Furthermore, each solution will generate multiple candidate solutions. The solutions are then added to the solution set. Simultaneously, the system verifies whether these solutions are located within the drivable area using the BEV map, i.e., satisfying the following conditions:

[0073] in To minimize the safe distance, ensure that waypoints do not collide with obstacles. All solutions that satisfy this condition will be retained and used as a navigation tree. The child nodes in the current node This creates multiple branch paths.

[0074] This step is applicable to robots performing zero-shot visual-language navigation tasks in unknown, continuous environments. By combining real-time RGB-D data with constraint solving, the system can quickly adjust the path in dynamically changing environments, avoiding navigation failures caused by perceptual errors or command ambiguity. The construction of the navigation tree provides structured support for path backtracking and multi-solution exploration, significantly improving the system's robustness and adaptability.

[0075] The technical advantage of this step lies in achieving high-precision estimation of pathpoint coordinates through the joint solution of nonlinear optimization and BEV constraints. Simultaneously, the navigation tree structure manages multiple solutions, enhancing the system's fault tolerance in complex environments. In practical deployment, this method can achieve reliable navigation based on natural language commands in continuous space without requiring environment-specific training.

[0076] Furthermore, S3 includes: S31, the problem of finding the coordinates of path points is modeled as maximizing the constraint function. The nonlinear optimization problem, in which This represents the summation function for the constraints.

[0077] Specifically, this step models the problem of finding the coordinates of path points as maximizing the constraint function. The core of this nonlinear optimization problem (P1) lies in transforming the spatial constraints extracted from natural language instructions into a computable mathematical form, and then solving for the coordinates of the optimal path points using an optimization algorithm. In some implementations, the construction of this optimization problem relies on graph constraints. In and the current path point Related constraint set , where each constraint From constraint library It provides coverage of both angle and distance dimensions.

[0078] Specifically, each constraint Summation function By angle constraint and distance constraints It consists of two parts, and their mathematical expressions are as follows:

[0079]

[0080] in, and These represent the tolerance thresholds for angle and distance, respectively, used to measure the degree of matching between the current solution and the constraints. Indicates navigation direction, This represents the desired distance. By maximizing... The system is able to find nodes with known parent nodes while satisfying spatial semantics. Coordinates of the best matching path point .

[0081] In an embodiment of the present invention, Usually set to Angle range, The actual environmental scale is set to [0.1, 1.0] meters. In addition, the optimization process must also meet the driving area constraints provided by the BEV (bird's-eye view), that is, the waypoints must be located in the barrier-free area to avoid collisions with obstacles.

[0082] This step plays a crucial role in zero-shot visual-language navigation in continuous environments. By transforming the spatial relationships in the instructions into an optimizable mathematical form, the system can achieve accurate solutions for waypoints without environment-specific training. Simultaneously, the construction of this optimization problem lays the foundation for subsequent navigation tree generation and backtracking mechanisms. This allows the agent to achieve fault tolerance and path recovery by exploring different branch paths in the presence of multiple solutions or no solution, thereby significantly improving the robustness and success rate of navigation.

[0083] S32 generates a bird's-eye view using real-time acquired RGB-D environmental observation data, verifies whether the obtained coordinates are within the drivable area, and excludes solutions that are in contact with obstacles.

[0084] Specifically, in this invention, generating a bird's-eye view (BEV) using real-time acquired RGB-D environmental observation data and verifying whether the coordinates of the solved path points are within the drivable area, while excluding solutions that contact obstacles, is one of the key steps in achieving zero-sample visual language navigation. This step plays a core role in environmental perception and path feasibility verification during the construction of the navigation tree and path optimization.

[0085] This step first relies on RGB-D sensors (such as depth cameras) to acquire environmental images and depth information in real time. Through point cloud projection or depth map conversion methods, obstacle and ground information in 3D space is mapped onto a 2D BEV space, forming a top-down representation of the environment. The construction of the BEV typically uses a fixed-resolution mesh representation, for example... An image of pixels, where each pixel corresponds to a pixel in actual space. Areas. During the construction process, ground points are marked as drivable areas, while obstacle points are marked as impassable areas.

[0086] In this embodiment of the invention, the generation of BEV requires setting key parameters, such as the truncation distance of the depth map. To limit the range of environmental perception; threshold for ground points Used to distinguish between the ground and obstacles; and grid resolution. To ensure spatial accuracy. Furthermore, the coordinate verification of path points must meet the following conditions: path points The coordinates must satisfy ,in This represents the solution from the previous stage. The minimum safe distance is usually set to To avoid contact with obstacles.

[0087] This step is primarily used for navigation path verification in continuous, unknown environments. For example, when an indoor service robot executes commands such as "cross the living room, go around the sofa, and reach the dining table," BEV (Browsing Path Validation) is used to determine in real time whether the robot is within a walkable area and to ensure that path points do not collide with obstacles such as sofas. This verification mechanism is combined with a navigation tree structure; when a path point solution fails, the system can backtrack to the nearest common parent node and try other feasible solutions, thereby improving navigation robustness.

[0088] The technical advantage of this step lies in its ability to effectively filter out infeasible path points through real-time perception and constraint verification of the BEV (Browser Active Vehicle), ensuring the safety and feasibility of the navigation path in physical space. Combining nonlinear optimization and a navigation tree structure, this method achieves high-precision path planning for complex instructions under zero-sample conditions, significantly improving the system's adaptability and execution success rate in real-world environments.

[0089] S4 uses the multi-solution path management mechanism of the navigation tree to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and selects an alternative path to continue navigation until the target location is reached.

[0090] Specifically, the navigation tree and backtracking mechanism proposed in this invention are one of the key steps in achieving robustness of zero-shot visual language navigation. Its technical implementation principle is based on a graph-constrained optimization framework, which manages path infeasibility issues caused by constraint solution uncertainties or environmental perception errors during navigation by constructing a multi-solution path tree. In the navigation tree, each path point... It may correspond to multiple candidate coordinate solutions These solutions are obtained by the constraint solver while satisfying the graphical constraints. It is generated under the premise of BEV drivability area restrictions. The structure of the navigation tree reflects the multiple solution space of the navigation path, with each level corresponding to a navigation stage and each node representing the coordinates of a possible waypoint.

[0091] When the agent cannot continue on the current path (e.g., the constraint solver fails to find a solution that satisfies the conditions), the system will initiate a backtracking mechanism. Specifically, the agent backtracks to an unexplored path node in the current path that shares the deepest common parent node with the failed node, and selects the next untried solution at that node as an alternative path to continue navigation. This process ensures the continuity of path exploration and avoids the entire navigation task failing due to local errors. For example, if the current path is... And in If no solution is found, the system will backtrack to... Try other unexplored child nodes, such as or And continue to plan the path from that node.

[0092] In this embodiment of the invention, the backtracking mechanism depends on the size of the solution set of the constraint solver. That is, each waypoint can generate a maximum of There are several candidate solutions. Furthermore, the construction of the navigation tree must adhere to graph constraints. Topological sorting results This ensures that the order of waypoints is consistent with the semantics of the instructions. During backtracking, the system prioritizes unexplored paths that share the deepest common parent node with the current path to minimize the cost of path reconstruction and improve navigation efficiency.

[0093] This step plays a crucial role in the overall technical solution. On one hand, it effectively addresses the issues of multiple solutions and no solutions in graph constraint solving, enhancing the system's fault tolerance in complex, dynamic, or perceptually noisy environments. On the other hand, through an intelligent backtracking strategy, the system can dynamically adjust the path without re-parsing instructions, thereby significantly improving the navigation success rate. In practical applications, this mechanism is particularly suitable for systems such as indoor service robots and autonomous navigation vehicles that execute natural language commands in unknown environments, ensuring that they can still complete navigation tasks when faced with environmental changes or perceptual errors.

[0094] The zero-shot visual language navigation method based on graph constraint optimization in this invention can achieve structured modeling and high-precision path solving of spatial constraint relationships in natural language instructions without the need for environment-specific training, significantly improving the accuracy and robustness of visual language navigation in continuous environments.

[0095] Furthermore, S4 includes: S41, when the current path branch has no solution in the constraint solver, backtrack to the nearest unexplored path branch with the deepest common parent node, and resolve the coordinates of subsequent path nodes based on that branch.

[0096] Specifically, when the current path branch has no solution in the constraint solver, the system backtracks to the nearest unexplored path branch with the deepest common parent node and resolves the coordinates of subsequent path nodes based on that branch. This step is a core component of the navigation tree mechanism in the GC-VLN method, designed to improve the system's fault tolerance and navigation robustness in complex environments or environments with perceived errors.

[0097] In this embodiment of the invention, the backtracking mechanism relies on a navigation tree. The structural design of the navigation tree is based on graph constraints. Topological sorting The constructed multi-branch tree structure, where each node It may correspond to multiple coordinate solutions This creates multiple child nodes. When the constraint solver cannot find a solution that satisfies all constraints on the current path branch, the system will backtrack to the unexplored path branch with the deepest common parent node from the previous stage. Here, the "deepest common parent node" refers to the last common ancestor node shared by the current failed path and the backtrackable path in the navigation tree, which is usually a waypoint. Backtracking is implemented using either depth-first search (DFS) or breadth-first search (BFS) strategies, depending on the system's trade-off between real-time performance and exploration efficiency.

[0098] Furthermore, the triggering condition for the backtracking mechanism is usually based on the output state of the constraint solver. If in the... waypoints The constraint solver failed to complete the preset number of iterations. Find the satisfying formula inside If no feasible solution is found, then the branch is determined to be infeasible. This represents the minimum safe distance threshold, which is usually set to... Meters are used to ensure a safe distance between waypoints and obstacles. During backtracking, the system selects the next unexplored solution from the solution set of that waypoint. The coordinates of subsequent path points are recalculated to verify the feasibility of the new path.

[0099] This step is particularly suitable for complex indoor environments or scenarios with occlusion or recognition errors. For example, when the visual perception module fails to accurately identify a certain object node. In some cases, the constraints of the current path branch may not be satisfied, thus triggering a backtracking mechanism. The system will backtrack to the nearest alternative path branch, re-match object nodes, and solve for path point coordinates, thereby avoiding navigation failure due to local errors.

[0100] The technical benefit of this step is a significant improvement in the robustness and fault tolerance of the navigation system. Through the backtracking mechanism, the system can explore alternative paths without re-initializing the navigation process, thus enabling it to complete navigation tasks even in the face of environmental uncertainties or ambiguous instructions. Furthermore, this mechanism enhances the flexibility of path planning, giving GC-VLN stronger adaptability under zero-sample conditions.

[0101] The zero-shot visual language navigation method based on graph constraint optimization in this invention can achieve structured modeling and high-precision path solving of spatial constraint relationships in natural language instructions without the need for environment-specific training, significantly improving the accuracy and robustness of visual language navigation in continuous environments.

[0102] S5 dynamically adjusts the tolerance thresholds in the constraint library based on real-time acquired RGB-D environmental observation data. and To adapt to the spatial characteristics of the current environment.

[0103] Specifically, this step involves dynamically adjusting the tolerance thresholds in the constraint library based on real-time acquired RGB-D environmental observation data. and This step is crucial for achieving high robustness and environmental adaptability in the GC-VLN method, adapting to the spatial characteristics of the current environment.

[0104] This step involves acquiring RGB-D images in real time through the environmental perception module and extracting the BEV representation and drivable area information of the current environment. The system first uses SLAM or a depth estimation model to calculate the spatial layout of the current environment, including obstacle distribution, free space regions, and the relative positions of objects. Subsequently, this information is compared with the object nodes involved in the current navigation phase. and its spatial constraints A comparison is performed to assess the degree of fit between the current constraints and the actual environment. If an environmental change is detected (such as object displacement, the addition of new obstacles, or differences in spatial structure), a dynamic adjustment mechanism for the tolerance threshold is triggered.

[0105] In an embodiment of the present invention, This indicates the tolerance range of the angle constraint, which is usually set to... This is used to allow for a certain degree of directional deviation when the robot actually turns; This represents the tolerance range for distance constraints, typically set to [0.1, 0.5] meters, allowing for a certain degree of positional error when the robot approaches a target object or waypoint. These parameters are adjusted based on the perception accuracy of the current environment, object recognition confidence, and the feasibility assessment of the navigation path. For example, when the object recognition confidence in the depth map is lower than a set threshold (e.g., 0.7), the system will appropriately relax the tolerance. This is to avoid path solving failures caused by perception errors.

[0106] This step is applicable to robots performing zero-sample navigation tasks in unknown, continuous environments. For example, in home service robots, warehouse logistics robots, or urban inspection robots, when there are discrepancies between the environmental layout and the instruction description, the system can dynamically adjust... and This ensures the feasibility and stability of navigation paths. Especially in complex indoor environments, such as when furniture is moving, passageways are narrow, or lighting changes, this mechanism can effectively mitigate navigation interruptions caused by environmental uncertainties.

[0107] This step significantly enhances the system's adaptability and fault tolerance in real-world environments. Through real-time perception and dynamic adjustment, the system can flexibly handle navigation constraints without relying on environment-specific training, thereby improving the success rate and accuracy of path solving. Furthermore, this mechanism also strengthens the navigation tree. The improved branch generation efficiency provides a more reasonable solution space basis for subsequent path backtracking and multi-solution exploration.

[0108] S6, select the path that satisfies the multiple solutions from the navigation tree. The path with the smallest function value is selected as the priority path for exploration. The function representing the minimum value of the constraint condition.

[0109] Specifically, during the construction of the navigation tree, the system uses a constraint solver to solve each navigation waypoint. Generate multiple possible coordinate solutions These solutions constitute the navigation tree. The branch nodes in the navigation path. Due to the uncertainty of the navigation path and the existence of environmental perception errors, each waypoint may correspond to multiple feasible sub-paths. In order to select the optimal exploration direction among these multiple solution paths, this invention introduces a method based on... The function path selection mechanism, where This represents the minimum value function of the constraints, used to evaluate how well the constraints are satisfied on the path.

[0110] This step involves traversing the branch paths of the navigation tree and defining the constraint functions for each path. The evaluation is performed. Each path corresponds to a series of navigation waypoint coordinate solutions, and each solution must satisfy the condition that it is related to the parent node. Spatial constraints between them. Constraint functions. By angle constraint and distance constraints The components, and their mathematical expressions are as follows:

[0111]

[0112] in, and These represent the tolerance thresholds for angle and distance, respectively, used to measure the deviation between the actual solution and the spatial relationship described in the instructions. This is achieved by calculating the minimum value of all constraint functions on each path. The system can identify the branch with the strictest constraints and the path most likely to meet the instruction requirements.

[0113] Furthermore, Typically set to to , Set as to The specific value can be adjusted based on environmental complexity and sensor accuracy. Furthermore, the route selection process must also consider the drivable area constraints of the BEV map to ensure the selected route is physically feasible.

[0114] This step is of great significance in practical applications. In continuous environments, due to the high uncertainty of navigation paths, traditional single-path planning methods are prone to getting trapped in local optima or infeasible paths. By selecting... By prioritizing the path with the shortest path, the system can explore the path that best fits the instruction semantics and spatial constraints in the multi-solution space, thereby improving the accuracy and robustness of navigation. Furthermore, this mechanism, combined with the backtracking strategy of the navigation tree, enables the system to quickly backtrack and try other branches when a path fails, significantly enhancing the fault tolerance and adaptability of zero-shot navigation.

[0115] The zero-shot visual language navigation method based on graph constraint optimization in this invention dynamically adjusts the tolerance threshold in the constraint library. and To adapt to real-time RGB-D environmental observation data, and based on the minimum constraint cost function in the navigation tree. Choosing the optimal path further enhances the navigation system's adaptability and path planning efficiency in complex and ever-changing environments.

[0116] Example 2 This invention relates to a zero-shot visual-language navigation method based on graph constraint optimization. Figure 2 This is an architecture diagram of a zero-shot visual language navigation method based on graph constraint optimization provided in an embodiment of the present invention. Figure 3 This is a path diagram illustrating a zero-shot visual language navigation method based on graph constraint optimization, provided as an embodiment of the present invention. Figure 2 and Figure 3 As shown, the method specifically includes: In one embodiment of the present invention, an innovative solution is proposed to address key problems existing in current visual language navigation technologies. Regarding the technical problems, three main difficulties are addressed: first, existing zero-shot navigation methods struggle to adapt to continuous environments, and traditional discrete environment methods cannot meet the needs of real-world scenarios; second, the spatial constraints implicit in natural language commands are difficult to extract and utilize effectively, leading to inaccurate path planning; and third, there is a lack of effective fault-tolerance mechanisms, meaning that when there are errors in environmental perception or deviations in command interpretation, the system is prone to getting stuck on incorrect paths and unable to recover. These problems severely restrict the application effectiveness and reliability of visual language navigation technology in real-world scenarios.

[0117] The core protection points of this invention focus on three key technical aspects. First, an innovative instruction decomposition method transforms natural language descriptions into structured graph constraint representations. Leveraging the semantic understanding capabilities of a large language model, navigation instructions are decomposed into directed acyclic graphs containing path points and object nodes, and spatial constraint relationships between nodes are extracted. This technology allows the spatial semantics originally implicit in natural language to be explicitly expressed, laying the foundation for subsequent accurate navigation. Second, a constraint library covering various spatial relationships is constructed, transforming common spatial relationships such as orientation and distance into computable mathematical constraints. The optimal path point coordinates are dynamically determined through a constraint solver. This method overcomes the limitations of traditional rule-based or end-to-end learning, achieving high-fidelity transformation of instruction spatial relationships. Finally, a navigation tree and backtracking mechanism are designed. By constructing a multi-possibility solution space tree, uncertainty in constraint solving is handled. When the current path is infeasible, intelligent backtracking is possible to try alternative solutions. This mechanism significantly improves the system's robustness in complex environments, resulting in a marked improvement in navigation success rate. These technological innovations together constitute a complete zero-sample visual language navigation solution, enabling accurate navigation based on natural language commands in continuous environments without requiring any environment-specific training.

[0118] The zero-shot visual language navigation method (GC-VLN) based on graph constraint optimization proposed in this invention focuses on transforming natural language instructions into computable graph constraints and achieving accurate path planning through constraint solving. The implementation process consists of three stages: First, a large language model (LLM) is used to parse the instructions and construct a multi-stage directed acyclic graph (DAG), where nodes represent path points or environmental objects, and edges represent spatial constraint relationships. Second, semantic relationships are transformed into mathematical constraints using a predefined constraint library. Finally, a nonlinear optimization method is used to solve for the node coordinates, and a navigation tree is combined to handle multiple solutions, ultimately outputting an executable navigation path. This method completely eliminates the dependence on environment-specific training data, achieving true zero-shot adaptation capability.

[0119] 1. Task definition and overall framework.

[0120] The technical solution of this invention mainly addresses the problem of zero-sample visual-language navigation for intelligent agents in unknown continuous environments. Specifically, the task is defined as follows: In an unknown 3D environment, the intelligent agent parses navigation paths described by natural language instructions (including direction, distance, and spatial relationships with objects in the environment), combines this with real-time acquired RGB-D environmental observation data, outputs a sequence of low-level control commands (forward, turn, stop, etc.), and ultimately accurately reaches the target location within a limited number of steps. The success criterion for the task is that the intelligent agent reaches a distance of d meters from the target within T steps, and no environment-specific training or fine-tuning is required throughout the entire process.

[0121] like Figure 2 As shown, the overall technical framework of this invention adopts a three-stage architecture of "instruction parsing - constraint modeling - path solving". First, a Large Language Model (LLM) is used to parse natural language instructions into structured representations, constructing a directed acyclic graph containing path point nodes, object nodes, and their spatial constraint relationships. Second, based on a predefined library of six types of spatial constraints, semantic relationships are transformed into computable mathematical constraints. Finally, a nonlinear optimization method is used to solve for the path point coordinates, and an innovative navigation tree structure manages multiple solutions, coupled with a backtracking mechanism to ensure path feasibility. This framework achieves end-to-end conversion from natural language instructions to executable paths, guaranteeing the accuracy and robustness of navigation under completely zero-sample conditions.

[0122] 2. Establishing graph constraints.

[0123] To fully utilize the information in human natural language instructions and transform them into a form suitable for zero-shot input, GC-VLN converts instructions into graph constraints that include navigation information. Furthermore, considering the sufficiency and uniqueness of instruction utilization, necessary rules are added to the instruction-to-graph-constraint transformation process, namely: (1) Instructions must not be lost. Any information in; (2) All objects mentioned in the instructions must be explicitly included; (3) The navigation direction and the spatial relationship between the object and the navigation path must be clearly given.

[0124] With the strong guarantee of these three provisions, the following steps can make full use of the navigation information in the instructions.

[0125] 3. Instruction decomposition.

[0126] Large Language Models (LLMs), guided by prompts, break down human natural language instructions into process-friendly formats. This paper requires LLMs to break down natural language instructions... The process involves multi-stage decomposition, with each stage containing only one linear displacement and one or fewer rotational movements.

[0127] To ensure the uniqueness of the decomposition, the instructions are required. Each object mentioned belongs to one and only one navigation stage. Through this process, a Directed Acyclic Graph (DAG) storing navigation information can be obtained. By topologically sorting the DAG, the multi-stage, multi-node instruction decomposition results can be transformed into a connected navigation sequence, thus naturally indicating the navigation direction and object matching order for the Agent.

[0128] The process of decomposing large models can be described as follows:

[0129] in ,in The nodes in the navigation phase can be divided into navigation waypoints. and object nodes The navigation waypoints here are the path points during the navigation process. If we consider... If there are several stages, then the nodes in the DAG can be represented as: .and This represents an edge between nodes, storing the relative positions of the nodes at both ends of the edge. Similarly, edges can be... Divided into edges connecting two waypoints and the edges connecting waypoints and object nodes .consider In each stage, the edges in the DAG can be represented as:

[0130] Here and They represent the first The waypoints of the first stage and the second Stages and objects Connected object nodes.

[0131] 4. Constraint library.

[0132] The LLM decomposes the instructions and stores the results in the corresponding DAG, which contains the most important navigation directions and landmarks. To standardize the representation of the constraints obtained from the decomposition of navigation instructions and form an optimizable form, a constraint library is constructed. This is used to summarize and represent all the constraint relationships.

[0133] The obtained DAG will be added to the constraint library according to its attributes. Query the corresponding constraint type, and then construct the specific spatial constraints within the scene based on the position of the parent node involved in the constraint. ,in This represents the coordinates of the point that needs to be optimized by constraints, while This indicates the parent node used by the constraint. For example, if it is the second type of constraint in the diagram, then the constraint can be represented as:

[0134]

[0135]

[0136] in and These represent the tolerance thresholds for the constraint in terms of angle and distance, respectively. and These represent angular and distance constraints in space, respectively. Since the optimization solution requires sequentially matching constraints step-by-step, the constraints... When solving, all parent nodes The coordinates are all known.

[0137] 5. Establishing graph constraints.

[0138] Human natural language navigation commands will eventually be transformed from natural language into a graph structure represented by normalized constraints in a constraint library, i.e., graph constraints. Nodes in the diagram Represents waypoints and object nodes, edges Used to represent the constraint relationships between nodes. This process can be represented as:

[0139] 6. Constraint solver.

[0140] Graph Constraints The navigation information in the instructions was preserved and a normalized constraint relationship was established, which was then used to solve for waypoints. The coordinates are determined by the coordinates of the object node (which acts as the parent node in the constraint relationship), the waypoint coordinates, and the specific range of the constraint.

[0141] For any waypoint The general constraint is specified as follows: These are the constraints between the previous path nodes and their corresponding object nodes. These constraints correspond to graph constraints. Zhongyu All connected edges. Therefore, the problem of finding the coordinates of this point can be transformed into a nonlinear constrained optimization problem (P1), which can be expressed as:

[0142]

[0143]

[0144] Here express The There are several solutions. And the function... and Let represent the summation function and the minimum function, respectively. For each solution, the above expression will be solved multiple times, and the result will be obtained in the th iteration. During the second solution, the first... Add a solution Within the solution set.

[0145] In addition to the optimization problem described in P1, the points obtained from the solution must also satisfy the constraints of the BEV obtained from the depth map, that is, their coordinates must be located within the drivable area of ​​the BEV and cannot be in contact with obstacles. Therefore, all solutions that satisfy the above conditions will be used as... A solution for the coordinates. To a certain extent, each solution can represent a possible exploration path in a real-world navigation process.

[0146] 7. Navigation tree.

[0147] Due to the inherent characteristics of the algorithm and issues such as errors and duplicates in object node recognition during actual navigation, a constraint solver is used to solve for any point... Each constraint solver can yield several possible solutions or no solution. In fact, all solutions obtained by the constraint solver are equal, and their quality or correctness cannot be immediately determined after the solution is completed. Furthermore, no solution only indicates that an incorrect solution was chosen when selecting a particular waypoint. Therefore, to address the issues of multiple solutions and no solution, GC-VLN introduces the concept of a navigation tree. Through the navigation tree... This is used to store the possible coordinates of all nodes in order to handle path selection and error correction.

[0148] Specifically, this paper sets graph constraints. The topological sort result is So, based on each node Number of solutions and its parent nodes on the path , It will be in the The layer is expanded to indivual Each child node of a node represents the spatial coordinates of a corresponding solution. This means that... At a waypoint, the Agent will have n branches pointing in different directions. These branches of the navigation tree represent different navigation paths. If the constraint solver can find a solution for the next spatial constraint on the current navigation path, navigation will continue, and the Agent will select a child node from the navigation tree to continue navigating. If, during navigation along a path, the region solver fails to find a solution, the node at that level will not have a branch, indicating that this specific navigation path has failed.

[0149] When a navigation path fails, it indicates that the waypoint coordinates or matched object nodes from previous stages are inaccurate. In this case, the agent will revisit the navigation tree. Backtracking involves exploring unexplored nodes that share the deepest common parent with the current node in an attempt to find the correct path. This backtracking method allows the robot to explore alternative paths without restarting the entire navigation process, giving the agent the ability to re-explore missed correct paths, while also making the entire method more robust and flexible. The navigation tree reaches the endpoint defined in the instruction DAG. Navigation ends at that time.

[0150] Example 3 This invention also provides a zero-shot visual language navigation device based on graph constraint optimization, such as... Figure 4 As shown, the device 10 includes: The instruction decomposition and graph construction module 100 is used to decompose natural language instructions in multiple stages using a large language model, construct a directed acyclic graph containing path point nodes and object nodes, where each path point node corresponds to one linear displacement and one or less rotational motion, and determines the navigation sequence through topological sorting. The semantic constraint conversion module 200 is used to convert the semantic constraint relationship in the directed acyclic graph into mathematical constraint conditions based on a predefined constraint library. The mathematical constraint conditions include angle constraints and distance constraints, and the constraint conditions of each path node are constructed based on the known coordinates of its parent node. The coordinate optimization and verification module 300 is used to solve the coordinates of path point nodes using a nonlinear optimization method, and to verify whether the coordinates meet the drivable area constraints by combining real-time acquired RGB-D environmental observation data, and to generate a navigation tree structure containing multiple possible solutions. The multi-solution path management module 400 is used to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and select an alternative path to continue navigation until the target location is reached.

[0151] Example 4 To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the methods described above.

[0152] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0153] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A zero-shot visual-language navigation method based on graph constraint optimization, characterized in that, Includes the following steps: S1 utilizes a large language model to decompose natural language instructions in multiple stages, constructs a directed acyclic graph containing path point nodes and object nodes, where each path point node corresponds to one linear displacement and one or less rotational motion, and determines the navigation sequence through topological sorting. S2, based on a predefined constraint library, the semantic constraint relationships in the directed acyclic graph are transformed into mathematical constraint conditions. The mathematical constraint conditions include angle constraints and distance constraints, and the constraint conditions of each path node are constructed based on the known coordinates of its parent node. S3. The coordinates of the path point nodes are solved by nonlinear optimization method. The coordinates are verified by combining the real-time acquired RGB-D environmental observation data to see if they meet the drivable area constraints, and a navigation tree structure containing multiple possible solutions is generated. S4 uses the multi-solution path management mechanism of the navigation tree to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and selects an alternative path to continue navigation until the target location is reached.

2. The method according to claim 1, characterized in that, The method utilizes a large language model to perform multi-stage decomposition of natural language instructions, constructing a directed acyclic graph containing path node nodes and object nodes. Each path node node corresponds to one linear displacement and one or fewer rotational movements. The navigation sequence is determined through topological sorting, including: S11 requires that each object node mentioned in the instruction must be uniquely assigned to a navigation phase, and that the path point nodes and object nodes of each navigation phase establish explicit spatial constraint relationships through edges. S12, when converting a multi-stage directed acyclic graph into a linear navigation sequence through topological sorting, the navigation direction and object matching order are determined sequentially according to the topological order of the path point nodes.

3. The method according to claim 1, characterized in that, The predefined constraint library transforms the semantic constraints in the directed acyclic graph into mathematical constraints. These mathematical constraints include angle constraints and distance constraints, and the constraints for each path node are constructed based on the known coordinates of its parent node, including: S21, transform semantic constraints into angular constraints. and distance constraints ,in and These represent the tolerance thresholds for angle and distance, respectively. S22 dynamically generates corresponding mathematical constraints based on six types of spatial relationships in the constraint library, and combines all constraints into a computable optimization objective function.

4. The method according to claim 1, characterized in that, The method employs a nonlinear optimization approach to solve for the coordinates of path points, and combines this with real-time acquired RGB-D environmental observation data to verify whether the coordinates satisfy the drivable area constraints. This generates a navigation tree structure containing multiple possible solutions, including: S31, the problem of finding the coordinates of path points is modeled as maximizing the constraint function. The nonlinear optimization problem, in which The summation function represents the constraint conditions; S32 generates a bird's-eye view using real-time acquired RGB-D environmental observation data, verifies whether the obtained coordinates are within the drivable area, and excludes solutions that are in contact with obstacles.

5. The method according to claim 1, characterized in that, The multi-solution path management mechanism using the navigation tree, when the current path is infeasible, backtracks to the unexplored path with the deepest common parent node, selects an alternative path, and continues navigation until the target location is reached, including: S41, when the current path branch has no solution in the constraint solver, backtrack to the nearest unexplored path branch with the deepest common parent node, and resolve the coordinates of subsequent path nodes based on that branch.

6. The method according to claim 1, characterized in that, Also includes: S5 dynamically adjusts the tolerance thresholds in the constraint library based on real-time acquired RGB-D environmental observation data. and To adapt to the spatial characteristics of the current environment; S6, select the path that satisfies the multiple solutions from the navigation tree. The path with the smallest function value is selected as the priority path for exploration. The function representing the minimum value of the constraint condition.

7. A zero-shot visual language navigation device based on graph constraint optimization, characterized in that, include: The instruction decomposition and graph construction module is used to decompose natural language instructions in multiple stages using a large language model, construct a directed acyclic graph containing path point nodes and object nodes, where each path point node corresponds to one linear displacement and one or less rotational motion, and determines the navigation sequence through topological sorting. The semantic constraint conversion module is used to convert the semantic constraint relationships in the directed acyclic graph into mathematical constraint conditions based on a predefined constraint library. The mathematical constraint conditions include angle constraints and distance constraints, and the constraint conditions of each path node are constructed based on the known coordinates of its parent node. The coordinate optimization and verification module is used to solve the coordinates of path point nodes using nonlinear optimization methods, and to verify whether the coordinates meet the drivable area constraints by combining real-time acquired RGB-D environmental observation data, and to generate a navigation tree structure containing multiple possible solutions. The multi-solution path management module is used to backtrack to the unexplored path with the deepest common parent node when the current path is not feasible, and select an alternative path to continue navigation until the target location is reached.

8. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a zero-sample visual language navigation method based on graph constraint optimization as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a zero-sample visual language navigation method based on graph constraint optimization as described in any one of claims 1-6.

Images