An open-vocabulary topological semantic graph construction and navigation method and system for unknown environments

Abstract

The application belongs to the technical field of artificial intelligence and robot autonomous navigation, and discloses a method and system for open vocabulary topological semantic graph construction and navigation in unknown environment, which comprises the following steps: acquiring environment observation information at the current time and constructing a topological map composed of nodes and edges in an online incremental manner based on spatial position change and visual scene change, wherein the nodes represent key positions in the environment and the edges represent passable connections between the positions; for each node in the topological map, a corresponding visual observation image is encoded into a semantic feature vector by using a pre-trained visual language model; the semantic feature vector is fused by a graph neural network and cross-modal alignment with a natural language instruction is performed to generate a global map representation; and based on the global map representation, a navigation action is generated to drive an agent to move towards a target. The application constructs an environment representation deeply integrating spatial topology and semantic information, and makes intelligent decisions accordingly.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robot autonomous navigation technology, specifically relating to an open vocabulary topological semantic graph construction and navigation method and system for unknown environments. Background Technology

[0002] Autonomous navigation of embodied intelligent agents in unknown environments is a fundamental yet challenging task. Successful navigation heavily relies on the ability of intelligence to construct representations of its internal environment, which need to capture both spatial structure and encode rich semantic cues. Topological graphs, with nodes representing key locations and edges representing reachability, have become a mainstream, compact, and robust method for representing the environment. Building upon this, to achieve more advanced semantic navigation tasks (such as "finding the microwave in the kitchen"), researchers have developed semantic topological graphs by labeling object or scene information on topological nodes. However, despite significant progress, existing methods still face two key bottlenecks in moving towards truly intelligent and general-purpose navigation systems: (1) Limited semantic generalization ability: Existing semantic topology maps usually rely on predefined, closed sets of object categories (e.g., the 80 categories in the COCO dataset). This "closed world" assumption fails in real, diverse environments. When agents encounter objects not seen during training (e.g., "air fryer") or need to understand fine-grained instructions, existing models cannot incorporate these new concepts into their environmental representations, severely limiting their application in open-vocabulary scenarios.

[0003] (2) Lack of graph-level semantic consistency: In many methods, semantic information is merely attached to the topology graph as node labels or organized in a semantic layer separate from the topology graph. This loose coupling leads to a lack of global consistency between semantics and spatial structure. For example, when executing instructions such as "find the backpack next to the chair in the study", the agent needs to understand the spatial and semantic relationships between "study", "chair" and "backpack". Loosely coupled graph structures cannot provide this global contextual reasoning ability, limiting the effectiveness of long-distance, cross-regional semantic planning and navigation.

[0004] In summary, existing methods still face problems such as poor semantic generalization and lack of global information association. Based on the shortcomings of existing technologies, this invention proposes a topological semantic graph construction method that can be used for open-world positioning and navigation. It aims to deeply integrate semantic information and spatial topology to support robust navigation in open vocabulary scenarios. Summary of the Invention

[0005] To address the technical problems of navigation failure and low efficiency caused by limited semantic understanding and insufficient global planning capabilities of mobile robots (such as home service robots and warehouse logistics robots) in real and complex environments, this invention provides an open lexical topological semantic graph construction and navigation method and system for unknown environments. The core of this invention is to construct an environment representation that deeply integrates spatial topology and semantic information through a hardware and software collaborative system, and make intelligent decisions based on this representation.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for constructing and navigating an open lexical topological semantic graph for unknown environments, the method comprising: The system acquires environmental observation information at the current moment and, based on changes in spatial location and visual scene, incrementally constructs a topological map consisting of nodes and edges, where nodes represent key locations in the environment and edges represent traversable connections between locations. For each node in the topology map, the corresponding visual observation image is encoded into a semantic feature vector using a pre-trained visual language model; Semantic feature vectors are fused using graph neural networks and cross-modal alignment with natural language instructions is performed to generate a global map representation; Based on the global map representation, navigation actions are generated to drive the agent to move toward the target.

[0007] Preferably, the judgment condition for creating a new node based on spatial location change is: the distance between the current agent position and the position of any existing node exceeds a preset distance threshold; the judgment condition for creating a new node based on visual scene change is: the visual feature similarity between the current observed image and the corresponding image of any existing node is lower than a preset similarity threshold. The construction of edges includes connecting the new node to the previous node of the agent, and connecting the new node to existing nodes that can be traversed within a predetermined spatial range.

[0008] Preferably, for each node in the topology map, the method of encoding the corresponding visual observation image into a semantic feature vector using a pre-trained visual language model includes: For each node in the topology, the visual language model is invoked to represent each node. RGB visual observation Encode as a fixed-dimensional open-vocabulary semantic embedding vector The calculation formula is: ,in .

[0009] Preferably, the method for generating a global map representation by fusing semantic feature vectors with graph neural networks and performing cross-modal alignment with natural language instructions includes: A graph neural network is used to propagate and fuse the semantic feature vectors of all nodes to generate enhanced node features that contain global context information. Natural language navigation instructions are input into a visual language model to obtain the text feature vector of the instructions; The semantic correlation between the text feature vector of the calculation instruction and the enhanced node features containing global context information is calculated, and the features of all nodes in the topology map are weighted and fused based on the semantic correlation to obtain a global map representation aligned with the instruction.

[0010] Preferably, methods for using graph neural networks to propagate and fuse the semantic feature vectors of all nodes to generate enhanced node features containing global contextual information include: Initial semantic embedding of nodes is performed using a multi-layer graph attention network. To propagate information, given the adjacency matrix A and the initial node semantic embedding matrix... ,in, For the first N Generate context-enhanced node embeddings from the semantic embedding vectors of each node. ,in, This represents an L-layer graph attention network (GAT), where each layer of GAT updates node features through a weighted summation of neighboring features: ,in The learned neighbor attention weights, For trainable weight matrix, For activation function, For the first l +1 layer node i Features For the first l Layer nodes j Its characteristics.

[0011] Preferably, the method for calculating the semantic correlation between the text feature vector of the instruction and the enhanced node features containing global context information, and then weighting and fusing the features of all nodes in the topological map based on the semantic correlation to obtain a global map representation aligned with the instruction includes: Obtain natural language navigation instructions and encode them as text embeddings The semantic relevance score between the instruction embedding vector and the context-enhanced features of all nodes along the path is calculated using a cross-modal attention mechanism: ,in It is the first i The context-enhanced feature vectors of each node, from the output of GAT. The i OK, These are two trainable linear projection weight matrices, used to map the instruction vector and node features to the same query-key space, respectively. It is the feature dimension after projection. It is the first i Semantic relevance score between each node and instruction; The semantic relevance scores are used to weight and aggregate the context-enhanced node features to obtain the final graph representation: .

[0012] The present invention also provides an open vocabulary topological semantic graph construction and navigation system for unknown environments. The system is used to implement the aforementioned method and includes: a topological map construction module, a semantic embedding module, a graph fusion and alignment module, and a navigation decision module. The topology map construction module is used to acquire environmental observation information at the current moment and, based on changes in spatial location and visual scene, incrementally construct a topology map consisting of nodes and edges online, where nodes represent key locations in the environment and edges represent traversable connections between locations. The semantic embedding module is used to encode the corresponding visual observation image into a semantic feature vector for each node in the topology map using a pre-trained visual language model. The graph fusion and alignment module is used to perform graph neural network fusion on semantic feature vectors and perform cross-modal alignment with natural language instructions to generate a global map representation. The navigation decision module is used to generate navigation actions based on the global map representation, driving the intelligent agent to move towards the target.

[0013] Preferably, the judgment condition for creating a new node based on spatial location change is: the distance between the current agent position and the position of any existing node exceeds a preset distance threshold; the judgment condition for creating a new node based on visual scene change is: the visual feature similarity between the current observed image and the corresponding image of any existing node is lower than a preset similarity threshold. The construction of edges includes connecting the new node to the previous node of the agent, and connecting the new node to existing nodes that can be traversed within a predetermined spatial range.

[0014] Preferably, for each node in the topology map, the process of encoding the corresponding visual observation image into a semantic feature vector using a pre-trained visual language model includes: For each node in the topology, the visual language model is invoked to represent each node. RGB visual observation Encode as a fixed-dimensional open-vocabulary semantic embedding vector The calculation formula is: ,in .

[0015] Preferably, the graph fusion and alignment module includes: an enhancement unit, a transformation unit, and a fusion unit; The enhancement unit is used to propagate and fuse the semantic feature vectors of all nodes using a graph neural network to generate enhanced node features containing global context information. The conversion unit is used to input natural language navigation instructions into a visual language model to obtain the text feature vector of the instructions; The fusion unit is used to calculate the semantic correlation between the text feature vector of the instruction and the enhanced node features containing global context information, and to perform weighted fusion of the features of all nodes in the topology map based on the semantic correlation to obtain a global map representation aligned with the instruction.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Powerful open vocabulary capability: By utilizing a visual-language model, this invention can identify and represent object categories that have never been seen during training, breaking through the limitations of traditional methods that rely on a fixed list of categories, and greatly improving the semantic generalization capability in real open-world environments.

[0017] (2) Superior graph-level semantic consistency: Through the attention mechanism of context propagation and language guidance via graph attention networks, this invention achieves deep coupling between semantic information and spatial topology. This enables the constructed topological semantic graph to have global consistency, allowing the agent to perform effective long-distance, cross-regional semantic reasoning.

[0018] (3) Significantly Improved Navigation Performance: Experimental results show that on standard datasets such as MP3D and Gibson, especially in challenging tasks involving unknown objects and environments, the key indicators of this invention, such as navigation success rate and path efficiency, are significantly better than existing mainstream methods such as SPTM, TSGM, InstructNav, and GAMap. For example, in the MP3D unknown object navigation task, the success rate is significantly improved relative to the best baseline.

[0019] (4) High system integration and scalability: This invention provides a complete end-to-end solution from perception, mapping to decision-making. The modules work together and can be further improved as basic components such as vision-language models are upgraded. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a method for constructing and navigating an open lexical topological semantic graph for an unknown environment, according to an embodiment of the present invention. Figure 2 This is a system overall framework diagram of an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Example 1 like Figure 1 As shown, this invention provides a method for constructing and navigating an open lexical topological semantic graph for unknown environments, the method comprising: The agent explores the unknown environment, acquires environmental observation information at the current moment (including RGB images, depth map memory, agent pose information), and incrementally constructs a topological map composed of nodes and edges online based on changes in spatial position and visual scene. Nodes represent key locations in the environment, and edges represent traversable connections between locations. For each node in the topology map, the corresponding visual observation image is encoded into a semantic feature vector using a pre-trained visual language model; Semantic feature vectors are fused using graph neural networks and cross-modal alignment with natural language instructions is performed to generate a global map representation; Based on the global map representation, navigation actions are generated to drive the agent to move toward the target.

[0025] The specific implementation process includes: A. Online topology graph construction: Mobile robots explore their environment and perform localization simultaneously using environmental perception sensors such as RGB-D cameras and LiDAR.

[0026] An embedded processor on the robot incrementally constructs a sparse topology graph, where the node set V represents key spatial locations on the robot's exploration path, and the edge set E represents traversable path connections between nodes.

[0027] Node creation: Node Representing a key "viewpoint" or "location" in the environment, the agent evaluates at each time step during exploration whether a new node needs to be created. The creation of new nodes is based on a dual criterion: a) Spatial novelty: The Euclidean distance between the current robot pose and all existing nodes is greater than a threshold. ,Right now This principle is implemented using odometer data to ensure that the topology map effectively covers the environmental space, avoids creating redundant nodes in small areas, and saves computing and storage resources.

[0028] b) Visual Novelty: The image observed by the current RGB camera, after visual feature extraction, is compared with existing nodes within a certain spatial range. If the maximum cosine similarity of the visual embeddings of all nodes currently observed in this local region is less than a threshold... ,Right now This principle is achieved through the processor's image processing capabilities, ensuring that new nodes are created at locations where the visual appearance changes significantly (such as entering a room from a corridor), thereby more accurately characterizing the environmental structure.

[0029] If any condition is met, the processor will create a new node at the current location. And record the position of the node. and visual observation .

[0030] Create on the side: This represents the direct reachability between nodes, automatically determined by the processor based on the robot's motion trajectory and spatial reachability analysis. This includes connecting new nodes to the previous node to maintain motion continuity, and connecting new nodes to the radius. It can reach neighboring nodes through paths to form an efficient path network.

[0031] B. Node-level semantic embedding To address the semantic generalization problem that robots cannot recognize objects outside the training set (such as specific home appliances or tools) using traditional methods, this system utilizes a large-scale vision-language model deployed locally on the robot to improve semantic generalization.

[0032] For each node in the topology, call the model and assign each node... RGB visual observation Encode it as a fixed-dimensional open-vocabulary semantic embedding vector (semantic feature vector). The formula for its calculation is: ,in This embedding vector It shares the same multimodal semantic space as the text embedding vectors generated by the VLM's text encoder. This means that... Rich semantic information aligned with natural language.

[0033] By performing this operation on all nodes in the graph, we transform a purely topological graph into an open-vocabulary semantically enhanced topological graph. At this point, each node... They have all been given a flexible and powerful semantic representation. It can directly semantically match any natural language instruction in subsequent tasks without relying on any predefined label set. For example, even if the robot has never learned the category of "air fryer," the node can generate a semantic vector that highly corresponds to the textual description of "air fryer." This lays the perceptual foundation for achieving true open vocabulary navigation, solves the technical obstacle of "ignoring" unknown objects, and provides high-quality initial node features for the next stage of graph-level semantic alignment.

[0034] C. Graph-level semantic alignment To address the global consistency problem caused by the limited perspective of a single node and the isolation of semantic information, which prevents robots from performing cross-regional semantic reasoning (such as understanding the relationship between "chair in the bedroom" and "desk in the study", or "chair in the bedroom" and "chair in the living room"), this invention introduces graph neural networks for advanced information processing.

[0035] Methods for generating global map representations by fusing semantic feature vectors using graph neural networks and performing cross-modal alignment with natural language instructions include: A graph neural network is used to propagate and fuse the semantic feature vectors of all nodes to generate enhanced node features that contain global context information. Natural language navigation instructions are input into a visual language model to obtain the text feature vector of the instructions; The semantic correlation between the text feature vector of the calculation instruction and the enhanced node features containing global context information is calculated, and the features of all nodes in the topology map are weighted and fused based on the semantic correlation to obtain a global map representation aligned with the instruction.

[0036] Specifically: Contextual feature propagation: using a multi-layer graph attention network for initial semantic embedding of nodes. To propagate information, given an adjacency matrix and the initial node semantic embedding matrix ,in For the first N The semantic embedding vector of each node, ImageEncoder() is an image encoder for VLM that generates context-enhanced node embeddings (node ​​features enhanced with global context information). ,in This represents an L-layer graph attention network (GAT), where each GAT layer updates node features through a weighted summation of neighboring features: ,in For the first l +1 layer node i Features The learned neighbor attention weights, For trainable weight matrix, For the first l Layer nodes j Features is the activation function. By dynamically aggregating the semantic information of neighboring nodes according to the attention weights among the incoming nodes, an enhanced feature containing the global environmental context is generated for each node. This process deeply couples the semantic content of "what the object is" with the graph topology of "where the object is," generating a globally semantically consistent environment model.

[0037] Language-guided graph attention: Upon receiving natural language navigation instructions, these instructions are encoded into text embeddings (text feature vectors). , The semantic relevance score between the instruction embedding vector and the context-enhanced features of all nodes along the path is calculated using a cross-modal attention mechanism: ,in It is the first i The context-enhanced feature vectors of each node, from the output of GAT. The i OK, These are two trainable linear projection weight matrices, used to map the instruction vector and node features to the same query-key space, respectively. It is the feature dimension after projection. It is the first i A semantic relevance score between each node and the instruction is used. This score is then used to weight and aggregate the context-enhanced node features to obtain the final graph representation. This mechanism enables the system to dynamically and selectively activate the environmental regions most relevant to the current task, while suppressing interference from irrelevant regions, thereby focusing the global map representation on the core of the task.

[0038] D. Navigation Strategy The system represents the global graph of task focus obtained above. This data, combined with the robot's real-time visual information and historical states, is input into the navigation decision model to predict navigation actions. In this process, the system first needs to construct a state. ,in It is the visual observation feature at the current moment. This represents the hidden state of the policy network at the previous time step. This encodes the robot's current pose. Action prediction is then performed based on this information. ,in For a parameterized navigation strategy network, output the current action. and the updated hidden state The model runs on the robot's processor, outputting specific motion control commands to drive the robot's movement, and the system then performs a perception-mapping-decision cycle for the next moment. The entire system operates through a composite loss function. This function performs end-to-end training and simultaneously optimizes node-level semantic alignment, graph-level consistency, and navigation strategy. , For node-level alignment loss, For graph-level semantic consistency loss, To mitigate navigation strategy losses, ensure the accuracy of semantic perception, the consistency of the environment model, and the effectiveness of the final navigation actions, they work together to improve navigation success rate and path efficiency in complex environments.

[0039] Alternative solution: In the node-level semantic embedding part, in addition to the CLIP model, other visual-language models with open vocabulary capabilities (such as BLIP-2) can also be used to generate semantic embeddings.

[0040] In the graph-level semantic alignment part, in addition to graph attention networks, other graph neural network structures such as graph convolutional networks or graph Transformers can also be considered for the propagation of contextual information.

[0041] Example 2 Figure 2 This is the overall system framework diagram for this embodiment. The diagram illustrates how the agent processes observations and instructions through a vision-language encoder, sequentially performing online topology graph construction, node-level semantic embedding, and graph-level semantic alignment, ultimately outputting a graph representation for navigation decisions. (Where G is the topology graph, V is the node set, E is the edge set, and V...) (GAT is a graph attention network, where nodes are used for semantic enhancement.) Taking the instruction to "go to the kitchen and get an apple" in an indoor environment as an example, the implementation process of the present invention is described in detail (this process is a simplified version for illustrative purposes): (1) Exploration and Mapping: The agent moves in an unknown environment. When entering a new area (such as a living room), nodes are created according to spatial and visual novelty criteria. The visual observations are then converted into semantic embeddings using CLIP's image encoder. The embedding may contain semantic information such as "sofa" or "coffee table", but no predefined tags are required.

[0042] (2) Semantic injection and enhancement: The agent continues to explore, enters the kitchen area, and creates nodes. Its semantic embedding This may contain information such as "cabinet" or "refrigerator." The graph attention network then begins to work, focusing on... and The embedding is propagated and enhanced, making the semantic information of the "living room" and "kitchen" areas interconnected.

[0043] (3) Language guidance and decision-making: The instructions “kitchen” and “apple” are encoded as text embeddings Language-guided graph attention mechanism computation The relevance score of the (kitchen area) to the instructions was much higher than... The weighted aggregation graph representation The navigation strategy network will strongly point towards the kitchen area, and based on this, it will generate actions that move towards the kitchen.

[0044] (4) Target localization: In the kitchen, the agent creates new nodes. Its observations included "apples". At this time, The semantic embedding highly matches the word "apple" in the instruction, and the graph attention weights are directed to... Concentrate and guide the intelligent agent to ultimately reach the goal.

[0045] Example 3 The present invention also provides an open vocabulary topological semantic graph construction and navigation system for unknown environments. The system is used to implement the method described in Embodiment 1. The system includes: a topological map construction module, a semantic embedding module, a graph fusion and alignment module, and a navigation decision module. The topology map building module is used to acquire environmental observation information at the current moment and, based on changes in spatial location and visual scene, incrementally build a topology map online, consisting of nodes and edges. Nodes represent key locations in the environment, and edges represent walkable connections between locations. The semantic embedding module is used to encode the corresponding visual observation image into a semantic feature vector for each node in the topology map using a pre-trained visual language model. The graph fusion and alignment module is used to fuse semantic feature vectors using graph neural networks and perform cross-modal alignment with natural language instructions to generate a global map representation. The navigation decision module is used to generate navigation actions based on the global map representation, driving the agent to move towards the target.

[0046] In this embodiment, the judgment condition for creating a new node based on spatial location change is: the distance between the current agent position and the position of any existing node exceeds a preset distance threshold; the judgment condition for creating a new node based on visual scene change is: the visual feature similarity between the current observed image and the corresponding image of any existing node is lower than a preset similarity threshold. The construction of edges includes connecting the new node to the previous node of the agent, and connecting the new node to existing nodes that can be traversed within a predetermined spatial range.

[0047] In this embodiment, for each node in the topology map, the process of encoding the corresponding visual observation image into a semantic feature vector using a pre-trained visual language model includes: For each node in the topology, the visual language model is invoked to represent each node. RGB visual observation Encode as a fixed-dimensional open-vocabulary semantic embedding vector The calculation formula is: ,in .

[0048] In this embodiment, the graph fusion and alignment module includes: an enhancement unit, a transformation unit, and a fusion unit; The enhancement unit is used to propagate and fuse the semantic feature vectors of all nodes using a graph neural network to generate enhanced node features that contain global context information. The conversion unit is used to input natural language navigation instructions into the visual language model and obtain the text feature vector of the instructions; The fusion unit is used to calculate the semantic correlation between the text feature vector of the instruction and the enhanced node features containing global context information, and to perform weighted fusion of the features of all nodes in the topology map based on the semantic correlation to obtain a global map representation aligned with the instruction.

[0049] In this embodiment, the process of using a graph neural network to propagate and fuse the semantic feature vectors of all nodes to generate enhanced node features containing global context information includes: Initial semantic embedding of nodes is performed using a multi-layer graph attention network. To propagate information, given the adjacency matrix A and the initial node semantic embedding matrix... ,in, For the firstN Generate context-enhanced node embeddings from the semantic embedding vectors of each node. ,in, This represents an L-layer graph attention network (GAT), where each layer of GAT updates node features through a weighted summation of neighboring features: ,in The learned neighbor attention weights, For trainable weight matrix, For activation function, For the first l +1 layer node i Features For the first l Layer nodes j Its characteristics.

[0050] In this embodiment, the process of calculating the semantic correlation between the text feature vector of the instruction and the enhanced node features containing global context information, and then weighting and fusing the features of all nodes in the topology map based on the semantic correlation to obtain a global map representation aligned with the instruction includes: Obtain natural language navigation instructions and encode them as text embeddings The semantic relevance score between the instruction embedding vector and the context-enhanced features of all nodes along the path is calculated using a cross-modal attention mechanism: ,in It is the first i The context-enhanced feature vectors of each node, from the output of GAT. The i OK, These are two trainable linear projection weight matrices, used to map the instruction vector and node features to the same query-key space, respectively. It is the feature dimension after projection. It is the first i Semantic relevance score between each node and instruction; The semantic relevance scores are used to weight and aggregate the context-enhanced node features to obtain the final graph representation: .

[0051] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for constructing and navigating an open lexical topological semantic graph for unknown environments, characterized in that, The method includes: The system acquires environmental observation information at the current moment and, based on changes in spatial location and visual scene, incrementally constructs a topological map consisting of nodes and edges, where nodes represent key locations in the environment and edges represent traversable connections between locations. For each node in the topology map, the corresponding visual observation image is encoded into a semantic feature vector using a pre-trained visual language model; Semantic feature vectors are fused using graph neural networks and cross-modal alignment with natural language instructions is performed to generate a global map representation; Based on the global map representation, navigation actions are generated to drive the agent to move toward the target; Methods for generating global map representations by fusing semantic feature vectors using graph neural networks and performing cross-modal alignment with natural language instructions include: A graph neural network is used to propagate and fuse the semantic feature vectors of all nodes to generate enhanced node features that contain global context information. Natural language navigation instructions are input into a visual language model to obtain the text feature vector of the instructions; The semantic correlation between the text feature vector of the calculation instruction and the enhanced node features containing global context information is calculated, and the features of all nodes in the topology map are weighted and fused based on the semantic correlation to obtain a global map representation aligned with the instruction. Methods that use graph neural networks to propagate and fuse semantic feature vectors of all nodes to generate enhanced node features containing global context information include: Initial semantic embedding of nodes is performed using a multi-layer graph attention network. To propagate information, given the adjacency matrix A and the initial node semantic embedding matrix... ,in, For the first N Generate context-enhanced node embeddings from the semantic embedding vectors of each node. ,in, This represents an L-layer graph attention network (GAT), where each layer of GAT updates node features through a weighted summation of neighboring features: ,in The learned neighbor attention weights, For trainable weight matrix, For activation function, For the first l +1 layer node i Features For the first l Layer nodes j Features; Methods for calculating the semantic relevance between the text feature vector of the instruction and the enhanced node features containing global context information, and then weighting and fusing the features of all nodes in the topological map based on this semantic relevance to obtain a global map representation aligned with the instruction, include: Obtain natural language navigation instructions and encode them as text embeddings The semantic relevance score between the instruction embedding vector and the context-enhanced features of all nodes along the path is calculated using a cross-modal attention mechanism: ,in It is the first i The context-enhanced feature vectors of each node, from the output of GAT. The i OK, These are two trainable linear projection weight matrices, used to map the instruction vector and node features to the same query-key space, respectively. It is the feature dimension after projection. It is the first i Semantic relevance score between each node and instruction; The semantic relevance scores are used to weight and aggregate the context-enhanced node features to obtain the final graph representation: .

2. The method according to claim 1, characterized in that, The criteria for creating a new node based on changes in spatial location are: the distance between the current agent's location and the location of any existing node exceeds a preset distance threshold; the criteria for creating a new node based on changes in visual scene are: the visual feature similarity between the currently observed image and the corresponding image of any existing node is lower than a preset similarity threshold. The construction of edges includes connecting the new node to the previous node of the agent, and connecting the new node to existing nodes that can be traversed within a predetermined spatial range.

3. The method according to claim 1, characterized in that, For each node in the topological map, the method of encoding the corresponding visual observation image into a semantic feature vector using a pre-trained visual language model includes: For each node in the topology, the visual language model is invoked to represent each node. RGB visual observation Encode as a fixed-dimensional open-vocabulary semantic embedding vector The calculation formula is: ,in .

4. A system for constructing and navigating an open lexical topological semantic graph for unknown environments, the system being used to implement the method described in any one of claims 1-3, characterized in that, The system includes: a topology map construction module, a semantic embedding module, a graph fusion and alignment module, and a navigation decision module; The topology map construction module is used to acquire environmental observation information at the current moment and, based on changes in spatial location and visual scene, incrementally construct a topology map consisting of nodes and edges online, where nodes represent key locations in the environment and edges represent traversable connections between locations. The semantic embedding module is used to encode the corresponding visual observation image into a semantic feature vector for each node in the topology map using a pre-trained visual language model. The graph fusion and alignment module is used to perform graph neural network fusion on semantic feature vectors and perform cross-modal alignment with natural language instructions to generate a global map representation. The navigation decision module is used to generate navigation actions based on the global map representation, driving the intelligent agent to move towards the target.

5. The system according to claim 4, characterized in that, The criteria for creating a new node based on changes in spatial location are: the distance between the current agent's location and the location of any existing node exceeds a preset distance threshold; the criteria for creating a new node based on changes in visual scene are: the visual feature similarity between the currently observed image and the corresponding image of any existing node is lower than a preset similarity threshold. The construction of edges includes connecting the new node to the previous node of the agent, and connecting the new node to existing nodes that can be traversed within a predetermined spatial range.

6. The system according to claim 4, characterized in that, For each node in the topological map, the process of encoding the corresponding visual observation image into a semantic feature vector using a pre-trained visual language model includes: For each node in the topology, the visual language model is invoked to represent each node. RGB visual observation Encode as a fixed-dimensional open-vocabulary semantic embedding vector The calculation formula is: ,in .

7. The system according to claim 6, characterized in that, The graph fusion and alignment module includes: an enhancement unit, a transformation unit, and a fusion unit; The enhancement unit is used to propagate and fuse the semantic feature vectors of all nodes using a graph neural network to generate enhanced node features containing global context information. The conversion unit is used to input natural language navigation instructions into a visual language model to obtain the text feature vector of the instructions; The fusion unit is used to calculate the semantic correlation between the text feature vector of the instruction and the enhanced node features containing global context information, and to perform weighted fusion of the features of all nodes in the topology map based on the semantic correlation to obtain a global map representation aligned with the instruction.

Images