Visual language navigation planning method and device based on topological semantic map prompts

Abstract

This invention relates to a visual language navigation planning method and device based on topological semantic map cues. The method first uses a large language model to decompose long instructions into shorter instructions and extract specific landmark words. These landmarks are extracted and classified from images by a multimodal model, generating corresponding semantic masks and corresponding textual descriptions. The 2D semantic encoding is simplified based on the textual descriptions. Then, a semantic matrix in natural language form is constructed / updated by combining UAV position and attitude information and image depth information. Finally, the planning result is output based on the large language model. This invention provides an end-to-end framework based on a large language model, which requires no additional training or action planner. Compared with existing technologies, this invention has advantages such as eliminating the need for action planners or additional training, reducing computational load while improving derivation accuracy, and providing strong spatial awareness.

Description

Technical Field

[0001] This invention relates to the field of robot task planning, and in particular to a visual language navigation planning method and device based on topological semantic map prompts. Background Technology

[0002] Visual and verbal navigation (VLN) for aerial unmanned aerial vehicles (UAVs) is a groundbreaking task that enables UAVs to navigate effectively in outdoor environments by interpreting user-inputted natural language commands and first-person visual information. This technology can eliminate the need for manual operation of UAVs, significantly reducing the barriers to human-UAV interaction and potentially benefiting tasks such as search and rescue, and logistics delivery. While VLN has been extensively researched, most methods are designed for indoor or ground-based outdoor environments, neglecting aerial VLN. The large-scale three-dimensional space, complex environments, and rich semantics make existing VLN methods ill-suited for urban aerial scenarios. Furthermore, it must be emphasized that in aerial VLN, the number of potential exploration areas increases significantly with the expansion of the command range. Incorrect predictions are more likely to accumulate, ultimately leading to irreversible deviations from the correct path.

[0003] In recent years, given the powerful reasoning capabilities of Large Language Models (LLMs), some VLN methods have begun to use LLMs as agents to parse language instructions and perform high-level planning. Due to the limited spatial understanding capabilities of LLMs, existing research attempts to enhance their scene understanding capabilities by utilizing descriptive text based on visual observations. For example, converting visual scene semantics into textual descriptions to enable high-level planning by LLMs. However, in complex outdoor environments, these methods may lead to incorrect judgments due to limitations of local perspectives, or deviate from the correct direction due to inconsistencies between the high-level LLM planner and the low-level action planner.

[0004] Therefore, a method is needed to solve both of the above problems simultaneously. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a visual language navigation planning method and device based on topological semantic map prompts. This aims to solve or partially solve the problems of incorrect judgments caused by limitations of local perspectives, and deviations from the correct direction due to inconsistencies between high-level LLM planners and low-level action planners, which lead to unsatisfactory LLM inference accuracy.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] One aspect of the present invention provides a visual language navigation planning method based on topological semantic map cues, comprising the following steps:

[0008] Based on the input command information, landmark words are extracted;

[0009] Based on the landmark vocabulary and the image information captured by the drone, the 2D semantic mask of the image and the corresponding text description information are obtained through landmark extraction and classification.

[0010] The 2D semantic mask is simplified by calculating the similarity between the vectorized landmark words and the text description information;

[0011] Based on the input UAV position and attitude information and image depth information, the simplified 2D semantic mask is projected onto the 3D point cloud space to form a topological bird's-eye view 2D semantic map, and the preset natural language semantic matrix is ​​updated.

[0012] Based on the semantic matrix and the text description information, a large-scale language model is used to realize visual language navigation planning.

[0013] As a preferred technical solution, the process of obtaining the 2D semantic mask and corresponding text description information of the image based on the landmark vocabulary and the input image information captured by the drone includes the following steps:

[0014] Based on the landmark vocabulary, a landmark extractor based on a large visual language model is used to extract and classify the landmark information captured by the drone, and the landmark type description in the image is obtained through landmark extraction.

[0015] Based on the landmark type description, the boundary information of each landmark in the image is obtained using a detection model;

[0016] Using the boundary information of each landmark in the image as a cue, the TAP model is used to segment the image to obtain a 2D semantic mask and corresponding text description information.

[0017] As a preferred technical solution, the process of simplifying the 2D semantic mask by calculating the similarity between vectorized landmark words and text description information includes the following steps:

[0018] For each landmark word, the landmark word and the text description information are vectorized separately, and the cosine similarity between the vectors is calculated. If the cosine similarity is greater than a preset threshold, the current landmark word is visible in the image information, thus simplifying the 2D semantic mask.

[0019] As a preferred technical solution, the semantic matrix is ​​used to represent the navigation space. The size of the matrix grows dynamically as the UAV explores, and the relative position and spacing between any two elements in the matrix correspond to the actual distance.

[0020] As a preferred technical solution, the process of updating the preset semantic matrix in natural language form includes the following steps:

[0021] The landmark types in the two-dimensional semantic map are mapped to the semantic matrix by nearest neighbor interpolation. The landmark types under each element in the semantic matrix are sorted according to frequency. The landmark type with the highest frequency is taken as the final landmark type of the element, and the numerical label of the element is updated.

[0022] As a preferred technical solution, the process of visual language navigation planning based on the semantic matrix and the text description information using a large language model includes the following steps:

[0023] Based on historical prior information, environmental definition information, the semantic matrix, and the text description information, prompt information matching a preset format is constructed and used as input to a large language model to obtain navigation planning results, thereby realizing visual language navigation planning.

[0024] As a preferred technical solution, the environment definition information includes an environment description and an output format, and the historical prior information includes sub-goal planning and historical actions.

[0025] As a preferred technical solution, the extraction of landmark words from the input instruction information includes the following steps:

[0026] The input instruction information is decomposed to obtain one or more sub-targets, and landmark words are extracted from the sub-targets.

[0027] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned visual language navigation planning method based on topological semantic map cues.

[0028] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the aforementioned visual language navigation planning method based on topological semantic map cues.

[0029] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0030] (1) No action planner or additional training required: For visual language navigation tasks of UAVs, this invention first acquires the image information of the UAV, obtains 2D semantic mask and corresponding text description information through landmark extraction, detection and segmentation, and simplifies the 2D semantic encoding based on the text description information. Then, it constructs / updates the semantic matrix in natural language form by combining the UAV position and attitude information and image depth information. Finally, it outputs the planning result based on a large language model. This invention provides an end-to-end framework based on a large language model. This framework does not require additional training or any action planner.

[0031] (2) Reduce computational load while improving inference accuracy: To address the problem of overclassification, this invention simplifies the 2D semantic mask by calculating the similarity between vectorized landmark words and text description information, thereby simplifying the semantic mask and reducing LLMs inference misleading caused by a large number of open word categories.

[0032] (3) Strong spatial perception capability: Based on the input UAV position and attitude information and image depth information, the simplified 2D semantic mask is projected onto the three-dimensional point cloud space to form a topological bird's-eye view two-dimensional semantic map and update the preset natural language semantic matrix. The topological structure, semantic information and distance information between objects of the environment are unified into a natural language matrix, which effectively assists the large language model in spatial reasoning. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the end-to-end framework of semantic-topology-metric representation of LLMs in the embodiment;

[0034] Figure 2 This is a schematic diagram of the 2D visual perception reasoning process in the embodiment;

[0035] Figure 3 This is a visualization of UAV visual language navigation using the method described in the embodiment in a real-world environment. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] Example 1

[0038] One of the major challenges facing LLMs today is their tendency to generate illusions—that is, to confidently produce seemingly reasonable but incorrect and unrealistic outputs. This illusion problem is particularly severe in LLMs' spatial understanding and reasoning. This high degree of confidence in erroneous outputs poses a significant challenge to LLM-based visual language navigation tasks. Furthermore, natural language instructions in real-world environments often contain inherent or unintentional high levels of ambiguity, and confidently following a flawed plan can lead to maladaptive or even unsafe behavior.

[0039] To address the aforementioned issues and the problems in the background section, this embodiment provides a visual language navigation planning method based on topological semantic map prompts, employing a novel end-to-end framework based on LLMs.

[0040] The framework takes natural language commands, RGB images, depth maps, and pose information as inputs and performs low-level action prediction directly through single-step inference updates (e.g., in an aerial VLN mission, a drone flying forward 1 meter or rotating 45 degrees to the left). This framework has powerful zero-shot inference capabilities and does not require an additional action prediction module.

[0041] Furthermore, this embodiment provides a semantic-topological-metric representation (STMR) for LLMs prompts. First, landmark words related to instructions are extracted using LLMs, and corresponding semantic masks are obtained using visual language models (VLMs). Then, these landmarks are projected onto a new bird's-eye view two-dimensional semantic map using depth-semantic mapping and pose transformation. This map encompasses the UAV's flight path and spatial information, i.e., the positions of surrounding objects.

[0042] To integrate rich visual information and topological structure into text prompts while maintaining simplicity, this framework divides the map into a grid and replaces each grid with a semantic number, thereby converting it into a matrix representation in natural language form, which serves as the spatial prompt for LLMs. This matrix representation includes spatial topological structure information, environmental semantic information, and distance metric information, and is input into LLMs along with information such as historical actions and sub-target instructions to infer the next action. Compared to methods that only use natural language instruction prompts, this method effectively improves the accuracy of LLMs in spatial reasoning.

[0043] See Figure 1The algorithm framework in this embodiment consists of three modules: sub-target extraction, semantic-topology-metric representation, and LLM planner. First, LLMs decompose the input long instruction information into short instructions (i.e., sub-targets) and extract specific landmark words. These landmarks are extracted and classified on the image by the VLM model, generating corresponding semantic masks through 2D visual language perception. Then, these masks are projected onto a new bird's-eye view 2D semantic map using the UAV's position and attitude information and first-view image depth information. This map is further compressed into a semantic matrix. Finally, the LLMs are given carefully designed instruction prompts to achieve effective task planning.

[0044] For details, see Figure 2 This is a flowchart for 2D visual language perception, corresponding to the sub-target extractor and the semantic-topological-metric representation. In the 2D visual language perception inference process, a Landmark Extractor (Rec(·)) driven by a large visual language model (VLM) is first used to obtain detailed landmark category descriptions from a single RGB image. Then, a detection model identifies the bounding box (i.e., landmark boundary information) for each landmark. Utilizing the localization capabilities of the large visual model Grounding DINO, and then employing the segmentation function of the TAP model (tokenize anything via prompting, a multimodal model), each bounding box serves as a cue, outputting a set of 2D semantic masks and corresponding text descriptions for the current RGB image. To simplify the semantic masks and reduce inference misleading by a large number of open-ended vocabulary categories in LLMs, this invention provides a TF-IDF-based text matching method to mitigate overclassification. Specifically, landmark words extracted from user language commands and text descriptions extracted from each frame of color image via a visual perceptron are vectorized, and the cosine similarity between these vectorized landmarks and descriptions is calculated. If the similarity score exceeds a predefined threshold τ, the landmark is considered visible in the current view. By directly removing the portion of the initial semantic mask below the threshold and retaining only the part related to language instructions, a simplified semantic mask is obtained, thus simplifying subsequent processing.

[0045] In this embodiment, the categories include buildings, pedestrian crossings, roads, streets, vehicles, horses, guardrails, trees, vegetation, etc., and can be adjusted according to actual needs.

[0046] To address the issue that two-dimensional semantic masks cannot accurately represent three-dimensional objects, this embodiment further combines depth images to generate precise object segmentation. These masks are then projected onto a three-dimensional point cloud space using the UAV's position and attitude information and its first-person perspective depth information. This is further processed into a new bird's-eye view two-dimensional semantic map. This map is then further compressed into a semantic matrix in natural language form.

[0047] Specifically, to improve the spatial reasoning capabilities of Large Language Models (LLMs), this embodiment uses a semantic matrix to represent the entire navigation space. The matrix initially has an n×n size and dynamically grows as the UAV explores. The relative positional relationships and spacing between each pair of elements correspond to actual distances. A nearest neighbor interpolation algorithm maps semantic categories on the semantic map to this matrix. All semantic categories falling within each element are sorted by frequency, and only the most frequent category is displayed as the final category for that element. Each category (e.g., buildings and cars) is distinguished by a different numerical label. Furthermore, unexplored areas and areas flown by the UAV are identified with specific numbers, and the distances between elements are also input into the LLM as metric information. In this case, the LLM can directly infer the actual positional relationships between each landmark.

[0048] Finally, by providing carefully designed iterative prompts to LLMs, including task definition, sub-goal planning, historical behavior, and map matrix representation, these components are integrated into a formatted prompt to supplement the LLM's spatial awareness and scene understanding capabilities. This includes defining the environment and UAV's tasks, as well as formatted input / output instructions to achieve effective task planning.

[0049] To verify the effectiveness of this method, experiments were conducted in both simulation and real-world environments. The simulation experiments used the validation set of the publicly available AerialVLN dataset as test data. Three metrics were used as evaluation standards: navigation error (NE), which quantifies the distance between the UAV's stopping point and the actual destination; success rate (SR), which measures the proportion of navigation operations that successfully reach the destination within 20 meters; and idealized success rate (OSR), an idealized measurement that considers any point on the trajectory that reaches the destination within 20 meters as a success. The results are shown in Table 1.

[0050] Table 1 Simulation Experiment Results

[0051]

[0052] As can be seen from Table 1, the method in this embodiment has significantly improved over some recent visual language navigation methods in most metrics.

[0053] Furthermore, this method was also tested in a real outdoor environment, demonstrating that it can achieve good results in the real world. Figure 3 The demonstration showed a drone following voice commands, taking off from a grassy area, crossing a road, making a slight turn to cross a river, and then flying to the vicinity of an orange road. When a large white letter became visible, it made a slight left turn and flew past the letter "R," then continued forward and landed on a cobblestone path. It can be seen that this method successfully achieved the sub-objectives at each step and ultimately completed the mission.

[0054] In summary, this method provides an end-to-end framework based on a large language model for visual language navigation tasks of UAVs. This framework requires no additional training or action planner. Furthermore, it provides a Semantic-Topo-Metric Representation (STMR) mechanism based on LLMs, unifying the topological structure, semantic information, and distance information between objects in the environment into a matrix in natural language form. This effectively assists the large language model in spatial reasoning and effectively solves the problems of spatial reasoning illusion and error accumulation during UAV exploration.

[0055] Example 2

[0056] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the visual language navigation planning method based on topological semantic map prompts as described in Embodiment 1.

[0057] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0058] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0059] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0060] Example 3

[0061] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing a visual language navigation planning method based on topological semantic map prompts as described in Embodiment 1.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual language navigation planning method based on topological semantic map cues, characterized in that, Includes the following steps: Based on the input command information, landmark words are extracted; Landmark category descriptions are extracted from a single RGB image using a landmark extractor driven by a large visual language model; By using the detection model, the bounding box of each landmark is identified, the visual large model is used for localization, the TAP model is used for segmentation, and each bounding box is used as a cue to output a set of 2D semantic masks and corresponding text description information of the current RGB image. The landmark words and text description information are vectorized, the cosine similarity between the vectorized landmarks and descriptions is calculated, and the part with similarity score below the threshold in the semantic mask is removed to obtain the simplified 2D semantic mask. Based on the input UAV position and attitude information and image depth information, the simplified 2D semantic mask is projected onto the 3D point cloud space to form a topological bird's-eye view 2D semantic map, and the preset natural language semantic matrix is ​​updated. Based on the semantic matrix and the text description information, a large-scale language model is used to realize visual language navigation planning.

2. The visual language navigation planning method based on topological semantic map cues according to claim 1, characterized in that, The semantic matrix is ​​used to represent the navigation space. The size of the matrix grows dynamically as the UAV explores. The relative position and spacing between any two elements in the matrix correspond to the actual distance.

3. The visual language navigation planning method based on topological semantic map prompts according to claim 1, characterized in that, The process of updating the preset semantic matrix in natural language form includes the following steps: The landmark types in the two-dimensional semantic map are mapped to the semantic matrix by nearest neighbor interpolation. The landmark types under each element in the semantic matrix are sorted according to frequency. The landmark type with the highest frequency is taken as the final landmark type of the element, and the numerical label of the element is updated.

4. The visual language navigation planning method based on topological semantic map cues according to claim 1, characterized in that, Based on the semantic matrix and the text description information, the process of implementing visual language navigation planning using a large language model includes the following steps: Based on historical prior information, environmental definition information, the semantic matrix, and the text description information, prompt information matching a preset format is constructed and used as input to a large language model to obtain navigation planning results, thereby realizing visual language navigation planning.

5. The visual language navigation planning method based on topological semantic map cues according to claim 4, characterized in that, The environmental definition information includes environmental description and output format, and the historical prior information includes sub-goal planning and historical actions.

6. The visual language navigation planning method based on topological semantic map prompts according to claim 1, characterized in that, The extraction of landmark words from the input instruction information includes the following steps: The input instruction information is decomposed to obtain one or more sub-targets, and landmark words are extracted from the sub-targets.

7. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the visual language navigation planning method based on topological semantic map cues as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the visual language navigation planning method based on topological semantic map cues as described in any one of claims 1-6.

Images