Visual language navigation planning method and equipment based on topological semantic map prompt

By using a topological semantic map-based approach, a 3D topological map is generated and a semantic matrix in natural language form is constructed. This solves the problems of local perspective limitations and planner inconsistency in airborne VLN missions, and achieves efficient and accurate airborne navigation planning.

CN118999554AActive Publication Date: 2024-11-22SHANGHAI ARTIFICIAL INTELLIGENCE INNOVATION CENT +1

Patent Information

Application Number
CN202410914652.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-22
Estimated Expiration
2044-07-09

AI Technical Summary

Technical Problem

Existing aerial visual language navigation methods are easily affected by local viewpoint limitations and inconsistencies between high-level LLM planners and low-level motion planners in complex outdoor environments, leading to error accumulation and navigation deviation.

Method used

A visual language navigation planning method based on topological semantic map prompts is adopted. A 2D semantic mask is generated by extracting and classifying landmarks. Combined with the position, attitude and image depth information of the UAV, a topological map of the three-dimensional point cloud space is formed. A large language model is used for navigation planning, and a semantic matrix in the form of natural language is constructed to assist LLMs in spatial reasoning.

Benefits of technology

It eliminates the need for additional action planner training, reduces computational load, improves navigation accuracy, enhances spatial awareness, effectively solves the misdirection problem in aerial VLN missions, and achieves end-to-end navigation planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118999554A_ABST
    Figure CN118999554A_ABST
Patent Text Reader

Abstract

The invention relates to a visual language navigation planning method and equipment based on topological semantic map prompt.The method comprises the steps that firstly, a long instruction is decomposed into short instructions through a large language model, specific landmark vocabularies are extracted, and landmarks are extracted and classified on an image through a multi-modal model; the method comprises the steps of generating a corresponding semantic mask and corresponding text description information, simplifying 2D semantic coding based on the text description information, then constructing / updating a semantic matrix in a natural language form in combination with unmanned aerial vehicle position attitude information and image depth information, and finally outputting a planning result based on a large language model. The invention provides an end-to-end framework based on a large language model, and the framework does not need additional training and does not need any action planner. Compared with the prior art, the method has the advantages that an action planner or additional training is not needed, the calculation amount is reduced, the derivation accuracy is improved, and the spatial perception capability is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot task planning, and in particular to a visual language navigation planning method and device based on topological semantic map prompts. Background Art

[0002] Vision and language navigation (VLN) for aerial drones is a pioneering task that enables unmanned aerial vehicles (UAVs) to navigate effectively in outdoor environments by interpreting natural language commands input by users and visual information from the first-person perspective of the drone. This technology can eliminate the need for manual operation of drones, significantly alleviate the barriers to human-drone interaction, and may be beneficial for tasks such as search and rescue and express logistics. Although VLN tasks have been widely studied, most methods are designed for indoor or ground-based outdoor environments, ignoring aerial vision-language navigation tasks. Large-scale three-dimensional spaces, complex environments, and rich semantics make it difficult for existing VLN methods to adapt to aerial urban scenes. In addition, it must be emphasized that in aerial VLN tasks, the number of potential exploration areas increases significantly as the scope of instructions increases. Wrong predictions are more likely to lead to error accumulation, ultimately leading to irreversible deviations from the correct path.

[0003] In recent years, considering the powerful reasoning ability of large language models (LLMs), some VLN methods have begun to use LLMs as agents to parse language instructions and perform high-level planning. Due to the limited spatial understanding ability of LLMs, existing studies have attempted to enhance the scene understanding ability of LLMs by using descriptive text based on visual observations. For example, the semantics of visual scenes are converted into text descriptions so that LLMs can perform high-level planning. However, in complex outdoor environments, these methods may lead to misjudgment due to the limitation of local perspective, or deviate from the correct direction due to the inconsistency between the high-level LLMs planner and the low-level action planner.

[0004] Therefore, a method is needed to solve the above two problems at the same time. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a visual language navigation planning method and device based on topological semantic map prompts, so as to solve or partially solve the problem of misjudgment caused by the limitation of local perspective, deviation from the correct direction due to inconsistency between high-level LLMs planner and low-level action planner, and thus unsatisfactory LLM reasoning accuracy.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a visual language navigation planning method based on topological semantic map prompts, comprising the following steps:

[0008] According to the input instruction information, landmark words are obtained by extraction;

[0009] Based on the landmark vocabulary and the input image information taken by the drone, a 2D semantic mask of the image and corresponding text description information are obtained by landmark extraction and classification;

[0010] Simplifying the 2D semantic mask by calculating the similarity between the vectorized landmark vocabulary and the text description information;

[0011] Based on the input drone position and posture information and image depth information, the simplified 2D semantic mask is projected into the 3D point cloud space to form a topological 2D semantic map from a bird’s-eye view, and the preset semantic matrix in natural language form is updated.

[0012] Based on the semantic matrix and the text description information, a large language model is used to implement visual language navigation planning.

[0013] As a preferred technical solution, based on the landmark vocabulary and the input image information taken by the drone, the process of obtaining the 2D semantic mask of the image and the corresponding text description information through landmark extraction and classification includes the following steps:

[0014] Based on the landmark vocabulary, a landmark extractor based on a large visual language model is used to extract and classify the input image information taken by the drone, and a description of the landmark type in the image is obtained through landmark extraction;

[0015] Based on the landmark type description, using a detection model to obtain boundary information of each landmark in the image;

[0016] Using the boundary information of each landmark in the image as a hint, the TAP model is used to segment the image to obtain the 2D semantic mask and corresponding text description information.

[0017] As a preferred technical solution, the process of simplifying the 2D semantic mask by calculating the similarity between the vectorized landmark vocabulary and the text description information comprises the following steps:

[0018] For each landmark word, the landmark word and the text description information are vectorized respectively, and the cosine similarity between the vectors is calculated. If the cosine similarity is greater than a preset threshold, the current landmark word is visible in the image information, thereby simplifying the 2D semantic mask.

[0019] As a preferred technical solution, the semantic matrix is ​​used to represent the navigation space. The size of the matrix grows dynamically with the exploration of the UAV, and the relative position and spacing between any two elements in the matrix correspond to the actual distance.

[0020] As a preferred technical solution, the process of updating the preset semantic matrix in natural language form includes the following steps:

[0021] The landmark types in the two-dimensional semantic map are mapped to the semantic matrix through nearest neighbor interpolation, the landmark types under each element in the semantic matrix are sorted according to frequency, the landmark type with the highest frequency is used as the final landmark type of the element, and the digital label of the element is updated.

[0022] As a preferred technical solution, based on the semantic matrix and the text description information, the process of implementing visual language navigation planning using a large language model includes the following steps:

[0023] Based on historical prior information, environment definition information, the semantic matrix and the text description information, prompt information matching a preset format is constructed and used as input of a large language model to obtain navigation planning results, thereby realizing visual language navigation planning.

[0024] As a preferred technical solution, the environment definition information includes environment description and output format, and the historical prior information includes sub-goal planning and historical actions.

[0025] As a preferred technical solution, the extraction of landmark vocabulary from input instruction information includes the following steps:

[0026] The input instruction information is decomposed to obtain one or more sub-goals, and landmark words are extracted for the sub-goals.

[0027] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned visual language navigation planning method based on topological semantic map prompts.

[0028] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned visual language navigation planning method based on topological semantic map prompts.

[0029] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0030] (1) No action planner or additional training is required: For the visual language navigation task of the UAV, the present invention first obtains the image information of the UAV, obtains the 2D semantic mask and the corresponding text description information through landmark extraction, detection and segmentation, simplifies the 2D semantic coding based on the text description information, and then constructs / updates the semantic matrix in natural language form by combining the UAV position and posture information and the image depth information. Finally, the planning result is output based on the large language model. The present invention provides an end-to-end framework based on the large language model. This framework does not require additional training and does not require any action planner.

[0031] (2) Reducing the amount of computation while improving the accuracy of derivation: To address the problem of over-classification, the present invention simplifies the 2D semantic mask by calculating the similarity between the vectorized landmark vocabulary and the text description information, thereby simplifying the semantic mask and reducing the misleading LLMs reasoning caused by a large number of open vocabulary categories.

[0032] (3) Strong spatial perception capability: Based on the input drone position and posture information and image depth information, the simplified 2D semantic mask is projected into the three-dimensional point cloud space to form a topological two-dimensional semantic map from a bird's-eye view and update the preset semantic matrix in natural language form. The topological structure, semantic information, and distance information between objects in the environment are unified into a matrix in natural language form, effectively assisting the large language model in spatial reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of an end-to-end framework for semantic-topological-metric representation of LLMs in an embodiment;

[0034] Figure 2 This is a schematic diagram of the 2D visual perception reasoning process in the embodiment;

[0035] Figure 3 This is a visualization diagram of drone visual language navigation using the embodiment method in a real environment in the embodiment. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] Example 1

[0038] One of the main challenges facing current LLMs is their tendency to produce hallucinations, i.e. confidently generate outputs that appear plausible but are incorrect and out of touch with reality. The hallucination problem is particularly severe for LLMs in spatial understanding and reasoning. This high confidence in erroneous outputs poses a significant challenge to visual-language navigation tasks based on LLMs. In addition, natural language instructions in real-world environments often contain a high degree of ambiguity that is either inherent or unintentional to humans, and confidently following an incorrectly constructed plan may lead to undesirable or even unsafe behavior.

[0039] In response to the aforementioned problems and the problems in the background technology section, this embodiment provides a visual language navigation planning method based on topological semantic map prompts, using a new end-to-end framework based on LLMs.

[0040] The framework takes natural language instructions, RGB images, depth maps, and pose information as input, and directly predicts low-level actions (for example, the drone flies forward 1 meter or rotates 45 degrees to the left in an aerial VLN mission) through single-step reasoning updates. The framework has strong zero-shot reasoning capabilities and does not require additional action prediction modules.

[0041] In addition, this embodiment provides a semantic-topological-metric representation (STMR) for LLMs prompts. First, the landmark vocabulary related to the instruction is extracted through LLMs, and the corresponding semantic masks are obtained through visual language models (VLMs). Then, these landmarks are projected onto a new bird's-eye view 2D semantic map using depth-semantic mapping and pose transformation, which covers the flight trajectory of the drone and spatial information, i.e., the location of surrounding objects.

[0042] In order to integrate rich visual information and topological structure into text prompts while maintaining their simplicity, this framework divides the map into grids and replaces each grid with a semantic number, thereby converting it into a matrix representation in natural language form as a spatial prompt input for LLMs. This matrix representation contains spatial topological structure information, environmental semantic information, and distance measurement information, and is input into LLMs together with information such as historical actions and sub-goal instructions to infer the next action. Compared with methods that only use natural language instruction prompts, this method effectively improves the accuracy of LLMs in spatial reasoning.

[0043] See also Figure 1, the algorithm framework of this embodiment consists of three modules, namely, sub-goal extraction, semantic-topological-metric representation, and LLM planner. First, LLMs decompose the input long instruction information into short instructions (i.e., sub-goals) and extract specific landmark vocabulary, which will be extracted and classified by the VLM model on the image, and the corresponding semantic masks will be generated through 2D visual language perception. Then, these masks are projected onto a new two-dimensional semantic map from a bird's-eye view using the drone's position and posture information and the image depth information from the first perspective. The map is further compressed into a semantic matrix. Finally, LLMs are prompted by carefully designed instructions to achieve effective mission planning.

[0044] For details, see Figure 2 It is a flowchart of 2D visual language perception, corresponding to the content of sub-target extractor and semantic-topological-metric representation. In the 2D visual language perception reasoning process, a landmark extractor (Landmark Extractor: Rec(·)) driven by a large visual language model (VLM) is first used to obtain a detailed landmark category description (Category) from a single RGB image, and then the bounding box of the target (i.e., landmark boundary information) is identified for each landmark through the detection model. Using the positioning capability of the large visual model Grounding DINO, and then using the segmentation function of the TAP model (tokenize anything via prompting, a multimodal model), a set of 2D semantic masks and corresponding text descriptions of the current RGB image are output with each bounding box as a prompt. In order to simplify the semantic mask and reduce the misleading of LLMs reasoning caused by a large number of open vocabulary categories, the present invention provides a TF-IDF-based text matching method to alleviate the problem of over-classification. Specifically, the landmark vocabulary extracted from the user's language instruction and the text description extracted by the visual perceptron for each frame of the color image are vectorized, and the cosine similarity between these vectorized landmarks and descriptions is calculated. If the similarity score exceeds a predefined threshold τ, the landmark is considered visible in the current view. By directly removing the part of the initial semantic mask that is below the threshold and retaining only the part related to the language instruction, a simplified semantic mask is obtained to simplify subsequent processing.

[0045] In this embodiment, the categories include buildings, crosswalks, roads, streets, vehicles, horses, guardrails, trees, vegetation, etc., which can be adjusted according to actual needs.

[0046] To address the problem that two-dimensional semantic masks cannot accurately express three-dimensional objects, this embodiment further combines depth images to generate accurate object segmentation, and uses the drone's position and posture information and the drone's first-person depth information to project these masks into a three-dimensional point cloud space, and then further processes them into a new bird's-eye view two-dimensional semantic map. The map is further compressed into a semantic matrix in the form of natural language.

[0047] Specifically, in order to improve the spatial reasoning ability of large language models (LLMs), this embodiment uses a semantic matrix to represent the entire navigation space. The initial size of the matrix is ​​n×n, which can grow dynamically as the drone flies and explores. The relative position relationship and spacing between each two elements correspond to the actual distance. The semantic categories on the semantic map are mapped to this matrix through the nearest neighbor interpolation algorithm, and all semantic categories falling in each element are sorted according to frequency, and only the most frequent category is displayed as the final category of the element. Each category (such as buildings and cars) is distinguished by a different digital label. In addition, unexplored areas and areas flown by drones are marked with specific numbers, and the distance between elements is also input into the LLM as metric information. In this case, the LLM can directly infer the actual position relationship between each landmark.

[0048] Finally, the LLMs are prompted with carefully designed step-by-step iterative instructions, including task definition, sub-goal planning, historical behavior, and map matrix representation, integrating these components into a formatted prompt to complement the LLM's spatial perception and scene understanding capabilities. This includes defining the environment and the UAV's task, as well as formatted input and output instructions for effective task planning.

[0049] To verify the effectiveness of this method, experiments were conducted in both simulation and real environments. The simulation experiment used the validation set of the public AerialVLN dataset as test data. Three indicators were used as evaluation criteria, namely navigation error (NE), which quantifies the distance between the drone's stop point and the actual destination; success rate (SR), which measures the proportion of navigations that successfully reach the destination within 20 meters; and idealized success rate (OSR), which considers any point on the trajectory that reaches the destination within 20 meters as an idealized measurement of success. The results are shown in Table 1.

[0050] Table 1 Simulation results

[0051]

[0052] According to Table 1, it can be seen that the method of this embodiment has significant improvements in most indicators compared with some recent visual language navigation methods.

[0053] In addition, this method has also been tested on a real machine in a real outdoor environment, proving that the proposed method can also achieve good results in the real world. Figure 3 It shows a drone following language instructions, taking off from the grass, crossing a road, then turning to cross the river, and then flying near an orange road. When large white letters are visible from the field of vision, it turns slightly left and flies over the letter "R", then flies forward and lands on the stone road. It can be seen that this method successfully achieves sub-goals at each step and finally completes the task.

[0054] In summary, this method provides an end-to-end framework based on a large language model for the visual language navigation task of UAVs. This framework does not require additional training or any action planner. In addition, it provides a semantic-topo-metric representation (STMR) mechanism based on LLMs, which unifies the topological structure, semantic information and distance information between objects in the environment into a matrix in the form of natural language, effectively assisting the large language model in spatial reasoning, and effectively solving the spatial reasoning illusion and error accumulation problems in the exploration process of UAVs.

[0055] Example 2

[0056] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the visual language navigation planning method based on topological semantic map prompts as described in Example 1.

[0057] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0058] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0059] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0060] Example 3

[0061] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the visual language navigation planning method based on topological semantic map prompts as described in Example 1.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visual language navigation planning method based on topological semantic map prompts, characterized in that: The steps include: According to the input instruction information, landmark words are obtained by extraction; Based on the landmark vocabulary and the input image information taken by the drone, a 2D semantic mask of the image and corresponding text description information are obtained by landmark extraction and classification; Simplifying the 2D semantic mask by calculating the similarity between the vectorized landmark vocabulary and the text description information; Based on the input drone position and posture information and image depth information, the simplified 2D semantic mask is projected into the 3D point cloud space to form a topological 2D semantic map from a bird’s-eye view, and the preset semantic matrix in natural language form is updated. Based on the semantic matrix and the text description information, a large language model is used to implement visual language navigation planning.

2. According to claim 1, a visual language navigation planning method based on topological semantic map prompts is characterized in that: Based on the landmark vocabulary and the input image information taken by the drone, the process of obtaining the 2D semantic mask of the image and the corresponding text description information by landmark extraction and classification includes the following steps: Based on the landmark vocabulary, a landmark extractor based on a large visual language model is used to extract and classify the input image information taken by the drone, and a description of the landmark type in the image is obtained through landmark extraction; Based on the landmark type description, using a detection model to obtain boundary information of each landmark in the image; Using the boundary information of each landmark in the image as a hint, the TAP model is used to segment the image to obtain the 2D semantic mask and corresponding text description information.

3. According to claim 1, a visual language navigation planning method based on topological semantic map prompts is characterized in that: The process of simplifying the 2D semantic mask by calculating the similarity between the vectorized landmark vocabulary and the text description information comprises the following steps: For each landmark word, the landmark word and the text description information are vectorized respectively, and the cosine similarity between the vectors is calculated. If the cosine similarity is greater than a preset threshold, the current landmark word is visible in the image information, thereby simplifying the 2D semantic mask.

4. According to claim 1, a visual language navigation planning method based on topological semantic map prompts is characterized in that: The semantic matrix is ​​used to represent the navigation space. The size of the matrix grows dynamically with the exploration of the UAV. The relative position and spacing of any two elements in the matrix correspond to the actual distance.

5. The visual language navigation planning method based on topological semantic map prompts according to claim 1 is characterized in that: The process of updating the preset semantic matrix in natural language form comprises the following steps: The landmark types in the two-dimensional semantic map are mapped to the semantic matrix through nearest neighbor interpolation, the landmark types under each element in the semantic matrix are sorted according to frequency, the landmark type with the highest frequency is used as the final landmark type of the element, and the digital label of the element is updated.

6. The visual language navigation planning method based on topological semantic map prompts according to claim 1 is characterized in that: Based on the semantic matrix and the text description information, the process of implementing visual language navigation planning using a large language model includes the following steps: Based on historical prior information, environment definition information, the semantic matrix and the text description information, prompt information matching a preset format is constructed and used as input of a large language model to obtain navigation planning results, thereby realizing visual language navigation planning.

7. The visual language navigation planning method based on topological semantic map prompts according to claim 6 is characterized in that: The environment definition information includes environment description and output format, and the historical prior information includes sub-goal planning and historical actions.

8. The visual language navigation planning method based on topological semantic map prompts according to claim 1 is characterized in that: The method of extracting landmark words from input instruction information comprises the following steps: The input instruction information is decomposed to obtain one or more sub-goals, and landmark words are extracted for the sub-goals.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the visual language navigation planning method based on topological semantic map prompts as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the visual language navigation planning method based on topological semantic map prompts as described in any one of claims 1-8.

Citation Information

Patent Citations

  • Indoor multi-balance-car autonomous navigation and scheduling system for car safe driving

    CN109029463A

  • Visual positioning method based on remote sensing images

    CN110032965A

  • Unmanned aerial vehicle semantic map construction method of deep convolutional neural network based on multi-scale prediction

    CN111798538A

  • Navigation method and device, electronic equipment and storage medium

    CN113465600A

  • Mobile robot indoor repositioning method, computer device and product

    CN114526739A

Cited By

  • Mobile robot target navigation system and navigation method thereof

    CN119879968A

  • A mobile robot target navigation system and method of navigation thereof

    CN119879968B

  • Self-adaptive robot sensing, guiding and controlling integrated system facing strange environment

    CN120085659A

  • Power grid work ticket automatic generation method, terminal device and storage medium

    CN120450396A

  • A method for automatically generating power grid work tickets, terminal equipment and storage medium

    CN120450396B