Vision and language navigation method and system based on perception enhancement and task decomposition of large language model

Abstract

The application discloses a large language model visual language navigation method and system based on perception enhancement and task decomposition, comprising: obtaining natural language navigation instructions and decomposing them into a sub-target sequence; obtaining an environment discrete view to obtain a panoramic text description set and generating a near-field object list; extracting a physical adjacency point set, constructing a legal candidate node set, and marking the access state of the candidate node; constructing a decision prompt word and adding the near-field object list; inputting the prompt word into a large language model, selecting a navigation point as a navigation action through space-time thinking chain reasoning; performing format checking and legality checking, if any checking fails, reselecting a navigation point, if both checks pass, executing the navigation action, and the agent moving and updating; verifying semantic progress conditions and near-field object confirmation conditions. The application can realize fine-grained environment perception and accurate stopping judgment, avoid the repeated exploration dead loop of the agent in the local area, and completely eliminate the illusion risk of the action space.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robot intelligent navigation technology, specifically relating to a visual language navigation method and system based on a large language model with enhanced perception and task decomposition. Background Technology

[0002] Vision-and-Language Navigation (VLN) is an important research direction in the field of Embodied AI. Its goal is to enable intelligent agents to perform path planning and autonomous navigation in complex environments based on natural language instructions. This technology integrates key technologies such as computer vision, natural language processing, and intelligent decision-making, and has broad application value in scenarios such as home service robots, intelligent inspection systems, and unmanned mobile platforms.

[0003] With the development of artificial intelligence technology, visual-language navigation methods have gradually evolved. Existing technologies mainly fall into the following categories: The first category is methods based on supervised learning or reinforcement learning, which train on manually labeled navigation trajectory data to achieve path planning and action decision-making. This type of method performs well in specific environments but is highly dependent on data and has limited generalization ability. The second category is methods based on multimodal pre-trained models, which jointly train on large-scale image and text data to align visual and linguistic information, thereby improving navigation performance. However, this type of method typically requires complex model structures and high training costs. The third category is navigation methods based on Large Language Models (LLMs), which convert visual information into textual descriptions and utilize large language models for reasoning and decision-making. This type of method has a certain degree of adaptability without requiring a large amount of task-specific training data and is gradually becoming a key research direction.

[0004] However, existing visual language navigation methods based on large language models still have the following shortcomings in practical applications: First, insufficient environmental perception and long-sequence planning capabilities. Existing methods typically rely on coarse-grained text descriptions generated from panoramic images for environmental representation, making it difficult to accurately acquire fine-grained spatial information, leading to biases in environmental understanding. Simultaneously, when processing long sequences of natural language instructions containing multiple sub-objectives, insufficient information processing capabilities easily arise, resulting in unreasonable task decomposition or discontinuous planning processes. Second, insufficient path memory capabilities. In complex topological environments, existing methods typically lack explicit mechanisms for recording historical paths, causing the decision-making process to rely primarily on current observation information and limited context, making it difficult to form a global path cognition. This easily leads to repeated movements in local areas, forming path loops and reducing navigation efficiency. Third, insufficient constraints on navigation actions. In actual navigation, the agent's action selection should be constrained by truly accessible paths in the environment. However, existing methods lack constraints on the environmental topology when generating navigation actions, potentially generating non-existent or unreachable navigation points, leading to navigation failure and affecting system stability.

[0005] Based on the above problems, there is an urgent need to provide a visual language navigation method that can improve the accuracy of environmental perception, enhance path memory capabilities, and effectively constrain navigation actions. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a visual language navigation method and system based on a large language model with enhanced perception and task decomposition. This method decomposes natural language instructions into sub-target sequences, introduces spatiotemporal thought chain execution progress analysis, spatial imagination, and cross-modal matching explicit reasoning, and combines this with near-field object detection to achieve fine-grained environmental perception and precise stop judgment. By recording the access status of each node in the navigation topology map in real time, it provides a quantified historical trajectory memory for the large language model, effectively avoiding the agent's repeated exploration loops in local areas. Furthermore, it extracts a set of physical adjacency points from the navigation environment, binds candidate nodes with local scene features, and inputs them into the large language model, completely eliminating the risk of illusions in the action space.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: Visual language navigation methods based on large language models with perceptual enhancement and task decomposition include: S1. Obtain natural language navigation instructions The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences ; S2. At the current decision-making time step of the agent, acquire discrete views of the environment and visually encode them to obtain a set of panoramic text descriptions. Then, perform instance-level object detection on the discrete views to generate a list of near-field objects. S3. Extract the current node from the agent's state in the navigation simulation environment or the real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. ; S4. Query the historical access table to obtain the access frequency of each node, and then analyze the current candidate node set. All candidate nodes Mark the access status; S5. Using prompt word constructor Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. middle; S6. Decision prompt words Input a large language model and perform three-stage explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination and cross-modal matching, and select the navigation point that best matches the expected scenario as the navigation action; S7. Perform format and validity checks on the navigation point selected in S6. If either check fails, return to S6 to select a new navigation point. If both checks pass, proceed to S8. S8. Execute the navigation action to control the agent from the current node. Move to target node Update the agent state Access status and historical trajectory It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to S2. If it has been reached, it outputs the current position as the navigation endpoint.

[0008] To optimize the above technical solution, the specific measures also include: The specific steps of S1 above are as follows: Obtain natural language navigation instructions Then input it into a large language model to extract the action transition points and target landmarks in the instructions; Based on the turning points and target landmarks of each action, organize each action into a logical sub-target sequence according to temporal dependencies. Each sub-target corresponds to a navigation action unit with a clear spatial semantics.

[0009] The aforementioned S2 specifically includes: A total of 24 discrete views were acquired in 8 horizontal directions and 3 pitch dimensions; Extracting the first through a visual encoder Visual semantic features of a discrete view And interpret it into a text description containing landmark information. This forms a panoramic text description set. ,in For the number of discrete views, Indicates the first Text descriptions corresponding to each discrete view; With the agent's current location as the center, a short-range perception threshold of 3 meters is set; Instance detection is performed on 24 discrete views, and the category label, confidence score and depth estimation distance of the detected targets in each view are extracted. Filter targets with a depth estimation distance of less than 3 meters and generate a list of near-field objects, including object category and location.

[0010] The aforementioned intelligent agents operate in a navigation simulation environment or a discrete topology graph constructed from a real environment. middle; in: Represents the set of navigable nodes in physical space; This represents the set of edges connecting adjacent nodes; At the decision-making time step The agent's state is determined by triples. Description, in which, Indicates the current node, Indicates the horizontal yaw angle. Indicates the vertical pitch angle.

[0011] The aforementioned S3 specifically includes: extracting the current node from the agent's state in the navigation simulation environment or the real environment. The complete set of physical adjacency points Each adjacent candidate node Visual semantic features in the corresponding direction Perform explicit binding to construct the current set of legal candidate nodes. , It is a candidate node.

[0012] In S4 above, if the access frequency Then candidate nodes The access status is marked as "never accessed". If the access frequency is low... Then candidate nodes The access status is marked with a warning message "accessed n times".

[0013] The aforementioned prompt word constructor presents each candidate node in the action selection block in the format of "Node ID: panoramic text description of the corresponding direction, access status". The access status follows each node entry. This allows for the application of a preset positive bias to the initial semantic similarity between the scene description and the expected scene for candidate nodes marked as "n times visited" during the cross-modal matching stage of the spatiotemporal thought chain. A preset negative bias is also applied to the initial semantic similarity between the scene description and the expected scene for candidate nodes marked as "n times visited". This adjusts the priority of each candidate node selection based on the biased semantic similarity. Simultaneously, during the progress analysis stage of the spatiotemporal thought chain, the large language model analyzes whether there is a directional deviation in the current path based on the access records of repeatedly visited nodes in the historical trajectory and provides a path correction judgment, thereby transforming the agent's exploration behavior from random walk to directional path convergence.

[0014] In step S6 above, three-stage explicit reasoning is performed through the spacetime thought chain, specifically including: Progress Analysis Phase: Based on Historical Trajectory Review the navigation steps already performed and compare them with the logical sub-target sequence. Assess the progress of each sub-objective and determine the current task progress and the sub-objectives that should be executed now. ; Spatial visualization phase: Based on the sub-goals that should be executed at present. Based on the semantic content and common sense of the physical world, we can infer and predict the type of target object, scene layout and spatial orientation relationship that should be perceived next, and form a description of the expected scene. Cross-modal matching stage: Matches the expected scene description generated in the spatial imagination stage with the current set of legal candidate nodes. The panoramic text descriptions corresponding to each candidate navigation point are compared semantically, and the navigation point that best matches the expected scenario is selected as the navigation action output.

[0015] The aforementioned S7 specifically includes: Format validation: Checks whether the navigation point ID selected by S6 conforms to the predefined format specification; if so, the format validation passes. Validity check: Verify whether the navigation point ID selected by S6 belongs to the current set of valid candidate nodes. If so, the validity check passes. If any of the above checks fail, return to S6 to select a new navigation point; The dual verification includes: Semantic progress condition verification: If the sub-target sequence is obtained through progress analysis If all sub-goals have been completed, then the semantic progress condition verification passes. Near-field object confirmation condition verification: If there is a target object in the near-field object list that is related to the final sub-target, the near-field object confirmation condition verification is successful.

[0016] This invention also provides a visual language navigation system based on a large language model using perception enhancement and task decomposition, comprising: The input processing module is used to obtain natural language navigation instructions. The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences ; The visual semanticization module is used to acquire discrete views of the environment and visually encode them at the current decision time step of the agent to obtain a set of panoramic text descriptions, and to perform instance-level object detection on the discrete views to generate a list of near-field objects. The adjacency extraction module is used to extract the current node from the agent's state in a navigation simulation environment or a real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. ; The historical access table module is used to query the historical access table to obtain the access frequency of each node, and to analyze the current candidate node set. All candidate nodes Mark the access status; The prompt word construction module is used to construct prompt words through the prompt word constructor. Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. middle; The large language model inference module is used to process decision prompts. Input a large language model and perform three-stage explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination and cross-modal matching, and select the navigation point that best matches the expected scenario as the navigation action; The execution verification module is used to perform format verification and legality verification on the navigation point selected by the large language model inference module. If either verification fails, the system returns to the large language model inference module to reselect the navigation point. If both verifications pass, the system enters the action execution module. The action execution module is used to execute the navigation action and control the agent to move from the current node. Move to target node Update the agent state Access status and historical trajectory It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to the visual semantic module. If it has been reached, it outputs the current position as the navigation endpoint.

[0017] The present invention has the following beneficial effects: (1) The present invention introduces a sub-target decomposition mechanism to divide long sequence natural language navigation instructions into sub-tasks with temporal relationships, effectively alleviating the cognitive load problem of large language models in complex navigation tasks, avoiding logical breaks in the planning process, and improving the continuity and stability of the navigation process.

[0018] (2) The present invention constructs a perception enhancement mechanism, introduces a local key object extraction strategy on the basis of panoramic semantic description, realizes fine-grained semantic expression of the environment, makes up for the problem of insufficient environmental perception granularity in traditional methods, and improves the ability to understand complex scenes.

[0019] (3) The present invention introduces a historical access table mechanism to explicitly record the number of visits to each node during the navigation process and participate in the decision-making, effectively avoiding the problem of repeated exploration by the agent in a local area, reducing the probability of path looping and improving navigation efficiency.

[0020] (4) This invention uses an adjacency navigation point selection strategy to constrain the action space of a large language model from an unbounded, freely generated space to a physically legal candidate set, thus completely eliminating the core pain point of action illusion in zero-sample large language model navigation; at the same time, by explicitly binding the candidate navigation points with the corresponding directional scene features, the large language model can perceive the local semantic information of the candidate region when selecting navigation points, thereby improving the spatial accuracy of action selection.

[0021] (5) This invention organically combines the three stages of progress analysis, spatial prediction and scene matching through the spatiotemporal thinking chain, so that the large language model can establish a clear logical link in each decision, which significantly improves the instruction compliance capability and decision interpretability in long-range navigation tasks. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the overall framework of the present invention; Figure 2 This is a schematic diagram of the sub-target decomposition and spatial prediction of the present invention; Figure 3 This is the core reasoning flowchart of the Spacetime Thinking Chain (ST-CoT) of this invention; Figure 4 This is a flowchart comparing the unconstrained generation strategy and the adjacent navigation point selection strategy of the present invention. Figure 5 This is a bar chart comparing the SPL index and SR / OSR ratio of different zero-shot methods of the present invention on the R2R validation no-seen set; Figure 6 This is a schematic diagram of the cross-modal sensing bias of the present invention; Figure 7 This is a diagram illustrating the PENav navigation case analysis in the R2R dataset of this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0025] This invention presents a visual language navigation method based on perception enhancement and task decomposition using a large language model. Based on embodied intelligence, visual language navigation, and zero-shot reasoning technology using a large language model, this invention addresses the problems of insufficient perception granularity, local dead loops, and action illusions in existing zero-shot navigation methods. It proposes three core mechanisms: a perception enhancement module, which decomposes natural language instructions into sub-target sequences, introduces spatiotemporal thought chain (ST-CoT) execution progress analysis, spatial imagination, and cross-modal matching explicit reasoning, combined with near-field object detection, to achieve fine-grained environmental perception and accurate stop judgment; a historical access table mechanism, which records the access status of each node in the navigation topology graph in real time, providing quantified historical trajectory memory for the LLM, effectively avoiding the agent's repeated exploration dead loops in local areas; and an adjacency navigation point selection strategy, which extracts a set of physical adjacency points from the navigation environment, binds candidate nodes with local scene features, and inputs them into the LLM, completely eliminating the risk of action space illusions. This solves the problems of insufficient environmental perception, path looping, and lack of physical constraints in navigation actions in complex environments found in existing visual language navigation methods. The method includes: S1. Obtain natural language navigation instructions The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences ; S2. At the current decision-making time step of the agent, acquire discrete views of the environment and visually encode them to obtain a set of panoramic text descriptions. Then, perform instance-level object detection on the discrete views to generate a list of near-field objects. S3. Extract the current node from the agent's state in the navigation simulation environment or the real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. ; S4. Query the historical access table to obtain the access frequency of each node, and then analyze the current candidate node set. All candidate nodes Mark the access status; S5. Using prompt word constructor Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. middle; S6. Decision prompt words Input a large language model and perform three-stage explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination and cross-modal matching, and select the navigation point that best matches the expected scenario as the navigation action; S7. Perform format and validity checks on the navigation point selected in S6. If either check fails, return to S6 to select a new navigation point. If both checks pass, proceed to S8. S8. Execute the navigation action, controlling the agent to move from the current node. Move to target node Update the agent state Access status and historical trajectory It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to S2. If it has been reached, it outputs the current position as the navigation endpoint.

[0026] In this embodiment, S1 obtains natural language navigation instructions. The instructions are derived using a large language model. Decomposed into sub-target sequences with temporal dependencies. The specific steps for sub-target decomposition are as follows: Original natural language navigation commands Input a large language model and have it extract the action transition points and target landmarks from the instructions; Organize multiple actions into a logical sub-target sequence based on temporal dependencies. Each sub-target corresponds to a navigation action unit with clear spatial semantics; In subsequent navigation, the sub-target sequence As a basis for explicit progress tracking, it is input into the ST-CoT inference chain along with current perceived information and historical trajectories to assist the model in accurate progress positioning and path planning.

[0027] In the embodiment, S2 is at the agent's current decision-making time step The system acquires 24 discrete views of the environment through a panoramic acquisition module, and then uses an image-text cross-modal understanding model (taking BLIP-2 as an example) to convert the images from each viewpoint into a set of text descriptions. It then calls the instance-level object detection module (taking Faster R-CNN as an example) to extract information on salient target objects within a 3-meter range of the current location; The visual semantic pipeline specifically includes: The panoramic acquisition module acquires a total of 24 discrete views in 8 horizontal directions (45° intervals) and 3 tilt dimensions; The image-text cross-modal understanding model (BLIP-2) is invoked as the core visual encoder. This model interprets local image features from various viewpoints into text descriptions containing landmark information, and the descriptions from each direction are then aggregated to form a panoramic description set. This allows LLM reasoning to be based on complete environmental priors.

[0028] The local object enhancement mechanism is as follows: With the agent's current location as the center, a short-range perception threshold of 3 meters is set; The instance-level object detection module (using the Faster R-CNN pre-trained detector as an example) is used to perform instance detection on 24 discrete views, and the class label, confidence and depth estimation distance of the detected objects in each view are extracted. Filter targets with a depth estimation distance of less than 3 meters and generate a list of near-field objects, including object category and location; The near-field object list serves as the basis for the second verification (near-field object confirmation condition) in step S8, and together with the ST-CoT progress analysis results in step S6, constitutes a dual verification mechanism for stopping detection.

[0029] Specifically, the list of near-field objects is attached separately to the cue words. In step S6, when the ST-CoT progress analysis determines that all sub-targets have been completed (first-level verification passed), the system further checks whether there are target objects related to the final sub-target in the near-field object list. If they exist, the second-level verification passes, triggering a stop detection and outputting the current position as the navigation endpoint. If they do not exist, the second-level verification fails, and the agent continues to advance, waiting for the next round of perception updates before re-performing the verification.

[0030] The intelligent agent operates in a navigation simulation environment or a real environment (taking Matterport3D simulation environment as an example) and constructs a discrete topology graph. middle; in: Represents the set of navigable nodes in physical space; Represents the set of edges connecting adjacent nodes, defining the connectivity of the agent between navigation points; At any decision time step The agent's state is determined by triples. Description, in which, Indicates the current navigation node. Indicates the horizontal yaw angle. Indicates the vertical pitch angle.

[0031] In this embodiment, S3 extracts the current node from the navigation simulation environment or the real environment (taking the Matterport3D simulation environment as an example). The complete set of physical adjacency points Each adjacent candidate node Corresponding directional visual semantic features generated in step S2 Perform explicit binding to construct the current set of legal candidate nodes. The action space of the large language model is presented to it in the form of a list, which strictly constrains the action space of the large language model from the unbounded free text generation space to a physically reachable candidate set.

[0032] In this embodiment, the historical access table mechanism of S4 is specifically as follows: The system maintains a global access count table and a topology map constructed for the navigation simulation environment or the real environment (taking Matterport3D simulation environment as an example). Each navigable node in Real-time recording of the cumulative number of times it is accessed. ; The system queries the historical access table to obtain the access frequency of each node, and then selects the current candidate node set. All candidate nodes Perform status labeling: like Then it will be marked as "Never visited", that is, the node will be marked as "Never visited" in the prompt. The message will be marked as "visited n times" and will be displayed as a warning message. ;

[0033] The status annotation is appended to each candidate node entry in the format "Node ID: panoramic text description of access status in the corresponding direction" during the prompt word construction stage. This allows the large language model to synchronously perceive the historical access status of each node while reading the candidate node list, and records the node status with explicit negative feedback annotations as... Output to the prompt word constructor; The prompt word constructor presents each candidate node in the action selection block in a triple format of "Node ID: Panoramic text description of access status in the corresponding direction". The access status label follows each node entry, with unvisited nodes labeled "Never visited" and visited nodes labeled "Visited n times". ; The status label is appended to each candidate node entry in the format of "Node ID: panoramic text description of the corresponding direction, access status" during the prompt word construction stage, so that the large language model can synchronously obtain the historical access status of each candidate node when reading the candidate node list. After each navigation action, the agent updates the historical access table, updating the number of visits to the reached node to [the new node's access count]. The current navigation node, number of visits, and corresponding decision results are written into the historical trajectory. This is for reference in subsequent decision-making steps; When a node is accessed repeatedly in the historical trajectory, the system records the node ID, the number of visits, and the movement direction when entering and leaving the node, and writes the record into the historical trajectory. .

[0034] In this embodiment, S5 is constructed by the prompt word constructor. Original instructions Sub-target sequence Environmental description Candidate Node Set Access status and historical information Integrate and construct complete decision prompts The components are defined as follows: These are original natural language navigation commands; For each sub-target sequence decomposed by the large language model preprocessing, each sub-target... Describe a navigation task within a specific semantic space; This is the set of text descriptions from various perspectives generated at the current moment by the image-text cross-modal understanding model (taking BLIP-2 as an example); This is the set of current physical adjacency candidate nodes extracted from a navigation simulation environment or a real environment (taking Matterport3D simulation environment as an example), where A unique identifier for a waypoint. Provide a scene feature description corresponding to this navigation point; This is a quantitative record of the access frequency of each candidate node at the current time step; This is to input the historical trajectory and historical decision-making information of the large language model; The cue word constructor combines the near-field object list generated by the instance-level object detection module (taking Faster R-CNN as an example) with the panoramic description. The near-field object list is placed in an independent block of the prompt words, and the prompt words clearly indicate the 3-meter perception range corresponding to the near-field object list. The near-field object list is used as the basis for judgment rather than the panoramic description in the stop detection trigger condition.

[0035] In the embodiment, S6 will display the prompt word. Inputting a large language model, three stages of explicit reasoning are performed via the Spatiotemporal Thinking Chain (ST-CoT): progress analysis, spatial imagination, and cross-modal matching, from the set of legitimate candidate nodes. The output navigation action; the three-stage explicit reasoning of the Spacetime Thinking Chain (ST-CoT) specifically includes: Progress Analysis Phase: Large Language Model Based on Historical Trajectory Review the navigation steps already performed and compare them with the sub-target sequence. Assess the progress of each sub-objective and determine the current task progress and the sub-objectives that should be executed now. ; Spatial imagination stage: The large language model is based on the current sub-goal. Based on the semantic content and common sense of the physical world, we can infer and predict the type of target object, scene layout and spatial orientation relationship that should be perceived next. Cross-modal matching stage: This stage combines the expected scene description generated in the spatial imagination stage with the candidate node set. Environmental text descriptions corresponding to each candidate navigation point Perform semantic similarity comparison and select the navigation point that best matches the expected scenario as the action output.

[0036] In this embodiment, S7 performs format and validity checks on the LLM output. If the output navigation point is not present... Within the set, real-time error feedback is injected into the large language model, returning to step S6 to request a re-selection of navigation points from the list of valid candidates; the closed-loop verification mechanism specifically includes: First-level format validation: Verify whether the navigation point IDs output by the LLM conform to the predefined format specifications; Second-level validity check: Verify whether the generated navigation point ID belongs to the current candidate set. Domain; If any of the above checks fail, the action instruction is intercepted, a real-time error feedback message is injected into the LLM, indicating that the current selection is physically unreachable, and the LLM is instructed to reselect from the list of valid candidates; The real-time error feedback mechanism ensures that every navigation action performed is within a legal physical space.

[0037] In this embodiment, S8 executes a valid navigation action, controlling the agent to move from the current node. Move to target node Update the agent state Access status and historical information If the ST-CoT progress analysis in step S6 determines the sub-target sequence If all sub-targets have been completed (first verification: semantic progress condition), then further check whether there are target objects related to the final sub-target in the near-field object list generated in step S2 (second verification: near-field object confirmation condition); if both verifications pass, then trigger stop detection and output the current position as the navigation endpoint; otherwise, if the maximum number of steps has not been reached, return to step S2 to drive the next round of loop, and if the maximum number of steps is reached, output the current position as the navigation endpoint.

[0038] This invention relates to a large language model-based visual language navigation system based on perception enhancement and task decomposition, comprising: The input processing module is used to obtain natural language navigation instructions. The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences Specifically, the input processing module receives natural language navigation instructions. Sub-target decomposition is performed using a large language model to generate a sequence of sub-targets. The output is sent to the prompt word construction module; The visual semantics module is used to acquire discrete views of the environment at the agent's current decision time step and visually encode them to obtain a set of panoramic text descriptions. It also performs instance-level object detection on the discrete views to generate a list of near-field objects. Specifically, the visual semantics module acquires a 24-view panoramic image of the agent's current location and calls an image-text cross-modal understanding model (using BLIP-2 as an example) to generate a set of text descriptions. The list of near-field objects is output to the adjacency point extraction module and the prompt word construction module. The adjacency extraction module is used to extract the current node from the agent's state in a navigation simulation environment or a real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. Specifically, the adjacency extraction module extracts the current node in real time from the navigation simulation environment or the real environment (taking the Matterport3D simulation environment as an example). physical adjacency set And bind candidate points with scene features to construct a candidate set. The output is sent to the prompt word construction module; The historical access table module is used to query the historical access table to obtain the access frequency of each node, and to analyze the current candidate node set. All candidate nodes Access status is marked; specifically, the historical access table module maintains the historical access table and records the access status of each node in real time. And manage historical trajectory information The output is sent to the prompt word construction module; The prompt word construction module is used to construct prompt words through the prompt word constructor. Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. In the middle; specifically, the prompt word construction module will instruct Sub-targets ,describe Candidate set Access status Historical Information Integrate and build decision prompts ; The large language model inference module is used to process decision prompts. The input is a large language model, which performs three stages of explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination, and cross-modal matching, selecting the navigation point that best matches the expected scenario as the navigation action; specifically, the large language model reasoning module receives prompt words. It outputs navigation actions through the ST-CoT three-stage inference mechanism; The execution verification module is used to perform format and validity checks on the navigation points selected by the large language model inference module. If either check fails, the system returns to the large language model inference module to reselect a navigation point. If both checks pass, the system enters the action execution module. Specifically, the execution verification module performs format and validity checks on the LLM output. If the checks fail, the system returns an error feedback to the large language model inference module to request a re-inference. The action execution module is used to execute the navigation action and control the agent to move from the current node. Move to target node Update the agent state Access status and historical trajectory It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to the visual semantic module. If it has been reached, it outputs the current position as the navigation endpoint.

[0039] Specifically, the action execution module executes legal navigation actions, controls the agent to move in the topology map, and updates the agent's state. Based on the stop signal output by the large language model reasoning module and the near-field object list provided by the visual semantics module, it determines whether a stop detection is triggered. If triggered, it outputs the navigation endpoint; otherwise, it feeds back the updated state information to the historical access table module and the visual semantics module to drive the next round of perception-reasoning-execution loop.

[0040] Please refer to Figures 1 to 7 This paper illustrates a complete embodiment of the method and system of the present invention. This embodiment decomposes natural language navigation instructions into sub-target sequences, constructs a visual semantic pipeline to convert multi-view images into structured text descriptions, introduces a spatiotemporal thought chain (ST-CoT) to perform three-stage explicit reasoning, combines a historical access table mechanism to quantify and memorize historical trajectories, and strictly constrains the action space of the large language model to a physically reachable candidate set through an adjacency navigation point selection strategy. This achieves accurate visual language navigation in complex 3D indoor environments under zero-sample settings, effectively improving the perception accuracy, reasoning coherence, and action reliability of the large language model in navigation tasks. It is applicable to various embodied navigation scenarios such as home service robots, intelligent inspection systems, and unmanned mobile platforms.

[0041] This embodiment proposes a visual language navigation method based on a large language model with enhanced perception and task decomposition. This method decomposes the navigation decision-making process into three collaborative stages: perception and state management, reasoning and path planning, and action execution and verification. The overall process conforms to… Figure 1 The algorithm framework is shown.

[0042] Figure 1 This is a schematic diagram of the overall visual language navigation framework of the large language model based on perception enhancement and task decomposition of the present invention. It mainly shows the complete process of input processing and sub-target decomposition, visual semanticization pipeline, adjacent navigation point extraction, historical access table management, ST-CoT inference decision and closed-loop verification execution; the "panoramic description set" shown in the figure "Corresponding to the structured text description generated in step S2 of the text, that is, the set of text representations containing landmark information obtained after interpreting 24 discrete viewpoint images of the agent's current position through an image-text cross-modal understanding model (taking BLIP-2 as an example); the "legal candidate node set" shown in the figure." "This corresponds to step S3 in the text, which extracts the set of physical adjacency points of the current node in real time from the navigation simulation environment or the real environment (taking Matterport3D simulation environment as an example). That is, the set of structured entries of the navigation nodes that the agent can actually reach at the current location and their bound scene features."

[0043] The visual language navigation method based on a large language model with perception enhancement and task decomposition proposed in this embodiment includes the following steps: S1, Input Processing Module: Receives natural language navigation instructions. The large language model is invoked to perform sub-target decomposition and generate an ordered sequence of sub-targets. The decomposition results are then passed to the prompt word construction module.

[0044] As a preferred embodiment of step S1, the specific process includes: S11, Transmit the original natural language navigation instructions Input a large language model and have it identify the turning points and key target landmarks of each action in the instruction, and extract action node information.

[0045] S12. Organize multiple action nodes into a logical sub-target sequence according to their temporal dependencies. Each sub-target This corresponds to a navigation action unit with clear spatial semantics, such as moving towards the bar, turning left at the three tables and chairs, or stopping near the sofa.

[0046] S13, Sub-target Sequence In subsequent navigation, this will serve as a basis for explicit progress tracking, in conjunction with the current perceived information. and historical trajectory The ST-CoT inference chain is used as a common input to assist the model in accurate progress positioning and path planning, effectively alleviating the problems of attention distraction and cognitive overload that are prone to occur when large language models process indoor navigation instructions that last for dozens of steps.

[0047] Taking a typical R2R dataset navigation command as an example, such as Figure 2 As shown, the instruction is to go straight along the chair or stool to the bar, turn left and go straight to the three tables and chairs, then turn left again and stop near the sofa. After LLM decomposition, an ordered sequence containing 5 sub-targets is generated. Each sub-target is accompanied by a corresponding spatial prediction description, providing a clear navigation semantic reference for subsequent ST-CoT inference.

[0048] S2, Visual Semanticization Module: Acquires a 24-view panoramic image of the agent's current location and calls an image-text cross-modal understanding model (using BLIP-2 as an example) to generate a set of text descriptions. It also integrates an instance-level object detection module (taking Faster R-CNN as an example) to extract near-field object information and outputs it to the neighbor point extraction module and the prompt word construction module.

[0049] As a preferred embodiment of step S2, the specific process includes: S21. Panoramic Acquisition: The panoramic acquisition module acquires a total of 24 discrete views in 8 horizontal directions (45° intervals) and three pitch dimensions, covering 360° environmental information of the agent's current position.

[0050] S22. Image-Text Cross-Modal Understanding Encoding (Taking BLIP-2 as an Example): The system calls the BLIP-2 model as the core visual encoder to extract the first... Visual semantic features of a discrete view And interpret it into a text description containing landmark information. This forms a panoramic text description set. .

[0051] S23, Instance-level Object Detection Near Field Detection (taking Faster R-CNN as an example): Integrate the Faster R-CNN pre-trained detector to perform instance object detection on 24 viewpoint images and extract object category, confidence, orientation and depth distance information.

[0052] S24. Local Object Enhancement Mechanism: Centered on the agent's current position, a near-field perception threshold of 3 meters is set. Targets with a depth estimation distance of less than 3 meters are filtered to generate a near-field object list (including object category and location), which is then separately appended to the prompt. In this process, the perception basis for stopping the decision is used for the second verification (near-field object confirmation condition) in step S8.

[0053] The 3-meter threshold is set based on the standard stopping condition for target determination in the R2R dataset (i.e., the agent's final position being within 3 meters of the target is considered successful), which is consistent with the task evaluation metrics, thus ensuring precise alignment between the perception range of the near-field object enhancement module and the stopping decision criterion. In other application scenarios, this threshold can be adjusted accordingly based on the stopping determination criteria of the actual navigation task.

[0054] S3, Adjacency Point Extraction Module: Extracts the current node in real time through the interface of the navigation simulation environment or the real environment (taking Matterport3D simulation environment as an example). physical adjacency set The candidate points are bound to scene features to construct the current set of legal candidate nodes. The output is sent to the prompt word construction module.

[0055] As a preferred embodiment of step S3, the specific process includes: S31, at each decision step The system directly obtains the current node through the interface of the navigation simulation environment or the real environment (taking Matterport3D simulation environment as an example). The complete set of physical adjacency points This refers to the adjacent nodes that the current location can directly reach in the real-world topology map; S32. Bind each adjacent navigation point to the corresponding directional scene features generated by the image-text cross-modal understanding model (taking BLIP-2 as an example) to form a structured entry containing node ID and scene description; S33. Construct the current set of legal candidate nodes. ,in A unique identifier for a waypoint. Provide a scene feature description corresponding to this navigation point; S34. List in format The IDs of all candidate nodes and their associated scene features are presented in the prompts, guiding the LLM to make inference choices within the range of legal navigation points. This constrains the action space faced by the LLM from an infinite free text generation space to a strictly limited set of physical reachables.

[0056] S4. Historical Access Table Module: Maintains a global historical access count table and updates the access status of each node in real time. Managing historical trajectory information Record the navigation node ID and decision summary for each step.

[0057] As a preferred embodiment of step S4, the specific process includes: S41. The system maintains a global state table and records the topology graph in real time. Each navigable node Total number of visits During initialization, the access count of all nodes is 0.

[0058] S42, The system checks the current candidate node set. All candidate nodes Perform access status annotation and generate annotated node status records: If If it is, then mark it as "not visited"; if If it is, then it will be marked as "visited". The warning label "times" is among them. The output is sent to the prompt word construction module.

[0059] S43. The access status, as the historical access information of the candidate nodes, is input into the large language model along with the scene description corresponding to the candidate nodes. This is used in the cross-modal matching stage of the spatiotemporal thought chain to apply a preset positive bias to the initial semantic similarity between the scene description corresponding to the candidate nodes marked as "n times visited" and the expected scene; and to apply a preset negative bias to the initial semantic similarity between the scene description corresponding to the candidate nodes marked as "visited n times" and the expected scene, so as to adjust the selection priority of each candidate node according to the semantic similarity after bias adjustment.

[0060] S44. After each navigation action, the agent updates the historical access table: [Order / Instructions] and will Recorded to historical trajectory This is provided for reference in subsequent decision-making steps.

[0061] When a repeatedly visited node appears in the historical trajectory, the system records the node identifier and the corresponding number of visits for the repeatedly visited node. During the progress analysis stage of the spatiotemporal thinking chain, the large language model analyzes whether there is a directional deviation in the current path based on the access records of the repeatedly visited node and provides a path correction judgment. The path correction judgment is used to reduce the semantic matching priority of the candidate node corresponding to the repeatedly visited path in the subsequent decision-making process.

[0062] By using the above-mentioned historical access status annotation and path correction judgment, the probability of the agent repeatedly accessing local areas can be reduced.

[0063] Ablation experiments showed that removing the historical access table mechanism resulted in the largest decrease in the system's SPL index (from 38.35% to 33.51%), strongly validating the important value of this mechanism in suppressing path oscillations and optimizing exploration trajectories.

[0064] S5, Prompt word construction module: This module constructs the command... Sub-target sequence Text description Candidate set Access status Historical Information Integrate according to predefined templates to construct decision prompts containing complete context. .

[0065] As a preferred embodiment of step S5, the specific process includes: S51, Transfer the original instruction With sub-target sequence Write the task description block to present each sub-objective in an ordered list as an explicit progress reference for ST-CoT progress analysis; S52, Set up panoramic text descriptions Write the environmental perception blocks in directional order, and attach the near-field object list generated in step S2 separately as the perception basis for stopping decisions; S53. Write the set of legal candidate nodes into the action selection block in the form of a list. Each entry includes "Node ID: panoramic text description of the corresponding direction, access status". The access status label is generated by step S4. Unvisited nodes are labeled as "n times". Visited nodes are labeled as "n times".

[0066] The prompts require the large language model to calculate the initial semantic similarity between the expected scene description and the scene description of each candidate node during the cross-modal matching stage; apply a preset positive bias to "unvisited" nodes and a preset negative bias to "visited n times" nodes, so as to adjust the priority of each candidate node selection based on the semantic similarity after bias adjustment; the large language model selects navigation points from the current set of legal candidate nodes for output.

[0067] By tightly binding the access status with node entries, the large language model can synchronously perceive the historical exploration status of candidate nodes when selecting navigation points, and explicitly instruct the large language model to select navigation points from this list for output.

[0068] S54. Write the historical trajectory summary managed in step S4 into the historical status block, which includes the historical navigation node ID sequence and the decision summary of each step, to provide a traceable decision context for the ST-CoT progress analysis stage. S55, Transfer historical trajectory information Write the historical trajectory block, and encapsulate blocks S51 to S54 into complete decision prompt words according to the predefined template. The output is sent to the large language model inference module.

[0069] S6, Large Language Model Reasoning Module: Receives prompt words It calls a large language model (such as GPT-4) to generate navigation action output through the ST-CoT three-stage inference mechanism, which includes the inference process text and the selected navigation point ID.

[0070] As a preferred embodiment of step S6, the specific process includes: S61. Phase One: Progress Analysis; Large language models based on historical trajectories Review the navigation steps already performed and compare them with the sub-target sequence. Evaluate the completion status of each sub-objective one by one to determine the current task progress and the sub-objectives that should be executed at the current time. .

[0071] Specifically, the model needs to answer the following questions: Preceding sub-objectives to Have all tasks been completed? Which sub-goal are we currently in? Based on the current perceived information, what is the current sub-goal? What is the completion status? The output of the progress analysis is the current sub-objective index and progress assessment report, which serve as the starting point for subsequent reasoning.

[0072] When there are repeatedly visited nodes in the historical trajectory, the large language model further analyzes whether there is a directional deviation in the current path based on the access records of the repeatedly visited nodes. When it is determined that there is a directional deviation, a path correction judgment is output to reduce the selection priority of candidate nodes corresponding to the repeatedly visited path in subsequent navigation decisions.

[0073] S62, Second Stage: Spatial Imagination; The large language model is based on the current sub-target Based on the semantic content and common sense of the physical world, it infers and predicts the type of target object, scene layout and spatial orientation relationship that should be perceived next, and generates the expected scene description.

[0074] For example, if the current sub-objective is to stop near a sofa, the spatial imagination stage will predict that a living room or rest area with casual furniture such as sofas and armchairs should appear in front, possibly accompanied by decorative elements such as coffee tables and fireplaces. This expected description will serve as a semantic reference for the next stage of cross-modal matching.

[0075] S63, Third Stage: Cross-modal matching; The expected scene description generated in the spatial imagination stage is compared with the environmental text description corresponding to each candidate navigation point in the candidate node set. For candidate nodes marked as "n times visited", a preset positive bias is applied on the basis of the initial semantic similarity. For candidate nodes marked as "n times visited", a preset negative bias is applied on the basis of the initial semantic similarity. The priority of each candidate node is adjusted based on the bias-adjusted semantic similarity. The large language model combines the current sub-goal to be executed, the environmental text description of the candidate node, and the priority of the candidate node to select a navigation point from the current set of legal candidate nodes as the navigation action output.

[0076] During the comparison process, the model needs to take into account the following: Does the scene description of the candidate node contain the expected target object? Are the orientational relationships of the candidate nodes consistent with the movement direction of the sub-target? Historical access status of candidate nodes ( (Note: Does this indicate the existence of a better direction for exploration?)

[0077] Through this progressive thinking guidance, the model can effectively establish a logical mapping between current observations, future expectations, and candidate actions, significantly improving the robustness of decision-making.

[0078] When the progress analysis determines that all sub-objectives have been completed, the large language model outputs a stop signal, which is then double-verified by the subsequent S8 action execution module in conjunction with the near-field object list. If the verification passes, the detection is stopped; otherwise, the agent continues to perform the current navigation action and enters the next round of ST-CoT inference until the stopping condition is met or the maximum number of steps is reached.

[0079] S7. Execution Verification Module: Performs format and validity checks on the LLM output, identifies and intercepts illegal navigation points with format errors or exceeding the candidate set range, injects real-time error feedback into the large language model inference module, and requires the regeneration of valid output.

[0080] As a preferred embodiment of step S7, the specific process includes: S71, Format Validation: Verifies whether the navigation point IDs output by the LLM conform to the predefined string format specification (e.g., 32-bit hexadecimal UUID format). If the format does not conform, the system injects a format error message into the LLM, requiring it to regenerate output that conforms to the specification; S72. Legality Verification: Verify whether the generated navigation point ID belongs to the current candidate set. The domain of the definition, i.e., whether the generated ID matches... A legal node in Exact match. If ID does not exist. In the middle, the system injects a real-time error feedback message into the LLM, explicitly stating that the navigation point is unreachable in the physical topology map of the current location. Please reselect from the following list of valid candidates and submit the complete... The list is presented again, forcing the model to make a second selection within the valid space.

[0081] The aforementioned closed-loop verification mechanism ensures that every actual navigation action is performed within a legal physical space through hard rules, fundamentally eliminating the risk of action illusion caused by the generative randomness of LLM, and significantly improving the stability and reliability of the system.

[0082] S8, Action Execution Module: Receives the valid navigation point ID that has passed verification, controls the agent to move from the current node to the target node in the topology graph; and determines whether to trigger stop detection based on the stop signal output by the large language model inference module and the near-field object list provided by the visual semantics module. If triggered, it outputs the navigation endpoint; otherwise, it feeds back the execution result to the historical access table module and the visual semantics module to drive the next round of perception-inference-execution loop.

[0083] As a preferred embodiment of step S8, the specific process includes: S81. After verifying the validity of the navigation point through two layers of checks, the system executes the navigation action, controlling the agent to move from the current node. Move to the selected target node ; S82, Update agent state: Let Record the new location node and orientation information; S83, Update the historical access table: Order This marks the new node as having been visited once. S84. Update Historical Information: Add the navigation point ID, selection rationale (ST-CoT reasoning summary), and perception description of this decision-making step to the historical information. This is for use in analyzing the progress of the next decision-making step; S85. Loop Check: If the ST-CoT progress analysis determines that all sub-targets have been completed (first verification passed), then further check whether there are target objects related to the final sub-target in the near-field object list (second verification). If both verifications pass, the detection will stop, the current position will be output as the navigation endpoint, and the task will end. If the detection stop is not triggered and the current step number has not exceeded the maximum step limit, then steps S2 to S8 are re-executed to enter the perception-reasoning-execution loop of the next decision time step; If the maximum number of steps is reached, output the current position as the navigation endpoint and the task ends.

[0084] The following experiment will verify this.

[0085] To verify the effectiveness of the perception enhancement, historical access table, and adjacent navigation point selection strategies proposed in this invention, this invention was thoroughly tested on the mainstream VLN benchmark dataset R2R Validation Unseen (Val-Unseen).

[0086] The R2R verification unseen set contains 11 indoor scenarios that have never appeared during the training phase, enabling an objective assessment of the agent's generalized navigation capabilities in unknown environments.

[0087] Table 1 shows the comparative experimental results of the present invention and existing methods on the R2R verification unseen set.

[0088] Table 1. Comparison of methods on the R2R validation unseen set.

[0089] like Figure 5As shown, under the pure text zero-shot setting, the method of this invention (PENav), using GPT-4 as the core decision engine, achieves navigation performance of SR=44.79% and SPL=38.35% on the R2R verification unseen set. Compared with the pioneering zero-shot method NavGPT, the SR is improved by 10.79 percentage points and the SPL by 9.35 percentage points. Compared with MapGPT (GPT-4) which introduces a topological map structure, the SR (38.80%) and SPL (25.80%) are improved by 5.99 percentage points and the SPL by 12.55 percentage points. The SR / OSR ratio (70.7%) significantly outperforms existing similar methods, fully verifying the effectiveness and accuracy of the stopping detection mechanism of this invention. Among them, Trajectory Length (TL) is the total length of the actual path traveled by the agent; Navigation Error (NE) is the shortest path distance between the agent's final stopping position and the target position; Oracle Success Rate (OSR) measures whether the agent has reached the vicinity of the target; as long as the agent enters within 3 meters of the target position at any position in the entire navigation trajectory, it is considered a success; Success Rate (SR) is the proportion of samples in the total sample whose final stopping position is within 3 meters of the target position; Success Weighted by Path Length (SPL) is based on the success rate, weighted by the ratio of the reference path length to the actual path length, comprehensively measuring the accuracy and efficiency of navigation; SR / OSR represents the ratio of success rate to oracle success rate, used to measure the agent's ability to correctly execute the stopping action after reaching the vicinity of the target; the higher the value, the more accurate the stopping decision.

[0090] Table 2 shows the ablation experiment results of each core module of the present invention.

[0091] Table 2. Ablation experiment results (R2R validation not seen set)

[0092] Ablation experiments further revealed the independent contributions of each core module: after removing the perception enhancement module, SR decreased to 34.84%; after removing the historical access table, the SPL index showed the largest decrease (down to 33.51%); after removing the adjacency navigation point selection strategy, SR decreased to 40.32%. Each module made a significant contribution to the overall performance, and the best effect was achieved when the three modules worked together.

[0093] The following is a specific navigation case analysis.

[0094] Figure 7This paper presents a complete navigation execution trajectory of PENav on the R2R dataset, with the corresponding natural language instructions containing five consecutive sub-goals. To further evaluate the model's decision-making ability at the end of a long sequence of navigation tasks, this case focuses on the agent's last two key decision steps (t=4 and t=5) at the end of the task.

[0095] In the t=4 decision step, ST-CoT progress analysis confirms that the first four sub-objectives have been completed, and the current task is to stop near the sofa (s5). The spatial imagination stage predicts that a living room or rest area with sofas, armchairs, and other casual furniture should appear ahead, and anticipates the possible presence of decorative elements such as a coffee table. The cross-modal matching stage compares this prediction with the panoramic description of the current navigable viewpoint. The scene described in the right-front direction (a living room with a fireplace, sofa, yellow chair, and stained glass sculpture) highly matches the prediction. Therefore, the agent selects the corresponding candidate node and executes the navigation action, and the historical access table is then updated with the access status of the target node.

[0096] In the t=5 decision step, after the agent enters the new location, the panoramic perception module returns an updated scene description: the right rear direction describes a living room with a coffee table and sofa, and the direct rear direction also describes a modern building lobby scene with multiple sofas and coffee tables. ST-CoT progress analysis confirms that all sub-objectives have been completed (first layer of verification passed). The system further checks the near-field object list, and the local object enhancement module detects key objects such as armchairs and fireplaces within a 3-meter range (second layer of verification passed). Both verification conditions are met, and the stopping detection mechanism integrates the dual evidence of macroscopic semantic scene and microscopic object level to confirm that the agent has reached the target area. Finally, it executes the stopping action and completes the navigation task.

[0097] This case demonstrates the synergistic advantages of the three core mechanisms of this invention: the perception enhancement module provides traceable explicit decision-making basis through the ST-CoT inference chain, and achieves accurate stop judgment with dual verification of macro-semantics and micro-objects by combining local object enhancement; the historical access table effectively avoids repeated exploration paths by marking access status; and the adjacent navigation point selection strategy ensures the legality and spatial accuracy of each navigation action by constraining candidate nodes to the physical reachable set and binding them to scene features.

[0098] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0099] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A visual language navigation method based on a large language model with perceptual enhancement and task decomposition, characterized in that, include: S1. Obtain natural language navigation instructions The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences ; S2. At the current decision-making time step of the agent, acquire discrete views of the environment and visually encode them to obtain a set of panoramic text descriptions. Then, perform instance-level object detection on the discrete views to generate a list of near-field objects. S3. Extract the current node from the agent's state in the navigation simulation environment or the real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. ; S4. Query the historical access table to obtain the access frequency of each node, and then analyze the current candidate node set. All candidate nodes Mark the access status; S5. Using prompt word constructor Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. middle; S6. Decision prompt words Input a large language model and perform three-stage explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination and cross-modal matching, and select the navigation point that best matches the expected scenario as the navigation action; S7. Perform format and validity checks on the navigation point selected in S6. If either check fails, return to S6 to select a new navigation point. If both checks pass, proceed to S8. S8. Execute the navigation action to control the agent from the current node. Move to target node Update the agent state Access status and historical trajectory ; It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to S2. If it has been reached, it outputs the current position as the navigation endpoint.

2. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, The specific steps of S1 are as follows: Obtain natural language navigation instructions Then input it into a large language model to extract the action transition points and target landmarks in the instructions; Based on the turning points and target landmarks of each action, organize each action into a logical sub-target sequence according to temporal dependencies. Each sub-target corresponds to a navigation action unit with a clear spatial semantics.

3. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, S2 specifically includes: A total of 24 discrete views were acquired in 8 horizontal directions and 3 pitch dimensions; Extracting the first through a visual encoder Visual semantic features of a discrete view And interpret it into a text description containing landmark information. This forms a panoramic text description set. ,in For the number of discrete views, Indicates the first Text descriptions corresponding to each discrete view; With the agent's current location as the center, a short-range perception threshold of 3 meters is set; Instance detection is performed on 24 discrete views, and the category label, confidence score and depth estimation distance of the detected targets in each view are extracted. Filter targets with a depth estimation distance of less than 3 meters and generate a list of near-field objects, including object category and location.

4. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, The intelligent agent operates on a discrete topology graph constructed in a navigation simulation environment or a real environment. middle; in: Represents the set of navigable nodes within physical space; This represents the set of edges connecting adjacent nodes; At the decision-making time step The agent's state is determined by triples. Description, in which, Indicates the current node, Indicates the horizontal yaw angle. Indicates the vertical pitch angle.

5. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, S3 specifically includes: extracting the current node from the agent's state from the navigation simulation environment or the real environment. The complete set of physical adjacency points Each adjacent candidate node Visual semantic features in the corresponding direction Perform explicit binding to construct the current set of legal candidate nodes. , It is a candidate node.

6. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, In S4, if the access frequency Then candidate nodes The access status is marked as "never accessed". If the access frequency is low... Then candidate nodes The access status is marked with a warning message "accessed n times".

7. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, The prompt word constructor presents each candidate node in the action selection block in the format of "node ID: panoramic text description of the corresponding direction, access status". The access status follows each node entry. In the cross-modal matching stage of the spatiotemporal thought chain, a preset positive bias is applied to the initial semantic similarity between the scene description corresponding to the candidate node marked "n-times unvisited" and the expected scene, and a preset negative bias is applied to the initial semantic similarity between the scene description corresponding to the candidate node marked "visited n times" and the expected scene. This adjusts the selection priority of each candidate node based on the semantic similarity after bias adjustment. At the same time, in the progress analysis stage of the spatiotemporal thought chain, the large language model analyzes whether there is a directional deviation in the current path based on the access records of repeatedly visited nodes in the historical trajectory and gives a path correction judgment, thereby transforming the agent's exploration behavior from random walk to directional path convergence.

8. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, The S6 process employs a three-stage explicit reasoning mechanism via a spatiotemporal thought chain, specifically including: Progress Analysis Phase: Based on Historical Trajectory Review the navigation steps already performed and compare them with the logical sub-target sequence. Assess the progress of each sub-objective and determine the current task progress and the sub-objectives that should be executed now. ; Spatial visualization phase: Based on the sub-goals that should be executed at present. Based on the semantic content and common sense of the physical world, we can infer and predict the type of target object, scene layout and spatial orientation relationship that should be perceived next, and form a description of the expected scene. Cross-modal matching stage: Matching the expected scene description generated in the spatial imagination stage with the current set of legal candidate nodes. The panoramic text descriptions corresponding to each candidate navigation point are semantically similar, and the navigation point that best matches the expected scenario is selected as the navigation action output.

9. The visual language navigation method based on a large language model with perceptual enhancement and task decomposition according to claim 1, characterized in that, Specifically, S7 includes: Format validation: Checks whether the navigation point ID selected by S6 conforms to the predefined format specification; if so, the format validation passes. Validity check: Verify whether the navigation point ID selected by S6 belongs to the current set of valid candidate nodes. If so, the validity check passes. If any of the above checks fail, return to S6 to select a new navigation point; The dual verification includes: Semantic progress condition verification: If the sub-target sequence is obtained through progress analysis If all sub-goals have been completed, then the semantic progress condition verification passes. Near-field object confirmation condition verification: If there is a target object in the near-field object list that is related to the final sub-target, the near-field object confirmation condition verification is successful.

10. A visual language navigation system based on a large language model with perceptual enhancement and task decomposition, used to implement the method described in any one of claims 1 to 9, characterized in that, The system includes: The input processing module is used to obtain natural language navigation instructions. The natural language navigation instructions are derived using a large language model. Decomposed into logical sub-target sequences ; The visual semanticization module is used to acquire discrete views of the environment and visually encode them at the current decision time step of the agent to obtain a set of panoramic text descriptions, and to perform instance-level object detection on the discrete views to generate a list of near-field objects. The adjacency extraction module is used to extract the current node from the agent's state in a navigation simulation environment or a real environment. The set of all physical adjacency nodes is used to explicitly bind each candidate adjacency node to its corresponding directional visual semantic features, thus constructing the current set of legal candidate nodes. ; The historical access table module is used to query the historical access table to obtain the access frequency of each node, and to analyze the current candidate node set. All candidate nodes Mark the access status; The prompt word construction module is used to construct prompt words through the prompt word constructor. Natural language navigation instructions Logical sub-target sequence Panoramic text description collection Current set of legal candidate nodes Access status and historical trajectory Integrate and construct decision prompts And attach a list of near-field objects separately to the cue words. middle; The large language model inference module is used to process decision prompts. Input a large language model and perform three-stage explicit reasoning through a spatiotemporal thought chain: progress analysis, spatial imagination and cross-modal matching, and select the navigation point that best matches the expected scenario as the navigation action; The execution verification module is used to perform format verification and legality verification on the navigation point selected by the large language model inference module. If either verification fails, the system returns to the large language model inference module to reselect the navigation point. If both verifications pass, the system enters the action execution module. The action execution module is used to execute the navigation action and control the agent to move from the current node. Move to target node Update the agent state Access status and historical trajectory It performs two verifications: semantic progress condition and near-field object confirmation condition. If both verifications pass, it triggers a stop detection and outputs the current position as the navigation endpoint. Otherwise, it checks whether the maximum number of steps has been reached. If not, it returns to the visual semantic module. If it has been reached, it outputs the current position as the navigation endpoint.

Images