Advanced instruction understanding and long-range navigation method based on visual language model

Abstract

The invention provides an advanced instruction understanding and long-range navigation method based on a visual language model, and the method comprises the steps: carrying out the semantic object reasoning and spatial position prediction based on an advanced instruction and a global three-dimensional scene with annotations through a visual language model, and obtaining a target region; determining a target waypoint in the local boundary of the target area through a random sampling mode; and based on the panoramic view of the target waypoint, through the intelligent model with the body, carrying out detection and positioning to obtain the position information of the target object. Compared with the large difference between a navigation task and a real world demand in the prior art, the method comprises the following steps: performing semantic object reasoning and spatial position prediction through a visual language model, and determining the position of a target object; based on the panoramic view of the target waypoint, through the intelligent model with the body, detection and positioning are carried out, and a target object is determined; understanding of a high-level abstract instruction is realized, and a target object and a possible position are automatically reasoned; and the navigation capability of the robot in a complex real environment is obviously improved.

Description

Technical Field

[0001] This invention relates to the field of embodied navigation, and more particularly to a high-level command understanding and long-range navigation method based on a visual language model. Background Technology

[0002] Embodied navigation is a fundamental capability of embodied intelligence, crucial for robots to perform complex tasks in physical environments. This capability enables autonomous agents to navigate and interact in real-world space, laying the foundation for more complex embodied behaviors such as manipulation, exploration, and human-robot collaboration. Despite significant progress in this field, current research has primarily focused on relatively low-level tasks, such as following instructions and basic object navigation, which fail to adequately capture the nuances of human needs in dynamic environments.

[0003] Current embodied navigation methods can be broadly categorized into two main types: Visual and Verbal Navigation (VLN) and Object Navigation (ObjectNav). VLN tasks require agents to follow detailed, step-by-step instructions, such as "turn left, go out, then go straight." While these tasks require precise spatial understanding, they often rely on overly specific instructions that humans rarely provide in natural environments. In contrast, ObjectNav tasks aim to locate predefined object categories, such as "find any chair in the scene," and successfully execute whenever an instance of the target object is encountered, regardless of the spatial context or specific requirements.

[0004] However, real-world human commands often involve high-level intentions requiring complex reasoning and spatial awareness. For example, requests like "I want a cup of coffee" or "I want to eat the fruit on the left side of the tea room" not only require understanding the underlying goal but also reasoning about the spatial relationships between objects. This highlights a fundamental gap between current navigation tasks and real-world needs, severely hindering the development of embodied agents capable of advanced human-computer interaction. Summary of the Invention

[0005] This invention provides a high-level instruction understanding and long-range navigation method based on a visual language model, which addresses the shortcomings of existing technologies where there is a large gap between navigation tasks and real-world requirements, and significantly improves the robot's navigation capabilities in complex real-world environments.

[0006] This invention provides a high-level instruction understanding and long-range navigation method based on a visual language model, comprising the following steps.

[0007] Based on high-level instructions and annotated global 3D scenes, the target region is obtained by performing semantic object reasoning and spatial location prediction through a visual language model. Within the local boundaries of the target area, target waypoints are determined through random sampling. Based on the panoramic view of the target waypoint, the location information of the target object is obtained through detection and localization using an embodied intelligent model.

[0008] According to the present invention, a high-level instruction understanding and long-range navigation method based on a visual language model is provided. Based on high-level instructions and annotated global 3D scenes, semantic object reasoning and spatial location prediction are performed using a visual language model to obtain the target region, including: By using a visual language model, high-level instructions are semantically decomposed to infer the target object; Using a visual language model, based on annotated global 3D scenes and target objects, spatial semantic relationship analysis is performed to identify the target region where the target object is located.

[0009] According to the advanced command understanding and long-range navigation method based on a visual language model provided by the present invention, before obtaining the position information of the target object by detecting and locating it through an embodied intelligence model based on a panoramic view of the target waypoint, the method further includes: A panoramic view of the target waypoint is captured through a rotating scan of the embodied agent.

[0010] According to the present invention, a high-level command understanding and long-range navigation method based on a visual language model is provided. Based on a panoramic view of the target waypoint, an embodied intelligence model is used to detect and locate the target object, thereby obtaining its position information. The method includes: Based on the panoramic view of the target waypoint, the target center point coordinates are obtained through detection and localization using an embodied intelligent model. The target center point coordinates are converted into robot coordinates using the camera's intrinsic functions; The position information of the target object is determined based on the robot's coordinates.

[0011] According to the present invention, a high-level command understanding and long-range navigation method based on a visual language model is provided. Based on a panoramic view of the target waypoint, an embodied intelligent model is used for detection and localization to obtain the coordinates of the target center point, including: Based on the panoramic view of the target waypoints, the system uses an embodied intelligent model to detect and locate multiple target points, and outputs the coordinates of each point. The coordinates of the target center point are obtained by averaging the coordinates of multiple target points.

[0012] According to the present invention, a high-level command understanding and long-range navigation method based on a visual language model is provided. Based on a panoramic view of the target waypoint, an embodied intelligence model is used to detect and locate the target object, thereby obtaining its position information. The method includes: Based on the panoramic view of the target waypoint, detection and localization are performed through an embodied intelligent model; In the absence of a target object, a new target waypoint is obtained based on the annotated global 3D scene and target region through a two-stage decision processing method. Using an embodied intelligent model, based on a panoramic view of the new target waypoint, the location information of the target object is obtained through detection and localization.

[0013] According to the present invention, a high-level command understanding and long-range navigation method based on a visual language model obtains new target waypoints based on an annotated global 3D scene and target region through a two-stage decision processing method, including: The analysis results are obtained by using a visual language model based on annotated global 3D scene and target region. If the analysis results indicate the need to explore a new region, a new target region is obtained by using a visual language model to perform semantic object reasoning and spatial location prediction in the region outside the target region, based on high-level instructions and annotated global 3D scenes. Within the local boundaries of the new target region, new target waypoints are determined through random sampling. If the analysis results indicate that the target area should be explored further, a new target waypoint is determined through random sampling within the local boundary of the target area, among waypoints other than the target waypoint.

[0014] The present invention also provides a high-level instruction understanding and long-range navigation device based on a visual language model, comprising the following modules: The target region determination module is used to obtain the target region based on high-level instructions and annotated global 3D scenes, through a visual language model, semantic object reasoning and spatial location prediction. The target waypoint determination module is used to determine the target waypoint within the local boundary of the target area by means of random sampling; The target object location determination module is used to detect and locate the target object based on the panoramic view of the target waypoint and through an embodied intelligent model to obtain the target object's location information.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the advanced instruction understanding and long-range navigation method based on the visual language model as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the advanced instruction understanding and long-range navigation method based on the visual language model as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the advanced instruction understanding and long-range navigation method based on a visual language model as described above.

[0018] This invention provides a high-level instruction understanding and long-range navigation method based on a visual language model. Based on high-level instructions and annotated global 3D scenes, it uses a visual language model to perform semantic object reasoning and spatial location prediction to obtain the target region. Within the local boundaries of the target region, target waypoints are determined through random sampling. Based on a panoramic view of the target waypoints, an embodied intelligence model is used to detect and locate the target object, obtaining its position information. Compared to existing technologies where there is a significant gap between navigation tasks and real-world requirements, this method first uses a visual language model for semantic object reasoning and spatial location prediction to obtain the target region, i.e., the possible location of the target object. Then, using an embodied intelligence model, based on a panoramic view of the target waypoints in the target region, it detects and locates the target object, determining its position. This achieves the understanding of high-level abstract instructions and automatically infers the target object and its possible location, significantly improving the robot's navigation capabilities in complex real-world environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the advanced instruction understanding and long-range navigation method based on a visual language model provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of the advanced instruction understanding and long-range navigation device based on a visual language model provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The following is combined Figures 1-3 This invention is described.

[0025] Figure 1 This is a flowchart illustrating the advanced command understanding and long-range navigation method based on a visual language model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Based on advanced instructions and annotated global 3D scene, semantic object reasoning and spatial location prediction are performed through visual language model to obtain the target region.

[0026] In step 101 above, in order to address the challenges of long-range navigation, this invention proposes NavA. 3 This is a novel hierarchical framework that decomposes more complex embodied behaviors, such as manipulation, exploration, and human-machine collaboration, into two phases: global policy and local policy. The global policy uses Reasoning-VLM to interpret high-level instructions, such as "dry clothes" → clothes hanger; and leverages 3D scene understanding to identify the target location, such as the balcony. The local policy uses Pointing-VLM to navigate waypoints and employs the embodied intelligence model of this invention, such as the NaviAfford model, for precise object localization, which utilizes spatial affordance understanding to accurately locate the target object, the clothes hanger.

[0027] Reasoning-VLM identifies key objects that need to be located based on these instructions and uses an annotated global 3D scene to determine the space where the target object is most likely to appear. For example, when given the instruction "I want a cup of coffee", the global policy infers that the coffee machine is likely located in the break room, thus guiding the agent to this high-probability area.

[0028] Optionally, step 101 includes steps A1 to A3: Step A1: Using a visual language model, perform semantic decomposition on high-level instructions to infer the target object.

[0029] Step A2: Using a visual language model, based on the annotated global 3D scene and target object, perform spatial semantic relationship analysis to identify the target region where the target object is located.

[0030] In steps A1 to A2 above, the global strategy leverages the advanced reasoning capabilities of the visual language model (Reasoning-VLM) to bridge the semantic gap between advanced human instructions and navigation goals.

[0031] This invention treats the global reasoning task as a multimodal problem, where Reasoning-VLM simultaneously performs semantic object reasoning and spatial location prediction. To support systematic reasoning, this invention designs a structured prompt template to effectively guide Reasoning-VLM: "You need to complete the human instruction: I. Now, given this top-down scene view Sglobal and several selectable areas, think about what object you should find to complete the instruction, and where you should look for that object. Please show your thought process and give your answer at the end." Reasoning-VLM processes textual instructions and visual representations of annotated global scenes to achieve hierarchical reasoning. It first infers the target object O* required to complete the instruction through semantic decomposition: O* = f semantic (I). Then, the model analyzes spatial semantic relationships and identifies the target region R∗ where the target object is most likely located, defined as R∗ = arg maxRj ∈S global P(O∗|R j A j ), where P(O∗|R j A j ) indicates that in region R j Find the annotation A j The conditional probability of O*.

[0032] Step 102: Within the local boundary of the target area, determine the target waypoint by random sampling.

[0033] In step 102 above, after identifying the target region R∗, the present invention randomly samples a waypoint w ∈ R∗ within its local boundary and uses Pointing-VLM to guide the agent. This strategy promotes robust exploration while effectively narrowing the search space to relevant sub-regions where the target object may be located, thereby improving the efficiency of the search process.

[0034] Optionally, prior to step 103, the advanced instruction understanding and long-range navigation method based on a visual language model further includes step B1: Step B1: Capture a panoramic view of the target waypoint through a rotational scan of the embodied agent.

[0035] Step 103: Based on the panoramic view of the target waypoint, use the embodied intelligent model to detect and locate the target object to obtain its position information.

[0036] In step 103 above, after the global policy is completed, the local policy takes over, focusing on exploration and precise object localization within the identified target area. The VLM, called Pointing-VLM, selects pathpoints from the local 3D scene for exploration. At each pathpoint, the invention performs panoramic perception and utilizes a specially trained NaviAfford model (an implementation of Pointing-VLM) for precise target object recognition.

[0037] If a target object is detected, the invention transforms its position from the agent's perspective to the robot's coordinate system, thereby enabling navigation to the final target. The NaviAfford model is trained on a spatial object availability dataset containing 1 million sample pairs, enabling spatially aware object and availability-based localization. This allows the model to understand complex spatial relationships, such as "the cup by the window" or "the empty seat on the left side of the table."

[0038] Optionally, step 103 includes steps C1 to C3: Step C1: Based on the panoramic view of the target waypoint, use the embodied intelligent model to detect and locate the target center point coordinates.

[0039] Step C2: Convert the target center point coordinates to robot coordinates using the camera's intrinsic functions.

[0040] Step C3: Determine the position information of the target object based on the robot's coordinates.

[0041] Optionally, step C1 includes steps C11 to C12: Step C11: Based on the panoramic view of the target waypoints, use the embodied intelligent model to detect and locate multiple target points, and output the coordinates of multiple target points.

[0042] Step C12: Average the coordinates of multiple target points to obtain the coordinates of the target center point.

[0043] In steps C1 to C3 above, within the local strategy, the system of this invention employs a fine-grained object localization and navigation strategy based on systematic waypoint exploration. The agent captures a panoramic RGB view of each waypoint through rotational scanning. The NaviAfford model processes these views to detect and accurately locate the target object. After detecting the target object, the model outputs multiple point coordinates; this invention selects the center point by averaging to achieve robust localization. To convert pixel coordinates to robot coordinates, this invention uses camera intrinsic functions: Among them, f x and f yThese are the focal lengths, c x and c y Here, represents the principal point, and d represents the depth at pixel (u, v). This ensures effective navigation to the target object. Next, the invention uses rotation and translation to convert the camera coordinates to robot coordinates: Among them, (x r , y r , θ r (x) represents the robot's world pose. robot , y robot Derived from camera coordinates: x robot = Z cam , y robot = −X cam .

[0044] Optionally, step 103 includes steps D1 to D3: Step D1: Based on the panoramic view of the target waypoint, detection and localization are performed using an embodied intelligent model.

[0045] Step D2: If the target object is not located, a new target waypoint is obtained based on the annotated global 3D scene and target region using a two-stage decision processing method.

[0046] Step D3: Using the embodied intelligent model, based on the panoramic view of the new target waypoint, detect and locate the target object to obtain its position information.

[0047] In steps D1 through D3 above, if no target object is detected, the system will follow a two-stage decision-making process. First, Reasoning-VLM analyzes the local 3D scene and historical exploration data to decide whether to continue exploring the current area or transition to a new area. If the choice is to continue, the NaviAfford model will determine the next optimal exploration point. Otherwise, it will select the most promising room or space to explore based on previous searches, thus achieving an efficient transition.

[0048] Optionally, step D2 includes steps D21 to D23: Step D21: Analyze the annotated global 3D scene and target region using a visual language model to obtain the analysis results.

[0049] Step D22: If the analysis result indicates the exploration of a new region, a new target region is obtained by performing semantic object reasoning and spatial location prediction in the region outside the target region using a visual language model, based on high-level instructions and annotated global 3D scene; within the local boundary of the new target region, a new target waypoint is determined by random sampling.

[0050] Step D23: If the analysis result indicates that the target area should be explored further, a new target waypoint is determined by random sampling within the local boundary of the target area and among waypoints other than the target waypoint.

[0051] The NavA of the present invention 3 The framework employs a hierarchical global-to-local approach, combining semantic reasoning with precise spatial localization to handle long-view navigation tasks. The global strategy utilizes Reasoning-VLM to interpret advanced human commands (e.g., "I want a cup of coffee"), inferring the target object (coffee machine) and identifying possible rooms (e.g., tea room, kitchen area). Upon reaching the room, the local strategy uses the present invention's NaviAfford model (Pointing-VLM) to analyze the panoramic RGB observations and local map for each waypoint. This model determines whether the target object exists; if found, navigation proceeds to its location; if not found, the next optimal waypoint is predicted or the Reasoning-VLM is referenced to continue exploration until the target is found.

[0052] Extensive experimental evaluations have shown that NavA 3 It achieves optimal performance in long-distance navigation tasks in large-scale real-world environments. Furthermore, the system of this invention exhibits superior cross-instance capabilities, enabling it to adapt to various robot instances and highlighting its potential for practical applications.

[0053] To further explain the present invention, the following specific experimental examples are provided.

[0054] Experiment 1 To evaluate the performance of long-range navigation, this invention established a benchmark comprising five different scenarios: Conference Room A, Conference Room B, Tea Room, Workstation, and Balcony. Each scenario contained 10 navigation tasks, for a total of 50 tasks. For each method, this invention deployed each task 10 times to minimize randomness. Human experts defined high-level instructions and associated semantic objects to ensure the uniqueness of target objects in each scenario. Each task was tested five times under different initial conditions to ensure reliability. During execution, the agent is free to interact with the environment, utilizing egocentric RGB-D perception, waypoint selection, and action control. To evaluate different Pointing-VLM models, this invention selected 1,000 images not present in the training set.

[0055] This invention employs two standard metrics in embodied navigation: Navigation Error (NE) and Success Rate (SR). NE measures the Euclidean distance (in meters) between the agent's final position and the target; a lower value indicates better performance. SR reflects the percentage of navigation events in which the agent successfully reaches the target (defined as within 1 meter). This invention calculates the SR for 10 tasks, testing each task 5 times (for a total of 50 trials), and reports the average SR (Avg. SR). Furthermore, for PointVLM evaluation, this invention uses Accuracy (Acc), defined as the ratio of correctly predicted points within the ground reality mask to the total number of predicted points.

[0056] Existing navigation methods often struggle with long-field-of-view tasks involving advanced human commands. To ensure fair comparison, this invention modifies their task statements and command formats to make them more explicit: “You need to perform the following command: I want to drink coffee. Find the target object and perform the command, then stop near it.” Furthermore, this invention provides top-down global 3D scene information for the baseline models. This invention evaluates three types of baseline models: (1) closed-source general-purpose VLMs, including GPT-4o, etc.; (2) open-source general-purpose VLMs, such as Janus-Pro-7B, etc.; and (3) navigation-specific methods, including NaVid, etc., which require adaptation for long-range navigation tasks according to this invention.

[0057] Experimental results show that NavA 3 It significantly outperformed existing best-in-class methods in all evaluation scenarios, improving the success rate (SR) by 41.2 percentage points to an average of 66.4%, while the best baseline MapNav (Zhang et al. 2025b) had a success rate of only 25.2%. Specifically, NavA 3SR increased by 46.0% in Meeting Room A (72.0% vs. 26.0%), by 40.0% in Meeting Room B (64.0% vs. 24.0%), by 34.0% in the Tea Room (60.0% vs. 26.0%), by 48.0% in the Workstations (76.0% vs. 28.0%), and by 38.0% in the Balcony (60.0% vs. 22.0%). It also significantly reduced navigation errors (NE) across all scenarios: 5.98 meters in Conference Room A (1.23 meters vs. 7.21 meters), 6.49 meters in Conference Room B (1.45 meters vs. 7.94 meters), 7.23 meters in the tea room (1.89 meters vs. 9.12 meters), 5.22 meters in the workstation (1.56 meters vs. 6.78 meters), and 6.11 meters in the balcony (1.34 meters vs. 7.45 meters). While general-purpose VLMs (including closed-source and open-source) typically have near-zero success rates in this challenging long-range navigation task, the hierarchical approach of this invention effectively bridges the gap between high-level command understanding and accurate spatial navigation in real-world environments.

[0058] Experiment 2 To evaluate the annotation strategy of this invention, ablation studies were conducted in a tea room and a workstation. Compared with NavA without a map... 3 In comparison, NavA with complete annotations 3 (The solution of this invention) improved performance by 28.0% (60.0% vs. 32.0%) in the tea room, by 36.0% (76.0% vs. 40.0%) in the workstation, and by an average SR of 32.0% (68.0% vs. 36.0%). Compared with NavA (without labeling)... 3 In comparison, this invention found a 24.0% improvement in tea rooms (60.0% vs. 36.0%), a 32.0% improvement in workstations (76.0% vs. 44.0%), and an average improvement of 28.0%. This is in contrast to NavA, which does not include room-level labeling. 3 In comparison, the strategy of this invention improved performance by 24.0% (60.0% vs. 36.0%) in the tea room scenario and by 36.0% (76.0% vs. 40.0%) in the workstation scenario, with an average improvement of 30.0%. These findings confirm that detailed semantic annotation can improve Reasoning-VLM's understanding of spatial relationships.

[0059] To evaluate the impact of different Reasoning-VLM models on navigation performance, this invention conducted ablation studies in tea room and workstation scenarios. The Reasoning-VLM based on GPT-4o achieved the highest success rate (SR) of 68.0%. Closed-source models such as Claude3.5-Sonnet and Qwen-VL-Max showed reductions of 3.0% (65.0%) and 7.0% (61.0%), respectively. Open-source models such as Qwen2.5-VL72B showed a reduction of 9.0% (59.0%). Smaller 7B models, including Qwen2.5-VL-7B, JanusPro-7B, and LLaVA-NeXT-7B, showed reductions of 28.0%, 32.0%, and 36.0%, respectively. These findings highlight the importance of reasoning ability in complex space tasks.

[0060] To evaluate the effectiveness of different Pointing-VLMs in object localization, this invention compares the NaviAfford model with baseline methods. In the availability understanding benchmark, NaviAfford achieves a 13.0% improvement in average availability accuracy compared to the previous state-of-the-art RoboPoint (Yuan et al. 2025) (70.8% vs. 55.9%). This strong availability understanding translates into enhanced navigation performance, with NaviAfford achieving a 10.5% improvement in success rate (SR) compared to RoboPoint (68.0% vs. 57.5%), a 36.0% improvement compared to GPT4o (Hurst et al., 2024) (68.0% vs. 32.0%), and a 52.0% improvement compared to the best open-source model, Qwen2.5-VL-72B (Bai et al., 2025) (68.0% vs. 16.0%). These results demonstrate that the spatial availability training of this invention effectively connects accurate object localization with practical navigation execution.

[0061] The NavA³ framework proposed in this application overcomes the limitations of traditional embodied navigation methods in understanding high-level human commands and executing long-range navigation tasks. By constructing a hierarchical global-local policy architecture, it significantly improves the robot's navigation capabilities in complex real-world environments. This method innovatively combines semantic reasoning with precise spatial localization, realizing an end-to-end navigation system from abstract commands to concrete execution.

[0062] Compared to existing methods, NavA³ has significant advantages in the following aspects: Significantly improved instruction comprehension: Breaking through the limitations of traditional methods that can only handle low-level instructions, it can understand high-level abstract instructions such as "I want to drink coffee" and automatically deduce the target object (coffee machine) and possible location (tea room), significantly enhancing the complexity and practicality of instruction comprehension.

[0063] Significant improvement in navigation success rate: In 50 navigation tasks across 5 different scenarios, the average success rate reached 66.4%, an improvement of 41.2 percentage points compared to the best baseline method MapNav's 25.2%. In the meeting room A scenario, an even greater performance improvement of 46% was achieved (72.0% vs 26.0%).

[0064] Significantly improved spatial perception accuracy: The NaviAfford model, trained on a spatial perception object availability dataset of 1 million samples, improves object localization accuracy by 13% compared to the cutting-edge RoboPoint method, and can accurately understand complex spatial relationships such as "a cup by the window".

[0065] Highly adaptable across platforms: Successfully deployed on the wheeled robot RealMan and the quadruped robot Unitree Go2, demonstrating excellent cross-platform generalization capabilities and providing a unified navigation solution for different types of robots.

[0066] The system architecture has good versatility: the hierarchical design allows global and local strategies to be optimized independently, supports flexible replacement of different VLM models, and has good scalability and adaptability.

[0067] High feasibility of practical deployment: Based on standard RGB-D cameras and existing VLM technology, no additional special hardware is required, which reduces the technical threshold and cost investment for practical applications.

[0068] This invention provides a high-level instruction understanding and long-range navigation method based on a visual language model. Based on high-level instructions and annotated global 3D scenes, it uses a visual language model to perform semantic object reasoning and spatial location prediction to obtain the target region. Within the local boundaries of the target region, target waypoints are determined through random sampling. Based on a panoramic view of the target waypoints, an embodied intelligence model is used to detect and locate the target object, obtaining its position information. Compared to existing technologies where there is a significant gap between navigation tasks and real-world requirements, this method first uses a visual language model for semantic object reasoning and spatial location prediction to obtain the target region, i.e., the possible location of the target object. Then, using an embodied intelligence model, based on a panoramic view of the target waypoints in the target region, it detects and locates the target object, determining its position. This achieves the understanding of high-level abstract instructions and automatically infers the target object and its possible location, significantly improving the robot's navigation capabilities in complex real-world environments.

[0069] The advanced command understanding and long-range navigation device based on a visual language model provided by the present invention will be described below. The advanced command understanding and long-range navigation device based on a visual language model described below can be referred to in correspondence with the advanced command understanding and long-range navigation method based on a visual language model described above.

[0070] Figure 2 This is a flowchart illustrating the advanced command understanding and long-range navigation device based on a visual language model provided by the present invention, as shown below. Figure 2 As shown, the device includes the following: The target region determination module 201 is used to obtain the target region based on high-level instructions and annotated global 3D scene, through a visual language model, semantic object reasoning and spatial location prediction.

[0071] The target waypoint determination module 202 is used to determine the target waypoint within the local boundary of the target area by means of random sampling.

[0072] The target object location determination module 203 is used to detect and locate the target object based on the panoramic view of the target waypoint and through the embodied intelligent model to obtain the target object's location information.

[0073] This invention provides a high-level instruction understanding and long-range navigation device based on a visual language model. Based on high-level instructions and annotated global 3D scenes, it uses a visual language model to perform semantic object reasoning and spatial location prediction to obtain the target region. Within the local boundaries of the target region, target waypoints are determined through random sampling. Based on a panoramic view of the target waypoints, an embodied intelligence model is used to detect and locate the target object, obtaining its position information. Compared to existing technologies where there is a significant gap between navigation tasks and real-world requirements, this invention first uses a visual language model for semantic object reasoning and spatial location prediction to obtain the target region, i.e., the possible location of the target object. Then, using an embodied intelligence model, based on a panoramic view of the target waypoints in the target region, it detects and locates the target object, determining its position. This achieves the understanding of high-level abstract instructions and automatically infers the target object and its possible location, significantly improving the robot's navigation capabilities in complex real-world environments.

[0074] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions from the memory 830 to execute advanced instruction understanding and long-range navigation methods based on a visual language model.

[0075] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the advanced instruction understanding and long-range navigation methods based on visual language models provided by the above methods.

[0077] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the advanced instruction understanding and long-range navigation methods based on visual language models provided by the methods described above.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for advanced instruction understanding and long-range navigation based on visual language models, characterized in that, The method comprises the following steps: Based on the high-level instruction and the annotated global three-dimensional scene, semantic object reasoning and spatial position prediction are performed by a visual language model to obtain a target area; Within the local boundary of the target area, a target waypoint is determined by a random sampling method; Based on the panoramic view of the target waypoint, detection and positioning are performed by an embodied intelligent model to obtain the position information of the target object.

2. The method of claim 1, wherein, The method of obtaining the target area based on the high-level instruction and the annotated global three-dimensional scene by the visual language model comprises the following steps: The visual language model performs semantic decomposition on the high-level instruction to infer a target object; The visual language model performs spatial semantic relationship analysis based on the annotated global three-dimensional scene and the target object to identify the target area where the target object is located. 3.The visual language model based high-level instruction understanding and long-range navigation method of claim 1, wherein, Before the method of obtaining the position information of the target object based on the panoramic view of the target waypoint by the embodied intelligent model, the method further comprises the following step: The panoramic view of the target waypoint is captured by rotating scanning of the embodied agent.

4. The method of claim 3, wherein, The method of obtaining the position information of the target object based on the panoramic view of the target waypoint by the embodied intelligent model comprises the following steps: Based on the panoramic view of the target waypoint, the embodied intelligent model performs detection and positioning to obtain a target center point coordinate; The target center point coordinate is converted into a robot coordinate by a camera intrinsic function; According to the robot coordinate, the position information of the target object is determined.

5. The method of claim 4, wherein, The method of obtaining the target center point coordinate based on the panoramic view of the target waypoint by the embodied intelligent model comprises the following steps: Based on the panoramic view of the target waypoint, the embodied intelligent model performs detection and positioning to output a plurality of target point coordinates; The plurality of target point coordinates are averaged to obtain the target center point coordinate.

6. The method of claim 1, wherein, The method of obtaining the position information of the target object based on the panoramic view of the target waypoint by the embodied intelligent model comprises the following steps: Based on the panoramic view of the target waypoint, the embodied intelligent model performs detection and positioning; In the case where the target object is not located, a new target waypoint is obtained based on the annotated global three-dimensional scene and the target area by a two-stage decision processing method; Based on the panoramic view of the new target waypoint, the embodied intelligent model performs detection and positioning to obtain the position information of the target object.

7. The method of claim 6, wherein the visual language model is based on a recurrent neural network (RNN) language model. The method of obtaining the new target waypoint based on the annotated global three-dimensional scene and the target area by the two-stage decision processing method comprises the following steps: Based on the annotated global three-dimensional scene and the target area, the visual language model performs analysis to obtain an analysis result; In the case where the analysis result is to explore a new area, a new target area is obtained based on the high-level instruction and the annotated global three-dimensional scene by the visual language model in an area outside the target area; within the local boundary of the new target area, a new target waypoint is determined by a random sampling method. In a case where the analysis result is to continue exploring the target region, within a local boundary of the target region, a new target waypoint is determined in a random sampling manner among waypoints other than the target waypoint.

8. A device for advanced instruction understanding and long-range navigation based on visual language models, characterized by Comprise: A target region determination module is configured to perform semantic object reasoning and spatial position prediction through a visual language model based on high-level instructions and an annotated global three-dimensional scene to obtain a target region. A target waypoint determination module is configured to determine a target waypoint in a random sampling manner within a local boundary of the target region. A target object position determination module is configured to perform detection and positioning through an embodied intelligence model based on a panoramic view of the target waypoint to obtain position information of a target object.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the high-level instruction understanding and long-range navigation method based on the visual language model according to any one of claims 1 to 7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the high-level instruction understanding and long-range navigation method based on the visual language model according to any one of claims 1 to 7.

Images