Robot navigation method and device, electronic equipment and storage medium

Abstract

The invention provides a robot navigation method and device, an electronic device and a storage medium, and relates to the technical field of intelligent device.The method comprises the steps that multi-modal observation information is obtained, and a point-of-interest set is generated based on the multi-modal observation information; under the condition that a robot arrives at a current interest point in the interest point set, screening out a plurality of candidate interest points except the current interest point from the interest point set, and constructing a combined image containing the plurality of candidate interest points; inputting the combined image into a visual language model, outputting a target interest point, and controlling the robot to navigate to the target interest point; wherein the visual language model is used for understanding image and text input and outputting a decision based on multi-modal reasoning. According to the method provided by the invention, the calling frequency of the visual language model is greatly reduced, so that the navigation fluency and the overall calculation efficiency of the robot are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a robot navigation method, device, electronic device, and storage medium. Background Technology

[0002] In the current field of robot object navigation, the mainstream methods mainly follow two paradigms: one is based on vision-language navigation (VLN), which relies on a large visual language model (VLM) to reason frame by frame and output low-level action commands (such as "move forward 0.5 meters"); the other is based on boundary point exploration, which generates leading edge points through geometric information and selects them by the VLM.

[0003] However, the VLN method requires frequent calls to the VLM, resulting in a stop-and-go robot behavior and low navigation efficiency. Boundary point methods, on the other hand, suffer from an inherent mismatch between semantic and geometric information—VLM excels at handling high-level semantics but struggles to effectively understand purely geometric boundary points, leading to suboptimal decisions. Furthermore, existing methods often compress visual observations into text descriptions or topological maps, causing information loss and further limiting navigation success rates and environmental understanding. Therefore, a method to address these issues is urgently needed. Summary of the Invention

[0004] This invention provides a robot navigation method, device, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a robot navigation method, comprising the following steps: Acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, and a combined image containing the multiple candidate points of interest is constructed. The combined image is input into a visual language model, which outputs a target interest point and controls the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

[0006] According to a robot navigation method provided by the present invention, the acquisition of multimodal observation information includes: The system acquires RGB images, depth images, and robot pose information of the current environment; wherein, the depth image is used to measure the distance between the robot and obstacles and to determine the terrain. A grid map is constructed based on the RGB image of the current environment, the depth image, and the robot's pose information; wherein, the grid map is used to mark obstacles, passable areas, and unexplored areas.

[0007] According to a robot navigation method provided by the present invention, the step of generating a set of points of interest based on the multimodal observation information includes: Based on the multimodal observation information, a non-directional interest point set and a directional interest point set are generated; wherein, the non-directional interest point set is used to guide the robot to explore unexplored areas; the directional interest point set is used to guide the robot to conduct directional observation and confirmation of detected potential target objects, wherein the potential target objects are objects whose confidence in the robot's navigation target is greater than a preset threshold; The set of interest points is constructed based on the set of non-directional interest points and the set of directional interest points.

[0008] According to a robot navigation method provided by the present invention, the multimodal observation information is acquired in real time based on the movement of the robot, and the set of points of interest is dynamically updated according to the changes in the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, the step of filtering out multiple candidate points of interest from the set of points of interest, excluding the current point of interest, includes: Select the most recently generated interest point from the set of interest points as the preferred candidate interest points; If the number of preferred candidate points of interest is less than a preset number, at least one point of interest that is closest to the robot's current position is selected from the set of points of interest to supplement the list of candidate points of interest.

[0009] According to a robot navigation method provided by the present invention, constructing a combined image containing the plurality of candidate points of interest includes: Obtain the map location of each candidate point of interest among the plurality of candidate points of interest, and generate a unique numerical identifier corresponding to each candidate point of interest based on the map location; By projecting the coordinate transformation onto the corresponding observation image, the unique numerical identifier of each candidate point of interest is drawn onto the image, generating a labeled candidate point of interest image; The labeled candidate interest point image of each candidate interest point is stitched together with the environmental background image to generate the combined image.

[0010] According to a robot navigation method provided by the present invention, the step of inputting the combined image into a visual language model and outputting target interest points includes: The combined image and a text prompt containing navigation task instructions are input into the visual language model, and a text response result is output; wherein, the text response result includes a unique numerical identifier for the target point of interest; The text response result is parsed to extract the unique numerical identifier of the target interest point and to determine the corresponding interest point as the target interest point.

[0011] According to a robot navigation method provided by the present invention, after inputting the combined image into a visual language model, outputting a target interest point, and controlling the robot to navigate to the target interest point, the method further includes: Repeat the above steps of acquiring multimodal observation information, generating a set of points of interest based on the multimodal observation information, filtering out multiple candidate points of interest other than the current point of interest from the set of points of interest when the robot reaches the current point of interest, constructing a combined image containing the multiple candidate points of interest, inputting the combined image into the visual language model, outputting the target point of interest, and controlling the robot to navigate to the target point of interest, until the robot is controlled to navigate to the target location.

[0012] The present invention also provides a robot navigation device, comprising the following modules: The acquisition module is used to acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; The filtering module is used to filter out multiple candidate interest points other than the current interest point from the interest point set when the robot arrives at the current interest point in the interest point set, and to construct a combined image containing the multiple candidate interest points. The navigation module is used to input the combined image into a visual language model, output a target interest point, and control the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the robot navigation methods described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot navigation method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the robot navigation methods described above.

[0016] This invention provides a robot navigation method, apparatus, electronic device, and storage medium. It acquires multimodal observation information and generates a set of points of interest (POIs) based on this information. When the robot reaches a current POI in the set, it selects multiple candidate POIs (excluding the current POI) from the set and constructs a combined image containing these candidate POIs. The combined image is input into a visual language model, which outputs a target POI and controls the robot to navigate to that target POI. The visual language model is used to understand image and text input and outputs decisions based on multimodal reasoning. Therefore, this invention uses a visual language model to make high-level decisions on semantically rich POIs at key nodes, rather than outputting low-level action commands frame by frame. This significantly reduces the frequency of visual language model calls, thereby significantly improving the robot's navigation smoothness and overall computational efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the robot navigation method provided by the present invention.

[0019] Figure 2 This is a complete flowchart of the robot navigation method provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the robot navigation device provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following is combined Figures 1-4This invention describes a robot navigation method, apparatus, electronic device, and storage medium.

[0024] Before describing the present invention, the terms and concepts used in the present invention will be explained illustratively.

[0025] PIGEON: Point of Interest-Driven Object Navigation and Exploration.

[0026] VLM: Visual Language Model.

[0027] PoI: Point of Interest.

[0028] RLVR: Reinforcement learning with verifiable rewards.

[0029] VLN: Visual Language Navigation.

[0030] SR: Accuracy.

[0031] SPL: Path Length Weighted Success Rate.

[0032] LM: Large Model.

[0033] CoT: Mind Chain.

[0034] GRPO: Group Relative Strategy Optimization.

[0035] It's important to note that object navigation tasks typically require models to actively explore the unknown environment based on their own position and the robot's current RGBD (Red, Green, Blue, and Depth) observations, thereby discovering and navigating to a given class of objects. Related methods can generally be divided into two main categories. One mainstream approach follows the "Vision-Language Navigation (VLN)" paradigm, where a large visual language model (VLM) performs intensive reasoning to output fine, step-by-step action commands such as "move forward 75 centimeters" or "turn right 45 degrees." The other approach employs a "boundary point-based selection" paradigm, where the VLM directly selects the next target from frontier points explored on the map. These methods generally have the following shortcomings: The problem of low exploration efficiency: VLN-type methods typically utilize multimodal input directly, or use VLM to extract information from the scene into text descriptions or scene graphs. Since these methods require a large model to directly provide actions at each step, when deployed on real robots, this causes the robot to stop and wait for model inference after each step, exhibiting obvious "stop-and-go" behavior, which seriously affects the smoothness and efficiency of navigation.

[0036] The problem of semantic and geometric information mismatch: Frontier-based exploration methods suffer from a fundamental mismatch problem: boundary points contain low-level geometric information, while VLM excels at processing high-level semantic information. Forcing VLM to process geometric data it is not good at leads to suboptimal decision results. Some methods abstract the environment into textual memory, construct semantic maps in the form of grid maps, or abstract the environment into topological graphs. These approaches introduce additional processing, which results in information loss and limits the model's perceptual capabilities. Furthermore, frontier exploration only focuses on unexplored areas, and due to poor observation angles, information from explored areas may not be fully utilized, leading to a decrease in navigation success rate.

[0037] Based on this, the present invention proposes a robot navigation method to solve at least one of the above problems.

[0038] Figure 1 This is a flowchart illustrating the robot navigation method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 100: Obtain multimodal observation information and generate a set of points of interest based on the multimodal observation information.

[0039] Step 100 involves obtaining multimodal observation information, specifically including: Step 110: Obtain the RGB image, depth image, and robot pose information of the current environment; wherein, the depth image is used to measure the distance between the robot and obstacles and to determine the terrain.

[0040] Step 120: Construct a grid map based on the RGB image of the current environment, the depth image, and the robot pose information; wherein the grid map is used to mark obstacles, passable areas, and unexplored areas.

[0041] Specifically, multimodal observation information refers to various types of environmental data collected in real time by robot sensor systems, including RGB images (i.e., red, green and blue three-color images, used to provide rich color and texture information for object recognition and scene understanding), depth images (used to accurately measure the distance of the robot to the surface of each object in the environment, thereby determining the terrain and obstacles), and robot pose information (i.e. the position and orientation of the robot in the global coordinate system, usually provided by odometry or SLAM systems).

[0042] Based on this real-time, multi-dimensional data, the system can construct and dynamically update the internal representation map of the environment (such as a grid map), thereby generating a navigation option pool of points of interest (POIs). Each POI in this set is not a pure geometric coordinate point, but a navigable location endowed with semantic meaning. Specifically, it is divided into "non-directional POIs" that guide the exploration of unknown areas and "directional POIs" that guide the approach to and confirmation of suspected targets.

[0043] Step 100, which generates a set of points of interest based on the multimodal observation information, specifically includes: Step 130: Based on the multimodal observation information, generate a non-directional interest point set and a directional interest point set; wherein, the non-directional interest point set is used to guide the robot to explore unexplored areas; the directional interest point set is used to guide the robot to conduct directional observation and confirmation of detected potential target objects, wherein the potential target object is an object whose confidence level with the robot's navigation target is greater than a preset threshold.

[0044] Step 140: Construct the set of interest points based on the set of non-directional interest points and the set of directional interest points.

[0045] Specifically, non-directional points of interest are mainly generated based on a geometric exploration strategy. As the robot moves and updates its internal grid map using depth images and pose information, the system continuously generates these points on the boundary (i.e., the exploration front) between the "explored area" and the "unexplored area". They are essentially geometric landmarks that encourage the robot to move into unknown spaces. Their semantic meaning is "there may be new space or new objects in this direction". For example, points generated at the end of a corridor or in front of an unopened doorway. Their main function is to guide the robot to systematically explore unexplored areas and ensure that the environment is fully covered.

[0046] In contrast, the generation of directional points of interest (POIs) relies heavily on a high-level semantic understanding of RGB images: The system continuously runs an object detection model (such as YOLO or Faster R-CNN) in the background. When it identifies an object in the current field of view, and that object is semantically similar to the target category specified in the navigation command (e.g., the navigation target is "chair," but "stool" or "sofa" is detected), and its confidence score exceeds a preset threshold (e.g., 0.7), the object is identified as a "potential target object." The system then immediately generates a POI directly facing the object within the traversable area around it. For example, when a potted plant-like object is detected at the far end of a room, a directional point is generated to its side and front, with the semantic meaning "Please approach and observe this suspected target from the front." Its main function is to guide the robot to orient and confirm the detected potential target, thereby avoiding misjudgments and improving navigation accuracy.

[0047] Ultimately, the system dynamically merges these two subsets to construct a unified set of points of interest.

[0048] Step 200: When the robot reaches the current point of interest in the set of points of interest, select multiple candidate points of interest other than the current point of interest from the set of points of interest, and construct a combined image containing the multiple candidate points of interest.

[0049] It should be noted that the multimodal observation information is acquired in real time based on the robot's movement, and the set of points of interest is dynamically updated according to the changes in the multimodal observation information.

[0050] In step 200, when the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, specifically including: Step 210: Select the most recently generated interest point from the set of interest points as the preferred candidate interest point.

[0051] Step 220: If the number of preferred candidate points of interest is less than a preset number, at least one point of interest that is closest to the robot's current position is selected from the set of points of interest to supplement the plurality of candidate points of interest.

[0052] It should be noted that the entire system is based on real-time changes: as the robot moves continuously, its sensors acquire new multimodal observation information (such as new RGB images, depth data, and updated poses). The system uses this new data to update the internal map in real time (such as turning unknown areas into explored areas and marking newly discovered obstacles), and dynamically updates the set of points of interest based on this (for example, generating non-directional points of interest at new exploration frontiers, or generating directional points of interest for newly detected objects).

[0053] Specifically, after the robot arrives at each current point of interest, a key candidate point selection is triggered. In step 210, the most recent movement refers to the entire motion process from the previous decision point (i.e., the previous point of interest) to the current point of interest. During this process, due to the continuous changes in the robot's pose and observation range, a large amount of new environmental information emerges, resulting in a batch of newly generated points of interest. These new points represent the freshest, unevaluated exploration clues or potential targets in the environment, and are therefore prioritized as the first choice of candidate points of interest to ensure that the VLM's decision is based on the latest environmental understanding and prioritizes the direction of exploring new discoveries.

[0054] Considering that in some environments there may not be enough new points of interest to be generated in the short term (for example, in an empty hall, few new boundary points are discovered after one movement), a preset number (e.g., 5) is set as a threshold to avoid the model's judgment being affected by too few candidate options. When the number of preferred candidate points is insufficient, the system will supplement the entire set of points of interest by selecting those existing points of interest that are closest to the robot's current position. Here, "closest distance" refers to the path planning distance (such as the shortest path length calculated by the A* algorithm) based on map navigability. This ensures that the supplemented points are not only spatially proximate but also practically quickly reachable.

[0055] Step 200 involves constructing a combined image containing the multiple candidate interest points, specifically including: Step 230: Obtain the map location of each candidate point of interest among the multiple candidate points of interest, and generate a unique digital identifier corresponding to each candidate point of interest based on the map location.

[0056] Step 240: By projecting the coordinate transformation, the unique numerical identifier of each candidate point of interest is drawn onto the corresponding observation image to generate a labeled candidate point of interest image.

[0057] Step 250: The labeled candidate interest point image of each candidate interest point is stitched together with the environmental background image to generate the combined image.

[0058] Specifically, the precise map location of each candidate point of interest on the internal raster map (i.e., its two-dimensional coordinates [x, y] in the map coordinate system) is first obtained, and a unique numerical identifier is assigned to each point (for example, the numbers 1 to 5 are assigned to the 5 candidate points in sequence); this identifier is the key index for VLM to select and associate data in the future.

[0059] Furthermore, the system performs a crucial coordinate transformation projection: using the camera's intrinsic parameter matrix and the robot's current pose (i.e., the transformation relationship between the camera coordinate system and the map coordinate system), the map coordinates of each point of interest are precisely back-projected onto the pixel coordinate system of the observation image (i.e., the RGB image) corresponding to the frame in which it was initially generated, thereby determining the specific location of the point in the actual image (e.g., the point [x=3.5, y=2.1] might be projected to the image pixel coordinates [u=320, v=240]). Subsequently, at this pixel location, the system draws the previously assigned numerical identifier onto the image in a conspicuous manner (e.g., with a red circle and white numbers), thereby generating a series of labeled candidate point of interest images; each such image visually shows what a candidate point looks like in its original visual context, for example, the identifier "3" is drawn in the center of a porch image, indicating that this is an exploration point that leads to a new room.

[0060] Finally, the system stitches together all the labeled candidate point of interest images with an environmental background image (a snapshot intelligently sampled from historical points of interest, with minimal overlap with the current candidate point's field of view but providing an overview of the surrounding environment) to generate a combined image.

[0061] Step 300: Input the combined image into the visual language model, output the target interest point, and control the robot to navigate to the target interest point; wherein, the visual language model is used to understand the image and text input and output a decision based on multimodal reasoning.

[0062] Step 300 involves inputting the combined image into a visual language model and outputting target interest points, including: Step 310: Input the combined image and the text prompt containing navigation task instructions into the visual language model, and output the text response result; wherein, the text response result includes a unique numerical identifier of the target point of interest.

[0063] Step 320: Parse the text response result to extract the unique numerical identifier of the target interest point and determine the corresponding interest point as the target interest point.

[0064] Specifically, the combined image generated in the above embodiments, along with a text prompt containing navigation task instructions (e.g., "Task: Find and go to a potted plant. Please carefully analyze the image of each numbered point and select the direction most likely to be found or the direction leading to the target."), are used as multimodal inputs and fed into the visual language model. Upon receiving this information, the model not only identifies the visual content in the image of each numbered point (e.g., point 3 shows a narrow corridor, and point 5 shows a green, plant-like object at the far end), but also combines its built-in common-sense knowledge base (e.g., "potted plants are usually placed in the living room, balcony, or study, not the kitchen," and "the end of the corridor may be a bedroom") and the semantics of the text instructions to comprehensively compare, evaluate, and logically reason about all options. Finally, VLM outputs a structured natural language text response, which typically consists of two parts: first, a natural language explanation of the model's reasoning process (e.g., "I think point 5 is the most likely because the green object in its image is similar in height to a potted plant, and the room's decor is like a living room, where potted plants are common; although point 3 leads to an unknown area, it has a lower priority."), and second, a unique numerical identifier for the target point of interest (e.g., "Therefore, I choose point 5.").

[0065] Furthermore, the system parses the text response, extracts the numerical identifiers (such as "5") contained therein, determines the specific map coordinates of the interest point corresponding to the number in the candidate interest point list, and officially sets it as the next target interest point.

[0066] It should be noted that the system repeatedly executes the complete sequence of steps from "acquiring multimodal observation information" to "controlling the robot to navigate to the target point of interest." This loop continues until the loop termination condition is met, i.e., the robot is controlled to navigate to the target location. The advantages of the entire method are concentrated in this iterative process: VLM is sparsely invoked only at each key node (point of interest), ensuring efficiency; and each invocation is based on the latest and richest environmental information, ensuring the quality of decision-making.

[0067] Optionally, during navigation, if the robot generates and visits multiple "points of interest" around a highly suspected target object, the system will automatically collect all image snapshots of the object taken from these different locations and angles. Then, the system will stitch these multi-view images together and submit them again to the visual language model, issuing a confirmation instruction, such as, "Please confirm if this is a real potted plant?". By aggregating information from multiple perspectives, misjudgments caused by occlusion, lighting, or angle issues from a single perspective can be effectively overcome. This improves the accuracy of the final target confirmation.

[0068] Optionally, to make the model more efficient in navigation decisions, this embodiment also provides a training process for fine-tuning large models.

[0069] Data and Reward Generation: The navigation task is run in a simulator with a ground truth map and target location annotations. At each decision point, the system uses the ground truth map and the A* algorithm to calculate the shortest path distance from each "candidate point of interest" to the real target. Based on these distances, the system calculates a continuous "soft reward" score for each option—the closer to the target, the higher the score.

[0070] Policy Optimization: A reinforcement learning algorithm called "group-based policy optimization" is used to fine-tune a large visual language model. The goal of this algorithm is to enable the model to increase the probability of responses that receive higher "soft reward" scores (i.e., those that select better path points) after multiple responses are generated.

[0071] The above describes the steps of the robot navigation method provided by this invention. As can be seen from the above description, according to the robot navigation method provided by this invention, multimodal observation information is acquired, and a set of points of interest (POIs) is generated based on the multimodal observation information; when the robot reaches the current POI in the set of POIs, multiple candidate POIs other than the current POI are selected from the set of POIs, and a combined image containing the multiple candidate POIs is constructed; the combined image is input into a visual language model, a target POI is output, and the robot is controlled to navigate to the target POI; wherein, the visual language model is used to understand image and text input and output decisions based on multimodal reasoning. Therefore, this invention uses a visual language model to make one-time high-level decisions on semantically rich POIs at key nodes, rather than outputting low-level action commands frame by frame, significantly reducing the frequency of calling the visual language model, thereby significantly improving the robot's navigation smoothness and overall computational efficiency.

[0072] Figure 2 This is a complete flowchart of the robot navigation method provided by the present invention. The following is a summary of the process. Figure 2 The robot navigation method provided by the present invention will be described in detail below.

[0073] like Figure 2 As shown, this paper illustrates the complete reasoning process of the Visual Language Model (VLM) in the robot navigation method of this invention for interest point decision-making and verification, and the reinforcement learning optimization mechanism behind it. The process begins with the Object Verification task, where the VLM is required to select the next target point number to move to based on a combination of images with numerical identifiers (such as 1, 2, 3...). If uncertain, it returns 0, demonstrating its decision-making capability based on multimodal reasoning. The Thought process example shown in the figure illustrates the reasoning logic of the VLM: it starts by analyzing region number 1 ("a living area with a couch..."), and associates it with common sense ("potted plant could be"), finally outputting a selection ( <thought> <action> i <action>The mathematical formula below reveals the reinforcement learning mechanism for verifiable rewards used in the model training phase: the calculated soft reward value is negatively correlated with the path distance from each point of interest (PoI) to the real target, with higher rewards for closer distances; subsequently, by calculating the average reward and standardized advantage value of all candidate points, fine-grained optimization signals are provided for the Group Relative Policy Optimization (GRPO) algorithm, thereby efficiently improving the navigation strategy of the VLM. The left side of the figure shows the real-time state information of the system, including multimodal observation information such as Depth (depth image), Odom (odometry pose coordinates, such as (-3.7, 9.5)), ω (angular velocity), etc., as well as key elements in the map: Current Waypoint, New Directional PoI, PreviousSelectable PoI, and Unselectable PoI. The entire diagram clearly outlines a closed-loop system from perception (multimodal data) to decision-making (VLM inference) and then to optimization (GRPO training), highlighting its core innovation of closely integrating advanced semantic reasoning with underlying geometric control.

[0074] In summary, the PIGEON provided by this invention mainly includes the following aspects: 1. A sparse decision-making framework based on "Points of Interest (PoI)".

[0075] This framework decomposes the complex navigation task into a series of high-level decisions involving "selecting the next point of interest." The robot first uses a low-level planner (such as the A* algorithm) to navigate to a selected point of interest. Only after reaching that point is a high-cost, large-scale visual language model (VLM) invoked to analyze the current environment and select the next most promising location for finding the target from multiple candidate Points of Interest (PoIs). These PoIs are not merely geometric points on the map but are also endowed with semantic meaning, falling into two categories: 1) Non-directional PoI: Generated at the boundary of an already explored area to encourage exploration of unknown areas.

[0076] 2) Oriented PoI: When the object detector detects a suspected target, it generates a PoI facing the object in the unobstructed area around the object to guide subsequent close-up observation and confirmation.

[0077] Traditional VLN methods require the VLM to perform reasoning and output low-level commands such as "forward" or "turn" at every step. PIGEON's improvement lies in freeing the VLM from high-frequency, low-level action control, allowing it to focus on sparse, high-level path selection. Traditional boundary point selection methods provide the VLM with purely geometric information (Frontier Points), which does not match the VLM's strength in semantic understanding. PIGEON perfectly connects high-level semantic reasoning and low-level geometric planning by introducing semantically meaningful Points of Interest (especially oriented Points of Interest).

[0078] Because the VLM is called much less frequently, the "stop-and-go" phenomenon caused by the time-consuming model inference is avoided. Semantic PoIs provide VLM with higher quality decision options, enabling it to perform deeper reasoning (e.g., "there may be a study in this direction, and there are usually potted plants in the study").

[0079] 2. A multi-perspective memory and reasoning mechanism based on snapshots.

[0080] PIGEON does not compress environmental information into abstract text or symbols. Instead, it directly saves the visual observations (i.e., RGB images) at each Point of Interest (PoI) as "snapshots," forming a persistent memory rich in visual detail. When a decision needs to be made or an object needs to be confirmed, the system retrieves multiple snapshots from different perspectives (including views of candidate PoIs and related historical views) from the memory and inputs them together into the Virtual Model (VLM). Related technologies typically abstract visual information into 3D scene graphs, topology maps, or text descriptions, a process that loses a significant amount of original visual semantic information crucial for VLM inference. PIGEON's improvement lies in preserving this high-fidelity visual information.

[0081] By integrating images from multiple perspectives, VLM can make more reliable judgments. For example, PIGEON can accurately determine whether an object is a floral artwork on a wall or a real potted plant by observing from multiple angles, thus avoiding incorrect navigation destinations.

[0082] 3. Verifiable reinforcement learning (RLVR) based on soft rewards.

[0083] To improve the navigation inference capabilities of the VLM without relying on expensive manually labeled data, PIGEON employs an RLVR fine-tuning method. Its core lies in the design of the reward function: instead of using a simple "success / failure" binary reward, it adopts a "soft reward" mechanism. Specifically, for each Point of Interest (PoI) selected by the VLM, the system calculates a continuous reward value based on the shortest path distance from that PoI to the real target; the closer the distance, the higher the reward.

[0084] Traditional reinforcement learning methods often face problems such as sparse rewards (only rewarding successful completion) and low sample efficiency in navigation tasks. Using simple binary rewards (e.g., whether the selected Point of Interest (PoI) is closest to the target) also provides limited information. PIGEON's soft reward design provides the model with denser, more fine-grained learning signals.

[0085] This soft reward mechanism significantly improves the efficiency of training samples, enabling the model to efficiently learn complex spatial reasoning capabilities.

[0086] The robot navigation method provided by this invention has the following main advantages: 1. Significantly improved computational efficiency and navigation smoothness in the physical world.

[0087] This beneficial effect stems from an innovative "sparse decision-making" framework. Traditional solutions often require large visual language models (VLMs) to perform intensive, step-by-step action reasoning (e.g., "move forward 0.25 meters"), causing the robot to "stop and go" due to frequent waiting for calculations. This invention liberates the VLM from high-frequency low-level control, performing a high-level semantic decision (selecting the next point of interest) only at key nodes (after reaching a point of interest), while the actual movement between the two points is handled by an efficient low-level planner. This significantly reduces the number of calls to the high-cost VLM, thereby improving overall operational efficiency.

[0088] 2. Higher decision-making quality and leading navigation success rate.

[0089] This beneficial effect mainly stems from two improvements. First, it introduces semantic "Points of Interest (PoIs)." Compared to related solutions that only use geometric boundary points, the PoIs of this invention (especially "oriented PoIs" facing potential targets) provide VLM with task-relevant, semantically rich options, perfectly solving the problem of mismatch between high-level semantic models and low-level geometric information, thus fully leveraging its reasoning capabilities. Second, it employs "snapshot-based memory." This invention does not compress visual information into abstract symbols but retains original image snapshots, enabling VLM to perform multi-view reasoning. This allows for accurate differentiation of similar objects (such as a painting on a wall and a real potted plant), avoiding erroneous decisions and improving the success rate.

[0090] 3. The model training samples are more efficient.

[0091] This is thanks to the use of a "soft reward" mechanism in the reinforcement learning phase. Many related solutions employ simple binary rewards (1 for success, 0 for failure) or sparse final task rewards during model training, resulting in low learning efficiency. This invention calculates a continuous, normalized reward score based on the path distance between candidate interest points and the real target. This fine-grained reward signal provides the model with richer and denser learning feedback, enabling it to learn faster what constitutes a "better" decision, thus achieving higher sample efficiency during training.

[0092] 4. Possesses strong real-world generalization ability.

[0093] The architecture of this invention decouples high-level semantic reasoning (handled by the VLM) from low-level motion control (handled by the robot's own planner). This makes the spatial reasoning and navigation strategies learned by the VLM universal, without being overly tied to the physics engine or motion space of a specific simulator. Therefore, when the system is deployed on a physical robot, only the low-level navigation module needs to be replaced, and the top-level VLM decision-making brain can adapt well to the new platform, demonstrating a success rate of up to 95% in real, unknown physical environments.

[0094] Based on the above embodiments of the present invention, variations or optimizations can be made in the following directions: 1. Evolution towards three-dimensional map representation.

[0095] The current approach uses a two-dimensional grid map for planning. This two-dimensional method performs poorly when dealing with multi-story scenes, including staircases, which is why its performance is limited on some datasets.

[0096] Optimization Solution: A 3D map representation method can be adopted, such as a voxel grid, point cloud, or 3D mesh. Points of interest are upgraded to 3D spatial coordinates, and a 3D path planner is used for navigation. This will enable the solution to naturally handle cross-floor navigation tasks, greatly expanding its applicability in complex indoor environments (such as duplex apartments and office buildings).

[0097] 2. Construct a dynamic, hierarchical memory management system.

[0098] Current snapshot-based memory systems grow as the scope of exploration expands, which may pose challenges to retrieval efficiency and information redundancy in ultra-large-scale scenarios.

[0099] Optimization Solution: A hierarchical memory system can be designed. For key areas near the current location, high-resolution visual snapshots are retained; for distant or fully explored areas, VLM can be used to automatically summarize multiple snapshots into an abstract semantic node (e.g., a "second-floor bedroom" node containing a list of key objects). This reduces the information burden on VLM during decision-making while ensuring the long-term availability of memory.

[0100] 3. Explore end-to-end learnable underlying planners.

[0101] Current underlying planners (such as the A* algorithm) are standalone, non-learnable modules.

[0102] Optimization Scheme: A learnable neural network planner could be used to replace the traditional A* algorithm. This planner can take the high-level target (next point of interest) selected by the VLM and local map observations as input, directly outputting a smoother sequence of actions that better adapts to the robot's own dynamics. This promises to achieve higher-level end-to-end training, allowing for more seamless coordination between lower-level movements and higher-level decisions.

[0103] The robot navigation device provided by the present invention is described below. The robot navigation device described below can be referred to in correspondence with the robot navigation method described above.

[0104] Figure 3 This is a schematic diagram of the robot navigation device provided by the present invention, as shown below. Figure 3 As shown, the robot navigation device provided by the present invention includes: The acquisition module 301 is used to acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; The filtering module 302 is used to filter out multiple candidate interest points other than the current interest point from the interest point set when the robot arrives at the current interest point in the interest point set, and to construct a combined image containing the multiple candidate interest points. The navigation module 303 is used to input the combined image into the visual language model, output the target interest point, and control the robot to navigate to the target interest point; wherein, the visual language model is used to understand the image and text input and output a decision based on multimodal reasoning.

[0105] The robot navigation device provided by this invention acquires multimodal observation information and generates a set of points of interest (POIs) based on this information. When the robot reaches the current POI in the set, it selects multiple candidate POIs (excluding the current POI) from the set and constructs a combined image containing these candidate POIs. The combined image is then input into a visual language model, which outputs a target POI and controls the robot to navigate to that target POI. The visual language model is used to understand image and text input and outputs decisions based on multimodal reasoning. Therefore, this invention uses a visual language model to make high-level decisions on semantically rich POIs at key nodes, rather than outputting low-level action commands frame by frame. This significantly reduces the frequency of visual language model calls, thereby significantly improving the robot's navigation smoothness and overall computational efficiency.

[0106] Based on the above embodiments, in this embodiment, the acquisition module 301 is specifically used for: The system acquires RGB images, depth images, and robot pose information of the current environment; wherein, the depth image is used to measure the distance between the robot and obstacles and to determine the terrain. A grid map is constructed based on the RGB image of the current environment, the depth image, and the robot's pose information; wherein, the grid map is used to mark obstacles, passable areas, and unexplored areas.

[0107] Based on the above embodiments, in this embodiment, the device further includes a generation module, specifically used for: Based on the multimodal observation information, a non-directional interest point set and a directional interest point set are generated; wherein, the non-directional interest point set is used to guide the robot to explore unexplored areas; the directional interest point set is used to guide the robot to conduct directional observation and confirmation of detected potential target objects, wherein the potential target objects are objects whose confidence in the robot's navigation target is greater than a preset threshold; The set of interest points is constructed based on the set of non-directional interest points and the set of directional interest points.

[0108] Based on the above embodiments, in this embodiment, the multimodal observation information is acquired in real time according to the movement of the robot, and the set of points of interest is dynamically updated according to the changes in the multimodal observation information; The filtering module 302 is specifically used for: Select the most recently generated interest point from the set of interest points as the preferred candidate interest points; If the number of preferred candidate points of interest is less than a preset number, at least one point of interest that is closest to the robot's current position is selected from the set of points of interest to supplement the list of candidate points of interest.

[0109] Based on the above embodiments, in this embodiment, the device further includes a construction module, specifically used for: Obtain the map location of each candidate point of interest among the plurality of candidate points of interest, and generate a unique numerical identifier corresponding to each candidate point of interest based on the map location; By projecting the coordinate transformation onto the corresponding observation image, the unique numerical identifier of each candidate point of interest is drawn onto the image, generating a labeled candidate point of interest image; The labeled candidate interest point image of each candidate interest point is stitched together with the environmental background image to generate the combined image.

[0110] Based on the above embodiments, in this embodiment, the navigation module 303 is specifically used for: The combined image and a text prompt containing navigation task instructions are input into the visual language model, and a text response result is output; wherein, the text response result includes a unique numerical identifier for the target point of interest; The text response result is parsed to extract the unique numerical identifier of the target interest point and to determine the corresponding interest point as the target interest point.

[0111] Based on the above embodiments, in this embodiment, the navigation module 303 is further configured to: The process involves inputting the combined image into a visual language model, outputting a target interest point, and then controlling the robot to navigate to the target interest point. Repeat the above steps of acquiring multimodal observation information, generating a set of points of interest based on the multimodal observation information, filtering out multiple candidate points of interest other than the current point of interest from the set of points of interest when the robot reaches the current point of interest, constructing a combined image containing the multiple candidate points of interest, inputting the combined image into the visual language model, outputting the target point of interest, and controlling the robot to navigate to the target point of interest, until the robot is controlled to navigate to the target location.

[0112] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device can be a robot or other electronic device. This electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions from the memory 430 to execute robot navigation methods, including: Acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, and a combined image containing the multiple candidate points of interest is constructed. The combined image is input into a visual language model, which outputs a target interest point and controls the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

[0113] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the robot navigation methods provided by the above methods, including: Acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, and a combined image containing the multiple candidate points of interest is constructed. The combined image is input into a visual language model, which outputs a target interest point and controls the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

[0115] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the robot navigation methods provided by the methods described above, including: Acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, and a combined image containing the multiple candidate points of interest is constructed. The combined image is input into a visual language model, which outputs a target interest point and controls the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / action> < / action> < / thought>

Claims

1. A robot navigation method, characterized in that, include: Acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, multiple candidate points of interest other than the current point of interest are selected from the set of points of interest, and a combined image containing the multiple candidate points of interest is constructed. The combined image is input into a visual language model, which outputs a target interest point and controls the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

2. The robot navigation method according to claim 1, characterized in that, The acquisition of multimodal observation information includes: The system acquires RGB images, depth images, and robot pose information of the current environment; wherein, the depth image is used to measure the distance between the robot and obstacles and to determine the terrain. A grid map is constructed based on the RGB image of the current environment, the depth image, and the robot's pose information; wherein, the grid map is used to mark obstacles, passable areas, and unexplored areas.

3. The robot navigation method according to claim 1, characterized in that, The generation of the point of interest set based on the multimodal observation information includes: Based on the multimodal observation information, a non-directional interest point set and a directional interest point set are generated; wherein, the non-directional interest point set is used to guide the robot to explore unexplored areas; the directional interest point set is used to guide the robot to conduct directional observation and confirmation of detected potential target objects, wherein the potential target objects are objects whose confidence in the robot's navigation target is greater than a preset threshold; The set of interest points is constructed based on the set of non-directional interest points and the set of directional interest points.

4. The robot navigation method according to claim 1, characterized in that, The multimodal observation information is acquired in real time based on the robot's movement, and the set of points of interest is dynamically updated based on the changes in the multimodal observation information; When the robot reaches the current point of interest in the set of points of interest, the step of filtering out multiple candidate points of interest from the set of points of interest, excluding the current point of interest, includes: Select the most recently generated interest point from the set of interest points as the preferred candidate interest points; If the number of preferred candidate points of interest is less than a preset number, at least one point of interest that is closest to the robot's current position is selected from the set of points of interest to supplement the list of candidate points of interest.

5. The robot navigation method according to claim 1, characterized in that, The construction of a combined image containing the multiple candidate interest points includes: Obtain the map location of each candidate point of interest among the plurality of candidate points of interest, and generate a unique numerical identifier corresponding to each candidate point of interest based on the map location; By projecting the coordinate transformation onto the corresponding observation image, the unique numerical identifier of each candidate point of interest is drawn onto the image, generating a labeled candidate point of interest image; The labeled candidate interest point image of each candidate interest point is stitched together with the environmental background image to generate the combined image.

6. The robot navigation method according to claim 5, characterized in that, The step of inputting the combined image into a visual language model and outputting target interest points includes: The combined image and a text prompt containing navigation task instructions are input into the visual language model, and a text response result is output; wherein, the text response result includes a unique numerical identifier for the target point of interest; The text response result is parsed to extract the unique numerical identifier of the target interest point and to determine the corresponding interest point as the target interest point.

7. The robot navigation method according to any one of claims 1-6, characterized in that, After inputting the combined image into a visual language model, outputting a target interest point, and controlling the robot to navigate to the target interest point, the method further includes: Repeat the above steps of acquiring multimodal observation information, generating a set of points of interest based on the multimodal observation information, filtering out multiple candidate points of interest other than the current point of interest from the set of points of interest when the robot reaches the current point of interest, constructing a combined image containing the multiple candidate points of interest, inputting the combined image into the visual language model, outputting the target point of interest, and controlling the robot to navigate to the target point of interest, until the robot is controlled to navigate to the target location.

8. A robot navigation device, characterized in that, include: The acquisition module is used to acquire multimodal observation information and generate a set of points of interest based on the multimodal observation information; The filtering module is used to filter out multiple candidate interest points other than the current interest point from the interest point set when the robot arrives at the current interest point in the interest point set, and to construct a combined image containing the multiple candidate interest points. The navigation module is used to input the combined image into a visual language model, output a target interest point, and control the robot to navigate to the target interest point; wherein, the visual language model is used to understand image and text input and output a decision based on multimodal reasoning.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot navigation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot navigation method as described in any one of claims 1 to 7.

Images