A Zero-Shot Target Navigation Method and System Based on Object-Aware Memory

Abstract

This invention relates to a zero-shot target navigation method and system based on object-aware memory. The method includes: S1, acquiring environmental observation information at the current moment, constructing an occupancy map, and extracting leading edge points; S2, caching the environmental observation information in a memory buffer; S3, filtering relevant historical observation frames for each leading edge point from the memory buffer based on visibility and distance constraints; S4, extracting image sequences of the object region based on the relevant historical observation frames; S5, combining the natural language description of the target object with the image sequence to obtain a semantic matching score for each leading edge point, and selecting a target exploration point based on the semantic matching score; S6, performing path planning and navigation control in the occupancy map based on the target exploration point. When the target object is identified in the field of view, the navigation task is terminated; otherwise, return to S1, until navigation is successful or a preset step limit is reached. This invention can improve the accuracy of target positioning and the efficiency of navigation paths.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology, and in particular to a zero-sample target navigation method and system based on object-aware memory. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision, and robotics, object-oriented autonomous navigation (ObjectNav) has gradually become one of the key technologies in fields such as server robots, warehouse logistics robots, and disaster relief. This task requires robots to automatically explore and locate the position of a target object in an unknown environment, based on a user-provided description of the target object.

[0003] Existing ObjectNav tasks can be broadly categorized into two types:

[0004] 1. End-to-end learning methods: These methods directly map environmental perception to action control through reinforcement learning or imitation learning. These methods rely on large amounts of task-specific data for training, have limited generalization ability, lack interpretability, and have high practical deployment costs.

[0005] 2. Modular Approach: This approach decomposes the task by constructing modules such as semantic maps, global strategies, and local path planning. SemExp uses semantic segmentation to build a map and combines it with a goal-guided strategy for exploration. FSE optimizes navigation goal selection by constructing a frontier point semantic map. While these methods improve the interpretability of navigation decisions, they still heavily rely on supervised training in specific scenarios and lack adaptability to new targets and environmental units.

[0006] In recent years, zero-shot object navigation (ZSON) methods have been proposed, utilizing large-scale pre-trained vision-language models (VLMs) or large language models (LLMs) to achieve a general understanding of semantic targets, eliminating the need for large-scale training data to learn semantic knowledge. Examples include ESC, which combines a GLIP model with language inference to identify the room described by the target; and VLFM, which feeds the target category and image input into a vision-language model for semantic scoring. These methods achieve good transfer performance without requiring additional training.

[0007] However, existing zero-shot target navigation methods generally suffer from the following defects and shortcomings: single-frame observation limits the stability of semantic understanding. Most ZSON methods only utilize the current frame image for semantic matching, making them susceptible to interference from factors such as viewpoint bias, lighting changes, occlusion, or cluttered backgrounds, leading to inaccurate target recognition and frequent adjustments to the navigation path; they lack short-term memory mechanisms, failing to integrate historical observation information for reasoning and judgment like humans, resulting in low information utilization efficiency during navigation; and their leading edge selection strategies are coarse. Some methods rely solely on heuristic rules or overall image semantic scores for evaluation, failing to effectively distinguish between target-related and irrelevant regions, thus reducing exploration efficiency and accuracy.

[0008] In summary, existing zero-shot navigation technologies still face problems such as unstable semantic understanding, low navigation efficiency, and insufficient generalization. There is an urgent need for an intelligent navigation method that possesses human analogical memory capabilities and combines multimodal information for frontier point judgment and semantic reasoning, in order to improve the ability to find specific objects in unknown environments. Summary of the Invention

[0009] The purpose of this invention is to provide a zero-sample target navigation method and system based on object-aware memory, which solves the problems of limited target recognition, rough judgment of leading edge points and lack of time sequence awareness in the prior art, improves the accuracy of target positioning and the efficiency of navigation path, while maintaining good versatility and no training required, and is applicable to service robots, indoor detection equipment and other autonomous navigation systems.

[0010] To achieve the above objectives, on the one hand, the present invention provides a zero-sample target navigation method based on object-aware memory, comprising:

[0011] S1. Obtain environmental observation information at the current moment, construct an occupation map and extract leading edge points;

[0012] S2. Cache the environmental observation information into a memory buffer to construct a short-term spatiotemporal memory;

[0013] S3. Based on visibility constraints and distance constraints, filter relevant historical observation frames for each leading edge point from the memory buffer;

[0014] S4. Extract the image sequence of the target region based on the relevant historical observation frames;

[0015] S5. Combining the natural language description of the target object with the image sequence, obtain the semantic matching score of each front point, and select the target exploration point based on the semantic matching score;

[0016] S6. Based on the target exploration point, perform path planning and navigation control on the occupied map. When the target object is identified in the field of view, terminate the navigation task; otherwise, return to S1 until navigation is successful or the preset step limit is reached.

[0017] Optionally, the environmental observation information includes RGB images, depth maps, and pose information.

[0018] Optionally, constructing the occupancy map and extracting the frontier points includes:

[0019] The local 3D structure information of the environment is obtained by using a depth map and integrated into a global 2D grid map to construct the occupancy map;

[0020] Traverse the occupied graph, mark all grids in the free state, and obtain the free cells;

[0021] Check the N-neighborhood of the free element. If there is an unknown state in the neighborhood, the free element is marked as the leading edge point.

[0022] Spatial clustering is performed on the front points to obtain front clusters, and the geometric confidence of the front clusters is extracted to form a discrete front point set.

[0023] Optionally, the memory buffer adopts a first-in-first-out mechanism and is set with a fixed frame limit to store environmental observation information of the most recent M frames.

[0024] Optionally, the visibility constraint is that the relative angle between the camera orientation and the leading edge point in the historical observation frame is less than half of a preset field of view; the distance constraint is that the Euclidean distance between the pose position of the historical observation frame and the leading edge point is less than a preset threshold.

[0025] Optionally, by combining the natural language description of the target object with the image sequence, a semantic matching score is obtained for each front point, and target exploration points are selected based on the semantic matching scores, including:

[0026] The natural language description of the target object and the image sequence are input into a pre-trained BLIP-2 model to calculate the cosine similarity score;

[0027] The semantic matching score of each front point is calculated based on the cosine similarity score, and the front point with the highest semantic matching score is selected as the target exploration point.

[0028] Optionally, in step S6, during the path execution process, environmental observation information is continuously acquired, and the occupancy map and the set of leading points are updated at each step. After returning to step S1, the next round of calculation is performed with the latest environmental observation information, occupancy map, and set of leading points.

[0029] On the other hand, the present invention also provides a zero-sample target navigation system based on object-aware memory, comprising:

[0030] The perception module is used to acquire environmental observation information at the current moment, construct an occupancy map, and extract leading edge points;

[0031] The memory buffer module is used to cache the environmental observation information into the memory buffer to construct a short-term spatiotemporal memory;

[0032] The matching module is used to filter relevant historical observation frames for each front point from the memory buffer based on visibility constraints and distance constraints;

[0033] The extraction module is used to extract image sequences of the target region based on the relevant historical observation frames;

[0034] The semantic scoring module is used to combine the natural language description of the target object with the image sequence to obtain the semantic matching score of each front point, and select the target exploration point based on the semantic matching score;

[0035] The path planning and navigation module is used to perform path planning and navigation control on the occupied map based on the target exploration point. When the target object is identified in the field of view, the navigation task is terminated; otherwise, it returns to the perception module until navigation is successful or the preset step limit is reached.

[0036] The beneficial effects of this invention are as follows:

[0037] This invention introduces a short-term memory buffer mechanism for object awareness, comprehensively referencing multiple frames of historical images, depth maps, and robot pose information during the forefront point determination process. Through multi-view observation and spatial association, the stability of semantic understanding is significantly improved, exhibiting higher target recognition accuracy, especially in complex environments (such as occlusion, viewpoint shift, and uneven lighting). This invention performs object-level image-text matching on object image regions extracted from historical frames, avoiding the problem of semantic redundancy in a single frame's overall image, achieving more refined and accurate semantic association, thereby enhancing the intelligence of target guidance. Traditional navigation methods typically rely on large-scale training for specific environments and target categories, while this invention achieves zero-shot navigation by calling a general pre-trained model. That is, it can be deployed in entirely new scenes and target categories without any task-related training, significantly reducing deployment costs and improving system adaptability. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a zero-sample target navigation method based on object-aware memory according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] This embodiment provides a zero-sample target navigation method based on object-aware memory, such as... Figure 1 As shown, it includes:

[0043] S1. Obtain environmental observation information at the current moment, construct an occupation map and extract leading edge points;

[0044] S2. Cache the environmental observation information into a memory buffer to construct a short-term spatiotemporal memory;

[0045] S3. Based on visibility constraints and distance constraints, filter relevant historical observation frames for each leading edge point from the memory buffer;

[0046] S4. Extract the image sequence of the target region based on the relevant historical observation frames;

[0047] S5. Combining the natural language description of the target object with the image sequence, obtain the semantic matching score of each front point, and select the target exploration point based on the semantic matching score;

[0048] S6. Based on the target exploration point, perform path planning and navigation control on the occupied map. When the target object is identified in the field of view, terminate the navigation task; otherwise, return to S1 until navigation is successful or the preset step limit is reached.

[0049] Specifically, this embodiment introduces a short-term memory buffer mechanism for object awareness, comprehensively referencing multiple frames of historical images, depth maps, and robot pose information during the forefront point determination process. Through multi-view observation and spatial association, the stability of semantic understanding is significantly improved, especially demonstrating higher target recognition accuracy in complex environments (such as occlusion, viewpoint shift, and uneven lighting). This embodiment performs object-level image-text matching on object image regions extracted from historical frames, avoiding the problem of semantic redundancy in a single frame's overall image, achieving more refined and accurate semantic association, thereby enhancing the intelligence of target guidance.

[0050] Specifically, it includes the following:

[0051] Step 1: Obtain environmental observation information at the current moment, including RGB image, depth map, robot current pose information, construct occupancy mesh map and extract leading edge points.

[0052] Constructing the occupancy map and extracting the frontier points includes:

[0053] The local 3D structure information of the environment is obtained by using a depth map and integrated into a global 2D grid map to construct the occupancy map;

[0054] Traverse the occupied graph, mark all grids in the free state, and obtain the free cells;

[0055] Check the N-neighborhood of the free element. If there is an unknown state in the neighborhood, the free element is marked as the leading edge point.

[0056] Spatial clustering is performed on the front points to obtain front clusters, and the geometric confidence of the front clusters is extracted to form a discrete front point set.

[0057] Specifically, it includes:

[0058] (1) Construct the occupancy graph:

[0059] At each time step t, the robot receives an RGB image I from the sensor. t Depth map D t and the current pose p t By projecting depth information from the sensor coordinate system to the global coordinate system, a local 3D structure of the environment is obtained. This 3D structural information is then integrated into a global 2D raster map M. occ ∈{0, 1, -1} H×M In this context, 0 represents free space, 1 represents an obstacle, and -1 represents an unknown region.

[0060] (2) Constructing a frontier map;

[0061] The frontier is defined as: a grid point currently in free space that is adjacent to at least one unknown region. These points represent the boundaries between explored and unexplored regions and are highly valuable candidate target points for exploration. First, the entire map M is traversed. occ First, all grid cells in any state are marked as free. Then, the 8-neighborhood of these free cells is examined. If an unknown state exists in the neighborhood, the free cell is marked as a front point. To avoid generating a dense and redundant set of front points, spatial clustering is performed on the front points. Adjacent front points are merged into a front cluster, and the geometric confidence of each cluster is extracted as the detected front point, forming a discrete front point set: F = {f1, f2, ..., f...} N}, f i ∈R 2 .

[0062] Step 2: Cache the RGB image, depth map and pose information into the object-aware memory buffer to build a short-term spatiotemporal memory containing multiple frames of historical observation information.

[0063] Maintain an observation queue of length M using a first-in, first-out (FIFO) approach: H = {(I t-M+1 D t-M+1 p t-M+1 ), ..., (I t D t p t Each of these items contains an RGB image I. t Depth Image D t And the agent's global pose p at that moment t .

[0064] Step 3: For each front point, based on the constraints of the viewing angle and spatial distance, select the historical observation frames associated with it from the memory buffer.

[0065] The visibility constraint is that the relative angle between the camera orientation and the leading edge point in the historical observation frame is less than half of the preset field of view angle; the distance constraint is that the Euclidean distance between the pose position of the historical observation frame and the leading edge point is less than a preset threshold.

[0066] Specifically as follows:

[0067] (1) Visibility constraint: Consider whether the view cone angle of the robot's observation direction in the historical frame covers the current front point position. Let the pose at a certain moment in the historical frame be p. t =(x t ,y t ,θ t The position of the leading edge is: f i =(x f ,y fThe azimuth angle of the leading edge point relative to the camera is defined as: φ t =arctan2(y f -y t ,x f -x t If the angle falls within the camera's field of view, that is: Where α is the horizontal field of view of the camera, it is assumed that the leading edge point may be seen in this frame.

[0068] (2) Distance constraint: It further requires that the Euclidean distance between the leading edge point and the historical frame acquisition is within a certain threshold d. th Within:

[0069] ||f i -(x t y t )||2≤d th ;

[0070] To ensure that observation frames close to and visible to the leading edge are selected, and to reduce interference from irrelevant data, historical frames meeting the following conditions will be retained for semantic analysis of the leading edge:

[0071]

[0072] Step 4: Perform object detection on relevant historical images and extract the image sequence of the object region.

[0073] For image I in each associated frame, a frozen object detection model (such as Grounding DIDO) is used to extract potential object regions in the image, generating an image sequence set.

[0074] Step 5: Input the image sequence and the natural language description of the target object into the pre-trained visual language model to obtain the semantic matching score of each front point.

[0075] The visual language model BLIP-2 is used to measure the semantic relevance between leading points and navigation targets by calculating the similarity between images and text. The core innovation of this embodiment lies in patch-level semantic matching based on object memory, avoiding redundant processing of the entire image. Firstly, for each... Construct a text prompt based on a natural language description, for example: T = "Seems like there is a<target object> "ahead." The patch and text prompt T are then input into the BLIP-2 model to calculate the cosine similarity score:

[0076] s j =BLP(p j ,T);

[0077] To obtain the frontier point f i The overall semantic score is averaged across all its patches:

[0078]

[0079] Where S(f) i ) represents the semantic score of the frontier point.

[0080] Finally, the point with the highest semantic score among all candidate frontiers is selected as the next exploration target:

[0081]

[0082] Where F is the set of all currently detected frontier points.

[0083] Emphasizing a memory-driven semantic perception strategy, this approach effectively improves the accuracy and efficiency of semantic reasoning by retrospectively analyzing previously observed object regions.

[0084] Step 6: Select the frontier point with the highest semantic matching score as the target exploration point, and perform path planning and navigation control in the current occupied map. When the target object is identified in the field of view, the navigation task is terminated; otherwise, repeat the above steps until navigation is successful or the preset step limit is reached.

[0085] After determining the navigation points, the classic Fast Movement Method (FMM) is used to perform local path planning on the currently constructed occupancy grid map. FMM efficiently generates the shortest collision-free path from the robot's current position to the target forefront. Subsequently, navigation actions are executed along this path using a deterministic strategy. During path execution, the robot continuously perceives the environment and updates the occupancy map and the forefront set at each step. A target detection strategy from VLFM is employed. When an object matching the target semantics g is detected, the navigation task is considered complete, and the process terminates; if no target is detected, the semantic score of the forefront is re-evaluated, a new target is selected, and the above process is repeated.

[0086] On the other hand, this embodiment also provides a zero-sample target navigation system based on object-aware memory, including:

[0087] The perception module is used to acquire environmental observation information at the current moment, construct an occupancy map, and extract leading edge points;

[0088] The memory buffer module is used to cache the environmental observation information into the memory buffer to construct a short-term spatiotemporal memory;

[0089] The matching module is used to filter relevant historical observation frames for each front point from the memory buffer based on visibility constraints and distance constraints;

[0090] The extraction module is used to extract image sequences of the target region based on the relevant historical observation frames;

[0091] The semantic scoring module is used to combine the natural language description of the target object with the image sequence to obtain the semantic matching score of each front point, and select the target exploration point based on the semantic matching score;

[0092] The path planning and navigation module is used to perform path planning and navigation control on the occupied map based on the target exploration point. When the target object is identified in the field of view, the navigation task is terminated; otherwise, it returns to the perception module until navigation is successful or the preset step limit is reached.

[0093] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A zero-shot goal navigation method based on object perception memory, characterized in that, The method comprises the following steps: S1, acquiring environment observation information at the current moment, constructing an occupancy grid and extracting frontiers, comprising: acquiring local 3D structure information of the environment through a depth map and integrating it into a global 2D grid map to construct the occupancy grid; traversing the occupancy grid, marking all grids in a free state to obtain free cells; checking the N-neighborhood of the free cells, if there is an unknown state in the neighborhood, the free cell is marked as a frontier; spatially clustering the frontiers to obtain frontier clusters and extract the geometric confidence of the frontier clusters to form a discrete frontier set; S2, caching the environment observation information to a memory buffer to construct short-term spatiotemporal memory; S3, according to the visibility constraint and the distance constraint, screening relevant historical observation frames of each frontier from the memory buffer; S4, extracting object region image sequences based on the relevant historical observation frames; S5, combining the natural language description of the target object with the image sequences to obtain a semantic matching score of each frontier and selecting a target exploration point based on the semantic matching score, comprising: inputting the natural language description of the target object and the image sequences into a pre-trained BLIP-2 model to calculate a cosine similarity score; calculating a semantic matching score of each frontier based on the cosine similarity score and selecting the frontier with the highest semantic matching score as the target exploration point; S6, path planning and navigation control according to the target exploration point in the occupancy grid, terminating the navigation task when the target object is identified in the field of view, otherwise returning to S1 until the navigation is successful or the preset step limit is reached.

2. The object perception memory based zero-shot goal navigation method of claim 1, wherein, The environment observation information comprises an RGB image, a depth map and pose information.

3. The object perception memory based zero-shot goal navigation method of claim 1, wherein, The memory buffer adopts a first-in-first-out mechanism and is provided with a fixed frame number limit for storing the latest M frames of environment observation information.

4. The object perception memory based zero-shot goal navigation method of claim 1, wherein, The visibility constraint is that the relative angle between the camera orientation in the historical observation frame and the frontier is less than half of the preset field of view angle; and the distance constraint is that the Euclidean distance between the pose position of the historical observation frame and the frontier is less than a preset threshold.

5. The object perception memory based zero-shot goal navigation method of claim 1, wherein, In S6, during path execution, environment observation information is continuously acquired and the occupancy grid and the frontier set are updated at each step, and the latest environment observation information, occupancy grid and frontier set are used for the next round of calculation after returning to S1.

6. An object perception memory based zero-shot goal navigation system, characterized in that, The method comprises the following steps: a perception module for acquiring environment observation information at the current moment, constructing an occupancy grid and extracting frontiers, comprising: acquiring local 3D structure information of the environment through a depth map and integrating it into a global 2D grid map to construct the occupancy grid; traversing the occupancy grid, marking all grids in a free state to obtain free cells; checking the N-neighborhood of the free cells, if there is an unknown state in the neighborhood, the free cell is marked as a frontier; spatially clustering the frontiers to obtain frontier clusters and extract the geometric confidence of the frontier clusters to form a discrete frontier set; a memory buffer module for caching the environment observation information to a memory buffer to construct short-term spatiotemporal memory; a matching module configured to filter relevant historical observation frames of each frontier point from the memory buffer according to the visibility constraint and the distance constraint; an extraction module configured to extract an object region image sequence based on the relevant historical observation frames; a semantic scoring module configured to combine a natural language description of a target object with the image sequence to obtain a semantic matching score of each frontier point, and select a target exploration point based on the semantic matching score, including: inputting the natural language description of the target object and the image sequence into a pre-trained BLIP-2 model to calculate a cosine similarity score; calculating a semantic matching score of each frontier point based on the cosine similarity score, and selecting a frontier point with the highest semantic matching score as the target exploration point; a path planning and navigation module configured to perform path planning and navigation control in the occupancy map according to the target exploration point, terminate the navigation task when the target object is identified in the field of view, or return to the perception module until the navigation is successful or the upper limit of the preset step number is reached.

Images