A zero-shot embodied target navigation method and system based on prior semantic map

Abstract

The application discloses a zero-sample embodied target navigation method and system based on prior semantic maps. The method uses scene prior data to infer the existence probability of target objects in different room types through a large language model to construct a prior value map. In the navigation process, based on the current observation image, the similarity score of the observation and the semantic description of the target object is calculated using a visual language model, and an observation value map is generated. The prior value map and the observation value map are dynamically fused, the prior value weight is adaptively adjusted with the accumulation of observation information, and a fused value map is obtained. Based on the fused value map, the room-level semantic prior points and the front points are evaluated, and the dual-layer search navigation of the upper room guidance and the lower front exploration is alternately performed. When the target object is detected, the target navigation is switched. Through the value fusion and hierarchical navigation architecture, the application effectively utilizes the environmental prior semantic information, and improves the efficiency and stability of target search under the zero-sample condition.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot environmental perception and autonomous navigation technology, specifically involving key technologies such as embodied intelligent navigation, visual semantic understanding, and autonomous exploration decision-making. In particular, it relates to a zero-shot embodied target navigation method that integrates prior semantic information and online observation information. This method can be widely applied to scenarios where service robots, indoor mobile robots, virtual intelligent agents, and automated inspection systems need to perform target search and navigation tasks in unknown environments. Background Technology

[0002] With the continuous development of artificial intelligence technology, embodied AI has gradually become one of the important research directions in the field of robotics. Embodied target navigation tasks require intelligent agents to search for and locate target objects in unknown or partially unknown environments, based on given target semantic descriptions (such as "find the refrigerator" or "find the sofa"), through processes such as autonomous perception, environment mapping, semantic understanding, and path planning.

[0003] Traditional navigation methods primarily rely on Simultaneous Localization and Mapping (SLAM) technology to plan paths by constructing a geometric map of the environment. However, these methods depend mainly on the geometric structure information of the environment and lack the ability to understand semantic information, resulting in low efficiency when performing target search tasks, especially when the target location is unknown, requiring large-scale, targetless exploration.

[0004] To address these issues, the research community has proposed learning-based methods that jointly model object detection and navigation strategies using deep neural networks. However, these methods typically rely on large amounts of labeled data and can only identify object categories that have appeared during training, making it difficult to generalize to unknown categories and limiting their widespread application in real-world scenarios.

[0005] In recent years, with the development of visual language models, zero-shot object detection and navigation methods have gradually become a research hotspot. These methods utilize the open vocabulary capabilities of pre-trained models to achieve object recognition even without having seen the target category before. However, existing zero-shot embodied navigation methods still have the following shortcomings: First, most methods rely solely on real-time observation information for decision-making, lacking the utilization of prior semantic information about the environment (such as room type and object distribution patterns), resulting in low search efficiency. Second, in unknown environments, they typically employ a frontier-based exploration strategy, which relies solely on geometric information, lacks semantic guidance, and easily generates a large number of invalid exploration paths. Third, existing methods do not effectively separate global path planning from local exploration during navigation, lacking a hierarchical decision-making mechanism, leading to unstable navigation behavior. Finally, there is a lack of an effective fusion mechanism between prior information and observation information, making it impossible to dynamically adjust the decision-making strategy according to environmental changes. Therefore, how to fully utilize prior semantic information under zero-shot conditions and combine it with real-time observation information to construct an efficient navigation decision-making mechanism has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the problems of low efficiency, insufficient utilization of semantic information, and unreasonable decision-making structure in existing zero-sample embodied target navigation, this invention provides a zero-sample embodied target navigation method based on a priori semantic maps. By introducing prior semantic information, constructing a value map, and designing a hierarchical navigation strategy, it achieves efficient and stable target search and navigation.

[0007] According to one aspect of the present invention, a zero-sample embodied target navigation method based on a priori semantic maps is provided, comprising: By using pre-acquired scene prior data, the probability of the existence of target objects in different room types is inferred through a large language model, and a prior value map that assigns semantic value to grid areas is constructed. During navigation, based on the current observed image, the similarity score between the current observed image and the semantic description of the target object is calculated using a visual language model, and an observation value map is generated by combining the field of view cone confidence score. The prior value map and the observation value map are dynamically fused together, and the fusion weight is adaptively adjusted as the observation information accumulates to obtain the fused value map. Based on the fused value map, room-level semantic prior points and frontier points are evaluated, and a two-layer search navigation is performed, alternating between upper-level room guidance and lower-level frontier exploration. When a target object is detected using the open vocabulary target detection model, switch to target navigation with the target object's location as the navigation target.

[0008] As a further technical solution, the large language model includes a first language model and a second language model that are independent of each other; The first language model is used to infer the room type based on the categories of objects in the room; The second language model is used to infer the probability of the existence of the target object in each room based on the semantic description of the target object and the room type.

[0009] As a further technical solution, the construction of the prior value map also includes: Generate a top-down view of scene accessibility, project objects from the scene prior data onto the top-down view, and assign a room ID to each accessible pixel in the top-down view through multi-source breadth-first search to obtain room area division; The centroid of each room region is calculated as the room-level semantic prior point. If the centroid is in an impassable area, it is projected to the nearest passable pixel through breadth-first search.

[0010] As a further technical solution, the field of view cone confidence level Based on the angle between the current pixel and the camera's optical axis and camera horizontal field of view The calculation formula is as follows: .

[0011] As a further technical solution, the generation of the observation value map includes: Calculate the similarity score between the current RGB image and the semantic description of the target object using a visual language model; The product of the similarity score and the field-of-view cone confidence score is projected into the map space as the new value generated by the current observation; For pixels in regions that overlap with historical observations, a weighted update is performed as follows: Value channel updated to ; Confidence channel updated to ;in, The pixel at time t-1 Updated confidence level The pixel at time t-1 The updated observational value, Pixel at time t The confidence level obtained from the new observation Pixel at time t The observational value obtained from the new observations.

[0012] As a further technical solution, the prior value weight of the dynamic fusion The value is 1 minus the maximum value in the current observation value map, and the fusion formula is: , For pixels The a priori value of a place. For pixels The observational value at time t.

[0013] As a further technical solution, room-level semantic priors are evaluated, including: For each room-level semantic prior point, calculate the traversable distance from the agent's current position to that room-level semantic prior point. and normalized to ,in Distance threshold; Room-level semantic prior review is divided into ,in This represents the fusion value at the room-level semantic prior point. These are the weighting coefficients; The room-level semantic prior point with the highest score is selected as the navigation target for upper-level room guidance.

[0014] As a further technical solution, the frontier points are evaluated, including: The sum of the fusion values ​​of the fusion value maps within the neighborhood area centered on the frontier point is calculated as the frontier point score; The highest-scoring frontier point is selected as the navigation target for exploration of the lower frontier.

[0015] According to one aspect of the present invention, a zero-sample embodied target navigation system based on a priori semantic maps is provided, comprising: The prior semantic map construction module is used to infer the probability of the existence of a target object in different room types by using scene prior data and a large language model, and to construct a prior value map. The observation value map construction module is used to calculate the similarity score between the current observation image and the semantic description of the target object using a visual language model during navigation, and to generate an observation value map by combining the field of view cone confidence score. The value fusion module is used to dynamically fuse the prior value map and the observed value map. The fusion weight is adaptively adjusted as the observed information accumulates to obtain the fused value map. The dual-layer search and navigation module is used to evaluate room-level semantic prior points and frontier points based on the fused value map, perform dual-layer search and navigation that alternates between upper-layer room guidance and lower-layer frontier exploration, and switch to target navigation with the target object's location as the navigation target when a target object is detected by the open vocabulary target detection model.

[0016] According to one aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the aforementioned zero-sample embodied target navigation method based on a priori semantic maps.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: First, by introducing a prior semantic map, environmental semantic information is effectively utilized to improve target search efficiency. Second, by constructing a value map and guiding exploration, invalid paths are significantly reduced, improving exploration efficiency. Third, by dynamically fusing prior information and observation information, adaptive adjustment of the navigation strategy is achieved. Furthermore, by constructing a two-layer search architecture, global navigation and local exploration are coordinated, improving system stability and robustness. Finally, this method supports zero-shot target detection, has good generalization ability, and can adapt to various complex environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating the principle of a zero-sample embodied target navigation method based on a priori semantic map, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the prior semantic information processing flow provided in an embodiment of the present invention; Figure 3 A schematic diagram of a two-layer search architecture of "prior information guidance - frontier exploration" provided for embodiments of the present invention. Detailed Implementation

[0020] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0022] In practical implementation, the method of this invention can be deployed in a mobile robot system. The system first loads prior semantic information and constructs an initial value map. During navigation, it acquires environmental observation data in real time, dynamically updates the frontier map and the observation value map, and performs fusion processing. Subsequently, based on the fused value distribution, it selects the optimal navigation target and executes the navigation task through a two-layer search architecture. When a target object is detected, the system immediately switches to target navigation mode and completes path planning. Through the above process, efficient search and localization of target objects are achieved.

[0023] See Figure 1 As shown, the sample-embodied target navigation method provided in this embodiment of the invention includes the following steps.

[0024] Step 1: Data Preprocessing and Map Initialization. In this invention, the system first receives RGB images, depth images, and agent pose information from sensors. For the input depth image, it is converted into 3D point cloud data using camera intrinsic parameters, and the point cloud data is projected onto a global coordinate system. Simultaneously, the environment is rasterized to construct a 2D occupancy raster map, where each raster cell is marked as a free area, an occupied area, or an unknown area based on observation results.

[0025] In step 1, after the navigation task begins, the system first receives the RGB image, depth image, and pose information collected by the agent at the current moment. Let the current moment be... RGB image is Depth image is The pose of the intelligent body is For any pixel in the depth image The system backprojects the camera's intrinsic parameters into three-dimensional space to obtain the corresponding spatial points: , in, For the camera intrinsic parameter matrix, For pixels The depth value at that location, where T is the matrix transpose. Then, it is combined with the agent's pose. Transform the 3D point to the global coordinate system: , The system filters the converted 3D points, removing those that are too close, have abnormal heights, or do not meet obstacle detection criteria, and projects the remaining points onto a 2D raster map. Let the front edge map be... Each grid cell stores one of three states: , In this system, the free area represents the traversable area for the agent, the occupied area represents the obstacle area, and the unknown area represents the area that has not yet been observed. Through the above processing, the system completes the initialization of the spatial coordinate reference required for the basic occupied map, the frontier map, and the subsequent value map.

[0026] Step 2: Prior Semantic Map Construction. To fully utilize the prior semantic information of the environment, this invention designs a prior semantic map construction method, such as... Figure 2 As shown in the diagram. Specifically, the input scene prior data is first parsed to extract information such as object categories, spatial locations, and room affiliations in the environment. Then, the scene top-down view is fused with the object location information, and the object coordinates are projected onto a grid map through spatial mapping. Based on the object distribution, the environment is divided into rooms. After obtaining the room division results, the semantic description of the target object and each room type are input into a large language model, which infers the probability of the target object's presence in different rooms. Based on this probability distribution, each grid cell in the map is assigned a corresponding semantic value, thereby generating an initial prior value map. Simultaneously, the center position of each room is extracted as a room-level semantic prior point, providing candidate targets for subsequent global navigation.

[0027] In step 2, the construction of the prior semantic map does not directly rely on real-time observation. Instead, it utilizes existing prior information such as rooms, objects, and their spatial locations in the scene to generate an initial value map and room-level semantic prior points. The core idea of ​​this process is to first convert prior information from different sources into a top-down view with room type labels, and then further generate an initial value map and candidate room points that the agent can directly access.

[0028] Scene prior data can be stored in semantic text files (e.g., semantic.txt) and 3D model files (e.g., semantic.glb). The semantic.txt file contains object identity information, category, and room ID, while the semantic.glb file, after being processed by the simulator, yields the object ID and its global coordinates. Based on the uniqueness of the object ID, the system matches the information from the two types of files to form the following object semantic set: , in, Let be the identification number of the i-th object. For object categories, For its room ID, Let N be the position of the object in the global coordinate system, and N be the total number of objects. Then, the system generates a top-down view of scene accessibility using a simulation environment (such as Habitat-sim). This top-down view represents scene accessibility in binary form, where accessible areas are valued as 1 and inaccessible areas as 0. The global coordinates of the objects are converted to map coordinates and projected onto the top-down view. At this point, the top-down view contains several object points labeled with room IDs. To obtain a complete room area division, the system uses a multi-source breadth-first search (BFS) algorithm to assign a room ID to each pixel in the top-down view, which can be represented as: , in, Represents map pixels The room ID. After completing the room area division, the system inputs the set of object categories within the same room into the first language model. It infers the room type based on the composition of the objects inside the room: .

[0029] After obtaining the room types, the system further describes the semantics of the target objects. Input the second language model with each room type Infer the probability of the target object appearing in different rooms: , Based on this probability result, an initial prior value is assigned to all map pixels belonging to the corresponding room: .

[0030] Simultaneously, the system calculates the centroid of each room region as room-level semantic prior points: , in, For room The set of pixels. If the centroid of a room is not located in a passable area, it is projected to the nearest passable pixel using the BFS algorithm, thus ensuring that the candidate room point can be actually navigated to by the agent. It is worth noting that in this invention... and Maintaining independent design, the former is used to infer the room type based on objects in the room, while the latter is used to infer the probability of existence based on the semantic description of the target object and the room type, thereby avoiding interference from specific object information in the room in determining the probability of the target's existence.

[0031] Step 3: Construction of the Frontier Map and Observational Value Map. During navigation, the system continuously updates the frontier map based on current observations. Frontier points are defined as the boundaries between explored and unknown areas. By detecting changes in the state of adjacent areas in the raster map, all frontier points are extracted and a set of candidate exploration points is formed. Simultaneously, the system constructs an observational value map. Specifically, it calculates the similarity between the current RGB image and the target semantic description based on a visual language model, and introduces a field-of-view cone confidence model, assigning different weights to different pixel positions. Subsequently, the similarity results are projected onto the map space to form an observational value distribution, while simultaneously maintaining both the value channel and the confidence channel to improve the reliability of the observational information.

[0032] In step 3, the frontier map provides geometric candidate points for exploration, while the observation value map provides semantic exploration priorities. During frontier point extraction, the system detects the boundary line between explored and unknown regions and takes the midpoint of the boundary line as a potential frontier path point. Let the set of frontier points be: , in, This represents the number of foreground points extracted at the current moment. As the agent continues to explore, the number and spatial distribution of foreground points will change continuously until the entire environment is completely explored and no remaining foreground points exist. The observation value map consists of two parts: a value channel and a confidence channel. The system first generates a cone-shaped region based on the current camera pose to represent the current camera field of view from a top-down perspective. In this region, pixels closer to the camera's optical axis have higher confidence, while pixels at the edge of the field of view have lower confidence. Let the angle between a pixel and the camera's optical axis be θ. The horizontal field of view of the intelligent agent camera is Then the confidence level of the pixel can be expressed as: , When the pixel is located in the optical axis direction When the pixel is located at the edge of the field of view Subsequently, the system updates the cone region using the depth image, excluding the field of view areas obscured by obstacles. For the remaining visible regions, a pre-trained BLIP-2 visual language model is used to calculate the current RGB image and the text prompt "Seems like there is a...".<target_object> Cosine similarity score between "ahead". , in, Representing image features, Indicates the target text prompt features. This indicates the degree of matching between the current observation and the target semantics. The system projects this similarity score onto the value channel of the cone region to obtain the new value generated by the current observation: .

[0033] When an agent moves to a new location, its field of view may overlap with previously observed areas. In this case, for each pixel within the overlapping area, the system updates the observation value weighted by the confidence levels of the old and new observations: , Simultaneously update the confidence channel: .

[0034] The confidence update method described above favors high-confidence observations, resulting in higher confidence for regions closer to the optical axis, with less occlusion, and more stable semantic matching in the observation value map. It should be noted that prior values ​​do not participate in the update process of the observation value channel; the observation value map is obtained solely from real-time observations.

[0035] Step 4: Fusion of Prior Value and Observational Value. To effectively combine prior information with real-time observational information, this invention proposes a value fusion mechanism. Specifically, the prior value map and the observational value map are weighted and fused to obtain the final value map used for navigation decision-making. During the fusion process, the weights are dynamically adjusted according to the observational confidence level, and a time decay mechanism is introduced, so that the system initially relies more on prior information and gradually relies on observational information in the later stages, thereby achieving adaptive optimization of the navigation strategy.

[0036] In step 4, to achieve the synergistic utilization of prior value and real-time observation value, this invention updates the observation value. with a priori value By merging the data, a final value map is obtained for navigation decisions. : , in, The prior value weights are dynamically adjusted according to the fusion strategy: , The final fusion result is: , Through the above fusion strategy, the system can generate a fused value map frame by frame throughout the navigation process, realizing the dynamic integration of prior semantic information and real-time observation information.

[0037] Step 5: Candidate Navigation Point Generation and Evaluation. Based on the fused value map, the system generates two types of key navigation points. The first type is room-level target points. By comprehensively considering the semantic value and distance factors of rooms, all room points are ranked, and the optimal room point is selected as the global navigation target. The second type is front-line points. By calculating the accumulated value of the area surrounding the front-line point, candidate front-line points are evaluated, and the optimal front-line point is selected as the local exploration target.

[0038] In step 5, candidate navigation points include two categories: room-level semantic prior points and frontier points. Room-level semantic prior points are used in the upper-layer guidance mode, while frontier points are used in the lower-layer exploration mode. For room point selection, the system considers both distance and value factors. The distance factor represents the shortest passable distance between the agent's current position and the target room point, while the value factor represents the fusion value of the room point's location. To ensure that both factors are of the same magnitude, the distance is first normalized: , in, This represents the shortest accessible distance from the agent's current position to the i-th candidate room. This is the passable distance threshold. Theoretically, It should be no less than the passable distance between any two points in the scene to ensure that the normalized distance is no greater than 1; however, in practical applications, this threshold is difficult to obtain accurately, so this invention takes the smaller value between the original normalized result and 1.

[0039] After obtaining the normalized distance, the system integrates the distance factor with the value factor to obtain the comprehensive value of each room point. Let the integrated value at the i-th candidate room point be . Then a linear weighted strategy can be adopted: , in, Parameters used to adjust the weighting ratio of value factors and distance factors.

[0040] For the evaluation of frontier points, the system dynamically accumulates candidate frontier points during the observation process and uses the weighted value of the surrounding area of ​​each candidate frontier point as the evaluation index. Let the k-th frontier point be... Its neighboring region is Then the frontier score can be expressed as: , The system selects the frontier point with the highest score as the optimal frontier point for lower-level local exploration.

[0041] Step 6: Two-layer search navigation strategy. To improve navigation efficiency, this invention constructs a two-layer search mechanism of "prior information guidance - frontier point exploration," such as... Figure 3 As shown in the diagram, the upper layer is the guidance mode, primarily responsible for global path planning, guiding the agent to high-probability areas by selecting the optimal room point; the lower layer is the exploration mode, performing free exploration within a local area based on a leading edge point. Once the agent reaches the target room, it switches to exploration mode; when exploration meets certain conditions or is completed, it switches back to guidance mode. This cyclical switching between the two modes achieves coordinated optimization of global and local navigation.

[0042] In step 6, the present invention constructs a two-layer search mechanism of "prior information guidance - frontier point exploration" to make full use of the best room point and the best frontier point. In this architecture, the upper layer is the guidance mode, which mainly decides the next room to be visited based on the comprehensive value of the agent's current position and each room point; the lower layer includes an initialization mode and an exploration mode, which mainly drive the agent to freely explore within the room through a value map.

[0043] When the task starts, the agent first enters the guidance mode. In the first frame of this guidance mode, the system sorts the candidate rooms and selects the room with the highest score as the current guidance target. , in, Indicates candidate room points, This represents the overall score. During the guidance process, the current best room point remains fixed and does not frequently switch with changes in the agent's position to avoid navigation target jitter. When the agent reaches the vicinity of the target room, the guidance mode ends, and the system removes the current room point from the candidate sequence to prevent subsequent repeated exploration.

[0044] Subsequently, the agent switches to initialization mode, rotating once in place to obtain the leading edge point of the surrounding environment and updating the leading edge map and observation value map. After initialization, the system enters exploration mode, using the optimal leading edge point as the navigation target: , Indicates candidate frontier points, This represents the set of all candidate frontier points at the current moment. When the number of exploration steps reaches a preset threshold or there are insufficient effective frontier points in the current area, the system switches back to the guidance mode and selects the next candidate room to continue exploration. Through periodic switching between guidance mode, initialization mode, and exploration mode, the agent can achieve hierarchical navigation from room-level global guidance to local exploration within a room. If a target object is detected at any stage, the system directly switches to navigation mode and uses the target object's location as the final navigation target.

[0045] Step 7: Target Detection and 3D Localization. During zero-shot navigation, the system needs to determine in real-time whether a target object appears in the current field of view. This invention uses an open-vocabulary target detection model to detect the input image and obtain the bounding box information of the target candidate region. Subsequently, a segmentation model is used to finely segment the candidate region, obtaining a pixel-level mask of the target. Combined with depth image information, the pixels within the mask region are mapped to 3D space, thereby calculating the 3D position of the target object relative to the agent. When a target object is detected, the system switches the current navigation target from the exploration point to the target object's position and enters the target navigation phase.

[0046] In step 7, the object detection module determines whether a target object appears in the current field of view, which is a crucial step for the successful completion of the zero-shot embodied target navigation task. Unlike conventional object detection, zero-shot object detection cannot rely on training data specific to a particular target category. Therefore, this invention combines a conventional object detection model with an open-vocabulary object detection model. Specifically, for common object categories included in the COCO dataset, the system uses the YOLOv7 model for detection; for target objects outside the COCO categories, the GroundingDINO open-vocabulary object detection model based on natural language prompts is introduced as a supplementary detector. GroundingDINO supports object detection through natural language text descriptions without requiring additional fine-tuning for target categories.

[0047] Let the target object text description be The current RGB observation is The set of detection bounding boxes output by GroundingDINO can be represented as: , in, This represents the position and size of the i-th detection bounding box. This indicates the corresponding detection confidence level. To detect the number of bounding boxes, These represent the x-coordinate, y-coordinate, width, and height of the detected bounding box. The system selects the bounding box with the highest confidence or that meets the threshold requirement as the target candidate region, and uses the lightweight image segmentation model MobileSAM to perform fine segmentation on this region, obtaining the pixel-level mask of the target object. , in, For the selected target, detect the bounding box. Let be the target mask. After obtaining the target mask, the system acquires the depth information of pixels within the mask region using a depth image and calculates the 3D position of the target object relative to the agent. Let the set of valid pixels within the mask region be denoted as . Then the three-dimensional position of the target can be represented as: , Once the three-dimensional position of the target object is determined, the system will switch the navigation target from the current exploration point or room point to the target object's position and enter the target navigation phase.

[0048] Step 8: Path Planning and Execution. After determining the navigation target point, the system adopts a point-target navigation strategy, generating an action sequence based on the target point's position on the grid map and controlling the agent to execute the movement. This process can be combined with classic path planning algorithms to achieve path optimization, thereby ensuring the safety and efficiency of the navigation process.

[0049] In step 8, the system employs a unified point-target navigation strategy for motion control. Regardless of whether the agent is currently in guided mode, exploration mode, or target navigation mode, its core task is to move from its current position to a designated target point; the only difference lies in the source of the target point. When no target object is detected, the target point originates from the optimal room point or the optimal front edge point; once a target object is detected, the target point switches to the three-dimensional position of the target object. The navigation target can be represented as: , in, For the best room location, As the best frontier point, The PointNav policy takes the coordinates of the target point in a 2D grid map as input and outputs the sequence of actions the agent should perform: , in, Indicates the current state of the agent. Indicates the current navigation target. This represents the action performed by the agent. Through this unified navigation strategy, the system can smoothly switch the source of the target point between different task stages, thereby realizing a complete navigation process of prior guidance, local exploration, and target arrival.

[0050] The present invention also provides a zero-sample embodied target navigation system based on a priori semantic map, which can be used to perform the navigation methods described in the above method embodiments.

[0051] The system includes: The prior semantic map construction module utilizes pre-acquired scene prior data and a large language model to infer the probability of a target object's presence in different room types, constructing a prior value map that assigns semantic value to grid regions. Specifically, the module parses object category, room affiliation, and global location information from the scene prior data, generates a scene accessibility top-down view, projects objects into the view, and assigns a room ID to each accessible pixel through multi-source breadth-first search, resulting in room region division. The module contains independent first and second language models. The first language model infers the room type based on object categories within the room, while the second language model infers the probability of a target object's presence in each room based on the target object's semantic description and room type, assigning prior value to grid cells within the corresponding room region based on this probability. Simultaneously, the module calculates the centroid of each room region as a room-level semantic prior point; if the centroid is located in an inaccessible area, it is projected to the nearest accessible pixel.

[0052] The observation value map construction module is used during navigation to calculate the similarity score between the current observed image and the semantic description of the target object using a visual language model, and then combine this with the field-of-view cone confidence score to generate an observation value map. Specifically, the module uses a visual language model to calculate the similarity score between the current RGB image and the semantic description of the target object, and projects the product of the similarity score and the field-of-view cone confidence score as the new value generated by the current observation onto the map space. For pixels overlapping with historical observations, the module updates the observation value with a weighted average based on the confidence scores of the new and old observations, giving higher-confidence observations a higher weight in the value map. The field-of-view cone confidence score is calculated based on the angle between the current pixel and the camera's optical axis and the camera's horizontal field of view, with the highest confidence score at the center of the optical axis, gradually decreasing towards the edge of the field of view.

[0053] The value fusion module dynamically fuses the prior value map with the observed value map. The prior value weights adaptively adjust as observed information accumulates, resulting in a fused value map. Specifically, the value fusion module employs a weighted fusion strategy, where the prior value weight is 1 minus the maximum value in the current observed value map. In the initial navigation phase, with limited observed information, the prior value weights approach 1, and navigation decisions rely more on prior values ​​for global guidance. As observed information accumulates, the observed value weights adaptively increase, and navigation decisions gradually transition to relying on observed values ​​for precise local positioning.

[0054] A dual-layer search and navigation module is used to evaluate room-level semantic prior points and frontier points based on the fused value map, and to perform a dual-layer search and navigation that alternates between upper-layer room guidance and lower-layer frontier exploration. When a target object is detected by the open-vocabulary target detection model, it switches to target navigation with the target object's location as the navigation target. Specifically, the dual-layer search and navigation module includes: a room point evaluation submodule, which calculates and normalizes the traversable distance from the agent's current position to each room-level semantic prior point, calculates a room point score by combining distance factors and fused value, and selects the room point with the highest score as the navigation target for upper-layer guidance; a frontier point evaluation submodule, which calculates the cumulative fused value of the fused value map in the neighborhood area centered on the frontier point as the frontier point score, and selects the frontier point with the highest score as the navigation target for lower-layer exploration; and a mode switching submodule, which controls the cyclic switching between guidance mode and exploration mode. When the task starts, it enters guidance mode, switches to exploration mode after reaching the target room through initialization mode, returns to guidance mode to select the next room point after exploration is completed, and switches to target navigation mode with the highest priority when a target object is detected.

[0055] The path planning and execution module is used to generate an action sequence based on a determined navigation target point and a point-target navigation strategy, thereby controlling the agent to move.

[0056] The various modules of the system communicate with each other through a data bus or message mechanism to collaboratively complete the entire process from prior loading, real-time perception, value fusion, navigation decision-making to path execution.

[0057] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the zero-sample embodied target navigation method based on prior semantic maps described in the above method embodiments.

[0058] The computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical disk, hard disk, flash memory, etc. The computer program may be written using any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, C++, Python, etc.) and procedural programming languages ​​(such as C language, etc.).

[0059] Specifically, when the computer program is executed by the processor, it can perform the following steps: using pre-acquired scene prior data, inferring the probability of the existence of the target object in different room types through a large language model, and constructing a prior value map that assigns semantic value to the grid area; during navigation, based on the current observed image, using a visual language model to calculate the similarity score between the current observed image and the semantic description of the target object, and combining the field of view cone confidence to generate an observation value map; dynamically fusing the prior value map and the observation value map, with the prior value weight adaptively adjusted as the observation information accumulates, to obtain a fused value map; based on the fused value map, evaluating room-level semantic prior points and frontier points, and performing a two-layer search navigation that alternates between upper-level room guidance and lower-level frontier exploration; when a target object is detected through an open-vocabulary target detection model, switching to target navigation with the target object's location as the navigation target.

[0060] This storage medium can be mounted on the vehicle-mounted computing platform of service robots, indoor mobile robots, and automatic inspection systems, or it can be deployed on a cloud server to interact with robot terminals via a network.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A zero-sample embodied target navigation method based on prior semantic maps, characterized in that, include: By using pre-acquired scene prior data, the probability of the existence of target objects in different room types is inferred through a large language model, and a prior value map that assigns semantic value to grid areas is constructed. During navigation, based on the current observed image, the similarity score between the current observed image and the semantic description of the target object is calculated using a visual language model, and an observation value map is generated by combining the field of view cone confidence score. The prior value map and the observation value map are dynamically fused together, and the fusion weight is adaptively adjusted as the observation information accumulates to obtain the fused value map. Based on the fused value map, room-level semantic prior points and frontier points are evaluated, and a two-layer search navigation is performed, alternating between upper-level room guidance and lower-level frontier exploration. When a target object is detected using the open vocabulary target detection model, switch to target navigation with the target object's location as the navigation target.

2. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, The large language model includes mutually independent first language models and second language models; The first language model is used to infer the room type based on the categories of objects in the room; The second language model is used to infer the probability of the existence of the target object in each room based on the semantic description of the target object and the room type.

3. The zero-sample embodied target navigation method based on prior semantic maps according to claim 2, characterized in that, The construction of the prior value map also includes: Generate a top-down view of scene accessibility, project objects from the scene prior data onto the top-down view, and assign a room ID to each accessible pixel in the top-down view through multi-source breadth-first search to obtain room area division; The centroid of each room region is calculated as the room-level semantic prior point. If the centroid is in an impassable area, it is projected to the nearest passable pixel through breadth-first search.

4. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, The field of view cone confidence level Based on the angle between the current pixel and the camera's optical axis and camera horizontal field of view The calculation formula is as follows: .

5. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, The generated observation value map includes: Calculate the similarity score between the current RGB image and the semantic description of the target object using a visual language model; The product of the similarity score and the field-of-view cone confidence score is projected into the map space as the new value generated by the current observation; For pixels in regions that overlap with historical observations, a weighted update is performed as follows: Value channel updated to ; Confidence channel updated to ;in, The pixel at time t-1 Updated confidence level The pixel at time t-1 The updated observational value, Pixel at time t The confidence level obtained from the new observation Pixel at time t The observational value obtained from the new observations.

6. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, The prior value weights of the dynamic fusion The value is 1 minus the maximum value in the current observation value map, and the fusion formula is: , For pixels The a priori value of a place. For pixels The observational value at time t.

7. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, Evaluate room-level semantic priors, including: For each room-level semantic prior point, calculate the traversable distance from the agent's current position to that room-level semantic prior point. and normalized to ,in Distance threshold; Room-level semantic prior review is divided into ,in This represents the fusion value at the room-level semantic prior point. These are the weighting coefficients; The room-level semantic prior point with the highest score is selected as the navigation target for upper-level room guidance.

8. The zero-sample embodied target navigation method based on prior semantic maps according to claim 1, characterized in that, Assessment frontiers include: The sum of the fusion values ​​of the fusion value maps within the neighborhood area centered on the frontier point is calculated as the frontier point score; The highest-scoring frontier point is selected as the navigation target for exploration of the lower frontier.

9. A zero-sample embodied target navigation system based on a priori semantic map, characterized in that, include: The prior semantic map construction module is used to infer the probability of the existence of a target object in different room types by using scene prior data and a large language model, and to construct a prior value map. The observation value map construction module is used to calculate the similarity score between the current observation image and the semantic description of the target object using a visual language model during navigation, and to generate an observation value map by combining the field of view cone confidence score. The value fusion module is used to dynamically fuse the prior value map and the observed value map. The fusion weight is adaptively adjusted as the observed information accumulates to obtain the fused value map. The dual-layer search and navigation module is used to evaluate room-level semantic prior points and frontier points based on the fused value map, perform dual-layer search and navigation that alternates between upper-layer room guidance and lower-layer frontier exploration, and switch to target navigation with the target object's location as the navigation target when a target object is detected by the open vocabulary target detection model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements a zero-sample embodied target navigation method based on a priori semantic map as described in any one of claims 1 to 8.

Images