Zero sample target navigation method and system based on functional Bayesian network

Abstract

The invention discloses a zero sample target navigation method based on a functional Bayesian network, and the method comprises the following steps: at each time step, an agent projects an RGB image observed in real time into a probability semantic graph of an open vocabulary taking itself as a center, and carries out the weighted fusion of the probability semantic graph and a global map; based on the probability semantic map, a function Bayesian network with the function as the core is constructed in real time, and the function Bayesian network is a dynamic directed acyclic graph; and performing reasoning and anti-fact reasoning by using a large language model on the functional Bayesian network to finally obtain a probability map used for navigation, and realizing zero-cost target navigation for the intelligent agent under the guidance of the probability map. The navigation success rate and the path efficiency of the method are superior to those of an existing zero-sample target navigation method.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robotics in artificial intelligence, and particularly to a zero-shot target navigation method and system. Background Technology

[0002] Visual target navigation is an important research direction in the field of computer vision within artificial intelligence. This task aims to enable an intelligent agent to autonomously navigate to a user-specified target object in an unknown environment without a global map. In this task, the agent starts from a random starting point and acquires RGB observation image information, depth observation image information, and its own pose information in real time. Based on its visual observation of the environment, its own state, and its understanding of the environment, the agent autonomously plans and takes actions such as moving forward, turning left, turning right, and stopping. Ultimately, the agent should autonomously stop within a finite number of steps; a task is considered successful if the distance to the target is less than a specified threshold and the target is within the agent's field of vision when stopping.

[0003] In recent years, zero-shot target navigation methods have become a research hotspot. Compared with conventional target navigation methods, zero-shot target navigation methods have greater flexibility and generalization ability. Conventional target navigation methods are often limited to a closed set of target categories, requiring retraining of the model for new target categories, making it difficult to adapt to real-world scenarios with diverse and dynamically changing target types. In contrast, zero-shot target navigation methods allow for free specification of target categories, eliminating the need for additional training or fine-tuning, and can directly handle any target in an open vocabulary, greatly expanding their applicability in practical applications.

[0004] Since the environment is unknown and the target is initially invisible, the agent needs to learn knowledge about the scene and objects to infer the target's likely location. Previous target navigation methods construct this knowledge by learning scene representations, potential functions, waypoint predictions, and unexplored area predictions. These methods can be categorized into end-to-end methods relying on reinforcement learning (RL) or imitation learning (IL) and modular methods that localize via geometric semantic maps. Zero-shot target navigation methods, on the other hand, use visual-language models (VLMs) and large language models (LLMs) as training-free knowledge to infer the target's potential location. VLM-based methods compute a relevance score between visual observation features and the target semantics encoded by the VLM, thus transforming the navigation task into a semantic-visual fundamental problem. LLM-based methods convert the 3D environment into a textual representation, such as observed object categories, potential trajectory descriptions, and textual scene maps. This text is then used to cue the LLM, which determines the most likely location of the target.

[0005] Previously, visual navigation algorithms that required training with non-zero samples needed a large amount of training data to learn the features of the environment and targets, which limited their ability to generalize in new environments. These methods typically could only handle target categories seen during the training phase and could not handle new categories in the open vocabulary. At the same time, these methods mainly relied on the similarity of visual features or general knowledge obtained from language models, lacking a deep understanding of the target's function and the semantic relationship between the scene, which may lead to inefficient navigation in complex environments.

[0006] Existing zero-shot visual object navigation algorithms primarily utilize visual language models or large language models as training-free knowledge to infer the potential location of targets. For methods using visual language models, they rely on observing visual information semantically similar to the target; when such semantically relevant visual cues are lacking, inferring the correct exploration direction becomes difficult. For methods using large language models, representing the 3D environment in textual form inevitably leads to information loss (such as geometric location, context, and perceptual confidence). Furthermore, the knowledge gained from large language models is general and environment-independent, while the contextual relationships between objects are environment-dependent and vary with their functional roles. For example, the probability of finding a plate near a chair depends on the chair's function in the environment: the probability of finding a plate near a chair is high when it is used for dining, and low when it is used for resting.

[0007] In summary, given the current limitations of the VLM method in zero-shot navigation and the loss of environmental information in the LLM method, there is an urgent need to research a novel functional Bayesian network-based approach and apply it to visual target navigation tasks. Summary of the Invention

[0008] To address the shortcomings of existing zero-shot target navigation methods, such as insufficient accuracy in inferring potential target locations in open-vocabulary and dynamically changing environments due to the lack of a function-centric causal reasoning mechanism, and the low navigation efficiency and poor generalization in complex and unconventional layouts caused by the reliance on superficial semantics or general knowledge by traditional visual-language models or large language models, which struggle to integrate environment-specific causal dependencies between scene, function, and object in real time, this application proposes a visual target navigation method and system based on functional Bayesian networks.

[0009] In a first aspect, embodiments of this application provide a zero-shot target navigation method based on functional Bayesian networks, applied to an artificial intelligence-based intelligent agent, the method comprising:

[0010] Probabilistic semantic map construction steps: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map;

[0011] Steps for real-time construction of functional Bayesian networks: Based on a probabilistic semantic map, construct functional Bayesian networks with functions at their core in real time. The functional Bayesian network is a dynamic directed acyclic graph. ,in, Represents variables, This represents a directed edge representing the probability between variables. Variable V includes: scene, object, and functional group. The directed edge representing the probability is determined using a counterfactual reasoning thought chain.

[0012] The steps for target navigation based on functional Bayesian networks are as follows: a large language model is used on the functional Bayesian network for reasoning and counterfactual reasoning to obtain a probability map for navigation. Under the guidance of the probability map, the agent achieves zero-cost target navigation.

[0013] In a specific embodiment of the present invention, the above-mentioned probabilistic semantic map construction step includes:

[0014] A set of indoor object categories and scene categories are preset, and the object categories are updated according to the preset target;

[0015] At each time step, the agent receives RGB visual observation images, detects objects belonging to the COCO category for RGB images, and uses an open-vocabulary object detector to identify scene categories and other object categories, and obtains segmentation semantic masks using bounding box cues.

[0016] Based on the obtained depth image, semantic mask and its corresponding semantic label and confidence score, each pixel and its semantic label are converted into a point cloud. Points falling within the height range of the agent are classified as obstacles. The point cloud is then projected onto the top-down map space through geometric operations to generate a map centered on the agent. The semantic channel of the egocentric map records each pixel as a probability value.

[0017] Based on the agent's coordinates, the egocentric map is aggregated with the existing global map to generate a probabilistic semantic map.

[0018] In a specific embodiment of the present invention, the above-mentioned step of real-time construction of functional Bayesian networks includes:

[0019] By introducing functional groups to model the relationship between objects and scenes, the directed acyclic graph structure of the functional Bayesian network adopts object nodes pointing to functional nodes, or scene nodes pointing to functional nodes; based on the unidirectionality of the directed acyclic graph, bidirectional edges cannot be directly established between objects and scenes.

[0020] The reasoning in the counterfactual reasoning chain includes: identifying variables; determining the type of edge between variables, such as an object-to-function or a scene-to-function edge; assuming the edge exists and defining the factual and counterfactual situations in symbolic form; inferring the probabilities of the factual and counterfactual situations; comparing the probabilities of the factual and counterfactual situations and deriving estimates; and finally forming the edge between variables by evaluating whether the edge exists and the magnitude of the probability value.

[0021] In a specific embodiment of the present invention, the above-mentioned step of real-time construction of functional Bayesian networks further includes:

[0022] At each navigation time step, for an object observed in the probabilistic semantic map, it is determined whether it is a newly observed object. If the projection of an object does not overlap with the corresponding category channel of the object in the previous map, the object is considered a newly observed object.

[0023] If an object has been previously observed, update the object's probability in the previous map;

[0024] For each newly observed object, register it as a new object node; determine the edges between the new node and other nodes in the functional Bayesian network.

[0025] In a specific embodiment of the present invention, the step of determining the edges between a new node and other nodes in the functional Bayesian network in the above-mentioned real-time construction of the functional Bayesian network further includes:

[0026] For an edge from an object to a function, identify each existing function group node within the reachable distance specified by the new node; for each function group node within the reachable distance threshold, use counterfactual reasoning thought chain reasoning to determine whether the newly observed object establishes a directed edge with an existing function node and estimate the confidence probability, and update it in the functional Bayesian network.

[0027] For a scene-to-function edge, given a newly registered function group node, the scene associated with the newly registered function group node is determined by querying the corresponding scene channel at the matching coordinates in the probabilistic semantic map. The counterfactual reasoning chain is used to evaluate whether the scene establishes a directed edge with the existing function group node and to estimate the confidence probability, and the function Bayesian network is updated. Each object node is connected to a maximum of a preset number of function group nodes.

[0028] In a specific embodiment of the present invention, the above-mentioned target navigation steps based on functional Bayesian networks include:

[0029] The confidence scores of scenes / objects in the probabilistic semantic map are used as prior inputs to a functional Bayesian network. The posterior probabilities inferred by the functional Bayesian network are used to update the target occurrence probabilities of each pixel in the map, forming a continuously optimized closed loop to generate a probabilistic map.

[0030] In a specific embodiment of the present invention, the above-mentioned target navigation steps based on functional Bayesian networks further include:

[0031] The functional Bayesian network is divided into multiple sub-directed acyclic graphs centered on functional group nodes; for each functional group of the central node, a subgraph containing all connected parent nodes is extracted, preserving the relational context; where parent nodes include object nodes and scene nodes;

[0032] Based on Bayesian inference, calculate the posterior probability of each subgraph containing the target object;

[0033] For each functional group node, construct a bounding box centered on the coordinates of the functional group node, encompassing the positions of connected objects;

[0034] A probability distribution is formed on the map; the probability maps of all functional group nodes are aggregated to form a probability map of the target's appearance.

[0035] Secondly, embodiments of this application provide a zero-shot target navigation system based on functional Bayesian networks. Employing the aforementioned zero-shot target navigation method based on functional Bayesian networks, the system is applied to an artificial intelligence-based intelligent agent. The system includes:

[0036] Probabilistic Semantic Map Construction Module: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map;

[0037] Real-time construction of functional Bayesian network module: Based on probabilistic semantic map, a functional Bayesian network centered on functions is constructed in real time. The functional Bayesian network is a dynamic directed acyclic graph. ,in, Represents variables, This represents a directed edge representing the probability between variables. Variable V includes: scene, object, and functional group. The directed edge representing the probability is determined using a counterfactual reasoning thought chain.

[0038] The target navigation module based on functional Bayesian networks uses a large language model for reasoning and counterfactual reasoning on a functional Bayesian network to obtain a probability map for navigation. Guided by the probability map, the agent achieves zero-cost target navigation.

[0039] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a zero-shot target navigation method based on a functional Bayesian network.

[0040] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the zero-sample target navigation method based on functional Bayesian networks as described above.

[0041] Compared with existing technologies, it has the following outstanding advantages:

[0042] 1) This invention proposes a zero-shot target navigation method based on functional reasoning. This method constructs a functional Bayesian network in real-time during navigation, explicitly modeling the scene, object, and function groups as a dynamic directed acyclic graph. It utilizes a large language model for counterfactual causal reasoning, estimating the conditional probabilities of "object → function" and "scene → function" online, thereby transforming abstract functional probabilities into a navigable spatial heatmap. Its technical advantage lies in the fact that even if the target category never appeared in the training, it can accurately infer the region where the target is most likely to appear using functional intermediate variables, significantly improving the navigation success rate under open vocabulary.

[0043] 2) The method and system of this invention propose Fourier causal inference. This method, through the design of a "counterfactual thought chain" prompt template, allows the large language model to perform do-calculus-style causal comparison at the symbolic level. Specifically, it compares the changes in functional probabilities under two scenarios: "edge exists" and "edge does not exist," quickly providing causal strength and confidence, and outputting interpretable conditional probability values. Its technical advantage lies in the fact that, without retraining or fine-tuning the model, online registration of causal edges between a new observed object and a functional group can be completed with only 3–8 LLM calls, achieving linear complexity response and ensuring real-time navigation.

[0044] 3) The method and system of this invention propose a bidirectional real-time coupled update mechanism of probabilistic semantic map and functional Bayesian network. At each time step, this mechanism projects newly observed RGB-D evidence into an egocentric probabilistic semantic map and weightedly fuses it with the global map. Simultaneously, it uses the confidence level of scenes / objects in the map as a priori input to the functional Bayesian network, and in turn, uses the functional posterior probability inferred by the functional Bayesian network to update the "target occurrence probability" of each pixel in the map, forming a rolling optimization closed loop. Its technical effect is that it ensures that geometric-semantic information does not decay over time, and that abstract functional probabilities are "solidified" in real time into a two-dimensional navigable space, realizing a navigation link of "seeing where you want to see where you want to see where you want to see where you want to see where you want to see where you want to see." Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a schematic diagram of the zero-shot target navigation method based on functional Bayesian networks according to the present invention;

[0047] Figure 2 This is a schematic diagram of the overall architecture of the target navigation method according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the probabilistic semantic map of open vocabulary in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of a functional Bayesian network according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of target navigation based on functional Bayesian networks according to an embodiment of the present invention;

[0051] Figure 6 This is a diagram of the zero-shot target navigation system based on functional Bayesian networks of this invention;

[0052] Figure 7 This is a schematic diagram of the computer hardware of the present invention. Detailed Implementation

[0053] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0054] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0055] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0056] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0057] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0058] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0059] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0060] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0061] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0062] In its research on zero-shot visual target navigation tasks, this invention addresses the shortcomings of the reasoning ability of the VLM method and the loss of environmental information in the LLM method. It designs a functional Bayesian network and applies it to visual target navigation tasks.

[0063] Compared to the apparent features of objects, their functions are a more crucial basis for human target localization. The invention team discovered that using functional groups as intermediate variables can accurately correlate scene semantics with object distribution. For example, when "gas stove" is detected, counterfactual reasoning quantifies its contribution probability to "cooking function" (if the stove is removed, the probability of cooking in the kitchen drops from 0.9 to 0.2). This mechanism solves the problem of integrating environment-specific functions with general knowledge of LLM. However, realizing this concept faces three major technical challenges: first, how to establish a dynamic probabilistic network to express the time-varying relationship between function, scene, and object; second, how to achieve real-time registration of open-vocabulary objects through lightweight computation; and third, how to transform abstract functional probabilities into a navigable spatial heatmap. These difficulties were ultimately overcome through the innovative design of functional Bayesian networks.

[0064] The system of this application embodiment will be described in detail below with reference to specific embodiments:

[0065] Example 1

[0066] like Figure 1 and Figure 2 As shown, this invention proposes a novel visual target navigation method using a function-based Bayesian network. The design is based on a modular target navigation framework. First, a probabilistic semantic map of an open vocabulary is generated during the navigation process. A function-centric Bayesian network is extracted from this map. A large language model is then used on this Bayesian network for reasoning and counterfactual reasoning, ultimately yielding a probabilistic map of the value used in navigation. The overall navigation architecture is then guided by this probabilistic map, as follows: Figure 2 As shown. It should be clarified that the specific implementation method described is only for illustrating the technical solution of the present invention and is not intended to limit the present invention. The specific implementation process is as follows:

[0067] like Figure 1 As shown, this invention proposes a zero-shot target navigation method based on functional Bayesian networks, applied to an artificial intelligence-based intelligent agent. The method includes:

[0068] Step 101 of constructing the probabilistic semantic map: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map.

[0069] Step 102 of real-time construction of functional Bayesian networks: Based on the probabilistic semantic map, construct a functional Bayesian network with functions as the core in real time. The functional Bayesian network is a dynamic directed acyclic graph. ,in, Represents variables, This represents a directed edge representing the probability between variables. Variable V includes: scene, object, and functional group. The directed edge representing the probability is determined using a counterfactual reasoning thought chain.

[0070] Step 103 of target navigation based on functional Bayesian network: Use a large language model on the functional Bayesian network to perform reasoning and counterfactual reasoning, and finally obtain the probability map used for navigation. Under the guidance of the probability map, the agent achieves zero-cost target navigation.

[0071] In a specific embodiment of the present invention, the above-mentioned probabilistic semantic map construction step 101 includes:

[0072] A set of indoor object categories and scene categories are preset, and the object categories are updated according to the preset target;

[0073] At each time step, the agent receives RGB visual observation images, detects objects belonging to the COCO category for RGB images, and uses an open-vocabulary object detector to identify scene categories and other object categories, and obtains segmentation semantic masks using bounding box cues.

[0074] Based on the obtained depth image, semantic mask and its corresponding semantic label and confidence score, each pixel and its semantic label are converted into a point cloud. Points falling within the height range of the agent are classified as obstacles. The point cloud is then projected onto the top-down map space through geometric operations to generate a map centered on the agent. The semantic channel of the egocentric map records each pixel as a probability value.

[0075] Based on the agent's coordinates, the egocentric map is aggregated with the existing global map to generate a probabilistic semantic map.

[0076] Specifically, in a specific embodiment of the present invention, the above-mentioned probabilistic semantic map construction step 101 for open vocabulary includes:

[0077] The overall process is as follows Figure 3As shown, the functional Bayesian network of this invention is constructed based on observed object and scene instances. To this end, this invention maintains an open-vocabulary probabilistic semantic map, which continuously records the geometry and probabilities of observed objects and scenes throughout the navigation process. Formally, at the beginning of each navigation round, this invention considers a set of common indoor object categories. and scene categories It's important to note that these categories can be freely specified. Given a target... Since the target is an open vocabulary, the set of object categories will be updated to... At each time step The agent receives RGB-D visual observations. For RGB images, this invention utilizes YOLOv7 to detect objects belonging to the COCO category and uses Grounding-DINO to identify scene categories and other object categories. Grounding-DINO is an open-vocabulary object detector that achieves arbitrary object detection by aligning image regions with phrases in text prompts and outputs the alignment score between the region and the text as a confidence score. Subsequently, this invention utilizes bounding box prompts Mobile SAM to obtain a more accurate segmentation mask.

[0078] Based on the obtained depth image, semantic mask, and their corresponding semantic labels and confidence scores, each pixel and its semantic label are converted into a point cloud. Points falling within the agent's height range are classified as obstacles. Then, geometric operations are used to project the point cloud onto a top-down map space to generate an egocentric map. The semantic channel of an egocentric map records each pixel as a probability value (i.e., a confidence score).

[0079] Based on the agent's coordinates, this egocentric map is aggregated with the existing global map, i.e. It's worth noting that if a pixel has been previously observed (i.e., has a non-zero score), its value is updated using a weighted average based on the number of times it has been observed. This ensures that the confidence score is continuously updated across observations from different perspectives. In summary, probabilistic semantic maps... In the structure, Represents a set The number of object categories in the data. Represents a set The number of scene categories is represented by H, the length of the map is H, the width of the map is W, and R is a real number, indicating that the semantic probability map is a numerical structure. The other two channels correspond to obstacles and explored areas, respectively.

[0080] In a specific embodiment of the present invention, the above-mentioned step 102 of real-time construction of functional Bayesian network includes:

[0081] By introducing functional groups to model the relationship between objects and scenes, the directed acyclic graph structure of the functional Bayesian network adopts object nodes pointing to functional nodes, or scene nodes pointing to functional nodes; based on the unidirectionality of the directed acyclic graph, bidirectional edges cannot be directly established between objects and scenes.

[0082] The reasoning in the counterfactual reasoning chain includes: identifying variables; determining the type of edge between variables, such as an object-to-function or a scene-to-function edge; assuming the edge exists and defining the factual and counterfactual situations in symbolic form; inferring the probabilities of the factual and counterfactual situations; comparing the probabilities of the factual and counterfactual situations and deriving estimates; and finally forming the edge between variables by evaluating whether the edge exists and the magnitude of the probability value.

[0083] In a specific embodiment of the present invention, the above-mentioned step 102 of real-time construction of functional Bayesian network further includes:

[0084] At each navigation time step, for an object observed in the probabilistic semantic map, it is determined whether it is a newly observed object. If the projection of an object does not overlap with the corresponding category channel of the object in the previous map, the object is considered a newly observed object.

[0085] If an object has been previously observed, update the object's probability in the previous map;

[0086] For each newly observed object, register it as a new object node; determine the edges between the new node and other nodes in the functional Bayesian network.

[0087] In a specific embodiment of the present invention, the step of determining the edges between a new node and other nodes in the functional Bayesian network in the above-mentioned real-time construction of the functional Bayesian network further includes:

[0088] For an edge from an object to a function, identify each existing function group node within the reachable distance specified by the new node; for each function group node within the reachable distance threshold, use counterfactual reasoning thought chain reasoning to determine whether the newly observed object establishes a directed edge with an existing function node and estimate the confidence probability, and update it in the functional Bayesian network.

[0089] For a scene-to-function edge, given a newly registered function group node, the scene associated with the newly registered function group node is determined by querying the corresponding scene channel at the matching coordinates in the probabilistic semantic map. The counterfactual reasoning chain is used to evaluate whether the scene establishes a directed edge with the existing function group node and to estimate the confidence probability, and the function Bayesian network is updated. Each object node is connected to a maximum of a preset number of function group nodes.

[0090] Specifically, in a specific embodiment of the present invention, step 102 of constructing a functional Bayesian network specifically includes:

[0091] like Figure 4 As shown, the functional Bayesian network of this invention is defined as a directed acyclic graph. ,in Represents variables, This represents a directed edge representing the probability between these variables. Regarding the variables... The definition includes three types: scene, object, and function group, denoted as .here, This represents a scene instance whose category is contained in the scene category set, i.e. its scope Recorded in the probabilistic semantic map. Represents an object instance, where Specify its category, Indicate its coordinates. This represents a function group instance. Its category belongs to a predefined set of common functions, i.e. .gather This includes common functions that support daily human activities, such as rest, entertainment, and cooking. (Regarding the side...) By definition, this invention considers the following two types of edges: 1) Edges from objects to functions : This edge represents the function that an object can provide for human activities, such as a bed providing a sleeping function. 2) Edge from scene to function This edge indicates that the semantic context of a scene can be decomposed into functional groups. For example, a kitchen can be decomposed into cooking, cleaning, and storage functions. Introducing functional groups helps to more rationally model the relationships between objects and the scene.

[0092] 1) The prior scene knowledge constructed for target navigation assumes a bidirectional relationship between scenes and objects, meaning that objects tend to appear in relevant scenes, and scenes may contain relevant objects. However, in real-world scenes, unusual layouts often occur, such as objects appearing in semantically unexpected scenes (e.g., a yoga mat in a bedroom), or scenes lacking typical objects (e.g., a living room without a coffee table). These situations violate the above assumptions. However, such scenes can be represented when functions are incorporated into a directed acyclic graph (DAG). Specifically, the DAG structure allows object nodes to point to function nodes (where the function does not belong to the current scene), or scene nodes to point to function nodes (where a function of the current scene is not provided by any object within it).

[0093] 2) Due to the unidirectional nature of directed acyclic graphs (DAGs), bidirectional edges cannot be directly established between objects and scenes. By introducing functional groups, scene and object nodes naturally form a V-shaped structure: object → function ← scene. With this intermediate functional node, objects and scenes appear to be interconnected, which aligns with the intuitive concept of a bidirectional relationship between scenes and objects. Therefore, the introduction of functions not only conforms to the intuitive understanding of the relationship between scene objects but also allows for the structured representation of uncommon layouts.

[0094] Counterfactual Reasoning Thinking Chain. Large Language Models (LLMs) possess powerful reasoning capabilities and prior knowledge, which this invention leverages to determine edges in Functional Bayesian Networks. To derive plausible causal reasoning from LLMs, this invention proposes a thinking chain prompting strategy: the counterfactual reasoning thinking chain, which prompts LLMs to perform counterfactual reasoning according to predefined formal rules. The main steps of the counterfactual reasoning thinking chain are as follows: 1) Identify variables; 2) Determine the type of edge (i.e., object-to-function or scene-to-function edge); 3) Assume the edge exists and formulate factual and counterfactual situations in symbolic form; 4) Infer the probabilities of factual and counterfactual situations (e.g., assuming...). Indicates sleep function, Indicates bed, Represents the probability of other adjacent objects. The question asks, "If a bed exists, what is the probability that the sleep function will exist?", while the counterfactual probability... (5) Compare the probabilities of factual and counterfactual scenarios and derive estimates; (6) Evaluate the existence and probability of edges.

[0095] In summary, given the variables in a functional Bayesian network and another variable and including Subgraph of directly connected and related nodes and edges The counterfactual reasoning chain, by prompting LLM to compare two contrasting scenarios (scenarios with and without edges), determines whether an object or scenario node is a causal factor for a given function and infers the corresponding confidence probability. The process of the counterfactual reasoning chain can be described as follows:

[0096]

[0097] Real-time construction of functional Bayesian networks. The functional Bayesian network of this invention is constructed in real-time during navigation, inferring relationships between scenes, objects, and functional groups in the navigation environment based on a continuously accumulating probabilistic semantic map. At each navigation time step... For self-centered maps Upon observing objects in a map, the present invention first determines whether they are newly observed objects. If an object's projection is similar to its projection on a previous map... If the corresponding category channels do not have partial geometric overlap, then the object is considered a newly observed object. If an object has been previously observed, the present invention only needs to update its category channel. The probability in the middle. For each newly observed object. It is registered as a new object node, with its group coordinates and category. Then, the invention determines... Edges between objects and other nodes in the Bayesian network. Edges from objects to functions. The present invention identifies in Specified reachable distance Each existing functional group node within (e.g., 1.5 meters) The reachable distance is calculated based on obstacles and pathways to explored areas recorded in the probabilistic semantic map. The coordinates of a functional group node are defined as the average position of all its associated object nodes. For each functional node within a distance threshold, this invention utilizes... To determine whether a newly observed object establishes a directed edge with an existing functional node, where, Represents a containing The counterfactual thought chain is a local functional subgraph of a node and its directly connected parent nodes (including object nodes and scene nodes), indicating whether it shares a function with surrounding objects. If edges exist, the counterfactual thought chain also estimates the confidence probability. If this relationship is confirmed, then from arrive Edges and their probabilities Add to the Bayesian network, and The coordinates are updated to the average coordinates of all causal object nodes. Furthermore, this invention uses... and The LLM is suggested to explore additional features and probabilities. Once identified, these features are registered as new nodes. Its coordinates are inherited from Furthermore, the corresponding edges and conditional probabilities are updated in the Bayesian network. To avoid excessive redundancy in the Bayesian network structure leading to inference delays, each object is limited to connecting a maximum of five functional group nodes.

[0098] For the edge from scene to function Given a newly registered functional node This invention uses query The scene is determined by matching the corresponding scene channel at the coordinates. As with the previous steps, this invention uses... The existence of this relationship is evaluated, and the functional Bayesian network is updated accordingly. Represents the current scene node The subgraph centered on this. Furthermore, the invention will further utilize... and The LLM is used to infer other potential functions of the scene. These inferred functions are also incorporated into the functional Bayesian network, while limiting each scene node to a maximum of five different function categories.

[0099] The inference cost of the functional Bayesian network in this invention primarily stems from querying the LLM to establish edges for newly observed object nodes. To manage this complexity, this invention employs two constraints: 1) object nodes are only connected to functional nodes within a distance threshold; 2) each object node is connected to a maximum of five functional nodes. Statistically, the number of LLM calls required per object node is between 3 and 8, ensuring linear complexity.

[0100] In a specific embodiment of the present invention, the target navigation step 103 based on a functional Bayesian network includes:

[0101] The confidence scores of scenes / objects in the probabilistic semantic map are used as prior inputs to a functional Bayesian network. The posterior probabilities inferred by the functional Bayesian network are used to update the target occurrence probabilities of each pixel in the map, forming a continuously optimized closed loop to generate a probabilistic map.

[0102] In a specific embodiment of the present invention, the target navigation step 103 based on a functional Bayesian network further includes:

[0103] The functional Bayesian network is divided into multiple sub-directed acyclic graphs centered on functional group nodes; for each functional group of the central node, a subgraph containing all connected parent nodes is extracted, preserving the relational context; where parent nodes include object nodes and scene nodes;

[0104] Based on Bayesian inference, calculate the posterior probability of each subgraph containing the target object;

[0105] For each functional group node, construct a bounding box centered on the coordinates of the functional group node, encompassing the positions of connected objects;

[0106] A probability distribution is formed on the map; the probability maps of all functional group nodes are aggregated to form a probability map of the target's appearance.

[0107] Specifically, in a specific embodiment of the present invention, the target navigation step 103 based on a functional Bayesian network specifically includes:

[0108] like Figure 5 As shown, at each navigation time step Given a target The location can be inferred based on a dynamically updated functional Bayesian network. Formally, the functional Bayesian network is divided into several sub-directed acyclic graphs centered around functional group nodes. For each central node... This invention extracts a subgraph containing all connected parent nodes (including object and scene nodes). This preserves the relational context. Based on Bayesian inference, each containing the target object... Posterior probability of a subgraph It can be represented as follows:

[0109]

[0110] Where prior probability Represents the target object Provide functions The probability. This probability is obtained by using... and all The prompt indicates that LLMs are used to determine this, ensuring... Each functional group probability of occurrence It can be obtained from the structure and conditional probabilities recorded in the functional Bayesian network:

[0111]

[0112] in and These represent the observed scenes. and objects The probabilities are recorded in a probabilistic semantic map. This is the probability normalization factor. Based on the above formula, the target appears in the functional group. The posterior probability formula is:

[0113]

[0114] in and Recorded in the probabilistic semantic map, it represents the actual observation results, while and Recorded in the functional Bayesian network, representing prior knowledge. Subsequently, for each functional group... The present invention is constructed based on The bounding box centered at the coordinates of the point encompasses the positions of connected objects. The central probability is defined as... As a result, the probability value gradually decreases from the center to the edge, thus forming a probability distribution on the 2D map. Then, aggregate all The probability map forms the probability map of the target's appearance. .

[0115] During navigation, select The point with the highest probability is selected as the navigation waypoint. The path from the current position to the waypoint is calculated using the FMM algorithm, and then the agent is driven to move towards the waypoint. The waypoint is updated at every moment of navigation until the agent finds the target.

[0116] As described above, the method of the present invention can be implemented well.

[0117] Compared with the prior art, the present invention has the following outstanding advantages and beneficial effects:

[0118] Compared to existing technologies, this invention constructs a functional Bayesian network and combines it with CounterfactCoT to accurately model the causal relationships between scenes, objects, and functional groups. Furthermore, relying on the bidirectional real-time coupling update mechanism of the probabilistic semantic map and the functional Bayesian network, it solves the navigation failure problem caused by the lack of functional reasoning in traditional visual-language models (VLMs) when clues are missing. It also avoids the information loss and environment irrelevance defects brought about by large language models (LLMs) in textualized 3D environments. Under the premise of no vocabulary target requiring training and fine-tuning and support for open vocabulary targets, its navigation success rate (SR) and path efficiency (SPL) are superior to existing zero-shot target navigation methods on MP3D and HM3D datasets. Its navigation success rate even surpasses some training-dependent closed-set methods. Simultaneously, it maintains stable performance in non-standard open vocabulary target tests, significantly improving the generalization and accuracy of zero-shot target navigation in complex dynamic environments.

[0119] Example 2

[0120] like Figure 6 As shown, this application provides a zero-shot target navigation system based on functional Bayesian networks. It employs the zero-shot target navigation method based on functional Bayesian networks described above and is applied to an artificial intelligence-based intelligent agent. The system includes:

[0121] Probabilistic Semantic Map Construction Module 201: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map.

[0122] Module 202 for Real-Time Construction of Functional Bayesian Networks: Based on probabilistic semantic maps, this module constructs functional Bayesian networks with functions at their core in real time. The functional Bayesian network is a dynamic directed acyclic graph. ,in, Represents variables, This represents a directed edge representing the probability between variables. Variable V includes: scene, object, and functional group. The directed edge representing the probability is determined using a counterfactual reasoning thought chain.

[0123] The target navigation module 203 based on functional Bayesian networks uses a large language model to perform reasoning and counterfactual reasoning on the functional Bayesian network, and finally obtains a probability map for navigation. Under the guidance of the probability map, the agent achieves zero-cost target navigation.

[0124] Example 3

[0125] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a zero-shot target navigation method based on a functional Bayesian network.

[0126] Example 4

[0127] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the zero-sample target navigation method based on functional Bayesian networks as described above.

[0128] In addition, combined Figure 1 The zero-shot target navigation method based on functional Bayesian networks described in this application can be implemented by electronic devices, such as computer devices. Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0129] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. For example, Figure 7 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0130] Specifically, the processor 81 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0131] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0132] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the zero-shot target navigation methods based on functional Bayesian networks in the above embodiments.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A zero-shot target navigation method based on functional Bayesian networks, characterized in that, Applied to artificial intelligence-based intelligent agents, the method includes: Probabilistic semantic map construction steps: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map; A function-based Bayesian network is constructed in real time based on the probabilistic semantic map, and the function-based Bayesian network is a dynamic directed acyclic graph wherein, denotes a variable, denotes a probabilistic directed edge between the variables, and the variable V includes a scene, an object, and a function group, and the probabilistic directed edge is determined by using a counterfactual reasoning thought chain; The target navigation steps based on functional Bayesian networks are as follows: a large language model is used on the functional Bayesian network for reasoning and counterfactual reasoning to obtain a probability map for navigation. Under the guidance of the probability map, the agent achieves zero-cost target navigation.

2. The zero-shot target navigation based on functional Bayesian networks of claim 1, wherein, The probabilistic semantic map construction steps include: A set of indoor object categories and scene categories are preset, and the object categories are updated according to the preset target; At each time step, the agent receives RGB visual observation images, detects objects belonging to the COCO category for RGB images, and uses an open-vocabulary object detector to identify scene categories and other object categories, and obtains segmentation semantic masks using bounding box cues. Based on the obtained depth image, semantic mask and its corresponding semantic label and confidence score, each pixel and its semantic label are converted into a point cloud. Points falling within the height range of the agent are classified as obstacles. The point cloud is then projected onto the top-down map space through geometric operations to generate a map centered on the agent. The semantic channel of the centered map records each pixel as a probability value. Based on the agent's coordinates, the egocentric map is aggregated with the existing global map to generate the probabilistic semantic map.

3. The zero-shot target navigation based on functional Bayesian networks of claim 1, wherein, The real-time construction of the functional Bayesian network includes: By introducing functional groups to model the relationship between objects and scenes, the directed acyclic graph structure of the functional Bayesian network adopts object nodes pointing to functional nodes, or scene nodes pointing to functional nodes; based on the unidirectionality of the directed acyclic graph, bidirectional edges cannot be directly established between objects and scenes. The reasoning in the counterfactual reasoning chain includes: identifying the variables; determining the type of the edges between the variables as object-to-function or scene-to-function edges; assuming the existence of the edges and defining the factual and counterfactual situations in symbolic form; inferring the probabilities of the factual and counterfactual situations; comparing the probabilities of the factual and counterfactual situations and deriving estimated values; and finally forming the edges between the variables by evaluating whether the edges exist and the magnitude of their probability values.

4. The zero-shot target navigation based on functional Bayesian networks of claim 1, wherein, The real-time construction of the functional Bayesian network step also includes: At each navigation time step, for an object observed in the probabilistic semantic map, it is determined whether it is a newly observed object. If the projection of an object does not overlap with the corresponding category channel of the object in the previous map, then the object is considered a newly observed object. If an object has been previously observed, update the probability of that object in the previous map; For each newly observed object, register it as a new object node; determine the edges between the new node and other nodes in the functional Bayesian network.

5. The zero-shot target navigation based on functional Bayesian networks of claim 4, wherein, In the step of constructing the functional Bayesian network in real time, determining the edges between the new node and other nodes in the functional Bayesian network further includes: For an edge from an object to a function, identify each existing function group node within the reachable distance specified by the new node; for each function group node within the reachable distance threshold, use counterfactual reasoning thought chain reasoning to determine whether the newly observed object establishes a directed edge with an existing function node and estimate the confidence probability, and update it in the functional Bayesian network. For a scene-to-function edge, given a newly registered function group node, the scene associated with the newly registered function group node is determined by querying the corresponding scene channel at the matching coordinates in the probabilistic semantic map. The counterfactual reasoning chain is used to evaluate whether the scene establishes a directed edge with the existing function group node and to estimate the confidence probability. The function Bayesian network is then updated. Each object node is connected to a maximum of a preset number of function group nodes.

6. The zero-shot target navigation based on functional Bayesian networks of claim 1, wherein, The target navigation steps based on functional Bayesian networks include: The confidence scores of scenes / objects in the probabilistic semantic map are used as prior inputs to the functional Bayesian network. The posterior probabilities of the functions inferred by the functional Bayesian network are used to update the target occurrence probabilities of each pixel in the map, forming a continuously optimized closed loop to generate a probabilistic map.

7. The zero-shot target navigation based on functional Bayesian networks of claim 6, wherein, The target navigation step based on functional Bayesian networks further includes: The functional Bayesian network is divided into multiple sub-directed acyclic graphs centered on functional group nodes; for each functional group of the central node, a subgraph containing all connected parent nodes is extracted, preserving the relational context; wherein, the parent nodes include object nodes and scene nodes; Based on Bayesian inference, calculate the posterior probability of each subgraph containing the target object; For each of the functional group nodes, construct a bounding box centered on the coordinates of the functional group node, covering the positions of connected objects; A probability distribution is formed on the map; the probability maps of all the nodes of the aforementioned functional groups are aggregated to form a probability map of the target's appearance.

8. A zero-shot target navigation system based on functional Bayesian networks, employing the zero-shot target navigation method based on functional Bayesian networks according to any one of claims 1 to 7, characterized in that, The system, applied to artificial intelligence-based intelligent agents, includes: Probabilistic Semantic Map Construction Module: At each time step, the agent projects the real-time observed RGB image into a probabilistic semantic map with an open vocabulary centered on itself, and then weightedly fuses it with the global map; A function Bayesian network module is constructed in real time: based on the probability semantic map, a function Bayesian network is constructed in real time, and the function Bayesian network is a dynamic directed acyclic graph wherein, denotes a variable, denotes a probability directed edge between the variables, and the variable V includes a scene, an object and a function group, and the probability directed edge is determined by using a counterfactual reasoning thought chain; The target navigation module based on functional Bayesian networks uses a large language model to perform reasoning and counterfactual reasoning on the functional Bayesian network, and finally obtains a probability map for navigation. Under the guidance of the probability map, the agent achieves zero-cost target navigation.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the steps of the zero-shot target navigation method based on functional Bayesian networks as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the zero-shot target navigation method based on functional Bayesian networks as described in any one of claims 1-7.

Images