Zero-shot target navigation method based on fusion of graph matching and key point detection

Abstract

The application belongs to the technical field of robot autonomous navigation, and discloses a zero sample target navigation method based on fusion of graph matching and key point detection, which reconstructs the traditional target determination mode which only relies on visual similarity around the overall technical path of "perception - modeling - matching - decision", proposes a double determination mechanism of graph matching and image feature point fusion, and extracts target related features from the semantic layer and the fine-grained visual layer respectively. Among them, the semantic information is used to describe the object categories and their spatial relationship in the scene, and the key point feature is used to depict the local structure details of the target appearance, so as to form a multi-layer feature expression with complementarity, and reduce the navigation determination error caused by rotation and dynamic interference through a quantitative weighted fusion model. Further, a dynamic confidence and position confidence double attenuation mechanism is designed, and a node dynamic updating method is combined to realize real-time adaptation of the scene graph to the dynamic environment, and guarantee the navigation robustness.

Description

Technical Field

[0001] This invention belongs to the field of robot autonomous navigation technology, and more specifically, relates to a zero-shot target navigation method based on the fusion of graph matching and key point detection. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, autonomous navigation in complex environments has gradually become an important research direction in the fields of computer vision and intelligent systems. Target-oriented navigation, in particular, aims to enable intelligent agents to autonomously complete path planning and target search tasks in unknown environments based on a given objective, and has been widely applied in service robots, intelligent security, and human-computer interaction scenarios. In recent years, with the development of deep learning, researchers have proposed various vision-based navigation methods, among which end-to-end reinforcement learning methods and semantic modeling-based methods are representative, improving navigation efficiency and generalization capabilities to a certain extent.

[0003] In existing technologies, zero-shot target navigation, as a vision-based navigation method, has gradually become a research hotspot. This type of method typically utilizes pre-trained vision-language models or general visual representations to achieve the identification and navigation of unseen target categories. By introducing large models for semantic understanding, natural language descriptions are mapped to target features, and similarity matching is performed in conjunction with visual observations, thus achieving target navigation capabilities without additional training. However, the performance of this type of method in real-world complex environments still has significant shortcomings.

[0004] Most existing methods rely on global visual features or keypoint matching for target identification, essentially measuring the consistency between the current observation and the target through image similarity. While these methods perform well in ideal environments, in real-world scenarios, image features often change significantly due to factors such as viewpoint, scale, lighting, and occlusion, leading to unstable matching results. Furthermore, in dynamic environments, interference factors such as pedestrian movement and object position changes can easily cause misjudgments in matching methods based on single-frame visual features, thus affecting the reliability of navigation decisions.

[0005] Existing navigation systems have weak environmental modeling capabilities, with most methods making decisions based solely on current observations and lacking effective utilization of scene structure information. Although some studies have introduced semantic maps or topological graphs to represent the environment, these typically focus on spatial location modeling (SLAM), making it difficult to form stable high-level semantic understanding in complex scenes. This deficiency results in the system lacking effective reasoning ability when faced with scene reconstruction or when key targets are temporarily invisible, leading to decreased navigation efficiency or even mission failure.

[0006] In terms of adapting to dynamic environments, existing methods generally lack effective temporal modeling and information update mechanisms. Traditional scene representations typically employ a static accumulation approach, continuously overlaying historical observations onto the map, but fail to consider the information obsolescence caused by environmental changes over time. For example, when an object is moved or removed, the system may still rely on outdated information for decision-making, resulting in erroneous navigation behavior. Furthermore, the lack of differentiation and modeling of dynamic objects further reduces the system's robustness in real-world environments.

[0007] Furthermore, existing methods for target identification often employ a single threshold strategy, where a target is considered to have arrived when the visual similarity exceeds a set threshold. This simplistic approach ignores the complementarity between multiple sources of information, is susceptible to noise interference in complex environments, and results in a high false positive rate, making it difficult to meet the stability and accuracy requirements of practical applications. Summary of the Invention

[0008] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a zero-shot target navigation method that integrates graph matching and key point detection. Its purpose is to alleviate the matching instability caused by changes in viewpoint, lighting and dynamic interference, and improve navigation efficiency and the reliability of navigation decisions.

[0009] To achieve the above objectives, this invention provides a zero-shot target navigation method based on the fusion of graph matching and keypoint detection, comprising: S1. During navigation, the observation image at time t is converted into a node graph; the node graph is then topologically stitched and the observation scene graph at time t-1 is aligned with the nodes to form the observation scene graph at time t. S2. Calculate the node similarity and edge similarity between the target scene graph and the observed scene graph at time t. Then, weight and fuse the node similarity and edge similarity to obtain the graph matching confidence score. Wherein, the target scene graph is a node graph transformed from the target image to be navigated; the nodes in the node graph are objects in image P, the node attributes are the categories of the objects, the edges are the positional relationships between the objects, and image P is the target image or the observed image; S3. Calculate the feature consistency score between the target image and the observed image. Match the graph with confidence level. Consistency score with the aforementioned features The weighted fusion is then used as the navigation success determination function. ; S4. If Then, the navigation success determination function is improved. Obtain the observed image at time t+1 in the direction of [the method], and let t = t+1, then return to S1 for the next iteration; if If so, the navigation is considered successful.

[0010] Furthermore, the node similarity and edge similarity are respectively: ; ; in, , These represent node similarity and edge similarity, respectively. , These are the target scene diagrams. The set of nodes and the set of edges; , The observation scene diagrams at time t are shown below. The set of nodes and the set of edges; and They are nodes exist and The degree in the middle, for and Number of public nodes for and The number of common edges, , They are respectively The number of nodes and the number of edges in the middle. This indicates finding the intersection; Correspondingly, the graph matching confidence level for: ; in, It is a scene complexity factor, which is positively correlated with the sparsity of environmental objects.

[0011] Furthermore, the node similarity and edge similarity are respectively: ; ; ; in, , These represent node similarity and edge similarity, respectively. , These are the target scene diagrams. The set of nodes and the set of edges; , The observation scene diagrams at time t are shown below. The set of nodes and the set of edges; and They are nodes exist and The degree in the middle, for and Number of public nodes for and The number of common edges, for The number of nodes in the middle, This indicates finding the intersection; for any pair of nodes Connected edges; For nodes exist Relative space vectors in For nodes exist Relative space vectors in; Angle sensitivity factor; Correspondingly, the graph matching confidence level for: ; in, It is a scene complexity factor, which is positively correlated with the sparsity of environmental objects.

[0012] Furthermore, the feature consistency score for: ; ; in, This is the set of feature matching point pairs between the target image and the observed image. For the first The distance between two features in a pair of matching feature points, the th feature... The nth feature matching point pair is the nth feature in the target image. A pair of matching points formed by each feature point and any feature point in the observed image; For the corresponding matching confidence weight, when the first When a feature point is in the observed image ,otherwise ; The length of the current time window.

[0013] Furthermore, before calculating the node similarity and edge similarity, the observed scene graph at time t is updated, and the update method includes: For object i in the observed image, its corresponding node in the observed scene graph at time t is... compute nodes Dynamic confidence : ; in, For the Sigmoid function; For nodes The attribute information represents the category of object i. The category of object i Inherent dynamic score; Let be the velocity of object i in the world coordinate system. Based on the category of object i The 3D coordinates of object i are calculated using the optical flow method, and the 3D coordinates of object i are determined by the depth information corresponding to the observed image. To set weights; compute nodes Location confidence at time t ,in, With dynamic confidence Positive correlation; Constructing fused location confidence Evaluation function of time decay : ; in, Represents a node The time interval from the last observation to time t; These are the weighting coefficients. This is the time decay factor; when Higher than the preset threshold At that time, determine the node For stable objects, they are retained in the observed scene graph at time t; otherwise, the nodes are... Delete the observation scene graph at time t and delete the corresponding edges to obtain the updated observation scene graph at time t; Correspondingly, the node similarity and edge similarity are calculated based on the updated observation scene graph at time t.

[0014] Furthermore, nodes Location confidence at time t for: , ; in, is the attenuation constant.

[0015] Furthermore, the node graph is topologically stitched and node-aligned with the observation scene graph at time t-1 to obtain the observation scene graph at time t, including: For the observed target in the image at time t It corresponds to the node in the node diagram. compute nodes Euclidean distances between nodes in the observation scene diagram at time t-1; When the minimum Euclidean distance exceeds a preset threshold At time t-1, a new node is created in the observation scene graph to represent the node. Otherwise, the node Match the nearest neighbor node in the observation scene graph at time t-1 to obtain the observation scene graph at time t; wherein, the nearest neighbor node is the node in the observation scene graph at time t-1 corresponding to the minimum Euclidean distance.

[0016] The present invention also provides a zero-sample target navigation robot, including a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the zero-shot target navigation method based on graph matching and key point detection fusion as described above.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the zero-shot target navigation method based on graph matching and keypoint detection fusion as described in any of the preceding claims.

[0018] The present invention also provides a computer program product, including a computer program that, when the computer program is run on a computer, causes the computer to execute the zero-shot target navigation method based on graph matching and key point detection fusion as described in any of the preceding claims.

[0019] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention models the environment by acquiring observation information of the current scene and constructing the current observation scene graph structure. It abstracts the observed object instances into nodes and establishes edge connections through spatial relationships (the positional relationships of objects), realizing the transformation from pixel-level representation to structured semantic representation. It makes full use of the semantic relationships between objects and scene structure information to enhance the ability to model the environment and can form a stable high-level semantic understanding in complex scenes. Based on this, when making target determination, it breaks through the traditional single visual matching strategy and proposes a dual determination mechanism that integrates feature point matching and scene graph matching: on the one hand, it measures the appearance consistency between the target image and the current observation through key point matching (feature consistency score representation); on the other hand, it measures the degree of matching between the two at the semantic layer and the spatial relationship layer through scene graph structure similarity (graph matching confidence representation). By weighted fusion of the two types of matching results, a unified matching confidence evaluation function (navigation success determination function) is constructed, thereby effectively alleviating the problem of unstable matching caused by changes in viewpoint, lighting, and dynamic interference. In the navigation decision-making process, the current state is evaluated in real time based on the above matching results, and the next action is generated in combination with the strategy model to realize continuous exploration guided by the target. When the fusion matching confidence reaches a preset threshold, navigation is deemed successful and the task is terminated; otherwise, the scene graph is continuously updated and the navigation strategy is iteratively executed, improving navigation efficiency and the reliability of navigation decisions. Through the above mechanism, this invention achieves unified modeling from low-level visual perception to high-level semantic reasoning.

[0020] (2) Furthermore, relative geometric topological constraints are introduced in the scene graph matching process, namely, node In the target scene diagram And the current observation scene diagram Moderate, and and The number of common edges requires not only that the object categories be consistent (common nodes be consistent), but also that the spatial relative orientations between objects be consistent (common edges be consistent). This avoids misjudgment when the environment layout is similar, such as symmetrical rooms or when the camera rotates significantly, and further improves the discrimination accuracy of zero-sample navigation in complex large scenes.

[0021] (3) Furthermore, when calculating edge similarity, an orientation consistency factor is introduced. The constructed edge similarity index not only counts the number of "common edges" but also quantifies the consistency of the spatial structure represented by these edges, eliminating the influence of the absolute coordinate system during robot navigation and focusing only on the relative geometric topology between objects. Even if the robot rotates violently in a dynamic environment, as long as the relative spatial relationship between the target object and its surrounding references does not change abruptly, the navigation decision function remains valid. It can still maintain a high level of confidence output, effectively reducing the false trigger rate.

[0022] (4) Furthermore, when calculating the feature consistency score, matching quality information is incorporated into the traditional matching quantity. Specifically, a matching confidence weight is introduced based on the matching point pairs. Matching confidence weights In Characterize the feature points to be matched in the target image In the The presence of indicator variables in the frame observation image can highlight high-confidence matching point pairs, thereby suppressing the influence of noisy matching pairs on the overall judgment and further improving the reliability of navigation decisions.

[0023] (5) Furthermore, in order to adapt to the changes in the position and uncertainty of the state of objects in a dynamic environment, this invention continuously corrects the node attributes in the observed scene graph by calculating the dynamic confidence of the nodes and the confidence of the node position, and suppresses the interference of outdated information on the global graph structure by using a confidence decay strategy, so that the scene graph always maintains consistency with the real environment.

[0024] In summary, this invention, by constructing a multimodal feature modeling and structured scene expression mechanism, enables target determination and navigation decision-making for unknown targets in complex dynamic environments. This alleviates the instability problem of matching caused by changes in viewpoint, lighting, and dynamic interference, and improves navigation efficiency and the reliability of navigation decisions. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a dynamic environment zero-shot target navigation method that integrates graph matching and keypoint detection in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the dual determination mechanism of graph matching and image feature point fusion in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of node dynamic updates in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram illustrating the preprocessing of target images in the dataset in an embodiment of the present invention.

[0029] Figure 5 for Figure 4 A schematic diagram of feature point matching between the target image and the currently observed image.

[0030] Figure 6 This is an example of a robot performing a target navigation in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] Example 1 This invention provides a zero-shot target navigation method based on graph matching and keypoint detection fusion. Following the overall technical path of "perception-modeling-matching-decision," it reconstructs the traditional target determination method that relies solely on visual similarity. A dual determination mechanism integrating graph matching and image feature point fusion is proposed, extracting target-related features from both the semantic layer (currently observed scene graph) and the fine-grained visual layer (feature points). Semantic information describes the categories of objects in the scene and their spatial relationships (positional relationships), while keypoint features characterize the local structural details of the target's appearance, thus forming a complementary multi-layered feature representation. A quantized weighted fusion model is used to reduce navigation determination errors caused by rotation and dynamic interference. Furthermore, a dual decay mechanism of dynamic confidence and positional confidence is designed, combined with a dynamic node update method, to achieve real-time adaptation of the scene graph to the dynamic environment, ensuring navigation robustness.

[0033] I. Target determination based on graph matching method and key point feature fusion.

[0034] To address the issue that existing graph matching frameworks, which rely solely on visual feature keypoint matching for navigation success determination, are susceptible to interference from rotation, dynamic objects, and other factors, this invention proposes a dual-determination method that integrates feature point matching and scene graph matching. By quantifying the confidence level of visual feature matching and the structural similarity of the scene graph, a multimodal determination criterion is constructed.

[0035] like Figure 1 As shown, the target navigation robot first acquires an image of the target to be navigated. Then, the robot uses an RGB-D sensor to acquire RGB-D video frame information in real time during the navigation process. Each frame of RGB-D information simultaneously contains multimodal data of the observed image (RGB color image) and the depth image at the current moment; this stage is the robot's perception stage. In the next stage, the robot uses a multimodal language model to transform the target image into a node graph with semantic and positional information, i.e., a target scene graph. In this node graph, nodes represent objects in the target image, node attribute information indicates the object type, and connecting edges represent the relative positional relationships between objects. Preferably, the input to the multimodal language model also includes textual information describing the positions of target objects in the target image. Simultaneously, for the observation image of the current frame acquired in real-time by the robot, node graph modeling is also performed. The observation image of the current frame (corresponding to the current time) is input into the multimodal language model, and the corresponding node graph is output. In this node graph, nodes represent objects in the observation image at the current time, node attribute information indicates the object type, and connecting edges represent the relative positional relationships between objects. Furthermore, the node graph at the current time t is compared with... After performing topology stitching and node alignment on the observation scene graph at time t, the observation scene graph at time t is obtained. (Local semantic map), i.e., the current observation scene map; the initial observation scene map is the node map corresponding to the initial observation image. In the third stage, matching and decision-making are performed. A dual-determination mechanism fusing feature point matching and scene graph matching is used to perform feature point and node graph fusion matching to determine whether the current frame observation image matches the target image. If a match is found, the robot navigation is considered successful, meaning the robot has navigated to the target image's location. Otherwise, the robot continues navigation, moving in the direction with higher confidence until navigation ends. Addressing the problem that existing graph matching frameworks relying solely on visual feature keypoint matching for navigation success determination are susceptible to interference from object rotation, occlusion, and dynamic objects, this invention proposes a dual-determination method based on scene graphs, representing the node attributes and edge spatial relationships of objects in a structured manner. This method integrates visual feature matching and graph structure matching to construct a multimodal determination criterion. By quantifying the confidence of visual feature matching and the similarity of scene graph structure, this method constructs a multimodal determination criterion, effectively reducing errors caused by dynamic interference and geometric transformations, and improving the robustness of zero-sample target-oriented navigation in real-world scenarios.

[0036] Specifically, feature point matching is performed between the target image and the observed image of the current frame. This step yields the feature point matching information between the target image and the observed image of the current frame.

[0037] At the visual level, local feature points in the target image and the currently observed image describe the fine-grained appearance information of the object. To improve matching robustness, this invention proposes a weighted feature consistency modeling mechanism, incorporating matching quality information into the traditional matching quantity. Specifically, let the set of feature matching point pairs between the target image and the currently observed image be... The feature consistency score between the target image and the currently observed image is... for: ; ; in, Represents the set of feature matching point pairs. The descriptor distance of the matching pair, i.e., the th matching pair The distance between two features in a matching pair; the distance between the first and second features. Each matching pair consists of feature points to be matched in the target image. A feature point pair formed with any feature point in the currently observed image; This corresponds to the matching confidence weight. This mechanism can highlight high-confidence matching pairs, thereby suppressing the influence of noisy matching pairs on the overall judgment. This depends on the inter-frame stability metric. The length of the current time window. Represents the feature points to be matched in the target image. In the The presence of indicator variables in the frame observation image, when the feature points to be matched are in the target image. In the When it exists in the frame observation image, ,otherwise This scheme improves system stability by extracting stable feature points through time series analysis. This indicates taking the absolute value. The metric (feature consistency score) This reflects the degree of visual matching between the target image and the currently observed image. The value range is [0,1]. The higher the value, the more complete the visual feature matching.

[0038] When performing scene graph matching metrics, it's divided into two parts: node similarity and edge similarity. This applies to the constructed target scene graph. Compared with the current observation scene diagram ,in For a set of nodes, i.e., objects. Let be the set of edges, representing the positional relationships between objects; where, For the target scene diagram The set of nodes in the target image, i.e., the set of objects in the target image; Image of the current observation scene Middle node set; This represents the positional relationships between objects in the target image. This represents the positional relationships between objects in the currently observed scene graph. Therefore, node similarity... Similarity between edges The calculations are as follows: ; ; in, and They are nodes exist and The degree in the middle, for and Number of public nodes. for and The number of common edges; common edges must satisfy the condition that the source node, target node, and edge type are all the same. This indicates taking the intersection. This indicates taking the absolute value. , They are respectively The number of nodes and edges in the graph is considered. However, traditional graph matching only focuses on node categories and connectivity, which can easily lead to misjudgments when the environment layout is similar, such as symmetrical rooms or when the camera rotates significantly. This invention introduces relative geometric topological constraints, i.e., nodes... In the target scene diagram And the current observation scene diagram Moderate, and and The number of common edges requires not only that the object categories are consistent (consistent common nodes), but also that the spatial relative orientations between objects remain consistent (consistent common edges), thus improving the discrimination accuracy of zero-shot navigation in complex large-scale scenes. This is in line with the aforementioned node similarity. Similarity with edges Based on this, the present invention further introduces a relative orientation consistency factor. For any pair of connected nodes in the scene graph Its target scene map And the current observation scene diagram The relative space vectors in are respectively and Define the orientation consistency function: ; in, The angle sensitivity factor is determined empirically; the symbol is "". The "" indicates dot product calculation. This term ensures the relative orientation between objects, such as "the cup is on the left side of the table," which has rotational invariance across different viewpoints.

[0039] The edge similarity formula after incorporating geometric constraints is updated as follows: ; This indicator not only counts the number of "common edges," but also quantifies the consistency of the spatial structure represented by these edges. The final graph matching confidence score... Introducing a scenario complexity factor To achieve adaptive weight allocation: ; in, This represents the confidence level of the graph matching between the target scene graph and the currently observed scene graph; Dynamically adjust based on the currently observed node density: when environmental objects are sparse, increase the edge similarity weight (decrease it). ), that is, structural information weight; when objects are densely packed, increase the node similarity weight (increase (i.e., category information weight).

[0040] By introducing a unit vector ( and Orientation consistency of dot product calculation (Relative spatial vectors only represent direction), the system eliminates the influence of the robot's absolute coordinate system, focusing only on the relative geometric topology between objects. Even if the robot rotates violently in a dynamic environment, as long as the relative spatial relationship between the target object and its surrounding references does not change abruptly, the navigation decision function... It can still maintain a high level of confidence output, effectively reducing the false trigger rate.

[0041] like Figure 2 As shown, the target scene map is matched with the currently observed scene map, and the resulting map matching confidence score is... As the similarity confidence between the target scene map and the currently observed scene map.

[0042] Finally, the visual feature matching confidence score (i.e., feature consistency score) will be fused. ) Confidence of matching graph Construct a navigation success determination function : ; in, The weight coefficients for visual features, when If navigation is successful, the location of the current frame image captured by the robot is determined to be the location of the target image; otherwise, navigation is determined to be unsuccessful, and the robot continues to probe along the path defined by the navigation success determination function. The system moves in the direction of τ to obtain the next frame of the observed image, where τ is the decision threshold, which is empirically determined.

[0043] II. Dynamic Environment Adaptation and Graph Structure Attenuation Mechanism.

[0044] As a further design of this invention, the statically incrementally constructed scene graph only considers new observations but does not handle the movement of objects in the environment, such as chairs being moved or items being taken away in dynamic scenes. This causes the current observed scene graph structure to become disconnected from the real environment, leading to the accumulation of inference errors. To solve this problem, this embodiment of the invention will design a dynamic environment adaptation and graph structure decay mechanism through "dynamic node tracking + temporal graph update + re-perception triggering" to achieve real-time alignment between the current observed scene graph and the real environment.

[0045] (1) Dynamic confidence decay.

[0046] To identify dynamic nodes in the current observation scene graph, the attribute dimensions of the nodes are expanded to include dynamically relevant states. Then, dynamic target recognition is achieved through multimodal sensor fusion. Each node in the current observation scene graph... Includes the following attributes: status label ,in, Represents a node For static nodes, Represents a node For dynamic nodes, Represents a node The state is unknown. Dynamic confidence level. , which represents the probability that a node is a dynamic object (the probability that the object may move).

[0047] By fusing RGB-D images with optical flow, dynamic objects in the local scene graph are identified and their state labels are updated, including: detecting object instances in the current frame of the observed image, which are nodes in the current observed scene graph. Get the object (node) ) category and 3D coordinates , where 3D coordinates Determined based on depth information corresponding to the observed image in the current frame; then, based on category. and 3D coordinates The optical flow method is used to calculate the pixel motion vectors of the current frame observation image and its previous adjacent frame observation image, in order to locate the object (node). Convert the velocity to the world coordinate system Then calculate the nodes in the current observation scene graph. Dynamic confidence (Dynamic confidence decay), the formula is as follows: ; in For the Sigmoid function, To set weights, Category of objects The inherent dynamic scores are empirically determined, such as 0.8 for "cup" and 0.1 for "wall". Then it represents the actual speed of the object. The impact, The larger, the more it means right The greater the impact. This represents the influence weight of the semantic category of the object. The larger the size, the more semantic category the object belongs to. The greater the weight it occupies.

[0048] (2) Node location confidence decay.

[0049] In constructing the current observation scene graph, some nodes become unreliable over time. For example, a water glass on a table might move when not being observed, but the probability of the table itself moving is relatively small. Therefore, a method is needed to delete or reduce the confidence of such a node. The core objective is to remove outdated and unreliable information from the current observation scene graph in real time, and synchronously update the node and edge relationships in the dynamic environment. This ensures that the current observation scene graph remains consistent with the real environment, providing a precise semantic and spatial foundation for subsequent graph matching and multi-stage exploration. Combining the temporal characteristics of dynamic nodes, a three-level mechanism—"confidence decay - node / edge update - temporal consistency maintenance"—is used to achieve lightweight, high-precision dynamic adaptation of the current observation scene graph.

[0050] For static nodes in the current observation scene graph, their positions remain almost unchanged, and the confidence decays very slowly or not at all. For dynamic nodes, their positions are easily changed, and the reliability of their coordinates decreases rapidly when not observed by the robot. The decay rate of node position confidence is similar to that of dynamic confidence. Positive correlation, meaning the stronger the dynamics, the faster the decay. The design formula is as follows: ; in, Represents a node The confidence score of the location at time t, and its range. , , It is the attenuation constant; Represents a node The duration from the last observation by the robot to time t. Since the coordinates were completely reliable during the first observation, the initial values... This formula shows that the more dynamic a node is, the faster its position confidence decays. This is especially true when the robot has not observed a node. After a period of time, it will approach 0, providing a basis for subsequent node deletion.

[0051] (3) Node dynamic update method.

[0052] In dynamic environments, to ensure the accuracy and real-time performance of the current observation scene graph structure, this invention designs a node dynamic update mechanism based on the fusion of spatial consistency and temporal confidence, which performs addition, retention, and deletion operations on nodes in the current observation scene graph.

[0053] like Figure 3 As shown, the confidence level of a node's position decays to varying degrees over time. When the confidence level of a node's position decays to less than a threshold, the node is deleted. Simultaneously, edges are updated based on the confidence level of the nodes' positions, thereby deleting edges with low confidence levels. Furthermore, new nodes and edges are updated in real-time as the robot observes.

[0054] Specifically, for time Observation target in the acquired observation image (Nodes in the node graph corresponding to the currently observed image) First, based on its spatial location and the current set of graph nodes in the observed scene. Perform matching. Let the spatial location of the node be represented as... Then, by calculating the observed target and Observation scene diagram at time (based on time) The Euclidean distance between nodes in the previously constructed observation image is used to determine the observation target. exist The attribution relationship in the observation scene graph at any given time. When the minimum Euclidean distance exceeds a preset threshold. At that time, determine the observation target For new goals, and in The observation scene diagram at a given time shows the observation target. Create a new node; otherwise, observe the target. Match to The nearest neighbor node in the observation scene graph at any given time, where the nearest neighbor node is the node in the current observation scene graph that is closest to the observed target. The node with the smallest Euclidean distance between the nodes is used to update its attributes. The updated observation scene graph is then used as the observation scene graph at time t, i.e., the current observation scene graph. This process can be formally represented as: ; Introducing location confidence during node update process With time interval The node status in the current observation scene graph is dynamically evaluated. This includes location confidence. This is used to measure the stability of a node through continuous observation; the time interval represents the duration since the node was last observed by the robot. To achieve robust management of dynamic targets, this invention further constructs a unified evaluation function that integrates confidence level and time decay: ; in, These are the weighting coefficients. These are time decay factors, all empirically determined. Based on this evaluation function, the node states in the current observation scene graph are determined: ; When the node's overall score Above the threshold At that time, determine the node To ensure the stability of the target, it is retained; conversely, when a node has not been observed for a long time or its confidence level continues to decrease, it is deleted from the current observation scene graph, thereby avoiding redundant nodes from interfering with navigation decisions.

[0055] Calculate node similarity in the updated current observation scene graph. Similarity between edges ; then calculate the graph matching confidence score This leads to updating the navigation success determination function. This is used to determine whether navigation was successful.

[0056] Through the above mechanism, this invention achieves adaptive modeling of target states in dynamic environments, enabling the scene graph to be updated in real time as the environment changes. On the one hand, newly emerging targets can be quickly introduced into the graph structure, enhancing the expressive power of the environment; on the other hand, nodes that have disappeared or are no longer reliable can be promptly removed, thereby ensuring the compactness and accuracy of the graph structure. Furthermore, this method introduces a continuous-time decay mechanism (…). This effectively avoids the misleading effect of short-term occlusion or sensor noise on node status judgment, and improves the overall robustness and stability of the system.

[0057] In this embodiment of the invention, to verify the effectiveness of the method, simulation experiments are conducted using a general navigation dataset, selecting the target image to be navigated by the robot, multiple frames of observation images of the robot along the navigation path, and corresponding depth images. To demonstrate universality, such as... Figure 4 As shown, the target image is first preprocessed in different ways to increase interference, in order to verify the applicability of the method of the present invention in various scenarios and demonstrate the strong robustness of the method of the present invention: 1) Preprocessing method one: Adding extra objects to the target image to introduce interference noise, such as Figure 4 As shown in a. 2) Preprocessing method two: Reconstruct the target scene, that is, only focus on the semantic information of the target scene, such as Figure 4 As shown in b. 3) Preprocessing method three: Increase the scene illumination intensity in the target image, such as... Figure 4 As shown in c. 4) Preprocessing method four: Reduce the scene illumination intensity in the target image, such as Figure 4 As shown in d.

[0058] like Figure 5 As shown, in different preprocessing methods, feature point matching is performed between the target image and the observed image of the current frame to obtain feature point matching information between the target image and the current observed image. Specifically, in the original target image, the number of matching feature points between the target image and the current observed image is 131. After processing with four different preprocessing methods, the number of matching feature points is 83, 41, 106, and 72, respectively. Figure 5 As shown in the ad.

[0059] Through simulation, this invention can quickly and accurately navigate to the location of the target image.

[0060] like Figure 6 The image shows an example of target navigation. In a simulated environment, the robot explores with zero samples and no pre-training, accurately and quickly finding the location corresponding to the target image.

[0061] Example 2 This invention provides a target navigation robot, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the zero-shot target navigation method that fuses graph matching and key point detection in Embodiment 1 above.

[0062] The relevant technical solutions are the same as above, and will not be repeated here.

[0063] Example 3 This invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the zero-shot target navigation method that fuses graph matching and keypoint detection in Embodiment 1 above.

[0064] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0065] The relevant technical solutions are the same as above, and will not be repeated here.

[0066] Example 4 This invention provides a computer program product, including a computer program that, when run on a computer, causes the computer to perform the steps of the zero-shot target navigation method that fuses graph matching and key point detection in Embodiment 1.

[0067] The relevant technical solutions are the same as above, and will not be repeated here.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A zero-shot target navigation method based on graph matching and keypoint detection fusion, characterized in that, include: S1. During navigation, the observation image at time t is converted into a node graph; the node graph is then topologically stitched and the observation scene graph at time t-1 is aligned with the nodes to form the observation scene graph at time t. S2. Calculate the node similarity and edge similarity between the target scene graph and the observed scene graph at time t. Then, weight and fuse the node similarity and edge similarity to obtain the graph matching confidence score. Wherein, the target scene graph is a node graph transformed from the target image to be navigated; the nodes in the node graph are objects in image P, the node attributes are the categories of the objects, the edges are the positional relationships between the objects, and image P is the target image or the observed image; S3. Calculate the feature consistency score between the target image and the observed image. Match the graph with confidence level Consistency score with the aforementioned features The weighted fusion is then used as the navigation success determination function. ; S4. If Then, the navigation success determination function is improved. Obtain the observed image at time t+1 in the direction of [the method], and let t = t+1, then return to S1 for the next iteration; if If so, the navigation is considered successful.

2. The zero-sample target navigation method according to claim 1, characterized in that, The node similarity and edge similarity are respectively: ； ； in, , These represent node similarity and edge similarity, respectively. , These are the target scene diagrams. The set of nodes and the set of edges; , The observation scene diagrams at time t are shown below. The set of nodes and the set of edges; and They are nodes exist and The degree in the middle, for and Number of public nodes for and The number of common edges, , They are respectively The number of nodes and the number of edges in the middle. This indicates finding the intersection; Correspondingly, the graph matching confidence level for: ； in, It is a scene complexity factor, which is positively correlated with the sparsity of environmental objects.

3. The zero-sample target navigation method according to claim 1, characterized in that, The node similarity and edge similarity are respectively: ； ； ； in, , These represent node similarity and edge similarity, respectively. , These are the target scene diagrams. The set of nodes and the set of edges; , The observation scene diagrams at time t are shown below. The set of nodes and the set of edges; and They are nodes exist and The degree in the middle, for and Number of public nodes for and The number of common edges, for The number of nodes in the middle, This indicates finding the intersection; for any pair of nodes Connected edges; For nodes exist Relative space vectors in For nodes exist Relative space vectors in; Angle sensitivity factor; Correspondingly, the graph matching confidence level for: ； in, It is a scene complexity factor, which is positively correlated with the sparsity of environmental objects.

4. The zero-sample target navigation method according to any one of claims 1-3, characterized in that, The feature consistency score for: ； ； in, This is the set of feature matching point pairs between the target image and the observed image. For the first The distance between two features in a pair of matching feature points, the th feature... The nth feature matching point pair is the nth feature in the target image. A pair of matching points formed by a feature point and any feature point in the observed image; For the corresponding matching confidence weight, when the first When a feature point is in the observed image ,otherwise ; The length of the current time window.

5. The zero-sample target navigation method according to any one of claims 1-3, characterized in that, Before calculating the node similarity and edge similarity, the observed scene graph at time t is updated, and the update method includes: For object i in the observed image, its corresponding node in the observed scene graph at time t is... compute nodes Dynamic confidence : ； in, For the Sigmoid function; For nodes The attribute information represents the category of object i. The category of object i Inherent dynamic score; Let be the velocity of object i in the world coordinate system. Based on the category of object i The 3D coordinates of object i are calculated using the optical flow method, and the 3D coordinates of object i are determined by the depth information corresponding to the observed image. To set weights; compute nodes Location confidence at time t ,in, With dynamic confidence Positive correlation; Constructing fused location confidence Evaluation function of time decay : ； in, Represents a node The time interval from the last observation to time t; These are the weighting coefficients. This is the time decay factor; when Higher than the preset threshold At that time, determine the node For stable objects, they are retained in the observed scene graph at time t; otherwise, the nodes are... Delete the observation scene graph at time t and delete the corresponding edges to obtain the updated observation scene graph at time t; Correspondingly, the node similarity and edge similarity are calculated based on the updated observation scene graph at time t.

6. The zero-sample target navigation method according to claim 5, characterized in that, node Location confidence at time t for: ， ； in, is the attenuation constant.

7. The zero-sample target navigation method according to any one of claims 1-3, characterized in that, The node graph is topologically stitched and node-aligned with the observation scene graph at time t-1 to form the observation scene graph at time t, including: For the observed target in the image at time t It corresponds to the node in the node diagram. compute nodes Euclidean distances between nodes in the observation scene diagram at time t-1; When the minimum Euclidean distance exceeds a preset threshold At time t-1, a new node is created in the observation scene graph to represent the node. Otherwise, the node Match the nearest neighbor node in the observation scene graph at time t-1 to obtain the observation scene graph at time t; wherein, the nearest neighbor node is the node in the observation scene graph at time t-1 corresponding to the minimum Euclidean distance.

8. A zero-sample target navigation robot, characterized in that, Includes computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the zero-shot target navigation method based on graph matching and key point detection fusion as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the zero-shot target navigation method based on graph matching and key point detection fusion as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the zero-shot target navigation method based on graph matching and keypoint detection fusion as described in any one of claims 1-7.

Images