Robot navigation method and device based on multimodal semantic topological memory map

Abstract

This invention discloses a robot navigation method and apparatus based on a multimodal semantic topology memory map, relating to the field of intelligent robots. One specific implementation of the method includes: receiving natural language commands issued to the robot and encoding them as text embedding vectors; acquiring semantic feature vectors of each node in the multimodal semantic topology memory map, determining the semantic correlation between the text embedding vectors and each semantic feature vector, and identifying the node with the highest semantic correlation as the target node; determining the robot's current node, planning a global navigation path from the current node to the target node based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map, and controlling the robot to perform navigation operations based on the global navigation path. This implementation ensures the physical accessibility and obstacle avoidance safety of the global navigation path, achieving an efficient closed loop from natural language commands to safe autonomous navigation.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots, and in particular to a robot navigation method and apparatus based on a multimodal semantic topological memory map. Background Technology

[0002] Traditional robot navigation methods often rely on lidar or visual sensors to construct dense point clouds or maps with abundant feature points. These methods suffer from limitations such as high storage requirements, poor adaptability to dynamic environments, and susceptibility to interference from moving objects, leading to decreased positioning accuracy. While existing technologies have proposed semantic mapping methods that combine geometric and semantic information, these methods struggle to effectively balance semantic extraction accuracy and computational efficiency in real-time scenarios. Furthermore, they lack robust map update mechanisms to adapt to dynamic environmental changes, thus limiting the robot's navigation efficiency and positioning accuracy in complex environments. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a robot navigation method and apparatus based on a multimodal semantic topological memory map, which can at least solve the problems of large storage consumption, poor adaptability to dynamic environments, and insufficient semantic understanding ability in the prior art.

[0004] To achieve the above objectives, according to one aspect of the present invention, a robot navigation method based on a multimodal semantic topological memory map is provided, comprising: Receive natural language instructions issued to the robot and encode the natural language instructions into text embedding vectors; Read the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node; The robot's current node is determined, and a global navigation path from the current node to the target node is planned based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map. The robot is then controlled to perform navigation operations based on the global navigation path.

[0005] To achieve the above objectives, according to another aspect of the present invention, a robot navigation device based on a multimodal semantic topological memory map is provided, comprising: The encoding module is used to receive natural language instructions issued to the robot and encode the natural language instructions into text embedding vectors; The filtering module is used to obtain the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node. The planning module is used to determine the current node of the robot and plan a global navigation path from the current node to the target node based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map. The control module is used to control the robot to perform navigation operations based on the global navigation path.

[0006] To achieve the above objectives, according to another aspect of the present invention, a robot navigation electronic device based on a multimodal semantic topological memory map is provided.

[0007] The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the above-described robot navigation methods based on multimodal semantic topological memory maps.

[0008] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the robot navigation methods based on multimodal semantic topological memory maps described above.

[0009] To achieve the above objectives, according to another aspect of the present invention, a computing program product is provided. The computing program product of the present invention includes a computer program, which, when executed by a processor, implements the robot navigation method based on a multimodal semantic topological memory map provided in the present invention.

[0010] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: by encoding natural language instructions into text embedding vectors and calculating their semantic correlation with node semantic feature vectors to determine target nodes, accurate target anchoring of user intent is achieved based on cross-modal similarity calculation of text and images; by combining path planning with the local two-dimensional occupancy grid map of each node in the multimodal semantic topological memory map, the physical traversability and obstacle avoidance safety of the global navigation path are ensured, thereby realizing an efficient closed loop from natural language instructions to safe autonomous navigation. Furthermore, by replacing traditional dense point cloud maps or dense feature point maps with multimodal semantic topological memory maps, the storage footprint of the navigation system can be significantly reduced and the efficiency of environmental representation can be improved.

[0011] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0012] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main process for constructing a multimodal semantic topological memory map according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of constructing a multimodal semantic topological memory map; Figure 3 This is a schematic diagram of the main process of a robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating an optional robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating another optional robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the main modules of a robot navigation device based on a multimodal semantic topological memory map according to an embodiment of the present invention; Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0013] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0015] Traditional robot navigation methods primarily rely on lidar or visual sensors (such as cameras) for environmental perception and localization, which demonstrate high reliability in specific scenarios. On one hand, lidar-based navigation methods construct dense point cloud maps through high-precision distance measurements, providing accurate geometric information to support robot localization and path planning. However, this method typically requires significant storage space and has poor adaptability to dynamic environments, where dynamic objects can easily cause mismatches and localization errors.

[0016] On the other hand, navigation methods based on visual sensors utilize image data acquired by cameras for SLAM (Simultaneous Localization and Mapping), offering advantages such as low cost and rich information. However, this method exhibits weak robustness in dynamic scenes; the presence of dynamic objects can interfere with feature extraction and matching processes, thereby reducing the accuracy of localization and mapping. Furthermore, traditional navigation methods largely rely on high-resolution dense point cloud maps or dense feature point maps. These maps not only require significant storage but also struggle to adapt quickly to dynamic environmental changes, limiting their application in complex scenarios.

[0017] In recent years, semantic maps have attracted widespread attention due to their potential to enhance the intelligence of robot navigation. By combining geometric and semantic information in the environment, semantic maps can provide robots with a higher level of environmental understanding. In terms of semantic information extraction, research focuses on how to efficiently and accurately extract semantic labels from sensor data using deep learning techniques. For example, semantic segmentation methods based on CNNs (Convolutional Neural Networks) have been widely applied in the field of visual SLAM to achieve pixel-level classification of environmental data.

[0018] In semantic map construction, researchers have proposed various methods to integrate geometric and semantic information. For example, some studies have shown that by improving the DeepLabv3+ network and combining it with multi-view geometric information, the motion state of dynamic objects can be effectively detected and a topological map containing semantic information can be generated, such as whether there is an obstacle 3 meters ahead or a person 3 meters ahead. Furthermore, semantic map research also involves how to utilize lightweight models (such as MobileNetV2) to maintain high semantic segmentation accuracy while reducing computational complexity. Nevertheless, current semantic map research still faces some challenges, such as how to balance the accuracy of semantic information extraction with computational efficiency in complex scenarios with high real-time requirements, and how to design robust semantic map update mechanisms to adapt to changes in the dynamic environment.

[0019] See Figure 1The diagram shows the main flowchart of constructing a multimodal semantic topological memory map according to an embodiment of the present invention, including the following steps: S101: Environmental data of keyframes is acquired through a data acquisition device mounted on the robot; wherein, the environmental data includes color images and depth information; S102: Encode the color image to obtain a semantic feature vector; and construct a local two-dimensional occupancy raster map based on the depth information; S103: The color image, the semantic feature vector, the local two-dimensional occupancy grid map, and the robot's pose information when the key frame is acquired are fused together to serve as the attribute of the node corresponding to the key frame. S104: Connect the nodes in sequence according to the acquisition time of the key frames to obtain the multimodal semantic topology memory map.

[0020] In the above implementation, for step S101, the environmental data in this solution is mainly collected by a data acquisition device mounted on the robot. Therefore, in the process of constructing a multimodal semantic topological memory map, the robot efficiently collects environmental data in front of it at a specific data acquisition frequency using the onboard data acquisition device, obtaining continuous frames of environmental data. The preferred data acquisition device is an RGB (Red Green Blue)-D camera, which can simultaneously capture color images and depth information of the scene (including the specific distance values ​​of each pixel in the image, such as the pixel in the center being 1.5 meters from the lens and the pixel in the upper left corner being 3 meters from the lens), providing rich raw material for subsequent data processing.

[0021] In addition, the data acquisition frequency is preferably set to 10-30 frames / second to ensure that sufficiently dense environmental details can be captured during robot movement, while avoiding data redundancy and computational burden caused by excessively high frequency. The acquisition range covers a horizontal field of view of approximately 120-180 degrees and a vertical field of view of approximately 90 degrees in front of the robot. This range ensures comprehensive perception of the surrounding environment while reducing the introduction of invalid data. In an optional implementation, the acquired environmental data can be preprocessed, such as through preliminary filtering, to eliminate the influence of noise and outliers, thereby improving the accuracy of subsequent node generation and feature extraction.

[0022] To reduce the computational load on the system, keyframes can be extracted from consecutive frames. Keyframe extraction is a crucial step in constructing a multimodal semantic topological memory map, triggered by precise pose constraints provided by odometry (such as Fast-LIO (Fast LiDAR-Inertial Odometry)). Odometry is primarily used to estimate the robot's motion state in real time, obtaining the robot's accumulated translational distance and rotational angle. Fast-LIO, a tightly coupled LiDAR-Inertial Odometry method, achieves high-precision pose estimation by deeply fusing environmental 3D point cloud data acquired by LiDAR with high-frequency acceleration and angular velocity data acquired by an IMU (Inertial Measurement Unit).

[0023] Therefore, the first frame in a series of frames can be used as the starting frame. Based on the environmental data of the current frame and the starting frame, the robot's movement distance and / or turning angle relative to the starting frame in the current frame can be determined. If the movement distance exceeds a preset distance threshold (e.g., 1.5 meters) and / or the turning angle exceeds a preset turning angle threshold (e.g., 30 degrees), the system will automatically extract the current frame as a keyframe.

[0024] In this invention, the threshold settings comprehensively consider the robot's motion characteristics and environmental complexity. For example, the preset distance threshold aims to ensure that keyframes are evenly distributed in space, avoiding waste of storage resources due to oversampling in local areas. The preset angle threshold is used to capture changes in environmental direction, especially in complex scenes where even small angles can cause significant changes in perspective, thus requiring the recording of new keyframes. As an optional implementation, by combining the dual conditions of movement distance and angle, the keyframe extraction mechanism can effectively reduce the generation of redundant frames while ensuring the integrity of the topology map, laying a solid foundation for subsequent multimodal data mounting and feature extraction.

[0025] In step S102, one keyframe corresponds to one node, and each node is forced to attach the RGB image of the corresponding keyframe as raw visual observation data. The main reason for adopting this design is that RGB images contain the most intuitive and richest visual information in the scene, providing a "baseline map" for subsequent cross-modal alignment, historical scene backtracking verification, and geometric feature point matching. In the semantic path reasoning stage, the keyframe RGB image, through cross-modal alignment with natural language commands, helps the system accurately understand the user's intent and anchor the target node; in the dynamic obstacle intervention stage, the keyframe RGB image is used for accessibility decisions and alternative path verification in the backtracking mechanism; in the pose closure correction stage, the keyframe RGB image achieves high-precision pose correction through fine-grained geometric feature point matching. Therefore, attaching the keyframe RGB image is not only a core component of the multimodal semantic topology memory map, but also an important foundation for ensuring the smooth execution of subsequent navigation tasks.

[0026] To achieve efficient processing of large-scale environmental data, the system utilizes a visual foundation model (such as DINOv2 or CLIP (Contrastive Language-Image Pre-training)) to encode keyframe RGB images to extract high-dimensional semantic feature vectors. Specifically, the visual foundation model maps the keyframe RGB images to a fixed-dimensional semantic feature space to generate compact and expressive semantic feature vectors. These semantic feature vectors preserve the visual details of the images while achieving effective representation of the high-dimensional semantic space, providing a data foundation for subsequent cross-modal alignment operations.

[0027] Among them, the visual foundation model is pre-trained on a large number of image-text pairs through self-supervised learning or contrastive learning, thus possessing powerful semantic understanding capabilities. For example, the widely used CLIP model's image encoder can extract fine-grained features of images and map them into a semantic space shared with text embeddings, thereby achieving accurate alignment of images and text in a unified semantic space.

[0028] Furthermore, the system projects the depth information of keyframes onto a two-dimensional plane to construct a local 2D occupancy grid. This local 2D occupancy grid not only intuitively reflects the distribution of obstacles and geometric access information around nodes, but also possesses high computational efficiency, making it perform excellently in real-time navigation tasks. During the local path planning phase, the system utilizes this local 2D occupancy grid for collision detection and feasible path search, ensuring the robot moves safely and efficiently in complex environments (such as indoor environments). The specific generation process is as follows: First, based on this depth information, the specific distance between each pixel in the RGB image and the camera of the acquisition device is determined. Then, by combining the horizontal and vertical coordinates of each pixel and the preset camera intrinsic parameters (used to transform from the image coordinate system to the camera coordinate system), the three-dimensional coordinates of each pixel in the camera coordinate system are obtained.

[0029] Then, based on the preset camera extrinsic parameters, these three-dimensional coordinates are projected onto the robot coordinate system centered on the robot body, obtaining the three-dimensional coordinates of each pixel in the robot coordinate system. This operation can eliminate the spatial offset caused by the camera installation position, enabling the system to accurately represent the actual spatial position of the obstacle relative to the robot body (e.g., "the obstacle is 1.5 meters in front of the robot"), rather than just the local position relative to the camera lens (e.g., "1 meter in front of the camera lens").

[0030] Finally, a two-dimensional grid is constructed in the robot coordinate system. The three-dimensional coordinates of each pixel in the robot coordinate system are projected onto the two-dimensional grid. The two-dimensional grid is divided into multiple sub-grids with a preset resolution (e.g., 0.1 m / pixel). The occupancy status of each sub-grid is determined according to whether there is a projection point in each sub-grid. If there is an obstacle, it is marked as occupied (or 1); otherwise, it is marked as idle (or 0).

[0031] For steps S103 and S104, the RGB image and semantic feature vector of the keyframe, the local 2D occupancy grid map (which may also include depth information), and the robot's pose information (the camera's global absolute pose) at the time of keyframe acquisition are fused as attributes of the corresponding node for that keyframe. Finally, the nodes are concatenated according to the keyframe acquisition sequence to obtain a multimodal semantic topological memory map. The overall process is described in [link to documentation]. Figure 2 As shown.

[0032] In one optional implementation, the system also utilizes deep learning models (such as YOLO (You Only LookOnce, a single-stage real-time object detection algorithm) and other techniques known to those skilled in the art) to identify semantically meaningful targets (such as pedestrians, vehicles, traffic signs, etc.) in the RGB images of each keyframe, and generates 2D bounding boxes and category labels with confidence scores for each semantic target, thereby providing data for subsequent determination of alternative paths. Therefore, each node in the multimodal semantic topological memory map also encompasses the semantic targets in the RGB images and their two-dimensional bounding boxes and category labels.

[0033] It is understood that the specific values ​​listed in the above hypothetical description of robot navigation based on multimodal semantic topological memory maps are merely illustrative examples and not restrictive. In practice, other values ​​can be set according to requirements. Similarly, the specific values ​​listed in the subsequent embodiments are also just examples and are not the only limitation.

[0034] The method provided in the above embodiments constructs a multimodal semantic topological memory map with complete attributes by fusing multimodal data such as visual semantics (RGB images, semantic feature vectors, semantic targets) and spatial geometry (depth information, local 2D raster images, pose information). This not only provides data support for subsequent cross-modal semantic retrieval, high-precision visual positioning and dynamic obstacle avoidance, but also significantly improves the robot's environmental understanding and autonomous navigation capabilities in complex scenarios.

[0035] See Figure 3 The diagram illustrates a robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention, including the following steps: S301: Receive natural language instructions issued to the robot and encode the natural language instructions into a text embedding vector; S302: Obtain the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node; S303: Determine the current node of the robot, and based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map, plan a global navigation path from the current node to the target node, and control the robot to perform navigation operations based on the global navigation path.

[0036] In the above implementation, for step S301, when the system receives a natural language command issued by the user to the robot, the semantic reasoning and target anchoring process is initiated. First, the natural language command is input into a locally deployed lightweight parsing model (such as an LLM (Large Language Model)). The LLM performs syntactic and semantic analysis on the natural language command to extract key information and understand its contextual meaning, thereby obtaining structured information. Then, the structured information is encoded into a text embedding vector.

[0037] For example, when a user issues the command "Go to the third floor and find the rest area with the red sofa", the LLM recognizes the verbs (such as "go" and "find"), nouns (such as "third floor", "red sofa", and "rest area"), and modification relationships (such as "with red sofa" modifying "rest area"), thereby converting the natural language command into structured information.

[0038] Furthermore, LLM possesses a certain degree of contextual reasoning capability, enabling it to handle ambiguous or incomplete instructions in complex scenarios, thereby enhancing the system's robustness and usability. Normally, user-issued natural language instructions are valid. However, if LLM fails to identify key information or understand its contextual meaning, it sequentially traces back the user's historical natural language instructions, combining them with the current instruction for re-analysis of syntax and semantics until the key information is extracted and its contextual meaning is understood.

[0039] For step S302, while parsing the natural language instructions, the system traverses the semantic feature vectors of each node in the multimodal semantic topological memory map. After completing the parsing of the natural language instructions and the extraction of semantic feature vectors, the system enters the image-text cross-modal similarity calculation stage. The cosine similarity between the text embedding vector and the semantic feature vector of each node is calculated through LLM to measure the semantic relevance between the text and the image. Finally, the node with the highest semantic relevance is taken as the target node (i.e., the location that best meets the requirements of the instruction).

[0040] By calculating semantic relevance, this invention can directly capture the degree of matching between detailed descriptions in natural language instructions (such as "red sofa") and the content of RGB images. Compared to relying solely on abstract semantic labels, this invention directly utilizes the semantic feature vectors of RGB images for cross-modal alignment, which can more accurately reflect visual features in the environment, thereby effectively avoiding matching errors introduced by oversimplification of semantic labels. Experiments have verified that this mechanism exhibits significant robustness in complex scenarios, especially in tasks requiring the identification of specific targets (such as "red sofa"), significantly improving the anchoring accuracy of target nodes.

[0041] For step S303, after determining the target node, the system determines the robot's current node in the multimodal semantic topology memory map based on the robot's current location information. For example, based on the robot's hardware information such as GPS (Global Positioning System), the system determines its current location and selects the node in the multimodal semantic topology memory map whose pose information is closest to or matches that location information as the current node. Alternatively, it can use... Figure 5 The operation shown is not restricted.

[0042] Through path planning algorithms, such as A The algorithm, or other algorithms known to those skilled in the art, plans a global navigation path from the current node to the target node. This global navigation path is a "concept path" consisting of a series of nodes, each node representing a key location with clear semantic information, such as a "corridor intersection" or a "meeting room entrance," thereby providing reliable global guidance for subsequent local navigation.

[0043] During path planning, the local 2D occupancy grid of each node in the multimodal semantic topological memory map is used to analyze the connectivity and traversability between nodes, ensuring the feasibility and efficiency of the path. This method enables the robot to achieve efficient semantic navigation in complex environments, while avoiding the path planning problems caused by a lack of semantic information in traditional methods.

[0044] The method provided in the above embodiments determines the target node by encoding natural language instructions into text embedding vectors and calculating their semantic correlation with the semantic feature vectors of nodes. This achieves accurate target anchoring of user intent based on cross-modal similarity calculation of text and images. By combining the local two-dimensional occupancy grid map of each node in the multimodal semantic topological memory map for path planning, the physical accessibility and obstacle avoidance safety of the global navigation path are ensured, thereby realizing an efficient closed loop from natural language instructions to safe autonomous navigation.

[0045] See Figure 4 The diagram illustrates an optional robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention, including the following steps: S401: In response to the failure to generate a collision-free trajectory through the local planner, it is determined that the path ahead is blocked, and the previous node in the global navigation path that is before the robot's latest current node is determined, so as to determine an alternative path based on the color image of the previous node. S402: Control the robot to retreat from the latest current node to the previous node, then acquire a real-time color image from the robot's current perspective, and perform feature matching between the real-time color image and the color image of the previous node; S403: In response to the inconsistency in the matching result, continue backtracking to the previous node to perform backtracking and matching verification operations until a passable alternative path is determined. Control the robot to continue performing navigation operations based on the determined alternative path.

[0046] For steps S401 and S402, in complex dynamic environments, the robot may encounter sudden obstacles during navigation, rendering the originally planned global navigation path impassable. To address this issue, this invention provides an obstruction state determination mechanism based on local planner feedback. During the robot's navigation operation based on the global navigation path, the system determines the obstruction state of the path ahead based on feedback from the local planner. When the local planner fails to generate a collision-free trajectory within a preset time window (e.g., 500 milliseconds), the system determines that the path ahead is obstructed and triggers a backtracking mechanism.

[0047] First, the system re-determines the robot's current node based on its current location information. Then, based on the strong relationships between nodes in the multimodal semantic topological memory map, it identifies the previous node in the originally planned global navigation path that precedes the current node and acquires the RGB image of the keyframe attached to that previous node. Here, the RGB image records environmental information collected during the robot's previous exploration, providing a reference for alternative path planning.

[0048] When the path ahead is blocked, the system utilizes visual language models (such as VLM (Vision-Language Model) and GPT4 (Generative Pre-trained Transformer 4)) to deeply analyze RGB images to identify potential alternative paths from historical observations. This method integrates the visual features of RGB images with the semantic understanding capabilities of natural language processing, significantly enhancing the accuracy and robustness of scene parsing through cross-modal information complementarity, and effectively improving the robot's adaptive navigation capabilities in dynamic and complex environments.

[0049] Here, VLM integrates the capabilities of a visual encoder and a large language model, enabling it to extract fine-grained semantic information from RGB images and describe it in natural language. For example, for an RGB image containing objects such as corridors, tables, and chairs, VLM can generate a description such as "there is an empty corridor on the left and tables and chairs on the right." By analyzing this natural language description, the system can infer which directions had previously been passable during historical observations and, based on this, decide on alternative exploration directions for retrospective exploration. In an alternative implementation, the process of identifying alternative paths may also consider semantic targets and their two-dimensional bounding boxes and category labels.

[0050] For step S403, to ensure the feasibility of the alternative path, this invention provides a visual consistency verification mechanism based on fine-grained feature matching. After controlling the robot to retreat to the previous node before the current node, the system controls the robot to make tentative movements according to the previously determined alternative path. During this process, a real-time RGB image of the robot's current viewpoint is acquired and high-precision feature matching is performed with the RGB image of the previous node.

[0051] Specifically, this invention preferably uses the SuperPoint algorithm to extract local feature points in the RGB image, and then uses the LightGlue method to match the local feature points in the real-time RGB image with the local feature points in the previous node's RGB image. If the matching result indicates success or consistency, it means that the current environment and the historical observation environment are consistent, and the alternative path is still open and passable. In this case, the robot is controlled to continue performing navigation operations along the alternative path. Conversely, if the matching result indicates failure or inconsistency, it means that the alternative path may have been blocked by new obstacles. The system will abandon the path and try other alternative paths or continue backtracking.

[0052] The method provided in the above embodiments involves the system monitoring the obstruction status of the path ahead in real time through a local planner during robot navigation. When an obstruction is detected, the system backtracks to the previous node and determines an alternative path based on historical observation data such as the RGB image of the previous node. Subsequently, a verification mechanism is used to determine whether the alternative path is traversable. This method enables the robot to quickly and accurately select safe and traversable alternative paths in complex environments, thereby effectively addressing the challenges posed by dynamic obstacles and ensuring the continuity of the robot's navigation task.

[0053] See Figure 5 The diagram illustrates another optional robot navigation method based on a multimodal semantic topological memory map according to an embodiment of the present invention, including the following steps: S501: Real-time environmental data is collected by a data acquisition device mounted on the robot, and the real-time color image in the real-time environmental data is encoded to obtain a real-time semantic feature vector. S502: Determine the similarity between the real-time semantic feature vector and the semantic feature vector of each node in the multimodal semantic topology memory map, and select multiple candidate nodes based on the similarity. S503: Perform geometric feature point matching on the real-time color image and the color images of each candidate node to determine the number of successfully matched geometric feature points and / or the matching ratio. S504: In response to the quantity exceeding a preset quantity threshold and / or the matching ratio exceeding a preset ratio threshold, the corresponding candidate node is determined as the current node of the robot. S505: Based on the geometric feature point matching relationship between the real-time color image and the color image of the current node, determine the matching point pair; S506: Determine the homography matrix or the essential matrix based on the matching point pairs; estimate the pose deviation parameters based on the homography matrix or the essential matrix, so as to use the pose deviation parameters to correct the robot's current pose information.

[0054] During robot navigation, the cumulative drift of the odometry leads to a gradual increase in pose estimation error. To address this issue, this invention provides a two-level verification mechanism of "coarse screening + fine verification" to correct the robot's pose through closed-loop detection. In step S501, during navigation, the robot continuously collects real-time environmental data from its current perspective using onboard acquisition devices (such as an RGB-D camera), and encodes the real-time RGB images using a visual basic model (such as DINOv2 or CLIP) to extract high-dimensional real-time semantic feature vectors. These semantic feature vectors have low computational complexity and storage requirements, while effectively capturing semantic information in the scene, such as furniture layout and wall texture.

[0055] In step S502, the system performs a fast cosine similarity calculation on the real-time semantic feature vector and the semantic feature vectors of each node in the multimodal semantic topological memory map to measure the semantic similarity between the two. Then, based on the cosine similarity from highest to lowest, the system sequentially selects the top K candidate nodes, which are the nodes most similar to the current perspective. Alternatively, a similarity threshold can be pre-set, and nodes exceeding this threshold are considered candidate nodes. This "coarse screening" mechanism can select the K most similar candidate nodes in a very short time, significantly reducing computational complexity and quickly locating potential closed-loop nodes in complex environments, providing high-quality initial screening results for subsequent fine-tuning.

[0056] For steps S503 and S504, after completing the global coarse screening based on semantic feature vectors, a geometric refinement mechanism using keyframe RGB images is needed to further verify whether these K candidate nodes truly correspond to the robot's current position. Specifically, the system reads the keyframe RGB images attached to these K candidate nodes from the multimodal semantic topological memory map, and then performs geometric feature point matching between the real-time RGB image and the RGB image of each candidate node. Here, classic algorithms such as ORB (Oriented FAST and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform) can be used to extract feature points from the two RGB images and calculate descriptors. Then, matching algorithms such as brute-force matching or FLANN (Fast Library for Approximate Nearest Neighbors) are used to determine the matching relationship between the two sets of feature points.

[0057] Compared to algorithms like LightGlue, the algorithm used here has a finer-grained local feature matching capability, which is more suitable for the current geometric accuracy requirements. To further reduce the impact of mismatches, the system can also use RANSAC (Random Sample Consensus) to optimize the matching results and eliminate outliers. If the number of successfully matched feature points exceeds a preset threshold, or the matching ratio exceeds a preset ratio threshold, the system determines it as a "successful revisit," that is, it determines that the candidate node is the node where the robot is currently located.

[0058] Through a geometric verification mechanism, the system can accurately estimate the geometric transformation relationship between the real-time RGB image and the RGB images of each candidate node, thereby determining whether the two RGB images originate from the same physical location, such as a rest area or the same room. This not only improves the accuracy of loop closure detection but also effectively avoids false loop closures caused by semantic similarity but different spatial locations, thus enhancing the accuracy of determining the robot's current node.

[0059] For steps S505 and S506, after determining the robot's current node, a pose forced correction process is triggered. Specifically, based on the geometric feature point matching relationship between the real-time RGB image and the RGB image of the current node, matching point pairs are determined, and the homography matrix or essential matrix is ​​calculated based on the matching point pairs. For planar scenes, the homography matrix can directly describe the projective transformation relationship between two RGB images; while for non-planar scenes, the rotation matrix and translation vector need to be decomposed from the essential matrix in combination with camera intrinsic parameters to recover the pose transformation in three-dimensional space.

[0060] Therefore, after calculating the homography matrix or the essential matrix, the system also needs to estimate the robot's pose deviation parameters based on them, and then use these pose deviation parameters to correct the robot's current pose information so that the current pose information is aligned with the global absolute coordinates of the current node, thereby eliminating the cumulative drift error of the odometry.

[0061] Furthermore, to ensure the reliability of the correction, the system can introduce constraints. The forced pose correction process is only executed if these constraints are met. For example, the calculation results of the homography matrix or essential matrix must pass a validity check (using conventional techniques in the field, such as interior point scaling checks or reprojection error assessments). By introducing these constraints, the system can effectively avoid erroneous corrections caused by feature mismatches or environmental noise interference while ensuring correction accuracy, thereby significantly improving the long-term stability and positioning accuracy of robot navigation.

[0062] The method provided in the above embodiments adopts a two-level verification mechanism of "semantic feature vector coarse screening + RGB image geometric fine verification". While ensuring the real-time positioning, it significantly improves the repositioning accuracy in complex scenarios. Based on the matching point pairs, it calculates homography or essential matrix to estimate pose deviation parameters, thereby realizing forced correction of the robot's current pose information, effectively eliminating the cumulative error of the system odometry, and significantly enhancing the positioning accuracy and robustness of the robot's long-term navigation.

[0063] In summary, addressing the shortcomings of existing robot navigation technologies such as large storage requirements, poor adaptability to dynamic environments, and insufficient semantic understanding, this invention provides a robot navigation method based on a multimodal semantic topological memory map. This multimodal semantic topological memory map integrates semantic feature vectors from RGB images and a local two-dimensional occupancy grid map of depth information to replace traditional dense point clouds or dense feature maps, significantly reducing system storage overhead and improving environmental representation efficiency. During actual navigation, the system can not only monitor the obstruction status of the path ahead in real time and quickly plan alternative paths through a local planner, effectively coping with dynamic environmental challenges; it can also accurately estimate pose deviation parameters and perform closed-loop correction operations based on the geometric feature point matching relationship between real-time RGB images and historical node RGB images, thereby comprehensively ensuring the efficiency, safety, and long-term positioning accuracy of robot navigation.

[0064] See Figure 6 The diagram shows the main modules of a robot navigation device 600 based on a multimodal semantic topological memory map provided in an embodiment of the present invention, including: The encoding module 601 is used to receive natural language instructions issued to the robot and encode the natural language instructions into a text embedding vector; The filtering module 602 is used to obtain the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node. The planning module 603 is used to determine the current node of the robot and plan a global navigation path from the current node to the target node based on the local two-dimensional occupancy grid map of each node in the multimodal semantic topology memory map. The control module 604 is used to control the robot to perform navigation operations based on the global navigation path.

[0065] The apparatus for implementing this invention further includes a construction module for constructing the multimodal semantic topological memory map, the construction process of which includes: Environmental data for keyframes is acquired using a data acquisition device mounted on the robot; the environmental data includes color images and depth information. The color image is encoded to obtain a semantic feature vector; and a local two-dimensional occupancy raster map is constructed based on the depth information; The color image, the semantic feature vector, the local two-dimensional occupancy grid map, and the robot's pose information when the key frame is acquired are fused together to serve as the attributes of the node corresponding to the key frame. By concatenating the nodes according to the acquisition sequence of the keyframes, the multimodal semantic topology memory map is obtained.

[0066] In the apparatus of this invention, the construction module is used for: Collect environmental data from consecutive frames, and determine the robot's movement distance and / or rotation angle in the current frame relative to the starting frame based on the environmental data of the current frame and the environmental data of the starting frame. In response to the movement distance exceeding a preset distance threshold and / or the turning angle value exceeding a preset turning angle threshold, the current frame is determined as a key frame.

[0067] In the apparatus of this invention, the construction module is used for: Based on the depth information, the distance between each pixel in the color image and the acquisition device is determined. Combining the horizontal and vertical coordinates of each pixel in the color image and the preset camera intrinsic parameters, the three-dimensional coordinates of each pixel in the camera coordinate system are obtained. Based on the preset camera extrinsic parameters, each 3D coordinate is projected onto the robot coordinate system to obtain the 3D coordinates of each pixel in the robot coordinate system. A two-dimensional grid is constructed in the robot coordinate system, and the three-dimensional coordinates of each pixel in the robot coordinate system are projected onto the two-dimensional grid. The two-dimensional grid is divided into multiple sub-grids according to a preset resolution. The occupancy status of each sub-grid is determined based on whether there is a projection point in each sub-grid, so as to construct a local two-dimensional occupancy grid map.

[0068] The apparatus for implementing this invention further includes a rollback substitution module, used for: In response to the failure to generate a collision-free trajectory through the local planner, it is determined that the path ahead is blocked. The previous node in the global navigation path that is located before the robot's latest current node is determined, and an alternative path is determined based on the color image of the previous node. The robot is controlled to regress from the latest current node to the previous node, and then a real-time color image of the robot's current view is acquired. The real-time color image is then matched with the color image of the previous node. In response to inconsistent matching results, the robot continues to backtrack to earlier nodes for backtracking and matching verification until a viable alternative path is determined. The robot then continues navigation based on the determined alternative path.

[0069] In the apparatus of this invention, the planning module 603 is used for: Real-time environmental data is collected by a data acquisition device mounted on a robot, and the real-time color image in the real-time environmental data is encoded to obtain a real-time semantic feature vector. Determine the similarity between the real-time semantic feature vector and the semantic feature vector of each node in the multimodal semantic topology memory map, and filter out multiple candidate nodes based on the similarity. Geometric feature point matching is performed on the real-time color image and the color images of each candidate node to determine the number of successfully matched geometric feature points and / or the matching ratio. In response to the number exceeding a preset number threshold and / or the matching ratio exceeding a preset ratio threshold, the corresponding candidate node is determined as the current node of the robot.

[0070] The apparatus of the present invention further includes a calibration module, used for: Based on the geometric feature point matching relationship between the real-time color image and the color image of the current node, a matching point pair is determined; Determine the homography matrix or essential matrix based on the matching point pairs; Based on the homography matrix or the essential matrix, pose deviation parameters are estimated to correct the robot's current pose information.

[0071] In the apparatus of this invention, the encoding module 601 is used for: The natural language instructions are input into a preset parsing model for syntactic and semantic analysis to extract key information and contextual meaning, and obtain structured information to be encoded as text embedding vectors. When the preset parsing model fails to extract key information or contextual meaning, the historical natural language instructions issued are traced back in sequence, and the syntactic and semantic analyses are re-performed in combination with the natural language instructions until the key information and contextual semantics are successfully extracted.

[0072] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0073] Figure 7 An exemplary system architecture 700 to which embodiments of the present invention can be applied is shown, including terminal devices 701, 702, 703, network 704, and server 705 (only an example).

[0074] Terminal devices 701, 702, and 703 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc.

[0075] Network 704 is a medium used to provide a communication link between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0076] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (this is just an example). The backend management server can analyze and process received data such as product information query requests, and feed back the processing results (e.g., target push information, product information—this is just an example) to the terminal devices. It should be noted that the method provided in this embodiment of the invention is generally executed by server 705, and correspondingly, the apparatus is generally set in server 705.

[0077] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0078] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0079] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0080] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0081] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0082] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including an encoding module, a filtering module, a planning module, and a control module. The names of these modules do not necessarily limit the module itself; for example, a filtering module may also be described as a "computation module."

[0085] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the above-described robot navigation methods based on multimodal semantic topological memory maps.

[0086] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the robot navigation method based on a multimodal semantic topological memory map in the embodiments of the present invention.

[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A robot navigation method based on a multimodal semantic topological memory map, characterized in that, include: Receive natural language instructions issued to the robot and encode the natural language instructions into text embedding vectors; Obtain the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node; The robot's current node is determined, and a global navigation path from the current node to the target node is planned based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map. The robot is then controlled to perform navigation operations based on the global navigation path.

2. The method according to claim 1, characterized in that, The method also includes a process of constructing the multimodal semantic topological memory map, including: Environmental data for keyframes is acquired using a data acquisition device mounted on the robot; the environmental data includes color images and depth information. The color image is encoded to obtain a semantic feature vector; and a local two-dimensional occupancy raster map is constructed based on the depth information; The color image, the semantic feature vector, the local two-dimensional occupancy grid map, and the robot's pose information when the key frame is acquired are fused together to serve as the attributes of the node corresponding to the key frame. By concatenating the nodes according to the acquisition sequence of the keyframes, the multimodal semantic topological memory map is obtained.

3. The method according to claim 2, characterized in that, The environmental data of the acquired keyframes includes: Collect environmental data from consecutive frames, and determine the robot's movement distance and / or rotation angle in the current frame relative to the starting frame based on the environmental data of the current frame and the environmental data of the starting frame. In response to the movement distance exceeding a preset distance threshold and / or the turning angle value exceeding a preset turning angle threshold, the current frame is determined as a key frame.

4. The method according to claim 2 or 3, characterized in that, The construction of a local two-dimensional occupancy grid map based on the depth information includes: Based on the depth information, the distance between each pixel in the color image and the acquisition device is determined. Combining the horizontal and vertical coordinates of each pixel in the color image and the preset camera intrinsic parameters, the three-dimensional coordinates of each pixel in the camera coordinate system are obtained. Based on the preset camera extrinsic parameters, each 3D coordinate is projected onto the robot coordinate system to obtain the 3D coordinates of each pixel in the robot coordinate system. A two-dimensional grid is constructed in the robot coordinate system, and the three-dimensional coordinates of each pixel in the robot coordinate system are projected onto the two-dimensional grid. The two-dimensional grid is divided into multiple sub-grids according to a preset resolution. The occupancy status of each sub-grid is determined based on whether there is a projection point in each sub-grid, so as to construct a local two-dimensional occupancy grid map.

5. The method according to claim 2, characterized in that, During the process of the controlled robot performing navigation operations based on the global navigation path, the method further includes: In response to the failure to generate a collision-free trajectory through the local planner, it is determined that the path ahead is blocked. The previous node in the global navigation path that is located before the robot's latest current node is determined, and an alternative path is determined based on the color image of the previous node. The robot is controlled to regress from the latest current node to the previous node, and then a real-time color image of the robot's current view is acquired. The real-time color image is then matched with the color image of the previous node. In response to inconsistent matching results, the robot continues to backtrack to earlier nodes for backtracking and matching verification until a viable alternative path is determined. The robot then continues navigation based on the determined alternative path.

6. The method according to claim 1 or 5, characterized in that, Determining the current node where the robot is located includes: Real-time environmental data is collected by a data acquisition device mounted on a robot, and the real-time color image in the real-time environmental data is encoded to obtain a real-time semantic feature vector. Determine the similarity between the real-time semantic feature vector and the semantic feature vector of each node in the multimodal semantic topology memory map, and filter out multiple candidate nodes based on the similarity. Geometric feature point matching is performed on the real-time color image and the color images of each candidate node to determine the number of successfully matched geometric feature points and / or the matching ratio. In response to the number exceeding a preset number threshold and / or the matching ratio exceeding a preset ratio threshold, the corresponding candidate node is determined as the current node of the robot.

7. The method according to claim 6, characterized in that, After determining the corresponding candidate node as the current node of the robot, the method further includes: Based on the geometric feature point matching relationship between the real-time color image and the color image of the current node, a matching point pair is determined; Determine the homography matrix or essential matrix based on the matching point pairs; Based on the homography matrix or the essential matrix, pose deviation parameters are estimated to correct the robot's current pose information.

8. The method according to claim 1, characterized in that, Encoding the natural language instructions into text embedding vectors includes: The natural language instructions are input into a preset parsing model for syntactic and semantic analysis to extract key information and contextual meaning, and obtain structured information to be encoded as text embedding vectors. When the preset parsing model fails to extract key information or contextual meaning, the historical natural language instructions issued are traced back in sequence, and the syntactic and semantic analyses are re-performed in combination with the natural language instructions until the key information and contextual semantics are successfully extracted.

9. A robot navigation device based on a multimodal semantic topological memory map, characterized in that, include: The encoding module is used to receive natural language instructions issued to the robot and encode the natural language instructions into text embedding vectors; The filtering module is used to obtain the semantic feature vectors of each node in the multimodal semantic topology memory map, determine the semantic correlation between the text embedding vector and each semantic feature vector, and determine the node with the highest semantic correlation as the target node. The planning module is used to determine the current node of the robot and plan a global navigation path from the current node to the target node based on the local two-dimensional occupancy grid of each node in the multimodal semantic topology memory map. The control module is used to control the robot to perform navigation operations based on the global navigation path.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images