Autonomous vision semantic navigation system and method based on interactive semantic mapping

Abstract

The invention discloses an autonomous visual semantic navigation system and method based on interactive semantic mapping, a semantic scene graph is constructed by sensing the environment through a first visual angle of a robot and combining top-down environment semantic priori, and the method comprises regional spatial distribution identification, object attribute and layout analysis and room type clustering. And in combination with priori knowledge, complete scene representation is formed, the scene graph can be dynamically adapted and updated according to real-time observation of the robot, and the accuracy and timeliness of scene representation are ensured. Besides, a high-level and low-level AI agent module cooperation mechanism is adopted, and the high-level AI agent module is responsible for understanding and decomposing navigation tasks and distributing subtasks and supervising the execution process; the low-layer AI agent module is responsible for further dynamic decomposition and execution of the subtasks and feeds back execution results to the high-layer Agent, a complete task closed-loop monitoring mechanism is formed, and the execution efficiency and robustness of the navigation tasks are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual language navigation, and particularly relates to an autonomous visual semantic navigation system and method based on interactive semantic mapping. Background Art

[0002] Visual Language Navigation (VLN) aims to enable an agent to autonomously navigate in a complex visual environment by understanding natural language instructions. The current technologies mainly rely on the following two methods:

[0003] Geometric mapping and path planning methods: construct a geometric environment map based on sensors such as lidar and cameras, and complete path planning through path search algorithms (such as A* algorithm or Dijkstra algorithm). Such methods rely on accurate geometric mapping, have poor adaptability to dynamic environments, and are not conducive to the robot's understanding of natural language.

[0004] End-to-end navigation methods based on deep learning: use deep learning models to directly output navigation paths from raw sensor data. Such methods can learn path planning in complex scenarios to a certain extent, but have high requirements for the quality of input data, and have deficiencies in interpretability and generalization ability.

[0005] In addition, semantic navigation systems have gradually received attention, and enhance navigation capabilities through semantic perception technologies (such as object detection or semantic segmentation). However, the existing semantic navigation methods have limited semantic understanding and reasoning capabilities, insufficient support for complex natural language instructions or multi-task scenarios, and low generalization ability and deployment efficiency, making it difficult to navigate efficiently in unknown environments, and the actual deployment and scalability are limited. Summary of the Invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides an autonomous visual semantic navigation system and method based on interactive semantic mapping.

[0007] The technical problems to be solved by the present invention are realized through the following technical solutions:

[0008] The present invention provides an autonomous visual semantic navigation system based on interactive semantic mapping, which is applied to a robot and includes:

[0009] The instruction parsing and planning module is used to parse the obtained natural language navigation instructions of the user into structured task instructions, obtain the task-related sub-graphs from the scene graph according to the structured task instructions, and obtain the environmental information currently observed by the robot, and send the structured task instructions, the sub-graphs and the environmental information to the high-level AI agent module; the scene graph is a semantic scene graph constructed from the perspective of the robot and composed of nodes and edges, including a scene layer, a room layer, a functional area layer, a key object layer and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects;

[0010] The high-level AI agent module is used to generate single or at least two structured semantic subtasks arranged in sequence according to the scene graph, the structured task instructions, the sub-graphs and the environmental information, and re-obtain the environmental information and the current pose currently observed by the robot, and send the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtasks and the sub-graphs to the low-level AI agent module;

[0011] The low-level AI agent module is used to generate the physical action sequence of the robot according to the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtasks and the sub-graphs and control the movement of the robot until the subtask execution result is obtained and fed back to the high-level AI agent module;

[0012] The high-level AI agent module is further used to, when the subtask execution result indicates successful execution, continue to determine the next to-be-executed structured semantic subtask, and obtain the environmental information and pose currently observed by the robot and then send them to the low-level AI agent module, and so on, until all the structured semantic subtasks are executed.

[0013] The present invention also provides an autonomous visual semantic navigation method based on interactive semantic mapping, which is applied to a robot and includes:

[0014] Obtain the natural language navigation instructions of the user;

[0015] Parse the natural language navigation instructions into structured task instructions;

[0016] Obtain the task-related sub-graphs from the scene graph according to the structured task instructions, and obtain the environmental information currently observed by the robot; wherein, the scene graph is a multi-level heterogeneous semantic scene graph constructed from the perspective of the robot and composed of nodes and edges, and the scene graph includes a global scene layer, a room layer, a functional area layer, a key object layer and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects;

[0017] Generate one or at least two structured semantic subtasks arranged in order according to the scene graph, the structured task instruction, the sub-graph, and the environmental information, and re-obtain the environmental information and the current pose currently observed by the robot;

[0018] Generate a physical action sequence of the robot and control the movement of the robot according to the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtask, and the sub-graph until a subtask execution result is obtained;

[0019] When the subtask execution result indicates successful execution, continue to obtain the environmental information and pose currently observed by the robot, and continue to generate the physical action sequence of the robot and continue to control the movement of the robot according to the continuously obtained environmental information and pose, the next to-be-executed structured semantic subtask, and the sub-graph. In this way, continue until all structured semantic subtasks are executed.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1) The present invention constructs a multi-level and language-interpretable indoor scene graph (RC-ISG) from the perspective of the robot center, breaking through the semantic deviation problem of the traditional third-person perspective scene graph. By organically integrating the relative position relationships among spatial topologies, functional regions, and key objects, the system can more accurately reflect what the robot actually sees, thereby improving the accuracy and execution efficiency of navigation decisions.

[0022] 2) The present invention proposes a dual-agent framework that combines a high-level semantic planning agent (i.e., the high-level AI agent module) and a low-level execution control agent (i.e., the low-level AI agent module). The high-level semantic planning agent is based on the indoor scene graph and environmental perception information and is responsible for decomposing complex tasks into structured semantic sub-goals, while the low-level execution control agent performs specific action execution and path planning based on real-time perception data. This hierarchical design not only reduces the task complexity but also significantly improves the adaptability and task success rate of the system in a dynamic environment.

[0023] 3) While retaining necessary geometric details, the indoor scene graph of the present invention fully improves the system's parsing ability for natural language instructions by hierarchically encoding the semantic relationships among rooms, regions, and key objects, enhances the semantic understanding and reasoning abilities, realizes an efficient mapping from abstract semantics to specific spatial positioning, and improves the accuracy and robustness of navigation decisions.

[0024] 4) The present invention utilizes the representation method from the robot's perspective and the hierarchical planning strategy, and can still maintain high navigation performance in unknown or complex environments. The system exhibits excellent advantages in reducing computational resource consumption and achieving real-time dynamic planning, has high generalization ability and deployment efficiency, and provides higher reliability and scalability for actual deployment.

[0025] The following will further elaborate on the present invention in conjunction with the accompanying drawings and specific embodiments. Description of the Drawings

[0026] Figure 1 It is a schematic structural diagram of an autonomous visual semantic navigation system based on interactive semantic mapping provided by an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of the process of an autonomous visual semantic navigation method based on interactive semantic mapping provided by an embodiment of the present invention. Detailed Embodiments

[0028] The following further describes the present invention in detail with specific embodiments, but the embodiments of the present invention are not limited thereto.

[0029] The present invention provides an autonomous visual semantic navigation system based on interactive semantic mapping. The system is applied to a robot, which is equipped with an RGB-D camera and an inertial measurement unit (IMU). The internal parameters (focal length, principal point, distortion coefficient) of the RGB-D camera and the external parameters (rotation and translation transformation matrix) relative to the robot's base coordinate system (Base Frame) are known. The RGB-D camera is used to collect RGB-D data frames, and the IMU is used for real-time attitude estimation of the robot. As Figure 1As shown in the figure, the system includes: an instruction parsing and planning module, a high-level AI agent module, and a low-level AI agent module. The instruction parsing and planning module is used to obtain the user's natural language navigation instruction, parse the natural language navigation instruction into a structured task instruction, obtain the task-related sub-graph from the scene graph according to the structured task instruction, and obtain the environmental information currently observed by the robot, and send the structured task instruction, the sub-graph, and the environmental information to the high-level AI agent module. The high-level AI agent module is used to generate a single or at least two structured semantic sub-tasks arranged in sequence according to the scene graph, the structured task instruction, the sub-graph, and the environmental information, and re-obtain the environmental information currently observed by the robot and the current pose, and send the re-obtained environmental information and current pose, the currently to-be-executed structured semantic sub-task, and the sub-graph to the low-level AI agent module. The low-level AI agent module is used to generate a physical action sequence of the robot and control the movement of the robot according to the re-obtained environmental information, the currently to-be-executed structured semantic sub-task, the current pose, and the sub-graph until the sub-task execution result is obtained and fed back to the high-level AI agent module. The high-level AI agent module is further used to, when the sub-task execution result indicates successful execution, continue to obtain the environmental information and pose currently observed by the robot, and send the continuously obtained environmental information and pose, the next to-be-executed structured semantic sub-task, and the sub-graph to the low-level AI agent module, and so on until all the structured semantic sub-tasks are executed.

[0030] In the present invention, the scene graph is a multi-level heterogeneous semantic scene graph constructed from the perspective of the robot and composed of nodes and edges. The scene graph includes a global scene layer (i.e., the scene layer), a room layer, a functional area layer, a key object layer, and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects. The scene graph is a multi-level, heterogeneous, and semantically rich scene graph (RC-ISG) gradually constructed by using sensor data and through depth preprocessing, object detection, and scene segmentation. Among them, relative spatial relationships (such as front, back, left, and right) are used for encoding in the scene graph to ensure that the robot can still maintain an accurate spatial understanding during the movement process. In this scene graph, the global scene layer is used to describe the macroscopic layout of the entire environment, the room layer is used to divide each independent room or doorway area; the functional area layer is used to further subdivide the functional areas inside each room; the key object layer is used to identify the object nodes crucial for navigation (for example, tables, obstacles, etc.).

[0031] The scene graph includes a root node, at least two first nodes, at least two second nodes, and at least two third nodes. The first nodes are children of the root node, the second nodes are children of the first nodes, and the third nodes are children of the second nodes. There is an edge between each parent node and its child node. Among them, the root node represents the global scene layer, the root node represents the map scene, at least two first nodes represent the room layer, and at least two first nodes are used to represent at least two different rooms. At least two second nodes represent the functional area layer, and one second node represents a functional area within a room. At least two third nodes represent the key object layer, and one third node represents a key object within the functional area.

[0032] The attribute information of each node in the scene graph includes: node id, node type, node semantic label, 3D position of the node, size of the node, id of the parent node of the node, and id of the child node of the node. Among them, the node type is used to describe whether the node is a scene node, a room node, a functional area node, a door opening area node, or a key object node. The node semantic label is used to describe the semantics of the node. The 3D position of a node refers to the 3D position coordinates in the base coordinate system of the robot. The attribute information of each edge in the scene graph includes: edge id, id of the source node of the edge, id of the target node of the edge, type of the edge, and descriptor of the edge. Among them, the type of the edge is used to describe whether the edge represents an inclusion relationship, an adjacent relationship, a relative relationship, or a transition relationship. The descriptor of the edge is used to describe the relative position between the two nodes connected by this edge from the perspective of the robot. Id represents an identifier, and the node identifier of each node and the edge id of each edge are unique.

[0033] In some embodiments, the attribute information of each node and each edge further includes: a timestamp for indicating the creation time or update time.

[0034] In the present invention, for a preset scene, for example, a certain indoor scene, the scene graph of this scene is constructed by using the methods described in S1 to S8 as follows:

[0035] S1. Create a root node with the node type of scene node according to the predefined scene information, set the attribute information of the root node, and then control the robot to enter the preset scene corresponding to the predefined scene information.

[0036] Here, the predefined scene information can be the scene name of this scene, for example, xx shopping mall, xx hotel, xx restaurant, and so on.

[0037] Specifically, the node semantic label of the root node is the predefined scene information, the 3D position, size, and id of the parent node of the node are all empty, and the id of the child node of this root node is temporarily empty and will be added in the subsequent steps.

[0038] S2. Control the robot to collect images of the environment it enters, determine a panoramic image of the environment based on the collected images, and perform target detection and scene area understanding, as well as functional area understanding and segmentation on the panoramic image in combination with preset object prompt information and functional area prompt information, to obtain an object instance list and functional area division results for the panoramic image; the object instance list includes the category label, 2D position, posture and size of each detected object in the image; the functional area division result includes the geometric range of each functional area and the semantic label of each functional area.

[0039] Here, after the robot enters the central area of the environment, it performs one or more in-place rotations or small-range mobile scans, and while rotating or scanning, uses an RGB-D camera to capture images of the environment, thereby obtaining x RGB-D data frames, where x is a positive integer greater than or equal to 1. Among them, when the robot performs a 360° in-place rotation, one RGB-D data frame is obtained. This RGB-D data frame contains a panoramic view of the environment entered by the robot. When the robot performs multiple in-place rotations at small angles or multiple small-range mobile scans, two or more RGB-D data frames are obtained. The image after splicing these RGB-D data frames in the order of generation contains a panoramic view of the environment entered by the robot.

[0040] Here, when the robot collects x RGB-D data frames of the current environment, a panoramic image of the current environment can be determined through steps S11 to S15:

[0041] S11. For each of the x RGB-D data frames, synchronize and align the RGB image and the depth image contained in the RGB-D data frame to obtain a synchronized and aligned RGB image and a synchronized and aligned depth image.

[0042] Here, synchronizing and aligning the RGB image and the depth image can strictly align the RGB image and the depth image in time and space. The specific synchronization and alignment processing can be implemented by using existing software timestamp matching and image registration methods, or by using other existing methods, which will not be elaborated in the present invention.

[0043] S12, performing invalid value processing, noise filtering processing, and structural noise processing on the synchronously aligned RGB image and the synchronously aligned depth image in sequence to obtain a data-cleaned RGB image and a data-cleaned depth image.

[0044] Here, the invalid value processing can remove or mark invalid depth readings (such as specific error values returned by the sensor, values outside the range). The specific implementation of the noise filtering processing can be: applying a median filter based on neighborhood statistics or a bilateral filter considering edge preservation to remove salt-and-pepper noise and random fluctuations in the image while trying to retain the edge information of the object. The structural noise processing can filter out measurement errors or voids caused by the material of the object surface (such as high reflectivity, transparency) or sensor limitations (such as specific incident angles).

[0045] S13. Perform depth masking processing on the RGB image after data cleaning and the depth image after data cleaning respectively to obtain the processed RGB image and the processed depth image.

[0046] Here, when performing depth masking, it is necessary to define the minimum and maximum effective distances (for example, the minimum effective distance is 0.5 m and the maximum effective distance is 8 m). Then, the depth pixels outside the range formed by the minimum and maximum effective distances in the image are marked as invalid. Through depth masking, it is possible to focus on the medium and short distance spaces that the robot can perceive and interact with, remove distant irrelevant backgrounds and potential long-distance measurement errors of the sensor, and exclude the interference of too close areas (such as the robot's own components).

[0047] S14. Identify the door opening areas in the processed RGB image and the processed depth image respectively, and mask or mark the identified door opening areas to obtain the processed RGB image in the transition area and the processed depth image in the transition area. Among them, the processed RGB image in the transition area and the processed depth image in the transition area form a processed RGB-D data frame.

[0048] Here, geometric features (such as plane normal changes, rectangular contours) can be used to identify the door opening areas in the image, or object detection methods can be used to identify the door opening areas in the image. The present invention does not limit this. After identifying the door opening areas in the image, each door opening area in the image can be logically marked or masked. In this way, it is possible to prevent the information of the environment on the other side of the door from being wrongly associated with the sub-map of the current environment (for example, the current room) when constructing the scene sub-map corresponding to the image.

[0049] S15. Stitch the obtained x processed RGB-D data frames to obtain a panoramic image of the environment.

[0050] Specifically, after the x RGB-D data frames are preprocessed through the above steps, x processed RGB-D data frames can be correspondingly obtained. By stitching these x processed RGB-D data frames in the shooting order, the panoramic image of the current environment can be obtained.

[0051] Here, after obtaining the panoramic image of the current environment where the robot is located, the panoramic image and the preset object prompt information are input into the Grounding DINO model. The Grounding DINO model is used to detect and locate the objects contained in the panoramic image, and an object instance list is obtained. Among them, the preset object prompt information is used to prompt the Grounding DINO model about the objects that need to be detected and located in the panoramic image. For example, the preset object prompt information can be a text-based prompt, and it can be the category of a specific object, such as "table", "red chair", or it can be a more generalized description, such as "any furniture", "all obstacles". This model utilizes its powerful vision-language alignment ability to locate (output 2D bounding boxes or masks) object instances in the image that match the semantics of the text prompt, and generates an object instance list. Each instance contains: the object category label, the 2D position in the image (i.e., the box or mask), and the 3D position (e.g., the average value of all valid depth points within the mask or the 3D coordinate point obtained by back-projecting the centroid) and size in the robot base coordinate system calculated using the corresponding depth information.

[0052] Here, after obtaining the panoramic image of the current environment where the robot is located, the panoramic image and the preset functional area prompt information are also input into the Vision-Language Model (VLM). The VLM is used to understand and segment the functional areas in the panoramic image, and a functional area division result is obtained based on the output of the VLM. Among them, the preset functional area prompt information is used to prompt the VLM about the functional areas that need to be understood and divided in the panoramic image. The functional area prompt information is in text form. For example, it can be "Identify the kitchen area in this image.", "Where is the main working space?". The output of the VLM is in a structured form, such as directly outputting the bounding boxes and key point coordinates indicating the ranges of each area. According to the bounding boxes and key point coordinates of each area in the output of the VLM, the geometric range of each functional area that needs to be recognized from the panoramic image and the semantic label indicating what specific function the area is can be determined.

[0053] S3. When it is determined that the environment entered by the robot is a room, create a node with the node type of room node, connect it to the root node using an edge representing the containment relationship, and set the attribute information of the room node and the edge.

[0054] Here, when the robot enters an environment and obtains the list of object instances and / or the result of functional area division of the panoramic image of the environment, it can determine whether the environment is a room based on prior information (e.g., the architectural floor plan of the preset scenario), or it can determine whether the environment is a room based on the obtained list of object instances and / or the result of functional area division and the landmark objects in the room (e.g., bed, wardrobe, etc.). When the environment is a room, a node of the room node type is created, and an edge representing the containment relationship is used to connect it to the root node. Then, a node id and an edge id are created for the room node and this edge respectively. The node id of the room node is used as the child node id of the root node, the node id of the root node is used as the source node id of this edge, the node id of the room node is used as the target node id of this edge. The type of this edge is set to the containment relationship, the descriptor of this edge is set to be empty. Then, the node semantic label of the room node is set to "Room 1", representing the first room, or the node semantic label of the room node is set to the name of this room in the prior information (e.g., the architectural floor plan of the preset scenario). The 3D position and the size of the node of the room node are set to be empty, the parent node id of the room node is set to the root node id, and the child node id of the room node is temporarily set to be empty for adding as needed in subsequent steps.

[0055] S4. Create corresponding number of nodes of the functional area node type according to the number of functional areas included in the result of functional area division, set the attribute information for each of the created corresponding number of nodes, use the corresponding number of nodes as the child nodes of the room node, and connect them to the room node with an edge representing the containment relationship and set the attribute information of the edge.

[0056] Here, when the result of functional area division includes n functional areas, n nodes of the functional area node type are created, where n is a positive integer greater than or equal to 1. A node id is generated for each functional area node, and the obtained n node ids are used as the child node ids of the room node. A node semantic label is set for each functional area node to indicate which functional area each functional area node represents (e.g., "Resting Area"). Then, the 3D position of each functional area node is set to be empty, the node size of each functional area node is set to the geometric range of the functional area represented by this functional area node. The node id of the room node is set as the parent node id of each functional area node, and the child node id of each functional area node is temporarily set to be empty for adding as needed in subsequent steps. Then, these n functional area nodes are all connected to the room node with n edges. Then, the types of these n edges are all set to the containment relationship, and the descriptors of the edges are all set to be empty.

[0057] S5. According to the object instance list, the predefined key object categories, and the functional area division result, determine whether there are key objects in each functional area and the number of key objects present. When there are key objects in a functional area, create nodes of the key object node type with the corresponding number, set the attribute information of each key object node, then use edges representing the inclusion relationship to connect each key object node to the corresponding functional area node, set the attribute information of the edges, and when a functional area node has two or more key object nodes, determine the directed edges between every two key object nodes of the functional area node and the attribute information of the directed edges.

[0058] Here, after establishing the attribute information of the functional area nodes and their related edges, use the 3D positions of the objects in the object instance list and the geometric ranges of each functional area in the functional area division result to determine which functional area each object in the object instance list is located in, and then combine the predefined key object categories to identify whether the objects in each functional area belong to the key objects in the predefined key object categories. Exemplarily, the key object can be furniture, fixed obstacles, etc. When there are m key objects in a functional area, create m nodes of the key object node type, generate node IDs for each key object node, and use the obtained m node IDs as the m child node IDs of the functional area node corresponding to the functional area where they are located. Set a node semantic label for each key object node to indicate the object category represented by the key object node (for example, wardrobe), then set the 3D position of each key object node to the corresponding 3D position recorded in the object instance list, set the node size of each key object node to the corresponding size recorded in the object instance list, use the node ID of the functional area node corresponding to the functional area where these m key objects are located as the parent node ID of each key object node, and temporarily set the child node IDs of each key object node to be empty for adding as needed in subsequent steps. Then, use m edges to connect these m key object nodes to the above-mentioned functional area node; then, set the types of these m edges to the inclusion relationship and the descriptors of the edges to be empty. Next, judge the number of key objects in each functional area where key objects exist. When a functional area node has two or more key object nodes, set directed edges between every two key object nodes of the functional area node, determine the relative positions between every two key object nodes, use the relative positions as the descriptors of the directed edges, and use the adjacent relationship as the type of the directed edges.

[0059] Specifically, for any two key object nodes in a functional area node, the specific principle for determining the directed edge between these two key object nodes and the attribute information of this directed edge is as follows: Take one of the two key object nodes as the first node and the other as the second node; According to the 3D positions of the first node and the second node, calculate the 3D vector from the first node to the second node, and project this 3D vector onto the horizontal plane to obtain a 2D direction vector, where the horizontal plane is a two-dimensional plane perpendicular to the gravity direction provided by the robot's IMU; Calculate the angle between the obtained 2D direction vector and the current viewing direction of the robot, and determine the corresponding azimuth from the preset correspondence between the angle and the azimuth; Establish a directed edge from the first node to the second node, and use the corresponding azimuth of the angle as the descriptor of the directed edge from the first node to the second node. Moreover, the source node id of this directed edge is the node id of the first node, and the target node id of this directed edge is the node id of the second node; According to the 3D positions of the first node and the second node, calculate the 3D vector from the second node to the first node, and project this 3D vector onto the horizontal plane to obtain another 2D direction vector; Calculate the angle between the obtained 2D direction vector and the current viewing direction of the robot (exemplarily, the viewing angle of the robot is 180°, and the current viewing direction of the robot is the "front" direction vector of the robot), and determine the corresponding azimuth from the preset correspondence between the angle and the azimuth; Establish a directed edge from the second node to the first node, and use the corresponding azimuth of the angle as the descriptor of the directed edge from the second node to the first node. Moreover, the source node id of this directed edge is the node id of the second node, and the target node id of this directed edge is the node id of the first node. Exemplarily, the preset correspondence between the angle and the azimuth is specifically: the azimuth corresponding to [0°, 45°) is "front", the azimuth corresponding to (45°, 90°) is "right front", the azimuth corresponding to [90°, 135°) is "right rear", the azimuth corresponding to [135°, 180°) is "rear", the azimuth corresponding to [180°, 225°) is "left rear", the azimuth corresponding to [225°, 270°) is "left", the azimuth corresponding to [270°, 315°) is "left front", and the azimuth corresponding to [315°, 360°) is "front". These azimuths are the azimuths centered on the robot's viewing angle. Through such a preset correspondence between the angle and the azimuth, the relative azimuth between the objects represented by the two nodes can be converted into the relative azimuth from the robot's viewing angle.

[0060] Specifically, the "front" direction vector of the robot currently can be determined by one of the following methods:

[0061] (1) Robot body coordinate system: First, the robot itself defines a base coordinate system (Base Frame). In this coordinate system, the "front" direction is predefined, usually along a certain axis of the coordinate system. For example, if the X-axis points forward, then the current "front" direction of the robot is the positive direction of the X-axis. In the robot's base coordinate system, its current "front" direction vector can be represented as a fixed vector. For example, v_local = [1, 0, 0] T , " T " is the transpose symbol.

[0062] (2) Obtain the current pose: The robot can obtain its current pose information in real time through the IMU. The pose information is usually represented by Euler angles (such as yaw angle Yaw, pitch angle Pitch, roll angle Roll) or quaternions. For the "front" direction on the horizontal plane, the most important parameter is the yaw angle (Yaw), which represents the orientation of the robot on the horizontal plane. Therefore, according to the current yaw angle Yaw provided by the IMU, the current "front" direction vector of the robot can be obtained.

[0063] (3) Coordinate system transformation: If the current "front" direction vector of the robot is the current "front" direction vector in the world coordinate system (or a unified reference coordinate system), then the current "front" direction vector v_local of the robot in the base coordinate system needs to be transformed to the world coordinate system according to the current pose, so as to obtain the current "front" direction vector of the robot.

[0064] (4) Use of IMU data: The IMU can directly output the pose angles (including the yaw angle) processed by the fusion algorithm. This yaw angle represents the current orientation of the robot's head or body in the horizontal plane, that is, the current "front" direction vector of the robot can be obtained according to this current yaw angle.

[0065] It should be noted that if only considering the direction in the "two-dimensional plane perpendicular to the gravity direction provided by the robot's IMU" (i.e., the horizontal plane), then the yaw angle (Yaw, denoted as θ) is mainly concerned. Assume that the X-Y plane of the world coordinate system is this horizontal plane, and the initial "front" direction of the robot is consistent with the X-axis of the world coordinate system. Then, the current "front" direction vector v of the robot on the horizontal plane world_2D can be obtained by rotation. If the yaw angle θ is the rotation angle relative to the X-axis of the world coordinate system, then this two-dimensional vector v world_2D can be expressed as: v world_2D = [cos(θ), sin(θ)] T , where T is the transpose symbol; or, if the current "front" direction vector of the robot body is [1, 0] TIn the 2D case, its current 2D "forward" direction vector in the world coordinate system is:

[0066] S6. Obtain information about the marked door opening area from the panoramic image of the environment, create a node of the door opening area node type, connect it to the room node with an edge representing the inclusion relationship, set the attribute information of the door opening area node and the edge, and determine the directed edge and the attribute information of the directed edge between the door opening area node and each adjacent functional area node in the room node, thus completing the construction of the scene sub-graph of the environment; where the functional area node adjacent to the door opening area node refers to the functional area node corresponding to the functional area that is adjacent to the door opening area represented by the door opening area node in terms of position.

[0067] Here, after creating the key object nodes of all functional area nodes of a room node, when there are marked or masked door opening areas in the panoramic image of the room, it is also necessary to obtain the information of each marked or masked door opening area from the panoramic image of the room, and according to the information of each marked door opening area, create a node of the node type of door opening area node accordingly, generate a node ID for each door opening area node, and use the obtained node ID as the child node ID of the room node. Set the node semantic label for each door opening area node to indicate that each door opening area node represents a door. Then, set the 3D position of each door opening area node to be empty, set the node size of each door opening area node to the geometric range of the door opening. The geometric range of the door opening area can be the 3D bounding box in the robot base coordinate system calculated according to the depth information of the 2D area of the door opening area in the image. Then, set the node ID of the room node as the parent node ID of each door opening area node, and temporarily set the child node ID of each door opening area node to be empty for adding as needed in subsequent steps. Then, connect each door opening area node to the room node with each edge, and then set the types of these edges to be inclusion relationships. Then, for each door opening area node, determine the functional area node corresponding to the functional area adjacent to the door opening area represented by the door opening area node according to the geometric range of the door opening area node and the geometric ranges of all functional area nodes of the room node of the door opening area node. Then, establish a directed edge between the functional area node corresponding to each functional area adjacent to the door opening area represented by the door opening area node and the door opening area node, and determine the attribute information of this directed edge. It should be noted that the specific principle of determining the directed edge and the attribute information of the directed edge between the door opening area node and each adjacent functional area node is the same as the principle of determining the directed edge and the attribute information of the directed edge between every two key object nodes described above. Just analogize the door opening area node to a key object node, and each adjacent functional area node to another key object node, and then execute according to the above principle.

[0068] S7. Control the robot to enter the new environment in the preset scenario according to the information of the door opening area node and the information of the relevant edges, and then return to the above S2 to continue execution until all environments in the preset scenario are traversed, and the initial scene graph of the preset scenario is obtained.

[0069] S8. Check the initial scene graph, and use the checked scene graph as the final scene graph of the preset scenario.

[0070] In some embodiments, a rule engine or a logic programming method can be employed to check the topological consistency of the initial scene graph of a preset scene, so as to check whether the parent-child relationships, node type constraints, etc. in the initial scene graph meet the preset rules. The preset rules can be set according to actual needs. For example, they can be texts used to describe the types of functional areas included in each type of room and the types of key objects included in each type of functional area. When nodes that do not meet the preset rules are found, manual inspection can be prompted to correct the attribute information of the nodes that do not meet the preset rules manually, so as to obtain a verified scene graph.

[0071] In some embodiments, a large model (e.g., the LLM model) can also be used to perform semantic consistency checking based on the common sense of the large model on the initial scene graph of the obtained preset scene. Specifically, during the checking, the nodes in the initial scene graph and the relationships between the nodes are transformed into natural language descriptions (e.g., "the sofa node is included in the kitchen area node"), and the transformed natural language descriptions are input into the large model, so that the large model can judge whether the things stated in the input natural language descriptions are reasonable, and determine whether there are errors in the semantic relationships between the nodes related to the natural language descriptions according to the judgment of the statement rationality of the input natural language descriptions output by the large model (e.g., "impossible", "reasonable", "common"). When nodes with incorrect semantic relationships are found, manual inspection can be prompted to correct the attribute information of the nodes with incorrect semantic relationships manually, so as to obtain a verified scene graph. In this way, common sense can be used to correct or avoid some obvious mapping errors at the semantic level.

[0072] In some embodiments, the rule engine or the logic programming method can be first used to check the topological consistency of the initial scene graph of the obtained preset scene, and the initial scene graph can be corrected according to the checking results. Then, the large model can be used to perform semantic consistency checking based on the common sense of the large model on the corrected initial scene graph, and the corrected initial scene graph can be corrected according to the checking results, so as to obtain a verified scene graph.

[0073] The above method proposed by the present invention constructs a semantic scene graph from bottom to top through the perception of the environment from the first perspective of the robot, including regional space distribution recognition, object attribute and layout analysis, room type clustering, and forms a complete scene representation in combination with prior knowledge. This method can dynamically adapt to and update the scene graph according to the real-time observation of the robot, ensuring the accuracy and timeliness of the scene representation.

[0074] In the present invention, after obtaining the final scene graph of the preset scene, the scene graph can be serialized into a file and stored in a database for efficient query and use by downstream modules such as navigation, planning, and interaction. The specific storage format can be set according to actual needs, and the present invention does not limit this.

[0075] In the present invention, the instruction parsing and planning module is specifically used to obtain the user's natural language navigation instruction, and parse the natural language navigation instruction into a structured task instruction through the LLM model. Among them, the structured task instruction includes: task type, semantic information of the target location, semantic information of the regional location, semantic information of the target object, and constraint conditions; and find a subgraph (hereinafter represented by RC-ISG(P)) in the scene graph that contains the semantic information of the target location, the semantic information of the regional location, and the semantic information of the target object in the structured task instruction. Then, control the robot to collect the RGB-D data frame of the current environment, and send the structured task instruction, RC-ISG(P), and the RGB-D data frame of the current environment to the high-level AI agent module. Here, through instruction parsing, the user's natural language navigation instruction can be converted into a machine-understandable structured task representation, that is, a structured task instruction. For example, when the user's natural language navigation instruction is "go to the living room to find my glasses", the corresponding structured task instruction is: "{task type: navigation and search, target location semantics: living room, regional location semantics: leisure area, target object semantics: glasses, constraint: null}". Here, when finding a subgraph in the scene graph that contains the semantic information of the target location, the semantic information of the regional location, and the semantic information of the target object in the structured task instruction, if there are two or more subgraphs in the scene graph that all contain the semantic information of the target location, the semantic information of the regional location, and the semantic information of the target object in the structured task instruction, then the RC-ISG(P) found from the scene graph is the large subgraph composed of these two or more subgraphs. Then, when the low-level AI agent module uses this RC-ISG(P) subsequently, it further selects the required information from this RC-ISG(P).

[0076] In the present invention, the high-level AI agent module is one of the decision-making modules of the present invention and is responsible for global and semantic-level navigation planning. The high-level AI agent module is implemented based on a large language model (LLM), adopts a planning algorithm, and has the ability to call tools. Specifically, the high-level AI agent module is used to determine the current room-level position of the robot according to the scene graph and the RGB-D data frame of the current environment, and then input the current room-level position of the robot, the structured task instruction, and the RC-ISG(P) into the LLM model to obtain a single or at least two structured semantic subtasks arranged in sequence. Then, it controls the robot to obtain the current pose output by the IMU and re-obtain the RGB-D data frame of the current environment, and sends the current pose, the re-obtained RGB-D data frame, the current structured semantic subtask to be executed, and the RC-ISG(P) to the low-level AI agent module. Here, the high-level AI agent module is specifically used to perform understanding and matching inference according to the scene graph and the RGB-D data frame of the current environment by using existing positioning tools, so as to determine the current room-level position of the robot (that is, which room in the scene it is located in). Here, the at least two structured semantic subtasks arranged in sequence are a series of structured semantic subtasks (Sub-goal set St) that are logically continuous and easier to execute. Through the LLM model, a complex original task that may require long-distance movement or multi-step operations can be decomposed into a series of structured semantic subtasks that are logically continuous and easier to execute, so as to facilitate the execution of the task and improve the execution efficiency. Here, each structured semantic subtask also includes: task type, semantic information of the target position, semantic information of the regional position, semantic information of the target object, and constraint conditions. For example, the above-mentioned "go to the living room to find my glasses" can be decomposed into structured semantic subtask 1 (Sub-goal 1): "navigate to the living room", and structured semantic subtask 2 (Sub-goal 2): "search for glasses in the leisure area of the living room". Here, when obtaining at least two structured semantic subtasks arranged in sequence, the high-level AI agent module will select one structured semantic subtask that needs to be executed currently in sequence, and then control the robot to obtain the current pose output by the IMU and re-obtain the RGB-D data frame of the current environment, and send the current pose, the re-obtained RGB-D data frame, the current structured semantic subtask to be executed, and the RC-ISG(P) to the low-level AI agent module for execution.

[0077] In the present invention, the low-level AI agent module is responsible for converting the structured semantic subtasks issued by the high-level agent into specific physical action sequences. It is also implemented based on the LLM or other policy models. Specifically, the low-level AI agent module is used to retrieve, from the RC-ISG(P), the information required to execute the currently pending structured semantic subtask according to the newly acquired RGB-D data frame and the currently pending structured semantic subtask. The required information includes: the path information required to execute the currently pending structured semantic subtask, the node, sub-node, and edge attribute information corresponding to the target position and / or target object in the currently pending structured semantic subtask. Then, using the LLM model and combining the newly acquired RGB-D data frame, the current pose, the currently pending structured semantic subtask, and the required information, an optimal physical action is selected from the predefined action space and the robot is controlled to execute. After the robot executes the optimal physical action, the robot is continuously controlled to acquire a new current RGB-D data frame and a new current pose. Using the LLM model and combining the new current RGB-D data frame and the new current pose, an optimal physical action is continuously selected from the predefined action space and the robot is controlled to execute. This iterative loop continues until the subtask execution result is obtained and fed back to the high-level AI agent module. Specifically, for example, when the currently pending structured semantic subtask is "navigate to the living room" as described above, the specific information required to execute this subtask can be retrieved from the RC-ISG(P) according to this structured semantic subtask, the current RGB-D data frame, and the current pose. For example, the node id of the living room in the RC-ISG(P), the areas and objects included in the living room, the path nodes leading to the living room, etc. Here, the predefined action space is the set of actions that the robot can execute, which can be set according to actual needs. For example, it can be {move forward 0.25m, turn left 15 degrees, turn right 15 degrees, stop, check object, steer around obstacle}. Here, during and after the robot executes an action, the robot continuously collects real-time RGB-D data frames and poses. The low-level AI agent module continues to select actions from the predefined action space and controls the robot to execute according to the new RGB-D data frame and the new current pose. Since the low-level AI agent module determines the robot's actions based on the real-time updated RGB-D data frames and poses. Therefore, if an unexpected dynamic obstacle or environmental change (detected through real-time sensor data) is encountered during execution, the new RGB-D data frame and pose will reflect this change. Thus, in the next decision step, the low-level AI agent module will naturally select a suitable action (such as stop, steer around obstacle, etc.) according to this new state, thereby realizing real-time and reactive navigation strategy adjustment.

[0078] During the execution of the structured semantic subtasks, when a subtask is completed, the lower-level AI agent module will feedback the task execution result to the higher-level AI agent module. When the higher-level AI agent module receives the task execution result and learns that the subtask has been successfully executed based on this result, it will continue to obtain the current RGB-D data frame and the current pose of the robot, and send the continuously obtained RGB-D data frame and pose, the next structured semantic subtask to be executed, and the RC-ISG(P) to the lower-level AI agent module. In this way, until all the structured semantic subtasks are executed. When all the structured semantic subtasks are successfully executed, it means that the entire task has been successfully executed, and thus the task execution result can be output, such as "The task has been successfully completed, and the glasses have been found on the coffee table in the living room."

[0079] In some embodiments, during the execution of the structured semantic subtasks, the lower-level AI agent module will continuously evaluate whether the current subtask is still feasible. If it is determined that the target is unreachable (for example, the path is completely blocked), an evaluation function will be called to generate a failure report, which describes the reason for the failure of the subtask execution through the failure report, and the failure report will be fed back to the higher-level AI agent module. In some embodiments, when the higher-level AI agent module receives a failure report of a subtask, it will re-perform semantic planning based on the information in the failure report and the entire scene graph. For example, re-decompose the remaining unexecuted structured semantic subtasks or select a new navigation route, or when it is determined that the entire task cannot be completed, regenerate new structured semantic subtasks.

[0080] The present invention also provides an autonomous visual semantic navigation method based on interactive semantic mapping, which is applied to a robot. As Figure 2 shown, the method includes:

[0081] S101. Obtain the natural language navigation instruction of the user.

[0082] S102. Parse the natural language navigation instruction into a structured task instruction.

[0083] S103. Obtain the sub-graph related to the task from the scene graph according to the structured task instruction, and obtain the environmental information currently observed by the robot; wherein, the scene graph is a multi-level heterogeneous semantic scene graph constructed from the perspective of the robot and composed of nodes and edges, and the scene graph includes a global scene layer, a room layer, a functional area layer, a key object layer, and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects.

[0084] S104. Generate a single or at least two structured semantic subtasks arranged in sequence according to the scene graph, the structured task instruction, the sub-graph, and the environmental information, and re-obtain the environmental information currently observed by the robot and the current pose.

[0085] S105. Generate a physical action sequence for the robot and control the movement of the robot according to the newly obtained environmental information, the current pose, the current structured semantic subtask to be executed, and the subgraph until the execution result of the subtask is obtained.

[0086] S106. When the execution result of the subtask indicates success, continue to obtain the environmental information and pose currently observed by the robot. According to the continuously obtained environmental information and pose, the next structured semantic subtask to be executed, and the subgraph, continue to generate a physical action sequence for the robot and continue to control the movement of the robot. In this way, until all structured semantic subtasks are executed.

[0087] It should be noted that the specific details in the above navigation method have been described in the above navigation system section and will not be elaborated here.

[0088] The technical solution of the present invention has the following advantages:

[0089] 1) The present invention constructs a multi-level, language-interpretable indoor scene graph (RC-ISG) from the robot's central perspective, breaking through the semantic deviation problem of traditional third-person perspective scene graphs. By organically integrating the relative position relationships among spatial topology, functional areas, and key objects, the system can more accurately reflect what the robot actually sees, thereby improving the accuracy and execution efficiency of navigation decisions.

[0090] 2) Hierarchical dual AI-Agent framework: The present invention proposes a dual-agent framework that combines a high-level semantic planning agent and a low-level execution control agent. The high-level agent is based on the robot's central scene graph and environmental perception information and is responsible for decomposing complex tasks into structured semantic sub-goals; the low-level agent performs specific action execution and path planning for the preset sub-goals based on real-time perception data. This hierarchical design not only reduces the task complexity but also significantly improves the adaptability and task success rate of the system in dynamic environments.

[0091] 3) Dynamic environment adaptation and real-time update: Different from the problems of existing technologies mainly facing static scenarios, the present invention can update the scene graph and navigation strategy in a timely manner during operation, so as to maintain efficient task execution and dynamic adaptation capabilities.

[0092] 4) Enhanced semantic understanding and reasoning capabilities: While retaining the necessary geometric details, the multi-level scene graph of the present invention fully improves the system's parsing ability for natural language instructions by hierarchically encoding the semantic relationships among rooms, regions, and key objects. This method realizes an efficient mapping from abstract semantics to specific spatial positioning, improving the accuracy and robustness of navigation decisions.

[0093] 5) High generalization ability and deployment efficiency: The present invention utilizes the representation method from the robot's perspective and the hierarchical planning strategy, and can still maintain high navigation performance in unknown or complex environments. The system shows excellent advantages in reducing the consumption of computing resources and realizing real-time dynamic planning, providing higher reliability and scalability for actual deployment.

[0094] It should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0095] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0096] In the specification, the term "including" does not exclude other components or steps, and "a" or "one" does not exclude the case of a plurality. Certain measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0097] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. An autonomous visual semantic navigation system based on interactive semantic mapping, which is applied to a robot, and is characterized in that, Including: An instruction parsing and planning module, configured to parse the obtained natural language navigation instruction of the user into a structured task instruction, obtain a task-related sub-graph from the scene graph according to the structured task instruction, and obtain the environmental information currently observed by the robot, and send the structured task instruction, the sub-graph and the environmental information to a high-level AI agent module; the scene graph is a semantic scene graph constructed from the perspective of the robot and composed of nodes and edges, including a scene layer, a room layer, a functional area layer, a key object layer and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects; The high-level AI agent module is configured to generate one or at least two structured semantic subtasks arranged in sequence according to the scene graph, the structured task instruction, the sub-graph and the environmental information, and re-obtain the environmental information and the current pose currently observed by the robot, and send the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtask and the sub-graph to a low-level AI agent module; The low-level AI agent module is configured to generate a physical action sequence of the robot according to the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtask and the sub-graph, and control the movement of the robot until a sub-task execution result is obtained and fed back to the high-level AI agent module; The high-level AI agent module is further configured to, when the sub-task execution result indicates successful execution, continue to determine the next to-be-executed structured semantic subtask, and obtain the environmental information and pose currently observed by the robot and then send them to the low-level AI agent module, and so on, until all structured semantic subtasks are executed.

2. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 1, wherein The scene graph includes a root node, at least two first nodes, at least two second nodes and at least two third nodes, and the first nodes are child nodes of the root node, the second nodes are child nodes of the first nodes, the third nodes are child nodes of the second nodes, and there is an edge between each parent node and its child node. Among them, the root node represents the scene layer, the root node represents the map scene, the at least two first nodes represent the room layer, and the at least two first nodes are used to represent at least two different rooms, the at least two second nodes represent the functional area layer, and one second node represents a functional area in the room, the at least two third nodes represent the key object layer, and one third node represents a key object in the functional area.

3. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 2, characterized in that, The attribute information of the node includes: node id, node type, node semantic label, 3D position of the node, size of the node, id of the parent node of the node, id of the child node of the node; where the node type is used to describe whether the node is a scene node, a room node, a functional area node, a door opening area node or a key object node; the node semantic label is used to describe the semantics of the node. The attribute information of the edge includes: edge id, source node id of the edge, target node id of the edge, edge type, and edge descriptor; among them, the edge type is used to describe whether the edge represents an inclusion relationship, an adjacent relationship, a relative relationship, or a transition relationship; the edge descriptor is used to describe the relative position between the two nodes connected by the edge from the perspective of the robot.

4. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 1, characterized in that, The scene graph is constructed by the following method: S1. Create a root node with the node type of scene node according to the predefined scene information, set the attribute information of the root node, and then control the robot to enter the preset scene corresponding to the predefined scene information; S2. Control the robot to collect images of the entered environment, determine the panoramic image of the environment according to the collected images, and perform object detection and scene area understanding, as well as functional area understanding and segmentation on the panoramic image respectively in combination with the preset object prompt information and functional area prompt information, to obtain an object instance list and a functional area division result of the panoramic image; The object instance list contains the category label, 2D position, pose, and size of each detected object in the image; the functional area division result contains the geometric range of each functional area and the semantic label of each functional area; S3. When it is determined that the environment entered by the robot is a room, create a node with the node type of room node, connect it to the root node with an edge representing an inclusion relationship, and set the attribute information of the room node and the edge; S4. According to the number of functional areas included in the functional area division result, create the corresponding number of nodes with the node type of functional area node, set the attribute information for each of the created corresponding number of nodes, use the nodes as the child nodes of the room node, connect them to the room node with an edge representing an inclusion relationship, and set the attribute information of the edge; S5. According to the object instance list, the predefined key object categories, and the functional area division result, judge whether there are key objects and the number of existing key objects in each functional area, and when there are key objects in a functional area, create the corresponding number of nodes with the node type of key object node, set the attribute information of each key object node, then connect each key object node to the corresponding functional area node with an edge representing an inclusion relationship, and set the attribute information of the edge, and when a functional area node has two or more key object nodes, determine the directed edge between every two key object nodes of the functional area node and the attribute information of the directed edge; S6. Obtain the information of the marked door opening area from the panoramic image of the environment, create a node with the node type of door opening area node, connect it to the room node with an edge representing the inclusion relationship, set the attribute information of the door opening area node and the edge, and determine the directed edge between the door opening area node and each adjacent functional area node in the room node and the attribute information of the directed edge, completing the construction of the scene sub-graph of the environment; wherein, the functional area node adjacent to the door opening area node refers to the functional area node corresponding to the functional area adjacent to the door opening area represented by the door opening area node in terms of position. S7. According to the information of the door opening area node and the information of the relevant edges, control the robot to enter the new environment in the preset scene, and then return to the above S2 to continue execution until all the environments in the preset scene are traversed, obtaining the initial scene graph of the preset scene. S8. Verify the initial scene graph, and use the verified scene graph as the final scene graph of the preset scene.

5. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 4, characterized in that, The image collected by the robot is x RGB-D data frames, where x is a positive integer greater than or equal to 1. When x = 1, the RGB-D data frame contains the panorama of the environment. When x > 1, the x RGB-D data frames contain the panorama of the environment after splicing; the determination of the panoramic image of the environment according to the collected image includes: For each RGB-D data frame in the x RGB-D data frames, synchronize and align the RGB image and the depth image contained in the RGB-D data frame to obtain the synchronized and aligned RGB image and the synchronized and aligned depth image. Perform invalid value processing, noise filtering processing, and structural noise processing on the synchronized and aligned RGB image and the synchronized and aligned depth image in sequence to obtain the RGB image after data cleaning and the depth image after data cleaning. Perform depth masking processing on the RGB image after data cleaning and the depth image after data cleaning respectively to obtain the processed RGB image and the processed depth image. Perform door opening area recognition on the processed RGB image and the processed depth image respectively, and mask or mark the recognized door opening area to obtain the RGB image after transition area processing and the depth image after transition area processing, where the RGB image after transition area processing and the depth image after transition area processing constitute a processed RGB-D data frame. Splice the obtained x processed RGB-D data frames to obtain the panoramic image of the environment.

6. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 4, wherein The combined use of the preset object prompt information and functional area prompt information to perform object detection and scene area understanding, as well as functional area understanding and segmentation on the panoramic image respectively, to obtain an object instance list and a functional area division result of the panoramic image, including: Input the panoramic image and the preset object prompt information into the Grounding DINO model, and detect and locate the objects contained in the panoramic image through the Grounding DINO model to obtain a list of object instances; wherein, the preset object prompt information is used to prompt the Grounding DINO model to detect and locate the objects to be detected and located from the panoramic image. Input the panoramic image and the preset functional area prompt information into the VLM model, understand and segment the functional areas in the panoramic image through the VLM model, and obtain a functional area division result according to the result output by the VLM model; wherein, the preset functional area prompt information is used to prompt the VLM model to understand and divide the functional areas to be understood and divided from the panoramic image.

7. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 4, characterized in that, Determining the directed edges between every two key object nodes of the functional area node and the attribute information of the directed edges includes: For any two key object nodes in the functional area node, take one of the two key object nodes as the first node and the other as the second node. According to the 3D position of the first node and the 3D position of the second node, calculate the 3D vector from the first node to the second node, and project the 3D vector onto the horizontal plane to obtain a 2D direction vector, where the horizontal plane is a two-dimensional plane perpendicular to the gravity direction provided by the IMU of the robot. Calculate the angle between the obtained 2D direction vector and the current viewing direction of the robot, and determine the azimuth corresponding to the angle from the preset correspondence between the angle and the azimuth. Establish a directed edge from the first node to the second node, and use the azimuth corresponding to the angle as the descriptor of the directed edge from the first node to the second node. According to the 3D position of the first node and the 3D position of the second node, calculate the 3D vector from the second node to the first node, and project the 3D vector onto the horizontal plane to obtain another 2D direction vector. Calculate the angle between the obtained 2D direction vector and the current viewing direction of the robot, and determine the azimuth corresponding to the angle from the preset correspondence between the angle and the azimuth. Establish a directed edge from the second node to the first node, and use the azimuth corresponding to the angle as the descriptor of the directed edge from the second node to the first node.

8. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 1, characterized in that The instruction parsing and planning module is specifically configured to obtain the user's natural language navigation instruction and parse the natural language navigation instruction into a structured task instruction through an LLM model, where the structured task instruction includes: task type, semantic information of the target location, semantic information of the regional location, semantic information of the target object, and constraint conditions; and find a sub-graph in the scene graph that contains the semantic information of the target location, the semantic information of the regional location, and the semantic information of the target object in the structured task instruction, and then control the robot to collect the RGB-D data frame of the current environment, and send the structured task instruction, the sub-graph, and the RGB-D data frame of the current environment to the high-level AI agent module; The high-level AI agent module is specifically configured to determine the current room-level location of the robot according to the scene graph and the RGB-D data frame of the current environment, and then input the current room-level location of the robot, the structured task instruction, and the sub-graph into the LLM model to obtain at least two structured semantic subtasks that are single or arranged in sequence. Then, control the robot to obtain the current pose and re-obtain the RGB-D data frame of the current environment, and send the current pose, the re-obtained RGB-D data frame, the currently to-be-executed structured semantic subtask, and the sub-graph to the low-level AI agent module.

9. The autonomous visual semantic navigation system based on interactive semantic mapping according to claim 8, characterized in that, The low-level AI agent module is specifically configured to retrieve the information required to execute the currently to-be-executed structured semantic subtask from the sub-graph according to the re-obtained RGB-D data frame and the currently to-be-executed structured semantic subtask, where the required information includes: path information required to execute the currently to-be-executed structured semantic subtask, and attribute information of nodes, sub-nodes, and edges corresponding to the target location and / or target object in the currently to-be-executed structured semantic subtask; then, use the LLM model and combine the re-obtained RGB-D data frame, the current pose, the currently to-be-executed structured semantic subtask, and the required information to select an optimal physical action from the predefined action space and control the robot to execute it. After the robot executes the optimal physical action, continue to control the robot to obtain a new current RGB-D data frame and a new current pose, and use the LLM model and combine the new current RGB-D data frame and the new current pose to continue to select an optimal physical action from the predefined action space and control the robot to execute it. Such iterative loop is performed until the subtask execution result is obtained and fed back to the high-level AI agent module.

10. An autonomous visual semantic navigation method based on interactive semantic mapping, applied to a robot, characterized in that It includes: Obtain the user's natural language navigation instruction; Parse the natural language navigation instruction into a structured task instruction; Obtain task-related subgraphs from the scene graph according to the structured task instructions, and obtain the environmental information currently observed by the robot; wherein, the scene graph is a multi-level heterogeneous semantic scene graph constructed from the perspective of the robot and composed of nodes and edges, and the scene graph includes a global scene layer, a room layer, a functional area layer, a key object layer and the semantic relationships between the layers, as well as the relative position relationships between adjacent functional areas and adjacent key objects; Generate single or at least two structured semantic subtasks arranged in sequence according to the scene graph, the structured task instructions, the subgraph and the environmental information, and re-obtain the environmental information currently observed by the robot and the current pose; Generate the physical action sequence of the robot and control the movement of the robot according to the re-obtained environmental information, the current pose, the currently to-be-executed structured semantic subtask and the subgraph until the subtask execution result is obtained; When the subtask execution result indicates successful execution, continue to obtain the environmental information and pose currently observed by the robot, and continue to generate the physical action sequence of the robot and continue to control the movement of the robot according to the continuously obtained environmental information and pose, the next to-be-executed structured semantic subtask and the subgraph. In this way, continue until all structured semantic subtasks are executed.

Images