Semantic mapping method and device based on scene preset, and electronic device

Abstract

This application belongs to the field of computer vision, specifically disclosing a semantic mapping method, apparatus, and electronic device based on scene pre-setting. The method includes: acquiring the pose information of a robot at the current moment in a first scene, and determining the trigger for semantic processing based on the stored pose state; acquiring the current LiDAR point cloud data and image data, performing semantic segmentation on the image data based on an open-set object detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels; projecting the semantic labels onto the three-dimensional spatial coordinates of the LiDAR point cloud data, and performing a coordinate-weighted average of the dynamic object positions corresponding to the semantic segmentation result to obtain a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic labels, and semantic segmentation instance IDs, thereby realizing 3D semantic mapping. This application can improve the effect and accuracy of 3D semantic mapping.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and more specifically, relates to a semantic mapping method, apparatus and electronic device based on scene pre-setting. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) is an important technology for enabling robots to autonomously locate and navigate in unknown environments. It uses the robot's position and map during movement to perform self-localization, and builds incremental maps based on its self-localization. It is widely used in many application scenarios such as 3D modeling and autonomous driving.

[0003] In the field of mobile robotics, traditional SLAM mainly uses devices such as LiDAR, stereo vision, or depth sensors to acquire 3D point cloud data of the environment to construct geometric maps that accurately reflect the spatial structure of the scene. Although this can assist robots in path planning to some extent, the limited geometric information makes it difficult for robots to achieve a deep understanding of the environment and make intelligent decisions. This is especially true in complex and unstructured scenarios, such as homes and offices where there are many items and they are arranged irregularly. Robots cannot identify object categories or understand the relationships between objects by relying solely on geometric maps, thus limiting the further improvement of the robot's intelligence level.

[0004] In existing technologies, LiDAR algorithms or vision algorithms are typically used to construct 3D point clouds. However, due to data differences, issues such as inconsistent semantic information, inaccurate spatial location, and redundant or missing semantic labels can arise, resulting in poor semantic mapping performance and low accuracy. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this application is to provide a semantic mapping method, device and electronic device based on scene preset, which aims to solve the problems of poor effect and low accuracy of semantic mapping.

[0006] To achieve the above objectives, in a first aspect, this application provides a semantic mapping method based on scene pre-defined parameters, comprising:

[0007] Obtain the robot's pose information at the current moment in the first scenario, and determine the trigger for semantic processing based on the stored pose state;

[0008] Acquire the current lidar point cloud data and image data, and perform semantic segmentation on the image data based on the open set target detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels;

[0009] The semantic labels are projected onto the three-dimensional spatial coordinates of the lidar point cloud data, and the coordinate weighted average of the dynamic object positions corresponding to the semantic segmentation results is performed to obtain a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic labels, and semantic segmentation instance IDs, so as to realize 3D semantic mapping.

[0010] Optionally, the 3D semantic mapping process includes:

[0011] Visual language models are used to perform scene graph understanding on single-frame images of 3D semantic point clouds to determine the positional relationship nodes between objects in a single-frame image;

[0012] Visual language models are used to understand scene attributes of 3D semantic point clouds and determine the functional attribute nodes in the scene where the robot is located.

[0013] A topological graph of the hierarchical relationships between objects and rooms is constructed based on location relationship nodes and functional attribute nodes to achieve 3D semantic mapping.

[0014] Optionally, it also includes:

[0015] The DBSCAN clustering algorithm is used to segment the 3D semantic point cloud into instances, and the K nearest neighbor algorithm is combined to remove noise points and outliers.

[0016] Using the RANSAC algorithm and combining point cloud normal constraints and z-axis height constraints, the ground and ceiling planes in the first scene are extracted and separated.

[0017] An optimized 3D semantic point cloud is generated through clustering, denoising, and planar extraction operations.

[0018] Optionally, the step of performing semantic segmentation on the image data based on the open-set object detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels includes:

[0019] The image data is input into an open-set object detection and segmentation algorithm model for pixel-level semantic segmentation to obtain an initial segmentation result; the initial segmentation result includes semantic labels corresponding to objects in the image.

[0020] Noise and invalid detection regions in the segmentation results are removed by area threshold filtering and depth verification;

[0021] The initial segmentation result is subjected to scene-adaptive secondary verification using a lightweight object detection model to correct semantic label misjudgments and obtain the verified semantic segmentation result.

[0022] Optionally, it also includes:

[0023] The verified semantic segmentation results are back-projected to the three-dimensional world coordinate system through cross-modal coordinate transformation to obtain the global centroid coordinates of the semantic instance;

[0024] The three-dimensional index relationship is determined based on image identifiers, global centroid coordinates, and semantic tags for semantic query retrieval.

[0025] Optionally, it also includes:

[0026] When the robot crosses the passable boundary of the first scene, it matches the current 3D semantic point cloud with the historical 3D point cloud to achieve spatial continuity.

[0027] Logical consistency verification is performed on nodes and edges of connected scenes in the scene semantic topology, and data reliability is evaluated by dual indicators of overlap rate and semantic consistency.

[0028] When the robot enters the second scene, it determines the environmental information of the second scene and identifies target objects with high distinguishability, determines the scene type of the second scene, and infers the functional attributes of the second scene through zero-shot classification or scene graph matching.

[0029] The corresponding nodes of the functional attributes of the second scene are integrated into the global semantic graph and connected with the corresponding nodes of the first scene.

[0030] Optionally, the method for determining the passable boundary includes:

[0031] Based on the lidar point cloud data, the geometric plane features of the wall in the first scene are fitted to obtain the laser notch features corresponding to the passable boundary.

[0032] Determine the gate region semantic labels in the semantic labels, and verify the spatial rationality of the gate region semantic labels based on the confidence threshold of the gate region semantic labels;

[0033] Project the geometric planar features detected by the lidar and the semantic labels of the gate regions onto the same coordinate system;

[0034] When the spatial overlap between the laser notch feature and the semantic label of the gate region exceeds a preset threshold, it is determined to be a valid passable boundary.

[0035] Optionally, the determination of the stored pose state triggers semantic processing, including:

[0036] Compare the pose information at the current moment with the pose of the previous processed frame that has been stored.

[0037] When the detected pose change exceeds both the angle threshold and the displacement threshold simultaneously, the semantic processing flow is automatically triggered.

[0038] After semantic processing is completed, the stored pose state of the processed frame is updated to the current pose.

[0039] Secondly, this application also provides a semantic mapping device based on scene pre-setting, comprising:

[0040] The semantic triggering module is used to obtain the robot's pose information at the current moment in the first scenario, and determine the trigger for semantic processing based on the stored pose state.

[0041] The semantic segmentation module is used to acquire the current lidar point cloud data and image data, and to perform semantic segmentation on the image data based on the open set target detection and segmentation algorithm to obtain semantic segmentation results carrying semantic labels.

[0042] The semantic fusion module is used to project the semantic labels onto the three-dimensional spatial coordinates of the lidar point cloud data, and to perform a coordinate-weighted average of the dynamic object positions corresponding to the semantic segmentation results to obtain a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic labels and semantic segmentation instance IDs, so as to realize 3D semantic mapping.

[0043] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0044] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0045] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0046] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0047] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0048] (1) This application effectively reduces the interference of dynamic objects on the stability of map construction by projecting semantic labels onto three-dimensional spatial coordinates and by performing coordinate weighted averaging on the positions of dynamic objects, thereby enhancing the semantic consistency of static scenes. During the fusion process, the lidar point cloud provides high-precision coordinates obtained from the measurement, which correspond to the labels provided by the visual semantic segmentation results. Thus, it can generate a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic labels and instance IDs. This not only achieves accurate identification and positioning of object categories in the scene, but also improves the semantic richness and structured expression capability of the map, thereby improving the semantic mapping effect and accuracy.

[0049] (2) This application introduces an incremental semantic verification mechanism, uses a lightweight detection model to verify label consistency twice, and uses DBSCAN clustering combined with deep information to filter outout points, eliminate redundant or erroneous labels, and further improve the semantic mapping effect.

[0050] (3) This application reduces misjudgment caused by people or furniture blocking the way by integrating laser geometric features, such as wall gaps and visual semantics, such as door labels, and improves the recognition accuracy of passable boundaries (such as doors).

[0051] (4) This application constructs a global semantic graph with complete topological relationships by establishing semantic connection edges across scenes, enabling the robot to not only understand the layout of a single scene, but also to grasp the spatial logical relationship of the entire environment and realize cross-regional task planning. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the semantic mapping method based on scenario preset provided in the embodiments of this application;

[0053] Figure 2 This is a flowchart of the single-frame semantic point cloud acquisition process provided in the embodiments of this application;

[0054] Figure 3 This is a flowchart of the multi-layer scene graph construction process provided in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the semantic mapping device based on scene preset provided in the embodiments of this application;

[0056] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0059] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0060] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0062] The embodiments of this application are described below with reference to the accompanying drawings.

[0063] Reference Figure 1 This application provides a semantic mapping method based on scene pre-defined criteria, including:

[0064] S101. Obtain the robot's pose information at the current moment in the first scenario, and determine the trigger for semantic processing based on the stored pose state.

[0065] S102. Obtain the current lidar point cloud data and image data, and perform semantic segmentation on the image data based on the open set target detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels.

[0066] S103. Project the semantic label onto the three-dimensional spatial coordinates of the lidar point cloud data, and perform a coordinate-weighted average on the dynamic object position corresponding to the semantic segmentation result to obtain a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic label and semantic segmentation instance ID, so as to realize 3D semantic mapping.

[0067] Specifically, in this embodiment, the core objective of step S101 is to achieve an intelligent balance between map update requirements and computational efficiency. The robot does not perform time-consuming semantic processing on every frame of the image, but rather adaptively triggers based on its motion state.

[0068] The robot continuously receives pose information (including 3D position and orientation) from odometry. The current pose is compared with the pose stored during the last semantic processing (referred to as a "processing frame"). When a pose change is detected to simultaneously exceed a preset angle threshold (e.g., 10 degrees) and a displacement threshold (e.g., 0.3 meters), it is determined that the robot has moved to a position sufficient to provide a new perspective, thus triggering subsequent semantic processing. This strategy effectively avoids the significant computational redundancy caused by processing consecutive similar frames, ensuring the system's real-time performance.

[0069] In step S102, 3D point cloud data generated by the LiDAR and RGB image data captured by the camera are acquired simultaneously. It should be noted that the RGB image data is acquired using a binocular camera. Compared with a traditional monocular camera, a binocular camera can provide additional depth information through parallax calculation. This depth information can be used for redundancy verification or fusion with the LiDAR point cloud, especially in areas with rich texture, which can help improve the density and integrity of the point cloud.

[0070] Subsequently, the image data is input into an open-vocabulary semantic segmentation model such as Grounded SAM. In practical applications, due to differences in object attributes caused by scene characteristics, it is often necessary to retrain the target recognition and segmentation model for different environments. To solve the problem of repetitive training, this application proposes to use the open-set object detection and segmentation algorithm Grounded SAM for segmentation. However, in practical use, it often encounters problems such as inaccurate recognition and label disorder. This application also proposes to use the lightweight algorithm YOLO-fastest for correction, which only requires fine-tuning in different environments, greatly reducing the initial training cost required for deployment in different environments.

[0071] Furthermore, in step S103, the 2D pixel-level semantic labels obtained in S102 can be accurately projected onto the corresponding coordinates of the 3D LiDAR point cloud using the pre-calibrated extrinsic parameter matrix between the camera and the LiDAR. This ensures that each 3D point not only has a geometric position (x, y, z) but is also assigned a semantic label (m).

[0072] In this embodiment, semantic segmentation assigns a unique instance ID (n) to each independent object. The overlap between the newly detected object point cloud and existing object point clouds in the map is compared to determine if they are the same object. If so, the original ID is inherited, and an incremental update strategy (such as coordinate-weighted averaging) is used to smooth the object's position, effectively tracking dynamic objects and eliminating noise; otherwise, a new ID is assigned. The final result is a 3D semantic point cloud map in the format (x, y, z, m, n). This map not only describes the geometric structure of the environment but also clearly labels "where there are things" (semantics) and "which number this thing is" (instance), achieving true 3D semantic mapping.

[0073] Optionally, the 3D semantic mapping process includes:

[0074] Visual language models are used to perform scene graph understanding on single-frame images of 3D semantic point clouds to determine the positional relationship nodes between objects in a single-frame image;

[0075] Visual language models are used to understand scene attributes of 3D semantic point clouds and determine the functional attribute nodes in the scene where the robot is located.

[0076] A topological graph of the hierarchical relationships between objects and rooms is constructed based on location relationship nodes and functional attribute nodes to achieve 3D semantic mapping.

[0077] Specifically, this embodiment achieves high-level environmental cognition by introducing a Visual Language Model (VLM). First, VLM analyzes single-frame images, going beyond simple object recognition to further analyze the spatial and functional relationships between objects. Utilizing the Visual Language Model (VLM) to analyze the spatial and functional relationships between objects in a single-frame image overcomes the limitation of traditional SLAM in understanding the semantic logic of the environment. After the system inputs an image, VLM outputs a structured relationship graph (e.g., "the cup is on the table"), containing spatial relationships (on / under / near) and functional relationships (used-for / part-of). Through graph structure storage (nodes = objects, edges = relationships), the robot can infer the interaction logic of the scene (e.g., "the door needs to be pushed open" rather than "walk around").

[0078] Secondly, the VLM (Virtual Modeling Library) infers the functional attributes of the entire area by synthesizing the set of objects and their relationships within the scene. This step, through VLM, determines the functional attributes of the room where the agent is located (e.g., "kitchen" or "meeting room"), solving the core problem that geometric maps cannot distinguish functional areas. The system combines the object list and scene graph, dynamically generating room labels through multimodal reasoning (such as direct classification or question-and-answer analysis). For example, when "stove" and "refrigerator" are detected, they are automatically labeled as "kitchen".

[0079] Ultimately, based on these nodes, a semantic topology graph describing the hierarchical relationship between objects and rooms is constructed, so that the map not only contains geometric and object information, but also a semantic network that can be used for logical reasoning.

[0080] Optionally, it also includes:

[0081] The DBSCAN clustering algorithm is used to segment the 3D semantic point cloud into instances, and the K nearest neighbor algorithm is combined to remove noise points and outliers.

[0082] Using the RANSAC algorithm and combining point cloud normal constraints and z-axis height constraints, the ground and ceiling planes in the first scene are extracted and separated.

[0083] An optimized 3D semantic point cloud is generated through clustering, denoising, and planar extraction operations.

[0084] Specifically, this application embodiment performs post-processing optimization on the initially generated 3D semantic point cloud to improve map quality. The DBSCAN clustering algorithm is used to segment the point cloud into instances, aggregating scattered points belonging to the same object into a whole, and the K-Nearest Neighbor (KNN) algorithm is combined to identify and remove noise points and outliers, thereby obtaining point cloud clusters with clear boundaries and independent instances.

[0085] Furthermore, using the RANSAC algorithm supplemented with normal constraints and z-axis height constraints, the main horizontal planes, such as the ground and ceiling, are efficiently and accurately extracted from the point cloud. Through this series of clustering, denoising, and plane extraction operations, a final optimized 3D semantic point cloud with clear structure, low noise, and distinct main structures is output.

[0086] Optionally, the step of performing semantic segmentation on the image data based on the open-set object detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels includes:

[0087] The image data is input into an open-set object detection and segmentation algorithm model for pixel-level semantic segmentation to obtain an initial segmentation result; the initial segmentation result includes semantic labels corresponding to objects in the image.

[0088] Noise and invalid detection regions in the segmentation results are removed by area threshold filtering and depth verification;

[0089] The initial segmentation result is subjected to scene-adaptive secondary verification using a lightweight object detection model to correct semantic label misjudgments and obtain the verified semantic segmentation result.

[0090] Specifically, in practical application scenarios, due to differences in object attributes caused by scene characteristics, it is often necessary to retrain the target recognition and segmentation model for different environments. To solve the problem of repetitive training, the open-set object detection and segmentation algorithm Grouded SAM is proposed for segmentation. However, in practical use, it often encounters problems such as inaccurate recognition and label disorder. Therefore, it is proposed that the lightweight algorithm YOLO-fastest can be used for correction, requiring only fine-tuning in different environments, which greatly reduces the initial training cost required for deployment in different environments. The specific steps are as follows:

[0091] The Grounded SAM model is used to achieve open vocabulary semantic segmentation, which supports dynamic recognition of object categories outside the training set (new home appliances, temporary obstacles, etc.). Sensor noise and small-sized false detections are eliminated by area threshold filtering (masks with <400 pixels) and invalid regions (such as floating point clouds caused by reflection) are removed by combining depth information.

[0092] Using the YOLO-fastestv2 model (<1MB), fine-tuning was performed for specific scenarios (home / hospital / warehouse), and secondary verification was performed on the SAM output to correct obvious mislabeling (such as misidentifying "fire extinguisher" as "water bottle").

[0093] Optionally, the method further includes:

[0094] The verified semantic segmentation results are back-projected to the three-dimensional world coordinate system through cross-modal coordinate transformation to obtain the global centroid coordinates of the semantic instance;

[0095] The three-dimensional index relationship is determined based on image identifiers, global centroid coordinates, and semantic tags for semantic query retrieval.

[0096] Specifically, this application's embodiments achieve accurate semantic-geometric alignment and dynamic target tracking through a dual mechanism of cross-modal coordinate mapping and temporal data association. At the coordinate mapping level, the 2D semantic segmentation mask is back-projected into the 3D point cloud space using the radar-world calibration matrix obtained from the camera-LiDAR calibration matrix and odometry (Tcam2world). The object's centroid coordinates are calculated (with an error controlled within 3cm), and a three-dimensional index relationship of "image ID - global coordinates - semantic label" is constructed to ensure the unique positioning of each detected instance in the robot's global coordinate system. At the data association level, the Hungarian algorithm is employed, with a complexity of [missing information - likely related to complexity]. Instance matching is performed based on multi-frame detection results. The object ID is maintained across frames using a cost matrix (containing weights for IoU, semantic similarity, motion consistency, etc.). Simultaneously, a Kalman filter (state vector) is integrated. It can predict the motion of dynamic targets, effectively solve the ID jump problem caused by short-term occlusion, and improve the accuracy of trajectory tracking of dynamic obstacles, such as to more than 92%.

[0097] Optionally, it also includes:

[0098] When the robot crosses the passable boundary of the first scene, it matches the current 3D semantic point cloud with the historical 3D point cloud to achieve spatial continuity.

[0099] Logical consistency verification is performed on nodes and edges of connected scenes in the scene semantic topology, and data reliability is evaluated by dual indicators of overlap rate and semantic consistency.

[0100] When the robot enters the second scene, it determines the environmental information of the second scene and identifies target objects with high distinguishability, determines the scene type of the second scene, and infers the functional attributes of the second scene through zero-shot classification or scene graph matching.

[0101] The corresponding nodes of the functional attributes of the second scene are integrated into the global semantic graph and connected with the corresponding nodes of the first scene.

[0102] Specifically, in this embodiment, when the robot passes through a door or other passable boundary to enter a new scene (the second scene), the current observation is first aligned with the historical 3D point cloud using a point cloud matching algorithm to ensure the continuity of the geometric space. Then, logical consistency verification is performed on the connected nodes and edges in the existing semantic topology (e.g., checking whether the room types on both sides of the door are reasonable), and the reliability of historical data is evaluated. For the new scene, the system identifies typical objects using a Virtual Model (VLM) and infers the functional attributes of the area (e.g., "bedroom") based on these zero-shot examples. Finally, the nodes of this new scene are integrated into the global semantic graph and connected to the first scene, thereby achieving the coherent construction of a large-scale environmental semantic map.

[0103] Through a dual mechanism of cross-room topology verification and intelligent initialization of new scenes, this system systematically addresses two core challenges of long-term autonomous operation of mobile robots in multi-room environments: First, addressing the issue of scene consistency disruption during cross-regional navigation, point cloud registration (ICP error <10cm) and semantic topology verification (such as connectivity logic verification) eliminate positioning drift and topology disorder caused by dynamic environmental changes, improving the reliability of cross-room path planning by 80%. Second, addressing the cold start problem of real-time recognition of unknown areas, typical object recognition (such as bed → ward) and zero-shot semantic reasoning enable rapid functional annotation of new scenes within 3 seconds, overcoming the fundamental limitation of traditional methods that rely on pre-stored maps and cannot adapt to newly added spaces. The specific steps are as follows:

[0104] Multimodal data alignment solves the scene consistency verification problem during cross-room navigation. When the agent detects a passable boundary (such as a door), the system performs the following operations: First, it uses the ICP algorithm to accurately register the currently observed 3D point cloud with the historical 3D point cloud to ensure spatial continuity (matching error controlled within 10cm); then, it verifies the rationality of nodes and edges of connected rooms in the scene semantic topology graph, such as verifying the logical consistency of typical connectivity relationships like "living room-door-bedroom"; finally, it evaluates the reliability of historical data using both overlap rate (required >70%) and semantic consistency (such as whether the room types on both sides of the door match).

[0105] Rapid semantic annotation of unknown areas is achieved through typical object recognition, solving the problem of real-time cognition in new scenarios. When an agent enters a new room space, the system initiates an intelligent initialization process: first, it scans the environment and prioritizes the identification of typical objects with high discriminative power. For example, if a "hospital bed" is detected, it is inferred to be a ward area, and if a "stove" is found, it is determined to be a kitchen space. Then, based on the zero-shot classification capability of VLM (Llama3.2-vision) or a pre-built scene graph matching mechanism, it performs deep reasoning on the functional attributes of the room. Finally, the new room node is seamlessly integrated into the global semantic graph, and reasonable connection edges are established with adjacent areas.

[0106] Optionally, the method for determining the passable boundary includes:

[0107] Based on the lidar point cloud data, the geometric plane features of the wall in the first scene are fitted to obtain the laser notch features corresponding to the passable boundary.

[0108] Determine the gate region semantic labels in the semantic labels, and verify the spatial rationality of the gate region semantic labels based on the confidence threshold of the gate region semantic labels;

[0109] Project the geometric planar features detected by the lidar and the semantic labels of the gate regions onto the same coordinate system;

[0110] When the spatial overlap between the laser notch feature and the semantic label of the gate region exceeds a preset threshold, it is determined to be a valid passable boundary.

[0111] Specifically, this embodiment employs a decision-making strategy that combines LiDAR and vision: on the LiDAR side, candidate doorways are identified by fitting the wall surface and detecting its continuity interruption regions (gap areas) and combining this with point cloud density analysis; on the vision side, a semantic segmentation model is used to identify "door" labels in the image and verify their spatial rationality. The detection results of the two modalities are projected onto the same coordinate system, and only when the LiDAR gap features and the visual semantic labels highly overlap in space are they ultimately determined to be a valid passable boundary. This cross-validation mechanism greatly improves the accuracy and robustness of the detection, effectively avoiding the limitations of a single sensor.

[0112] Robust detection and functional status determination of doorways were achieved through laser-vision multimodal fusion technology. At the lidar level, the system uses the RANSAC algorithm to fit the geometric features of the wall (planar equation fitting error <2cm) to detect discontinuous areas (gaps with width >0.8m and abrupt depth changes >1.2m), and combines point cloud density analysis (density decrease in the door frame area >50%) to eliminate false openings. At the visual semantic level, the "door" label is identified using an open vocabulary segmentation model (Grounded SAM) (confidence threshold >0.65), and its spatial rationality is verified (e.g., the vertical angle deviation between the door frame and the ground <5°). The system projects the features of both modalities onto a unified coordinate system using an extrinsic parameter matrix (Tcam2world). When the spatial overlap between the laser gap feature and the visual semantic label exceeds 80%, it is determined to be a valid doorway, and the opening and closing status of the door is recorded (based on point cloud dynamic analysis). The technology achieved a detection accuracy of 98% in hospital corridor tests, with a false detection rate of less than 2 per thousand square meters, and supports robots to update the passable topology map in real time in dynamic environments.

[0113] In this embodiment, door recognition is not only for determining the passable boundaries of the agent, but also for the boundary constraints of the agent's understanding of the scene. That is, when using VLM to semantically understand room attribute information, the boundary door is used as a constraint to avoid using instance objects outside the room space. At the same time, the room door also serves as an important criterion for the agent to enter the new room space. If the agent is detected to have crossed the door boundary and is continuously detected for more than t>10s, it is considered to be already in the new room node.

[0114] Optionally, the determination of the stored pose state triggers semantic processing, including:

[0115] Compare the pose information at the current moment with the pose of the previous processed frame that has been stored.

[0116] When the detected pose change exceeds both the angle threshold and the displacement threshold simultaneously, the semantic processing flow is automatically triggered.

[0117] After semantic processing is completed, the stored pose state of the processed frame is updated to the current pose.

[0118] Specifically, to optimize computational resource utilization, the system employs a dynamic triggering mechanism based on odometry offset: the single-frame semantic processing flow is activated only when the agent's pose change (including displacement Δd > 0.2 meters or rotation Δθ > 10 degrees) exceeds a preset threshold. This strategy, by monitoring odometry increments in real time, restricts computationally intensive semantic analysis to key motion nodes, reducing the load by more than 50% compared to the full-frame processing mode. Simultaneously, kinematic constraints ensure the spatiotemporal continuity of environmental perception. In the specific implementation, quaternion differences are used to calculate the angle offset. This is achieved by using a circular buffer to cache historical pose states, thus balancing computational efficiency and perception accuracy on resource-constrained platforms. The specific steps are as follows:

[0119] Odometry is used for localization to calculate the position of the intelligent robot.

[0120] Starting from time 0, odometer information is received and read in chronological order, with initial odometer information d0. Save it as the initial processing frame;

[0121] The pose information at the nth time step is compared with the stored pose state. If the angle deflection is greater than 10 degrees and the displacement deflection is greater than 0.2m, semantic processing is triggered and the pose state of the processing frame is updated.

[0122] Reference Figure 2 , Figure 2 This is a flowchart of a single-frame semantic point cloud acquisition process according to an embodiment of this application, including the following steps:

[0123] Odometer positioning;

[0124] Initial frame determination;

[0125] Determine the cumulative pose changes of the agent;

[0126] Semantic segmentation using GroundedSAM; new room initialization;

[0127] Continue to the next moment;

[0128] YOLO-fastest validation;

[0129] Establish a global mapping relationship for the centroid;

[0130] Identify accessible doors in a room;

[0131] Remove dynamic objects.

[0132] Reference Figure 3 , Figure 3 This is a flowchart of the multi-layer scene graph construction process according to an embodiment of this application, including the following steps:

[0133] Semantic point cloud mapping;

[0134] Scene diagram understanding;

[0135] Room attribute reasoning;

[0136] Hierarchical relationship construction;

[0137] DBSCAN clustering;

[0138] Ground extraction;

[0139] Boundary alignment verification;

[0140] New room initialization.

[0141] Reference Figure 4 This application also provides a semantic mapping device based on scene preset, including:

[0142] The semantic triggering module 410 is used to obtain the robot's pose information at the current moment in the first scene, and determine the triggering of semantic processing based on the stored pose state.

[0143] The semantic segmentation module 420 is used to acquire the current lidar point cloud data and image data, and perform semantic segmentation on the image data based on the open set target detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels.

[0144] The semantic fusion module 430 is used to project the semantic label onto the three-dimensional spatial coordinates of the lidar point cloud data, and to perform a coordinate-weighted average of the dynamic object positions corresponding to the semantic segmentation results to obtain a 3D semantic point cloud carrying three-dimensional spatial coordinates, semantic labels and semantic segmentation instance IDs, so as to realize 3D semantic mapping.

[0145] Optionally, the 3D semantic mapping process includes:

[0146] Visual language models are used to perform scene graph understanding on single-frame images of 3D semantic point clouds to determine the positional relationship nodes between objects in a single-frame image;

[0147] Visual language models are used to understand scene attributes of 3D semantic point clouds and determine the functional attribute nodes in the scene where the robot is located.

[0148] A topological graph of the hierarchical relationships between objects and rooms is constructed based on location relationship nodes and functional attribute nodes to achieve 3D semantic mapping.

[0149] Optionally, an optimization module is also included for:

[0150] The DBSCAN clustering algorithm is used to segment the 3D semantic point cloud into instances, and the K nearest neighbor algorithm is combined to remove noise points and outliers.

[0151] Using the RANSAC algorithm and combining point cloud normal constraints and z-axis height constraints, the ground and ceiling planes in the first scene are extracted and separated.

[0152] An optimized 3D semantic point cloud is generated through clustering, denoising, and planar extraction operations.

[0153] Optionally, the step of performing semantic segmentation on the image data based on the open-set object detection and segmentation algorithm to obtain a semantic segmentation result carrying semantic labels includes:

[0154] The image data is input into an open-set object detection and segmentation algorithm model for pixel-level semantic segmentation to obtain an initial segmentation result; the initial segmentation result includes semantic labels corresponding to objects in the image.

[0155] Noise and invalid detection regions in the segmentation results are removed by area threshold filtering and depth verification;

[0156] The initial segmentation result is subjected to scene-adaptive secondary verification using a lightweight object detection model to correct semantic label misjudgments and obtain the verified semantic segmentation result.

[0157] Optionally, it also includes an index relationship module for:

[0158] The verified semantic segmentation results are back-projected to the three-dimensional world coordinate system through cross-modal coordinate transformation to obtain the global centroid coordinates of the semantic instance;

[0159] The three-dimensional index relationship is determined based on image identifiers, global centroid coordinates, and semantic tags for semantic query retrieval.

[0160] Optionally, a cross-scene module is also included for:

[0161] When the robot crosses the passable boundary of the first scene, it matches the current 3D semantic point cloud with the historical 3D point cloud to achieve spatial continuity.

[0162] Logical consistency verification is performed on nodes and edges of connected scenes in the scene semantic topology, and data reliability is evaluated by dual indicators of overlap rate and semantic consistency.

[0163] When the robot enters the second scene, it determines the environmental information of the second scene and identifies target objects with high distinguishability, determines the scene type of the second scene, and infers the functional attributes of the second scene through zero-shot classification or scene graph matching.

[0164] The corresponding nodes of the functional attributes of the second scene are integrated into the global semantic graph and connected with the corresponding nodes of the first scene.

[0165] Optionally, the method for determining the passable boundary includes:

[0166] Based on the lidar point cloud data, the geometric plane features of the wall in the first scene are fitted to obtain the laser notch features corresponding to the passable boundary.

[0167] Determine the gate region semantic labels in the semantic labels, and verify the spatial rationality of the gate region semantic labels based on the confidence threshold of the gate region semantic labels;

[0168] Project the geometric planar features detected by the lidar and the semantic labels of the gate regions onto the same coordinate system;

[0169] When the spatial overlap between the laser notch feature and the semantic label of the gate region exceeds a preset threshold, it is determined to be a valid passable boundary.

[0170] Optionally, the determination of the stored pose state triggers semantic processing, including:

[0171] Compare the pose information at the current moment with the pose of the previous processed frame that has been stored.

[0172] When the detected pose change exceeds both the angle threshold and the displacement threshold simultaneously, the semantic processing flow is automatically triggered.

[0173] After semantic processing is completed, the stored pose state of the processed frame is updated to the current pose.

[0174] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0175] Reference Figure 5 Based on the methods in the above embodiments, this application provides an electronic device that may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions stored in the memory 530 to execute the methods in the above embodiments.

[0176] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0177] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0178] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0179] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0180] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0181] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0182] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0183] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A semantic mapping method based on scene preset, characterized in that, The method comprises the following steps: acquiring the current pose information of the robot in the first scene, and determining the triggering of semantic processing based on the stored pose state; acquiring the laser radar point cloud data and image data at the current time, performing semantic segmentation on the image data based on an open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic label; projecting the semantic label to the three-dimensional space coordinates of the laser radar point cloud data, and performing coordinate weighted averaging on the position of the dynamic object corresponding to the semantic segmentation result to obtain a 3D semantic point cloud carrying three-dimensional space coordinates, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping; the process of coordinate weighted averaging further comprises: comparing the detected object point cloud with the existing object point cloud in the map to determine whether they are the same object; if yes, the original semantic segmentation instance ID is inherited and incremental coordinate weighted averaging is adopted to update the position of the object; if not, a new ID is generated; when the robot passes through the passable boundary of the first scene, the current 3D semantic point cloud is matched with the historical 3D point cloud to realize spatial continuity; performing logical consistency verification on the nodes and edges of the connected scenes in the scene semantic topology, and evaluating the data reliability through the double indexes of overlap rate and semantic consistency; the determination method of the passable boundary comprises: fitting the wall surface geometric plane features of the first scene based on the laser radar point cloud data to obtain the laser gap features corresponding to the passable boundary; determining the door area semantic label in the semantic label, and verifying the spatial rationality of the door area semantic label based on the confidence threshold of the door area semantic label; projecting the geometric plane features detected by the laser radar and the door area semantic label to the same coordinate system, and determining the effective passable boundary when the spatial overlap degree of the laser gap features and the door area semantic label exceeds the preset threshold.

2. The method of claim 1, wherein, the process of 3D semantic mapping comprises: performing scene graph understanding on the single frame image of the 3D semantic point cloud by using a visual language model to determine the position relationship nodes between objects in the single frame image; performing scene attribute understanding on the 3D semantic point cloud by using a visual language model to determine the functional attribute nodes in the scene where the robot is located; constructing a topology graph of the subordinate relationship between objects and rooms based on the position relationship nodes and the functional attribute nodes to realize 3D semantic mapping.

3. The method of claim 1, wherein, further comprising: performing instance segmentation on the 3D semantic point cloud by using a DBSCAN clustering algorithm, and removing noise points and outliers by combining a K nearest neighbor algorithm; extracting and separating the floor and ceiling planes in the first scene by using a RANSAC algorithm and combining point cloud normal constraints and z-axis height constraints; generating an optimized 3D semantic point cloud through clustering, denoising and plane extraction operations.

4. The method of claim 1, wherein, the process of performing semantic segmentation on the image data based on the open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic label comprises: inputting the image data into an open set target detection segmentation algorithm model for pixel-level semantic segmentation to obtain an initial segmentation result; the initial segmentation result comprises a semantic label corresponding to an object in the image; The noise and invalid detection area of the segmentation result are removed through area threshold filtering and depth verification; The initial segmentation result is subjected to scene adaptive secondary checking through a lightweight target detection model to correct semantic label misjudgment and obtain a checked semantic segmentation result.

5. The method of claim 4, wherein, Further comprising: The checked semantic segmentation result is back-projected to a three-dimensional world coordinate system through cross-modal coordinate transformation to obtain global centroid coordinates of a semantic instance; A three-dimensional index relationship is determined based on image identification, global centroid coordinates and semantic labels for semantic query retrieval.

6. The method of claim 1, wherein, Further comprising: When the robot enters a second scene, environment information of the second scene is determined and a target object with high distinguishability is identified, the scene type of the second scene is determined, and the functional attribute of the second scene is inferred through zero-shot classification or scene graph matching; The corresponding node of the functional attribute of the second scene is integrated into the global semantic graph, and a connection relationship is established with the corresponding node of the first scene.

7. The method of claim 1, wherein, The determination of triggering semantic processing based on the stored pose state comprises: comparing the pose information at the current time with the stored last processing frame pose; when the pose change amount exceeds both the angle threshold and the displacement threshold, automatically triggering the semantic processing flow; after completing the semantic processing, updating the stored processing frame pose state to the current pose. 8.A semantic mapping device based on scene preset, characterized in that, Comprising: a semantic triggering module configured to acquire the pose information of the robot at the current time in the first scene, and determine triggering of semantic processing based on the stored pose state; a semantic segmentation module configured to acquire the laser radar point cloud data and image data at the current time, perform semantic segmentation on the image data based on an open set target detection segmentation algorithm, and obtain a semantic segmentation result carrying a semantic label; a semantic fusion module configured to project the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and perform coordinate weighted averaging on the position of a dynamic object corresponding to the semantic segmentation result, to obtain a 3D semantic point cloud carrying a three-dimensional space coordinate, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping; the process of coordinate weighted averaging further comprises: determining whether the detected object point cloud is the same object as the existing object point cloud in the map by comparing the overlap degree, if yes, inheriting the original semantic segmentation instance ID and adopting incremental coordinate weighted averaging to update the position of the object, if not, generating a new ID; when the robot passes through the passable boundary of the first scene, matching the current 3D semantic point cloud with the historical 3D point cloud to realize spatial continuity; performing logical consistency verification on the nodes and edges of the connected scenes in the scene semantic topology, and evaluating the data reliability through the double indexes of overlap rate and semantic consistency; the determination method of the passable boundary comprises: fitting the wall surface geometric plane feature of the first scene based on the laser radar point cloud data to obtain the laser gap feature corresponding to the passable boundary; determining the door region semantic label in the semantic label, and verifying the spatial rationality of the door region semantic label based on the confidence threshold of the door region semantic label; The geometric plane features detected by the laser radar are projected to the same coordinate system as the door area semantic label, and when the spatial coincidence degree of the laser gap feature and the door area semantic label exceeds a preset threshold, the effective passable boundary is determined.

9. An electronic device, comprising: The application relates to a computer program product comprising: at least one memory for storing a computer program; at least one processor for executing the program stored by the memory, the processor being configured to execute the method according to any one of claims 1-7 when the program stored by the memory is executed.

Images