Semantic mapping method and device based on scene presetting and electronic equipment

Abstract

The invention belongs to the field of computer vision, and particularly discloses a semantic mapping method and device based on scene presetting and electronic device.The method comprises the steps that pose information of a robot at the current moment in a first scene is obtained, and semantic processing is determined to be triggered based on the stored pose state; acquiring laser radar point cloud data and image data at the current moment, and performing semantic segmentation on the image data based on an open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic tag; and projecting the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and performing coordinate weighted average on a dynamic object position corresponding to a semantic segmentation result to obtain a 3D semantic point cloud carrying the three-dimensional space coordinate, the semantic label and a semantic segmentation instance ID so as to realize 3D semantic mapping. According to the method and the device, the effect and the accuracy of 3D semantic mapping can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly, relates to a semantic mapping method and device based on scene preset and an electronic device. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) is an important technology for realizing autonomous positioning and navigation of robots in unknown environments. According to the position during the movement of the robot and the map, the robot performs self-positioning, and meanwhile, an incremental map is built based on the self-positioning, which is widely used in 3D modeling, unmanned driving and many other application scenarios.

[0003] In the field of mobile robots, traditional SLAM mainly obtains three-dimensional point cloud data of the environment through devices such as laser radar, stereo vision or depth sensor to construct a geometric map that can accurately reflect the spatial structure of the scene. Although this can assist the robot in path planning to a certain extent, the single geometric information makes it difficult for the robot to achieve deep understanding of the environment and intelligent decision-making, especially in complex and unstructured scenes such as homes and offices where there are many items placed irregularly. The robot cannot identify object categories and understand the relationship between objects relying only on the geometric map, thereby limiting the further improvement of the intelligent level of the robot.

[0004] In the prior art, three-dimensional point cloud construction is usually performed using laser radar algorithms or vision algorithms, which can cause inconsistent semantic information, inaccurate spatial position, and redundant or missing semantic labels, thereby resulting in poor semantic mapping effect and poor precision. SUMMARY

[0005] In view of the defects of the prior art, the purpose of the present application is to provide a semantic mapping method, device and electronic equipment based on scene preset, aiming to solve the problems of poor semantic mapping effect and poor precision.

[0006] To achieve the above-mentioned purpose, in a first aspect, the present application provides a semantic mapping method based on scene preset, comprising: obtaining the pose information of the robot at the current time in the first scene, and determining to trigger semantic processing based on the stored pose state; obtaining laser radar point cloud data and image data at the current time, performing semantic segmentation on the image data based on an open set target detection segmentation algorithm to obtain a semantic segmentation result carrying semantic labels; projecting the semantic labels to the three-dimensional space coordinates of the laser radar point cloud data, and performing coordinate weighted averaging on the dynamic object positions corresponding to the semantic segmentation result to obtain 3D semantic point cloud carrying three-dimensional space coordinates, semantic labels and semantic segmentation instance ID, so as to realize 3D semantic mapping.

[0007] Optionally, the process of 3D semantic mapping comprises: performing scene graph understanding on the single-frame image of the 3D semantic point cloud by using a visual language model to determine the position relationship nodes between objects in the single-frame image; performing scene attribute understanding on the 3D semantic point cloud by using a visual language model to determine the functional attribute nodes in the scene where the robot is located; constructing a topological graph of the subordination relationship between objects and rooms based on the position relationship nodes and the functional attribute nodes to achieve 3D semantic mapping.

[0008] Optionally, it further comprises: performing instance segmentation on the 3D semantic point cloud by using a DBSCAN clustering algorithm, and removing noise points and outliers in combination with a K-nearest neighbor algorithm; extracting and separating the ground and ceiling planes in the first scene by using a RANSAC algorithm in combination with point cloud normal constraints and z-axis height constraints; generating an optimized 3D semantic point cloud through clustering, denoising and plane extraction operations.

[0009] Optionally, the open-set target detection segmentation algorithm performs semantic segmentation on the image data to obtain a semantic segmentation result carrying semantic labels, comprising: performing pixel-level semantic segmentation on the image data by inputting the image data into an open-set target detection segmentation algorithm model to obtain an initial segmentation result; the initial segmentation result includes semantic labels corresponding to objects in the image; filtering and removing noise and invalid detection areas of the segmentation result through area threshold filtering and depth verification; performing scene-adaptive secondary verification on the initial segmentation result by using a lightweight target detection model to correct semantic label misjudgments and obtain a verified semantic segmentation result.

[0010] Optionally, it further comprises: projecting the verified semantic segmentation result back to a three-dimensional world coordinate system through cross-modal coordinate transformation to obtain global centroid coordinates of semantic instances; determining three-dimensional index relationships based on image identification, global centroid coordinates and semantic labels for semantic query retrieval.

[0011] Optionally, it further comprises: when the robot passes through the passable boundary of the first scene, matching the current 3D semantic point cloud with historical 3D point clouds to achieve spatial continuity; performing logical consistency verification on the nodes and edges of the connected scenes in the scene semantic topology, and evaluating data reliability through double indicators of overlap rate and semantic consistency; When the robot enters a second scene, environment information of the second scene is determined and a target object with high distinguishability is identified, a scene type of the second scene is determined, and a functional attribute of the second scene is inferred through zero-shot classification or scene graph matching; A corresponding node of the functional attribute of the second scene is integrated into a global semantic graph, and a connection relationship is established with a corresponding node of the first scene.

[0012] Optionally, the determination method of the passable boundary comprises: A wall surface geometric plane feature of the first scene is fitted based on laser radar point cloud data, and a laser gap feature corresponding to the passable boundary is obtained; A door region semantic label in the semantic label is determined, and spatial reasonableness of the door region semantic label is verified based on a confidence threshold of the door region semantic label; The geometric plane feature detected by the laser radar and the door region semantic label are projected to the same coordinate system; When a spatial coincidence degree of the laser gap feature and the door region semantic label exceeds a preset threshold, it is determined as an effective passable boundary.

[0013] Optionally, the determination of triggering semantic processing based on the stored pose state comprises: The pose information at the current time is compared with a stored last processing frame pose; When it is detected that the pose change amount exceeds both an angle threshold and a displacement threshold, the semantic processing flow is automatically triggered; After the semantic processing is completed, the stored processing frame pose state is updated to the current pose.

[0014] In a second aspect, the application further provides a semantic mapping device based on scene preset, comprising: A semantic triggering module is configured to acquire pose information of a robot at a current time in a first scene, and determine triggering of semantic processing based on a stored pose state; A semantic segmentation module is configured to acquire laser radar point cloud data and image data at the current time, perform semantic segmentation on the image data based on an open set target detection segmentation algorithm, and obtain a semantic segmentation result carrying a semantic label; A semantic fusion module is configured to project the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and perform coordinate weighted averaging on a dynamic object position corresponding to the semantic segmentation result, to obtain a 3D semantic point cloud carrying a three-dimensional space coordinate, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping.

[0015] In a third aspect, the present application provides an electronic device, comprising: at least one memory configured to store programming; and at least one processor configured to execute the programming stored in the memory, wherein the processor is configured to perform the method described in the first aspect or any possible implementation manner of the first aspect when the programming stored in the memory is executed.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is run on a processor, the processor is caused to perform the method described in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fifth aspect, the present application provides a computer program product, which, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation manner of the first aspect.

[0018] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0019] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects: (1) The present application effectively reduces the interference of dynamic objects on mapping stability by projecting semantic labels to three-dimensional space coordinates and performing coordinate weighted averaging on the positions of dynamic objects, and enhances the semantic consistency of static scenes. In the fusion process, the laser radar point cloud provides high-precision coordinates measured, and the labels provided by the visual semantic segmentation result are corresponded, so that 3D semantic point cloud carrying three-dimensional space coordinates, semantic labels and instance ID can be generated. Not only the accurate identification and positioning of object categories in the scene are realized, but also the semantic richness and structured expression ability of the map are improved, thereby improving the semantic mapping effect and accuracy.

[0020] (2) The present application introduces an incremental semantic verification mechanism, which verifies the label consistency through a lightweight detection model for the second time, and filters outliers by using DBSCAN clustering combined with depth information, so as to eliminate redundant or incorrect labels and further improve the semantic mapping effect.

[0021] (3) The present application reduces the misjudgment caused by the shielding of persons or furniture by fusing the laser geometric features such as wall gaps and the visual semantics such as door labels for double verification, and improves the recognition accuracy of passable boundaries (such as doors).

[0022] (4) The present application establishes a semantic connection edge across scenes to construct a global semantic map with complete topological relationship, so that the robot can not only understand the layout of a single scene, but also master the spatial logical relationship of the entire environment, and realize cross-regional task planning. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the semantic mapping method based on scenario preset provided in the embodiments of this application; Figure 2 This is a flowchart of the single-frame semantic point cloud acquisition process provided in the embodiments of this application; Figure 3 This is a flowchart of the multi-layer scene graph construction process provided in the embodiments of this application; Figure 4 This is a schematic diagram of the semantic mapping device based on scene preset provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0026] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0027] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0029] The embodiments of this application are described below with reference to the accompanying drawings.

[0030] Reference Figure 1 The application provides a semantic mapping method based on scene preset, comprising: S101, acquiring the pose information of the robot at the current time in a first scene, and determining to trigger semantic processing based on the stored pose state; S102, acquiring laser radar point cloud data and image data at the current time, performing semantic segmentation on the image data based on an open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic label; S103, projecting the semantic label to the three-dimensional space coordinates of the laser radar point cloud data, and performing coordinate weighted averaging on the position of a dynamic object corresponding to the semantic segmentation result to obtain a 3D semantic point cloud carrying three-dimensional space coordinates, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping.

[0031] Specifically, in the embodiment of the application, the core purpose of step S101 is to achieve intelligent balance between map updating demand and calculation efficiency. The robot does not perform time-consuming semantic processing on every frame of image, but performs adaptive triggering based on its motion state.

[0032] By continuously receiving pose information (including 3D position and orientation) from an odometry, the pose at the current time is compared with the stored pose (referred to as "processing frame") when the semantic processing is performed last time. When the detected pose change amount exceeds the preset angle threshold (such as 10 degrees) and displacement threshold (such as 0.3 meters) at the same time, it is determined that the robot has moved to a position sufficient to provide a new perspective, and the subsequent semantic processing process is triggered. Through the strategy of the application, the huge calculation redundancy caused by processing continuous similar frames is effectively avoided, and the real-time performance of the system is ensured.

[0033] In step S102, 3D point cloud data generated by a laser radar and RGB image data captured by a camera are synchronously acquired. It should be noted that the RGB image data acquired by the application is acquired by using a binocular camera. Compared with a traditional monocular camera, the binocular camera can provide additional depth information through parallax calculation, which can be redundantly checked or fused with the laser radar point cloud. Especially in the area rich in texture, it can assist to improve the density and integrity of the point cloud.

[0034] Subsequently, the image data is input into the Grounded SAM open-vocabulary semantic segmentation model. In actual application scenarios, due to the object attribute distinction caused by the scene characteristics, it is often necessary to retrain the target recognition segmentation model for different environments. In order to solve the problem of repeated training, the application proposes to use the open set target detection segmentation algorithm Groudded SAM for segmentation, but in actual use, it often encounters the problem of inaccurate recognition and label disorder, and the application also proposes that a lightweight algorithm YOLO-fastest can be used for correction, which only needs to be fine-tuned in different environments, greatly reducing the training cost demand in the early stage of deployment in different environments.

[0035] Further, in step S103, the 2D pixel-level semantic label obtained in S102 can be accurately projected onto the corresponding coordinates of the 3D laser radar point cloud through the pre-calibrated extrinsic matrix between the camera and the laser radar. This makes each 3D point not only have a geometric position (x, y, z), but also be assigned a semantic label (m).

[0036] In the embodiment, the semantic segmentation assigns a unique instance ID (n) to each independent object. By comparing the overlap of newly detected object point clouds with existing object point clouds in the map, it is determined whether it is the same object. If so, the original ID is inherited and an incremental update strategy (such as coordinate weighted average) is used to smooth the position of the object, effectively tracking dynamic objects and eliminating noise; if not, a new ID is assigned. Finally, a 3D semantic point cloud map with the format (x, y, z, m, n) is generated, which not only describes the geometric structure of the environment, but also clearly labels "where is what" (semantic) and "what is the number of this" (instance), realizing true 3D semantic mapping.

[0037] Optionally, the process of the 3D semantic mapping comprises: using a visual language model to perform scene graph understanding on a single frame of the 3D semantic point cloud to determine position relationship nodes between objects in the single frame; using a visual language model to perform scene attribute understanding on the 3D semantic point cloud to determine functional attribute nodes in the scene where the robot is located; constructing a topological graph of the subordinate relationship between objects and rooms based on the position relationship nodes and the functional attribute nodes to realize 3D semantic mapping.

[0038] Specifically, the embodiment realizes high-level cognition of the environment by introducing a visual language model (VLM). First, the VLM analyzes a single frame of image, beyond simple object recognition, and further analyzes the spatial and functional relationship between objects. The visual language model (VLM) is used to analyze the spatial and functional relationship between objects in a single frame of image, solving the defect that the traditional SLAM cannot understand the semantic logic of the environment. After the system inputs an image, the VLM outputs a structured relationship graph (such as "the cup is on the table"), which contains spatial relationships (on / under / near) and functional relationships (used-for / part-of). By storing the graph structure (node = object, edge = relationship), the robot can infer the interaction logic of the scene (such as "the door needs to be pushed open" rather than "bypassed").

[0039] Secondly, the VLM synthesizes the object set and its relationship in the scene, and infers the functional properties of the entire area. This step solves the core problem that the geometric map cannot distinguish the functional area by determining the functional properties of the room where the agent is located (such as "kitchen" or "meeting room") through the VLM. The system combines the object list and the scene graph to dynamically generate room labels through multi-modal reasoning (such as direct classification or question-and-answer analysis). For example, when "stove" and "refrigerator" are detected, it is automatically labeled as "kitchen".

[0040] Finally, a semantic topological graph describing the affiliation relationship between objects and rooms is constructed based on these nodes, so that the map not only contains geometric and object information, but also contains a semantic network that can be used for logical reasoning.

[0041] Optionally, it further comprises: The DBSCAN clustering algorithm is used to perform instance segmentation on the 3D semantic point cloud, and the K nearest neighbor algorithm is used to remove noise points and outliers; The RANSAC algorithm is used in combination with point cloud normal constraints and z-axis height constraints to extract and separate the ground and ceiling planes in the first scene; Through clustering, denoising and plane extraction operations, an optimized 3D semantic point cloud is generated.

[0042] Specifically, the embodiment of the present application performs post-processing optimization on the initially generated 3D semantic point cloud to improve the quality of the map. The DBSCAN clustering algorithm is used to perform instance segmentation on the point cloud, and the K nearest neighbor (KNN) algorithm is used to identify and remove noise points and outliers, so as to obtain a point cloud cluster with clear boundaries and independent instances.

[0043] And, using the RANSAC algorithm and supplemented by the normal constraint and z-axis height constraint, the main horizontal planes such as the ground and the ceiling are efficiently and accurately extracted from the point cloud. Through a series of clustering, denoising and plane extraction operations, an optimized 3D semantic point cloud with clear structure, few noises and distinct main structures is finally output.

[0044] Optionally, the open-set target detection segmentation algorithm performs semantic segmentation on the image data to obtain a semantic segmentation result carrying semantic labels, including: inputting the image data into an open-set target detection segmentation algorithm model to perform pixel-level semantic segmentation, to obtain an initial segmentation result; the initial segmentation result includes semantic labels corresponding to objects in the image; filtering and removing noises and invalid detection areas of the segmentation result through area threshold filtering and depth verification; performing scene-adaptive secondary verification on the initial segmentation result through a lightweight target detection model, to correct semantic label misjudgment, and obtaining a verified semantic segmentation result.

[0045] Specifically, in actual application scenarios, due to the object attribute difference caused by the scene characteristics, it is often necessary to retrain the target recognition segmentation model for different environments. In order to solve the problem of repeated training, the Grounded SAM open-set target detection segmentation algorithm is used for segmentation, but in actual use, the problem of inaccurate recognition and label disorder often occurs, therefore, the lightweight algorithm YOLO-fastest can be used for correction, which only needs to be fine-tuned in different environments, greatly reducing the training cost demand in the early stage of deployment in different environments. The specific steps are as follows: The Grounded SAM model is used to realize open-vocabulary semantic segmentation, and supports dynamic recognition of out-of-training-set object categories (new household appliances, temporary obstacles, etc.). The sensor noise and small size false detection are eliminated through area threshold filtering (mask with less than 400 pixels), and the invalid area (such as floating point cloud caused by reflection) is removed in combination with depth information.

[0046] The YOLO-fastestv2 model (<1MB) is used to fine-tune for specific scenes (home / hospital / warehouse), and the SAM output is verified again to correct obvious mislabeling (such as misidentifying "fire extinguisher" as "water bottle").

[0047] Optionally, the method further includes: projecting the verified semantic segmentation result back to a three-dimensional world coordinate system through cross-modal coordinate transformation to obtain global centroid coordinates of semantic instances; determining a three-dimensional index relationship based on image identification, global centroid coordinates and semantic labels for semantic query retrieval.

[0048] Specifically, this application's embodiments achieve accurate semantic-geometric alignment and dynamic target tracking through a dual mechanism of cross-modal coordinate mapping and temporal data association. At the coordinate mapping level, the 2D semantic segmentation mask is back-projected into the 3D point cloud space using the radar-world calibration matrix obtained from the camera-LiDAR calibration matrix and odometry (Tcam2world). The object's centroid coordinates are calculated (with an error controlled within 3cm), and a three-dimensional index relationship of "image ID - global coordinates - semantic label" is constructed to ensure the unique positioning of each detected instance in the robot's global coordinate system. At the data association level, the Hungarian algorithm is employed, with a complexity of [missing information - likely related to complexity]. Instance matching is performed based on multi-frame detection results. The object ID is maintained across frames using a cost matrix (containing weights for IoU, semantic similarity, motion consistency, etc.). Simultaneously, a Kalman filter (state vector) is integrated. It can predict the motion of dynamic targets, effectively solve the ID jump problem caused by short-term occlusion, and improve the accuracy of trajectory tracking of dynamic obstacles, such as to more than 92%.

[0049] Optionally, it also includes: When the robot crosses the passable boundary of the first scene, it matches the current 3D semantic point cloud with the historical 3D point cloud to achieve spatial continuity. Logical consistency verification is performed on nodes and edges of connected scenes in the scene semantic topology, and data reliability is evaluated by dual indicators of overlap rate and semantic consistency. When the robot enters the second scene, it determines the environmental information of the second scene and identifies target objects with high distinguishability, determines the scene type of the second scene, and infers the functional attributes of the second scene through zero-shot classification or scene graph matching. The corresponding nodes of the functional attributes of the second scene are integrated into the global semantic graph and connected with the corresponding nodes of the first scene.

[0050] Specifically, in this embodiment, when the robot passes through a door or other passable boundary to enter a new scene (the second scene), the current observation is first aligned with the historical 3D point cloud using a point cloud matching algorithm to ensure the continuity of the geometric space. Then, logical consistency verification is performed on the connected nodes and edges in the existing semantic topology (e.g., checking whether the room types on both sides of the door are reasonable), and the reliability of historical data is evaluated. For the new scene, the system identifies typical objects using a Virtual Model (VLM) and infers the functional attributes of the area (e.g., "bedroom") based on these zero-shot examples. Finally, the nodes of this new scene are integrated into the global semantic graph and connected to the first scene, thereby achieving the coherent construction of a large-scale environmental semantic map.

[0051] Through the dual mechanism of cross-room topology verification and new scene intelligent initialization, the two core challenges of long-term autonomous operation of mobile robots in multi-room environments are systematically solved: first, for the scene consistency break problem during cross-area navigation, through point cloud registration (ICP error <10 cm) and semantic topology verification (such as logical verification of connectivity relationship), the positioning drift and topology disorder caused by environmental dynamic changes are eliminated, and the reliability of cross-room path planning is improved by 80%; second, for the cold start problem of real-time cognition of unknown areas, typical object recognition (such as bed → ward) and zero-shot semantic reasoning are used to realize fast function labeling of new scenes within 3 seconds, solving the fundamental limitation of traditional methods that rely on pre-stored maps and cannot adapt to new spaces. The specific steps are as follows: The scene consistency verification problem during cross-room navigation is solved by multi-modal data alignment. When the agent detects a passable boundary (such as a door), the system performs the following operations: first, use the ICP algorithm to accurately register the current observed 3D point cloud with the historical 3D point cloud, ensuring spatial continuity (matching error controlled within 10 cm); then, verify the nodes and edges of the connected rooms in the scene semantic topology graph for reasonableness, for example, verify the logical consistency of typical connectivity relationships such as "living room-door-bedroom"; finally, evaluate the reliability of historical data through dual indicators of overlap rate (requirement >70%) and semantic consistency (such as whether the room types on both sides of the door match).

[0052] Typical object recognition is used to realize fast semantic labeling of unknown areas, solving the problem of real-time cognition in new scenes. When the agent enters a new room space, the system will start the intelligent initialization process: first, scan the environment and prioritize recognizing typical objects with high discriminability, such as detecting "bed" to infer as a ward area, or finding "stove" to determine as a kitchen space; then, use the zero-shot classification ability of VLM (Llama3.2-vision) or the pre-built scene graph matching mechanism for deep reasoning of room function attributes; finally, integrate the new room node seamlessly into the global semantic graph and establish reasonable connection edges with adjacent areas.

[0053] Optionally, the determination method of the passable boundary includes: Fitting the wall surface geometric plane features of the first scene based on the laser radar point cloud data to obtain the laser gap features corresponding to the passable boundary; Determining the door area semantic label in the semantic label, verifying the spatial reasonableness of the door area semantic label based on the confidence threshold of the door area semantic label; Projecting the geometric plane features detected by the laser radar and the door area semantic label to the same coordinate system; When the spatial coincidence degree of the laser gap features and the door area semantic label exceeds the preset threshold, it is determined as an effective passable boundary.

[0054] Specifically, the embodiment adopts a laser radar and vision fusion determination strategy: on the laser side, candidate doorways are found by fitting the wall surface and detecting the continuity interruption area (gap), combined with point cloud density analysis; on the visual side, the "door" label in the image is identified by a semantic segmentation model, and its spatial rationality is verified. The detection results of the two modalities are projected into the same coordinate system, and only when the laser gap feature and the visual semantic label are highly coincident in space, is it finally determined as an effective passable boundary. This cross-validation mechanism greatly improves the accuracy and robustness of detection, effectively avoiding the limitations of a single sensor.

[0055] Through laser-vision multi-modal fusion technology, robust detection and functional state determination of passable doorways are achieved. On the laser radar level, the RANSAC algorithm is used to fit the wall surface geometric features (plane equation fitting error <2 cm), detect the continuity interruption area (gap width >0.8 m and depth mutation >1.2 m), and combine point cloud density analysis (door frame area density drop >50%) to exclude false openings; on the visual semantic level, the "door" label (confidence threshold >0.65) is identified by an open vocabulary segmentation model (Grounded SAM), and its spatial rationality (such as the vertical angle deviation of the door frame and the ground <5°) is verified. The system projects the features of the two modalities into a unified coordinate system through the external parameter matrix (Tcam2world), and when the spatial coincidence degree of the laser gap feature and the visual semantic label exceeds 80%, it is determined as an effective passable door, and the opening and closing state of the door (based on point cloud dynamic analysis) is recorded. The technology achieves a detection accuracy of 98% in hospital corridor tests, with a false detection rate of less than 2 times per thousand square meters, supporting robots to update the passable topology map in real time in dynamic environments.

[0056] The recognition of the door in this embodiment is not only a determination of the passable boundary for the agent, but also a boundary constraint for the agent to understand the scene, i.e., when using VLM to understand the semantic understanding of room attribute information, the boundary door is used as a constraint to avoid using instance objects outside the room space, and the room door is also used as an important determination mark for the agent to enter a new room space, such as the agent being detected to cross the door boundary and being continuously detected for more than t>10s is considered to have been in a new room node.

[0057] Optionally, the determination of the trigger semantic processing based on the stored pose state comprises: comparing the pose information at the current time with the stored last processing frame pose; when the detected pose change amount exceeds both the angle threshold and the displacement threshold, automatically triggering the semantic processing flow; After completing semantic processing, update the stored processing frame pose state to the current pose.

[0058] Specifically, to optimize the utilization of computing resources, the system adopts a dynamic triggering mechanism based on odometry offset: when the pose change of the agent (including displacement Δd>0.2 meters or rotation Δθ>10 degrees) exceeds the preset threshold, the single-frame semantic processing flow is activated. This strategy limits computationally intensive semantic analysis to key motion nodes by monitoring odometry increments in real time, reducing load by more than 50% compared to the full-frame processing mode, while ensuring the spatiotemporal continuity of environmental perception through kinematic constraints. In specific implementation, the quaternion difference is used to calculate the angle offset ), and the historical pose state is cached in a ring buffer to balance the computing efficiency and perception accuracy on resource-constrained platforms. The specific steps are as follows: Use odometry to locate the position of the agent robot; From time 0, read the odometry information in chronological order, the initial odometry information d0, , is saved as the initial processing frame; Compare the pose information at the nth time with the stored pose state. If the angle deflection is greater than 10 degrees and the displacement deflection is greater than 0.2m, trigger semantic processing and update the processing frame pose state.

[0059] Referring to Figure 2 , Figure 2 is a single-frame semantic point cloud acquisition flowchart of an embodiment of the present application, comprising the following steps: Odometry positioning; Initial frame determination; Judge the cumulative pose change of the agent; Use GroundedSAM semantic segmentation; initialize a new room; Continue to the next time; YOLO-fastest verification; Establish the global mapping relationship of the centroid; Identify the passable doors of the room; Remove dynamic objects.

[0060] Referring to Figure 3 , Figure 3 is a multi-layer scene graph construction flowchart of an embodiment of the present application, comprising the following steps: Semantic point cloud mapping; Scene graph understanding; Room attribute reasoning; Hierarchical relationship construction; DBSCAN clustering; Ground extraction; Boundary alignment verification; New room initialization.

[0061] Referring Figure 4 The application also provides a semantic mapping device based on scene preset, comprising: The semantic trigger module 410 is configured to acquire pose information of the robot at the current moment in the first scene, and determine a trigger semantic processing based on a stored pose state; The semantic segmentation module 420 is configured to acquire laser radar point cloud data and image data at the current moment, perform semantic segmentation on the image data based on an open set target detection segmentation algorithm, and obtain a semantic segmentation result carrying a semantic label; The semantic fusion module 430 is configured to project the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and perform coordinate weighted averaging on a dynamic object position corresponding to the semantic segmentation result, to obtain a 3D semantic point cloud carrying a three-dimensional space coordinate, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping.

[0062] Optionally, the process of 3D semantic mapping comprises: Performing scene graph understanding on a single frame image of the 3D semantic point cloud by using a visual language model, to determine a position relationship node between objects in the single frame image; Performing scene attribute understanding on the 3D semantic point cloud by using the visual language model, to determine a functional attribute node in the scene where the robot is located; Constructing a topological graph of the subordination relationship between objects and rooms based on the position relationship node and the functional attribute node, to realize 3D semantic mapping.

[0063] Optionally, the device further comprises an optimization module configured to: Perform instance segmentation on the 3D semantic point cloud by using a DBSCAN clustering algorithm, and remove noise points and outliers in combination with a K nearest neighbor algorithm; Extract and separate a ground plane and a ceiling plane in the first scene by using a RANSAC algorithm and in combination with a point cloud normal constraint and a z-axis height constraint; Generate an optimized 3D semantic point cloud through clustering, denoising and plane extraction operations.

[0064] Optionally, the open set target detection segmentation algorithm performs semantic segmentation on the image data to obtain a semantic segmentation result carrying a semantic label, comprising: Input the image data into an open set target detection segmentation algorithm model for pixel-level semantic segmentation, to obtain an initial segmentation result; the initial segmentation result comprises a semantic label corresponding to an object in the image; Remove noise and invalid detection areas of the segmentation result through area threshold filtering and depth verification. The initial segmentation result is subjected to scene self-adaptive secondary verification by a lightweight target detection model, semantic label misjudgment is corrected, and a verified semantic segmentation result is obtained.

[0065] Optionally, the method further comprises an index relationship module, configured to: The verified semantic segmentation result is back-projected to a three-dimensional world coordinate system through cross-modal coordinate transformation, and a global centroid coordinate of a semantic instance is obtained. A three-dimensional index relationship is determined based on the image identification, the global centroid coordinate, and the semantic label for semantic query retrieval.

[0066] Optionally, the method further comprises a cross-scene module, configured to: When the robot passes through the passable boundary of the first scene, the current 3D semantic point cloud is matched with the historical 3D point cloud to realize spatial continuity; Logical consistency verification is performed on the nodes and edges of the connected scenes in the scene semantic topology, and data reliability is evaluated through double indicators of overlap rate and semantic consistency; When the robot enters the second scene, the environmental information of the second scene is determined, the target object with high distinguishability is identified, the scene type of the second scene is determined, and the functional attribute of the second scene is inferred through zero-shot classification or scene graph matching; The corresponding node of the functional attribute of the second scene is integrated into the global semantic graph, and a connection relationship is established with the corresponding node of the first scene.

[0067] Optionally, the determination method of the passable boundary comprises: The wall surface geometric plane feature of the first scene is fitted based on the laser radar point cloud data, and the laser notch feature corresponding to the passable boundary is obtained; The door region semantic label in the semantic label is determined, and the spatial rationality of the door region semantic label is verified based on a confidence threshold of the door region semantic label; The geometric plane feature detected by the laser radar is projected to the same coordinate system as the door region semantic label; When the spatial coincidence degree of the laser notch feature and the door region semantic label exceeds a preset threshold, it is determined as an effective passable boundary.

[0068] Optionally, the determination of triggering semantic processing based on the stored pose state comprises: The pose information at the current time is compared with the stored last processing frame pose; When the pose change amount is detected to simultaneously exceed an angle threshold and a displacement threshold, the semantic processing flow is automatically triggered; After the semantic processing is completed, the stored processing frame pose state is updated to the current pose.

[0069] It should be understood that the above device is used to execute the method in the above embodiment, the corresponding program module in the device, the implementation principle and technical effect are similar to the description in the above method, the working process of the device can refer to the corresponding process in the above method, and details are not described here.

[0070] With reference to Figure 5 Based on the method in the above embodiment, the electronic device provided in the embodiment of the present application can include a processor (Processor) 510, a communications interface (Communications Interface) 520, a memory (Memory) 530 and a communications bus 540, wherein the processor 510, the communications interface 520 and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can call the logical instructions in the memory 530 to execute the method in the above embodiment.

[0071] In addition, the logical instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part or the technical solutions of the present application that make contributions to the prior art. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0072] Based on the method in the above embodiment, the computer readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program runs on the processor, the processor executes the method in the above embodiment.

[0073] Based on the method in the above embodiment, the computer program product provided in the embodiment of the present application, when the computer program product runs on the processor, makes the processor execute the method in the above embodiment.

[0074] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0075] The method steps in the embodiments of the present application can be implemented in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0076] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0077] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0078] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A semantic mapping method based on scene preset, characterized in that, The method comprises the following steps: acquiring the current pose information of the robot in a first scene, and determining to trigger semantic processing based on the stored pose state; acquiring laser radar point cloud data and image data at the current time, performing semantic segmentation on the image data based on an open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic label; projecting the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and performing coordinate weighted averaging on the position of a dynamic object corresponding to the semantic segmentation result to obtain a 3D semantic point cloud carrying a three-dimensional space coordinate, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping.

2. The method of claim 1, wherein, The process of the 3D semantic mapping comprises: performing scene graph understanding on a single frame image of the 3D semantic point cloud by using a visual language model to determine the position relationship nodes between objects in the single frame image; performing scene attribute understanding on the 3D semantic point cloud by using the visual language model to determine the functional attribute nodes in the scene where the robot is located; constructing a topological graph of the subordination relationship between objects and rooms based on the position relationship nodes and the functional attribute nodes to realize 3D semantic mapping.

3. The method of claim 1, wherein, The method further comprises: performing instance segmentation on the 3D semantic point cloud by using a DBSCAN clustering algorithm, and removing noise points and outliers in combination with a K nearest neighbor algorithm; extracting and separating the floor and ceiling planes in the first scene by using a RANSAC algorithm in combination with point cloud normal constraint and z-axis height constraint; generating an optimized 3D semantic point cloud through clustering, denoising and plane extraction operations.

4. The method of claim 1, wherein, The method of performing semantic segmentation on the image data based on the open set target detection segmentation algorithm to obtain a semantic segmentation result carrying a semantic label comprises: inputting the image data into an open set target detection segmentation algorithm model for pixel-level semantic segmentation to obtain an initial segmentation result; the initial segmentation result comprises semantic labels corresponding to objects in the image; removing noise and invalid detection areas of the segmentation result through area threshold filtering and depth verification; performing scene adaptive secondary verification on the initial segmentation result by using a lightweight target detection model to correct semantic label misjudgment to obtain a verified semantic segmentation result.

5. The method of claim 4, wherein, The method further comprises: back-projecting the verified semantic segmentation result to a three-dimensional world coordinate system through cross-modal coordinate transformation to obtain global centroid coordinates of a semantic instance; determining a three-dimensional index relationship based on image identification, global centroid coordinates and semantic labels for semantic query retrieval.

6. The method of claim 1, wherein, The method further comprises: when the robot passes through a passable boundary of the first scene, matching the current 3D semantic point cloud with historical 3D point clouds to realize spatial continuity; performing logical consistency verification on the nodes and edges of the connected scenes in the scene semantic topology, and evaluating the data reliability through an overlap rate and semantic consistency double index; when the robot enters a second scene, determining the environmental information of the second scene and identifying target objects with high distinguishability, determining the scene type of the second scene, and inferring the functional attributes of the second scene through zero-shot classification or scene graph matching; integrating the corresponding nodes of the functional attributes of the second scene into a global semantic graph, and establishing a connection relationship with the corresponding nodes of the first scene.

7. The method of claim 6, wherein, The method for determining the passable boundary comprises: Fitting a wall surface geometric plane feature of the first scene based on laser radar point cloud data to obtain a laser gap feature corresponding to the passable boundary; Determining a door region semantic label in the semantic label, verifying spatial rationality of the door region semantic label based on a confidence threshold of the door region semantic label; Projecting the geometric plane feature detected by the laser radar and the door region semantic label to the same coordinate system, and determining an effective passable boundary when spatial coincidence degree of the laser gap feature and the door region semantic label exceeds a preset threshold.

8. The method of claim 1, wherein, The method for determining the passable boundary comprises: Comparing the current time pose information with a stored last processing frame pose; When detecting that the pose change amount exceeds both an angle threshold and a displacement threshold, automatically triggering a semantic processing flow; After completing the semantic processing, updating the stored processing frame pose state to the current pose. 9.A semantic mapping device based on scene preset, characterized in that, The method comprises: A semantic trigger module is configured to acquire current time pose information of a robot in a first scene, and determine triggering of semantic processing based on a stored pose state; A semantic segmentation module is configured to acquire laser radar point cloud data and image data at the current time, perform semantic segmentation on the image data based on an open set target detection segmentation algorithm, and obtain a semantic segmentation result carrying a semantic label; A semantic fusion module is configured to project the semantic label to a three-dimensional space coordinate of the laser radar point cloud data, and perform coordinate weighted average on a dynamic object position corresponding to the semantic segmentation result, to obtain a 3D semantic point cloud carrying a three-dimensional space coordinate, a semantic label and a semantic segmentation instance ID, so as to realize 3D semantic mapping.

10. An electronic device, comprising: The method comprises: At least one memory for storing a computer program; At least one processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1-8.

Images