Map building methods and systems based on VLM and SLAM positioning

Abstract

This application discloses a map construction method and system based on VLM and SLAM localization, belonging to the field of map construction technology. The method includes: preprocessing current environmental data collected by multimodal sensors to obtain multimodal target data; performing environmental localization and map construction on the multimodal target data using SLAM technology to obtain an initial 3D environmental map; when blank areas exist in the initial 3D environmental map, inferring environmental semantic information in the current environmental data using a visual language model to obtain a semantically complete map; and fusing the semantically complete map with the initial 3D environmental map to obtain a dense map. Because this application combines visual language models with SLAM technology, it can automatically complete blank areas in the initial 3D environmental map generated by SLAM technology through the understanding and inference of environmental semantic information using visual language models, thereby improving the accuracy and robustness of map construction.

Description

Technical Field

[0001] This application relates to the field of map construction technology, and in particular to a map construction method and system based on VLM and SLAM positioning. Background Technology

[0002] Existing SLAM (Simultaneous Localization and Mapping) technologies collect environmental data through sensors and combine it with feature extraction and matching algorithms to achieve real-time localization and map building of the environment. Common SLAM systems include graph-based SLAM and extended Kalman filtering. Under ideal conditions, these systems can effectively construct 3D maps of the environment and achieve accurate localization.

[0003] However, in complex environments (such as dynamic or occluded environments), existing SLAM technology has certain limitations. It relies on sensor data to build 3D maps, and when data is missing or the positioning error is large, the 3D map is easily incomplete or the accuracy is reduced. Summary of the Invention

[0004] The main purpose of this application is to provide a map building method and system based on VLM and SLAM positioning, which aims to solve the technical problem that traditional SLAM technology relies on sensor data to build 3D maps, and data loss or positioning errors occur in complex or occluded environments, resulting in incomplete 3D maps or reduced accuracy.

[0005] To achieve the above objectives, this application proposes a map construction method based on VLM and SLAM positioning, the method comprising:

[0006] The current environmental data is preprocessed to obtain the corresponding multimodal target data, which is acquired based on preset multimodal sensors;

[0007] The multimodal target data is used to perform environmental localization and map construction using SLAM technology to obtain an initial three-dimensional environment map;

[0008] When there are blank areas in the initial 3D environment map, the environmental semantic information in the current environment data is inferred through a visual language model to obtain a semantically complete map.

[0009] The semantic completion map is fused with the initial 3D environment map to obtain a dense map corresponding to the current environment data.

[0010] In one embodiment, the multimodal sensor includes a lidar and a depth camera, and the step of preprocessing the current environmental data to obtain the corresponding multimodal target data includes:

[0011] Extract sparse point cloud data, color frames, and depth maps from the current environmental data. The sparse point cloud data is obtained by the LiDAR scanning, and the color frames and depth maps are captured by the depth camera.

[0012] The sparse point cloud data is subjected to ground plane removal and downsampling to obtain the target point cloud;

[0013] The depth map is subjected to bilateral filtering for denoising to obtain a denoised depth map;

[0014] The color frame is processed using histogram equalization to obtain a contrast-enhanced image.

[0015] The target point cloud, the denoised depth map, and the contrast-enhanced map are used as multimodal target data.

[0016] In one embodiment, the step of performing environmental localization and map construction on the multimodal target data using SLAM technology to obtain an initial 3D environment map includes:

[0017] The pose is solved based on the contrast enhancement map and the target point cloud to obtain the front-end pose data;

[0018] Based on the front-end pose data, the target point cloud and the denoised depth map are fused using SLAM technology to generate a locally dense 3D mesh.

[0019] Detect blank areas in the current environment data corresponding to view occlusion or feature loss;

[0020] An initial 3D environment map is generated based on the local dense 3D mesh and the blank areas.

[0021] In one embodiment, the step of detecting blank areas corresponding to view occlusion or feature loss in the current environmental data includes:

[0022] Detect missing voxels in the current environment data, where the missing voxels are caused by viewpoint occlusion or feature loss;

[0023] The void ratio is determined based on the missing voxels and the total expected voxels of the current environmental data;

[0024] When the void ratio reaches a preset threshold, the missing voxels are treated as blank areas.

[0025] In one embodiment, the step of obtaining a semantically complete map by reasoning about the environmental semantic information in the current environmental data through a visual language model when there are blank areas in the initial 3D environment map includes:

[0026] When there are blank areas in the initial 3D environment map, the CLIP model is used to perform image recognition on the contrast enhancement map to generate an instance mask.

[0027] The instance mask is projected onto the denoised depth map, and semantic anchor nodes are constructed using an industry database;

[0028] Based on the semantic anchor nodes, the environmental semantic information in the current environmental data is inferred through a visual language model to obtain a semantic completion map.

[0029] In one embodiment, the step of projecting the instance mask onto the denoised depth map and constructing semantic anchor nodes in conjunction with an industry database includes:

[0030] Based on the mapping relationship between image pixels and the denoised depth map, the depth information of the instance mask is extracted to obtain the instance point cloud;

[0031] Traverse the instance point cloud to determine the axis-aligned bounding box and the corresponding center point;

[0032] Based on the class label of the instance mask, the size of the axis-aligned bounding box is compared with the standard size in the industry database to obtain a size confidence mark. The industry database stores standard size information for various devices.

[0033] The class label, the axis-aligned bounding box, the center point, and the size confidence flag are integrated to construct a semantic anchor node.

[0034] In one embodiment, the step of fusing the semantically completed map with the initial 3D environment map to obtain a dense map corresponding to the current environment data includes:

[0035] Based on the current environmental data and the multimodal sensor, determine the geometric TSDF weights;

[0036] Based on the size confidence flag, determine the semantic completion TSDF weights of the semantic completion map;

[0037] Based on the geometric TSDF weights and the semantic completion TSDF weights, the semantic completion map is fused with the initial 3D environment map to obtain a dense map.

[0038] Furthermore, to achieve the above objectives, this application also proposes a map building system based on VLM and SLAM positioning, wherein the system performs the method described above, and the system includes:

[0039] The data acquisition module is used to preprocess the current environmental data to obtain the corresponding multimodal target data, which is acquired based on preset multimodal sensors.

[0040] The map building module is used to perform environmental localization and map building on the multimodal target data using SLAM technology to obtain an initial three-dimensional environment map;

[0041] The semantic reasoning module is used to reason about the environmental semantic information in the current environmental data through a visual language model when there are blank areas in the initial 3D environment map, so as to obtain a semantically complete map.

[0042] The map fusion module is used to fuse the semantically completed map with the initial 3D environment map to obtain a dense map corresponding to the current environment data.

[0043] In one embodiment, the multimodal sensor includes a lidar and a depth camera;

[0044] The data acquisition module is also used to extract sparse point cloud data, color frames and depth maps from the current environmental data. The sparse point cloud data is obtained by the LiDAR scanning, and the color frames and depth maps are obtained by the depth camera.

[0045] The data acquisition module is also used to perform ground plane removal and downsampling processing on the sparse point cloud data to obtain the target point cloud;

[0046] The data acquisition module is also used to perform bilateral filtering and denoising on the depth map to obtain a denoised depth map;

[0047] The data acquisition module is also used to perform image processing on the color frame using histogram equalization technology to obtain a contrast enhancement image;

[0048] The data acquisition module is also used to use the target point cloud, the denoised depth map, and the contrast enhancement map as multimodal target data.

[0049] In one embodiment, the map building module is further configured to perform pose calculation based on the contrast enhancement map and the target point cloud to obtain front-end pose data;

[0050] The map building module is also used to fuse the target point cloud and the denoised depth map based on the front-end pose data using SLAM technology to generate a locally dense 3D mesh.

[0051] The map building module is also used to detect blank areas in the current environment data corresponding to view occlusion or feature loss;

[0052] The map building module is also used to generate an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank area.

[0053] One or more technical solutions proposed in this application have at least the following technical effects: The map construction method based on VLM and SLAM positioning in this application includes: preprocessing the current environmental data to obtain corresponding multimodal target data, wherein the current environmental data is collected based on a preset multimodal sensor; performing environmental positioning and map construction on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map; when there are blank areas in the initial three-dimensional environmental map, reasoning about the environmental semantic information in the current environmental data using a visual language model to obtain a semantically complete map; and fusing the semantically complete map with the initial three-dimensional environmental map to obtain a dense map corresponding to the current environmental data.

[0054] Because this application combines visual language models with SLAM technology, it can automatically fill in the blank areas in the initial 3D environment map generated by SLAM technology by understanding and reasoning about the semantic information of the environment through visual language models, thereby improving the accuracy and robustness of map construction. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating an embodiment of the map construction method based on VLM and SLAM positioning in this application.

[0058] Figure 2 This is a flowchart illustrating a second embodiment of the map construction method based on VLM and SLAM positioning in this application.

[0059] Figure 3 A schematic diagram of the scene for map construction based on VLM and SLAM positioning proposed in this application;

[0060] Figure 4 This is a schematic diagram of the modules of the map building system based on VLM and SLAM positioning in this application.

[0061] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0062] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0063] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0064] It should be noted that the executing entity in this embodiment can be a computing service device with multimodal data acquisition, map building, semantic reasoning, and map fusion functions, such as an industrial robot, an autonomous vehicle, or a surveying drone, or an electronic device capable of performing the above functions, or a map building device based on VLM and SLAM positioning that executes the map building method based on VLM and SLAM positioning of this application (hereinafter referred to as a map building device, for example, it can be mounted on a robot), etc. This embodiment does not limit this. The following uses a map building device as an example to describe this embodiment and the following embodiments.

[0065] Based on this, Embodiment 1 of this application is proposed. This embodiment provides a map construction method based on VLM and SLAM positioning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the map construction method based on VLM and SLAM positioning in this application.

[0066] In this embodiment, the map construction method based on VLM and SLAM positioning includes steps S10 to S40:

[0067] Step S10: Preprocess the current environmental data to obtain the corresponding multimodal target data, wherein the current environmental data is acquired based on preset multimodal sensors.

[0068] It should be noted that a multimodal sensor is a combination of sensors capable of simultaneously acquiring multiple types of data, including cameras, infrared sensors, radar, inertial measurement equipment, etc.

[0069] Understandably, current environmental data is a collection of raw information obtained from the surrounding environment through pre-set multimodal sensors. For example, in autonomous driving scenarios, multimodal sensors may include cameras, radar, inertial measurement units, etc., with cameras acquiring video images, radar collecting distance data, and inertial measurement units collecting vehicle linear velocity, etc.

[0070] It should be understood that multimodal target data is data obtained after preprocessing the original current environment data (e.g., denoising, filtering, etc.). Since the original current environment data may contain noise, inconsistent data formats, or excessively large amounts of data, preprocessing can solve these problems. For example, for image data captured by a camera, denoising can be performed to reduce random noise points in the image and improve image quality; or standardization can be performed to convert data of different formats and ranges into a form suitable for subsequent processing.

[0071] In practice, after the multimodal sensor transmits the collected current environmental data to the map building device, the map building device first performs noise reduction, filtering and other operations on the current environmental data to obtain the multimodal target data to be processed.

[0072] In one feasible implementation, the multimodal sensor includes a lidar and a depth camera. Step S10 in this embodiment may include the following steps: extracting sparse point cloud data, color frames, and depth maps from the current environmental data, wherein the sparse point cloud data is obtained by the lidar scanning, and the color frames and depth maps are obtained by the depth camera; performing ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; performing bilateral filtering denoising on the depth map to obtain a denoised depth map; performing image processing on the color frames using histogram equalization technology to obtain a contrast enhancement map; and using the target point cloud, the denoised depth map, and the contrast enhancement map as multimodal target data.

[0073] It should be noted that LiDAR (Light Detection and Ranging) is a sensor that acquires three-dimensional information about the surrounding environment by emitting laser beams and measuring the laser reflection time. LiDAR can quickly scan the surrounding environment by emitting laser pulses, generating a large amount of point cloud data.

[0074] Sparse point cloud data is a collection of a large number of discrete points obtained by scanning the surrounding environment with LiDAR. These points are relatively sparsely distributed in space and can depict three-dimensional information such as the shape, position and distance of surrounding objects.

[0075] Understandably, ground-plane removal is an operation that removes ground-plane point cloud data from sparse point cloud data. This is because ground-plane point clouds generally interfere with the identification and analysis of target objects. By using specific algorithms (such as planar model fitting methods) to identify and remove point cloud data belonging to the ground plane, other objects in the environment can be highlighted, unnecessary data volume can be reduced, and the efficiency and accuracy of subsequent processing can be improved.

[0076] Downsampling (VoxelGrid) is a method of compressing sparse point cloud data without affecting key information. Since the original sparse point cloud data may contain too many points, downsampling reduces the number of points and thus the data volume without affecting key information.

[0077] For example, the three-dimensional space can be divided into small cubic units (called voxels), assuming each voxel has a side length of 0.05 meters. Within each voxel, a point is selected according to certain rules (such as taking the centroid of points within the voxel) to represent all points within that voxel, thereby reducing the number of points in the point cloud and achieving downsampling. This can reduce the amount of data and improve the efficiency of subsequent processing while preserving the overall shape and features of the sparse point cloud data.

[0078] Therefore, by removing ground planes and downsampling the sparse point cloud data, the target point cloud with ground planes removed and the data volume reduced can be obtained.

[0079] It should be noted that a depth camera (RGB-D camera for short) is a camera that can simultaneously capture color images of a scene and depth information corresponding to each pixel. In addition to providing visual images (color frames) of the scene like a regular camera, it can also sense the distance between objects and the camera, output a depth map, and acquire two-dimensional and three-dimensional data of the environment.

[0080] Color frames are image data captured by a depth camera, measured in pixels. Each pixel contains both brightness and color information. Color frames are typically represented in RGB (red, green, blue) color models, reflecting the color distribution in the environment.

[0081] A depth map is an image captured by a depth camera that represents the distance from each point in the scene to the camera. The value of each pixel represents the distance from the scene point corresponding to that pixel to the camera. Combining depth maps with color frames can provide a more comprehensive description of the 3D structure of the environment.

[0082] Understandably, bilateral filtering is a non-linear filtering method. Depth maps are susceptible to noise interference during acquisition. Bilateral filtering considers not only the spatial distance between pixels but also the differences in their grayscale values ​​(depth values ​​in the depth map). For each pixel in the depth map, bilateral filtering determines weights based on the similarity of its surrounding pixels' spatial location and depth values, then calculates a weighted average to obtain the filtered depth value. This allows for noise removal while effectively preserving important features such as edges in the depth map, recognizing these differences without over-smoothing edges.

[0083] It should be understood that Contrast Limited Adaptive Histogram Equalization (CLAHE) is an image processing technique that adjusts the histogram distribution of color frames to make the pixel values ​​of the color frames more evenly distributed, thereby enhancing the contrast of the image and obtaining a contrast-enhanced image.

[0084] For example, to facilitate understanding of the above data acquisition and processing process, this embodiment and subsequent embodiments use a robot as an example for illustration, but do not limit the scope of this solution. First, multimodal sensors collect environmental information, and sparse point cloud data is collected and output via LiDAR. Color frames are captured and output via an RGB-D camera. With depth map The output angular velocity is acquired through an inertial measurement unit (IMU). With linear acceleration Then, the above data is preprocessed, and a Bilateral Filter is applied to the depth map to remove noise, resulting in a denoised depth map. Ground plane removal and downsampling are performed on the sparse point cloud data to obtain the target point cloud. Applying CLAHE to the color frame improves low-light details, resulting in a contrast-enhanced image. The data obtained after the above collection and processing is aligned using hardware timestamps and software interpolation to form a synchronization data packet. .

[0085] In this embodiment, for data acquired by multimodal sensors, point cloud data can be optimized by removing ground planes and downsampling to reduce interference and data volume; denoising the depth map and enhancing the contrast of color frames can improve data quality.

[0086] Step S20: Use SLAM technology to perform environmental localization and map construction on the multimodal target data to obtain an initial three-dimensional environment map.

[0087] It should be noted that SLAM technology is a technique that enables robots and other devices to simultaneously build an environmental map and determine their own position in an unknown environment. It integrates information from multiple sensors, such as LiDAR, cameras, and IMUs. Through SLAM technology, the robot's pose can be accurately estimated and a consistent map can be constructed.

[0088] Understandably, the initial 3D environment map is a preliminary map obtained by using SLAM technology to perform environmental localization and map construction on multimodal target data. It is obtained using multimodal sensor data such as vision, laser, and inertial sensors through SLAM technology.

[0089] In the specific implementation, after obtaining the preprocessed data mentioned above, the front end of SLAM technology is used to extract features, and the back end is processed based on probabilistic algorithms to match and fuse the data, determine the location of the map building device and build the map, thereby obtaining the initial three-dimensional environment map.

[0090] In one feasible implementation, step S20 of this embodiment may include the following steps: performing pose calculation based on the contrast enhancement map and the target point cloud to obtain front-end pose data; based on the front-end pose data, fusing the target point cloud and the denoised depth map using SLAM technology to generate a locally dense 3D mesh; detecting blank areas corresponding to viewpoint occlusion or feature loss in the current environment data; and generating an initial 3D environment map based on the locally dense 3D mesh and the blank areas.

[0091] It should be noted that the front-end pose data is the result of visual inertial tracking at the front end in SLAM technology, and includes the robot's position and orientation information in 3D space. In SLAM-based map building, the front-end pose data can guide the fusion process of the target point cloud and the denoised depth map, ensuring the accuracy of the fusion.

[0092] For example, ORB (Oriented Fast and Rotated BRIEF) feature points can first be extracted from the contrast enhancement map, while simultaneously acquiring the measurement data from the aforementioned IMU. Then, the IMU data undergoes pre-integration processing to obtain relative motion constraints. The ORB feature points and relative motion constraints are then constructed into an optimization problem. Within a sliding window, an optimization algorithm (such as the Levenberg-Marquardt algorithm) is used to solve for the camera pose, minimizing the reprojection error of the feature points.

[0093] Meanwhile, for the target point cloud, a local sub-graph can first be determined. Then, for each point in the target point cloud, the nearest point (or set of points) is found in the local sub-graph. Based on these corresponding point pairs, an initial transformation estimate is determined. This transformation estimate is then iteratively improved through ICP (Iterative Closest Point) to minimize the distance between the two point clouds, ultimately obtaining the initial attitude value of the lidar.

[0094] Based on the camera pose and the initial pose value of the LiDAR, the current front-end pose data of the robot can be obtained by solving the front-end of the SLAM technology.

[0095] Understandably, a locally dense 3D mesh is a 3D structure generated by fusing the target point cloud and the denoised depth map using SLAM technology. It is constructed within a local area and contains relatively dense 3D information, accurately representing the geometry of the local environment, the position of objects, and other information. Locally dense 3D meshes are an important component in constructing a complete 3D environment map.

[0096] For example, in the process of constructing a locally dense 3D mesh, Surfel (Surface Element, an element representing a 3D surface that includes the surface's position, normal direction, and some other attributes) elements can first be extracted from the target point cloud and the denoised depth map to determine the attributes of each Surfel element. Then, based on the front-end pose data estimated by the front end, the target point cloud is merged... and the denoised depth map The Surfel elements from different perspectives or at different times are fused. For example, if a new target point cloud or a denoised depth map comes in, its Surfel elements are extracted and matched and fused with the Surfel elements in an existing locally dense mesh. Factors such as distance and normal direction can be used to determine whether to merge Surfel elements or update their properties, gradually constructing a locally dense 3D mesh. .

[0097] It should be understood that blank areas are regions in the current environmental data that are not covered or accurately described by normal sensor data due to viewpoint occlusion or feature loss. Viewpoint occlusion may be caused by objects in the environment obscuring each other, such as a large box blocking a wall behind it, making it difficult for sensors (such as cameras or LiDAR) to acquire information about the obscured portion. Feature loss may be due to certain areas in the environment lacking sufficient recognizable features such as texture and shape, making it difficult for sensors to accurately describe these areas, thus creating information gaps when building maps or modeling the environment.

[0098] In one feasible implementation, the step of detecting blank areas corresponding to view occlusion or feature loss in the current environmental data in this embodiment includes: detecting missing voxels in the current environmental data, the missing voxels being caused by view occlusion or feature loss; determining the void ratio based on the missing voxels and the total expected voxels of the current environmental data; and when the void ratio reaches a preset threshold, treating the missing voxels as blank areas.

[0099] It's important to note that a voxel is the smallest unit in three-dimensional space. In environmental data, due to viewpoint occlusion or feature loss, some voxel information cannot be accurately obtained or may not exist; these voxels are called missing voxels. The total expected voxels is a measure of the number of voxels that the current environmental data should ideally contain. It represents the total number of voxels required to fully describe the current environment without any occlusion or feature loss. For example, if the sensor's scanning range is known to be a specific three-dimensional spatial region, the theoretically expected total number of voxels can be calculated based on the size of this region and the voxel division precision.

[0100] Understandably, the void ratio is an indicator used to measure the completeness of current environmental data. It is calculated by the ratio of missing voxels to total expected voxels (i.e., missing voxels / total expected voxels = void ratio). The void ratio reflects the degree of voids (i.e., missing information portions) in environmental data caused by missing voxels.

[0101] The preset threshold is a numerical limit used to determine whether there are too many missing voxels. When detecting the porosity in the current environmental data, the calculated porosity is compared with this preset threshold. If the porosity reaches or exceeds this preset threshold, it means that there are too many missing voxels in the environmental data, requiring special treatment. In this case, the corresponding missing voxels are treated as blank areas. For example, if the porosity is 0, it means that the environmental data is complete and there are no missing voxels; if the preset threshold is 0.3, and the calculated porosity is 0.35, it indicates that there is a lot of missing information in the environmental data, which may affect the accuracy of subsequent map construction. This meets the condition of marking missing voxels as blank areas, thus triggering subsequent semantic reasoning.

[0102] For example, during the construction of a locally dense 3D mesh, the void ratio is continuously calculated. When viewpoint occlusion or feature loss leads to a void ratio increase... At that time, semantic reasoning is triggered.

[0103] In this embodiment, by solving the front-end pose data, the robot's initial position and attitude in the environment can be accurately determined. Based on the front-end pose data, SLAM technology is used to fuse the target point cloud and the denoised depth map to generate a locally dense 3D mesh, which helps to more accurately construct a local environment model and obtain richer environmental information. At the same time, detecting blank areas and generating an initial 3D environment map based on the locally dense 3D mesh and blank areas enables a more complete map construction.

[0104] Step S30: When there are blank areas in the initial three-dimensional environment map, the environmental semantic information in the current environment data is inferred through the visual language model to obtain a semantically complete map.

[0105] It should be noted that the Visual Language Model (VLM) is an artificial intelligence model that combines computer vision and natural language processing to understand and reason about the semantic information of visual information (such as images, videos, etc.).

[0106] Understandably, environmental semantic information refers to the semantically meaningful information contained in the current environmental data. Environmental semantic information can describe high-level information such as what objects are in the environment, the relationships between objects (such as relative positions, interactions, etc.), and the overall scene type of the environment (such as the scene in which the robot is located).

[0107] Understandably, a semantic completion map is a map obtained by reasoning about the semantic information of the environment using a visual language model, given that there are blank areas in the initial 3D environment map. This map fills in the blank areas in the initial 3D environment map, using semantic information to fill in the missing parts, making the map more complete. For example, in a blank area in the initial 3D environment map, the semantic completion map might infer from the surrounding semantic information that this area should be part of a bookshelf, thus improving the overall structure and information completeness of the map.

[0108] Furthermore, since there may be some errors in the aforementioned front-end pose data generation process, such as visual reprojection error (in the aforementioned SLAM technology, the feature points in the three-dimensional space are projected onto the image plane according to the estimated camera pose, and the error between this projected point and the actual detected feature point is the reprojection error), IMU pre-integration error (in the aforementioned IMU pre-integration process, the error between the actual motion and the motion estimated by pre-integration), ICP point-to-surface error (the error that exists when aligning the point cloud by iterating the nearest point through ICP), etc., these errors are constructed into a factor graph, where each factor corresponds to an error term, and the factor graph can also be optimized through semantic completion.

[0109] In a specific implementation, when there are blank areas in the initial 3D environment map, the VLM can be used to mine the environmental semantic information in the current environment data by leveraging its powerful semantic understanding and reasoning capabilities, and obtain the semantic completion map corresponding to the missing voxels.

[0110] In one feasible implementation, step S30 of this embodiment may include the following steps: when there are blank areas in the initial three-dimensional environment map, perform image recognition on the contrast enhancement map using the CLIP model to generate an instance mask; project the instance mask onto the denoised depth map and construct semantic anchor nodes in conjunction with an industry database; based on the semantic anchor nodes, infer the environmental semantic information in the current environment data using a visual language model to obtain a semantic completion map.

[0111] It should be noted that CLIP (Contrastive Language-Image Pretraining) is a model that can recognize various visual features in contrast-enhanced images and associate these features with corresponding semantic concepts based on pre-trained knowledge. For example, it can identify objects in an image as "chairs" or "tables," and can distinguish the categories and attributes of different objects, providing basic image recognition results for subsequent operations.

[0112] An instance mask is an image representation used to mark regions of specific objects in an image. The CLIP model generates instance masks after performing image recognition on the contrast enhancement map. This mask explicitly indicates the location and extent of each identified object in the contrast enhancement map. For example, if there are multiple chairs in the image, the instance mask will mark the area of ​​each chair, as if drawing a specific "outline" for each chair on the image. This "outline" is a manifestation of the instance mask, used to distinguish different instance objects.

[0113] Understandably, industry databases are collections of data related to the construction of 3D environment maps and object recognition. These data include information such as the standard size, shape, material, common uses, and functional attributes of objects, and store standard size information for various devices.

[0114] For example, firstly, different objects in the contrast enhancement image can be segmented using an image segmentation model (Segment-Anything, SAM) to generate instance masks. Then, using the text encoder part of the CLIP pre-trained model, each equipment name in the industrial equipment dictionary (e.g., "valve", "pump", "transformer") of the industry database is encoded to obtain the corresponding semantic vector. For instance masks The image encoding part of the CLIP model converts it into a semantic vector. During the recognition process, a semantic vector corresponding to each device name is generated. Calculate its semantic vector relative to the instance mask. The cosine similarity is compared with a threshold to determine the recognition result; for example, if cos( , If the value is greater than 0.3 (the set threshold), the corresponding device is considered successfully identified. After successfully identifying the corresponding device, the class label c can be determined directly based on the successfully identified device name; the confidence level q of the identification result is determined based on factors such as the closeness of the cosine similarity value to the threshold, for example, the cosine similarity value itself can be used as the confidence level; finally, the image recognition result is output as follows: <class label c, confidence level q, instance mask>. >.

[0115] It should be understood that a semantic anchor node is a node with semantic information constructed by projecting an instance mask onto a denoised depth map and combining it with an industry database. It integrates device instance information from the instance mask, depth information from the depth map, and relevant semantic knowledge from the industry database.

[0116] In another feasible implementation, the step of projecting the instance mask onto the denoised depth map and constructing semantic anchor nodes in conjunction with an industry database, as described in this embodiment, includes: extracting the depth information of the instance mask according to the mapping relationship between image pixels and the denoised depth map to obtain an instance point cloud; traversing the instance point cloud to determine the axis-aligned bounding box and the corresponding center point; comparing the size of the axis-aligned bounding box with the standard size in the industry database according to the class label of the instance mask to obtain a size confidence flag, wherein the industry database stores standard size information of various devices; and integrating the class label, the axis-aligned bounding box, the center point, and the size confidence flag to construct a semantic anchor node.

[0117] It's important to note that image pixels are the basic building blocks of an image, containing information such as color and brightness. A depth map, on the other hand, reflects the distance information of each point in the scene from the camera. The mapping relationship between image pixels and the denoised depth map describes how to find the corresponding depth information in the depth map from the pixel information of the image. Depth information refers to the depth values ​​of each point within the area defined by the instance mask, used to determine the spatial position and shape of the instance in the 3D scene.

[0118] An instance point cloud is a collection of points representing the surface of an object or scene in 3D space. It is obtained by extracting depth information from an instance mask and represents the set of points of an instance in 3D space. The instance point cloud contains the 3D spatial location information of the instance mask.

[0119] Understandably, an axis-aligned bounding box (AABB) is a simple geometric shape used to represent the boundary of a 3D object. For an object represented by an instance point cloud, the smallest axis-aligned cuboid that completely contains the object's point cloud is found by traversing the instance point cloud; this cuboid is the axis-aligned bounding box. The center point refers to the geometric center point inside the axis-aligned bounding box, used to represent the center position of the device. It is calculated by taking the average of the minimum and maximum coordinates of the bounding box along the three coordinate axes.

[0120] For example, based on the mapping relationship between image pixels and the noisy depth map, the depth information corresponding to the pixels in the instance mask can be extracted to construct three-dimensional coordinates, thereby converting the two-dimensional mask information into a three-dimensional instance point cloud. This yields the point cloud representation of the corresponding device in 3D space. Then, by traversing the points in the point cloud, the minimum and maximum values ​​along each coordinate axis are found to determine the range and center point of the axis-aligned bounding box (AABB). .

[0121] It should be understood that the size reliability flag is an identifier obtained by comparing the dimensions of the axis-aligned bounding box with standard dimensions in an industry database. It is used to assess the reliability of dimensional information obtained from instances. If the dimensions of the axis-aligned bounding box are close to the standard dimensions for the corresponding category in the industry database, the flag indicates that the dimensions are reliable; if the difference is significant, it may indicate a problem with detection or calculation, or that there are special circumstances surrounding the object.

[0122] For example, an industry database corresponding to different types of industrial equipment can be pre-established to store standard size information for various types of equipment. After identifying the equipment type, the standard size of the equipment type (determined by the class label) can be retrieved from the industry database. Then the AABB dimensions of the instance point cloud can be compared. The confidence level between the size and the standard size is determined by the comparison result. If the comparison is satisfactory, the size is considered reliable, and the size confidence flag β is set to 1; otherwise, it is set to 0. For example, if... Set the size confidence flag β=1, otherwise set it to 0. Finally, integrate the device's class label, size confidence flag, AABB, and center point information into a semantic anchor node. This is used by the aforementioned factor graph for subsequent completion and optimization.

[0123] After constructing the semantic anchor nodes, missing detection is performed first. The entire initial 3D environment map can be scanned using the Signed Distance Field (SDF) to perform SDF statistics. By analyzing the distance, the empty areas in the scene are found, and the empty set Ω is obtained.

[0124] Then, using a three-dimensional diffusion model (Latent-Diffusion-3D) to... Given the category information (i.e., class labels, which contain the device's category information) and the size vector (i.e., the size of the axis-aligned bounding box, which contains the device's size information), and the coordinate system of the void voxels in the void set, a complete voxel block capable of filling the voids is generated iteratively. Simultaneously, during the generation of the completed voxel block, to prevent the generated voxel block from deviating from its surroundings (drift), a consistency loss can be introduced. This involves calculating the changes in the SDF values ​​around the completed voxel block (e.g., calculating the SDF difference between adjacent voxels) to obtain the surrounding SDF difference field. The consistency of the generated voxel with its surroundings is measured by comparing the completed voxel block with the SDF difference field; for example, the consistency between the completed voxel block and the surrounding SDF difference field is measured by the diffusion reconstruction residual (a measure of the difference between the actual generation result and the expected result during voxel completion using a 3D diffusion model). Then it is consistent with the surrounding environment.

[0125] After successfully generating the above-mentioned complete primitive blocks, the confidence level q, size confidence flag β, and diffusion reconstruction residual generated in the above VLM can be integrated. Calculate the completion confidence γ:

[0126] ;

[0127] Only when γ > 0.6 (a threshold can be set) can the complete primitive blocks, semantic information of the device (class labels, etc.), and related spatial location information be written into the hole set. Generate semantic completion map .

[0128] In this embodiment, by combining depth map information with instance masks to construct instance point clouds, the position and shape of objects in 3D space can be determined more accurately. Axis-aligned bounding boxes and center points further clarify the spatial extent and central position of the object, helping to improve the accuracy of object localization. At the same time, the constructed semantic anchor nodes can integrate multiple important pieces of information, which helps to achieve semantic understanding at a higher level.

[0129] Step S40: Fuse the semantic completion map with the initial 3D environment map to obtain a dense map corresponding to the current environment data.

[0130] It should be noted that the dense map is generated by fusing the aforementioned semantically completed map with the initial 3D environment map, resulting in a map that can accurately describe the environment. Compared to the initial 3D environment map, the dense map has a higher information density and can provide a more accurate environmental description, including small obstacles, subtle terrain changes, and the completion of blank areas due to viewpoint occlusion or feature loss.

[0131] In the technical solution provided in this embodiment, after the multimodal sensor transmits the collected current environmental data to the map building device, the map building device first performs noise reduction and filtering operations on the current environmental data to obtain multimodal target data to be processed. Then, the front end uses SLAM technology to extract features from the multimodal target data, and the back end processes the data based on probabilistic algorithms to match and fuse the data, determine the location of the map building device, and build a map, thereby obtaining an initial 3D environmental map. When blank areas are detected in the initial 3D environmental map, the current environmental data is input into a visual language model to analyze and reason about the environmental semantic information, such as identifying object types and positional relationships, to construct a semantic completion map. Then, the semantic completion map is fused with the initial 3D environmental map. Information from the semantic completion map can be filled into the blank areas of the initial 3D environmental map according to a certain weight allocation, making the map information more complete, and finally obtaining a dense map corresponding to the current environmental data. Because this embodiment combines a visual language model with SLAM technology, the understanding and reasoning of environmental semantic information by the visual language model can automatically fill in the blank areas in the initial 3D environmental map generated by SLAM technology, thereby improving the accuracy and robustness of map building.

[0132] Based on the first embodiment of this application described above, a second embodiment of this application is proposed. In this second embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 2 of the map construction method based on VLM and SLAM positioning in this application.

[0133] This example, step S40, also includes steps S41 to S43:

[0134] Step S41: Determine the geometric TSDF weights based on the current environmental data and the multimodal sensor.

[0135] It should be noted that the geometric TSDF (Truncated Signed Distance Function) weights are parameters determined based on current environmental data and multimodal sensor data. Multimodal sensors can collect various types of information from the environment, such as depth and color. The current environmental data includes various raw data from the environment, such as the coordinates of points at different locations. When constructing a 3D environmental map, the importance of each data component can be considered from a geometric perspective. Appropriately setting the geometric TSDF weights allows for a more accurate representation of the geometric structure information in the environment.

[0136] Step S42: Determine the semantic completion TSDF weights of the semantic completion map based on the size confidence flag.

[0137] Step S43: Based on the geometric TSDF weights and the semantic completion TSDF weights, fuse the semantic completion map with the initial 3D environment map to obtain a dense map.

[0138] It should be noted that since the semantic completion map is obtained by inferring environmental semantic information from the initial 3D environment map with blank areas through a visual language model, the relative importance of the information in the semantic completion map can be reflected in the fusion process. Therefore, the semantic completion TSDF weights can be set based on factors such as the quality and reliability of the semantic completion map.

[0139] For example, for an initial 3D environment map, geometric TSDF weights can be set. Based on the completion credibility mentioned above, set the semantic completion TSDF weights. Finally, a uniform dense map is output. .

[0140] Furthermore, based on the aforementioned semantic anchor nodes, connections with other factors in the factor graph can be defined, and new edges (factors) can be constructed to connect semantic anchor nodes with other related nodes (such as pose nodes, feature point nodes, etc.). Then, when optimizing the factor graph, geometric factors and semantically related factors are considered. By adjusting the variables in the graph (such as pose, semantic information parameters, etc.), the objective function containing semantic factors is minimized, achieving joint geometric-semantic optimization and suppressing the drift of the generated dense map.

[0141] In the technical solution provided in this embodiment, by combining VLM semantic understanding with SLAM localization, environmental semantic information is used to correct errors and optimize accuracy when localization accuracy deviates, thereby improving the accuracy and robustness of the 3D map. VLM improves localization accuracy in real time by identifying information such as device location and function in the environment, especially in environments with occlusion, changes in lighting, or interference from dynamic objects.

[0142] For example, to help understand the implementation process of the map construction method based on VLM and SLAM localization obtained after Embodiments 1 and 2 above, please refer to... Figure 3 , Figure 3 This is a schematic diagram of a scene for map construction based on VLM and SLAM localization proposed in this application; specifically:

[0143] The map-building device is located on the ground, and the entire process is completed in parallel by two main lines: SLAM and VLM. SLAM first uses laser point clouds to construct a sparse point cloud map, resembling a skeleton. Then, color frames and depth maps are acquired from RGB images for texture and semantic reference. Due to factors such as viewpoint occlusion, sensor blind spots, or reflections in this scene, the sparse point cloud map may contain holes (missing regions). The raw depth information given in the depth map is also sparse or incomplete. At this point, the map-building device performs a semantic completion stage using VLM inference. The Visual Language Model (VLM) infers from the color frames, discovering "there should be a box here," thus inferring the semantic category corresponding to the geometrically missing regions. VLM not only provides "there is a box here," but also its approximate size and shape, generating the completed depth values, resulting in a dense depth map.

[0144] The process is as follows: SLAM first builds a geometric skeleton, then detects blank areas, VLM uses environmental semantic reasoning to fill in the blanks, and finally integrates the semantically completed geometric / semantic information back into the map, ultimately outputting a dense 3D map that has both complete geometry and semantic labels. By combining visual language models with SLAM technology, blank areas in the initial 3D environment map generated by SLAM technology can be automatically filled in, thereby improving the accuracy and robustness of map construction.

[0145] refer to Figure 4 The aforementioned map building method based on VLM and SLAM localization is applied to a map building system based on VLM and SLAM localization. Figure 4 This is a schematic diagram of the modules of the map building system based on VLM and SLAM positioning in this application.

[0146] like Figure 4 As shown, the system described in this embodiment includes: a data acquisition module 10, a map construction module 20, a semantic reasoning module 30, and a map fusion module 40.

[0147] The data acquisition module 10 is used to preprocess the current environmental data to obtain the corresponding multimodal target data, which is acquired based on preset multimodal sensors.

[0148] It should be noted that a multimodal sensor is a combination of sensors capable of simultaneously acquiring multiple types of data, including cameras, infrared sensors, radar, inertial measurement equipment, etc.

[0149] Understandably, current environmental data is a collection of raw information obtained from the surrounding environment through pre-set multimodal sensors. For example, in autonomous driving scenarios, multimodal sensors may include cameras, radar, inertial measurement units, etc., with cameras acquiring video images, radar collecting distance data, and inertial measurement units collecting vehicle linear velocity, etc.

[0150] It should be understood that multimodal target data is data obtained after preprocessing the original current environment data (e.g., denoising, filtering, etc.). Since the original current environment data may contain noise, inconsistent data formats, or excessively large amounts of data, preprocessing can solve these problems. For example, for image data captured by a camera, denoising can be performed to reduce random noise points in the image and improve image quality; or standardization can be performed to convert data of different formats and ranges into a form suitable for subsequent processing.

[0151] In practice, after the multimodal sensor transmits the acquired current environmental data to the data acquisition module, the data acquisition module first performs noise reduction, filtering and other operations on the current environmental data to obtain the multimodal target data to be processed.

[0152] In one embodiment, the multimodal sensor of this embodiment includes a lidar and a depth camera; the data acquisition module 10 is further configured to extract sparse point cloud data, color frames, and depth maps from the current environmental data, wherein the sparse point cloud data is obtained by the lidar scanning, and the color frames and depth maps are obtained by the depth camera; the data acquisition module 10 is further configured to perform ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; the data acquisition module 10 is further configured to perform bilateral filtering denoising on the depth map to obtain a denoised depth map; the data acquisition module 10 is further configured to perform image processing on the color frames using histogram equalization technology to obtain a contrast enhancement map; the data acquisition module 10 is further configured to use the target point cloud, the denoised depth map, and the contrast enhancement map as multimodal target data.

[0153] It should be noted that LiDAR is a sensor that acquires three-dimensional information about the surrounding environment by emitting laser beams and measuring the laser reflection time. LiDAR can quickly scan the surrounding environment by emitting laser pulses, generating a large amount of point cloud data.

[0154] Sparse point cloud data is a collection of a large number of discrete points obtained by scanning the surrounding environment with LiDAR. These points are relatively sparsely distributed in space and can depict three-dimensional information such as the shape, position and distance of surrounding objects.

[0155] Understandably, ground plane removal is an operation that removes ground plane point cloud data from sparse point cloud data. This is because ground plane point clouds generally interfere with the identification and analysis of target objects. By using specific algorithms (such as planar model fitting methods) to identify and remove point cloud data belonging to the ground plane, other objects in the environment can be highlighted, unnecessary data volume can be reduced, and the efficiency and accuracy of subsequent processing can be improved.

[0156] Downsampling is a method of compressing sparse point cloud data without affecting key information. Since the original sparse point cloud data may contain too many points, downsampling reduces the number of points and thus the data volume without affecting key information.

[0157] Therefore, by removing ground planes and downsampling the sparse point cloud data, the target point cloud with ground planes removed and the data volume reduced can be obtained.

[0158] It should be noted that a depth camera is a camera that can simultaneously capture color images of a scene and depth information corresponding to each pixel. In addition to providing visual images (color frames) of the scene like a regular camera, it can also sense the distance between objects and the camera, output a depth map, and acquire two-dimensional and three-dimensional data of the environment.

[0159] Color frames are image data captured by a depth camera, measured in pixels. Each pixel contains both brightness and color information. Color frames are typically represented in RGB (red, green, blue) color models, reflecting the color distribution in the environment.

[0160] A depth map is an image captured by a depth camera that represents the distance from each point in the scene to the camera. The value of each pixel represents the distance from the scene point corresponding to that pixel to the camera. Combining depth maps with color frames can provide a more comprehensive description of the 3D structure of the environment.

[0161] Understandably, bilateral filtering is a non-linear filtering method. Depth maps are susceptible to noise interference during acquisition. Bilateral filtering considers not only the spatial distance between pixels but also the differences in their grayscale values ​​(depth values ​​in the depth map). For each pixel in the depth map, bilateral filtering determines weights based on the similarity of its surrounding pixels' spatial location and depth values, then calculates a weighted average to obtain the filtered depth value. This allows for noise removal while effectively preserving important features such as edges in the depth map, recognizing these differences without over-smoothing edges.

[0162] It should be understood that histogram equalization is an image processing technique that adjusts the histogram distribution of color frames to make the pixel values ​​of the color frames more evenly distributed, thereby enhancing the contrast of the image and obtaining a contrast-enhanced image.

[0163] In this embodiment, for the data collected by the multimodal sensor, the point cloud data can be optimized by removing the ground plane and downsampling, which can reduce interference and data volume; the data quality can be improved by denoising the depth map and enhancing the contrast of the color frame.

[0164] The map building module 20 is used to perform environmental localization and map building on the multimodal target data using SLAM technology to obtain an initial three-dimensional environment map.

[0165] It should be noted that SLAM technology is a technique that enables robots and other devices to simultaneously build an environmental map and determine their own position in an unknown environment. It integrates information from multiple sensors, such as LiDAR, cameras, and IMUs. Through SLAM technology, the robot's pose can be accurately estimated and a consistent map can be constructed.

[0166] Understandably, the initial 3D environment map is a preliminary map obtained by using SLAM technology to perform environmental localization and map construction on multimodal target data. It is obtained using multimodal sensor data such as vision, laser, and inertial sensors through SLAM technology.

[0167] In the specific implementation, after obtaining the preprocessed data, the map building module first uses SLAM technology to extract features at the front end, and then processes the data based on probabilistic algorithms at the back end to match and fuse the data, determine the location of its own device, and build a map, thereby obtaining an initial three-dimensional environment map.

[0168] In one embodiment, the map building module 20 is further configured to perform pose calculation based on the contrast enhancement map and the target point cloud to obtain front-end pose data; the map building module 20 is further configured to fuse the target point cloud and the denoised depth map using SLAM technology based on the front-end pose data to generate a locally dense 3D mesh; the map building module 20 is further configured to detect blank areas corresponding to view occlusion or feature loss in the current environment data; the map building module 20 is further configured to generate an initial 3D environment map based on the locally dense 3D mesh and the blank areas.

[0169] It should be noted that the front-end pose data is the result of visual inertial tracking solution performed on the front end in SLAM technology, and includes the robot's position and orientation information in three-dimensional space.

[0170] Understandably, a locally dense 3D mesh is a 3D structure generated by fusing the target point cloud and the denoised depth map using SLAM technology. It is constructed within a local area and contains relatively dense 3D information, accurately representing the geometry of the local environment, the position of objects, and other information. Locally dense 3D meshes are an important component in constructing a complete 3D environment map.

[0171] It should be understood that blank areas are regions in the current environmental data that are not covered or accurately described by normal sensor data due to viewpoint occlusion or feature loss. Viewpoint occlusion may be caused by objects in the environment obscuring each other, such as a large box blocking a wall behind it, making it difficult for sensors (such as cameras or LiDAR) to acquire information about the obscured portion. Feature loss may be due to certain areas in the environment lacking sufficient recognizable features such as texture and shape, making it difficult for sensors to accurately describe these areas, thus creating information gaps when building maps or modeling the environment.

[0172] In this embodiment, by solving the front-end pose data, the robot's initial position and attitude in the environment can be accurately determined. Based on the front-end pose data, SLAM technology is used to fuse the target point cloud and the denoised depth map to generate a locally dense 3D mesh, which helps to more accurately construct a local environment model and obtain richer environmental information. At the same time, detecting blank areas and generating an initial 3D environment map based on the locally dense 3D mesh and blank areas enables a more complete map construction.

[0173] The semantic reasoning module 30 is used to reason about the environmental semantic information in the current environmental data through a visual language model when there are blank areas in the initial three-dimensional environment map, so as to obtain a semantically complete map.

[0174] It should be noted that visual language models are artificial intelligence models that combine computer vision and natural language processing to understand and reason about the semantic information of visual information (such as images, videos, etc.).

[0175] Understandably, environmental semantic information refers to the semantically meaningful information contained in the current environmental data. Environmental semantic information can describe high-level information such as what objects are in the environment, the relationships between objects (such as relative positions, interactions, etc.), and the overall scene type of the environment (such as the scene in which the robot is located).

[0176] Understandably, a semantic completion map is a map obtained by reasoning about the semantic information of the environment using a visual language model, given that there are blank areas in the initial 3D environment map. This map fills in the blank areas in the initial 3D environment map, using semantic information to fill in the missing parts, making the map more complete. For example, in a blank area in the initial 3D environment map, the semantic completion map might infer from the surrounding semantic information that this area should be part of a bookshelf, thus improving the overall structure and information completeness of the map.

[0177] In the specific implementation, when there are blank areas in the initial 3D environment map, the semantic reasoning module can use VLM to mine the environmental semantic information in the current environment data through its powerful semantic understanding and reasoning capabilities, and obtain the semantic completion map corresponding to the missing voxels.

[0178] The map fusion module 40 is used to fuse the semantically completed map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

[0179] It should be noted that the dense map is generated by fusing the aforementioned semantically completed map with the initial 3D environment map, resulting in a map that can accurately describe the environment. Compared to the initial 3D environment map, the dense map has a higher information density and can provide a more accurate environmental description, including small obstacles, subtle terrain changes, and the completion of blank areas due to viewpoint occlusion or feature loss.

[0180] In its implementation, the map fusion module merges the semantically completed map with the initial 3D environment map. It can fill the blank areas of the initial 3D environment map with information from the semantically completed map according to certain weights, making the map information more complete and ultimately obtaining a dense map corresponding to the current environment data.

[0181] Other embodiments or specific implementations of the map building system based on VLM and SLAM positioning in this application can be found in the above-described method embodiments, and will not be repeated here.

[0182] The map building system based on VLM and SLAM positioning provided in this application, employing the map building method based on VLM and SLAM positioning in the above embodiments, can solve the technical problem that traditional SLAM technology relies on sensor data to build 3D maps, resulting in data loss or positioning errors in complex or occluded environments, leading to incomplete 3D maps or reduced accuracy. Compared with the prior art, the beneficial effects of the map building system based on VLM and SLAM positioning provided in this application are the same as those of the map building method based on VLM and SLAM positioning provided in the above embodiments, and other technical features of the map building system based on VLM and SLAM positioning are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0183] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the map construction method and system based on VLM and SLAM positioning in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0184] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A map construction method based on VLM and SLAM positioning, characterized by, The method comprises: preprocessing the current environment data to obtain corresponding multi-modal target data, the current environment data being collected based on a preset multi-modal sensor; performing environment positioning and map construction on the multi-modal target data through SLAM technology to obtain an initial three-dimensional environment map; when there is a blank area in the initial three-dimensional environment map, inferring environmental semantic information in the current environment data through a visual language model to obtain a semantic completed map; fusing the semantic completed map and the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data; wherein the multi-modal target data comprises a denoised depth map and a contrast-enhanced map, the current environment data comprises a color frame and a depth map, the denoised depth map is obtained by denoising the depth map, and the contrast-enhanced map is obtained by equalizing the depth map; the step of, when there is a blank area in the initial three-dimensional environment map, inferring environmental semantic information in the current environment data through a visual language model to obtain a semantic completed map, comprises: when there is a blank area in the initial three-dimensional environment map, performing image recognition on the contrast-enhanced map through a CLIP model to generate an instance mask, the instance mask comprising spatial position information of a point cloud; identifying the instance mask through a text encoder in a CLIP pre-training model and an industrial equipment dictionary of an industry database to obtain an identification result, the identification result comprising a class label of an identified device and a confidence q; projecting the instance mask to the denoised depth map and constructing a semantic anchor node in combination with the industry database, the semantic anchor node comprising a size confidence flag β; Determine a hole set Ω by scanning the initial three-dimensional environment map through a signed distance field (SDF) statistics; generate a completed voxel block using a three-dimensional diffusion model, taking the class label and the size information as conditions, and given the coordinate system of the hole voxel in the hole set Ω ; Determine the change of the SDF value around the completed voxel block through a consistency loss to obtain a surrounding SDF difference field; when a diffusion reconstruction residual exists between the completed voxel block and the surrounding SDF difference field , determine that the completed voxel block is consistent with the surrounding environment; calculate a completion confidence γ according to the confidence q, the size confidence flag β, and the diffusion reconstruction residual , and the formula is as follows: ； when γ>0.6, writing the completed voxel block, the class label of the identified device, and the spatial position information into the hole set to generate a semantic completed map.

2. The method of claim 1, wherein, The multi-modal sensor comprises a laser radar and a depth camera, and the step of preprocessing the current environment data to obtain corresponding multi-modal target data comprises: extracting sparse point cloud data, a color frame, and a depth map in the current environment data, the sparse point cloud data being obtained by scanning the laser radar, and the color frame and the depth map being obtained by photographing the depth camera; performing ground plane removal and down-sampling processing on the sparse point cloud data to obtain target point cloud; performing bilateral filter denoising on the depth map to obtain a denoised depth map; performing image processing on the color frame through a histogram equalization technique to obtain a contrast-enhanced map; taking the target point cloud, the denoised depth map, and the contrast-enhanced map as multi-modal target data.

3. The method of claim 2, wherein, The step of performing environment positioning and map construction on the multi-modal target data through SLAM technology to obtain an initial three-dimensional environment map comprises: solving a pose according to the contrast-enhanced map and the target point cloud to obtain front-end pose data; fusing the target point cloud and the denoised depth map through SLAM technology based on the front-end pose data to generate a local dense three-dimensional grid; detecting a blank region corresponding to the visual angle occlusion or feature missing in the current environment data; generating an initial three-dimensional environment map according to the local dense three-dimensional grid and the blank region.

4. The method of claim 3, wherein, The step of detecting the blank region corresponding to the visual angle occlusion or feature missing in the current environment data comprises: detecting a missing voxel in the current environment data, the missing voxel being caused by visual angle occlusion or feature missing; determining a hole rate according to the missing voxel and a total expected voxel of the current environment data; when the hole rate reaches a preset threshold, regarding the missing voxel as a blank region.

5. The method of claim 1, wherein, The step of projecting the instance mask to the denoised depth map and constructing a semantic anchor node in combination with an industry database comprises: extracting depth information of the instance mask according to a mapping relationship between an image pixel and the denoised depth map to obtain an instance point cloud; traversing the instance point cloud to determine an axis-aligned bounding box and a corresponding center point; comparing a size of the axis-aligned bounding box with a standard size of an industry database according to a class label of the instance mask to obtain a size credible flag, the industry database storing standard size information of various devices; integrating the class label, the axis-aligned bounding box, the center point and the size credible flag to construct a semantic anchor node.

6. The method of claim 5, wherein, The step of fusing the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data comprises: determining a geometric TSDF weight according to the current environment data and the multi-modal sensor; determining a semantic completion TSDF weight of the semantic completion map according to the size credible flag; fusing the semantic completion map with the initial three-dimensional environment map based on the geometric TSDF weight and the semantic completion TSDF weight to obtain a dense map.

7. A map building system based on VLM and SLAM localization, characterized in that, The system performs the method of claim 1, and the system comprises: a data acquisition module configured to pre-process current environment data to obtain corresponding multi-modal target data, the current environment data being acquired based on a preset multi-modal sensor; a map construction module configured to perform environment positioning and map construction on the multi-modal target data by using a SLAM technology to obtain an initial three-dimensional environment map; a semantic reasoning module configured to, when there is a blank region in the initial three-dimensional environment map, perform reasoning on environmental semantic information in the current environment data by using a visual language model to obtain a semantic completion map; a map fusion module configured to fuse the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data. The multi-modal target data comprises a denoised depth map and a contrast-enhanced map, the current environment data comprises a color frame and a depth map, the denoised depth map being obtained by denoising the depth map, and the contrast-enhanced map being obtained by equalizing the depth map. The semantic reasoning module is further configured to, when there is a blank area in the initial three-dimensional environment map, perform image recognition on the contrast-enhanced image through a CLIP model to generate an instance mask, the instance mask including spatial position information of a point cloud; identify the instance mask through a text encoder in the CLIP pre-training model and an industrial equipment dictionary of an industry database to obtain an identification result, the identification result including a class label of an identified device and a confidence q; project the instance mask to the denoised depth map and construct a semantic anchor node in combination with the industry database, the semantic anchor node including a size confidence flag β; Determine a hole set Ω by scanning the initial three-dimensional environment map with a signed distance field (SDF) to perform SDF statistics; use a three-dimensional diffusion model to generate a completed voxel block by taking the class label and the size information as conditions and giving a coordinate system of a hole voxel in the hole set Ω ; determine the change of the SDF value around the completed voxel block by a consistency loss to obtain a surrounding SDF difference field; determine that the completed voxel block is consistent with the surrounding environment when a diffusion reconstruction residual between the completed voxel block and the surrounding SDF difference field is small ; and calculate a completion confidence γ according to the confidence q, a size confidence flag β, and the diffusion reconstruction residual, as follows: γ = q * β * e -β·e ； When γ>0.6, the completed voxel block, the class label of the identified device, and the spatial position information are written into the hole set to generate a semantic completed map.

8. The system of claim 7, wherein, The multi-modal sensor includes a laser radar and a depth camera; The data acquisition module is further configured to extract sparse point cloud data, a color frame, and a depth map from the current environment data, the sparse point cloud data being scanned by the laser radar, and the color frame and the depth map being captured by the depth camera; The data acquisition module is further configured to perform ground plane removal and down-sampling processing on the sparse point cloud data to obtain a target point cloud. The data acquisition module is further configured to perform bilateral filter denoising on the depth map to obtain a denoised depth map. The data acquisition module is further configured to perform image processing on the color frame through a histogram equalization technique to obtain a contrast-enhanced image. The data acquisition module is further configured to use the target point cloud, the denoised depth map, and the contrast-enhanced image as multi-modal target data.

9. The system of claim 8, wherein, The map construction module is further configured to perform pose solving based on the contrast-enhanced image and the target point cloud to obtain front-end pose data. The map construction module is further configured to fuse the target point cloud and the denoised depth map based on the front-end pose data through SLAM technology to generate a local dense three-dimensional grid. The map construction module is further configured to detect a blank area corresponding to a view angle occlusion or feature absence in the current environment data. The map construction module is further configured to generate an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank area.

Images