Map construction method and system based on VLM and SLAM positioning

Abstract

The invention discloses a map construction method and system based on VLM and SLAM positioning, and relates to the technical field of map construction, and the method comprises the steps: carrying out the preprocessing of current environment data collected by a multi-modal sensor, and obtaining multi-modal target data; performing environment positioning and map construction on the multi-modal target data through an SLAM technology to obtain an initial three-dimensional environment map; when a blank area exists in the initial three-dimensional environment map, reasoning environment semantic information in the current environment data through a visual language model to obtain a semantic completion map; and fusing the semantic complemented map with the initial three-dimensional environment map to obtain a dense map. Due to the fact that the visual language model is combined with the SLAM technology, the blank area in the initial three-dimensional environment map generated by the SLAM technology can be automatically complemented through understanding and reasoning of the visual language model on the environment semantic information, and therefore the precision and robustness of map building are improved.

Description

Technical Field

[0001] The present application relates to the field of map construction technology, and in particular to a map construction method and system based on VLM and SLAM positioning. Background Art

[0002] Existing SLAM (Simultaneous Localization and Mapping) technology uses sensors to collect environmental data and combines it with feature extraction and matching algorithms to achieve real-time localization and mapping of the environment. Common SLAM systems include graph-based SLAM (using graph optimization) and the extended Kalman filter. Under ideal conditions, these systems can effectively construct a three-dimensional map of the environment and achieve precise positioning.

[0003] However, in complex environments (such as dynamic environments or environments with occlusion), existing SLAM technology has certain limitations. It relies on sensor data to build three-dimensional maps. In the case of missing data or large positioning errors, it is easy to cause the three-dimensional map to be incomplete or the accuracy to decrease. Summary of the Invention

[0004] The main purpose of this application is to provide a map construction method and system based on VLM and SLAM positioning, aiming to solve the technical problem that traditional SLAM technology relies on sensor data to construct three-dimensional maps, and there are data missing or positioning errors in complex or occluded environments, resulting in incomplete three-dimensional maps or reduced accuracy.

[0005] To achieve the above objectives, the present application proposes a map construction method based on VLM and SLAM positioning, the method comprising: Preprocessing current environmental data to obtain corresponding multimodal target data, wherein the current environmental data is collected based on a preset multimodal sensor; Performing environmental positioning and mapping on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map; When there are blank areas in the initial three-dimensional environment map, reasoning about the environment semantic information in the current environment data using a visual language model to obtain a semantically completed map; The semantic completion map is fused with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

[0006] In one embodiment, the multimodal sensor includes a laser radar and a depth camera, and the step of preprocessing the current environment data to obtain corresponding multimodal target data includes: Extracting sparse point cloud data, color frames, and depth maps from the current environment data, wherein the sparse point cloud data is obtained by scanning the laser radar, and the color frames and depth maps are obtained by photographing the depth camera; Performing ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; Performing bilateral filtering to denoise the depth map to obtain a denoised depth map; performing image processing on the color frame by using a histogram equalization technique to obtain a contrast enhanced image; The target point cloud, the denoised depth map and the contrast-enhanced map are used as multimodal target data.

[0007] In one embodiment, the step of performing environmental positioning and mapping on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map includes: Performing pose solving according to the contrast enhanced image and the target point cloud to obtain front-end pose data; Based on the front-end pose data, the target point cloud and the denoised depth map are fused using SLAM technology to generate a local dense three-dimensional grid; Detecting blank areas corresponding to view angle occlusion or feature missing in the current environment data; An initial three-dimensional environment map is generated according to the local dense three-dimensional grid and the blank area.

[0008] In one embodiment, the step of detecting a blank area corresponding to a view angle occlusion or feature missing in the current environment data includes: Detecting missing voxels in the current environment data, where the missing voxels are caused by view occlusion or feature loss; determining a void rate based on the missing voxels and the total expected voxels of the current environment data; When the void ratio reaches a preset threshold, the missing voxels are regarded as blank areas.

[0009] In one embodiment, when there are blank areas in the initial three-dimensional environment map, the step of inferring the environment semantic information in the current environment data using a visual language model to obtain a semantically completed map includes: When there is a blank area in the initial three-dimensional environment map, performing image recognition on the contrast-enhanced image through a CLIP model to generate an instance mask; Projecting the instance mask onto the denoised depth map and constructing semantic anchor nodes in combination with an industry database; Based on the semantic anchor nodes, the environmental semantic information in the current environmental data is inferred through a visual language model to obtain a semantic completion map.

[0010] In one embodiment, the step of projecting the instance mask onto the denoised depth map and constructing a semantic anchor node in combination with an industry database includes: Extracting depth information of the instance mask according to a mapping relationship between image pixels and the denoised depth map to obtain an instance point cloud; Traversing the instance point cloud to determine the axis-aligned bounding box and the corresponding center point; According to the class label of the instance mask, the size of the axis-aligned bounding box is compared with the standard size of an industry database to obtain a size trustworthy flag, wherein the industry database stores standard size information of various devices; The class label, the axis-aligned bounding box, the center point, and the size-trusted marker are integrated to construct a semantic anchor node.

[0011] In one embodiment, the step of fusing the semantically completed map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data includes: Determining a geometric TSDF weight based on the current environment data and the multimodal sensor; Determining a semantic completion TSDF weight of the semantic completion map according to the size trustworthy flag; Based on the geometric TSDF weights and the semantic completion TSDF weights, the semantic completion map is fused with the initial three-dimensional environment map to obtain a dense map.

[0012] In addition, to achieve the above objectives, the present application also proposes a map construction system based on VLM and SLAM positioning, which executes the method described above, and includes: A data acquisition module is used to pre-process current environmental data to obtain corresponding multimodal target data, wherein the current environmental data is collected based on a preset multimodal sensor; A map construction module is used to perform environmental positioning and map construction on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map; A semantic reasoning module, configured to, when there are blank areas in the initial three-dimensional environment map, infer the environment semantic information in the current environment data using a visual language model to obtain a semantically completed map; A map fusion module is used to fuse the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

[0013] In one embodiment, the multimodal sensor includes a lidar and a depth camera; The data acquisition module is further configured to extract sparse point cloud data, color frames, and depth maps from the current environment data, wherein the sparse point cloud data is obtained by scanning the laser radar, and the color frames and depth maps are obtained by capturing the depth camera; The data acquisition module is further configured to perform ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; The data acquisition module is further configured to perform bilateral filtering and denoising on the depth map to obtain a denoised depth map; The data acquisition module is further configured to perform image processing on the color frame using a histogram equalization technique to obtain a contrast-enhanced image; The data acquisition module is further configured to use the target point cloud, the denoised depth map, and the contrast-enhanced map as multimodal target data.

[0014] In one embodiment, the map construction module is further configured to perform pose solving based on the contrast enhancement map and the target point cloud to obtain front-end pose data; The map construction module is further configured to fuse the target point cloud and the denoised depth map using SLAM technology based on the front-end pose data to generate a local dense three-dimensional grid; The map construction module is further configured to detect blank areas corresponding to view angle occlusion or feature missing in the current environment data; The map construction module is further configured to generate an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank area.

[0015] One or more technical solutions proposed in the present application have at least the following technical effects: The map construction method based on VLM and SLAM positioning of the present application includes: preprocessing the current environmental data to obtain corresponding multimodal target data, and the current environmental data is obtained based on the preset multimodal sensor collection; performing environmental positioning and map construction on the multimodal target data through SLAM technology to obtain an initial three-dimensional environmental map; when there are blank areas in the initial three-dimensional environmental map, inferring the environmental semantic information in the current environmental data through a visual language model to obtain a semantic completion map; fusing the semantic completion map with the initial three-dimensional environmental map to obtain a dense map corresponding to the current environmental data.

[0016] Since this application combines the visual language model with SLAM technology, the visual language model can automatically fill in the blank areas in the initial three-dimensional environment map generated by SLAM technology through understanding and reasoning of environmental semantic information, thereby improving the accuracy and robustness of map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of the first embodiment of the map construction method based on VLM and SLAM positioning of this application is provided; Figure 2 A flowchart of the second embodiment of the map construction method based on VLM and SLAM positioning of this application is provided; Figure 3 Schematic diagram of the map construction scenario based on VLM and SLAM positioning proposed in this application; Figure 4 This is a module diagram of the map construction system based on VLM and SLAM positioning in this application.

[0020] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the execution entity of this embodiment can be a computing service device with multimodal data acquisition, map construction, semantic reasoning, and map fusion capabilities, such as an industrial robot, self-driving car, or surveying drone, or an electronic device capable of performing the aforementioned functions, or a VLM and SLAM positioning-based map construction device (referred to as a map construction device, such as one that can be mounted on a robot) that executes the VLM and SLAM positioning-based map construction method of this application, and this embodiment is not limited thereto. The following uses a map construction device as an example to illustrate this embodiment and the following embodiments.

[0024] Based on this, the first embodiment of the present application is proposed. The embodiment of the present application provides a map construction method based on VLM and SLAM positioning, referring to Figure 1 , Figure 1This is a flowchart of the first embodiment of the map construction method based on VLM and SLAM positioning in this application.

[0025] In this embodiment, the map construction method based on VLM and SLAM positioning includes steps S10 to S40: Step S10: pre-processing the current environment data to obtain corresponding multimodal target data, wherein the current environment data is collected based on a preset multimodal sensor.

[0026] It should be noted that a multimodal sensor is a combination of sensors that can collect multiple types of data simultaneously, and may include cameras, infrared sensors, radars, inertial measurement devices, etc.

[0027] It is understood that current environmental data is a collection of raw information from the surrounding environment acquired through pre-set multimodal sensors. For example, in an autonomous driving scenario, multimodal sensors may include cameras, radars, and inertial measurement devices, with cameras capturing video images, radars collecting distance data, and inertial measurement devices collecting vehicle linear velocity.

[0028] It should be understood that multimodal target data is derived from raw current environment data after preprocessing (e.g., denoising, filtering, etc.). Because raw current environment data may contain noise, inconsistent data formats, or excessive data volume, these issues can be addressed through preprocessing. For example, image data captured by a camera can be denoised to reduce random noise points and improve image quality. Alternatively, standardization can be performed to convert data of different formats and ranges into a form suitable for subsequent processing.

[0029] In a specific implementation, after the multimodal sensor transmits the collected current environment data to the map building device, the map building device first performs operations such as denoising and filtering on the current environment data to obtain the multimodal target data to be processed.

[0030] In a feasible embodiment, the multimodal sensor includes a lidar and a depth camera. Step S10 of this embodiment may include the steps of: extracting sparse point cloud data, color frames and depth maps from the current environment data, wherein the sparse point cloud data is obtained by scanning the lidar, and the color frames and the depth maps are obtained by photographing the depth camera; removing the ground plane and downsampling the sparse point cloud data to obtain a target point cloud; performing bilateral filtering and denoising on the depth map to obtain a denoised depth map; performing image processing on the color frame through histogram equalization technology to obtain a contrast enhanced map; and using the target point cloud, the denoised depth map and the contrast enhanced map as multimodal target data.

[0031] It should be noted that LiDAR (Light Detection and Ranging) is a sensor that acquires three-dimensional information about its surroundings by emitting laser beams and measuring the laser reflection time. LiDAR can rapidly scan its surroundings with laser pulses, generating large amounts of point cloud data.

[0032] Among them, sparse point cloud data is a collection of a large number of discrete points obtained by scanning the surrounding environment through lidar. These points are relatively sparsely distributed in space and can depict three-dimensional information such as the shape, position and distance of surrounding objects.

[0033] Ground-Plane Removal (GPR) is an operation that removes ground-plane point cloud data from sparse point cloud data. This is because ground-plane point cloud data often interferes with the identification and analysis of target objects. Using specific algorithms (such as those based on plane model fitting) to identify and remove ground-plane point cloud data can highlight other objects in the environment, reduce unnecessary data volume, and improve the efficiency and accuracy of subsequent processing.

[0034] Downsampling (VoxelGrid) is a method of compressing sparse point cloud data without compromising key information. Since the original sparse point cloud data may contain too many points, downsampling is used to reduce the number of points and thus reduce the data volume without compromising key information.

[0035] For example, a 3D space can be divided into small cubic units (called voxels), assuming a voxel side is 0.05 meters long. Within each voxel, a point is selected based on a specific rule (such as the center of gravity of the points within the voxel) to represent all points within that voxel, thereby reducing the number of points in the point cloud and achieving downsampling. This method preserves the overall shape and characteristics of the sparse point cloud data while reducing the data volume and improving the efficiency of subsequent processing.

[0036] Therefore, by removing the ground plane and downsampling the sparse point cloud data, the target point cloud can be obtained by removing the ground plane and reducing the data volume.

[0037] It should be noted that a depth camera (RGB-D camera for short) is a camera that can simultaneously capture the color image of the scene and the depth information corresponding to each pixel. In addition to providing a visual image (color frame) of the scene like an ordinary camera, it can also sense the distance between the object and the camera, output a depth map, and obtain two-dimensional and three-dimensional data of the environment.

[0038] Color frames are image data captured by a depth camera, measured in pixels. Each pixel contains both brightness and color information. Color frames are typically represented in RGB (red, green, and blue) color modes, reflecting the color distribution in the environment.

[0039] A depth map is an image captured by a depth camera that shows the distance from each point in the scene to the camera. The value of each pixel represents the distance from the corresponding scene point to the camera. Combining the depth map with the color frame provides a more comprehensive description of the three-dimensional structure of the environment.

[0040] It's understandable that bilateral filtering (BF) denoising is a nonlinear filtering method. Depth maps may be subject to noise during the acquisition process. Bilateral filtering not only considers the spatial distance between pixels but also the differences in their grayscale values ​​(or, in the depth map, their depth values) during filtering. For each pixel in the depth map, bilateral filtering weights the pixels based on the spatial position and similarity of the depth values ​​of its surrounding pixels, then takes a weighted average to obtain the filtered depth value. This method removes noise while effectively preserving important features in the depth map, such as edges, and can identify these differences without over-smoothing them.

[0041] It should be understood that the Contrast Limited Adaptive Histogram Equalization (CLAHE) technology is an image processing technology that adjusts the histogram distribution of a color frame to make the pixel value distribution of the color frame more uniform, thereby enhancing the contrast of the image and obtaining a contrast-enhanced image.

[0042] For the sake of illustration, in order to facilitate the understanding of the above data collection and data processing process, this embodiment and subsequent embodiments are described using a robot as an example, but this does not limit the present solution. First, the multimodal sensor collects environmental information, and the laser radar collects and outputs sparse point cloud data. ; Collect and output color frames through RGB-D camera With Depth Map ; Collect and output angular velocity through inertial measurement unit (IMU) With linear acceleration Then pre-process the above data and perform Bilateral Filter denoising on the depth map to obtain the denoised depth map. ; Remove the ground plane and downsample the sparse point cloud data to obtain the target point cloud ; Perform CLAHE on the color frame to improve low-light details and obtain a contrast-enhanced image The data obtained after the above acquisition and processing is aligned by hardware timestamp + software interpolation to form a synchronous data packet .

[0043] In this embodiment, for data collected by multimodal sensors, point cloud data is optimized by removing ground planes and downsampling to reduce interference and data volume; depth map denoising and color frame contrast enhancement can improve data quality.

[0044] Step S20: Performing environmental positioning and mapping on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map.

[0045] It should be noted that SLAM technology allows robots and other devices to simultaneously build a map of the environment and determine their own position in an unknown environment. It integrates information from multiple sensors, such as lidar, cameras, and IMUs. SLAM technology can accurately estimate the robot's position and build a consistent map.

[0046] It is understood that the initial 3D environment map is a preliminary map obtained by using SLAM technology to locate and map multimodal target data. It is obtained through SLAM technology using multimodal sensor data such as vision, laser, and inertial.

[0047] In the specific implementation, after obtaining the above-mentioned preprocessed data, the SLAM technology front-end is first used to extract features, and the back-end is processed based on a probabilistic algorithm to match and fuse the data, determine the position of the map-building device and build a map, thereby obtaining an initial three-dimensional environment map.

[0048] In a feasible implementation, step S20 of this embodiment may include the steps of: performing pose solving based on the contrast-enhanced image and the target point cloud to obtain front-end pose data; based on the front-end pose data, fusing the target point cloud and the denoised depth map through SLAM technology to generate a local dense three-dimensional grid; detecting blank areas corresponding to perspective occlusion or feature missing in the current environment data; and generating an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank areas.

[0049] It's important to note that front-end pose data is the result of visual-inertial tracking in SLAM technology, and contains the robot's position and attitude in three-dimensional space. In SLAM-based map construction, front-end pose data guides the fusion process of the target point cloud and the denoised depth map, ensuring fusion accuracy.

[0050] For example, ORB (Oriented FAST and Rotated BRIEF) feature points can be extracted from the contrast-enhanced image while simultaneously acquiring the aforementioned IMU measurement data. The IMU data is then pre-integrated to obtain relative motion constraints. The ORB feature points and relative motion constraints are formulated as an optimization problem. Within a sliding window, an optimization algorithm (such as the Levenberg-Marquardt algorithm) is used to solve the camera pose, minimizing the feature point reprojection error.

[0051] For the target point cloud, a local subgraph is first determined. For each point in the target point cloud, the nearest point (or set of points) is found within the local subgraph. Based on these corresponding point pairs, an initial transformation estimate is determined. This transformation estimate is then iteratively refined using ICP (Iterative Closest Point) to minimize the distance between the two point clouds, ultimately yielding the initial LiDAR pose.

[0052] Based on the above camera pose and the initial value of the lidar pose, the current front-end pose data of the robot can be obtained by solving the front-end of the SLAM technology.

[0053] Understandably, a local dense 3D mesh is a 3D structure generated by fusing a target point cloud with a denoised depth map using SLAM technology. Constructed within a local area, it contains relatively dense 3D information and accurately represents the geometry of the local environment, the location of objects, and other information. This local dense 3D mesh is a crucial component in building a complete 3D environmental map.

[0054] For example, in the process of constructing a local dense 3D mesh, firstly, the Surfel (Surface Element, an element representing a 3D surface, which includes the position, normal direction and some other properties of the surface) element can be extracted based on the target point cloud and the denoised depth map, and the properties of each Surfel element can be determined. Then, the target point cloud is combined with the front-end pose data estimated by the front-end. And the denoised depth map The Surfel elements of the two from different perspectives or at different times are fused. For example, if a new target point cloud or denoised depth map comes in, the Surfel elements therein are extracted and matched and fused with the Surfel elements in the existing local dense grid. The Surfel elements can be merged or their attributes can be updated based on factors such as distance and normal direction, and a local dense 3D grid can be gradually constructed. .

[0055] It should be understood that blank areas are areas in the current environmental data that are not covered by normal sensor data or cannot be accurately described due to perspective occlusion or missing features. Perspective occlusion may be caused by objects in the environment blocking each other, such as a large box blocking the wall behind it, making it difficult for sensors (such as cameras or lidar) to obtain information about the obscured part. Missing features may be due to the lack of sufficient recognizable features such as texture and shape in certain areas of the environment, making it difficult for sensors to accurately describe these areas, resulting in areas of information blank when building maps or modeling the environment.

[0056] In a feasible implementation, the step of detecting the blank area corresponding to the perspective occlusion or feature missing in the current environment data in the steps of this embodiment includes: detecting the missing voxels in the current environment data, wherein the missing voxels are caused by perspective occlusion or feature missing; determining the void rate based on the missing voxels and the total expected voxels of the current environment data; when the void rate reaches a preset threshold, treating the missing voxels as a blank area.

[0057] It should be noted that voxels are the smallest units in three-dimensional space. In environmental data, due to view occlusion or missing features, some voxel information cannot be accurately obtained or does not exist. These voxels are called missing voxels. The total expected voxels is a measure of the number of voxels that the current environmental data should ideally contain. It represents the total number of voxels required to fully describe the current environment if there are no occlusions or missing features. For example, if it is known that the scanning range of the sensor is a specific three-dimensional spatial area, the total expected number of voxels that should theoretically exist can be calculated based on the size of this area and the accuracy of the voxel division.

[0058] The hole rate is an indicator used to measure the completeness of the current environment data. It is calculated by the ratio of missing voxels to the total expected voxels (i.e., missing voxels / total expected voxels = hole rate). The hole rate reflects the extent of holes (i.e., missing information) in the environment data caused by missing voxels.

[0059] The preset threshold is a numerical limit for judging whether there are too many missing voxels. When detecting the void rate in the current environmental data, the calculated void rate is compared with this preset threshold. If the void rate reaches or exceeds this preset threshold, it means that the number of missing voxels in the environmental data has reached a level that requires special treatment. At this time, the corresponding missing voxels are treated as blank areas. For example, if the void rate is 0, it means that the environmental data is complete and there are no missing voxels; if the preset threshold is 0.3, when the calculated void rate is 0.35, it means that there is a lot of missing information in the environmental data, which may affect the accuracy of subsequent map construction. The condition for marking the missing voxels as blank areas is met, and the subsequent semantic reasoning is triggered at this time.

[0060] For example, in the process of constructing a local dense 3D grid, the void rate is continuously counted. When the void rate is caused by view occlusion or feature loss, , triggering semantic reasoning.

[0061] In this implementation, by solving the front-end pose data, the robot's initial position and posture in the environment can be accurately determined. Based on this front-end pose data, SLAM technology is used to fuse the target point cloud and the denoised depth map to generate a local dense 3D mesh. This helps to more accurately construct a local environment model and obtain richer environmental information. Furthermore, detecting blank areas and generating an initial 3D environment map based on the local dense 3D mesh and blank areas can make map construction more complete.

[0062] Step S30: When there are blank areas in the initial three-dimensional environment map, the environment semantic information in the current environment data is inferred by a visual language model to obtain a semantically completed map.

[0063] It should be noted that the Visual Language Model (VLM) is an artificial intelligence model that combines computer vision and natural language processing to understand and reason about the semantic information of visual information (such as images, videos, etc.).

[0064] Understandably, environmental semantic information refers to information with semantic meaning contained in the current environmental data. This information describes high-level information such as the nature of objects in the environment, the relationships between objects (such as relative positions and interactions), and the overall scene type (e.g., the scene the robot is in).

[0065] The semantically completed map is a map created by inferring semantic information about the environment using a visual language model, even when there are blank areas in the initial 3D environment map. This map complements the blank areas in the initial 3D environment map, using semantic information to fill in the missing parts and make the map more complete. For example, in a blank area in the initial 3D environment map, the semantically completed map might infer that this area is part of a bookshelf based on the surrounding semantic information, thereby improving the overall structure and information integrity of the map.

[0066] Furthermore, since some errors may exist in the process of generating the above-mentioned front-end pose data, such as visual reprojection error (in the above-mentioned SLAM technology, the feature points in the three-dimensional space are projected onto the image plane according to the estimated camera pose. The error between this projection point and the actual detected feature point is the reprojection error), IMU pre-integration error (in the above-mentioned IMU pre-integration process, the error between the actual movement and the pre-integration estimated movement), ICP point-surface error (the error of the nearest point through ICP iteration when aligning the point cloud), and other error factors, these errors are constructed into a factor graph, in which each factor corresponds to an error term, and the factor graph can also be optimized through semantic completion.

[0067] In a specific implementation, when there are blank areas in the initial three-dimensional environment map, the VLM can utilize powerful semantic understanding and reasoning capabilities to mine the environmental semantic information in the current environment data and obtain a semantic completion map corresponding to the missing voxels.

[0068] In a feasible implementation, step S30 of this embodiment may include the following steps: when there is a blank area in the initial three-dimensional environment map, performing image recognition on the contrast-enhanced image through the CLIP model to generate an instance mask; projecting the instance mask onto the denoised depth map, and constructing a semantic anchor node in combination with the industry database; based on the semantic anchor node, inferring the environmental semantic information in the current environment data through a visual language model to obtain a semantic completion map.

[0069] It's important to note that the CLIP (Contrastive Language-Image Pretraining) model is a model that can identify various visual features in contrast-enhanced images and, based on pretrained knowledge, associate these features with corresponding semantic concepts. For example, it can identify objects in an image as "chair" or "table," and can distinguish between different object categories and attributes, providing foundational image recognition results for subsequent operations.

[0070] An instance mask is an image representation used to mark the area of ​​a specific object in an image. When the CLIP model performs image recognition on the contrast-enhanced image, it generates an instance mask. This mask clearly indicates the location and range of each identified object in the contrast-enhanced image. For example, if there are multiple chairs in an image, the instance mask will mark the area where each chair is located, like drawing a specific "outline" for each chair in the image. This "outline" is a manifestation of the instance mask and is used to distinguish different instance objects.

[0071] It can be understood that the industry database is a collection of data that has been collected and organized in the field of three-dimensional environment map construction and object recognition, including information such as the standard size, shape, material, common use, functional attributes, etc. of objects, and stores standard size information of various devices.

[0072] For example, first, the different objects in the contrast enhancement image can be segmented by the image segmentation model (Segment-Anything, SAM) to generate instance masks Then, the text encoder part of the CLIP pre-trained model is used to encode each device name in the industrial equipment dictionary (such as "valve", "pump", "transformer", etc.) of the industry database to obtain the corresponding semantic vector. For instance masks , which is converted into a semantic vector through the image encoding part of the CLIP model During the recognition process, the semantic vector corresponding to each device name , calculate its semantic vector with the instance mask The cosine similarity of is compared with the threshold to determine the recognition result; for example, if cos( , )>0.3 (set threshold), it is considered that the corresponding device has been successfully identified. After successfully identifying the corresponding device, the class label c can be directly determined based on the name of the successfully identified device; the confidence q of the recognition result is determined based on factors such as the degree of proximity between the above cosine similarity value and the threshold. For example, the cosine similarity value itself can be used as the confidence level; finally, the above image recognition result is output <class label c, confidence q, instance mask 〉.

[0073] It should be understood that a semantic anchor node is a node with semantic information, which is constructed by projecting the instance mask onto the denoised depth map and combining it with the industry database. It integrates the device instance information in the instance mask, the depth information in the depth map, and the relevant semantic knowledge in the industry database.

[0074] In another feasible implementation, the step of projecting the instance mask onto the denoised depth map and constructing a semantic anchor node in combination with the industry database described in this embodiment includes: extracting the depth information of the instance mask based on the mapping relationship between the image pixels and the denoised depth map to obtain an instance point cloud; traversing the instance point cloud to determine the axis-aligned bounding box and the corresponding center point; comparing the size of the axis-aligned bounding box with the standard size of the industry database based on the class label of the instance mask to obtain a size trustworthy mark, wherein the industry database stores standard size information of various devices; integrating the class label, the axis-aligned bounding box, the center point and the size trustworthy mark to construct a semantic anchor node.

[0075] It's important to note that image pixels are the basic unit of an image, containing information such as color and brightness. A depth map reflects the distance from each point in the scene to the camera. The mapping between image pixels and the denoised depth map describes how to find the corresponding depth information in the depth map from the image pixel information. Depth information is the depth value of each point within the area defined by the instance mask, used to determine the spatial position and shape of the instance in the 3D scene.

[0076] An instance point cloud is a collection of points representing the surface of an object or scene in 3D space. It is obtained by extracting depth information from an instance mask and represents the collection of points in 3D space representing the instance. The instance point cloud contains the 3D spatial position information of the instance mask.

[0077] It's understandable that an axis-aligned bounding box (AABB) is a simple geometric shape used to represent the boundaries of a three-dimensional object. For an object represented by an instance point cloud, the smallest axis-aligned cuboid that completely encompasses the object's point cloud is found by traversing the instance point cloud. This cuboid is the axis-aligned bounding box. The center point is the geometric center point within the axis-aligned bounding box, representing the center position of the device. It is calculated by taking the average of the minimum and maximum coordinates of the bounding box along the three coordinate axes.

[0078] For example, the depth information corresponding to the pixels in the instance mask can be extracted based on the mapping relationship between the image pixels and the depth map after noise, and the three-dimensional coordinates can be constructed, thereby converting the two-dimensional mask information into a three-dimensional instance point cloud. , that is, the point cloud representation of the corresponding device in three-dimensional space is obtained. Then, by traversing the points in the point cloud, the minimum and maximum values ​​in each coordinate axis direction are found to determine the range and center point of the axis-aligned bounding box (AABB) .

[0079] It should be understood that the size reliability flag is a marker obtained by comparing the size of the axis-aligned bounding box with the standard size in the industry database. It is used to assess the reliability of the size information obtained from the instance. If the size of the axis-aligned bounding box is close to the standard size of the corresponding category in the industry database, this flag indicates that the size is reliable. If the difference is large, it may indicate a problem with the detection or calculation, or a special case of the object.

[0080] For example, an industry database corresponding to the industrial equipment type can be pre-established to store standard size information of various equipment. After the equipment type is identified as above, the standard size of the equipment type (determined by the class label) can be retrieved from the industry database. . Then you can compare the size of the AABB of the instance point cloud If the comparison result is consistent with the standard size, the size is considered to be reliable and the size reliability flag β is set to 1, otherwise it is set to 0; for example, if , set the size trust flag β = 1, otherwise set it to 0. Finally, the device class label, size trust flag, AABB and center point information are integrated into the semantic anchor node , which is called by the above factor graph for subsequent completion optimization.

[0081] After constructing the semantic anchor nodes, missing nodes are detected first. The signed distance field (SDF) is used to scan the entire initial 3D environment map for SDF statistics. The hole areas in the scene are found by analyzing the distance and the hole set Ω is obtained.

[0082] Then the three-dimensional diffusion model (Latent-Diffusion-3D) is used to The category information (i.e., class label, including the category information of the device) and the size vector (i.e., the size of the axis-aligned bounding box, including the size information of the device) are embedded as conditions, and the coordinate system of the void voxels in the void set is given. The complementary voxel blocks that can fill the void are generated in an iterative manner. At the same time, in the process of generating the complementary voxel block, in order to prevent the generated voxel block from deviating from the surrounding environment (drift), a consistency loss (Consistency Loss) can be introduced to calculate the change of the SDF value around the complementary voxel block (for example, calculating the SDF difference between adjacent voxels) to obtain the surrounding SDF differential field. By comparing the complementary voxel block with the SDF differential field, it is measured whether the generated voxels are consistent with the surrounding environment; for example, if the diffusion reconstruction residual between the complementary voxel block and the surrounding SDF differential field (a measure of the difference between the actual generated result and the expected result in the process of voxel completion using a three-dimensional diffusion model) , it is consistent with the surrounding environment.

[0083] After successfully generating the above-mentioned complementary pixel blocks, the confidence q, size trust mark β, and diffusion reconstruction residual generated in the above VLM can be synthesized. Calculate the completion confidence γ: ; Only when γ>0.6 (the threshold can be set), the complementary pixel blocks, the semantic information of the device (class label, etc.) and the related spatial location information can be written into the empty set. , generate semantic completion map .

[0084] In this implementation, the depth map information combined with the instance mask constructs an instance point cloud, enabling more precise determination of the object's position and shape in 3D space. The axis-aligned bounding box and center point further clarify the object's spatial extent and center position, helping to improve object localization accuracy. Furthermore, the constructed semantic anchor nodes integrate multiple important pieces of information, facilitating higher-level semantic understanding.

[0085] Step S40: Fusing the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

[0086] It should be noted that the dense map is generated by fusing the semantically completed map with the initial 3D environment map, resulting in a precise description of the environment. Compared to the initial 3D environment map, the dense map has a higher information density and can provide a more accurate description of the environment, including small obstacles, subtle terrain changes, and the completion of blank areas due to view occlusion or missing features.

[0087] In the technical solution provided by this embodiment, after a multimodal sensor transmits the collected current environmental data to a map-building device, the map-building device first performs operations such as denoising and filtering on the current environmental data to obtain multimodal target data to be processed. The SLAM (Simultaneous Localization and Mapping) technology front-end then extracts features from the multimodal target data. The back-end, based on a probabilistic algorithm, matches and fuses the data, determines the location of the map-building device, and constructs a map, thereby obtaining an initial three-dimensional environmental map. When blank areas are detected in the initial three-dimensional environmental map, the current environmental data is input into a visual language model, which analyzes and infers the environmental semantic information contained therein, such as identifying object types and positional relationships, to construct a semantically completed map. The semantically completed map is then fused with the initial three-dimensional environmental map. Information from the semantically completed map can be weighted to fill in the blank areas of the initial three-dimensional environmental map, making the map information more complete and ultimately obtaining a dense map corresponding to the current environmental data. Because this embodiment combines a visual language model with SLAM technology, the visual language model's understanding and inference of environmental semantic information can automatically complete blank areas in the initial three-dimensional environmental map generated by the SLAM technology, thereby improving the accuracy and robustness of map construction.

[0088] Based on the above embodiment 1 of this application, the second embodiment of this application is proposed. In the second embodiment of this application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be repeated hereafter. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the map construction method based on VLM and SLAM positioning of this application.

[0089] In this example, step S40 further includes steps S41 to S43: Step S41: Determine the geometric TSDF weight according to the current environment data and the multimodal sensor.

[0090] It's important to note that geometric TSDF (Truncated Signed Distance Function) weights are parameters determined based on the current environmental data and multimodal sensors. Multimodal sensors can collect a variety of environmental information, such as depth and color. Current environmental data includes a variety of raw data, such as point coordinates at different locations. When constructing a 3D environmental map, the importance of each piece of data can be considered from a geometric perspective. Appropriately setting geometric TSDF weights allows for a more accurate representation of the geometric structure of the environment.

[0091] Step S42: Determine the semantic completion TSDF weight of the semantic completion map according to the size trustworthy flag.

[0092] Step S43: Based on the geometric TSDF weight and the semantic completion TSDF weight, the semantic completion map is fused with the initial three-dimensional environment map to obtain a dense map.

[0093] It should be noted that the semantic completion map is obtained by inferring the semantic information of the environment using a visual language model based on the initial 3D environment map with blank areas. To reflect the relative importance of the information in the semantic completion map during the fusion process, the semantic completion TSDF weight can be set based on factors such as the quality and credibility of the semantic completion map.

[0094] For example, for the initial 3D environment map, the geometric TSDF weights can be set , set the semantic completion TSDF weight according to the above completion credibility , and finally output a unified dense map .

[0095] Furthermore, based on the semantic anchor nodes, we can define connections with other factors in the factor graph, constructing new edges (factors) to connect the semantic anchor nodes with other related nodes (such as pose nodes and feature point nodes). Then, when optimizing the factor graph, we consider both geometric and semantic factors. By adjusting the variables in the graph (such as pose and semantic information parameters), we minimize the objective function that includes the semantic factors, achieving joint geometric and semantic optimization and suppressing drift in the resulting dense map.

[0096] The technical solution provided in this embodiment combines VLM semantic understanding with SLAM positioning. When positioning accuracy deviates, environmental semantic information is used to correct errors and optimize accuracy, thereby improving the accuracy and robustness of 3D maps. VLM identifies information such as the location and function of devices in the environment and makes real-time corrections to SLAM positioning results, improving positioning accuracy, especially in environments with occlusion, changing lighting, or interference from dynamic objects.

[0097] For example, in order to help understand the implementation process of the map construction method based on VLM and SLAM positioning obtained after the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 3 , Figure 3 Schematic diagram of the scenario of map construction based on VLM and SLAM positioning proposed in this application; specifically: The map-building equipment is located on the ground, and the entire process is completed in parallel using two main approaches: SLAM and VLM. SLAM uses the laser point cloud to first create a "skeleton" sparse point cloud map. The RGB image is then used to obtain a color frame and depth map for texture and semantic reference. Due to visual occlusion, sensor blind spots, or reflections in the scene, the sparse point cloud map may contain holes (missing areas). The raw depth information provided in the depth map is also sparse or has gaps. The map-building equipment then performs the semantic completion phase of VLM reasoning. The visual language model (VLM) infers the color frame and finds that "there should be a box here," thereby inferring the semantic category corresponding to the missing geometric area. The VLM not only indicates "there is a box here," but also provides the approximate size and shape, thereby generating the completed depth value and obtaining a dense depth map.

[0098] The process is as follows: SLAM first builds a geometric skeleton, then detects blank areas. VLM uses semantic reasoning about the environment to fill in the gaps, integrating the semantically completed geometric and semantic information back into the system. Ultimately, it outputs a dense 3D map with both complete geometry and semantic labels. By combining the visual language model with SLAM technology, it can automatically fill in blank areas in the initial 3D environment map generated by SLAM, thereby improving the accuracy and robustness of map construction.

[0099] refer to Figure 4 The above-mentioned map construction method based on VLM and SLAM positioning is applied to a map construction system based on VLM and SLAM positioning. Figure 4 This is a module diagram of the map construction system based on VLM and SLAM positioning in this application.

[0100] like Figure 4 As shown, the system described in this embodiment includes: a data acquisition module 10, a map construction module 20, a semantic reasoning module 30 and a map fusion module 40.

[0101] The data acquisition module 10 is used to pre-process the current environment data to obtain corresponding multimodal target data, wherein the current environment data is acquired based on a preset multimodal sensor.

[0102] It should be noted that a multimodal sensor is a combination of sensors that can collect multiple types of data simultaneously, and may include cameras, infrared sensors, radars, inertial measurement devices, etc.

[0103] It is understood that current environmental data is a collection of raw information from the surrounding environment acquired through pre-set multimodal sensors. For example, in an autonomous driving scenario, multimodal sensors may include cameras, radars, and inertial measurement devices, with cameras capturing video images, radars collecting distance data, and inertial measurement devices collecting vehicle linear velocity.

[0104] It should be understood that multimodal target data is derived from raw current environment data after preprocessing (e.g., denoising, filtering, etc.). Because raw current environment data may contain noise, inconsistent data formats, or excessive data volume, these issues can be addressed through preprocessing. For example, image data captured by a camera can be denoised to reduce random noise points and improve image quality. Alternatively, standardization can be performed to convert data of different formats and ranges into a form suitable for subsequent processing.

[0105] In a specific implementation, after the multimodal sensor transmits the collected current environmental data to the data acquisition module, the data acquisition module first performs operations such as denoising and filtering on the current environmental data to obtain the multimodal target data to be processed.

[0106] In one embodiment, the multimodal sensor described in this embodiment includes a laser radar and a depth camera; the data acquisition module 10 is also used to extract sparse point cloud data, color frames and depth maps from the current environment data, the sparse point cloud data is obtained by scanning the laser radar, and the color frames and the depth maps are obtained by photographing the depth camera; the data acquisition module 10 is also used to remove the ground plane and downsample the sparse point cloud data to obtain a target point cloud; the data acquisition module 10 is also used to perform bilateral filtering denoising on the depth map to obtain a denoised depth map; the data acquisition module 10 is also used to perform image processing on the color frame through histogram equalization technology to obtain a contrast enhanced map; the data acquisition module 10 is also used to use the target point cloud, the denoised depth map and the contrast enhanced map as multimodal target data.

[0107] It should be noted that LiDAR is a sensor that acquires three-dimensional information about its surroundings by emitting laser beams and measuring the time it takes for them to reflect back. LiDAR can rapidly scan its surroundings with laser pulses, generating a large amount of point cloud data.

[0108] Among them, sparse point cloud data is a collection of a large number of discrete points obtained by scanning the surrounding environment through lidar. These points are relatively sparsely distributed in space and can depict three-dimensional information such as the shape, position and distance of surrounding objects.

[0109] Understandably, ground plane removal is the process of removing ground plane point cloud data from sparse point cloud data. This is because ground plane point clouds often interfere with the identification and analysis of target objects. Using specific algorithms (such as those based on plane model fitting) to identify and remove ground plane point cloud data can highlight other objects in the environment, reduce unnecessary data, and improve the efficiency and accuracy of subsequent processing.

[0110] Downsampling is a method of compressing sparse point cloud data without compromising key information. Since the original sparse point cloud data may contain too many points, downsampling is used to reduce the number of points and thus reduce the data volume without compromising key information.

[0111] Therefore, by removing the ground plane and downsampling the sparse point cloud data, the target point cloud can be obtained by removing the ground plane and reducing the data volume.

[0112] It should be noted that a depth camera is a camera that can simultaneously capture the color image of the scene and the depth information corresponding to each pixel. In addition to providing a visual image (color frame) of the scene like an ordinary camera, it can also sense the distance between the object and the camera, output a depth map, and obtain two-dimensional and three-dimensional data of the environment.

[0113] Color frames are image data captured by a depth camera, measured in pixels. Each pixel contains both brightness and color information. Color frames are typically represented in RGB (red, green, and blue) color modes, reflecting the color distribution in the environment.

[0114] A depth map is an image captured by a depth camera that shows the distance from each point in the scene to the camera. The value of each pixel represents the distance from the corresponding scene point to the camera. Combining the depth map with the color frame provides a more comprehensive description of the three-dimensional structure of the environment.

[0115] It's understandable that bilateral filtering (BF) denoising is a nonlinear filtering method. Depth maps may be subject to noise during the acquisition process. Bilateral filtering not only considers the spatial distance between pixels but also the differences in their grayscale values ​​(or, in the depth map, their depth values) during filtering. For each pixel in the depth map, bilateral filtering weights the pixels based on the spatial position and similarity of the depth values ​​of its surrounding pixels, then takes a weighted average to obtain the filtered depth value. This method removes noise while effectively preserving important features in the depth map, such as edges, and can identify these differences without over-smoothing them.

[0116] It should be understood that the histogram equalization technology is an image processing technology that adjusts the histogram distribution of the color frame to make the pixel value distribution of the color frame more uniform, thereby enhancing the contrast of the image and obtaining a contrast-enhanced image.

[0117] In this embodiment, for the data collected by the multimodal sensor, the point cloud data is optimized by removing the ground plane, downsampling and other operations to reduce interference and data volume; the depth map is denoised and the color frame contrast is enhanced to improve the data quality.

[0118] The map construction module 20 is used to perform environmental positioning and map construction on the multimodal target data through SLAM technology to obtain an initial three-dimensional environmental map.

[0119] It should be noted that SLAM technology allows robots and other devices to simultaneously build a map of the environment and determine their own position in an unknown environment. It integrates information from multiple sensors, such as lidar, cameras, and IMUs. SLAM technology can accurately estimate the robot's position and build a consistent map.

[0120] It is understood that the initial 3D environment map is a preliminary map obtained by using SLAM technology to locate and map multimodal target data. It is obtained through SLAM technology using multimodal sensor data such as vision, laser, and inertial.

[0121] In the specific implementation, after obtaining the above-mentioned preprocessed data, the map construction module first uses the SLAM technology front-end to extract features, and the back-end is processed based on the probabilistic algorithm to match and fuse the data, determine the position of its own device and build a map, thereby obtaining an initial three-dimensional environment map.

[0122] In one embodiment, the map construction module 20 described in this embodiment is also used to solve the pose based on the contrast enhancement image and the target point cloud to obtain front-end pose data; the map construction module 20 is also used to fuse the target point cloud and the denoised depth map through SLAM technology based on the front-end pose data to generate a local dense three-dimensional grid; the map construction module 20 is also used to detect blank areas corresponding to perspective occlusion or feature missing in the current environment data; the map construction module 20 is also used to generate an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank areas.

[0123] It should be noted that the front-end pose data is the result obtained by visual inertial tracking at the front end in SLAM technology, which contains the position and pose information of the robot in three-dimensional space.

[0124] Understandably, a local dense 3D mesh is a 3D structure generated by fusing a target point cloud with a denoised depth map using SLAM technology. Constructed within a local area, it contains relatively dense 3D information and accurately represents the geometry of the local environment, the location of objects, and other information. This local dense 3D mesh is a crucial component in building a complete 3D environmental map.

[0125] It should be understood that blank areas are areas in the current environmental data that are not covered by normal sensor data or cannot be accurately described due to perspective occlusion or missing features. Perspective occlusion may be caused by objects in the environment blocking each other, such as a large box blocking the wall behind it, making it difficult for sensors (such as cameras or lidar) to obtain information about the obscured part. Missing features may be due to the lack of sufficient recognizable features such as texture and shape in certain areas of the environment, making it difficult for sensors to accurately describe these areas, resulting in areas of information blank when building maps or modeling the environment.

[0126] In this implementation, by solving the front-end pose data, the robot's initial position and posture in the environment can be accurately determined. Based on this front-end pose data, SLAM technology is used to fuse the target point cloud and the denoised depth map to generate a local dense 3D mesh. This helps to more accurately construct a local environment model and obtain richer environmental information. Furthermore, detecting blank areas and generating an initial 3D environment map based on the local dense 3D mesh and blank areas can make map construction more complete.

[0127] The semantic reasoning module 30 is used to infer the environmental semantic information in the current environmental data through a visual language model to obtain a semantically completed map when there are blank areas in the initial three-dimensional environmental map.

[0128] It should be noted that the visual language model is an artificial intelligence model that combines computer vision and natural language processing to understand and reason about the semantic information of visual information (such as images, videos, etc.).

[0129] Understandably, environmental semantic information refers to information with semantic meaning contained in the current environmental data. This information describes high-level information such as the nature of objects in the environment, the relationships between objects (such as relative positions and interactions), and the overall scene type (e.g., the scene the robot is in).

[0130] The semantically completed map is a map created by inferring semantic information about the environment using a visual language model, even when there are blank areas in the initial 3D environment map. This map complements the blank areas in the initial 3D environment map, using semantic information to fill in the missing parts and make the map more complete. For example, in a blank area in the initial 3D environment map, the semantically completed map might infer that this area is part of a bookshelf based on the surrounding semantic information, thereby improving the overall structure and information integrity of the map.

[0131] In a specific implementation, when there are blank areas in the initial three-dimensional environment map, the semantic reasoning module can use the powerful semantic understanding and reasoning capabilities of the VLM to mine the environmental semantic information in the current environmental data and obtain a semantic completion map corresponding to the missing voxels.

[0132] The map fusion module 40 is configured to fuse the semantically completed map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

[0133] It should be noted that the dense map is generated by fusing the semantically completed map with the initial 3D environment map, resulting in a precise description of the environment. Compared to the initial 3D environment map, the dense map has a higher information density and can provide a more accurate description of the environment, including small obstacles, subtle terrain changes, and the completion of blank areas due to view occlusion or missing features.

[0134] In the specific implementation, the map fusion module fuses the semantic completion map with the initial three-dimensional environment map. According to a certain weight distribution, the information in the semantic completion map can be filled into the blank areas of the initial three-dimensional environment map to make the map information more complete, and finally obtain a dense map corresponding to the current environment data.

[0135] Other embodiments or specific implementations of the map construction system based on VLM and SLAM positioning of this application can refer to the above-mentioned method embodiments and will not be repeated here.

[0136] The map construction system based on VLM and SLAM positioning provided by this application adopts the map construction method based on VLM and SLAM positioning in the above-mentioned embodiment, which can solve the technical problem that traditional SLAM technology relies on sensor data to construct three-dimensional maps, and there are data missing or positioning errors in complex or occluded environments, resulting in incomplete three-dimensional maps or reduced accuracy. Compared with the existing technology, the beneficial effects of the map construction system based on VLM and SLAM positioning provided by this application are the same as the beneficial effects of the map construction method based on VLM and SLAM positioning provided by the above-mentioned embodiment, and the other technical features of the map construction system based on VLM and SLAM positioning are the same as the features disclosed in the above-mentioned embodiment method, and are not repeated here.

[0137] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the map construction method and system based on VLM and SLAM positioning of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0138] The above description is only part of the embodiments of the present application and does not limit the scope of protection of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the scope of protection of the present application.

Claims

1. A map construction method based on VLM and SLAM positioning, characterized in that: The method comprises: Preprocessing current environmental data to obtain corresponding multimodal target data, wherein the current environmental data is collected based on a preset multimodal sensor; Performing environmental positioning and mapping on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map; When there are blank areas in the initial three-dimensional environment map, reasoning about the environment semantic information in the current environment data using a visual language model to obtain a semantically completed map; The semantic completion map is fused with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

2. The method according to claim 1, wherein The multimodal sensor includes a laser radar and a depth camera. The step of preprocessing the current environment data to obtain corresponding multimodal target data includes: Extracting sparse point cloud data, color frames, and depth maps from the current environment data, wherein the sparse point cloud data is obtained by scanning the laser radar, and the color frames and depth maps are obtained by photographing the depth camera; Performing ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; Performing bilateral filtering to denoise the depth map to obtain a denoised depth map; performing image processing on the color frame by using a histogram equalization technique to obtain a contrast enhanced image; The target point cloud, the denoised depth map and the contrast-enhanced map are used as multimodal target data.

3. The method according to claim 2, wherein The step of performing environmental positioning and mapping on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map includes: Performing pose solving according to the contrast enhanced image and the target point cloud to obtain front-end pose data; Based on the front-end pose data, the target point cloud and the denoised depth map are fused using SLAM technology to generate a local dense three-dimensional grid; Detecting blank areas corresponding to view angle occlusion or feature missing in the current environment data; An initial three-dimensional environment map is generated according to the local dense three-dimensional grid and the blank area.

4. The method according to claim 3, wherein The step of detecting a blank area corresponding to view angle occlusion or feature missing in the current environment data includes: Detecting missing voxels in the current environment data, where the missing voxels are caused by view occlusion or feature loss; determining a void rate based on the missing voxels and the total expected voxels of the current environment data; When the void ratio reaches a preset threshold, the missing voxels are regarded as blank areas.

5. The method according to claim 2, wherein When there is a blank area in the initial three-dimensional environment map, the step of inferring the environment semantic information in the current environment data by using a visual language model to obtain a semantically completed map includes: When there is a blank area in the initial three-dimensional environment map, performing image recognition on the contrast-enhanced image through a CLIP model to generate an instance mask; Projecting the instance mask onto the denoised depth map and constructing semantic anchor nodes in combination with an industry database; Based on the semantic anchor nodes, the environmental semantic information in the current environmental data is inferred through a visual language model to obtain a semantic completion map.

6. The method according to claim 5, wherein The step of projecting the instance mask onto the denoised depth map and constructing a semantic anchor node in combination with an industry database includes: Extracting depth information of the instance mask according to a mapping relationship between image pixels and the denoised depth map to obtain an instance point cloud; Traversing the instance point cloud to determine the axis-aligned bounding box and the corresponding center point; According to the class label of the instance mask, the size of the axis-aligned bounding box is compared with the standard size of an industry database to obtain a size trustworthy flag, wherein the industry database stores standard size information of various devices; The class label, the axis-aligned bounding box, the center point, and the size-trusted marker are integrated to construct a semantic anchor node.

7. The method according to claim 6, wherein The step of fusing the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data includes: Determining a geometric TSDF weight based on the current environment data and the multimodal sensor; Determining a semantic completion TSDF weight of the semantic completion map according to the size trustworthy flag; Based on the geometric TSDF weights and the semantic completion TSDF weights, the semantic completion map is fused with the initial three-dimensional environment map to obtain a dense map.

8. A map construction system based on VLM and SLAM positioning, characterized in that: The system performs the method according to claim 1, and the system comprises: A data acquisition module is used to pre-process current environmental data to obtain corresponding multimodal target data, wherein the current environmental data is collected based on a preset multimodal sensor; A map construction module is used to perform environmental positioning and map construction on the multimodal target data using SLAM technology to obtain an initial three-dimensional environmental map; A semantic reasoning module, configured to, when there are blank areas in the initial three-dimensional environment map, infer the environment semantic information in the current environment data using a visual language model to obtain a semantically completed map; A map fusion module is used to fuse the semantic completion map with the initial three-dimensional environment map to obtain a dense map corresponding to the current environment data.

9. The system according to claim 8, wherein The multimodal sensor includes a laser radar and a depth camera; The data acquisition module is further configured to extract sparse point cloud data, color frames, and depth maps from the current environment data, wherein the sparse point cloud data is obtained by scanning the laser radar, and the color frames and depth maps are obtained by capturing the depth camera; The data acquisition module is further configured to perform ground plane removal and downsampling processing on the sparse point cloud data to obtain a target point cloud; The data acquisition module is further configured to perform bilateral filtering and denoising on the depth map to obtain a denoised depth map; The data acquisition module is further configured to perform image processing on the color frame using a histogram equalization technique to obtain a contrast-enhanced image; The data acquisition module is further configured to use the target point cloud, the denoised depth map, and the contrast-enhanced map as multimodal target data.

10. The system according to claim 9, wherein: The map construction module is further used to perform pose solving based on the contrast enhancement map and the target point cloud to obtain front-end pose data; The map construction module is further configured to fuse the target point cloud and the denoised depth map using SLAM technology based on the front-end pose data to generate a local dense three-dimensional grid; The map construction module is further configured to detect blank areas corresponding to view angle occlusion or feature missing in the current environment data; The map construction module is further configured to generate an initial three-dimensional environment map based on the local dense three-dimensional grid and the blank area.

Images