Three-dimensional reconstruction and semantic map generation method and system based on Gaussian sputtering

Abstract

This application relates to the field of computer vision technology, and in particular to a method and system for 3D reconstruction and semantic map generation based on Gaussian sputtering. The method includes: acquiring a color image and depth point cloud data of a target scene; obtaining feature point cloud data based on the Gaussian sputtering algorithm and the depth point cloud data; aligning the depth point cloud data and feature point cloud data based on a pose calibration algorithm to obtain point cloud fusion data, and updating the Gaussian sphere kernel to perform 3D reconstruction of the target scene; mapping the updated Gaussian sphere kernel onto the color image to obtain a 2D point cloud texture; inputting the 2D point cloud texture into a semantic segmentation model to obtain a class mask for each pixel position in the 2D point cloud texture; and mapping the class mask back onto the 3D reconstructed target scene to obtain a semantic map corresponding to the target scene. This application reduces hardware procurement and maintenance costs while enabling online generation of semantic maps, providing convenience for subsequent applications.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for 3D reconstruction and semantic map generation based on Gaussian sputtering. Background Art

[0002] With the rapid development of smart city construction, industrial site digitization, and unmanned systems (such as drone inspection, unmanned vehicle inspection, indoor security robots, etc.), the demand for high-precision and high-efficiency 3D reconstruction and semantic annotation of target scenes or equipment is increasing.

[0003] Traditional 3D reconstruction methods typically employ LiDAR point cloud scanning, multi-view stereo reconstruction, or neural rendering. LiDAR point cloud scanning often requires high-precision static or mechanically rotating LiDAR, which is costly (a single unit can cost hundreds of thousands to millions of RMB). It also has strict requirements on manpower and the on-site environment. Furthermore, point cloud stitching, filtering, downsampling, and reconstruction often rely on offline batch processing, resulting in long cycles (usually several days) and high computational resource consumption. Multi-view stereo reconstruction requires high-precision positioning (GNSS / IMU / RTK) or strict multi-camera synchronous acquisition. It is also prone to reconstruction holes in weak textures (pure white walls, featureless scenes), and its rendering efficiency is low. The generated network / texture maps are large, which is not conducive to real-time rendering and mobile terminal browsing. While implicit scene methods such as neural rendering can achieve high-precision geometry and texture to a certain extent, they require a large number of training samples and training time, making it difficult to meet the needs of rapid on-site modeling, real-time iteration, and low-bandwidth transmission.

[0004] The aforementioned traditional methods struggle to achieve a good balance between cost, efficiency, and quality, resulting in problems such as high equipment procurement and maintenance costs, long modeling cycles, inability to meet the real-time requirements of unmanned systems, and lack of online semantic annotation and efficient visual interaction when deployed in real-world scenarios. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the existing technology, this application provides a method and system for 3D reconstruction and semantic map generation based on Gaussian sputtering. It solves the technical problems of high equipment procurement and maintenance costs, long modeling cycle and insufficient real-time performance of the current 3D reconstruction method, as well as the technical problems of lack of online semantic annotation and efficient visualization interaction, which prevent the full release of subsequent application value.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted in this application include:

[0009] In a first aspect, embodiments of this application provide a method for 3D reconstruction and semantic map generation based on Gaussian sputtering, including:

[0010] Acquire color and depth images of the target scene, and extract depth point cloud data of the target scene based on the depth images;

[0011] Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data.

[0012] Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel.

[0013] The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture.

[0014] The category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain the semantic map corresponding to the target scene.

[0015] Optionally, in one specific embodiment, the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel, including:

[0016] Based on pre-set device parameters and geographic tiling algorithms, the target scene is divided into multiple sub-regions. Each sub-region includes a set of point cloud locations corresponding to that region in the point cloud fusion data, and a set of Gaussian sphere kernels corresponding to that region.

[0017] Based on a pre-configured cluster environment, each sub-region is scheduled in parallel for 3D reconstruction. During this process, each sub-region is divided into LOD levels based on a pre-configured LOD strategy, and all sub-regions are stitched together based on the division results to complete the 3D reconstruction of the target scene.

[0018] The LOD strategy is used to dynamically filter the LOD level corresponding to each point cloud location based on the pre-set user viewpoint or bandwidth status.

[0019] LOD layers include distant layers, near layers, or the finest layer.

[0020] Optionally, in one specific embodiment, performing 3D reconstruction of the target scene based on the updated Gaussian sphere kernel further includes:

[0021] After performing 3D reconstruction of the target scene based on the updated Gaussian sphere kernel, the corresponding local point density / hole rate is obtained based on the point cloud fusion data, and the corresponding hole region is determined according to the local point density / hole rate.

[0022] Based on the location of the cavity region, the color image and point cloud data corresponding to the cavity region are selected from the color image and the feature point cloud data, respectively.

[0023] The color image and point cloud data corresponding to the void region to be compensated are input into a pre-trained lightweight neural implicit network to compensate the 3D reconstruction results of the target scene.

[0024] The lightweight neural implicit network is used to: fit a local neural implicit function representing the cavity region based on the color image to be compensated and the point cloud data to be compensated, and obtain the compensated point cloud corresponding to the cavity region by processing the local neural implicit function.

[0025] Based on the pre-set compensation weights, the compensation point cloud is superimposed on the 3D reconstruction result of the target scene in the form of a Gaussian sphere kernel. The Gaussian sphere kernel of the hole region and its adjacent region is subjected to Laplacian smoothing and normal reestimation to compensate for the 3D reconstruction result of the target scene.

[0026] Optionally, in a specific embodiment, based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data, including:

[0027] Based on the color image and / or the depth image, obtain the radiation weight corresponding to each point cloud position in the depth point cloud fusion data;

[0028] Based on the radiation weight corresponding to each point cloud position in the deep point cloud fusion data, and the pre-set Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the deep point cloud data to obtain the corresponding feature point cloud data.

[0029] Optionally, in a specific embodiment, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped onto the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain a semantic map corresponding to the target scene, including:

[0030] Based on the pre-set camera extrinsic and intrinsic parameters, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene to obtain the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0031] Based on the geometric features of the point cloud fusion data and the texture features of the color image, determine whether the target scene belongs to a weak texture scene;

[0032] If it belongs to the category, the point cloud fusion data is input into a pre-set 3D point cloud segmentation model to obtain the second semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0033] Based on a pre-set label filtering algorithm, the first semantic label and the second semantic label corresponding to each Gaussian sphere kernel are filtered to determine the final label and assign it to the corresponding Gaussian sphere kernel in order to obtain the semantic map corresponding to the target scene.

[0034] If it does not belong, then assign the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene to obtain the semantic map corresponding to the target scene;

[0035] The label filtering algorithm includes a confidence-weighted algorithm or a voting mechanism algorithm;

[0036] The semantic tags include a first semantic tag or a second semantic tag.

[0037] Optionally, in a specific embodiment, after obtaining the semantic map corresponding to the target scene, the method further includes:

[0038] Based on a pre-set label classification strategy, all semantic labels in the semantic map are classified, and each semantic label in the semantic map, the Gaussian sphere kernel corresponding to each semantic label, and the classification result are recorded as a set of semantic data in a pre-set semantic database.

[0039] In addition, assign a corresponding call / export interface to each set of semantic data.

[0040] Optionally, in a specific embodiment, after obtaining the semantic map corresponding to the target scene, the method further includes:

[0041] Based on the 3D reconstruction results and semantic map corresponding to the target scene, as well as the pre-set rendering strategy, the target scene is rendered in real time.

[0042] The rendering strategies include point cloud particle rendering, mesh rendering, PBR rendering, or LOD layered rendering.

[0043] Optionally, in one specific embodiment, acquiring a color image and a depth image of the target scene, and extracting depth point cloud data of the target scene based on the depth image, includes:

[0044] Determine whether the pre-set RAG knowledge base contains depth point cloud data corresponding to the target scene;

[0045] If it exists, a color image of the target scene is acquired through an image acquisition device, and the depth point cloud data corresponding to the target scene is called from the point cloud knowledge base;

[0046] If not, a color image and a depth image of the target scene are acquired through an image acquisition device, and depth point cloud data of the target scene is extracted based on the depth image.

[0047] Secondly, embodiments of this application provide a 3D reconstruction and semantic map generation system based on Gaussian sputtering, including: an image acquisition device and a cloud platform;

[0048] The image acquisition device is used to acquire color images and depth images of the target scene, and send the color images and depth images to the cloud platform;

[0049] The cloud platform is used to extract depth point cloud data of the target scene based on the depth image;

[0050] Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data.

[0051] Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel.

[0052] The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture.

[0053] The category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain the semantic map corresponding to the target scene.

[0054] Optionally, in one specific embodiment, the system further includes: a data twin platform;

[0055] The digital twin platform is used to render the target scene in real time based on the 3D reconstruction results and semantic map output by the cloud platform, as well as the received sensor dynamic data of the target scene, so as to realize the real-time mapping of the target scene.

[0056] (III) Beneficial Effects

[0057] This application presents a method for 3D reconstruction and semantic map generation based on Gaussian sputtering. It performs subsequent 3D reconstruction using color and depth images, eliminating the need for expensive high-precision LiDAR and significantly reducing hardware procurement and maintenance costs. Moreover, the 3D reconstruction using Gaussian sputtering greatly improves the speed compared to traditional methods such as MVS / SfM or NeRF. Furthermore, the online generation of semantic maps facilitates subsequent applications. Attached Figure Description

[0058] Figure 1 A flowchart of a 3D reconstruction and semantic map generation method based on Gaussian sputtering provided in an embodiment of this application;

[0059] Figure 2 A schematic diagram of the architecture of the 3D reconstruction and semantic map generation method based on Gaussian sputtering provided in the embodiments of this application;

[0060] Figure 3 A flowchart of the 3DGS three-dimensional reconstruction process provided in the embodiments of this application;

[0061] Figure 4 This is a schematic diagram of partitioned parallel computing and LOD management provided for an embodiment of this application;

[0062] Figure 5 This is a schematic diagram of the neural latent cavity compensation process provided in the embodiments of this application;

[0063] Figure 6 A schematic diagram of three-dimensional semantic segmentation and two-dimensional projection provided for embodiments of this application;

[0064] Figure 7 This is a schematic diagram of a single-device desktop-level automated shooting pipeline provided in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram of single-device automatic alignment and component-level semantic annotation provided in an embodiment of this application. Detailed Implementation

[0066] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0067] With the rapid development of smart city construction, industrial site digitization, and unmanned systems (such as drone inspection, unmanned vehicle inspection, indoor security robots, etc.), the demand for high-precision and high-efficiency 3D reconstruction and semantic annotation of target scenes or equipment is increasing.

[0068] Current 3D reconstruction methods typically employ LiDAR point cloud scanning, multi-view stereo reconstruction, or neural rendering. LiDAR point cloud scanning often requires the purchase of high-precision static or mechanically rotating LiDAR, which is costly (a single unit can cost hundreds of thousands to millions of RMB) and has strict requirements on manpower and the on-site environment. In contrast, the embodiments of this application only require color and depth images; that is, only a regular RGB camera and a lightweight depth sensor are needed, eliminating the need for expensive high-precision LiDAR and significantly reducing hardware procurement and maintenance pressure. Furthermore, LiDAR point cloud scanning, through point cloud stitching, filtering, downsampling, and reconstruction, often uses offline batch processing, resulting in long cycles (usually several days) and high computational resource consumption. In contrast, the embodiments of this application utilize the 3DGS Gaussian sputtering algorithm for 3D reconstruction, greatly improving reconstruction speed.

[0069] Multi-view stereo reconstruction requires high-precision positioning (GNSS / IMU / RTK) or strict multi-camera synchronous acquisition. Furthermore, weak textures (pure white walls, featureless scenes) are prone to reconstruction holes, resulting in low rendering efficiency and large network / texture maps, which are unfavorable for real-time rendering and mobile terminal browsing. While implicit scene methods such as neural rendering can achieve high-precision geometry and texture to some extent, they require a large number of training samples and training time, making it difficult to meet the needs of rapid on-site modeling, real-time iteration, and low-bandwidth transmission. This application's embodiment, by combining multi-view depth fusion and neural implicit hole compensation, can effectively restore weak texture areas and address large-scale occlusion holes, achieving centimeter-level accuracy for indoor dimensions. Through partitioned parallelism and LOD management, the system can elastically expand cluster resources based on data volume, completing 3D reconstruction and semantic annotation in batches online without manual intervention or cumbersome offline operations. Moreover, the embodiment of this application generates an online 3D semantic map with multi-level semantic labels, directly facilitating subsequent calls.

[0070] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0071] This application provides a method for 3D reconstruction and semantic map generation based on Gaussian sputtering, such as... Figure 1 and Figure 2 As shown, it includes:

[0072] Acquire color and depth images of the target scene, and extract depth point cloud data of the target scene based on the depth images;

[0073] Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data.

[0074] Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel.

[0075] The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture.

[0076] The category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain the semantic map corresponding to the target scene.

[0077] This embodiment uses color and depth images for subsequent 3D reconstruction, eliminating the need for expensive high-precision LiDAR and significantly reducing hardware procurement and maintenance costs. Moreover, the use of Gaussian sputtering for 3D reconstruction greatly improves the speed compared to traditional methods such as MVS / SfM or NeRF. Furthermore, the online generation of semantic maps facilitates subsequent applications.

[0078] Optionally, in one specific embodiment, acquiring the color image and depth image of the target scene includes:

[0079] For large-scale outdoor scenes, a multi-source collaborative acquisition method can be used, combining drones (aerial platform) and ground-based mobile mapping vehicles (ground platform). The drones, equipped with high-resolution RGB cameras and lightweight ToF / structured light depth cameras, perform low-altitude aerial photography along a pre-set flight path. The mobile mapping vehicles, equipped with airborne LiDAR and RGB-SLAM modules, supplement the data acquisition for obstructed / hollow areas on the ground. This method allows for the acquisition of color and depth images of large-scale outdoor scenes.

[0080] For enclosed indoor spaces, a handheld structured light depth camera (such as the Intel RealSense D4xx series) and a 360° panoramic camera are used together to acquire data, recording the camera pose in real time (using SLAM front-end positioning). The scanning process can be paused, resumed, and stitched together at any time. This method allows for the acquisition of color and depth images of the enclosed indoor space.

[0081] For single-object / single-device level, including programmable turntable and light source imaging device, the object under test is placed on the turntable, and the device automatically performs multi-view RGB+structured light depth acquisition to generate hundreds of high-definition color images and depth images.

[0082] This embodiment uses a regular RGB camera and a lightweight depth sensor, eliminating the need for expensive high-precision LiDAR and significantly reducing hardware procurement and maintenance costs.

[0083] Furthermore, the acquired RGB images (color images) and depth maps / coarse point clouds (depth images) are uploaded to a cloud platform for preprocessing. The preprocessing of the color images includes distortion correction, white balance, automatic contrast and noise reduction (such as bilateral filtering and CLAHE adaptive histogram equalization); the preprocessing of the depth images includes depth filtering, preliminary hole compensation (hole difference), and coarse normal estimation.

[0084] Optionally, in a specific embodiment, based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data, including:

[0085] Based on the color image and / or the depth image, obtain the radiation weight corresponding to each point cloud position in the depth point cloud fusion data;

[0086] Based on the radiation weight corresponding to each point cloud position in the deep point cloud fusion data, and the pre-set Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the deep point cloud data to obtain the corresponding feature point cloud data.

[0087] Specifically, the 3DGS 3D reconstruction flowchart is as follows: Figure 3 As shown, the corresponding depth point cloud data is extracted from each preprocessed depth image;

[0088] Based on each point cloud location in deep point cloud data (or structured light deep point cloud). Initialize a Gaussian splat, i.e., the Gaussian sputtering algorithm is expressed as:

[0089] ;

[0090] in, The coordinates of the point cloud position. The covariance matrix (which can be initially set to...) , (where I is the radius of the Gaussian sphere kernel and I is the identity matrix), and X is a random position on the Gaussian sphere kernel, with each point cloud position having a preset value. Color / radiance weights (obtained from the color of a color image or the depth confidence of a depth image).

[0091] This embodiment uses the 3DGS Gaussian sputtering algorithm, which can improve the reconstruction speed by several times compared with traditional MVS / SfM or NeRF methods. In typical outdoor scenes (several thousand square meters in radius), reconstruction can be completed in just a few minutes to several hours.

[0092] Optionally, in a specific embodiment, based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are respectively mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data, and the Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, including:

[0093] Using SLAM / IMU / GNSS pose calibration (pose calibration algorithm), the depth point cloud data of each frame is uniformly transformed to the world coordinate system; for outdoor scenes, coarse registration is first performed (based on LiDAR and RGB features), and then global optimization is performed through ICP (Iterative Closest Point); for indoor scenes, global graph optimization (Bundle Adjustment) is performed using RGB-SLAM front-end pose.

[0094] Align the point cloud fusion data and feature point cloud data based on the world coordinate system to obtain fused point cloud data. The aligned fused point cloud data includes normals. With confidence information This allows for further updating of the covariance matrix of each Gaussian sphere. With color / radiance weight Specifically:

[0095] ;

[0096] Wherein, the covariance matrix Adjustments can be made to ensure consistency between point cloud density and normals. To ensure matrix invertibility, the function... The weights are dynamically increased or decreased based on the confidence level.

[0097] This embodiment further optimizes the Gaussian sphere kernel to ensure the accuracy of subsequent 3D modeling.

[0098] Optionally, in one specific embodiment, the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel, including:

[0099] Based on pre-set device parameters and geographic tiling algorithms, the target scene is divided into multiple sub-regions. Each sub-region includes a set of point cloud locations corresponding to that region in the point cloud fusion data, and a set of Gaussian sphere kernels corresponding to that region.

[0100] Based on a pre-configured cluster environment, each sub-region is scheduled in parallel for 3D reconstruction. During this process, each sub-region is divided into LOD levels based on a pre-configured LOD strategy, and all sub-regions are stitched together based on the division results to complete the 3D reconstruction of the target scene.

[0101] The LOD strategy is used to dynamically filter the LOD level corresponding to each point cloud location based on the pre-set user viewpoint or bandwidth status.

[0102] LOD layers include, but are not limited to, distant layers, near layers, or the finest layers.

[0103] Specifically, partitioned parallel computing and LOD management, such as Figure 4 As shown, when the point cloud fusion data to be reconstructed exceeds the limitations of a single machine or video memory (device parameters), a geographic tiling strategy is adopted to divide the large-scale scene into several sub-regions (tiles) according to a grid. Each sub-region contains a Gaussian sphere kernel set corresponding to the local point cloud set.

[0104] In a cluster environment (Kubernetes / ECS / GPU elastic scaling), 3DGS computation tasks for each tile are scheduled in parallel. Each node independently runs Gaussian sputtering rendering to generate dense point clouds.

[0105] Multi-resolution LOD (Level of Detail) management is employed.

[0106] • Dynamically extract Gaussian radius layers of different sizes (e.g., long-distance layer, short-distance layer, finest layer) based on the user's view distance or bandwidth.

[0107] • Gaussian kernel data for each layer can be generated in parallel during the reconstruction phase and merged to form a LOD cache for the 3D scene, enabling rapid switching during real-time rendering on Web / Mobile devices.

[0108] This embodiment uses partitioned parallelism and LOD management, allowing the system to elastically expand cluster resources based on data volume and complete 3D reconstruction and semantic annotation in batches online without manual intervention or cumbersome offline operations.

[0109] Optionally, in one specific embodiment, performing 3D reconstruction of the target scene based on the updated Gaussian sphere kernel further includes:

[0110] After performing 3D reconstruction of the target scene based on the updated Gaussian sphere kernel, the corresponding local point density / hole rate is obtained based on the point cloud fusion data.

[0111] Based on the local point density / hole rate, the color image and point cloud data corresponding to the hole region are selected from the color image and the feature point cloud data, respectively.

[0112] The color image and point cloud data corresponding to the void region to be compensated are input into a pre-trained lightweight neural implicit network to compensate the 3D reconstruction results of the target scene.

[0113] The lightweight neural implicit network is used to: fit the corresponding local implicit scene field based on the color image to be compensated and the point cloud data to be compensated, and obtain the compensated point cloud corresponding to the hole region based on the local implicit scene field.

[0114] Based on the pre-set compensation weights, the compensation point cloud is superimposed onto the 3D reconstruction result of the target scene in the form of a Gaussian sphere kernel, and Laplacian smoothing and normal reestimation are performed to compensate for the 3D reconstruction result of the target scene.

[0115] Specifically, the neural implicit cavity compensation process is as follows: Figure 5 As shown, for weakly textured areas in indoor spaces (such as pure white walls and dark areas) and large-scale occlusion holes in outdoor spaces (tree shadows, building shadows), after initial 3DGS reconstruction, the areas requiring compensation are determined based on the "hole rate" or "local point density". For each hole sub-region, a lightweight neural implicit network (NeRFPatch) is constructed for local training and sampling:

[0116] Using the surrounding high-density point cloud and corresponding RGB image as input, a local implicit scene is quickly fitted to infer the geometric and texture information inside the hole.

[0117] • The sampled point cloud is superimposed onto the original 3DGS result using a small-weight Gaussian kernel to achieve a smooth transition.

[0118] Laplacian smoothing and normal reestimation are performed on the compensated point cloud to ensure the geometric and textural consistency of the final 3D model.

[0119] This embodiment combines multi-view depth fusion and neural implicit hole compensation, which can effectively restore weak texture areas and large-scale occlusion hole problems, and the indoor dimensional accuracy can reach the centimeter level (≤3 cm).

[0120] Optionally, in a specific embodiment, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped onto the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain a semantic map corresponding to the target scene, including:

[0121] Based on the pre-set camera extrinsic and intrinsic parameters, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene to obtain the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0122] Based on the point cloud fusion data, determine whether the target scene belongs to a weak texture scene;

[0123] If it belongs to the category, the point cloud fusion data is input into a pre-set 3D point cloud segmentation model to obtain the second semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0124] Based on a pre-set label filtering algorithm, the first semantic label and the second semantic label corresponding to each Gaussian sphere kernel are filtered to determine the final label and assign it to the corresponding Gaussian sphere kernel in order to obtain the semantic map corresponding to the target scene.

[0125] If it does not belong, then assign the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene to obtain the semantic map corresponding to the target scene;

[0126] The label filtering algorithm includes a confidence-weighted algorithm or a voting mechanism algorithm;

[0127] The semantic tags include a first semantic tag or a second semantic tag.

[0128] Specifically, 3D semantic segmentation and 2D projection, such as Figure 6 As shown, the dense point cloud (point cloud fusion data) generated by 3DGS is projected onto the corresponding RGB image (color image) acquired at the time of acquisition, forming several two-dimensional "point cloud textures". Specifically, the updated Gaussian sphere kernel from the point cloud fusion data is mapped onto the color image to obtain the two-dimensional point cloud textures.

[0129] A pre-trained Mask R-CNN or DeepLabV3+ model is loaded into the cloud to perform pixel-level semantic segmentation on each texture, obtaining pixel-level category masks (such as "building wall", "ground", "tree", "street lamp", "pipe", etc.). The 2D point cloud texture is then input into the pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the 2D point cloud texture.

[0130] Using camera extrinsic (pose) and intrinsic parameters, the 2D segmentation mask is inversely mapped back to the 3D point cloud, corresponding to each Gaussian kernel, and assigned a "semantic label" (such as Label_ID, Label_Name, confidence score). In other words, the category mask corresponding to each pixel position in the 2D point cloud texture is inversely mapped to the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian kernel in the 3D reconstructed target scene is assigned to obtain a semantic map corresponding to the target scene, including:

[0131] Based on the pre-set camera extrinsic and intrinsic parameters, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene to obtain the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0132] Based on the point cloud fusion data, determine whether the target scene belongs to a weak texture scene;

[0133] If it belongs to the category, the point cloud fusion data is input into a pre-set 3D point cloud segmentation model to obtain the second semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene.

[0134] Based on a pre-set label filtering algorithm, the first semantic label and the second semantic label corresponding to each Gaussian sphere kernel are filtered to determine the final label and assign it to the corresponding Gaussian sphere kernel in order to obtain the semantic map corresponding to the target scene.

[0135] If it does not belong, then assign the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene to obtain the semantic map corresponding to the target scene;

[0136] The label filtering algorithm includes a confidence-weighted algorithm or a voting mechanism algorithm;

[0137] The semantic tags include a first semantic tag or a second semantic tag.

[0138] This embodiment improves the accuracy of annotation by using dual-path fusion annotation: 2D Mask R-CNN segmentation re-projection and 3D PointNet++ segmentation are fused with confidence weights.

[0139] Furthermore, after obtaining the semantic map corresponding to the target scene, the method further includes:

[0140] Based on a pre-set label classification strategy, all semantic labels in the semantic map are classified, and each semantic label in the semantic map, the Gaussian sphere kernel corresponding to each semantic label, and the classification result are recorded as a set of semantic data in a pre-set semantic database.

[0141] In addition, assign a corresponding call / export interface to each set of semantic data.

[0142] Specifically, design a multi-level tagging system:

[0143] • Primary tags: Buildings / facilities, Ground / roads, Vegetation / greenery, Equipment / pipelines, Furniture / Appliances, etc.;

[0144] • Secondary tags: For example, under buildings, they can be divided into "walls / roofs / doors and windows", under equipment, they can be divided into "distribution cabinets / valves / pump stations", and under pipelines, they can be divided into "water supply / drainage / cable troughs";

[0145] • Level 3 labels: such as labels for single equipment or components, such as "internal circuit breaker of distribution cabinet ①" or "valve flange connection".

[0146] The label information is mapped one-to-one with the Gaussian kernel in the database to form a table structure record (PointID→LabelID→Score→CategoryPath→Timestamp), and it supports exporting to CSV / JSON / GeoJSON and other formats for integration with BIM, GIS, ERP and other systems.

[0147] This embodiment improves the accuracy of semantic annotation by pre-associating semantic tags: directly binding the tag tree during point cloud projection.

[0148] Specifically, the system obtains the first semantic label (derived from 2D projection) corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene; based on the geometric features (such as normal variation and curvature) of the point cloud fusion data and / or the texture features (such as gradient and entropy) of the color image, it determines whether there are weak texture regions in the target scene; if weak texture regions exist, the point cloud fusion data (or its subset / derived representation) corresponding to the region is input into a pre-trained 3D point cloud segmentation model to obtain the second semantic label corresponding to the Gaussian sphere kernel in that region; based on a pre-set label fusion algorithm (such as a confidence-based weighted fusion algorithm or a majority voting mechanism algorithm), the first and second semantic labels corresponding to each Gaussian sphere kernel in the weak texture region are fused to determine the final semantic label and assign it to the Gaussian sphere kernel; for non-weak texture regions, the first semantic label is directly used as the final label; thus obtaining a semantic map that covers the entire target scene and has more robust semantic annotation.

[0149] Furthermore, after obtaining the semantic map corresponding to the target scene, the method further includes:

[0150] Based on the 3D reconstruction results and semantic map corresponding to the target scene, as well as the pre-set rendering strategy, the target scene is rendered in real time.

[0151] The rendering strategies include point cloud particle rendering, mesh rendering, PBR rendering, or LOD layered rendering.

[0152] Specifically, it features real-time visualization, interaction, and export integration: a web-based 3D preview, powered by the Three.js + Potree engine, enables real-time rendering of Gaussian sputtering point clouds in the browser. It supports multiple rendering modes: point cloud particle rendering, displaying high-density, high-precision point clouds in high-bandwidth or local environments; Mesh / PBR rendering, generating triangular meshes from the point cloud and applying textures, suitable for displays requiring higher detail; and LOD layered rendering, dynamically switching between different resolution layers based on the user's viewpoint and device performance.

[0153] Furthermore, the configured interactive functions include: navigation and roaming, where a preset inspection path is provided in the 3D scene, and users can click the "roam" button to automatically fly or move along the trajectory; the viewpoint can also be manually controlled (mouse / touch + keyboard). Sectioning and measurement, users can draw sectioning planes (rectangle / arbitrary polygons) to generate cross-sections in real time, supporting functions such as "straight-line distance measurement," "area measurement," and "volume calculation." Semantic highlighting and filtering, a tag tree is displayed on the right panel, where users can select or search for keywords (such as "streetlight" or "fire extinguisher"), and the system will highlight or flash the corresponding Gaussian kernel; when the mouse hovers over or clicks a Gaussian kernel, a floating information box is displayed, containing attributes such as category, confidence level, coordinates, and tile / floor. Natural language / voice question answering: It calls large models deployed in the cloud (such as Qwen3 / DespSeek V3 and other multimodal large models), and users can input commands such as "Please show the location of the power distribution cabinet in Building 2" or "Mark all fire hydrants" using Chinese typing or voice input. The front end will parse and highlight the corresponding area and provide text or voice feedback.

[0154] Furthermore, this embodiment supports one-click export of the following three data formats: GLTF / GLB: suitable for web-based or lightweight visualization calls; OBJ / PLY: for further processing by offline professional 3D software (such as Meshlab, CloudCompare); OctoMap / ROS Topic: for unmanned vehicles / drones to directly subscribe to path planning and obstacle avoidance in the ROS environment; Cesium 3DTiles: easy to load in 3D GIS platforms; IFC (BIM format): encapsulates geometric models and semantic tags, facilitating direct import into BIM platforms such as Revit / ArchiCAD. Custom export filtering is supported, for example, exporting only sub-scenes of a certain category ("distribution cabinet", "pipeline"), or exporting only the model and attributes of a specific floor / tile.

[0155] This embodiment generates an online 3D semantic map with multi-level semantic tags, which can be directly used by drones / unmanned vehicles / inspection robots to achieve a closed loop of the entire process of "path planning → obstacle avoidance → target localization". The platform provides OctoMap / ROS Topic format export and calling examples, which greatly reduces the threshold for unmanned system integration. This embodiment also sets up multiple interaction methods such as voice / natural language question and answer, point selection highlighting, and section roaming on the web / mobile terminal, allowing non-professional users to quickly locate targets in 3D scenes. It also supports the export of multiple 3D data formats (GLTF, OBJ, PLY, 3D Tiles, IFC, OctoMap) and provides a standard RESTful API for easy integration with third-party platforms (BIM, ERP, GIS, unmanned systems).

[0156] Optionally, in one specific embodiment, acquiring a color image and a depth image of the target scene, and extracting depth point cloud data of the target scene based on the depth image, includes:

[0157] Determine whether the pre-set RAG knowledge base contains depth point cloud data corresponding to the target scene;

[0158] If it exists, a color image of the target scene is acquired through an image acquisition device, and the depth point cloud data corresponding to the target scene is called from the point cloud knowledge base;

[0159] If not, a color image and a depth image of the target scene are acquired through an image acquisition device, and depth point cloud data of the target scene is extracted based on the depth image.

[0160] Specifically, such as Figure 7 and Figure 8 As shown, this embodiment supports automated pipeline modeling at the single object / single device level, that is:

[0161] Desktop automated shooting hardware design: Supports a programmable turntable with stepper motor control, supporting 360° continuous rotation, and preset rotation angle steps (e.g., shooting every 5°, for a total of 72 images). Multi-angle adjustable light source array: Supports multi-directional supplementary lighting from the front, back, and sides, softening shadows and improving the shooting quality of complex materials (high reflectivity, transparency, dark colors). Dual integration of RGB camera and structured light depth camera: Simultaneously acquires high-resolution RGB images (color images) and depth images (depth images) at the same shooting station, ensuring pixel alignment; automatically triggers image capture during shooting and transmits data to local or cloud servers.

[0162] Subsequently, batch data is uploaded and sorted: each time an object or part is captured, it is automatically named and uploaded to the server as "device number_batch number_angle number". The cloud automatically categorizes data with the same device number and generates a list of shooting metadata (including: shooting timestamp, turntable angle, exposure parameters, and light source intensity).

[0163] Furthermore, 3DGS batch modeling and fusion: Gaussian kernel initialization associates each frame's depth point cloud with its corresponding RGB texture, generating an initial Gaussian kernel set. Historical model retrieval and rapid alignment: A RAG (Retrieval-Augmented Generation) knowledge base is maintained in the cloud, storing a large number of CAD models or 3D point clouds corresponding to common equipment (such as motors, valves, distribution cabinets, and pump sets). When the captured object's number is detected to match a known model, the most similar point cloud / CAD is automatically retrieved from the knowledge base and initially rigidly aligned with the currently captured Gaussian kernel, thereby accelerating subsequent high-precision registration and fusion. After alignment, 3DGS Gaussian sputtering is performed, and high-resolution texture mapping is introduced, further improving details through optional neural implicit compensation.

[0164] Furthermore, component-level semantic annotation and attribute generation: Multimodal dual-stream large models (such as a Qwen3-based image-text joint model) are used upstream and downstream of a single device to automatically identify device components. Inputting the captured imagery of the device and its name / model, the system outputs a list of corresponding components (such as "bolt interface," "heat sink," "nameplate," "flange connection," etc.). Component-level segmentation is performed on the single-object point cloud using Mask R-CNN or PointNet++, and each segmentation result is encapsulated as a sub-model, generating a component attribute table (including metadata such as name, material, key dimensions, weight, and manufacturer). Finally, the complete high-precision 3D model of the single device (including multi-resolution LOD) and component-level semantics are encapsulated together into a pluggable asset bundle, such as in GLB / IFC / FBX formats.

[0165] Furthermore, digital twin online integration and IoT docking: After importing a single device model into the digital twin platform, it can be dynamically overlaid with IoT sensor data (temperature, vibration, pressure, flow, etc.) to achieve a linkage between "real-time on-site status + 3D visualization". Maintenance personnel can select a device on the web or mobile device to view the current "3D model + real-time sensor data + historical trend chart", and it supports "virtual disassembly"—animated simulation of part disassembly based on the component semantic annotation model. For equipment failure scenarios, a fault prediction engine can be invoked, combining the digital twin model with time-series sensor data to perform analyses such as "hot spot analysis", "vibration anomaly detection", and "lifespan prediction", and to provide pre-maintenance / replacement suggestions for key components.

[0166] This embodiment uses preset templates and the RAG knowledge base to support "one-click parameterization" and "rapid alignment of similar devices," further shortening the modeling and annotation cycle. It also integrates 3D models and IoT time-series data to achieve digital twins of a single device throughout its entire lifecycle, as well as fault prediction and maintenance decisions for industrial equipment. This embodiment can meet the needs of smart cities (park inspection, road maintenance, indoor security) and can also be extended to industrial manufacturing (digital twins of workshop equipment, component replacement simulation) and building management (BIM), demonstrating strong industry versatility.

[0167] Specifically, in this embodiment, after performing 3D reconstruction based on the optimized Gaussian sphere kernel set (point cloud fusion data), the 3D reconstruction results are analyzed to calculate the point density distribution of local regions and detect geometric hole regions. For the identified hole regions, based on their 3D positions, corresponding image patches are extracted from the original color image as the color image to be compensated, and corresponding sparse point clouds are extracted from the optimized point cloud data (avoiding the use of feature point cloud data) as the point cloud data to be compensated. The color image to be compensated and the point cloud data to be compensated are input into a pre-trained lightweight neural implicit network. The lightweight neural implicit network is used to: learn to fit the local continuous geometry and appearance representation (implicit scene field) of the hole region, and generate a dense compensation point cloud corresponding to the hole region by sampling the implicit field. Based on the preset fusion weights, the compensation point cloud is initialized as a new Gaussian sphere kernel and fused into the original 3D reconstruction result (Gaussian sphere kernel set). The fused region (including the newly added and neighboring Gaussian sphere kernels) is subjected to Laplacian smoothing to optimize the distribution, and its normal direction is re-estimated to ensure geometric continuity and rendering quality.

[0168] In addition, this embodiment provides a 3D reconstruction and semantic map generation system based on Gaussian sputtering, including: an image acquisition device and a cloud platform;

[0169] The image acquisition device is used to acquire color images and depth images of the target scene, and send the color images and depth images to the cloud platform;

[0170] The cloud platform is used to extract depth point cloud data of the target scene based on the depth image;

[0171] Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data.

[0172] Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel.

[0173] The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture.

[0174] The category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene, and a semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain the semantic map corresponding to the target scene.

[0175] In addition to this, the system also includes: a data twin platform;

[0176] The digital twin platform is used to render the target scene in real time based on the 3D reconstruction results and semantic map output by the cloud platform, as well as the received sensor dynamic data of the target scene, so as to realize the real-time mapping of the target scene.

[0177] This embodiment provides a 3D reconstruction and semantic map generation system based on Gaussian sputtering. It performs subsequent 3D reconstruction using color and depth images, eliminating the need for expensive high-precision LiDAR and significantly reducing hardware procurement and maintenance costs. Moreover, the 3D reconstruction using Gaussian sputtering greatly improves the speed compared to traditional methods such as MVS / SfM or NeRF. Furthermore, the online generation of semantic maps facilitates subsequent applications.

[0178] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0179] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0180] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0181] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0182] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for 3D reconstruction and semantic map generation based on Gaussian sputtering, characterized in that, include: Acquire color and depth images of the target scene, and extract depth point cloud data of the target scene based on the depth images; Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data. Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel. The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture. Based on the pre-set camera extrinsic and intrinsic parameters, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene to obtain the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene. Based on the geometric features of the point cloud fusion data and the texture features of the color image, determine whether the target scene belongs to a weak texture scene; If it belongs to the category, the point cloud fusion data is input into a pre-set 3D point cloud segmentation model to obtain the second semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene. Based on a pre-set label filtering algorithm, the first semantic label and the second semantic label corresponding to each Gaussian sphere kernel are filtered to determine the final label and assign it to the corresponding Gaussian sphere kernel in order to obtain the semantic map corresponding to the target scene. The label filtering algorithm includes a confidence-weighted algorithm or a voting mechanism algorithm.

2. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 1, characterized in that, The target scene is reconstructed in 3D based on the updated Gaussian sphere kernel, including: Based on pre-set device parameters and geographic tiling algorithms, the target scene is divided into multiple sub-regions. Each sub-region includes a set of point cloud locations corresponding to that region in the point cloud fusion data, and a set of Gaussian sphere kernels corresponding to that region. Based on a pre-configured cluster environment, each sub-region is scheduled in parallel for 3D reconstruction. During this process, each sub-region is divided into LOD levels based on a pre-configured LOD strategy, and all sub-regions are stitched together based on the division results to complete the 3D reconstruction of the target scene. The LOD strategy is used to dynamically filter the LOD level corresponding to each point cloud location based on the pre-set user viewpoint or bandwidth status. LOD layers include distant layers, near layers, or the finest layer.

3. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 1, characterized in that, The 3D reconstruction of the target scene based on the updated Gaussian sphere kernel also includes: After performing 3D reconstruction of the target scene based on the updated Gaussian sphere kernel, the corresponding local point density / hole rate is obtained based on the point cloud fusion data, and the corresponding hole region is determined according to the local point density / hole rate. Based on the location of the cavity region, the color image and point cloud data corresponding to the cavity region are selected from the color image and the feature point cloud data, respectively. The color image and point cloud data corresponding to the void region to be compensated are input into a pre-trained lightweight neural implicit network to compensate the 3D reconstruction results of the target scene. The lightweight neural implicit network is used to: fit a local neural implicit function representing the cavity region based on the color image to be compensated and the point cloud data to be compensated, and obtain the compensated point cloud corresponding to the cavity region by processing the local neural implicit function. Based on the pre-set compensation weights, the compensation point cloud is superimposed on the 3D reconstruction result of the target scene in the form of a Gaussian sphere kernel. The Gaussian sphere kernel of the hole region and its adjacent region is subjected to Laplacian smoothing and normal reestimation to compensate for the 3D reconstruction result of the target scene.

4. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 1, characterized in that, Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data, including: Based on the color image and / or the depth image, obtain the radiation weight corresponding to each point cloud position in the depth point cloud fusion data; Based on the radiation weight corresponding to each point cloud position in the deep point cloud fusion data, and the pre-set Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the deep point cloud data to obtain the corresponding feature point cloud data.

5. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 1, characterized in that, The method further includes: If the target scene is determined not to be a weakly textured scene, then the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene is assigned to obtain the semantic map corresponding to the target scene.

6. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 5, characterized in that, After obtaining the semantic map corresponding to the target scene, the method further includes: Based on a pre-set label classification strategy, all semantic labels in the semantic map are classified, and each semantic label in the semantic map, the Gaussian sphere kernel corresponding to each semantic label, and the classification result are recorded as a set of semantic data in a pre-set semantic database. In addition, assign a corresponding call / export interface to each set of semantic data.

7. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 5, characterized in that, After obtaining the semantic map corresponding to the target scene, the method further includes: Based on the 3D reconstruction results and semantic map corresponding to the target scene, as well as the pre-set rendering strategy, the target scene is rendered in real time. The rendering strategies include point cloud particle rendering, mesh rendering, PBR rendering, or LOD layered rendering.

8. The method for 3D reconstruction and semantic map generation based on Gaussian sputtering according to claim 1, characterized in that, Acquire color and depth images of the target scene, and extract depth point cloud data of the target scene based on the depth images, including: Determine whether the pre-set RAG knowledge base contains depth point cloud data corresponding to the target scene; If it exists, a color image of the target scene is acquired through an image acquisition device, and the depth point cloud data corresponding to the target scene is called from the point cloud knowledge base; If not, a color image and a depth image of the target scene are acquired through an image acquisition device, and depth point cloud data of the target scene is extracted based on the depth image.

9. A 3D reconstruction and semantic map generation system based on Gaussian sputtering, characterized in that, include: Image acquisition devices and cloud platforms; The image acquisition device is used to acquire color images and depth images of the target scene, and send the color images and depth images to the cloud platform; The cloud platform is used to extract depth point cloud data of the target scene based on the depth image; Based on the depth point cloud data and the Gaussian sputtering algorithm, Gaussian sphere kernel initialization is performed on each point cloud position in the depth point cloud data to obtain the corresponding feature point cloud data. Based on a pre-set pose calibration algorithm, the depth point cloud data and the feature point cloud data are mapped to the world coordinate system for alignment to obtain corresponding point cloud fusion data. The Gaussian sphere kernel of each point cloud position is updated based on the point cloud fusion data, and the target scene is reconstructed in three dimensions based on the updated Gaussian sphere kernel. The updated Gaussian sphere kernel in the point cloud fusion data is mapped onto the color image to obtain a two-dimensional point cloud texture. The two-dimensional point cloud texture is then input into a pre-trained semantic segmentation model to obtain the category mask corresponding to each pixel position in the two-dimensional point cloud texture. Based on the pre-set camera extrinsic and intrinsic parameters, the category mask corresponding to each pixel position in the 2D point cloud texture is reverse-mapped to the 3D reconstructed target scene to obtain the first semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene. Based on the geometric features of the point cloud fusion data and the texture features of the color image, determine whether the target scene belongs to a weak texture scene; If it belongs to the category, the point cloud fusion data is input into a pre-set 3D point cloud segmentation model to obtain the second semantic label corresponding to each Gaussian sphere kernel in the 3D reconstructed target scene. Based on a pre-set label filtering algorithm, the first semantic label and the second semantic label corresponding to each Gaussian sphere kernel are filtered to determine the final label and assign it to the corresponding Gaussian sphere kernel in order to obtain the semantic map corresponding to the target scene. The label filtering algorithm includes a confidence-weighted algorithm or a voting mechanism algorithm.

10. The 3D reconstruction and semantic map generation system based on Gaussian sputtering according to claim 9, characterized in that, The system also includes: a digital twin platform; The digital twin platform is used to render the target scene in real time based on the 3D reconstruction results and semantic map output by the cloud platform, as well as the received sensor dynamic data of the target scene, so as to realize the real-time mapping of the target scene.

Images