Non-reconstruction sparse three-dimensional Gaussian semantic modeling method for large scene

Abstract

The invention discloses a non-reconstruction sparse three-dimensional Gaussian semantic modeling method for a large scene, and the method comprises the steps: constructing a time-space mask synchronization module containing a time sequence propagation fusion and space merging correction sub-module, and improving the multi-view consistency of a semantic supervision signal; sparse initialization is realized for point cloud downsampling of a motion recovery structure (SfM), and the number of Gaussian primitives is reduced; and skipping an appearance reconstruction step, directly adding a semantic feature vector for a Gaussian primitive, rendering a two-dimensional semantic graph through micro rasterization, and performing end-to-end optimization on Gaussian parameters and semantic vectors based on L1 loss. According to the method, the traditional normal form of'first reconstruction and then semantic 'is broken, geometric artifact interference is effectively avoided, model storage occupation and calculation time consumption are greatly reduced while semantic modeling precision is guaranteed, high efficiency and robustness are considered, and the method is suitable for application scenes related to large-scene three-dimensional semantic modeling.

Description

Technical Field

[0001] This invention belongs to the technical field of semantic modeling, specifically relating to a reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes. Background Technology

[0002] 3D scene semantic modeling is a core research direction in the field of computer vision and graphics. It is widely used in many fields such as autonomous driving, virtual reality, and scene understanding. Its core goal is to accurately obtain the semantic category and spatial distribution information of various objects in 3D space.

[0003] Early 3D semantic modeling mainly relied on point cloud data, using deep neural network architecture to extract features and segment disordered point clouds. Although it could identify object categories using geometric features, it had limitations in large-scale scenes, such as high computational complexity, lack of continuous surface description and texture information, and high dependence on the quality of input data, making it difficult to meet the requirements of high-quality rendering and robust semantic segmentation.

[0004] With the rise of Neural Radiation Field (NeRF) technology, implicit neural representation has become the mainstream approach. By jointly optimizing geometric, appearance, and semantic information through multilayer perceptrons, it has advantages in handling complex occlusions and fine structures. However, it relies on dense spatial sampling and massive neural network computation, resulting in long training time and low inference frame rate. Furthermore, the "black box" implicit representation makes it difficult to achieve local semantic editing and explicit geometric extraction, which limits its practical application.

[0005] Three-dimensional Gaussian sputtering (3DGS) technology, as an emerging explicit radiation field technology, combines the explicit characteristics of point clouds with the efficiency of rasterization pipelines. Recent research attempts to add semantic feature vectors to Gaussian spheres to achieve real-time semantic rendering, but it still faces key problems such as huge memory consumption, reliance on high-quality sparse point cloud initialization which easily produces geometric artifacts, and lack of efficient sparsification methods leading to resource waste. In particular, it is difficult to balance accuracy and efficiency in large-scene modeling.

[0006] Existing technologies generally adopt a cascaded strategy of "first reconstructing the appearance and then adding semantics", which results in the semantic modeling quality being limited by the accuracy of geometric reconstruction. In large scenes, it is easy to cause semantic segmentation errors due to geometric artifacts. Moreover, the training process is cumbersome and storage and computing resources are wasted, which cannot meet the requirements of efficiency and accuracy for large-scene 3D semantic modeling. Summary of the Invention

[0007] This invention addresses the problems of poor semantic understanding in large scenes and waste of storage space and computing resources in existing methods. It provides a reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes. Through sparse initialization, spatiotemporal mask synchronization and direct semantic field learning, it achieves efficient and accurate 3D semantic modeling in large scenes.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a reconstruction-free sparse 3D Gaussian semantic modeling method for large-scale scenes, comprising the following steps:

[0010] A spatiotemporal mask synchronization module is constructed, which includes a temporal propagation and fusion module and a spatial merging and correction module to improve the multi-view consistency of semantic supervision signals. The temporal propagation and fusion module generates a preliminary consistent semantic mask by utilizing the temporal correspondence between video frames. The spatial merging and correction module performs global correction by utilizing three-dimensional spatial projection.

[0011] A downsampling operation is performed on the point cloud generated by the motion recovery structure SfM, and some points are retained as the initial 3D Gaussian primitive centers to achieve sparse initialization;

[0012] Learnable semantic feature vectors are attached to each 3D Gaussian unit, and the 3D semantic features are rendered into a 2D semantic map using differentiable rasterization technology.

[0013] Using the semantic graph corresponding to the mask after spatiotemporal synchronization correction as the supervision signal, the Gaussian parameters and semantic feature vectors are directly optimized end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic graph and the supervision signal semantic graph, thus completing the construction of the three-dimensional semantic field.

[0014] As a preferred technical solution, the processing procedure of the timing propagation and fusion module is as follows:

[0015] Initialization: Using the first frame as the initial frame, a full-image instance mask set is generated using the segmentation model. ,in A unique instance ID of the object to be detected for each mask;

[0016] Temporal mask propagation: The mask of the current frame is propagated to the next frame through the video object tracker, preserving the identity of the object ID between adjacent frames;

[0017] Coverage detection and re-segmentation: Define the mask coverage for each frame, and the mask coverage of the next frame satisfies... When the time comes, the re-segmentation mechanism is triggered, and the segmentation model is applied again to the region in the (t+1)th frame that is not covered by the propagated mask, generating a new mask and merging it with the propagated mask to form the final mask set.

[0018] As a preferred technical solution, the formula for calculating the mask coverage of each frame is as follows:

[0019]

[0020] in, and Given the width and height of the input image, Represents all binary masks for frame t. Find the union of sets.

[0021] As a preferred technical solution, the processing procedure of the spatial merging and correction module is as follows:

[0022] Generate 3D instance proposals: for each time-aligned object It collects the 2D masks from all frames, back-projects the pixels contained in the masks onto the 3D point cloud, and aggregates them to generate a unified 3D instance proposal. ;

[0023] Space instance merging: Initialize the global proposal set Calculate all remaining proposals in sequence. With global proposal The 3D intersection-union ratio (IoU) between them; if the maximum IoU exceeds a given threshold If they are the same object, then the merge operation is performed; otherwise, It is added to the instance proposal set as a new proposal;

[0024] Spatial mask correction: merging instance proposals Back-projecting back into each frame of the image generates a corrected set of two-dimensional instance masks. ,in This represents the ID of each instance.

[0025] As a preferred technical solution, rendering the three-dimensional semantic features into a two-dimensional semantic graph specifically involves:

[0026]

[0027] in, For the first Gaussian opacity; For projection onto pixels Gaussian sets on, This is a semantic feature vector.

[0028] As a preferred technical solution, the formula for calculating the L1 loss is as follows:

[0029]

[0030] in, This represents the two-dimensional semantic graph generated by rendering, and S represents the semantic graph corresponding to the mask after spatiotemporal synchronization correction.

[0031] As a preferred technical solution, the segmentation model is the SAM model.

[0032] Secondly, the present invention provides a reconstruction-free sparse 3D Gaussian semantic modeling system for large scenes, which is applied to the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes, including a spatiotemporal mask synchronization module, a sparse initialization module, a semantic feature rendering module, and a semantic reconstruction module.

[0033] The spatiotemporal mask synchronization module includes a temporal propagation and fusion module and a spatial merging and correction module, which are used to improve the multi-view consistency of the semantic supervision signal. The temporal propagation and fusion module uses the temporal correspondence between video frames to generate a preliminary consistent semantic mask. The spatial merging and correction module uses three-dimensional spatial projection for global correction.

[0034] The sparse initialization module is used to perform downsampling on the point cloud generated by the motion recovery structure SfM, retaining some points as the initial three-dimensional Gaussian element centers to achieve sparse initialization.

[0035] The semantic feature rendering module is used to attach a learnable semantic feature vector to each three-dimensional Gaussian unit and render the three-dimensional semantic features into a two-dimensional semantic map using differentiable rasterization technology.

[0036] The semantic reconstruction module is used to use the semantic map corresponding to the mask after spatiotemporal synchronization correction as a supervision signal, and to optimize the Gaussian parameters and semantic feature vectors end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic map and the supervision signal semantic map, thereby completing the construction of the three-dimensional semantic field.

[0037] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0038] At least one processor; and,

[0039] A memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned method for large-scale scene-based non-reconstruction sparse 3D Gaussian semantic modeling.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] 1. This invention proposes a reconstruction-free direct learning strategy for 3D semantic fields. This strategy completely breaks the existing paradigm of "first reconstructing the appearance, then adding semantics," and overturns the traditional understanding that "high-quality geometric reconstruction is a prerequisite for semantic learning." Addressing the critical issue that appearance reconstruction in large scenes is easily affected by environmental factors and geometric quality is difficult to guarantee, thus limiting semantic accuracy, this invention pioneers a reconstruction-free pipeline, directly skipping the tedious and expensive photometric geometric reconstruction stage and directly constructing a 3D semantic field using multi-view 2D semantic supervision signals. This innovative design fundamentally avoids the interference of geometric artifacts on semantic modeling, effectively solving problems such as blurred semantic boundaries and classification errors, achieving end-to-end semantic optimization, and significantly improving the accuracy and reliability of semantic modeling in large scenes.

[0044] 2. The significant advantage of this invention lies in its highly efficient spatiotemporal mask synchronization module. Without 3D geometric priors, semantic masks generated by 2D vision-language models (VLMs) often exhibit severe inconsistencies across multiple viewpoints, becoming a key bottleneck affecting the quality of semantic field learning. To address this, this invention proposes a lightweight spatiotemporal mask synchronization strategy, which comprises two core components: temporal propagation and spatial correction. Temporal propagation preserves object identity and promptly supplements the masks of newly appearing objects. Spatial merging and correction then resolve the global consistency issue, enabling automatic alignment of semantic labels across viewpoints and time without relying on 3D geometric guidance. This design significantly improves the consistency of semantic supervision signals, providing strong support for the accurate convergence of the 3D semantic field and enhancing the model's adaptability to complex scenes.

[0045] 3. A key advantage of this invention is the introduction of a scientific sparse initialization strategy. Most existing methods directly use the high-density point cloud obtained from appearance reconstruction for initialization. However, appearance fitting requires extremely high-density point clouds to capture high-frequency details, which is incompatible with the low-frequency characteristics of semantic information, leading to significant computational redundancy and excessive memory consumption. Based on the low-frequency characteristics of semantic information, this invention proposes an initialization mechanism based on sparse point clouds, effectively representing the semantic structure of complex scenes using only a very small number of sparse Gaussian primitives. This strategy not only significantly reduces the model's storage requirements but also significantly shortens training time. While ensuring that the quality of semantic modeling is not compromised, it achieves an order-of-magnitude improvement in efficiency, making it possible to deploy the model on consumer-grade hardware and rapidly process large-scale scenes. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a comparison diagram of the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes in this invention and the methods in the prior art;

[0048] Figure 2 This is a flowchart of the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes according to an embodiment of the present invention;

[0049] Figure 3 This is a block diagram of a reconstruction-free sparse 3D Gaussian semantic modeling system for large scenes, according to an embodiment of the present invention.

[0050] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0053] Explanation of relevant terms:

[0054] 3DGS: 3D Gaussian Splatting

[0055] NeRF: Nerual Radiance Field

[0056] SfM: Structure from Motion

[0057] IoU: Intersect over Union

[0058] pA: Pixel accuracy

[0059] Figure 1 The differences in the process between the method of this invention and the prior art are intuitively compared. For example... Figure 1 As shown in the upper half, existing technologies generally employ a cascaded process of "reconstruction first, semantics later," that is, first reconstructing a dense 3D Gaussian point cloud using multi-view images as a geometric base, and then adding semantic features. However, in large-scale scenes, this geometric base, which relies on appearance reconstruction, is often of poor quality, containing a large number of artifacts and noise. As a result, the generated dense 3D semantic field inherits these geometric defects, leading to blurry and inaccurate final segmentation results. Conversely, Figure 1 The lower half of the document illustrates the "reconstruction-free" process proposed in this invention. This invention directly skips the appearance reconstruction step, which is susceptible to illumination and computationally expensive. It no longer relies on dense geometric priors and utilizes multi-view semantic graphs in conjunction with a consistency constraint module to directly construct a sparse three-dimensional semantic representation. This approach not only significantly reduces the number of Gaussian points and achieves high efficiency in storage and computation, but also effectively avoids interference from erroneous geometric structures, resulting in clearer and more accurate semantic segmentation boundaries.

[0060] like Figure 2 As shown, this embodiment provides a reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes, which includes two modules: spatiotemporal mask synchronization and sparse initialization, to achieve direct learning of the semantic field without reconstruction. The specific method is as follows:

[0061] S1. Construct a spatiotemporal mask synchronization module. The purpose of this module is to improve the multi-view consistency of semantic supervision signals in order to improve the performance of semantic field learning. The spatiotemporal mask synchronization module includes two parts: a temporal propagation and fusion module and a spatial merging and correction module.

[0062] Furthermore, the purpose of the temporal propagation and fusion module is to generate a preliminary consistent semantic mask by utilizing the temporal correspondence between video frames. The specific processing procedure is as follows:

[0063] (a) Initialization: Take the first frame (t=1) as the initial frame and generate a full-image instance mask set using a segmentation model (such as SAM). ;in The unique instance ID of the object to be detected for each mask.

[0064] (b) Temporal mask propagation: Using a video object tracker, the mask of the current frame is propagated to the next frame. This step explicitly preserves the consistency of object IDs across adjacent frames.

[0065] (c) Coverage detection and re-segmentation: To address the issue of newly appearing objects not being captured, a mask coverage rate is defined for each frame. ,in and Let be the width and height of the input image. When the mask coverage in the next frame significantly decreases, i.e., satisfying... When the re-segmentation mechanism is triggered, the first segment is re-segmented. The regions in the frame not covered by the propagated mask are segmented again using the segmentation model to generate new masks, which are then merged with the propagated masks to form the final mask set.

[0066] Furthermore, the spatial merging and correction module addresses the global consistency problem that temporal propagation cannot solve (such as an object temporarily disappearing and then reappearing and being identified as a new ID), by using three-dimensional spatial projection for global correction; the specific processing procedure is as follows:

[0067] (a) Generate a 3D instance proposal: for each time-aligned object It collects the 2D masks from all frames, back-projects the pixels they contain onto the 3D point cloud, and aggregates them to generate a unified 3D instance proposal. .

[0068] (b) Spatial instance merging: Initialize the global proposal set Calculate all remaining proposals in sequence. With global proposal The three-dimensional intersection-over-union ratio (IoU) between them. If the maximum IoU exceeds a given threshold. If they are the same object, then they are considered to be the same object, and a merge operation is performed. Otherwise, they will be... It is added to the instance proposal set as a new proposal.

[0069] (c) Spatial mask correction: The merged instance proposal Back-projecting back into each frame of the image generates a corrected set of two-dimensional instance masks. .in This represents the ID of each instance.

[0070] S2. Perform downsampling on the point cloud generated by the motion recovery structure SfM, retaining some points as the initial 3D Gaussian element centers to achieve sparse initialization.

[0071] Furthermore, addressing the problem that existing technologies using Structure of Motion (SfM) directly generate overly dense point clouds, leading to excessive computational and storage overhead for semantic modeling, this invention first performs sparsification: downsampling is performed on the SfM point cloud, retaining a small subset of points as the centers of the initialized 3D Gaussian Primitives. Experiments show that compared to the full SfM point cloud, using a sparser point cloud (e.g., downsampled to 10%) for initialization not only significantly reduces storage costs but also accelerates subsequent convergence without affecting the final quality of the semantic field.

[0072] S3. Attach a learnable semantic feature vector to each 3D Gaussian unit, and use differentiable rasterization technology to render the 3D semantic features into a 2D semantic graph.

[0073] Understandably, unlike existing technologies that employ a two-stage process of first reconstructing the appearance geometry and then attaching semantics, this invention directly optimizes the 3D semantic field. First, a learnable semantic feature vector is attached to each Gaussian unit. Then, using differentiable rasterization technology, the three-dimensional semantic features are rendered into a two-dimensional semantic map. :

[0074]

[0075] in For the first Gaussian opacity; For projection onto pixels A Gaussian set on.

[0076] S4. Use the mask that has undergone spatiotemporal synchronization correction in step S1. The obtained semantic graph As a supervisory signal, by minimizing the rendering of them The loss function directly optimizes the Gaussian parameters and semantic vectors end-to-end.

[0077]

[0078] In this process, the semantic field is learned directly without the need to pre-train the appearance geometry model, thus avoiding semantic learning bias caused by poor appearance reconstruction quality of large scenes (such as floating points and artifacts).

[0079] Through the above steps, the present invention can achieve fast, low-storage-occupancy, and highly semantically consistent 3D scene understanding in large-scale scenarios without relying on expensive geometric reconstruction calculations.

[0080] To comprehensively verify the practical performance of the reconstruction-free direct semantic modeling method proposed in this invention in large-scale scenes, this embodiment conducted rigorous closed-loop testing on six challenging real-world large-scale outdoor scenes (covering the GauU-Scene, UrbanScene3D, and Mill19 datasets), including LOWER CAMPUS, SMBU, SZIIT, Sci-Art, Residence, and Building. The evaluation system adopted two core metrics from the semantic segmentation field: mean intersection-over-union ratio (mIoU) to quantify the overlap between the predicted and ground truth regions; and mean pixel accuracy (mPA) to evaluate the accuracy of pixel-level classification, reflecting the system's understanding of fine-grained semantics. Furthermore, the number of Gaussian cells, storage usage, and overall process time were introduced as efficiency evaluation criteria.

[0081] Test results show that this invention significantly outperforms existing technologies in all key metrics, successfully breaking the traditional constraint that "high-quality semantics depends on high-quality geometry." Specifically, compared with state-of-the-art mainstream methods such as LangSplatV2, LangSplat, and OpenGaussian, this invention demonstrates superior segmentation accuracy in all large-scene test projects. Specifically, in the average tests across six scenarios, this system achieved 59.6% mIoU and 77.8% mPA, respectively. Compared to the strong benchmark LangSplatV2 method, this invention achieves a relative performance improvement of approximately 10% in mIoU (an absolute improvement of 5.7%), and a 33% improvement over LangSplat, as shown in Table 1. These experimental data fully demonstrate that this solution effectively avoids semantic boundary ambiguity caused by geometric reconstruction artifacts when processing complex outdoor scenes, exhibiting extremely high robustness.

[0082] Table 1: Comparison of Performance Indicators

[0083]

[0084] In addition to the significant improvement in segmentation performance, this method also demonstrates overwhelming advantages in training speed and computational resource consumption: the model storage footprint is only 0.4GB, the number of Gaussian units is only 1.1 million (1.1M), and the overall processing speed is about 7 times faster than existing technologies, as shown in Table 2. In this table, time is in minutes, the number of Gaussian units is in millions, and storage usage is in GB. In contrast, existing solutions typically require more than 5GB of storage space and tens of millions of Gaussian units, and training is extremely time-consuming.

[0085] Table 2: Comparison of Efficiency Indicators

[0086]

[0087] The main advantages of this invention lie in two core architectural innovations: First, the unique "de-reconstructionization" architecture. Compared to existing methods that require significant computational resources for photometric geometric reconstruction followed by the addition of semantics to dense point clouds—a serial and redundant architecture—this invention abandons reliance on unreliable geometric priors and directly initializes the semantic field using sparse point clouds. This design allows the system to focus on semantic learning itself, avoiding ineffective fitting to flat background regions, thus achieving a significant 13-fold reduction in model size while surpassing the accuracy requirement. Second, a specially optimized spatiotemporal mask synchronization module. Addressing the challenge of inconsistent 2D semantic signals without geometric guidance, this invention does not employ expensive, computationally-intensive strategies but instead designs a lightweight temporal propagation and spatial correction module. This module efficiently aligns semantic supervision signals across multiple perspectives, ensuring rapid convergence and global consistency of the 3D semantic field with almost no increase in computational burden.

[0088] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0089] Based on the same idea as the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes in the above embodiments, the present invention also provides a reconstruction-free sparse 3D Gaussian semantic modeling system for large scenes. This system can be used to execute the above-described reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes. For ease of explanation, the structural diagram of the embodiment of the reconstruction-free sparse 3D Gaussian semantic modeling system for large scenes only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0090] Please see Figure 3 In another embodiment of this application, a non-reconstruction sparse 3D Gaussian semantic modeling system 100 for large scenes is provided. The system includes a spatiotemporal mask synchronization module 101, a sparse initialization module 102, a semantic feature rendering module 103, and a semantic reconstruction module 104.

[0091] The spatiotemporal mask synchronization module 101 includes a temporal propagation and fusion module and a spatial merging and correction module, which are used to improve the multi-view consistency of the semantic supervision signal; the temporal propagation and fusion module uses the temporal correspondence between video frames to generate a preliminary consistent semantic mask; the spatial merging and correction module uses three-dimensional spatial projection for global correction.

[0092] The sparse initialization module 102 is used to perform downsampling operation on the point cloud generated by the motion recovery structure SfM, retaining some points as the initial three-dimensional Gaussian element centers to achieve sparse initialization.

[0093] The semantic feature rendering module 103 is used to attach a learnable semantic feature vector to each three-dimensional Gaussian unit and render the three-dimensional semantic features into a two-dimensional semantic map using differentiable rasterization technology.

[0094] The semantic reconstruction module 104 is used to use the semantic map corresponding to the mask after spatiotemporal synchronization correction as a supervision signal, and to optimize the Gaussian parameters and semantic feature vectors end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic map and the supervision signal semantic map, thereby completing the construction of the three-dimensional semantic field.

[0095] It should be noted that the non-reconstruction sparse 3D Gaussian semantic modeling system for large scenes of the present invention corresponds one-to-one with the non-reconstruction sparse 3D Gaussian semantic modeling method for large scenes of the present invention. The technical features and beneficial effects described in the embodiments of the non-reconstruction sparse 3D Gaussian semantic modeling method for large scenes described above are applicable to the embodiments of non-reconstruction sparse 3D Gaussian semantic modeling for large scenes. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0096] Furthermore, in the implementation of the large-scene non-reconstruction sparse 3D Gaussian semantic modeling system described above, the logical division of each program module is merely illustrative. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the large-scene non-reconstruction sparse 3D Gaussian semantic modeling system can be divided into different program modules to complete all or part of the functions described above.

[0097] Please see Figure 4 In one embodiment, an electronic device is provided for implementing a reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a reconstruction-free sparse 3D Gaussian semantic modeling program 203 for large scenes.

[0098] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of a large-scene non-reconstruction sparse 3D Gaussian semantic modeling program 203, but also to temporarily store data that has been output or will be output.

[0099] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0100] Figure 4 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 4 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0101] The first memory 202 in the electronic device 200 stores a large-scene, non-reconstruction sparse 3D Gaussian semantic modeling program 203, which is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0102] A spatiotemporal mask synchronization module is constructed, which includes a temporal propagation and fusion module and a spatial merging and correction module to improve the multi-view consistency of semantic supervision signals. The temporal propagation and fusion module generates a preliminary consistent semantic mask by utilizing the temporal correspondence between video frames. The spatial merging and correction module performs global correction by utilizing three-dimensional spatial projection.

[0103] A downsampling operation is performed on the point cloud generated by the motion recovery structure SfM, and some points are retained as the initial 3D Gaussian primitive centers to achieve sparse initialization;

[0104] Learnable semantic feature vectors are attached to each 3D Gaussian unit, and the 3D semantic features are rendered into a 2D semantic map using differentiable rasterization technology.

[0105] Using the semantic graph corresponding to the mask after spatiotemporal synchronization correction as the supervision signal, the Gaussian parameters and semantic feature vectors are directly optimized end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic graph and the supervision signal semantic graph, thus completing the construction of the three-dimensional semantic field.

[0106] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A large scene oriented sparse 3D Gaussian semantic modeling method without reconstruction, characterized in that, Includes the following steps: A spatiotemporal mask synchronization module is constructed, which includes a temporal propagation and fusion module and a spatial merging and correction module to improve the multi-view consistency of semantic supervision signals. The temporal propagation and fusion module generates a preliminary consistent semantic mask by utilizing the temporal correspondence between video frames. The spatial merging and correction module performs global correction by utilizing three-dimensional spatial projection. A downsampling operation is performed on the point cloud generated by the motion recovery structure SfM, and some points are retained as the initial 3D Gaussian primitive centers to achieve sparse initialization; Learnable semantic feature vectors are attached to each 3D Gaussian unit, and the 3D semantic features are rendered into a 2D semantic map using differentiable rasterization technology. Using the semantic graph corresponding to the mask after spatiotemporal synchronization correction as the supervision signal, the Gaussian parameters and semantic feature vectors are directly optimized end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic graph and the supervision signal semantic graph, thus completing the construction of the three-dimensional semantic field.

2. The method according to claim 1, wherein, The processing procedure of the timing propagation and fusion module is as follows: Initialization: take the first frame as the initial frame, and generate a set of full-image instance masks using the segmentation model wherein is a unique instance ID of the detected object corresponding to each mask Temporal mask propagation: The mask of the current frame is propagated to the next frame through the video object tracker, preserving the identity of the object ID between adjacent frames; Coverage detection and re-segmentation: Define the mask coverage for each frame, and the mask coverage of the next frame satisfies... When the time comes, the re-segmentation mechanism is triggered, and the segmentation model is applied again to the region in the (t+1)th frame that is not covered by the propagated mask, generating a new mask and merging it with the propagated mask to form the final mask set.

3. The method of claim 2, wherein, The formula for calculating the mask coverage of each frame is as follows: in, and Given the width and height of the input image, Represents all binary masks for frame t. Find the union of sets.

4. The reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes according to claim 1, characterized in that, The processing procedure of the spatial merging and correction module is as follows: Generate 3D instance proposals: for each time-aligned object It collects the 2D masks from all frames, back-projects the pixels contained in the masks onto the 3D point cloud, and aggregates them to generate a unified 3D instance proposal. ; Space instance merging: Initialize the global proposal set Calculate all remaining proposals in sequence. With global proposal The 3D intersection-union ratio (IoU) between them; if the maximum IoU exceeds a given threshold If they are the same object, then the merge operation is performed; otherwise, It is added to the instance proposal set as a new proposal; Spatial mask correction: the merged instance proposals Back-projecting back into each frame image, generating a set of corrected two-dimensional instance masks wherein denotes the ID of each instance.

5. The method of claim 1, wherein, The process of rendering three-dimensional semantic features into a two-dimensional semantic graph specifically involves: in, For the first Gaussian opacity; For projection onto pixels Gaussian sets on, This is a semantic feature vector.

6. The method of claim 1, wherein, The formula for calculating the L1 loss is as follows: wherein, denotes the rendered generated two-dimensional semantic map, S denotes the semantic map corresponding to the spatio-temporal synchronization corrected mask.

7. The method of claim 1, wherein, The segmentation model is the SAM model.

8. A large scene oriented reconstruction-free sparse 3D Gaussian semantic modeling system, characterized in that, The reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes, applied to any one of claims 1-7, includes a spatiotemporal mask synchronization module, a sparse initialization module, a semantic feature rendering module, and a semantic reconstruction module. The spatiotemporal mask synchronization module includes a temporal propagation and fusion module and a spatial merging and correction module, which are used to improve the multi-view consistency of the semantic supervision signal. The temporal propagation and fusion module uses the temporal correspondence between video frames to generate a preliminary consistent semantic mask. The spatial merging and correction module uses three-dimensional spatial projection for global correction. The sparse initialization module is used to perform downsampling on the point cloud generated by the motion recovery structure SfM, retaining some points as the initial three-dimensional Gaussian element centers to achieve sparse initialization. The semantic feature rendering module is used to attach a learnable semantic feature vector to each three-dimensional Gaussian unit and render the three-dimensional semantic features into a two-dimensional semantic map using differentiable rasterization technology. The semantic reconstruction module is used to use the semantic map corresponding to the mask after spatiotemporal synchronization correction as a supervision signal, and to optimize the Gaussian parameters and semantic feature vectors end-to-end by minimizing the L1 loss between the rendered two-dimensional semantic map and the supervision signal semantic map, thereby completing the construction of the three-dimensional semantic field.

9. An electronic device, comprising: The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the reconstruction-free sparse 3D Gaussian semantic modeling method for large scenes as described in any one of claims 1-7.

Images