Dynamic digital twinning three-dimensional Gaussian splash rendering system and method

Abstract

The invention discloses a dynamic digital twinning three-dimensional Gaussian splash rendering system and a dynamic digital twinning three-dimensional Gaussian splash rendering method. According to the system, a static scene is reconstructed by utilizing a three-dimensional Gaussian splashing technology through a semantic scene reconstruction module, semantic information is associated to Gaussian primitives in combination with spatial alignment and semantic injection, and a model with a semantic identifier is generated; multi-source dynamic data are fused and converted into a dynamic data field which can be accessed by a GPU through a multi-mode spatio-temporal data processing module; and through a rendering attribute dynamic modulation module, screening a target primitive according to the semantic identifier, and directly modulating internal rendering attributes such as color, transparency or shape of the target primitive based on a data field query value, thereby realizing internal rendering and visualization of dynamic data. According to the method, the problems that in the prior art, rendering reality and real-time performance are difficult to give consideration to and data and scene fusion is superficial are solved, and dynamic digital twinning rendering with high reality and strong immersion is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and digital twin technology, and particularly relates to a three-dimensional Gaussian splash rendering system and method for dynamic digital twins, which is suitable for immersive visualization of dynamic data. Background Technology

[0002] Digital twin technology, serving as a bridge between the physical and digital worlds, enables the dynamic mapping, simulation, and monitoring of physical entities, demonstrating immense application potential in areas such as smart cities, industrial operations and maintenance, and smart campus management. Achieving highly realistic and interactive digital twins is crucial for enhancing situational awareness and decision support capabilities.

[0003] Currently, the mainstream technical solutions in this field suffer from the following shortcomings: First, in terms of scene reconstruction and rendering, while Neural Radiation Field (NeRF)-based methods can generate photorealistic views, their rendering speed is slow, making it difficult to meet the needs of real-time interaction. Traditional polygon mesh modeling methods, although highly efficient, have limited capabilities in automating and cost-effectively constructing large-scale high-fidelity scenes, and their model accuracy and detail are often insufficient. Second, regarding the fusion of dynamic data and 3D scenes, existing technologies generally employ a "simple data overlay" approach, such as overlaying data like pedestrian heatmaps onto the surface of a 3D model as a 2D layer. This approach results in a severe disconnect between the data and the scene base, leading to a harsh visual presentation that not only obscures 3D information but also fails to provide an immersive data perception experience.

[0004] In summary, existing technologies struggle to achieve deep and organic integration of dynamic data and 3D scenes while ensuring high-fidelity real-time rendering. Furthermore, 3D models are often "dumb" geometries lacking semantic information, making them unsuitable for programmatic and precise association and driving with dynamic data. Therefore, there is an urgent need in this field for a new technological solution that can balance rendering efficiency and visual fidelity while embedding dynamic data inherently into the scene rendering process. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing digital twin systems in terms of visualization realism, real-time performance, and the depth of integration between dynamic data and 3D scenes. Embodiments of this invention provide a dynamic digital twin 3D Gaussian splash rendering system, comprising:

[0006] The semantic scene reconstruction module is used to reconstruct static 3D scenes based on 3D Gaussian splashing technology, and associates the preset 3D model data with semantic information to 3D Gaussian primitives through spatial alignment and semantic injection to generate a 3D Gaussian splashing model with semantic identifiers. The multimodal spatiotemporal data processing module is used to receive multi-source heterogeneous dynamic data, transform it into a dynamic data field through data fusion and spatial analysis, and store it in a data format accessible to the GPU. The rendering attribute dynamic modulation module is used to filter target Gaussian primitives based on semantic identifiers and, based on the data values ​​queried from the dynamic data field, directly modulate the intrinsic rendering attributes of the target Gaussian primitives through a mapping function, thereby realizing the endogenization and visualization of dynamic data in the 3D scene.

[0007] The semantic scene reconstruction module includes: The data acquisition and initial reconstruction unit is used to generate sparse point clouds through multi-view image acquisition and motion recovery structure technology, and to initialize and optimize the three-dimensional Gaussian splash model based on the sparse point clouds. Alignment unit, used to spatially align 3D Gaussian splash model with preset 3D model data with semantic information using spatial registration algorithm; The semantic query and injection unit is used to perform spatial attribution queries on each Gaussian element based on a spatially accelerated data structure, and inject the queried semantic identifiers into the corresponding three-dimensional Gaussian element.

[0008] In some embodiments, the semantic identifier is a key-value pair associated with an external metadata database; the metadata database stores metadata corresponding to the semantic identifier, including at least one of room name, functional attributes, or capacity information; the spatial acceleration data structure is an octree or a kd-tree.

[0009] In some embodiments, the alignment unit is configured to perform the following alignment process: Based on multiple pairs of corresponding feature points selected from a 3D Gaussian splash model and preset 3D model data, an initial affine transformation matrix is ​​calculated using the least squares method for coarse alignment. This initial affine transformation matrix refers to the global spatial transformation operator used to achieve coarse alignment in a progressive spatial registration strategy from coarse to fine. In a homogeneous coordinate system, this matrix... It manifests as a Matrix format:

[0010] in, represent The linear transformation matrix contains spatial rotation, scaling, and shear parameters. represent The translation vector; Using the initial affine transformation matrix as the initial value, the first point cloud composed of Gaussian metacenters and the second point cloud composed of surface samples of the preset 3D model data are precisely aligned using the iterative nearest point ICP algorithm until the root mean square error between the point pairs is lower than the preset centimeter-level accuracy threshold. Output the final fine-alignment transformation matrix to the semantic query and injection unit.

[0011] In some embodiments, the multimodal spatiotemporal data processing module includes: The data fusion unit is used to perform temporal and spatial alignment of dynamic data from different data sources and to fuse them using a weighted strategy, wherein the weights are dynamically adjusted deterministically based on the spatial location of each data source. The density field generation unit is used to generate a real-time density field based on the fused data using a Gaussian kernel density estimation algorithm. Data storage unit, used to store real-time density fields as floating-point textures in GPU memory; The density field generation unit is configured to dynamically select different kernel bandwidth parameter values ​​based on the spatial type indicated by semantic information.

[0012] In some embodiments, the rendering attribute dynamic modulation module includes: The filtering unit is used to pre-filter target Gaussian elements based on semantic identifiers during system initialization or semantic condition changes, and store their indexes in the GPU index buffer. The sampling unit is used to project the spatial coordinates of the target Gaussian primitives onto the texture coordinates corresponding to the density field, and to sample the data values ​​through the GPU shader. The modulation unit is used to calculate new parameters for at least one intrinsic rendering attribute of the target Gaussian primitive by passing the sampled data values ​​through a preset mapping function.

[0013] In some embodiments, the intrinsic rendering properties include at least one of color, transparency, and a covariance matrix representing the three-dimensional shape, represented by spherical harmonic coefficients; The modulation unit is configured to modulate the size or shape of the target Gaussian element by modifying the scaling vector representing the covariance matrix; The scaling vector is related to the sampled data values ​​and a global time variable, so that the target Gaussian unit produces a periodic visual pulsation effect. The mapping function is a smooth interpolation function used to achieve a smooth and non-linear transition of rendering attributes when the data value reaches a preset threshold range.

[0014] In some embodiments, the overall logic of the rendering attribute dynamic modulation module is implemented by the GPU compute shader and scheduled to be executed at an update frequency asynchronous with the main rendering thread.

[0015] On the other hand, a dynamic digital twin 3D Gaussian splash rendering method is provided, including the following steps: Step S1: Reconstruct the static 3D scene based on 3D Gaussian splashing technology, and associate the preset 3D model data with semantic information to the 3D Gaussian primitives through spatial alignment and semantic injection to generate a 3D Gaussian splashing model with semantic identifiers. Step S2: Receive multi-source heterogeneous dynamic data, transform it into a dynamic data field through data fusion and spatial analysis, and store it in a data format accessible to the GPU; Step S3: Filter target Gaussian primitives based on semantic identifiers, and based on the data values ​​queried from the dynamic data field, directly modulate the intrinsic rendering attributes of the target Gaussian primitives through a mapping function, thereby realizing the endogenization and visualization of dynamic data in the 3D scene.

[0016] On another front, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a three-dimensional Gaussian splash rendering method for any of the above-described dynamic digital twins.

[0017] On another front, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements a three-dimensional Gaussian splash rendering method for any of the above-described dynamic digital twins.

[0018] The above technical solution has the following technical effects: This application presents a dynamic digital twin 3D Gaussian splash rendering system and method. By integrating the high-fidelity, real-time rendering capabilities of 3D Gaussian splashing with the rich semantic information of engineering models, it constructs a semantic 3D scene database, fundamentally solving the problem of "dumb" geometry in traditional models. Furthermore, by transforming multi-source heterogeneous dynamic data into a GPU-native dynamic data field, and pioneering a semantically driven Gaussian meta-intrinsic rendering attribute direct modulation mechanism, it completely abandons the simple data layer overlay method, achieving a deep and seamless endogenous fusion of dynamic data and 3D scenes. Ultimately, this system not only provides users with an unprecedented immersive situational awareness experience through smooth interaction with photorealistic realism and high frame rates, but also forms an active digital twin platform capable of situational inference, decision support, and continuous self-optimization by integrating predictive models and feedback loops. This significantly improves the refined management and emergency response capabilities in scenarios such as smart campuses. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the dynamic digital twin 3D Gaussian splash rendering system of this application; Figure 2 The flowchart of the dynamic digital twin 3D Gaussian splash rendering method of this application is shown. Figure 3A schematic diagram of the structure for implementing the dynamic digital twin three-dimensional Gaussian splash rendering method of this application. Detailed Implementation

[0020] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0021] In the application of digital twins, especially in large-scale scenarios such as smart campuses, existing 3D visualization technology mainly relies on two types of solutions: one is the voxel rendering method based on neural radiation fields (NeRF). Although this method can generate new views with photorealistic feel, its rendering speed is slow and the training and inference costs are extremely high, making it difficult to meet the needs of real-time interaction and situational awareness at the management end; the other is the traditional polygon mesh modeling. Although this method has high rendering efficiency, the cost of automatically reconstructing large-scale scenes is high, and the model often lacks rich semantic information, making it a "dumb" model.

[0022] More importantly, existing technologies generally suffer from the drawback of "simple data overlay" in presenting dynamic data (such as pedestrian and vehicle traffic). The common practice is to overlay information such as heat maps obtained through sensor analysis onto a 3D model as a two-dimensional, semi-transparent layer. This approach creates a severe disconnect between dynamic data and static scenes, resulting not only in visual inconsistency and information occlusion, but also in the inability to achieve precise and deep correlation between data and scene elements in 3D space. This significantly limits the manager's intuitive understanding and cognitive efficiency of the overall situation.

[0023] In view of this, this application proposes a dynamic digital twin 3D Gaussian splash rendering system and method, constructing a complete technical closed loop from static high-fidelity reconstruction, dynamic intelligent data processing to final data-driven visualization. The system first uses a semantic scene reconstruction module to automatically and efficiently reconstruct photorealistic static 3D scenes from image sequences using 3D Gaussian splash technology. It then breaks through this limitation by precisely aligning and semantically injecting these scenes with semantically rich BIM / CAD engineering models, ensuring that each Gaussian primitive carries the semantic identifier of its spatial component. This transforms a "dumb" visual model into a queryable, programmable semantic scene database.

[0024] Subsequently, through the multimodal spatiotemporal data processing module, real-time dynamic data from multiple heterogeneous sources such as cameras, campus cards, and Wi-Fi on campus are spatiotemporally aligned and fused. Using spatial analysis methods such as Gaussian kernel density estimation, the data is uniformly transformed into a standard, continuous dynamic data field and directly stored in the GPU memory to ensure maximum access efficiency. Furthermore, the system can integrate time series prediction models to generate predictive data fields for future trends.

[0025] Finally, through the rendering attribute dynamic modulation module, target Gaussian primitives are accurately selected based on semantic identifiers. Then, based on data values ​​queried in real-time from the dynamic data field, the inherent rendering attributes of the Gaussian primitives are directly and efficiently modulated through a configurable mapping function. This mechanism allows the scene to dynamically change its appearance according to the data. For example, densely populated ground areas can spontaneously "emit" a color glow representing heat or produce subtle pulsating effects, thereby achieving a deep and seamless integration of dynamic data and the 3D scene visually and logically, providing users with an unprecedented immersive and intuitive situational awareness experience.

[0026] like Figure 1 As shown, one embodiment of this application provides a dynamic digital twin 3D Gaussian splash rendering system, comprising: The semantic scene reconstruction module is used to reconstruct static 3D scenes based on 3D Gaussian splashing technology, and associates the preset 3D model data with semantic information to 3D Gaussian primitives through spatial alignment and semantic injection to generate a 3D Gaussian splashing model with semantic identifiers. The multimodal spatiotemporal data processing module is used to receive multi-source heterogeneous dynamic data, transform it into a dynamic data field through data fusion and spatial analysis, and store it in a data format accessible to the GPU. The rendering attribute dynamic modulation module is used to filter target Gaussian primitives based on semantic identifiers and, based on the data values ​​queried from the dynamic data field, directly modulate the intrinsic rendering attributes of the target Gaussian primitives through a mapping function, thereby realizing the endogenization and visualization of dynamic data in the 3D scene.

[0027] The semantic scene reconstruction module includes: The data acquisition and initial reconstruction unit is used to generate sparse point clouds through multi-view image acquisition and motion recovery structure technology, and to initialize and optimize the three-dimensional Gaussian splash model based on the sparse point clouds. Alignment unit, used to spatially align 3D Gaussian splash model with preset 3D model data with semantic information using spatial registration algorithm; The semantic query and injection unit is used to perform spatial attribution queries on each Gaussian element based on a spatially accelerated data structure, and inject the queried semantic identifiers into the corresponding three-dimensional Gaussian element.

[0028] In the above embodiments, the endogenization and visualization of dynamic data are achieved through the precise collaboration of three core modules. First, the semantic scene reconstruction module uses 3D Gaussian splashing technology to reconstruct high-fidelity scenes from image sequences. Then, it uses a spatial registration algorithm to accurately align the visual model with semantically rich preset 3D model data, such as BIM or CAD engineering model data. Finally, based on a spatially accelerated data structure, it injects semantic identifiers into each Gaussian primitive. The technical purpose is to upgrade traditional "dumb geometry" into a semantic scene database that can be intelligently queried. The multimodal spatiotemporal data processing module reconstructs heterogeneous dynamic data into a continuous scalar field through Gaussian kernel density estimation and stores it as GPU native texture, aiming to solve the interaction bottleneck between multi-source data and the rendering engine. Finally, the rendering attribute dynamic modulation module directly converts data values ​​into spherical harmonic coefficients or shape parameters that control the appearance of Gaussian primitives through semantic filtering, GPU texture sampling, and mapping functions. The core technology of this module is to break through the traditional layer overlay mode and achieve deep coupling between data and geometry.

[0029] The technical effects produced by the above embodiments are significant and synergistic: First, they achieve true data endogenization and visualization, with dynamic data presented naturally as if it were the inherent optical properties of the scene. For example, the ground in a densely populated area spontaneously emits a halo, and this effect of natural integration with scene lighting and shadows completely eliminates the sense of fragmentation of traditional layer overlay, greatly improving the intuitiveness and immersion of situational awareness. Second, while maintaining the photorealistic feel of 3D Gaussian splashes, the system ensures real-time rendering performance under high data throughput by leveraging GPU texture sampling and parallel computing architecture, perfectly balancing visual quality and interactive efficiency. Third, the semantic-based precise filtering mechanism ensures that data visualization is strictly constrained within the effective semantic area, avoiding information misalignment and laying a technical foundation for refined management. Fourth, the modular system architecture defines clear data interfaces, enabling the system to have good scalability and flexibly adapt to new data sources and visualization needs, demonstrating excellent engineering application prospects.

[0030] The aforementioned pre-defined 3D model data with semantic information can be, but is not limited to, Building Information Modeling (BIM), Computer-Aided Design (CAD) models, point cloud models with semantic segmentation labels, triangular mesh models with identifiers, game engine scene diagrams, 3D GIS models, or simulation models specific to a particular field. This system (a dynamic digital twin 3D Gaussian splash rendering system) uses a spatial registration algorithm to align the 3D Gaussian splash model with any of the aforementioned semantic models to establish a precise spatial mapping relationship, providing a foundation for semantic injection.

[0031] As a high-level embodiment of the present invention, the dynamic data field includes not only a real-time data field but also a predicted data field generated based on historical data. The system utilizes time-series deep learning models such as Long Short-Term Memory Network (LST Net) for prediction. The model input is a multivariate time series, including: the overall campus pedestrian density map (flattened into a one-dimensional vector) for the past N time steps (e.g., N=12, representing the past hour), the day of the week and time information encoded by sine and cosine functions, and one-hot encoding features representing weather conditions and school calendar events. The model output is a predicted density map for the next M time steps. This predicted density field uses the same standardization process as the real-time density field and is ultimately stored as a floating-point texture in GPU memory for use by the rendering attribute dynamic modulation module, thereby achieving forward-looking visualization of future trends. The Long Short-Term Memory Network (LST Net) uses a hybrid architecture of CNN+RNN+Skip-RNN+AR to simultaneously capture long-term dependencies and short-term cyclic patterns in time-series data.

[0032] In some embodiments, the optimization of the 3D Gaussian splash model is performed iteratively through a differentiable rendering pipeline implemented using CUDA and PyTorch / TensorFlow frameworks. The loss function is a weighted average of L1 loss and D-SSIM (structural similarity) loss: Loss = (1 - λ) L1_Loss + λ D-SSIM_Loss, where λ is a hyperparameter ranging from 0.05 to 0.5, typically 0.2. L1_Loss refers to L1 loss, a method of measuring error, also known as "Least Absolute Deviations". D-SSIM_Loss refers to structural similarity loss; SSIM (Structural Similarity) is a metric for measuring the perceptual similarity between two images, and it is more in line with human visual perception than L1 or L2 (squared difference) losses. D-SSIM usually refers to 1-SSIM, transforming it into a loss that can be minimized. Through backpropagation, the gradient of this loss function is used to simultaneously optimize the parameters of each Gaussian unit, including: position μ (3D vector), covariance matrix Σ represented by rotation quaternions and 3D scaling vectors, color c represented by spherical harmonic coefficients, and transparency α (a scalar). The optimization process critically includes an adaptive density control step: the system periodically clones (splits) Gaussian primitives with large positional gradients to fill regions lacking detail; simultaneously, it prunes (removes) primitives with excessively high transparency α or excessively large size. This mechanism ensures that the model adaptively balances detail representation with storage efficiency. In the above embodiments, Structure-from-Motion (SfM) is a computer vision technique that automatically reconstructs the 3D sparse structure of a scene (i.e., a 3D model composed of sparse point clouds) by analyzing a set of 2D images taken from different perspectives and at different times with overlapping regions, and simultaneously estimates the camera's motion trajectory (i.e., the camera's position and attitude parameters) when each image was taken. Spherical harmonic functions are a complete set of orthogonal basis functions defined on a sphere. They are a universal language or coding system used to describe "how something looks in any direction on a sphere." Any complex function defined on a sphere (such as the color and intensity of light emitted from a point on an object's surface in all directions) can be decomposed into a superposition of a series of simple, fixed "fundamental waves." These "fundamental waves" are the basis functions of the spherical harmonic functions. One function of spherical harmonic functions is to provide an extremely compact and efficient way for each Gaussian unit to store and express this "view-dependent color." Spherical harmonic functions are a set of mathematical tools that allow each Gaussian unit to "remember" what color it should be at different viewing angles using a small set of numbers (coefficients). This allows the system to present realistic view-dependent lighting effects while maintaining extremely high rendering efficiency. More importantly, it provides a direct technical approach to dynamically change the appearance of a scene by modifying this set of numbers (i.e., data-driven visualization).

[0033] In some embodiments, the semantic identifier is a key-value pair associated with an external metadata database; the metadata database stores metadata corresponding to the semantic identifier, including at least one of room name, functional attributes, or capacity information; the spatial acceleration data structure is an octree or a kd-tree.

[0034] The above embodiments enhance the system's intelligence and efficiency through data separation and index optimization strategies. The core is to design semantic identifiers as key-value pairs linked to external metadata databases. This decouples geometric data from semantic information, making the Gaussian metadata structure lightweight while dynamically linking to external databases containing rich metadata such as room names and functional attributes through unique key-value pairs. Furthermore, octrees or kd-trees are used as spatial acceleration data structures. The principle behind this is leveraging the ability of these data structures to hierarchically partition 3D space, significantly reducing the algorithmic complexity of complex "points within volume" queries. This helps solve the performance bottleneck problem encountered when performing accurate, real-time spatial queries between massive Gaussian metadata and complex BIM components, and constructs an intelligent 3D scene that can be both quickly rendered and deeply queried.

[0035] In this embodiment, the externalized storage and association of semantic information greatly enhances the system's flexibility and scalability. Metadata can be updated and maintained independently of the 3D model without reconstructing or training the entire Gaussian splash model. This makes it easy to dynamically bind different business attributes (such as temporarily changing room functions or updating equipment information) to the same 3D scene. In addition, the introduction of octrees or kd-trees fundamentally ensures the efficiency of the semantic injection process. Even when facing a large-scale campus scene containing millions of Gaussian primitives, the system can complete the semantic annotation of all primitives within an acceptable time. This is not only a guarantee of technical feasibility but also a key foundation for achieving real-time or near-real-time response of the entire system, thereby supporting a truly usable and easy-to-use dynamic digital twin platform.

[0036] In some embodiments, the spatial semantic query and injection process is designed as a highly parallel GPU computing task. After constructing a spatial index (such as an octree) for the BIM / CAD model, the dynamic digital twin's 3D Gaussian splash rendering system traverses each Gaussian splatter element in the Gaussian splash model, obtains its center point coordinates in parallel on the GPU, and uses the spatial index to perform an efficient "point within volume" test to quickly determine the BIM component to which it belongs. The semantic identifier obtained from the query (such as the component's unique GUID) is written directly into the data structure of the Gaussian splatter element in the GPU memory as a new data attribute.

[0037] In some embodiments, the alignment unit is configured to perform the following alignment process: Based on multiple pairs of corresponding feature points selected from the 3D Gaussian splash model and the preset 3D model data, the initial affine transformation matrix is ​​calculated by the least squares method for coarse alignment. Using the initial affine transformation matrix as the initial value, the first point cloud composed of Gaussian metacenters and the second point cloud composed of surface samples of the preset 3D model data are precisely aligned using the iterative nearest point ICP algorithm until the root mean square error between the point pairs is lower than the preset centimeter-level accuracy threshold. Output the final fine-alignment transformation matrix to the semantic query and injection unit.

[0038] In the above embodiments, a progressive spatial registration strategy from coarse to fine is adopted, balancing the efficiency and accuracy of the alignment process through the combined application of different algorithms. The coarse alignment stage is based on manually selected sparse, identically named feature points, and uses the least squares method to calculate a global initial affine transformation matrix. The principle is to quickly establish a preliminary mapping that places the two models in the same coordinate system by minimizing the sum of squared coordinate errors between point pairs. The fine alignment stage is based on dense point clouds and optimizes using the Iterative Closest Point (ICP) algorithm. Starting from the coarse alignment result, it iteratively executes two steps: "finding the closest point pair" and "calculating the optimal rigid transformation." Its mathematical principle is to continuously solve for the transformation matrix that minimizes the root mean square error (RMSE) between the source and target point clouds using methods such as singular value decomposition (SVD) until it converges to a preset centimeter-level accuracy threshold.

[0039] The "initial affine transformation matrix" refers to the global spatial transformation operator used to achieve coarse alignment in a coarse-to-fine progressive spatial registration strategy. In a homogeneous coordinate system, this matrix... It manifests as a Matrix format:

[0040] in, represent The linear transformation matrix contains spatial rotation, scaling, and shear parameters. represent The translation vector. This matrix can initially map and unify the three-dimensional spatial coordinate system of the 3D Gaussian splash model to the coordinate system of a preset 3D model data (such as a BIM / CAD engineering model) with semantic information.

[0041] The affine transformation operation mainly involves the following technical steps to quickly establish a preliminary spatial mapping: Step 1: Extraction of Corresponding Feature Point Pairs. Select multiple pairs of corresponding feature points with clear spatial correspondences from the 3D Gaussian splash model (source model) and the pre-set 3D model data (target model). Let the source feature point set be... The target feature point set is .

[0042] Step 2: Error function construction and least squares solution. The source feature points are processed through a matrix... An error function is constructed based on the coordinate differences between the transformed points and the target feature points. Using the least squares method, this is achieved by minimizing the sum of squared coordinate errors between point pairs (i.e., minimizing...). ), calculate the globally optimal initial affine transformation matrix .

[0043] Step 3: Coarse alignment of the spatial coordinate system. Apply the calculated initial affine transformation matrix... The global affine transformation operation is performed on the first point cloud formed by the center points of Gaussian elements.

[0044] Step 4: Pass the benchmark downstream. This coarse alignment operation quickly places the two heterogeneous models in the same coordinate system, providing good initial values ​​for the subsequent iterative closest point (ICP) algorithm to perform fine alignment, effectively overcoming the shortcomings of the ICP algorithm that are sensitive to the initial position and are prone to getting trapped in local optima.

[0045] This embodiment aims to solve the challenge of automated high-precision alignment between large-scale, heterogeneous 3D models, laying a reliable spatial benchmark for subsequent semantic injection. First, the two-stage cascaded process design effectively overcomes the ICP algorithm's sensitivity to initial position and its tendency to get trapped in local optima, ensuring the robustness and success rate of the registration process. Second, by introducing a quantifiable centimeter-level precision threshold as an iteration termination condition, the alignment result is no longer a vague "good enough," but rather achieves an engineering-application-level precision usable for spatial queries, fundamentally guaranteeing the spatial accuracy of semantic identifier injection. Finally, this process achieves a leap from "rough overlap" to "precise locking" between the visual reconstruction model and the engineering design model, ensuring that each subsequent Gaussian primitive is assigned the correct semantic information. This is the core technological guarantee for the entire system to achieve accurate data attachment and visualization.

[0046] In some embodiments, the multimodal spatiotemporal data processing module includes: The data fusion unit is used to perform temporal and spatial alignment of dynamic data from different data sources and to fuse them using a weighted strategy, wherein the weights are dynamically adjusted deterministically based on the spatial location of each data source. The density field generation unit is used to generate a real-time density field based on the fused data using a Gaussian kernel density estimation algorithm. Data storage unit, used to store real-time density fields as floating-point textures in GPU memory; The density field generation unit is configured to dynamically select different kernel bandwidth parameter values ​​based on the spatial type indicated by semantic information.

[0047] The above embodiments construct an adaptive and hardware-accelerated dynamic data processing pipeline, improving the accuracy and rationality of the data field through weighted data fusion and semantically driven kernel density estimation. The data fusion unit dynamically allocates fusion weights based on the inherent spatial positioning uncertainties of each data source (such as cameras and Wi-Fi). Its principle is to use weighted averaging to suppress interference from low-precision data, thereby obtaining a more reliable estimate of people's locations in a probabilistic sense. The density field generation unit further introduces semantic context. Its principle is to adaptively adjust the smoothing parameter (bandwidth) in the Gaussian kernel density estimation based on scene structure knowledge (such as "open square" or "narrow corridor"), making the generated density field physically more closely match the actual distribution patterns of people in different spaces. Finally, by storing the data field as a GPU floating-point texture, the parallel sampling and interpolation capabilities of the GPU hardware are utilized, providing a near-zero-latency data access channel for real-time rendering.

[0048] In complex environments with diverse and heterogeneous multi-source data of varying precision and scene semantics, this embodiment is beneficial for generating dynamic data fields that are both accurate and physically intuitive, while meeting the demanding performance requirements of real-time rendering. A deterministic weighted fusion strategy effectively improves the overall reliability and robustness of the fused location data, avoiding the misleading influence of a single low-precision data source on the overall results. Dynamically adjusting kernel bandwidth based on semantics makes the generated density field highly intelligent, simulating the diffusion effect of crowds in open areas and accurately reflecting the focal point of congestion in narrow passages, significantly improving the accuracy and intuitiveness of visualization. Furthermore, the use of GPU textures fundamentally breaks down the bottleneck between big data processing and high-performance rendering, ensuring a highly efficient and smooth workflow from data updates to image presentation, which is conducive to achieving immersive, real-time dynamic digital twins.

[0049] In some embodiments, the rendering attribute dynamic modulation module includes: The filtering unit is used to pre-filter target Gaussian elements based on semantic identifiers during system initialization or semantic condition changes, and store their indexes in the GPU index buffer. The sampling unit is used to project the spatial coordinates of the target Gaussian primitives onto the texture coordinates corresponding to the density field, and to sample the data values ​​through the GPU shader. The modulation unit is used to calculate new parameters for at least one intrinsic rendering attribute of the target Gaussian primitive by passing the sampled data values ​​through a preset mapping function.

[0050] The above embodiments optimize dynamic modulation performance in large-scale scenes by using pre-screening indexes and GPU parallel pipelines, constructing an efficient visualization mechanism of hierarchical screening and parallel computing. During system initialization, the screening unit pre-screens target Gaussian primitives based on semantic identifiers and stores their indices in the GPU buffer. This works by using a space-for-time strategy, transforming runtime conditional judgments into a pre-prepared data list, avoiding traversing all primitives during each frame rendering. The sampling unit uses coordinate projection and GPU texture sampling, leveraging the native hardware acceleration capabilities of graphics hardware for texture mapping and interpolation calculations to efficiently convert 3D spatial coordinates into scalar values ​​in the data field. Finally, the modulation unit converts data values ​​into rendering attribute parameters using a preset mapping function. This process involves parallelizing the mapping calculation from data to visual attributes within the GPU computing shader, enabling the attribute updates of millions of primitives to be completed synchronously in a single computation task.

[0051] This embodiment helps address the performance bottleneck faced in achieving real-time, accurate data-driven visualization in scenarios containing massive amounts of primitives. The index buffer-based pre-filtering mechanism significantly reduces the number of primitives requiring real-time processing, saving valuable computing resources for subsequent intensive computations. By placing coordinate transformation and data querying entirely within the GPU shader pipeline, it fully leverages the parallel architecture of graphics hardware, making the data querying process extremely fast and consumes almost no CPU resources. Furthermore, the complete implementation of the entire "filter-sampling-modulation" process on the GPU ensures that dynamic visualization effects can be updated synchronously with the main rendering process at an extremely high frame rate. Ultimately, it achieves instant and smooth visual feedback on dynamic data changes while maintaining photorealistic visual realism, completely eliminating the stuttering or latency issues caused by insufficient performance in traditional solutions.

[0052] In some embodiments, the intrinsic rendering properties include at least one of color, transparency, and a covariance matrix representing the three-dimensional shape, represented by spherical harmonic coefficients; The modulation unit is configured to modulate the size or shape of the target Gaussian element by modifying the scaling vector representing the covariance matrix; The scaling vector is related to the sampled data values ​​and a global time variable, so that the target Gaussian unit produces a periodic visual pulsation effect. The mapping function is a smooth interpolation function used to achieve a smooth and non-linear transition of rendering attributes when the data value reaches a preset threshold range.

[0053] The above embodiments, through precise manipulation and temporal function modulation of the underlying rendering parameters of the 3D Gaussian splash layer, directly affect the spherical harmonic coefficients that control the optical properties of Gaussian primitives and the covariance matrix that determines their geometric shape, achieving dynamic visual effects that surpass traditional color changes. By modifying the scaling vector in the covariance matrix, the size and shape anisotropy of Gaussian primitives can be mathematically altered; and by associating this scaling vector with data values ​​and global time variables, based on the synthesis principle of periodic functions (such as sine functions), the primitives can produce regular "breathing" pulsations. Simultaneously, a smooth interpolation function (such as smoothstep) is used as the mapping function. Its mathematical principle is to create a transition curve with smooth first and second derivatives within a preset data threshold range, thereby avoiding abrupt changes in visual attributes.

[0054] This implementation breaks through the limitations of static or simple color-changing visualizations, creating a more expressive and guiding advanced visual language to improve the efficiency and intuitiveness of situational awareness. By modulating the periodic pulsation effect generated by the shape and size of primitives, a strong visual rhythm and focus cues are formed, instantly attracting the observer's attention to key data areas (such as congestion points), greatly enhancing the efficiency of alarms and prompts. The application of a smooth interpolation function ensures that all changes in visual attributes are continuous and natural, eliminating the mechanical and abrupt feeling that ordinary linear interpolation may bring, making the data-driven appearance evolution as smooth as a natural phenomenon, further enhancing the visual immersion. Ultimately, this deep and flexible control over the underlying rendering attributes allows abstract data to be transformed into rich and intuitive visual effects, significantly reducing the cognitive burden on users in interpreting complex spatiotemporal data.

[0055] In some embodiments, the overall logic of the rendering attribute dynamic modulation module is implemented by the GPU compute shader and scheduled to be executed at an update frequency asynchronous with the main rendering thread.

[0056] The above embodiments utilize the parallel computing architecture and asynchronous task scheduling mechanism of modern graphics processing units (GPUs) to completely decouple the core logic of rendering attribute modulation from traditional CPU serial processing or processes tightly coupled with rendering. Through the GPU compute shader—a programmable pipeline stage specifically designed for general-purpose parallel computing—target Gaussian pixel selection, data field sampling, and attribute modulation calculations are performed. The principle is to map the update tasks of millions of Gaussian pixels into tens of thousands of parallel execution threads, thereby achieving extreme computational throughput. Simultaneously, the asynchronous execution strategy ensures that the compute shader is triggered at a fixed frequency, typically lower than the rendering frame rate (e.g., 10Hz), in an independent command queue, maintaining independence from the frame cycle of the main rendering thread.

[0057] This embodiment, through asynchronous decoupling of computation and rendering, ensures that the main rendering thread for user interaction and camera movement can continuously run at a frame rate of 60 FPS or higher. Even if the GPU is performing time-consuming attribute update calculations in the background, it will not have any perceptible impact on the smoothness of front-end operations, thus providing an extremely smooth user experience. This design allows the system to intelligently allocate computing resources based on the frequency of change of the dynamic data itself, avoiding unnecessary over-computation. For example, for relatively slow-changing crowd data, it is not necessary to perform a full update in every rendering frame, thereby optimizing computational efficiency and reducing overall system power consumption. This helps to solve the rendering performance fluctuations and stuttering problems caused by dynamic data updates, ensuring the smoothness and stability of system interaction.

[0058] On the other hand, such as Figure 2 This application also provides a method for rendering three-dimensional Gaussian splashes of dynamic digital twins, including the following steps: Step S1: Reconstruct the static 3D scene based on 3D Gaussian splashing technology, and associate the preset 3D model data with semantic information to the 3D Gaussian primitives through spatial alignment and semantic injection to generate a 3D Gaussian splashing model with semantic identifiers. Step S2: Receive multi-source heterogeneous dynamic data, transform it into a dynamic data field through data fusion and spatial analysis, and store it in a data format accessible to the GPU; Step S3: Filter target Gaussian primitives based on semantic identifiers, and based on the data values ​​queried from the dynamic data field, directly modulate the intrinsic rendering attributes of the target Gaussian primitives through a mapping function, thereby realizing the endogenization and visualization of dynamic data in the 3D scene.

[0059] The above embodiments, through three logically rigorous steps, deeply integrate the traditionally fragmented static scene modeling, dynamic data processing, and real-time rendering, constructing an end-to-end technology chain that connects data perception, processing, and final visualization. In step S1, leveraging the explicit representation advantages of 3D Gaussian splashing and semantic injection technology, the visual model is upgraded to a semantic database with spatial indexing capabilities. Next, in step S2, the original heterogeneous data is normalized into a rendering-friendly dynamic data field through data fusion and spatial analysis, and the efficient organization and access of data are achieved by utilizing the characteristics of GPU hardware. Finally, in step S3, the traditional overlay layer method is abandoned, and a direct mapping function from the data field to the intrinsic properties of geometric primitives is established to achieve data-driven self-evolution of scene appearance.

[0060] This embodiment achieves a fundamental transformation of dynamic data from "overlay" to "endogenous," enabling abstract information such as crowd density to be presented naturally, like the inherent optical properties of the scene, completely eliminating visual disjointedness and bringing an unprecedented immersive cognitive experience. A semantically based precise filtering mechanism ensures the spatial accuracy of information visualization, avoiding data misalignment and providing a reliable basis for scientific decision-making. Furthermore, the full utilization of GPU parallel computing capabilities throughout the process ensures efficiency from real-time processing of massive amounts of data to high-fidelity scene rendering, making it possible to achieve a smooth, data-driven interactive experience in large-scale scenes. This embodiment effectively solves the problems of model "deafness," superficial data presentation, and system performance bottlenecks in existing digital twin technologies.

[0061] like Figure 3 This application also provides a dynamic digital twin 3D Gaussian splash rendering apparatus, comprising a processor 301, a memory 302, and a bus 303 in a computing device. The memory stores a computer program, which, when executed by the processor, implements the aforementioned dynamic digital twin 3D Gaussian splash rendering method. This apparatus can be a server, cloud computing platform, or edge computing device, supporting use by educational institutions or online learning platforms.

[0062] Furthermore, as an executable solution, the aforementioned dynamic digital twin 3D Gaussian splash rendering device can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit; it may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0063] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0064] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0065] In some embodiments, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described above in the embodiments of the present invention.

[0066] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0067] In some embodiments, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0068] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A dynamic digital-twin three-dimensional Gaussian spatter rendering system, comprising: The application comprises: a semantic scene reconstruction module for reconstructing a static three-dimensional scene based on a three-dimensional Gaussian splatting technique, and associating preset three-dimensional model data with semantic information to three-dimensional Gaussian primitives through spatial alignment and semantic injection to generate a three-dimensional Gaussian splatting model with semantic identifiers; a multi-modal spatio-temporal data processing module for receiving multi-source heterogeneous dynamic data, converting it into a dynamic data field through data fusion and spatial analysis, and storing it in a GPU accessible data format; a rendering attribute dynamic modulation module for filtering target Gaussian primitives according to the semantic identifiers, and directly modulating the intrinsic rendering attributes of the target Gaussian primitives through a mapping function based on the data values queried from the dynamic data field, thereby realizing the endogenesis and visualization of dynamic data in a three-dimensional scene; wherein the semantic scene reconstruction module comprises: a data acquisition and initial reconstruction unit for generating a sparse point cloud through multi-view image acquisition and motion structure from motion techniques, and initializing and optimizing a three-dimensional Gaussian splatting model based on the sparse point cloud; an alignment unit for spatially aligning the three-dimensional Gaussian splatting model with the preset three-dimensional model data with semantic information through a spatial registration algorithm; a semantic query and injection unit for performing a spatial attribution query on each Gaussian primitive based on a spatial acceleration data structure, and injecting the queried semantic identifiers into the corresponding three-dimensional Gaussian primitives.

2. The system of claim 1, wherein, The semantic identifiers are keys associated with an external meta database; the meta database stores metadata corresponding to the semantic identifiers, including at least one of room name, functional attribute, or capacity information; and the spatial acceleration data structure is an octree or a k-d tree.

3. The system of claim 1, wherein, The alignment unit is configured to perform the following alignment process: Based on the pairs of homonymic feature points selected from the three-dimensional Gaussian splash model and the preset three-dimensional model data, an initial affine transformation matrix is calculated by a least square method to perform coarse alignment, the initial affine transformation matrix is a global space transformation operator used to realize coarse alignment in a gradual space registration strategy from coarse to fine, and in a homogeneous coordinate system, the matrix is in a matrix format: ​ wherein represents a linear transformation matrix, comprising spatial rotation, scaling and skew parameters, represents a translation vector; taking the initial affine transformation matrix as the initial value, performing fine alignment on a first point cloud composed of Gaussian primitive center points and a second point cloud composed of surface samples of the preset three-dimensional model data through an iterative closest point (ICP) algorithm until the root mean square error between the point pairs is below a preset centimeter-level precision threshold; outputting a final fine alignment transformation matrix to the semantic query and injection unit.

4. The system of claim 1, wherein, The multi-modal spatio-temporal data processing module comprises: a data fusion unit for time and spatial alignment of dynamic data from different data sources, and fusion using a weighting strategy, wherein the weights are dynamically adjusted according to the spatial positioning certainty of each data source; a density field generation unit for generating a real-time density field based on the fused data through a Gaussian kernel density estimation algorithm; a data storage unit for storing the real-time density field as a floating-point number texture in GPU video memory; wherein the density field generation unit is configured to dynamically select different kernel bandwidth parameter values according to the spatial type indicated by the semantic information.

5. The system of claim 1, wherein, The rendering attribute dynamic modulation module comprises: a filtering unit for pre-filtering target Gaussian primitives based on the semantic identifiers and storing their indices in a GPU index buffer when the system is initialized or the semantic conditions change. a sampling unit configured to project the spatial coordinates of the target Gaussian cell to the texture coordinates corresponding to the density field, and sample the data value through a GPU shader; a modulation unit configured to calculate a new parameter for modulating at least one intrinsic rendering attribute of the target Gaussian cell through a preset mapping function of the sampled data value.

6. The system of claim 5, wherein, The intrinsic rendering attribute includes at least one of color represented by spherical harmonic function coefficients, transparency, and a covariance matrix representing a three-dimensional shape. The modulation unit is configured to modulate the size or shape of the target Gaussian cell by modifying a scaling vector representing the covariance matrix. The change of the scaling vector is related to the sampled data value and a global time variable, so as to make the target Gaussian cell produce a periodic visual pulsation effect. The mapping function is a smooth interpolation function for realizing smooth and nonlinear transition of the rendering attribute when the data value reaches a preset threshold interval.

7. The system of claim 5, wherein, The overall logic of the rendering attribute dynamic modulation module is realized by a GPU computing shader, and is executed at an asynchronous update frequency with the main rendering thread.

8. A method of dynamic digital-twinning three-dimensional Gaussian spatter rendering, comprising: The method comprises the following steps: reconstructing a static three-dimensional scene based on a three-dimensional Gaussian splash technology, and associating preset three-dimensional model data with semantic information to the three-dimensional Gaussian cell through spatial alignment and semantic injection to generate a three-dimensional Gaussian splash model with semantic identifiers; receiving multi-source heterogeneous dynamic data, converting it into a dynamic data field through data fusion and spatial analysis, and storing it in a GPU accessible data format; filtering a target Gaussian cell according to the semantic identifier, and directly modulating the intrinsic rendering attribute of the target Gaussian cell through a mapping function based on the data value queried from the dynamic data field, so as to realize the endogenization and visualization of dynamic data in a three-dimensional scene.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the method of claim 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of claim 8.

Images