A digital twin real scene three-dimensional scene construction method based on 3DGS

Abstract

This invention discloses a method for constructing a digital twin real-world 3D scene based on 3DGS, comprising: acquiring multi-source scene perception data of the scene to be constructed and generating a standardized scene input dataset; constructing an improved MobileSAM model and generating a cross-view semantic consistency index set; performing sparse 3D reconstruction to generate a scene semantic hierarchical point set; performing 3D Gaussian initialization on the scene semantic hierarchical point set to generate a hierarchical 3D Gaussian scene representation result; constructing an object-level digital twin mapping relationship to generate an object-level digital twin scene representation result; and performing real-time viewpoint projection rendering to generate a digital twin real-world 3D scene construction result. This invention, by introducing an improved MobileSAM model and an object-level digital twin mapping method, achieves high-precision 3D reconstruction of multi-view scene objects and state-driven real-time rendering of digital twin real-world 3D scenes.

Description

Technical Field

[0001] This invention relates to the field of digital twin technology, and in particular to a method for constructing a digital twin real-world 3D scene based on 3DGS. Background Technology

[0002] With the development of 3D visual reconstruction and digital twin technologies, methods for constructing realistic 3D models of scenes based on multi-view image data are increasingly being applied in fields such as smart cities, industrial monitoring, intelligent operation and maintenance, and virtual reality. Traditional technologies typically generate scene point cloud models or mesh models through multi-view image acquisition and 3D reconstruction algorithms, and then perform texture mapping and visualization rendering on these models to achieve a 3D representation of the realistic scene. However, traditional 3D modeling methods often rely on dense point clouds or mesh structures, resulting in large model data volumes and low update efficiency, making it difficult to meet the real-time and detailed representation requirements of digital twin scenes.

[0003] In existing technologies, basic methods for constructing digital twin 3D scenes typically employ multi-view images to perform sparse or dense 3D reconstruction, generating a 3D point cloud model. This model is then fused using a unified coordinate system, and texture mapping or volume rendering is used to generate the final 3D scene representation. The system collects state data from on-site sensors and binds this data to object regions within the 3D model to achieve digital twin scene display. However, existing methods usually only perform unified rendering based on the overall point cloud or mesh structure, lacking a layered representation method for the dynamic changes of different objects. Furthermore, in object-level state mapping and real-time viewpoint projection rendering, it is difficult to balance dynamic update efficiency and rendering accuracy, thus limiting the digital twin's expressive effect and real-time interactive capabilities in complex scenes.

[0004] Therefore, how to provide a method for constructing digital twin real-world 3D scenes based on 3DGS is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for constructing a digital twin real-world 3D scene based on 3DGS. This invention utilizes an improved MobileSAM model and a 3D Gaussian differentiable rendering method, detailing the entire process from multi-source scene perception data acquisition, cross-viewpoint semantic consistency merging, semantically constrained sparse 3D reconstruction to hierarchical 3D Gaussian scene representation construction. It also establishes object-level digital twin mapping relationships based on on-site state perception data, achieving real-time fusion representation of object-level dynamic states and 3D scenes. Hierarchical 3D Gaussian modeling is performed on static structural regions, dynamic disturbance regions, and boundary repair regions, and real-time viewpoint projection rendering technology is used to generate the digital twin real-world 3D scene. This invention can reduce model complexity while ensuring 3D reconstruction accuracy, improve dynamic scene update efficiency and real-time rendering capabilities, and possesses advantages such as high scene representation accuracy, strong object-level state mapping capabilities, and high real-time interaction of digital twins.

[0006] A method for constructing a digital twin real-world 3D scene based on 3DGS according to an embodiment of the present invention includes:

[0007] Collect multi-source scene perception data of the scene to be constructed, perform preprocessing on the multi-source scene perception data, and generate a standardized scene input dataset;

[0008] An improved MobileSAM model is built based on a standardized scene input dataset. A dual-mode image-text cue resonance path and a dynamically foldable hybrid temporal convolutional neck path are constructed. Object-level segmentation processing is performed on multi-view image data, and cross-view object regions are merged to generate a cross-view semantic consistency index set.

[0009] Based on the standardized scene input dataset, time-frequency dual-domain coherent geometric resampling is performed to generate a high-confidence 3D sampling point set. This set is then combined with a cross-view semantic consistency index set to form a semantically constrained sampling point set. The temporal variation features of objects are calculated and classified to generate a scene semantic hierarchical point set.

[0010] Perform 3D Gaussian initialization on the semantic layered point set of the scene, construct a static background Gaussian layer, a dynamic object Gaussian layer and a boundary repair Gaussian layer to form a layered 3D Gaussian expression structure, perform 3DGS differentiable rendering optimization, calculate the reconstruction error of each Gaussian layer and update the parameters to generate a layered 3D Gaussian scene expression result.

[0011] Based on the cross-perspective semantic consistency index set and the on-site state perception data, an object-level digital twin mapping relationship is constructed, and the state parameter data is written into the corresponding Gaussian unit set to generate the object-level digital twin scene expression result;

[0012] Real-time viewpoint projection rendering is performed based on the object-level digital twin scene representation results, and the corresponding state parameter data is retrieved according to the object index number to generate the digital twin real-scene 3D scene construction results.

[0013] Optionally, the multi-source scene perception data specifically includes multi-view image data, depth information data, camera pose data, time series data, scene structure perception data, and on-site state perception data.

[0014] Optionally, the preprocessing of multi-source scene perception data specifically includes data time synchronization, spatial coordinate alignment, abnormal data removal, data format unification, data scale normalization, and multi-source data registration processing.

[0015] Optionally, generating a cross-perspective semantically consistent index set includes:

[0016] An improved MobileSAM model is constructed, which includes a lightweight image coding module, a linear attention coding module, a parallel feedforward fusion module, a mask decoding module, and a cross-view semantic merging module. A bimodal image-text cue resonance path is constructed between the lightweight image coding module and the linear attention coding module, and a dynamically foldable hybrid temporal convolutional neck path is constructed between the parallel feedforward fusion module and the mask decoding module.

[0017] The standardized scene input dataset is input into the lightweight image encoding module, which performs multi-layer convolutional feature extraction processing on the multi-view image data. During the feature extraction process of each layer, the image features are subjected to fractional position embedding encoding processing. The spatial position of features at different scales is enhanced by recursive displacement encoding, generating a multi-view position enhancement feature set.

[0018] The set of multi-view location enhancement features is input into the linear attention encoding module. Implicit prompt features are generated based on the local feature regions of each view image in the image-text dual-mode prompt resonance path. Text prompt features are generated by combining the text semantic prompt information. The implicit prompt features and text prompt features are fused by channel resonance and then processed by single-layer linear attention calculation with the multi-view location enhancement features to generate an object-level semantic enhancement feature set.

[0019] The parallel feedforward fusion module performs multi-branch feedforward transformation and feature concatenation on the object-level semantic enhancement feature set to generate a multi-view fusion feature set. The multi-view fusion feature set is then input into a dynamically foldable hybrid temporal convolutional neck path. Temporal correlation modeling is performed on the multi-view fusion feature set to generate a continuous temporal attention feature flow. Based on the continuous temporal attention feature flow, feature enhancement processing and convolutional parameter folding processing are performed to generate a temporal enhancement fusion feature set.

[0020] The temporal enhancement fusion feature set is input into the mask decoding module. Dynamic decoding parameters are generated through the dynamic decoding path driven by the feature response. Based on the dynamic decoding parameters, object mask decoding processing is performed to obtain object segmentation masks for each viewpoint.

[0021] The cross-view semantic merging module extracts object regions based on object segmentation masks from each viewpoint, and maps the object regions from each viewpoint to a unified world coordinate system in combination with camera pose data. It then performs cross-view object merging processing on object regions from different viewpoints to generate a cross-view semantic consistency index set.

[0022] Optionally, the generation of the scene semantic hierarchical point set includes:

[0023] Based on a standardized scene input dataset, time-frequency dual-domain coherent geometric resampling processing is performed on multi-view image data to construct a time-frequency coherent spectrum cube, and high coherent spectrum peaks are detected. Spatial back projection processing is performed on the pixel positions corresponding to the high coherent spectrum peaks to generate a high-confidence three-dimensional sampling point set.

[0024] Perform a corresponding match between the object index number in the cross-view semantic consistency index set and the pixel projection position in the high-confidence 3D sampling point set. Write the matched object index number into the attribute field of the corresponding sampling point. Perform object clustering processing on the high-confidence 3D sampling point set according to the object index number to generate a semantic constraint sampling point set.

[0025] For the point clusters corresponding to the index numbers of each object in the semantic constraint sampling point set, the three-dimensional coordinate sequence is extracted according to the adjacent time segments, and frequency domain perturbation spectrum analysis is performed to calculate the spatial displacement change, density change amplitude and occlusion change degree, and generate the temporal change characteristics of the object.

[0026] Based on the temporal change characteristics of objects, the point clusters corresponding to the index numbers of each object are classified. Point clusters with spatial displacement changes less than the first threshold and density changes less than the second threshold are classified as static structure point sets. Point clusters with spatial displacement changes greater than the third threshold are classified as dynamic disturbance point sets. Point clusters with occlusion changes greater than the fourth threshold and density changes greater than the fifth threshold are classified as boundary repair point sets. For point clusters with occlusion changes greater than the fourth threshold and density changes less than the fifth threshold, a secondary boundary stability judgment is performed. When the boundary contour change rate within a continuous time segment is greater than the sixth threshold, the corresponding point cluster is classified as a boundary repair point set. A boundary repair point set is generated and combined to form a scene semantic hierarchical point set.

[0027] Optionally, generating the hierarchical 3D Gaussian scene representation result includes:

[0028] Read the static structure point set, dynamic disturbance point set and boundary repair point set of the scene semantic layer point set, perform three-dimensional Gaussian parameter initialization processing on the three-dimensional spatial coordinates, color information and neighborhood point distribution information of each sparse point in each point set, generate center position parameter, covariance parameter, color parameter and opacity parameter respectively, and convert each sparse point into the corresponding three-dimensional Gaussian unit.

[0029] The three-dimensional Gaussian units corresponding to the static structure point set are combined to form a static background Gaussian layer, the three-dimensional Gaussian units corresponding to the dynamic disturbance point set are combined to form a dynamic object Gaussian layer, and the three-dimensional Gaussian units corresponding to the boundary repair point set are combined to form a boundary repair Gaussian layer, thus forming a layered three-dimensional Gaussian representation structure.

[0030] Based on the hierarchical 3D Gaussian representation structure and the multi-view image data and camera pose data in the standardized scene input dataset, 3DGS differentiable rendering optimization processing is performed. Each 3D Gaussian unit is projected onto the corresponding view imaging plane, and the projection results are weighted and fused according to the depth order to generate the rendered image under each view.

[0031] The rendered images from each viewpoint are compared with the original input images from the corresponding viewpoints. The Gaussian layer reconstruction error of static background, Gaussian layer reconstruction error of dynamic object and Gaussian layer reconstruction error of boundary repair are calculated respectively. The reconstruction errors of each layer are weighted and fused according to the layer weight to generate layered reconstruction error values.

[0032] Based on the hierarchical reconstruction error value, the center position parameter, covariance parameter, color parameter and opacity parameter in the static background Gaussian layer, dynamic object Gaussian layer and boundary repair Gaussian layer are iteratively updated. When the error convergence condition is met, the updated hierarchical 3D Gaussian representation structure is output, generating the hierarchical 3D Gaussian scene representation result.

[0033] Optionally, the generated object-level digital twin scene representation result includes:

[0034] Based on the object index number in the cross-view semantic consistency index set, the corresponding object region is extracted from the hierarchical 3D Gaussian scene representation result, and the state parameter data corresponding to the object region is extracted from the on-site state perception data. The state parameter data includes state value, state timestamp information and state source identification information.

[0035] The object identification information in the field state perception data is matched with the object index number in the cross-view semantic consistency index set. The matched state parameter data is collected according to the object index number. The corresponding Gaussian unit set in the hierarchical 3D Gaussian scene representation result is located according to the object index number, and the correspondence between the object index number and the Gaussian unit set is generated.

[0036] Based on the correspondence between object index number and Gaussian cell set, the aggregated state parameter data is written into the attribute field of the corresponding Gaussian cell set, parameter mapping processing is performed on the state value, time association processing is performed on the state timestamp information, and source binding processing is performed on the state source identifier information to generate an object-level digital twin mapping record with the corresponding object index number.

[0037] Perform combination and encapsulation processing on the object-level digital twin mapping records corresponding to each object index number, and write all object-level digital twin mapping relationships into the hierarchical 3D Gaussian scene representation result to generate the object-level digital twin scene representation result.

[0038] Optionally, the generated digital twin real-world 3D scene construction result includes:

[0039] Read the object-level digital twin scene representation results, determine the target view parameters based on the current viewpoint position, viewpoint direction and viewpoint imaging range, and extract the corresponding Gaussian unit set and state parameter data from the object-level digital twin scene representation results according to the object index number;

[0040] Based on the target viewpoint parameters, viewpoint projection processing is performed on each Gaussian unit in the object-level digital twin scene representation result. Each Gaussian unit is mapped to the imaging plane corresponding to the target viewpoint. The corresponding two-dimensional projection area and pixel contribution value are calculated according to the spatial position, covariance parameter, color parameter and opacity parameter of each Gaussian unit, and a set of projection results under the target viewpoint is generated.

[0041] The projection result set under the target viewpoint is subjected to depth sorting and pixel fusion processing. The state parameter data is written into the display attribute field of the corresponding object index number. Based on the object index number, the corresponding state parameter data is called to perform object-level display mapping processing on the projection result set to generate a digital twin rendering image under the target viewpoint.

[0042] The digital twin rendered image from the target's perspective is output as a digital twin real-world 3D scene construction result, and the state parameter data, state timestamp information and state source identification information are output synchronously in the object area corresponding to the object index number.

[0043] The beneficial effects of this invention are:

[0044] This invention proposes a method for constructing a digital twin real-world 3D scene based on 3DGS. It utilizes multi-source scene perception data acquisition, multi-view image semantic segmentation, semantically constrained sparse 3D reconstruction, and hierarchical 3D Gaussian representation construction to perform object-level 3D modeling and dynamic state mapping processing on real-world scenes. An improved MobileSAM model is constructed to perform object-level segmentation on multi-view images, and a unified object index number is generated by cross-view semantic consistency merging. Then, a semantically constrained sparse point set is generated based on sparse 3D reconstruction. According to the temporal change characteristics of objects, the scene is divided into a static structure point set, a dynamic perturbation point set, and a boundary repair point set. A hierarchical 3D Gaussian representation structure is constructed, and 3D Gaussian differentiable rendering optimization processing is performed.

[0045] This invention establishes a correspondence between on-site state perception data and object index numbers, and writes state parameters into corresponding 3D Gaussian units to achieve object-level digital twin mapping. Real-time viewpoint projection rendering is then performed based on the viewpoint position to generate a digital twin real-world 3D scene. This invention achieves the fusion of scene object-level semantic consistency modeling and hierarchical 3D Gaussian representation, improving the accuracy and rendering efficiency of dynamic scene reconstruction. Simultaneously, it enhances the object-level state representation capability in the digital twin scene, making scene updates more real-time and 3D representation more refined. It offers the beneficial effects of high reconstruction accuracy, high dynamic update efficiency, and strong realism in digital twin representation. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of a method for constructing a digital twin real-world 3D scene based on 3DGS proposed in this invention;

[0048] Figure 2 This is a functional flowchart of an improved MobileSAM model for a 3D scene construction method based on 3DGS for digital twin reality proposed in this invention.

[0049] Figure 3 This is a functional flowchart of 3DGS for a digital twin real-world 3D scene construction method based on 3DGS proposed in this invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0051] refer to Figure 1 , Figure 2 and Figure 3 A method for constructing a digital twin real-world 3D scene based on 3DGS, comprising:

[0052] Collect multi-source scene perception data of the scene to be constructed, perform preprocessing on the multi-source scene perception data, and generate a standardized scene input dataset;

[0053] An improved MobileSAM model is built based on a standardized scene input dataset. A dual-mode image-text cue resonance path and a dynamically foldable hybrid temporal convolutional neck path are constructed. Object-level segmentation processing is performed on multi-view image data, and cross-view object regions are merged to generate a cross-view semantic consistency index set.

[0054] Based on the standardized scene input dataset, time-frequency dual-domain coherent geometric resampling is performed to generate a high-confidence 3D sampling point set. This set is then combined with a cross-view semantic consistency index set to form a semantically constrained sampling point set. The temporal variation features of objects are calculated and classified to generate a scene semantic hierarchical point set.

[0055] Perform 3D Gaussian initialization on the semantic layered point set of the scene, construct a static background Gaussian layer, a dynamic object Gaussian layer and a boundary repair Gaussian layer to form a layered 3D Gaussian expression structure, perform 3DGS differentiable rendering optimization, calculate the reconstruction error of each Gaussian layer and update the parameters to generate a layered 3D Gaussian scene expression result.

[0056] Based on the cross-perspective semantic consistency index set and the on-site state perception data, an object-level digital twin mapping relationship is constructed, and the state parameter data is written into the corresponding Gaussian unit set to generate the object-level digital twin scene expression result;

[0057] Real-time viewpoint projection rendering is performed based on the object-level digital twin scene representation results, and the corresponding state parameter data is retrieved according to the object index number to generate the digital twin real-scene 3D scene construction results.

[0058] In this embodiment, the multi-source scene perception data specifically includes multi-view image data, depth information data, camera pose data, time series data, scene structure perception data, and on-site state perception data.

[0059] In this embodiment, the preprocessing of multi-source scene perception data specifically includes data time synchronization, spatial coordinate alignment, abnormal data removal, data format unification, data scale normalization, and multi-source data registration processing.

[0060] In this embodiment, generating a cross-perspective semantic consistency index set includes:

[0061] An improved MobileSAM model is constructed, comprising a lightweight image encoding module, a linear attention encoding module, a parallel feedforward fusion module, a mask decoding module, and a cross-view semantic merging module. A bimodal image-text cue resonance path is constructed between the lightweight image encoding module and the linear attention encoding module, and a dynamically foldable hybrid temporal convolutional neck path is constructed between the parallel feedforward fusion module and the mask decoding module.

[0062] The improved MobileSAM model is constructed as follows:

[0063] In the traditional MobileSAM model image encoder, channel compression and position embedding calculation are performed on the convolutional feature extraction layer to form a lightweight image encoding module. A bimodal image-text cue resonance path is introduced into the traditional MobileSAM model cue encoder, and a single-layer linear attention calculation is added during feature fusion to form a linear attention encoding module. A parallel feedforward fusion module is added at the input of the mask decoding module to perform parallel feedforward transformation and feature concatenation processing on the encoded features. A dynamically foldable hybrid temporal convolutional neck path is added between the parallel feedforward fusion module and the mask decoding module. A cross-view semantic merging module is added at the output of the mask decoding module to obtain an improved MobileSAM model.

[0064] The standardized scene input dataset is fed into the lightweight image encoding module, which performs multi-layer convolutional feature extraction on the multi-view image data. During each layer's feature extraction, fractional position embedding encoding is performed on the image features. Spatial position enhancement is achieved for features at different scales using recursive displacement encoding, generating a multi-view position-enhanced feature set, where:

[0065] The execution of multi-layer convolutional feature extraction processing is as follows:

[0066] Multi-view image data is input into a lightweight image encoding module in viewpoint order. In the first convolutional layer, the convolution kernel slides across the image row by row and column by column, and the pixel values ​​within the kernel coverage area are multiplied and accumulated to obtain the first feature map. The first feature map is then input into the second convolutional layer, where convolution calculation is performed and downsampling is carried out with a fixed stride to obtain the second feature map. The second feature map is then input into the third convolutional layer, where convolution calculation and downsampling are performed again to obtain the third feature map. The first, second, and third feature maps are then combined in scale order to obtain a multi-layer convolutional feature set.

[0067] The execution of the positional embedding encoding process is as follows:

[0068] Read the spatial coordinate information of the feature maps of each layer in the multi-layer convolutional feature set. Calculate the row direction position encoding value and the column direction position encoding value according to the row and column positions of the feature points, respectively. Divide the row direction position encoding value by the number of rows in the feature map to obtain the normalized row coordinate value. Divide the column direction position encoding value by the number of columns in the feature map to obtain the normalized column coordinate value. Concatenate the normalized row coordinate value and the normalized column coordinate value according to the channel dimension to generate a position encoding matrix. Then, perform element-wise addition calculation with the corresponding layer feature map to obtain the position encoding feature map of each layer.

[0069] The spatial positioning enhancement is carried out as follows:

[0070] Read the location encoding feature maps of each layer, perform upsampling calculation on the low-resolution feature map according to the ratio, copy or interpolate each pixel value in the low-resolution feature map to expand to the target size, and stitch the feature maps of each layer with the same size according to the channel dimension to obtain a multi-scale fusion feature map. Perform channel weighted fusion calculation on the multi-scale fusion feature map to generate a multi-view location enhancement feature set.

[0071] The multi-view location enhancement feature set is input into the linear attention encoding module. Implicit cue features are generated based on local feature regions of the images from each viewpoint within the image-text dual-mode cue resonance path. Text cue features are then generated by combining these implicit cue features with textual semantic cue information. The implicit cue features and textual cue features are then fused using channel resonance and combined with the multi-view location enhancement features using a single-layer linear attention computation to generate an object-level semantic enhancement feature set, where:

[0072] The image-text dual-mode cue resonance path refers to establishing an image feature channel and a text semantic channel between the output of the lightweight image coding module and the input of the linear attention coding module.

[0073] The generation of implicit cue features is as follows:

[0074] Read the feature maps of each view in the multi-view location enhancement feature set, divide each view feature map into local regions of fixed size according to the row and column directions, perform summation calculation on the feature values ​​in each local region, sum the values ​​of all feature points in the region and divide by the number of feature points in the region to obtain the average feature value of the local region, arrange the average feature values ​​of all local regions under the same view in spatial order to form a local region feature sequence, and splice the local region feature sequences of each view in view number order to obtain the implicit cue feature set;

[0075] Textual semantic prompts refer to sequences of textual descriptions that correspond to object categories and object attributes in multi-view images;

[0076] Text prompt features are generated by combining semantic prompt information, specifically:

[0077] Read the character sequence in the semantic prompt information of the text, convert each character in the character sequence into a numerical code according to the character encoding table, perform vector mapping processing on the numerical encoding sequence, multiply each character code with the corresponding weight matrix to obtain a character vector, perform cumulative calculation on the character vector in character order and divide by the number of characters to obtain the text feature vector, and concatenate the text feature vectors corresponding to different text sequences in order to generate a text prompt feature set;

[0078] Generate an object-level semantically enhanced feature set, specifically:

[0079] The implicit cue feature set and the text cue feature set are concatenated along the channel dimension to generate image-text fusion cue features. The image-text fusion cue features and the multi-view position enhancement feature set are input into the linear attention encoding module. Single-layer linear attention calculation is performed on the input features. First, linear mapping calculation is performed on the input features to generate query features, key-value features and numerical features. The query features and key-value features are multiplied element-wise and accumulated to obtain attention weights. The attention weights are normalized and then weighted and summed with the numerical features to obtain the object-level semantic enhancement feature set.

[0080] The parallel feedforward fusion module performs multi-branch feedforward transformation and feature concatenation on the object-level semantic enhancement feature set to generate a multi-view fused feature set. This multi-view fused feature set is then input into a dynamically foldable hybrid temporal convolutional neck path. Temporal correlation modeling is performed on the multi-view fused feature set to generate a continuous temporal attention feature flow. Based on this continuous temporal attention feature flow, feature enhancement and convolution parameter folding are performed to generate a temporal enhancement fused feature set, where:

[0081] The execution of multi-branch feedforward transform and feature concatenation processing is as follows:

[0082] The object-level semantic enhancement feature set is read and input into the first feedforward branch, the second feedforward branch, and the third feedforward branch, respectively. In the first feedforward branch, a linear mapping calculation is performed on the input features, and the input features are multiplied with the first weight matrix to obtain the first branch features. In the second feedforward branch, a nonlinear feedforward calculation is performed on the input features, and the input features are first multiplied with the second weight matrix, and then a nonlinear function is performed on the product result to obtain the second branch features. In the third feedforward branch, a scaling transformation calculation is performed on the input features, and the third branch features are obtained by multiplying the input features with the third weight matrix and performing channel compression. The first branch features, the second branch features, and the third branch features are concatenated according to the channel dimension to generate a multi-view fusion feature set.

[0083] The dynamically foldable hybrid temporal convolutional neck path refers to establishing a temporal convolutional processing path between the parallel feedforward fusion module and the mask decoding module;

[0084] Generate a temporal enhancement fusion feature set, specifically as follows:

[0085] Read the multi-view fusion feature set of continuous time segments, stack the features of each time segment in chronological order to form a temporal feature matrix, perform convolution calculation on the temporal feature matrix, slide the convolution kernel in the time and spatial dimensions, perform multiplication and summation calculation on the feature values ​​and convolution kernel weights in the coverage area to obtain temporal convolution features, perform channel fusion calculation on the temporal convolution features, perform weighted summation of features of different time segments in the channel dimension, and use the fused features as the temporal enhancement fusion feature set;

[0086] The temporal enhancement fusion feature set is input into the mask decoding module. Dynamic decoding parameters are generated through a feature response-driven dynamic decoding path. Based on the dynamic decoding parameters, object mask decoding processing is performed to obtain object segmentation masks for each viewpoint, where:

[0087] Feature response-driven dynamic decoding path refers to the feature response analysis path established within the mask decoding module;

[0088] The generation of dynamic decoding parameters is as follows:

[0089] Read the temporal enhancement fusion feature set, perform average calculation on the feature values ​​of each channel in the feature map to obtain the channel response intensity, perform multiplication calculation on the channel response intensity and the decoding weight matrix to generate dynamic weight parameters, perform weighted sum calculation on the dynamic weight parameters and the basic decoding parameters to generate a dynamic decoding parameter set;

[0090] The execution of object mask decoding is as follows:

[0091] Deconvolution is performed on the temporal enhancement fusion feature set. The feature map is upsampled layer by layer and convolution is performed to generate a mask prediction map. Threshold segmentation is performed on the mask prediction map. Pixels greater than the threshold are marked as object regions and pixels less than the threshold are marked as background regions to obtain object segmentation masks for each view.

[0092] The cross-view semantic merging module extracts object regions based on object segmentation masks from each viewpoint, and maps these regions to a unified world coordinate system using camera pose data. It then performs cross-view object merging on these regions, generating a cross-view semantically consistent index set, where:

[0093] The extraction of the object region is specifically as follows:

[0094] The pixel values ​​in the object segmentation mask are scanned point by point. Pixels with pixel values ​​greater than the segmentation threshold are marked as object pixels, and pixels with pixel values ​​less than or equal to the segmentation threshold are marked as background pixels. Connectivity calculation is performed on the regions marked as object pixels, and spatially adjacent and continuous object pixels are grouped into the same object region. The boundary coordinates of each object region are calculated. The boundary range of the object region is obtained by statistically analyzing the minimum row coordinates, maximum row coordinates, minimum column coordinates, and maximum column coordinates of all pixels in the object region. The two-dimensional image coordinates of all pixels in the object region are read and the corresponding three-dimensional coordinates are calculated by combining the depth information data. The two-dimensional pixel coordinates and depth values ​​are back-projected to obtain the three-dimensional spatial point set corresponding to each object region. The three-dimensional spatial point set of the object region from each viewpoint is output as the object region extraction result.

[0095] Generate a set of cross-perspective semantically consistent indexes, specifically as follows:

[0096] The system reads the set of 3D spatial points corresponding to the object regions from each viewpoint and performs coordinate transformation calculations on the 3D spatial point sets from each viewpoint in conjunction with camera pose data. It then unifies the 3D coordinates of each viewpoint to the same world coordinate system. It performs spatial overlap calculations on the object regions under different viewpoints. By calculating the Euclidean distance between the 3D point sets of different object regions, when the average distance between the 3D points of two object regions is less than the distance threshold, the corresponding object regions are marked as the same object. It assigns a unified object index number to multiple viewpoint object regions marked as the same object and merges the viewpoint object regions according to the object index number. Finally, it combines the object index number, the corresponding viewpoint number, and the corresponding 3D point set to generate a cross-viewpoint semantically consistent index set.

[0097] In this embodiment, the generation of scene semantic hierarchical point sets includes:

[0098] Based on a standardized scene input dataset, time-frequency dual-domain coherent geometric resampling processing is performed on multi-view image data to construct a time-frequency coherent spectrum cube. High-coherence spectral peaks are detected, and spatial back-projection processing is performed on the pixel locations corresponding to these peaks to generate a high-confidence 3D sampling point set.

[0099] The execution of time-frequency dual-domain coherent geometric resampling processing is as follows:

[0100] Read multi-view image data from the standardized scene input dataset and construct a multi-view image sequence in chronological order. Multiply the pixel grayscale value matrix of each view image row by row and column by sine and cosine functions of different frequencies and accumulate them to obtain a spatial frequency amplitude matrix. Read the amplitude sequence of the same spatial frequency position in a continuous time segment and multiply it by the time frequency function and accumulate it to obtain a time frequency amplitude matrix. Combine the spatial frequency amplitude matrix and the time frequency amplitude matrix according to the frequency dimension to form a time-frequency coherence spectrum cube. Read the amplitude of the same frequency position of adjacent viewpoints, calculate the amplitude difference and divide it by the average amplitude to generate a coherence value. Extract the pixels corresponding to the positions where the coherence value is greater than the coherence threshold to generate a candidate sampling point set.

[0101] The detection of highly coherent spectral peaks is specifically as follows:

[0102] Read the coherence values ​​at each frequency position in the time-frequency coherence spectrum cube, construct neighborhood windows in the spatial frequency dimension and the time frequency dimension respectively, compare the coherence value at the current frequency position with all values ​​in the neighborhood window, mark it as a local spectral peak when the current value is greater than all values ​​in the neighborhood, then read the amplitude of the spectral peak and compare it with the spectral peak threshold, extract the pixel coordinates corresponding to the frequency position with the amplitude greater than the spectral peak threshold, and generate a set of high coherence spectral peak pixels;

[0103] The execution of spatial back-projection processing is as follows:

[0104] The horizontal scaling factor is obtained by subtracting the horizontal coordinate of the image center from the horizontal coordinate of the pixel and dividing by the focal length parameter. The vertical scaling factor is obtained by subtracting the vertical coordinate of the image center from the vertical coordinate of the pixel and dividing by the focal length parameter. The horizontal and vertical scaling factors are multiplied by the depth value to generate the 3D coordinates in the camera coordinate system. The 3D coordinates are transformed to the unified world coordinate system based on the rotation matrix and displacement vector in the camera pose. The Euclidean distance is calculated for the 3D coordinates generated from different viewpoints. The average value of the 3D coordinates with Euclidean distance less than the distance threshold is taken and fused to generate a high-confidence 3D sampling point set.

[0105] The object index numbers in the cross-view semantic consistency index set are matched with the pixel projection positions in the high-confidence 3D sampling point set. The matched object index numbers are written into the attribute field of the corresponding sampling point. Then, object clustering is performed on the high-confidence 3D sampling point set according to the object index numbers to generate a semantically constrained sampling point set, where:

[0106] The execution of object clustering is as follows:

[0107] Read the set of high-confidence 3D sampling points after the object index number, and group the sampling points according to the object index number. Calculate the Euclidean distance between the 3D coordinates of each sampling point in each group. Divide the sampling points whose Euclidean distance is less than the clustering radius threshold into the same object point cluster. Calculate the centroid coordinates of each point cluster and the distance between the centroids. Perform merging processing on point clusters whose centroid distance is less than the merging threshold to generate a set of semantically constrained sampling points.

[0108] For the point clusters corresponding to the index numbers of each object in the semantically constrained sampling point set, three-dimensional coordinate sequences are extracted according to adjacent time segments, and frequency domain perturbation spectrum analysis is performed to calculate the spatial displacement change, density change amplitude, and occlusion change degree, generating temporal change features of the objects, where:

[0109] Perform frequency domain perturbation spectrum analysis, specifically as follows:

[0110] Read the three-dimensional coordinates of each object cluster in the semantic constraint sampling point set in a continuous time segment, and construct a three-dimensional coordinate sequence in chronological order. Multiply the three-dimensional coordinate sequence with sine and cosine functions of different frequencies in the horizontal, vertical and depth directions respectively, and accumulate them to obtain the amplitude of each frequency. Square each frequency amplitude, sum and take the square root to obtain the frequency energy value. Select the frequency with the largest energy value as the dominant frequency, and record the corresponding frequency amplitude to generate frequency domain perturbation spectrum features.

[0111] The calculation of spatial displacement change, density change magnitude, and occlusion change degree is as follows:

[0112] Read the 3D coordinates of each object point cluster in adjacent time segments, calculate the centroid coordinates of each point cluster in each time segment, sum and take the square root of the square difference of the centroid coordinates of adjacent time segments to obtain the spatial displacement change, count the number of sampling points of each point cluster in each time segment, calculate the difference in the number of sampling points of adjacent time segments, divide the difference by the average number of sampling points of the two time segments to obtain the density change amplitude, project the point clusters onto the image plane and calculate the area of ​​the projection region, divide the area difference of adjacent time segments by the maximum area to obtain the occlusion change degree, and generate the temporal change features of the object.

[0113] Based on the temporal change characteristics of objects, the point clusters corresponding to the index numbers of each object are classified. Point clusters with spatial displacement changes less than the first threshold and density changes less than the second threshold are classified as static structure point sets. Point clusters with spatial displacement changes greater than the third threshold are classified as dynamic disturbance point sets. Point clusters with occlusion changes greater than the fourth threshold and density changes greater than the fifth threshold are classified as boundary repair point sets. For point clusters with occlusion changes greater than the fourth threshold and density changes less than the fifth threshold, a secondary boundary stability judgment is performed. When the boundary contour change rate within a continuous time segment is greater than the sixth threshold, the corresponding point cluster is classified as a boundary repair point set. A boundary repair point set is generated and combined to form a scene semantic hierarchical point set.

[0114] In this embodiment, generating the layered 3D Gaussian scene representation result includes:

[0115] Read the static structure point set, dynamic perturbation point set, and boundary repair point set from the scene semantic hierarchical point set. Perform 3D Gaussian parameter initialization processing on the 3D spatial coordinates, color information, and neighborhood point distribution information of each sparse point in each point set, generating center position parameters, covariance parameters, color parameters, and opacity parameters respectively. Then, convert each sparse point into a corresponding 3D Gaussian unit, where:

[0116] The execution of the 3D Gaussian parameter initialization process is as follows:

[0117] Read the three-dimensional spatial coordinates of each sparse point in the static structure point set, dynamic disturbance point set, and boundary repair point set. Use the horizontal coordinate value, vertical coordinate value, and depth coordinate value as the center position parameter. Read the color information of the corresponding sparse point. Divide the red component, green component, and blue component by 255 to obtain the color parameter. Select a fixed number of neighborhood points within the neighborhood of the sparse point. Calculate the coordinate difference between each neighborhood point and the center point in three directions. Square the coordinate difference in the same direction and average it to obtain the main diagonal value of the covariance matrix. Multiply the coordinate differences in different directions and average them to obtain the off-diagonal value of the covariance matrix. Generate the covariance parameter. Count the number of neighborhood points and divide it by the maximum number of neighborhood points of all sparse points to obtain the opacity parameter. Combine the center position parameter, covariance parameter, color parameter, and opacity parameter to form a three-dimensional Gaussian parameter set.

[0118] Each sparse point is converted into a corresponding three-dimensional Gaussian element, specifically:

[0119] Read the three-dimensional Gaussian parameter set, take the center position parameter as the three-dimensional Gaussian distribution center, the covariance parameter as the spatial diffusion range, the color parameter as the color attribute, and the opacity parameter as the transparency attribute. Write each parameter into the same data structure. Each sparse point forms a corresponding three-dimensional Gaussian unit, and combine them in the order of the point set to generate a three-dimensional Gaussian unit set.

[0120] The three-dimensional Gaussian units corresponding to the static structure point set are combined to form a static background Gaussian layer, the three-dimensional Gaussian units corresponding to the dynamic disturbance point set are combined to form a dynamic object Gaussian layer, and the three-dimensional Gaussian units corresponding to the boundary repair point set are combined to form a boundary repair Gaussian layer, thus forming a layered three-dimensional Gaussian representation structure.

[0121] Based on the hierarchical 3D Gaussian representation structure and the multi-view image data and camera pose data in the standardized scene input dataset, 3DGS differentiable rendering optimization processing is performed. Each 3D Gaussian unit is projected onto the corresponding viewpoint imaging plane, and the projection results are weighted and fused according to depth order to generate rendered images under each viewpoint, where:

[0122] The execution of 3DGS differentiable rendering optimization processing is as follows:

[0123] Read the center position parameters and covariance parameters of the three-dimensional Gaussian unit, and read the camera pose data. Perform directional transformation on the three-dimensional coordinates according to the camera rotation relationship, and then perform position translation according to the camera translation relationship to obtain the camera coordinates. Project the camera coordinates onto the two-dimensional imaging plane according to the camera intrinsic parameters to obtain the projection center position. Then calculate the two-dimensional diffusion range according to the covariance parameters and projection relationship to form the two-dimensional Gaussian projection area. Read the depth values ​​of each three-dimensional Gaussian unit and sort them in depth order to generate a set of projection results.

[0124] Generate rendered images from various viewpoints, specifically:

[0125] For each pixel in the image, calculate the horizontal and vertical distances between the pixel and the center of the two-dimensional Gaussian projection, and combine them with the two-dimensional diffusion range to obtain the weight value. Multiply the weight value with the color parameter and the opacity parameter to obtain the color contribution value. The color contribution values ​​of each Gaussian unit are successively accumulated in the order of depth to obtain the final color value of the pixel. Perform the same calculation on all pixels to generate rendering images from each viewpoint.

[0126] The rendered images from each viewpoint are compared with the corresponding original input images. Gaussian layer reconstruction errors for static backgrounds, dynamic objects, and boundary repair are calculated separately. These layer reconstruction errors are then weighted and fused according to their hierarchical weights to generate layered reconstruction error values.

[0127] The Gaussian layer reconstruction errors for static backgrounds, dynamic objects, and boundary repair were calculated separately.

[0128] Align the rendered image with the original image pixel by pixel, calculate the difference in red, green and blue components at corresponding pixel positions, square each difference and sum them, and average the sum of squares over all pixels to obtain the reconstruction error of the corresponding layer; perform the same calculation on the static background Gaussian layer, the dynamic object Gaussian layer and the boundary repair Gaussian layer respectively to obtain the reconstruction error of the three layers.

[0129] Based on the hierarchical reconstruction error values, iterative updates are performed on the center position parameters, covariance parameters, color parameters, and opacity parameters in the static background Gaussian layer, dynamic object Gaussian layer, and boundary repair Gaussian layer. When the error convergence condition is met, the updated hierarchical 3D Gaussian representation structure is output, generating a hierarchical 3D Gaussian scene representation result, where:

[0130] The execution of the iterative update process is as follows:

[0131] Read the layered reconstruction error and the 3D Gaussian parameters of each layer, apply a small change to a single parameter and recalculate the reconstruction error, divide the error difference before and after the change by the parameter change to obtain the error change rate, adjust the center position parameter, covariance parameter, color parameter and opacity parameter according to the error change rate, substitute the updated parameters back into the 3D Gaussian rendering process to calculate the new reconstruction error, repeat the same process until the error converges.

[0132] The error convergence condition is that the difference between the hierarchical reconstruction errors of two consecutive iterations is less than the error threshold, and the parameter update magnitude is less than the change threshold.

[0133] The generation of layered 3D Gaussian scene representation results is as follows:

[0134] Read the converged 3D Gaussian parameters, combine the updated 3D Gaussian units to form a static background Gaussian layer, a dynamic object Gaussian layer, and a boundary repair Gaussian layer, and combine them in hierarchical order to generate a layered 3D Gaussian scene representation result.

[0135] In this embodiment, the generation of object-level digital twin scene representation results includes:

[0136] Based on the object index number in the cross-view semantic consistency index set, the corresponding object region is extracted from the hierarchical 3D Gaussian scene representation result, and the state parameter data corresponding to the object region is extracted from the on-site state perception data. The state parameter data includes state value, state timestamp information and state source identification information.

[0137] The object identification information in the on-site state perception data is matched with the object index number in the cross-view semantic consistency index set. The matched state parameter data is then grouped according to the object index number. Based on the object index number, the corresponding Gaussian unit set in the hierarchical 3D Gaussian scene representation result is located, generating a correspondence between the object index number and the Gaussian unit set. Where:

[0138] The execution of the corresponding matching process is as follows:

[0139] The system reads object identification information from the field status perception data and object index numbers from the cross-view semantic consistency index set. It performs string comparison and number mapping processing on the object identification information and object index numbers. When the object identification information and object index numbers are consistent, the corresponding status parameter data is matched with the object index number. When the object identification information and object index number are not completely consistent, the system performs joint comparison processing based on the device name field, spatial location field, and category field in the object identification information. By calculating the spatial location difference and judging the category consistency, the system maps the object identification information that meets the condition that the location difference is less than the spatial threshold and the category is consistent to the corresponding object index number. After performing matching processing on all status parameter data, the system collects the status parameter data according to the object index number to generate the status parameter data set corresponding to the object index number.

[0140] The correspondence between the generated object index number and the Gaussian cell set is as follows:

[0141] Read the object index number attribute of each Gaussian unit in the layered 3D Gaussian scene representation result, and perform grouping processing on the Gaussian units according to the object index number. Combine Gaussian units with the same object index number to form a corresponding Gaussian unit set. Read the object index number in the aggregated state parameter data set, and establish a one-to-one correspondence between the object index number and the corresponding Gaussian unit set. Perform the same processing on all object index numbers to generate a mapping table of correspondence between object index numbers and Gaussian unit sets.

[0142] Based on the correspondence between object index numbers and Gaussian cell sets, the aggregated state parameter data is written into the attribute fields of the corresponding Gaussian cell set. Parameter mapping processing is performed on the state values, time association processing is performed on the state timestamp information, and source binding processing is performed on the state source identifier information. This generates an object-level digital twin mapping record corresponding to the object index number, where:

[0143] Perform parameter mapping processing on the state values, specifically as follows:

[0144] Read the state values ​​from the aggregated state parameter data and the Gaussian cell set corresponding to the object index number. Normalize the state values ​​according to their range. Subtract the minimum historical state value from the state value and divide by the difference between the maximum and minimum historical state values ​​to obtain the normalized state value. Calculate the display intensity parameter based on the normalized state value. Multiply the normalized state value by the maximum display intensity coefficient to obtain the display intensity value. Adjust the color and opacity parameters in the corresponding Gaussian cell set according to the display intensity value. Scale the luminance component of the color parameter according to the display intensity value and linearly adjust the opacity parameter according to the display intensity value. After performing the same processing on all Gaussian cells, write the updated parameters into the attribute field of the corresponding Gaussian cell set to generate the state value mapping result for the corresponding object index number.

[0145] Perform combination and encapsulation processing on the object-level digital twin mapping records corresponding to each object index number, and write all object-level digital twin mapping relationships into the hierarchical 3D Gaussian scene representation result to generate the object-level digital twin scene representation result.

[0146] In this embodiment, the generation of the digital twin real-world 3D scene construction result includes:

[0147] Read the object-level digital twin scene representation results, determine the target viewpoint parameters based on the current viewpoint position, viewpoint direction, and viewpoint imaging range, and extract the corresponding Gaussian unit set and state parameter data from the object-level digital twin scene representation results according to the object index number, where:

[0148] The target viewpoint parameters are determined as follows:

[0149] The system reads the three-dimensional coordinates of the current viewpoint, the viewpoint direction vector, and the viewpoint imaging range parameters. It then normalizes the viewpoint direction vector to obtain the standard line-of-sight direction. Based on the viewpoint imaging range parameters, it calculates the horizontal and vertical field-of-sight angles and constructs a viewpoint coordinate system according to the viewpoint position and line-of-sight direction. It then combines the horizontal, vertical, and line-of-sight directions in the viewpoint coordinate system to form the viewpoint rotation relationship. Finally, it combines the viewpoint position coordinates to construct the viewpoint pose parameters. Based on the imaging resolution parameters, it calculates the width and height of the imaging plane and calculates the pixel scale parameters based on the field-of-sight angle and the size of the imaging plane. Finally, it combines the viewpoint pose parameters, field-of-sight angle parameters, and pixel scale parameters to form the target viewpoint parameters.

[0150] Based on the target viewpoint parameters, viewpoint projection processing is performed on each Gaussian unit in the object-level digital twin scene representation result, mapping each Gaussian unit to the imaging plane corresponding to the target viewpoint. Then, according to the spatial position, covariance parameter, color parameter, and opacity parameter of each Gaussian unit, the corresponding two-dimensional projection area and pixel contribution value are calculated, generating a set of projection results under the target viewpoint, where:

[0151] The calculation of the corresponding two-dimensional projection area and pixel contribution value is as follows:

[0152] Read the center position parameters of each Gaussian unit, transform the center position from the world coordinate system to the target view coordinate system to obtain the view coordinates, calculate the two-dimensional projection center position based on the view coordinates and pixel scale parameters, read the covariance parameters, compress and transform the three-dimensional covariance parameters along the line of sight to obtain the two-dimensional diffusion range, and calculate the lateral and longitudinal diffusion radii of the two-dimensional projection area. For each pixel in the imaging plane, calculate the lateral and longitudinal distances between the pixel and the projection center, divide the square of the lateral distance between the pixel and the projection center by the square of the lateral diffusion radius, divide the square of the longitudinal distance by the square of the longitudinal diffusion radius, and add them together. Then take the negative exponential function value of the result to obtain the pixel weight value. Multiply the weight value by the color parameters and opacity parameters to obtain the pixel contribution value of the Gaussian unit to the corresponding pixel. Repeat the same calculation for all Gaussian units to generate a set of projection results under the target view.

[0153] The projection result set from the target viewpoint is subjected to depth sorting and pixel fusion processing. State parameter data is written to the display attribute field of the corresponding object index number. Based on the object index number, the corresponding state parameter data is called to perform object-level display mapping processing on the projection result set, generating a digital twin rendered image from the target viewpoint. Where:

[0154] The execution of depth sorting and pixel fusion processing is as follows:

[0155] Read the depth values ​​of each Gaussian cell in the projection result set, and sort all Gaussian cells in order of depth from far to near. Perform pixel fusion processing on each sorted Gaussian cell. First, read the pixel contribution value of the current Gaussian cell, then read the existing color value at the current pixel position, multiply the pixel contribution value by the opacity parameter to obtain the fusion weight, multiply the fusion weight by the color value of the current Gaussian cell to obtain the new color increment, and then add it to the existing color value of the current pixel to obtain the updated pixel color.

[0156] Generate a digital twin rendered image from the target's perspective, specifically as follows:

[0157] Read the fused pixel result set, write the status parameter data corresponding to the object index number into the display attribute field of the corresponding pixel area, call the status value data according to the object index number to adjust the color intensity and transparency parameters of the corresponding area, update the object display time identifier according to the status timestamp information, and rearrange all pixel results according to the imaging plane order to form a two-dimensional image matrix, which serves as the digital twin rendering image under the target view.

[0158] The digital twin rendered image from the target's perspective is output as a digital twin real-world 3D scene construction result, and the state parameter data, state timestamp information and state source identification information are output synchronously in the object area corresponding to the object index number.

[0159] Example 1: To verify the feasibility of this invention in practice, it was applied to a digital operation and maintenance scenario in a large-scale intelligent manufacturing plant. The invention's 3DGS-based digital twin real-scene 3D scene construction method was deployed in a production workshop of an equipment manufacturing enterprise. The workshop covers approximately 3200 square meters and contains 48 CNC machine tools, industrial robots, conveying equipment, and automated testing equipment. The equipment layout is dense. Traditional 3D modeling methods struggle to guarantee modeling accuracy and real-time update capabilities when equipment occlusion is severe and dynamic equipment moves frequently. Especially when equipment status changes frequently, the digital twin scene suffers from significant update delays and inaccurate object-level state representation, making it difficult for on-site maintenance personnel to promptly grasp the equipment's operating status.

[0160] Multi-view industrial cameras were deployed to collect multi-source scene perception data in the workshop. A total of 16 fixed cameras and 2 mobile inspection cameras were deployed, acquiring approximately 36 frames of multi-view image data per second. Simultaneously, equipment operating status data, including equipment temperature, vibration amplitude, operating current, and operating status indicators, were collected. An improved MobileSAM model was used to perform object-level segmentation on the multi-view images, and a unified object index number was generated through cross-view semantic merging. Subsequently, semantically constrained sparse 3D reconstruction was performed, generating approximately 1.28 million sparse points. These points were then divided into a static structure point set of approximately 920,000 points, a dynamic disturbance point set of approximately 260,000 points, and a boundary repair point set of approximately 100,000 points, based on the temporal variation characteristics of the objects. A hierarchical 3D Gaussian representation structure was constructed, generating approximately 1.28 million 3D Gaussian cells. Combined with real-time equipment status data, equipment temperature, vibration amplitude, and operating status information were written into the corresponding Gaussian cells, achieving object-level digital twin mapping.

[0161] Compared to traditional point cloud digital twin systems, the initial construction time for 3D scene creation in this invention is reduced from 27 minutes to 9 minutes, improving construction efficiency by approximately 66.7%. During dynamic device updates, the average update latency of the traditional system is 3.8 seconds, while the average update latency of this invention is reduced to 0.9 seconds, improving update efficiency by approximately 76.3%. In continuous operation tests, the system processed approximately 3.2TB of multi-view image data, generated approximately 56,000 dynamic update records, and successfully identified 17 abnormal device states, all of which were visualized in real time through digital twin scenes.

[0162] Table 1 Performance Comparison of 3DGS-Based Digital Twin Reality 3D Scene Construction Methods

[0163] Serial Number Performance indicators Traditional 3D point cloud methods 3DGS method of the present invention Increase 1 3D scene initialization time 27min 9min An increase of 66.7% 2 Average error of 3D scene reconstruction 4.6cm 1.7cm An increase of 63.0% 3 Dynamic scene update delay 3.8s 0.9 s An increase of 76.3% 4 Real-time rendering frame rate 18fps 42fps Increased by 133% 5 Completeness of reconstruction of occluded areas 78.4% 94.6% Increased by 20.7% 6 Number of sparse points 1.24 million points 1.28 million points Increased by 3.2% 7 Accuracy of dynamic disturbance region identification 82.6% 96.3% An increase of 16.6% 8 Digital twin state refresh cycle 4.2s 1.1s An increase of 73.8% 9 Number of device anomaly identifications 9 times 17 times An increase of 88.9% 10 Mean time to failure 8.5min 2.3min An increase of 72.9%

[0164] As can be seen from the data in Table 1, the method of this invention exhibits significant advantages in terms of 3D scene construction efficiency, dynamic update capability, and real-time rendering performance. Regarding 3D scene construction efficiency, the traditional 3D point cloud method takes 27 minutes to initialize the scene, while the method of this invention only takes 9 minutes, representing a 66.7% improvement in construction efficiency. This demonstrates that the hierarchical 3D Gaussian representation structure of this invention can effectively reduce the amount of redundant point cloud computing and improve the scene initialization speed.

[0165] Regarding dynamic update capabilities, the traditional method has a dynamic update latency of 3.8 seconds, while the method of this invention reduces it to 0.9 seconds, an improvement of 76.3%. Simultaneously, the digital twin state refresh cycle is shortened from 4.2 seconds to 1.1 seconds. In scenarios with severe occlusion, the traditional method achieves a reconstruction completeness of occluded areas of 78.4%, while the method of this invention reaches 94.6%, an improvement of 20.7%. This demonstrates that the layered modeling approach of boundary-fixing Gaussian layers and dynamically perturbed Gaussian layers effectively solves the problem of incomplete modeling caused by device occlusion and dynamic changes.

[0166] In summary, this invention enables higher precision 3D modeling, faster dynamic update speed, and more stable digital twin expression effects in complex industrial scenarios, fully demonstrating its practical application value and technical advantages in the field of digital twin real-world 3D scene construction.

[0167] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a digital-twin real-scene three-dimensional scene based on 3DGS, characterized in that, include: Collect multi-source scene perception data of the scene to be constructed, perform preprocessing on the multi-source scene perception data, and generate a standardized scene input dataset; An improved MobileSAM model is built based on a standardized scene input dataset. A dual-mode image-text cue resonance path and a dynamically foldable hybrid temporal convolutional neck path are constructed. Object-level segmentation processing is performed on multi-view image data, and cross-view object regions are merged to generate a cross-view semantic consistency index set. Based on the standardized scene input dataset, time-frequency dual-domain coherent geometric resampling is performed to generate a high-confidence 3D sampling point set. This set is then combined with a cross-view semantic consistency index set to form a semantically constrained sampling point set. The temporal variation features of objects are calculated and classified to generate a scene semantic hierarchical point set. Perform 3D Gaussian initialization on the semantic layered point set of the scene, construct a static background Gaussian layer, a dynamic object Gaussian layer and a boundary repair Gaussian layer to form a layered 3D Gaussian expression structure, perform 3DGS differentiable rendering optimization, calculate the reconstruction error of each Gaussian layer and update the parameters to generate a layered 3D Gaussian scene expression result. Based on the cross-perspective semantic consistency index set and the on-site state perception data, an object-level digital twin mapping relationship is constructed, and the state parameter data is written into the corresponding Gaussian unit set to generate the object-level digital twin scene expression result; Real-time viewpoint projection rendering is performed based on the object-level digital twin scene representation results, and the corresponding state parameter data is retrieved according to the object index number to generate the digital twin real-scene 3D scene construction results.

2. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The multi-source scene perception data specifically includes multi-view image data, depth information data, camera pose data, time series data, scene structure perception data, and on-site state perception data.

3. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The preprocessing of multi-source scene perception data specifically includes data time synchronization, spatial coordinate alignment, abnormal data removal, data format unification, data scale normalization, and multi-source data registration.

4. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The generation of the cross-perspective semantically consistent index set includes: An improved MobileSAM model is constructed, which includes a lightweight image coding module, a linear attention coding module, a parallel feedforward fusion module, a mask decoding module, and a cross-view semantic merging module. A bimodal image-text cue resonance path is constructed between the lightweight image coding module and the linear attention coding module, and a dynamically foldable hybrid temporal convolutional neck path is constructed between the parallel feedforward fusion module and the mask decoding module. The standardized scene input dataset is input into the lightweight image encoding module, which performs multi-layer convolutional feature extraction processing on the multi-view image data. During the feature extraction process of each layer, the image features are subjected to fractional position embedding encoding processing. The spatial position of features at different scales is enhanced by recursive displacement encoding, generating a multi-view position enhancement feature set. The set of multi-view location enhancement features is input into the linear attention encoding module. Implicit prompt features are generated based on the local feature regions of each view image in the image-text dual-mode prompt resonance path. Text prompt features are generated by combining the text semantic prompt information. The implicit prompt features and text prompt features are fused by channel resonance and then processed by single-layer linear attention calculation with the multi-view location enhancement features to generate an object-level semantic enhancement feature set. The parallel feedforward fusion module performs multi-branch feedforward transformation and feature concatenation on the object-level semantic enhancement feature set to generate a multi-view fusion feature set. The multi-view fusion feature set is then input into a dynamically foldable hybrid temporal convolutional neck path. Temporal correlation modeling is performed on the multi-view fusion feature set to generate a continuous temporal attention feature flow. Based on the continuous temporal attention feature flow, feature enhancement processing and convolutional parameter folding processing are performed to generate a temporal enhancement fusion feature set. The temporal enhancement fusion feature set is input into the mask decoding module. Dynamic decoding parameters are generated through the dynamic decoding path driven by the feature response. Based on the dynamic decoding parameters, object mask decoding processing is performed to obtain object segmentation masks for each viewpoint. The cross-view semantic merging module extracts object regions based on object segmentation masks from each viewpoint, and maps the object regions from each viewpoint to a unified world coordinate system in combination with camera pose data. It then performs cross-view object merging processing on object regions from different viewpoints to generate a cross-view semantic consistency index set.

5. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The generated scene semantic hierarchical point set includes: Based on a standardized scene input dataset, time-frequency dual-domain coherent geometric resampling processing is performed on multi-view image data to construct a time-frequency coherent spectrum cube, and high coherent spectrum peaks are detected. Spatial back projection processing is performed on the pixel positions corresponding to the high coherent spectrum peaks to generate a high-confidence three-dimensional sampling point set. Perform a corresponding match between the object index number in the cross-view semantic consistency index set and the pixel projection position in the high-confidence 3D sampling point set. Write the matched object index number into the attribute field of the corresponding sampling point. Perform object clustering processing on the high-confidence 3D sampling point set according to the object index number to generate a semantic constraint sampling point set. For the point clusters corresponding to the index numbers of each object in the semantic constraint sampling point set, the three-dimensional coordinate sequence is extracted according to the adjacent time segments, and frequency domain perturbation spectrum analysis is performed to calculate the spatial displacement change, density change amplitude and occlusion change degree, and generate the temporal change characteristics of the object. Based on the temporal change characteristics of objects, the point clusters corresponding to the index numbers of each object are classified. Point clusters with spatial displacement changes less than the first threshold and density changes less than the second threshold are classified as static structure point sets. Point clusters with spatial displacement changes greater than the third threshold are classified as dynamic disturbance point sets. Point clusters with occlusion changes greater than the fourth threshold and density changes greater than the fifth threshold are classified as boundary repair point sets. For point clusters with occlusion changes greater than the fourth threshold and density changes less than the fifth threshold, a secondary boundary stability judgment is performed. When the boundary contour change rate within a continuous time segment is greater than the sixth threshold, the corresponding point cluster is classified as a boundary repair point set. A boundary repair point set is generated and combined to form a scene semantic hierarchical point set.

6. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The generated hierarchical 3D Gaussian scene representation results include: Read the static structure point set, dynamic disturbance point set and boundary repair point set of the scene semantic layer point set, perform three-dimensional Gaussian parameter initialization processing on the three-dimensional spatial coordinates, color information and neighborhood point distribution information of each sparse point in each point set, generate center position parameter, covariance parameter, color parameter and opacity parameter respectively, and convert each sparse point into the corresponding three-dimensional Gaussian unit. The three-dimensional Gaussian units corresponding to the static structure point set are combined to form a static background Gaussian layer, the three-dimensional Gaussian units corresponding to the dynamic disturbance point set are combined to form a dynamic object Gaussian layer, and the three-dimensional Gaussian units corresponding to the boundary repair point set are combined to form a boundary repair Gaussian layer, thus forming a layered three-dimensional Gaussian representation structure. Based on the hierarchical 3D Gaussian representation structure and the multi-view image data and camera pose data in the standardized scene input dataset, 3DGS differentiable rendering optimization processing is performed. Each 3D Gaussian unit is projected onto the corresponding view imaging plane, and the projection results are weighted and fused according to the depth order to generate the rendered image under each view. The rendered images from each viewpoint are compared with the original input images from the corresponding viewpoints. The Gaussian layer reconstruction error of static background, Gaussian layer reconstruction error of dynamic object and Gaussian layer reconstruction error of boundary repair are calculated respectively. The reconstruction errors of each layer are weighted and fused according to the layer weight to generate layered reconstruction error values. Based on the hierarchical reconstruction error value, the center position parameter, covariance parameter, color parameter and opacity parameter in the static background Gaussian layer, dynamic object Gaussian layer and boundary repair Gaussian layer are iteratively updated. When the error convergence condition is met, the updated hierarchical 3D Gaussian representation structure is output, generating the hierarchical 3D Gaussian scene representation result.

7. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The generated object-level digital twin scene representation results include: Based on the object index number in the cross-view semantic consistency index set, the corresponding object region is extracted from the hierarchical 3D Gaussian scene representation result, and the state parameter data corresponding to the object region is extracted from the on-site state perception data. The state parameter data includes state value, state timestamp information and state source identification information. The object identification information in the field state perception data is matched with the object index number in the cross-view semantic consistency index set. The matched state parameter data is collected according to the object index number. The corresponding Gaussian unit set in the hierarchical 3D Gaussian scene representation result is located according to the object index number, and the correspondence between the object index number and the Gaussian unit set is generated. Based on the correspondence between object index number and Gaussian cell set, the aggregated state parameter data is written into the attribute field of the corresponding Gaussian cell set, parameter mapping processing is performed on the state value, time association processing is performed on the state timestamp information, and source binding processing is performed on the state source identifier information to generate an object-level digital twin mapping record with the corresponding object index number. Perform combination and encapsulation processing on the object-level digital twin mapping records corresponding to each object index number, and write all object-level digital twin mapping relationships into the hierarchical 3D Gaussian scene representation result to generate the object-level digital twin scene representation result.

8. The method for constructing a digital twin real-world 3D scene based on 3DGS according to claim 1, characterized in that, The generated digital twin real-world 3D scene construction results include: Read the object-level digital twin scene representation results, determine the target view parameters based on the current viewpoint position, viewpoint direction and viewpoint imaging range, and extract the corresponding Gaussian unit set and state parameter data from the object-level digital twin scene representation results according to the object index number; Based on the target viewpoint parameters, viewpoint projection processing is performed on each Gaussian unit in the object-level digital twin scene representation result. Each Gaussian unit is mapped to the imaging plane corresponding to the target viewpoint. The corresponding two-dimensional projection area and pixel contribution value are calculated according to the spatial position, covariance parameter, color parameter and opacity parameter of each Gaussian unit, and a set of projection results under the target viewpoint is generated. The projection result set under the target viewpoint is subjected to depth sorting and pixel fusion processing. The state parameter data is written into the display attribute field of the corresponding object index number. Based on the object index number, the corresponding state parameter data is called to perform object-level display mapping processing on the projection result set to generate a digital twin rendering image under the target viewpoint. The digital twin rendered image from the target's perspective is output as a digital twin real-world 3D scene construction result, and the state parameter data, state timestamp information and state source identification information are output synchronously in the object area corresponding to the object index number.

Images