A digital drainage twin flow data processing, AI cleaning and three-dimensional visualization rendering optimization method and system

Abstract

The application discloses a kind of digital drainage twin flow data processing, AI cleaning and three-dimensional visualization rendering optimization method and system, for streaming data and rendering different step and mass scene lag problem, constructs "preprocessing-reasoning-rendering trigger" pipeline, realizes data processing and twin scene frame level synchronization, adopts pruning quantization long short-term memory network combined with attention mechanism, accurately eliminates anomaly and jointly topological feature completes missing data, rendering end is through model pruning, instantiation batch drawing and dynamic LOD level switching improves frame rate, combined with terminal adaptive rendering pipeline and hot temperature cold three-level hierarchical storage, realize seamless adaptation and incremental synchronization of multiple ends, the data quality and rendering fluency of the application are significantly improved, reduce the power storage overhead, suitable for urban waterlogging early warning and intelligent management and control of pipe network.

Description

Technical Field

[0001] This invention relates to the fields of digital twins of urban drainage systems, streaming data processing, lightweight deployment of large artificial intelligence models, and 3D visualization rendering technology, and particularly to a method and system for optimizing streaming data processing, AI cleaning, and 3D visualization rendering of digital drainage twins. Background Technology

[0002] Urban drainage networks are a core component of urban municipal infrastructure, directly impacting urban flood control and waterlogging prevention, network operation and maintenance management, and urban water ecological security. With the continuous advancement of smart city construction, intelligent management and control models for urban drainage networks based on digital twin technology are gradually becoming the mainstream of the industry. Urban drainage network digital twin systems can construct virtual 3D scenes that are mapped in real-time to the physical network. Relying on core technologies such as real-time management of flow monitoring data and high-frame-rate visualization of 3D scenes, they achieve real-time perception of network operation status, dynamic simulation of operating conditions, early warning of waterlogging risks, and intelligent scheduling of operation and maintenance. This serves as a core technological carrier supporting the refined and intelligent management and control of urban drainage systems.

[0003] Currently, while existing digital twin technology for urban drainage networks can achieve basic network scene replication and data monitoring functions, there are still many technical bottlenecks in core aspects such as real-time collaborative control, accurate data processing, efficient scene rendering, data storage and retrieval, and lightweight terminal deployment. It cannot meet the actual application needs of high precision, low latency, and multi-terminal compatibility under complex drainage conditions. The specific technical defects are reflected in the following five aspects.

[0004] First, existing technologies cannot achieve frame-level synchronous control of streaming data processing and 3D scene rendering. The current data parsing and governance process and scene rendering process in digital twin systems are independent and have low temporal matching, resulting in the virtual twin scene's operational status updates lagging behind the actual operating status of the physical pipeline network. This leads to a significant disconnect between the scene and actual operating conditions, failing to meet the high-precision real-time control requirements for millisecond-level rapid early warning and dynamic source tracing needed for urban flood control, and severely limiting the system's emergency response capabilities.

[0005] Secondly, the precision and intelligence level of pipeline monitoring data cleaning and processing technology are insufficient. Existing data cleaning methods rely solely on traditional mathematical statistics algorithms for anomaly removal and noise reduction, lacking the semantic recognition and intelligent understanding capabilities of lightweight artificial intelligence models. For nonlinear monitoring data and sudden abnormal operating condition data generated during drainage pipeline operation, they cannot accurately identify abnormal data characteristics and effective information. The accuracy of anomaly data identification and redundant data cleaning is poor, making it difficult to guarantee the reliability of the data foundation for subsequent operating condition analysis and risk assessment.

[0006] Furthermore, the 3D scene rendering strategy fails to incorporate the unique topological characteristics of drainage pipe networks, resulting in poor rendering efficiency and multi-device compatibility. Drainage pipe networks are characterized by intricate pipelines, a large number of nodes, and scattered spatial distribution. However, existing 3D rendering technologies employ generalized loading and layering strategies without optimizing the LOD dynamic loading mechanism for the pipe network topology. This leads to high scene loading redundancy and high resource consumption during the rendering of massive pipe network scene data, directly causing problems such as low scene rendering frame rates and stuttering. Additionally, it cannot adapt to the display and operation needs of multiple terminals with different computing power and resolutions, including PCs, mobile devices, and large-screen terminals, resulting in extremely poor device compatibility.

[0007] Furthermore, the time-series monitoring data of the pipeline network lacks efficient compression storage and fast indexing mechanisms. Urban drainage pipeline networks continuously generate massive amounts of time-series operational data throughout the day. Existing storage technologies mostly adopt the method of directly storing raw data without specifically designing dedicated data compression algorithms and fast indexing architectures. This not only results in a serious waste of data storage resources, but also leads to time-consuming and computationally expensive retrieval of historical operating data, operational process backtracking, and operating condition comparison analysis, significantly reducing the efficiency of system operation and maintenance analysis.

[0008] Finally, existing processing and rendering models do not employ lightweight optimization strategies, limiting their terminal deployment and practicality. Currently, mainstream digital twin data processing and scene rendering models suffer from redundant structures and massive parameter volumes. They lack optimization techniques such as lightweight pruning and quantized perceptual training, resulting in slow inference speeds and high computational resource consumption. This limits their deployment to high-performance professional equipment, making them unsuitable for deployment on low-performance devices like mobile and edge terminals. This significantly restricts the application scope and practicality of digital twin systems for drainage networks.

[0009] In summary, existing general-purpose digital twin solutions are mostly designed for general scenarios and are not specifically optimized for the operating conditions, data characteristics, and maintenance needs of urban drainage pipe networks. Therefore, they cannot effectively address the unique technical challenges of digital twin systems for drainage pipe networks in areas such as real-time synchronization, data cleaning, scene rendering, data storage, and model deployment. Consequently, the industry urgently needs to develop an integrated technical solution that adapts to urban drainage pipe network scenarios, featuring efficient streaming data governance, AI-precise data cleaning, lightweight rendering, and multi-terminal compatibility. This solution aims to overcome existing technical bottlenecks and enhance the intelligence, high precision, and high adaptability of urban drainage pipe network digital twin systems. Summary of the Invention

[0010] To address the numerous technical shortcomings of existing digital twin technologies for urban drainage networks in practical applications, such as the inability to achieve frame-level synchronization between streaming data processing and 3D scene rendering, poor consistency between virtual and real-world condition mapping, the difficulty of adapting traditional statistical cleaning methods to complex nonlinear network data, inefficient 3D scene rendering strategies leading to low frame rates and weak multi-terminal adaptability in massive scenes, lack of efficient compression storage and fast indexing mechanisms for time-series data, high model redundancy, slow inference speed, and difficulty in deployment on low-computing-power terminals, this invention provides an optimized method for streaming data processing, AI cleaning, and 3D visualization rendering in digital drainage twins. This invention integrates a lightweight model architecture and quantitative optimization strategies, specifically optimizing the streaming data governance mechanism, intelligent data cleaning logic, 3D lightweight rendering strategy, massive time-series data storage and retrieval system, and multi-terminal synchronous interaction mechanism in the context of digital twins for drainage networks. This overcomes the technical limitations of traditional general-purpose digital twin technologies in adapting to the specific business scenarios of drainage networks. This invention can effectively improve the data processing accuracy, scene rendering real-time performance and stability of the twin system, reduce system resource consumption, enable multi-terminal adaptation and operation, and significantly improve the intelligence level of urban drainage network flood early warning and operation and maintenance management.

[0011] This invention provides a method for optimizing digital drainage twin flow cytometry data processing, AI cleaning, and 3D visualization rendering, including: Collect monitoring data of drainage pipe network and construct streaming data blocks according to a fixed sampling frequency; Lightweight quantization preprocessing is performed on streaming data blocks to obtain quantized data; The quantitative data is input into the data cleaning model for inference to obtain the cleaned data. Rendering trigger commands are constructed based on the cleaned data and pushed to the rendering queue. The rendering queue synchronously triggers the scene rendering engine to update the state of the twin scene. Monitor data processing latency, and when the latency exceeds a preset threshold, start the frame interpolation compensation algorithm to complete the rendering frame; After rendering is complete, the timestamp is written back to form a closed-loop scheduling.

[0012] In one embodiment of the present invention, the step of inputting quantized data into a data cleaning model for inference includes: Temporal features are extracted by pruning and quantizing the Long Short-Term Memory network. An attention mechanism is used to identify and remove outliers such as jumps, drifts, and zeroing. For missing data, the topological features of pipeline nodes and temporal features are fused together for joint prediction and completion. For sensor range overflow and quantization error, a preset error compensation coefficient is used to reverse correct the quantized data.

[0013] In one embodiment of the present invention, the process of the rendering queue synchronously triggering the scene rendering engine to update the twin scene state includes: Lightweight pruning is performed on the pipeline and equipment models to remove redundant faces and textures. For duplicate pipeline segments and equipment models, instantiation batch rendering and merging are used for drawing calls, and the model detail level is dynamically switched according to the camera's viewing distance.

[0014] In one embodiment of the present invention, the dynamic switching of model detail levels based on camera line-of-sight includes: The urban drainage network is divided into several data blocks according to a geographic grid and a spatial index is constructed. Define the model detail levels, including the full model level, the simplified model level, and the wireframe model level; Based on the distance between the camera and the node and the layer switching threshold, the data blocks of the corresponding layer are dynamically loaded and the layer data that is far from the viewpoint is unloaded.

[0015] In one embodiment of the present invention, it further includes: The terminal adaptive rendering pipeline automatically classifies terminals based on terminal device type, computing power level, screen resolution, and network bandwidth. It controls the deployment of INT8 quantization models on PCs and large screens, the deployment of INT4 ultra-lightweight models on mobile devices, reduces the resolution and disables anti-aliasing on mobile devices, improves the rendering accuracy on large screens, and pushes complex rendering tasks to the cloud for execution, while local devices are only responsible for lightweight rendering.

[0016] In one embodiment of the present invention, it further includes: A three-tiered storage architecture of hot, warm, and cold data was constructed to store nearly seven days' worth of raw quantitative data as hot data to support querying. Store compressed data from seven days to three months as warm data to support retrospective analysis; Archive data older than three months to object storage as cold data to support on-demand loading; Configure a data caching mechanism to prioritize reading cached hot data during visual queries, and read from the storage layer and update the cache if no data is found.

[0017] In one embodiment of the present invention, the time stamp back writing after rendering is completed forms a closed-loop scheduling, which uses the NTP protocol to calibrate the time of multiple terminals to ensure data timing consistency. The scene status of multiple terminals is compared periodically, and when the status is inconsistent, the data of the main terminal is used as the standard for synchronization. Only the changed data part is synchronized to perform incremental data synchronization.

[0018] In one embodiment of the present invention, the streaming data blocks undergo lightweight quantization preprocessing using INT4 or INT8 quantization levels, and the quantized data blocks are grouped according to a preset batch size.

[0019] This invention also provides a digital drainage twin flow cytometry data processing, AI cleaning, and 3D visualization rendering optimization system, comprising: The streaming data processing module is used to collect monitoring data of drainage pipe network, construct streaming data blocks at a fixed sampling frequency, and perform lightweight quantization preprocessing on the streaming data blocks to obtain quantized data; The lightweight AI cleaning module is used to input quantified data into the data cleaning model for inference and obtain cleaned data. The 3D quantization rendering module is used to construct rendering trigger commands based on the cleaned data and push them to the rendering queue. It controls the rendering queue to synchronously trigger the scene rendering engine to update the twin scene state and monitors data processing latency. The frame compensation scheduling module is used to start the frame interpolation compensation algorithm to complete the rendering frame when the data processing delay exceeds the preset threshold, and write back the timestamp after rendering to form a closed-loop scheduling.

[0020] In one embodiment of the present invention, it further includes: The multi-terminal adaptation module is used to automatically classify terminals according to terminal device type, computing power level, screen resolution and network bandwidth, control the deployment of INT8 quantization model on PC and large screen terminals, control the deployment of INT4 ultra-lightweight model on mobile terminals, push complex rendering tasks of mobile terminals to the cloud for execution, and perform lightweight rendering on local devices. The data compression and storage module is used to construct a three-level hierarchical storage architecture of hot, warm, and cold. The raw quantitative data of nearly seven days is stored as hot data, the compressed data of seven days to three months is stored as warm data, and the data of more than three months is archived to object storage as cold data. A data caching mechanism is configured to prioritize the response to visualization query requests. The multi-terminal synchronization module is used to calibrate the time of multiple terminals using the NTP protocol, periodically compare the scene status of multiple terminals and synchronize the data of the master terminal when there is inconsistency, and synchronize only the changed data part to perform incremental data synchronization.

[0021] The present invention has the following beneficial effects: (1) Significantly improved data quality: Through lightweight large model AI cleaning, the abnormal data removal rate is ≥99%, the missing value completion accuracy is ≥98%, and the quantization correction error is ≤0.5%, which solves the problems of high noise and many missing values ​​in drainage data.

[0022] (2) Rendering performance is greatly optimized: The combination of lightweight and quantitative rendering of 3D scenes is implemented, with a frame rate of ≥60fps on PC, ≥30fps on mobile, and ≥45fps on large screen, and memory usage is reduced by more than 65%.

[0023] (3) Streaming and rendering frame-level synchronization: Data processing-rendering synchronization latency ≤10ms, realizing real-time alignment between twin scenes and the physical world.

[0024] (4) Multi-terminal full-scene adaptation: One system automatically adapts to PC, large screen and mobile terminal without the need for separate development, and the adaptation efficiency is improved by 100%.

[0025] (5) Data storage and retrieval efficiency is doubled: the compression ratio of time series data is ≥8:1, the storage cost is reduced by 75%, and the speed of visual data retrieval is increased by more than 4 times.

[0026] (6) Strong consistency across multiple terminals: multi-terminal data synchronization error ≤10ms, interactive operation delay ≤50ms, and support for real-time collaborative operation by multiple users.

[0027] (7) Lightweight quantization model deep adaptation: The model size is reduced by 70%, the computing power consumption is reduced by 60%, and it can run stably on low-configuration edge devices. Attached Figure Description

[0028] Figure 1 A flowchart of a method for optimizing digital drainage twin flow data processing, AI cleaning, and 3D visualization rendering is shown in one embodiment of the present invention. Detailed Implementation

[0029] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific quantities, materials, and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0030] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0031] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0032] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1A flowchart of a method for optimizing digital drainage twin flow data processing, AI cleaning, and 3D visualization rendering is shown in one embodiment of the present invention.

[0035] like Figure 1 As shown, the method begins with S1, the drainage network monitoring data acquisition step, where sensor nodes transmit operating parameters such as liquid level, flow rate, and rainfall in real time. The acquisition module segments the raw data stream at a fixed sampling frequency to generate time-series continuous streaming data blocks. S2, the lightweight quantization preprocessing step, where the streaming data blocks then enter the lightweight quantization preprocessing unit, compressing the data precision using INT4 or INT8 quantization strategies and grouping and packaging them according to a preset batch size. S3, the data cleaning model inference step, where the quantized data is sent to the data cleaning model for forward inference, where anomaly detection, missing data completion, and quantization error inverse correction are performed internally, outputting standardized cleaned data. S4, the rendering trigger instruction construction step, where the cleaned data serves as the trigger source to generate rendering instructions, which are pushed to the rendering queue. The rendering queue synchronously wakes up the scene rendering engine, driving the twin scene's geometric mesh and material parameters to update their states. S5, the data processing latency monitoring step, where the frame compensation scheduling unit continuously monitors the data processing latency. When the latency exceeds a preset threshold, the frame interpolation compensation algorithm immediately intervenes, generating compensated rendering frames through interpolation of adjacent frame features. After the rendering pipeline outputs the final image, the timestamp is written back to the data header information, completing the closed-loop scheduling. This flowchart clearly illustrates the logical progression between data preprocessing, model inference, rendering triggering, latency monitoring, and closed-loop write-back, demonstrating the core architecture of synchronous updates between streaming processing and 3D rendering.

[0036] Given the extremely high real-time requirements of urban drainage network monitoring scenarios, this embodiment provides a method for optimizing digital drainage twin streaming data processing, AI cleaning, and 3D visualization rendering, along with a corresponding system for the same purpose. Since the operating status of drainage networks changes rapidly, this embodiment first collects monitoring data from sensors deployed at network nodes. The collected data covers key operating parameters such as liquid level, flow rate, and rainfall. To ensure consistency in the timing of subsequent processing, this embodiment segments the raw monitoring data at a fixed sampling frequency to construct streaming data blocks. Once the streaming data blocks are generated, lightweight quantization preprocessing is performed to reduce subsequent computational overhead by compressing data precision. Preferably, the lightweight quantization preprocessing uses INT4 or INT8 quantization levels, and the quantized data blocks are grouped according to a preset batch size to improve parallel inference throughput.

[0037] After obtaining the quantized data, the system inputs it into a data cleaning model for inference, thereby obtaining cleaned data. To ensure the timeliness of the twin scene updates, this embodiment constructs rendering trigger instructions based on the cleaned data and pushes them to the rendering queue. The rendering queue synchronously triggers the scene rendering engine to update the twin scene state. When the data processing latency exceeds a preset threshold, the system automatically starts a frame interpolation compensation algorithm to complete the rendering frame; otherwise, if the latency is within the threshold range, regular rendering is executed directly. After rendering is completed, the timestamp is written back, forming a closed-loop scheduling, thereby achieving frame-level synchronization between streaming data processing and scene rendering. Thanks to the above pipeline architecture, the consistency between data and scene is guaranteed, and the mapping latency between the twin scene and the physical world is strictly controlled within milliseconds.

[0038] Adapted to the above method, this embodiment also provides a digital drainage twin streaming data processing, AI cleaning, and 3D visualization rendering optimization system. This system includes a streaming data processing module, a lightweight AI cleaning module, a 3D quantization rendering module, and a frame compensation scheduling module. The streaming data processing module collects monitoring data from the drainage network, constructs streaming data blocks at a fixed sampling frequency, and performs lightweight quantization preprocessing on the streaming data blocks to obtain quantized data. The lightweight AI cleaning module inputs the quantized data into a data cleaning model for inference to obtain cleaned data. The 3D quantization rendering module constructs rendering trigger commands based on the cleaned data and pushes them to the rendering queue, controlling the rendering queue to synchronously trigger the scene rendering engine to update the twin scene state and monitor data processing latency. The frame compensation scheduling module initiates a frame interpolation compensation algorithm to complete the rendering frames when the data processing latency exceeds a preset threshold, and writes back the timestamp after rendering to form a closed-loop scheduling. The modules communicate with each other through an internal bus or network interface, reducing the risk of single-point failures and improving the overall scalability of the system.

[0039] Traditional data cleaning methods are insufficient in accurately handling the common outliers, drifts, and zero-value anomalies found in drainage network data. Therefore, this embodiment further incorporates a deep learning architecture into the data cleaning process. The input of quantized data into the data cleaning model for inference includes: extracting temporal features through pruning and quantized Long Short-Term Memory (LSTM) networks, and using an attention mechanism to identify and remove outliers, drifts, and zero-value anomalies. Due to the strong spatial correlation of the network structure, missing data is supplemented by a joint prediction using network node topology features and temporal features, thereby improving the physical interpretability of the completion logic. Furthermore, a preset error compensation coefficient is used to inversely correct the quantized data for sensor range overflow and quantization errors, eliminating the cumulative bias caused by hardware nonlinear response. Through these combined strategies, the anomaly removal rate and missing value completion accuracy of the data cleaning model are significantly improved.

[0040] In 3D visualization rendering, to address the issues of excessive memory consumption and frame rate fluctuations caused by massive pipeline models, this embodiment refines the steps for synchronously triggering the scene rendering engine to update the twin scene state in the rendering queue. Specifically, lightweight pruning is performed on pipeline and device models to remove redundant faces and textures, thereby significantly reducing the amount of geometric data while preserving visual features. For duplicate pipeline segments and device models, instantiated batch rendering and merged drawing calls are used, which not only reduces the frequency of rendering command issuance but also effectively alleviates the communication bottleneck between the CPU and GPU. When the camera position changes, the system dynamically switches the model detail level according to the camera's viewing distance, thereby achieving on-demand allocation of rendering resources.

[0041] To support the aforementioned dynamic switching mechanism, the dynamic switching of model detail levels based on camera view distance includes: dividing the urban drainage network into several data blocks according to a geographic grid and constructing a spatial index to quickly locate the visible area. This embodiment defines model detail levels, including a complete model level, a simplified model level, and a wireframe model level. Once the distance between the camera and a node approaches or exceeds the level switching threshold, the system dynamically loads the data blocks of the corresponding level and unloads data from levels far from the viewpoint. If the view distance continues to decrease, it gradually switches to a higher detail level; conversely, it gradually degrades to the wireframe level. Through this dynamic LOD loading and layering strategy, the rendering frame rate in massive scenes is stabilized, and the memory usage shows a linear decreasing trend.

[0042] In practical deployments, differences in computing power among terminal devices and the need for data backtracking remain key bottlenecks. To address this, this embodiment further provides multi-terminal adaptation, hierarchical storage, and multi-terminal synchronization mechanisms. The terminal adaptive rendering pipeline automatically categorizes terminals based on device type, computing power level, screen resolution, and network bandwidth, preventing both low-end devices from experiencing lag due to overload and high-end devices from being idle. Preferably, the system controls the deployment of INT8 quantization models on PCs and large-screen devices, and the deployment of INT4 ultra-lightweight models on mobile devices. To ensure smooth interaction on mobile devices, the system controls the mobile devices to reduce resolution and disable anti-aliasing, while controlling the large-screen devices to increase rendering accuracy. In the event of complex rendering tasks, the system pushes these tasks to the cloud for execution, with local devices only handling lightweight rendering, thus forming a cloud-edge-device collaborative architecture.

[0043] At the data persistence level, this embodiment constructs a three-tiered storage architecture of hot, warm, and cold data. Raw quantitative data from the past seven days is stored as hot data to support queries; compressed data from seven days to three months is stored as warm data to support backtracking analysis; and data older than three months is archived to object storage as cold data to support on-demand loading. A data caching mechanism is configured to prioritize reading cached hot data during visualization queries, and if a cache miss occurs, data is read from the storage layer and the cache is updated. Because query requests are highly concentrated on recent operational conditions, this caching strategy ensures that the hot data hit rate remains stable at over 90%, thereby significantly reducing visualization loading time.

[0044] In multi-terminal collaborative scenarios, after rendering is completed, the timestamp is written back to form a closed-loop scheduling mechanism. The NTP protocol is used to calibrate the time of multiple terminals to ensure data timing consistency. The scene states of multiple terminals are periodically compared, and synchronization is performed based on the data from the master terminal when states are inconsistent. Only the changed data is synchronized to perform incremental data synchronization. Given limited network bandwidth, the incremental synchronization mechanism effectively reduces redundant data transmission, thereby ensuring multi-terminal data consistency. In a system architecture adapted to the above method, a multi-terminal adaptation module, a data compression and storage module, and a multi-terminal synchronization module are also added. The multi-terminal adaptation module automatically categorizes terminals based on device type, computing power level, screen resolution, and network bandwidth. It controls the deployment of INT8 quantization models on PCs and large-screen devices, and the deployment of INT4 ultra-lightweight quantization models on mobile devices. Complex rendering tasks on mobile devices are pushed to the cloud for execution, while lightweight rendering is performed on the local device. The data compression and storage module constructs a three-tiered storage architecture: hot, warm, and cold. Nearly seven days of raw quantized data is stored as hot data; seven days to three months of compressed data is stored as warm data; and data older than three months is archived to object storage as cold data. A data caching mechanism is configured to prioritize visual query requests. The multi-terminal synchronization module uses the NTP protocol to calibrate the time of multiple terminals, periodically compares the scene status of multiple terminals, and synchronizes based on the master terminal data when inconsistencies arise. It also synchronizes only the changed data to perform incremental data synchronization. These modules work collaboratively with the core processing module, achieving full coverage of all business scenarios and improving the overall robustness of the system.

[0045] This invention introduces a combined implementation method of lightweight pruning, quantization-aware training, and model fusion throughout the entire data cleaning and rendering optimization process. The core steps and formulas are as follows: Lightweight pruning: Pruning non-core model channels based on an attention mechanism; pruning loss function: ; in, Losses due to data cleaning / rendering tasks For sparse regularization coefficients, For the first Channel attention weights This is a channel feature mapping.

[0046] Quantization-aware training: Employs INT4 / INT8 quantization to reduce model size and computational cost. Forward propagation formula: ; in, For model input and output, To quantify weights, For quantitative scale, This represents the number of quantization bits.

[0047] Model fusion scheduling: Optimal scheduling of lightweight and quantized models is achieved by fusing the loss function. Fusion formula: ; in, For load balancing losses, For inference performance loss, For data-rendering synchronization loss, .

[0048] In one embodiment of the present invention, a method for synchronizing real-time drainage monitoring data streaming processing and twin scene rendering constructs a frame-level synchronization queue for streaming data processing and rendering, and adopts a pipeline architecture of "data preprocessing - model inference - rendering triggering" to achieve synchronous updates of data processing and scene rendering. The core steps include: Collect drainage network monitoring data (liquid level, flow rate, rainfall, equipment status), and construct streaming data blocks at a frequency of 10ms / frame. , Batch size; Lightweight quantization preprocessing (INT4 quantization) is performed on the data block to obtain quantized data. ; Synchronously trigger data cleaning, model inference, and scene rendering tasks: After data cleaning is completed, immediately push the data to the rendering queue, and the scene state corresponding to the data is updated synchronously in the rendering frames; A delay compensation mechanism is adopted when the data processing delay exceeds a threshold. At the same time, frame interpolation is used to complete the rendering frames to ensure smooth visuals.

[0049] Synchronization formula: ; ; in, For data-rendering synchronization delay, Data processing / rendering time, Based on the rendering frame is the frame compensation coefficient.

[0050] In one embodiment of the present invention, a method for cleaning, imputing missing values, and quantizing drainage anomaly data based on a lightweight large model is constructed. A combined cleaning model of a lightweight large model and a quantization model is built to achieve the entire process of outlier removal, missing value imputation, and quantization value correction. The core steps and formulas are as follows: Outlier detection and removal: A lightweight large model is used to extract temporal and spatial features of the data. Outliers are identified through an attention mechanism. The outlier detection loss is: ; Missing value imputation: A lightweight imputation model trained based on quantization awareness imputation, which imputes missing data by fusing temporal and topological features. Imputation formula: ; in, Predicted values ​​for missing values. To quantify the perception of long short-term memory networks, These are the topological feature weight coefficients. This refers to the topological characteristics of pipeline nodes.

[0051] Quantization calibration: To address sensor range overflow and quantization error, an error compensation formula is used for correction. ; in, For quantified data, For raw floating-point data, This is the error compensation coefficient (value range: 0.01-0.1).

[0052] Model combination: A lightweight GPT-2 pruning model (for temporal features) + a quantized CNN model (for spatial features) are used, and the final cleaned result is obtained through weighted fusion. The fusion formula is as follows: ; in, , The results are from traditional statistical methods.

[0053] In one embodiment of the present invention, a method for lightweight rendering and frame rate optimization driven by a large model of a three-dimensional urban drainage scene is adopted. For a three-dimensional scene of drainage pipe network (pipe network, equipment, weather, special effects), a lightweight model-driven + quantitative rendering strategy is adopted. The frame rate is improved by model lightweighting, LOD level rendering, and batch rendering optimization.

[0054] The core steps and formulas are as follows: Lightweight scene modeling: Lightweight pruning is performed on the pipeline and equipment models, removing redundant surfaces and textures. Model volume compression formula: ; Quantization rendering optimization: Employs the INT4 quantization rendering model to improve the rendering speed of textures and models. Quantization rendering efficiency formula: ; Batch instance rendering: Instancing is used for repetitive network segments and device models to reduce draw calls. Batch rendering formula: ; in, To plot the number of calls, This is a batch of model collections.

[0055] Frame rate optimization goals: Achieve ≥60fps on PC, ≥30fps on mobile, and ≥45fps on large screens through a combination of lightweight and quantization.

[0056] In one embodiment of the present invention, a lightweight multi-terminal adaptation method for a digital drainage twin system on PCs, large screens, and mobile devices is proposed. This method constructs a terminal adaptive rendering pipeline and dynamically adjusts rendering parameters and model accuracy based on terminal computing power, screen resolution, and network bandwidth to achieve seamless multi-terminal adaptation.

[0057] Core adaptation strategy: Terminal type identification: Automatically classifies devices into PC, large screen, and mobile devices based on device information (CPU / GPU / memory, resolution, network); Model accuracy adaptive: Deploy lightweight quantized large model (INT8) on PC / large screen, and ultra-lightweight model (INT4) on mobile devices. Adaptation formula: ; Dynamic adjustment of rendering parameters: Reduce resolution and disable anti-aliasing on mobile devices, and improve rendering accuracy on large screens. Parameter adjustment formula: ; Resource adaptive allocation: Mobile devices utilize edge computing to push complex rendering tasks (such as flooding effects) to the cloud, while the local device only handles lightweight rendering. The cloud-based collaborative formula is as follows: ; In one embodiment of the present invention, the method for dynamic loading and LOD (Level of Detail) rendering optimization of drainage twin visualization data adopts a data block + LOD (Level of Detail) rendering strategy for massive drainage network data to achieve on-demand loading and hierarchical rendering of data, thereby optimizing memory usage and rendering performance.

[0058] Core implementation steps: Data partitioning and indexing: The urban drainage network is divided into geographic grids. Each data block is used to build a spatial index (R tree) to achieve fast location. LOD hierarchy definition: Divided into 3 levels based on data precision and complexity: LOD0 (High Precision): Complete pipeline model + high-definition texture, suitable for close-up viewing; LOD1 (Medium Precision): Simplified network model + ordinary texture, suitable for medium-distance viewing; LOD2 (Low Detail): Wireframe model + low-resolution texture, suitable for viewing at a distance; Dynamic loading and unloading: Based on the camera's viewpoint and distance, dynamically load data blocks at the corresponding LOD level and unload lower-level data that is far from the viewpoint. Loading formula: ; in, The distance between the camera and the node. The threshold for level switching ( , ).

[0059] Batch loading optimization: A preloading strategy is adopted to preload 2-3 data blocks in front of the viewpoint to avoid loading lag.

[0060] In one embodiment of the present invention, a multi-terminal data consistency synchronization and low-latency visual interaction method is proposed. This method constructs a distributed data synchronization mechanism based on incremental synchronization, timestamp calibration, and state consistency verification to achieve real-time data synchronization and low-latency interaction across multiple terminals. The core formula is as follows: Incremental data synchronization: Only the changed data is synchronized, not the entire dataset. The incremental synchronization formula is: ; ; Timestamp calibration: Multiple terminal times are calibrated using the NTP protocol to ensure data timing consistency. Time synchronization error formula: ; Interaction latency optimization: Local caching of interactive operations (such as device control), operation results are first rendered locally, then synchronized to the cloud, and the status is updated after feedback from the cloud. Interaction latency formula: ; State consistency verification: Periodically compare the scene states (device status, data values) of multiple terminals. If there is inconsistency, the main terminal (PC) shall be used for synchronization. The consistency verification formula is as follows: .

[0061] In one embodiment of the present invention, a method for compressing, storing, and rapidly visualizing drainage data employs a lightweight compression algorithm and a hierarchical storage strategy for drainage time-series monitoring data, thereby reducing storage costs and improving the speed of data reading and visualization.

[0062] Core methods and formulas: Data compression: A combination strategy of Zstandard lightweight compression and quantization encoding is used to compress time-series data. The compression ratio formula is as follows: ; Quantization compression formula: ; in, The quantification level is 4 / 8 / 16.

[0063] Tiered storage: Hot data (last 7 days): Stores raw quantitative data and supports fast querying; Warm data (7 days - 3 months): Stores compressed data and supports retrospective analysis; Cold data (over 3 months): Stored in object storage, supporting on-demand loading.

[0064] Fast call optimization: Build a data index + caching mechanism. During visualization queries, prioritize reading hot data from the cache; if no data is found, read and load from storage. Loading formula: ; in, For cache hit rate, the target is ≥90%.

[0065] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method for optimizing digital drainage twin flow cytometry data processing, AI cleaning, and 3D visualization rendering, characterized in that: include: Collect monitoring data of drainage pipe network and construct streaming data blocks according to a fixed sampling frequency; Lightweight quantization preprocessing is performed on streaming data blocks to obtain quantized data; The quantitative data is input into the data cleaning model for inference to obtain the cleaned data. Rendering trigger commands are constructed based on the cleaned data and pushed to the rendering queue. The rendering queue synchronously triggers the scene rendering engine to update the state of the twin scene. Monitor data processing latency, and when the latency exceeds a preset threshold, start the frame interpolation compensation algorithm to complete the rendering frame; After rendering is complete, the timestamp is written back to form a closed-loop scheduling.

2. The method according to claim 1, characterized in that, in, The step of inputting quantified data into the data cleaning model for inference includes: Temporal features are extracted by pruning and quantizing the Long Short-Term Memory network. An attention mechanism is used to identify and remove outliers such as jumps, drifts, and zeroing. For missing data, the topological features of pipeline nodes and temporal features are fused together for joint prediction and completion. For sensor range overflow and quantization error, a preset error compensation coefficient is used to reverse correct the quantized data.

3. The method according to claim 1, characterized in that, in, The synchronous triggering of the rendering queue to update the twin scene state by the scene rendering engine includes: Lightweight pruning is performed on the pipeline and equipment models to remove redundant faces and textures. For duplicate pipeline segments and equipment models, instantiation batch rendering and merging are used for drawing calls, and the model detail level is dynamically switched according to the camera's viewing distance.

4. The method according to claim 3, characterized in that, The dynamic switching of model detail levels based on camera line of view includes: The urban drainage network is divided into several data blocks according to a geographic grid and a spatial index is constructed. Define the model detail levels, including the full model level, the simplified model level, and the wireframe model level; Based on the distance between the camera and the node and the layer switching threshold, the data blocks of the corresponding layer are dynamically loaded and the layer data that is far from the viewpoint is unloaded.

5. The method according to claim 1, characterized in that, Also includes: The terminal adaptive rendering pipeline automatically classifies terminals based on terminal device type, computing power level, screen resolution, and network bandwidth. It controls the deployment of INT8 quantization models on PCs and large screens, the deployment of INT4 ultra-lightweight models on mobile devices, reduces the resolution and disables anti-aliasing on mobile devices, improves the rendering accuracy on large screens, and pushes complex rendering tasks to the cloud for execution, while local devices are only responsible for lightweight rendering.

6. The method according to claim 1, characterized in that, Also includes: A three-tiered storage architecture of hot, warm, and cold data was constructed to store nearly seven days' worth of raw quantitative data as hot data to support querying. Store compressed data from seven days to three months as warm data to support retrospective analysis; Archive data older than three months to object storage as cold data to support on-demand loading; Configure a data caching mechanism to prioritize reading cached hot data during visual queries, and read from the storage layer and update the cache if no data is found.

7. The method according to claim 1, characterized in that, in, After rendering is completed, the timestamp is written back to form a closed-loop scheduling. The NTP protocol is used to calibrate the time of multiple terminals to ensure data timing consistency. The scene status of multiple terminals is compared periodically, and when the status is inconsistent, the data of the main terminal is used for synchronization. Only the changed data is synchronized to perform incremental data synchronization.

8. The method according to claim 1, characterized in that, in, The streaming data blocks undergo lightweight quantization preprocessing using INT4 or INT8 quantization levels, and the quantized data blocks are grouped according to a preset batch size.

9. A digital drainage twin flow cytometry data processing, AI cleaning, and 3D visualization rendering optimization system, characterized in that, include: The streaming data processing module is used to collect monitoring data of drainage pipe network, construct streaming data blocks at a fixed sampling frequency, and perform lightweight quantization preprocessing on the streaming data blocks to obtain quantized data; The lightweight AI cleaning module is used to input quantified data into the data cleaning model for inference and obtain cleaned data. The 3D quantization rendering module is used to construct rendering trigger commands based on the cleaned data and push them to the rendering queue. It controls the rendering queue to synchronously trigger the scene rendering engine to update the twin scene state and monitors data processing latency. The frame compensation scheduling module is used to start the frame interpolation compensation algorithm to complete the rendering frame when the data processing delay exceeds the preset threshold, and write back the timestamp after rendering to form a closed-loop scheduling.

10. The system according to claim 9, characterized in that, Also includes: The multi-terminal adaptation module is used to automatically classify terminals according to terminal device type, computing power level, screen resolution and network bandwidth, control the deployment of INT8 quantization model on PC and large screen terminals, control the deployment of INT4 ultra-lightweight model on mobile terminals, push complex rendering tasks of mobile terminals to the cloud for execution, and perform lightweight rendering on local devices. The data compression and storage module is used to construct a three-level hierarchical storage architecture of hot, warm, and cold. The raw quantitative data of nearly seven days is stored as hot data, the compressed data of seven days to three months is stored as warm data, and the data of more than three months is archived to object storage as cold data. A data caching mechanism is configured to prioritize the response to visualization query requests. The multi-terminal synchronization module is used to calibrate the time of multiple terminals using the NTP protocol, periodically compare the scene status of multiple terminals and synchronize the data of the master terminal when there is inconsistency, and synchronize only the changed data part to perform incremental data synchronization.

Images