Water meter data processing method based on data visualization

Abstract

The present invention discloses a water meter data processing method based on data visualization, which relates to the field of data processing technology and includes the following steps: establishing a data acquisition channel based on water meters, and performing high-frequency real-time acquisition of the original water consumption data of each water meter to form continuous time series data; performing timestamp standardization processing to align all the collected time information to a unified format and time granularity to ensure that the time axis of each data point is consistent. The present invention improves the accuracy of water meter data aggregation through time standardization and boundary control to avoid trend misjudgment; adopts multiple anomaly rejection algorithms to enhance anomaly identification and early warning accuracy; introduces an index mapping mechanism between graphics and original data to achieve two-way traceability of visualization results, thereby improving the credibility of data display and the reliability of management decisions. The overall solution improves the stability, accuracy and responsiveness of the intelligent water system.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a water meter data processing method based on data visualization. Background Art

[0002] "Data visualization-based water meter data processing" integrates traditional water meter data collection and management with data visualization technology. This technology presents collected user water usage data (such as water consumption, water usage duration, and flow rate changes) through intuitive charts, curves, and heat maps. This allows managers and users to understand water usage in real time, identify abnormal water use behavior or leakage, and improve water resource management efficiency. This method not only supports data storage and analysis but also enhances data interpretation through dynamic, interactive graphics. It is widely used in scenarios such as smart water services and urban water supply monitoring.

[0003] Existing technologies have the following shortcomings: In existing water meter data processing based on data visualization, data aggregation anomalies are an easily overlooked but far-reaching issue. Specifically, when aggregating raw water meter data by time or regional dimensions (such as hourly averages, daily water consumption, and regional combined displays), if the processing logic contains time alignment errors, boundary data loss, or improper filtering of abnormal data, the generated visualization results may deviate from actual water use. This deviation directly affects the judgment of user water use behavior. For example, in the event of a sudden water leak, the system fails to identify and warn in a timely manner, thereby delaying disposal, resulting in wasted water resources and potential economic losses. Because such problems are often hidden in the underlying logic of data processing, they are difficult to directly reflect in front-end graphics, resulting in high risks and serious consequences.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a water meter data processing method based on data visualization. This method improves the accuracy of water meter data aggregation through time standardization and boundary control, avoiding trend misjudgments. It also employs a multiple anomaly rejection algorithm to enhance anomaly identification and early warning accuracy. Furthermore, it introduces an index mapping mechanism between graphics and raw data to achieve bidirectional traceability of visualization results, enhancing the credibility of data presentation and the reliability of management decisions. This overall solution improves the stability, accuracy, and responsiveness of intelligent water systems, addressing the aforementioned issues in the background art.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a water meter data processing method based on data visualization, comprising the following steps:

[0007] Establish a data collection channel based on water meters, collect the raw water consumption data of each water meter in real time at a high frequency, and form continuous time series data;

[0008] Perform timestamp standardization to align all collected time information to a unified format and time granularity, ensuring consistency of the time axis for each data point;

[0009] Set integrity check rules to verify the number of data items, data intervals, and boundary data in each collection cycle, and identify and mark missing and abnormal data segments;

[0010] A dynamic aggregation boundary control mechanism is introduced to automatically adjust the aggregation boundary based on the aggregation dimension and time window set by the user to avoid missing or duplicate statistics on boundary data.

[0011] Using multiple outlier rejection algorithms, based on a combination of historical statistical models and real-time threshold strategies, we can identify and eliminate outlier data values ​​that affect aggregation accuracy.

[0012] Generate graphical results for data that has been standardized, verified, aggregated, and anomaly eliminated, and simultaneously establish an index mapping between the chart and the original data to achieve traceability verification between the visualization results and the underlying data.

[0013] Preferably, the data acquisition channel is an asynchronous parallel structure built based on the Internet of Things communication module, and the data acquisition channel transmits the data collected by each water meter device to the cloud database in real time through the NB-IoT or LoRa communication protocol;

[0014] The data acquisition channel is equipped with an independent buffer area to avoid data loss due to network delays, and a unique identifier and device number are attached to each data for subsequent data clustering identification and cross-validation.

[0015] Preferably, the timestamp normalization process performed further includes normalizing the data at a minute-level time granularity and performing unified calibration using a global clock synchronization mechanism;

[0016] The global clock synchronization mechanism is based on the NTP protocol and combines dual synchronization with the local RTC redundancy calibration solution to ensure that the time error is controlled within ±1 second;

[0017] At the same time, each data is set to be calibrated before and after normalization;

[0018] During the calibration process, data with a time offset exceeding the set threshold will be automatically removed.

[0019] Preferably, the integrity check rule includes a periodic data backtracking mechanism, which performs a window scan of the previous hour's data every 10 minutes;

[0020] If the sampling frequency is abnormal, the supplementary sampling mechanism will be automatically triggered to fit the missing values ​​with the data of the adjacent time period;

[0021] In addition, for time points that cross daily boundaries, the difference calculation and verification of the previous and subsequent sampling points are automatically performed. If the difference exceeds the maximum allowable flow change value of the water meter, it is reported as a boundary error and eliminated.

[0022] Preferably, the dynamic aggregation boundary control mechanism is executed based on a rolling window algorithm, which automatically determines whether the current time slice is in a data stable region when each time window switches;

[0023] If the data at the boundary changes dramatically, the aggregation window is expanded forward or backward to include the complete data segment to avoid deviations in the aggregation results caused by boundary truncation.

[0024] Preferably, the multiple exception elimination algorithm includes:

[0025] Use the moving median smoothing algorithm to model historical data;

[0026] Real-time data is compared with the model prediction value through the deviation threshold mechanism. If the deviation exceeds the upper and lower limits, it is judged as abnormal;

[0027] A seasonal cycle correction coefficient is introduced to adjust the threshold boundary according to historical seasonal fluctuations to enhance adaptability to periodic water use patterns.

[0028] Preferably, an advanced data calculation model is set up to quantitatively analyze the collection integrity and abnormality of water meter data to improve the accuracy of identifying aggregated abnormalities. The specific steps are as follows:

[0029] Based on the data collection situation within the time period, the integrity vector is constructed to evaluate the integrity of the data collected in each time period relative to the expected number of samples. At the same time, the degree of data fluctuation is comprehensively considered. The calculation expression is as follows:

[0030] , where is the integrity vector, is the actual number of data points collected, that is, in the time period The number of water meter data successfully collected, is the expected number of data points, and are the standard deviation and mean of the water meter data in the current time period respectively;

[0031] In order to evaluate the data mutation phenomenon at the boundary of the time window, a boundary correction factor is introduced to measure the degree of drastic changes in water use data in adjacent time windows. The calculation expression is as follows:

[0032] , where is the boundary correction factor, Is the current time period Total water consumption, is the total water consumption in the next time period, is the time window width;

[0033] The comprehensive quality of data in each time period is evaluated by calculating the anomaly score. The calculation expression is as follows:

[0034] , where is the anomaly score, is the integrity weight parameter, is the boundary change weight parameter;

[0035] When the anomaly score Greater than the set threshold When It is the abnormality judgment threshold. The data in the current time period is judged as abnormal and automatically eliminated or alarmed to ensure the credibility of subsequent data aggregation.

[0036] Preferably, the generated graphical results are constructed using a responsive dynamic graphics engine, and the graphics include but are not limited to line graphs, bar graphs, heat maps and other types, and support multi-dimensional combination display;

[0037] A one-to-one mapping index is established for each graphic element and its corresponding original data. When the user clicks a node in the graphic on the front end, the back end will display the details of the original data associated with the node.

[0038] Preferably, during the visualization process, an index matching algorithm based on data structure similarity calculation is introduced to achieve accurate mapping between front-end graphic elements and back-end original data. The specific steps are as follows:

[0039] Construct a data scatter matrix to quantify the degree of difference between each graph node value and the candidate original data value. The calculation formula is as follows:

[0040] , where Is the first node in the graph Display value, The first The actual sample value, is an element in the data scatter matrix, indicating the Graphic nodes display values With the True sample values The normalized squared difference between

[0041] After constructing the divergence matrix, the optimal path search strategy is used to find a set of best-matching index paths. The optimization goal is to minimize the total divergence value. The formula is as follows:

[0042] , where is the global optimal path, Is any set of legal index paths, containing several pairs , representing a graph node The actual sample value One-to-one mapping relationship, is the sum of the divergences between all graph nodes on the index path and their original data mapping points;

[0043] In order to evaluate the overall matching quality of the index results, the index confidence score is introduced. The calculation expression is as follows:

[0044] , where Is the optimal path The total divergence value on is used to measure the total deviation value between all matching pairs. is the number of matching pairs in the optimal path, is the index confidence score;

[0045] When the rating value If it is higher than the preset threshold, it means that the current graphic display has highly restored the original data structure and is traceable, allowing users to conduct behavioral analysis and trend analysis; otherwise, the user will be prompted that there is a risk of mismatch in the graphic and the visualization chart will be regenerated.

[0046] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0047] By introducing timestamp standardization, integrity detection, and boundary correction mechanisms, this paper effectively avoids problems such as data misalignment, boundary omissions, and insufficient sampling before the raw water meter data is aggregated and processed. In particular, when performing hourly or daily summary analysis, the aggregation logic can automatically identify and repair discontinuities in edge data segments, ensuring that the visualization results accurately reflect actual water usage trends. This mechanism significantly reduces the risk of trend misjudgment due to data aggregation bias and improves the system's real-time response capabilities to water use anomalies. It is suitable for a variety of scenarios, including residential areas, industrial parks, and urban water supply networks.

[0048] This invention uses multiple anomaly rejection algorithms combined with historical models and real-time rules to accurately identify sudden changes, periodic fluctuations, and abnormal equipment sampling in water meter data. The rejection logic is adaptively adjustable, effectively eliminating irregular noise interference. It also tightly couples the anomaly determination process with subsequent data visualization, enabling anomaly annotation and retrospective analysis on visual charts. This deeply integrated data processing method significantly improves the accuracy of the system's anomaly warnings, enabling managers to detect emergencies such as leaks and pipe bursts earlier and quickly locate the source of risk.

[0049] This invention introduces an index mapping mechanism between graph nodes and raw data during the data visualization process, and uses a divergence matrix and an optimal path matching algorithm to establish a precise correspondence between the graph results and the data processing process. Users can click on any point in the front-end graphical interface, and the system will return its source data, processing flow, and exclusion marks in real time, forming a complete traceability chain for the graphical results. This mechanism not only ensures the verifiability of graphical displays, but also enhances the reliability of the platform's decision-making basis in scenarios such as water supply scheduling and resource allocation, effectively improving the management level of the entire smart water system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0051] Figure 1 This is a flow chart of a method for processing water meter data based on data visualization according to the present invention. DETAILED DESCRIPTION

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0053] The present invention provides Figure 1 The water meter data processing method based on data visualization shown includes the following steps:

[0054] Establish a data collection channel based on water meters, collect the raw water consumption data of each water meter in real time at a high frequency, and form continuous time series data;

[0055] The data acquisition channel is an asynchronous parallel structure built based on the IoT communication module. It transmits the data collected by each water meter device to the cloud database in real time through the NB-IoT or LoRa communication protocol.

[0056] The data acquisition channel is equipped with an independent buffer area to avoid data loss due to network delays, and a unique identifier and device number are attached to each data for subsequent data clustering identification and cross-validation.

[0057] In order to prevent noise interference during data acquisition, the channel installs a digital filter at the front end for preliminary denoising to improve the basic accuracy of subsequent data processing

[0058] Perform timestamp standardization to align all collected time information to a unified format and time granularity, ensuring consistency of the time axis for each data point;

[0059] The timestamp normalization process performed further includes normalizing the data at minute-level time granularity and using a global clock synchronization mechanism for unified calibration;

[0060] The global clock synchronization mechanism is based on the NTP protocol and combines dual synchronization with the local RTC (real-time clock) redundant calibration scheme to ensure that the time error is controlled within ±1 second;

[0061] At the same time, each piece of data is set to be calibrated before and after normalization for subsequent tracing;

[0062] During the calibration process, data with a time offset exceeding the set threshold is automatically removed to avoid misleading the aggregation calculation logic.

[0063] Set integrity check rules to verify the number of data items, data intervals, and boundary data in each collection cycle, and identify and mark missing and abnormal data segments;

[0064] The integrity check rules include a periodic data backtracking mechanism, which scans the data of the previous hour every 10 minutes to determine whether there are any anomalies such as missing periods, repeated sampling points, and sampling interval fluctuations exceeding thresholds.

[0065] If the sampling frequency is abnormal (for example, greater than 1.5 times the average sampling rate), the supplementary sampling mechanism will be automatically triggered to fit the missing values ​​with the data of the adjacent time period;

[0066] In addition, for time points that cross daily boundaries, the difference calculation and verification of the previous and subsequent sampling points are automatically performed. If the difference exceeds the maximum allowable flow change value of the water meter, it is reported as a boundary error and eliminated.

[0067] A dynamic aggregation boundary control mechanism is introduced to automatically adjust the aggregation boundary based on the aggregation dimension and time window set by the user to avoid missing or duplicate statistics on boundary data.

[0068] The dynamic aggregation boundary control mechanism is based on a rolling window algorithm, which automatically determines whether the current time slice is in the data stability area when switching between each time window.

[0069] If the data at the boundary changes dramatically, the aggregation window is expanded forward or backward to include the complete data segment to avoid deviations in the aggregation results caused by boundary truncation.

[0070] The system supports manual setting of the maximum range for expansion and provides boundary adjustment logs after aggregation for subsequent O&M review. Furthermore, the aggregation logic automatically excludes boundary values ​​that exceed measurement limits to ensure the physical plausibility of the aggregated data.

[0071] Using multiple outlier rejection algorithms, based on a combination of historical statistical models and real-time threshold strategies, we can identify and eliminate outlier data values ​​that affect aggregation accuracy.

[0072] Multiple outlier rejection algorithms include:

[0073] Use the moving median smoothing algorithm to model historical data;

[0074] Real-time data is compared with the model prediction value through the deviation threshold mechanism. If the deviation exceeds the upper and lower limits, it is judged as abnormal;

[0075] A seasonal cycle correction coefficient is introduced to adjust the threshold boundary according to historical seasonal fluctuations to enhance adaptability to periodic water use patterns.

[0076] All abnormal rejection processes record the data point number, deviation magnitude and rejection reason in the log, and support users to customize rejection rules and retrain models to cope with business expansion.

[0077] Set up an advanced data calculation model to quantitatively analyze the collection completeness and anomaly level of water meter data to improve the accuracy of identifying aggregate anomalies. The specific steps are as follows:

[0078] Based on the data collection situation within the time period, the integrity vector is constructed to evaluate the integrity of the data collected in each time period relative to the expected number of samples. At the same time, the degree of data fluctuation is comprehensively considered. The calculation expression is as follows:

[0079] , where Is the integrity vector, indicating that in a certain period of time The overall collection quality index of the data within a period. The closer the value is to 1, the more complete the data is and the smaller the fluctuation is. It is the core basis for evaluating whether the time period is suitable for aggregation. is the actual number of data points collected, that is, in the time period The number of water meter data successfully collected comes from actual communication records or database query results. is the expected number of data points, which is calculated based on the set sampling frequency and time period. For example, if sampling is once per minute and the time period is 1 hour, then , and are the standard deviation and mean of the water meter data in the current time period, Used to measure time periods The fluctuation range of internal data is a sensitive indicator used to detect whether there are drastic changes or anomalies. The larger the value, the more drastic the data fluctuation. It represents the average value of all sampling values ​​within the time period. It is a typical reference value for water meter flow data and serves as the denominator for normalizing the standard deviation to avoid exaggerating the volatility.

[0080] In order to evaluate the data mutation phenomenon at the boundary of the time window, a boundary correction factor is introduced to measure the degree of drastic changes in water use data in adjacent time windows. The calculation expression is as follows:

[0081] , where It is a boundary correction factor used to measure the degree of dramatic change in total water consumption between adjacent time periods. A larger value indicates a more abrupt boundary change, which may indicate that the boundary is truncated or data is missed. Is the current time period Total water consumption, is the total water consumption in the next time period, is the width of the time window, usually an aggregate unit of time (e.g., 1 hour, 30 minutes), used to normalize changes in water use to the rate of change per unit time, ensuring fair comparisons across different time scales;

[0082] The comprehensive quality of data in each time period is evaluated by calculating the anomaly score. The calculation expression is as follows:

[0083] , where Anomaly Score is the anomaly score that comprehensively reflects whether there are any abnormal phenomena that affect aggregation in the current time period. The higher the value, the less suitable the data is for visualization or trend analysis. is the integrity weight parameter, which is an empirical setting value. The importance of (data integrity) is adjusted between 0.4 and 0.7. It is the boundary change weight parameter, which is also an empirical setting value and is used to control the boundary change The sensitivity of the impact degree ranges from 0.3 to 0.6;

[0084] When the anomaly score Greater than the set threshold When It is the abnormality judgment threshold, which is an empirical threshold (such as 0.8). The data in the current time period is judged as abnormal and automatically eliminated or alarmed to ensure the credibility of subsequent data aggregation.

[0085] Generate graphical results from data that has been standardized, verified, aggregated, and anomaly eliminated, and simultaneously establish an index mapping between the chart and the original data to achieve traceability verification between the visualization results and the underlying data;

[0086] The generated graphical results are built using a responsive dynamic graphics engine. Graphics include but are not limited to line charts, bar charts, heat maps and other types, and support multi-dimensional combination display;

[0087] A one-to-one mapping index is established for each graphic element and its corresponding original data. When the user clicks a node in the graphic on the front end, the back end will display the details of the original data associated with the node.

[0088] To ensure the consistency of charts, the system synchronizes the index of the chart with the database every 30 minutes and generates a snapshot backup to avoid distortion of the chart information due to data rollback.

[0089] During the visualization process, an index matching algorithm based on data structure similarity calculation is introduced to achieve accurate mapping between front-end graphic elements and back-end raw data, and to ensure the traceability of graphic display and the reliability of analysis. The specific steps are as follows:

[0090] Construct a data scatter matrix to quantify the degree of difference between each graph node value and the candidate original data value. The calculation formula is as follows:

[0091] , where Is the first node in the graph A display value, which is usually the value of a point or column that users see on a visualization chart. It comes from aggregated water meter data, such as the total water consumption in a certain period of time. Its unit is usually cubic meters (m³) or liters (L), depending on the original data unit. The first A real sampling value, which is the underlying raw water meter data without any aggregation or visualization processing, representing the real collected data at a specific time point, such as flow data every 5 minutes. is an element in the data scatter matrix, indicating the Graphic nodes display values With the True sample values The normalized square difference between them represents the relative difference between the two. The smaller the value, the higher the similarity between the two, which can be used as the basis for subsequent index path selection;

[0092] Each element in this matrix represents the normalized squared difference between two values, and the smaller the value, the higher the similarity.

[0093] The index matching algorithm in the "Index Matching Algorithm Based on Data Structure Similarity Calculation" essentially borrows and improves the concept of the Dynamic Time Warping (DTW) algorithm, combining it with the data divergence matrix to construct the optimal matching path. Its core is to calculate the normalized difference between the graphical display value and the original data value to form a two-dimensional error matrix, and then find a path in this matrix that minimizes the total error to establish a one-to-one mapping relationship between the visualized data points and the original data. The purpose of this algorithm is to accurately restore the true source of the information presented in the chart, improve the traceability and accuracy of the data visualization system, and is particularly suitable for handling complex matching scenarios with time offsets, data sparsity, structural fluctuations, etc. Through this matching process, the system can identify the actual sampling basis behind the graphical data, ensuring that the graphical display truly reflects the characteristics of the underlying data within a controllable error range.

[0094] After constructing the divergence matrix, the optimal path search strategy is used to find a set of best matching index paths. Indicates that a data point in the graph corresponds to an item in the original data sequence, and the optimization goal is to minimize the total divergence value. The formula is as follows:

[0095] , where is the global optimal path, which is the path that minimizes the total divergence value among all possible paths. It represents the set of original data indexes that best matches the graphical display results. Is any set of legal index paths, containing several pairs , representing a graph node The actual sample value The path must maintain a monotonically increasing matching (for example, no jumps or backoffs). It is the sum of the divergences between all graph nodes on the index path and their original data mapping points. Minimizing this sum means selecting a set of matching points with the smallest overall difference;

[0096] The optimal path search strategy refers to the process of constructing a matching relationship between graphic data and original data, in which a path with the smallest total matching error is selected from all possible matching paths. This path is defined as the "optimal path." Technically, this strategy is usually implemented based on dynamic programming or graph search algorithms (such as A* and Dijkstra). The goal is to find a continuous, monotonically increasing sequence of nodes in the error matrix so that the sum of the errors of all elements on the path is minimized. Its function is to ensure that every data point in the visual graphics can be accurately indexed with the original data that best matches its numerical structure, thereby minimizing the deviation between the graphics and the actual data. Through this strategy, the system can achieve high-precision and high-reliability graphics traceability, so that data analysis and decision-making are based on real data rather than visual errors.

[0097] In order to evaluate the overall matching quality of the index results, the index confidence score is introduced. The calculation expression is as follows:

[0098] , where Is the optimal path The total divergence value on is used to measure the total deviation value between all matching pairs. Is the number of matching pairs in the optimal path, that is, the number of graph nodes matched. If there are 10 points in the graph participating in the matching, the value is 10. is the index confidence score, with a value range of [0, 1]. The closer the score is to 1, the more accurate the match between the graph and the original data, and the smaller the error. If the score is less than a certain threshold (such as 0.95), it means that there is a significant deviation in the graph and it lacks strong traceability. The system can issue a warning to the user to remind the user that the graph credibility is low.

[0099] When the rating value If the value is higher than the preset threshold (such as 0.95), it means that the current graphic display has highly restored the original data structure and is traceable, allowing users to conduct behavioral analysis and trend analysis. Otherwise, the user will be prompted that there is a risk of mismatch in the graphic and the visualization chart will be regenerated.

[0100] Implementation 1: To achieve intelligent water management in urban residential communities, water companies have widely deployed smart water meters with remote communication capabilities. These meters use NB-IoT (Narrowband Internet of Things) technology to upload real-time water usage data to a cloud-based data processing platform. Each meter uploads raw data points, such as water consumption, flow rate, and device status, every five minutes. The platform first processes data from all meters using a unified timestamp standardization mechanism, ensuring that even if upload times vary between devices, data is aligned to a common time dimension. This time alignment mechanism compares the server's NTP time source with the water meter's local RTC clock, interpolating or correcting for time offsets to ensure consistency across all data on a minute-by-minute timeline.

[0101] After standardization, the system enters the integrity check phase. Each hour constitutes a data integrity check window. Based on pre-set sampling density requirements, the system determines whether there are any issues such as missing data, duplicate data, or abnormal sampling intervals. If more than 5% of data is missing, that period is marked as a "data integrity anomaly," providing a warning during subsequent aggregation. For brief data interruptions, the system uses a time series interpolation algorithm to perform fitting and completion, recording the difference between the interpolated and original points to ensure traceability.

[0102] Before data visualization, the system constructs a spatial aggregation dimension based on the building number and device binding information, and uses a rolling time window to perform aggregation operations. The rolling window width can be configured to range from 30 minutes to 3 hours, and all water meter data within the window are aggregated and analyzed based on indicators such as average value, maximum value, and total water consumption. In order to deal with the problem of boundary data fluctuations that occur in reality, the system has designed a dynamic boundary expansion mechanism. When a sudden increase in water flow value occurs within the boundary of a window, but a sudden drop occurs in the next window, the system recognizes that the data has a boundary anomaly and automatically extends the window forward and backward to include the complete data segment, ensuring the continuity and authenticity of the aggregation results. This mechanism is particularly suitable for detecting sudden water leaks or abnormal water use behaviors at night.

[0103] The final aggregated data is rendered into heat maps and trend line graphs through the graphics engine. Users can interactively view the water usage details of any building and any time period. When a user clicks on a chart node, the system automatically pops up all the raw data details corresponding to that point, and indicates whether it is interpolated data or an abnormal elimination point. In addition, to support long-term analysis, the graphics engine regularly stores chart snapshots and their data indexes in historical archives for subsequent trend comparison and abnormal playback. This implementation method significantly improves the visualization capabilities and response efficiency of urban community water management.

[0104] Implementation 2: Water usage patterns in industrial parks differ significantly from those in residential communities. Industrial equipment relies heavily on water and requires consistent operation, necessitating a higher frequency of data collection. To meet this demand, smart water meters are configured to collect water usage data every minute and transmit it to edge computing devices via LoRa gateways or dedicated industrial Ethernet networks. The edge devices perform preliminary data cleaning before aggregating and uploading it to a cloud platform.

[0105] Due to the high frequency of data acquisition and the massive amount of data, time alignment accuracy is required to be even higher. This system utilizes a dual time synchronization mechanism: First, a central control server issues a reference time command, while second, each local data acquisition device performs an RTC calibration readback every hour. To ensure the validity of the time-aligned data, the platform immediately enters the anomaly elimination phase after timestamp standardization.

[0106] The anomaly rejection algorithm is based on a historical behavior model: the system records each industrial user's hourly, daily, and weekly water usage patterns, calculates the mean and standard deviation, and uses a moving median algorithm to filter out sudden excursions. When new data arrives, the system compares the deviation between the current value and the historical model. If the deviation exceeds three standard deviations, it is immediately identified as an anomaly. This rejection logic also incorporates time period characteristics. For example, if a nighttime production schedule causes a sudden increase in water consumption, the system dynamically adjusts the threshold to avoid misjudgments. Each rejected data point records the reason for rejection, the model deviation value, and the reference value, making them available for user query and review.

[0107] During the visualization process, the graphics engine displays hourly water flow changes via line charts, using color to indicate normal and excluded values. To ensure the credibility of the charts, the system incorporates a reverse validation mechanism. Each node in the chart has a unique ID that maps to the original database data point, allowing users to click on a chart point to view its source and whether it has undergone exception processing. This mechanism is particularly useful for production site safety management. For example, if a sudden drop in the chart is detected, the cause can be immediately identified, facilitating timely resolution and mitigating the risk of accidents.

[0108] Implementation Method 3: Regional water supply management systems typically involve multiple urban districts and thousands of water meters simultaneously uploading data. This system faces the pressure of processing massive amounts of data and requires complex spatial and temporal cross-analysis. This system uses a distributed database storage architecture at the data receiving end. Each sub-region has an edge gateway. Data is first cached in the gateway before being uploaded in batches to a central server to ensure high availability.

[0109] After data is uploaded, the system first constructs a completeness vector indicator—the ratio of the expected number of data items collected to the actual number of data items received within each time period. The completeness factor is then calculated by combining the standard deviation and the mean. If the completeness factor falls below a set threshold (e.g., 0.85), the data for that time period is marked as "partially reliable." The system also incorporates a boundary control mechanism. When an aggregation window includes a time period, the start and end positions of the window are automatically adjusted to prioritize the inclusion of complete data segments for improved accuracy.

[0110] To enhance the accuracy and traceability of graphical displays, the system implements an index matching algorithm during the visualization construction process. The data scatter matrix is ​​used to calculate the similarity between the displayed values ​​and the original data, and the optimal matching path is identified. If a chart segment deviates significantly from the original data, the system prompts the user to manually verify or re-aggregate. Furthermore, the system incorporates index confidence scores into the graphic metadata. Only graphics with a confidence score greater than 0.95 are allowed to be pushed to the water supply scheduling interface, ensuring the reliability of data-driven decision-making.

[0111] Through the above measures, the regional water supply platform not only realizes the structured management of large-scale data, but also ensures the authenticity, accuracy and availability of the data through integrity control and visual traceability, which is of great significance to improving the operating efficiency and intelligence of the entire water supply system.

[0112] By introducing timestamp standardization, integrity detection, and boundary correction mechanisms, this paper effectively avoids problems such as data misalignment, boundary omissions, and insufficient sampling before the raw water meter data is aggregated and processed. In particular, when performing hourly or daily summary analysis, the aggregation logic can automatically identify and repair discontinuities in edge data segments, ensuring that the visualization results accurately reflect actual water usage trends. This mechanism significantly reduces the risk of trend misjudgment due to data aggregation bias and improves the system's real-time response capabilities to water use anomalies. It is suitable for a variety of scenarios, including residential areas, industrial parks, and urban water supply networks.

[0113] This invention uses multiple anomaly rejection algorithms combined with historical models and real-time rules to accurately identify sudden changes, periodic fluctuations, and abnormal equipment sampling in water meter data. The rejection logic is adaptively adjustable, effectively eliminating irregular noise interference. It also tightly couples the anomaly determination process with subsequent data visualization, enabling anomaly annotation and retrospective analysis on visual charts. This deeply integrated data processing method significantly improves the accuracy of the system's anomaly warnings, enabling managers to detect emergencies such as leaks and pipe bursts earlier and quickly locate the source of risk.

[0114] This invention introduces an index mapping mechanism between graph nodes and raw data during the data visualization process, and uses a divergence matrix and an optimal path matching algorithm to establish a precise correspondence between the graph results and the data processing process. Users can click on any point in the front-end graphical interface, and the system will return its source data, processing flow, and exclusion marks in real time, forming a complete traceability chain for the graphical results. This mechanism not only ensures the verifiability of graphical displays, but also enhances the reliability of the platform's decision-making basis in scenarios such as water supply scheduling and resource allocation, effectively improving the management level of the entire smart water system.

[0115] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0116] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

[0117] It should be noted that, in this document, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0118] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0123] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0124] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A water meter data processing method based on data visualization, characterized in that: The following steps are involved: Establish a data collection channel based on water meters, collect the raw water consumption data of each water meter in real time at a high frequency, and form continuous time series data; Perform timestamp standardization to align all collected time information to a unified format and time granularity, ensuring consistency of the time axis for each data point; Set integrity check rules to verify the number of data items, data intervals, and boundary data in each collection cycle, and identify and mark missing and abnormal data segments; A dynamic aggregation boundary control mechanism is introduced to automatically adjust the aggregation boundary based on the aggregation dimension and time window set by the user to avoid missing or duplicate statistics on boundary data. Using multiple outlier rejection algorithms, based on a combination of historical statistical models and real-time threshold strategies, we can identify and eliminate outlier data values ​​that affect aggregation accuracy. Generate graphical results from data that has been standardized, verified, aggregated, and anomaly eliminated, and simultaneously establish an index mapping between the chart and the original data to achieve traceability verification between the visualization results and the underlying data; Set up an advanced data calculation model to quantitatively analyze the collection completeness and anomaly level of water meter data to improve the accuracy of identifying aggregate anomalies. The specific steps are as follows: Based on the data collection situation within the time period, the integrity vector is constructed to evaluate the integrity of the data collected in each time period relative to the expected number of samples. At the same time, the degree of data fluctuation is comprehensively considered. The calculation expression is as follows: , where is the integrity vector, is the actual number of data points collected, that is, in the time period The number of water meter data successfully collected, is the expected number of data points, and are the standard deviation and mean of the water meter data in the current time period respectively; In order to evaluate the data mutation phenomenon at the boundary of the time window, a boundary correction factor is introduced to measure the degree of drastic changes in water use data in adjacent time windows. The calculation expression is as follows: , where is the boundary correction factor, Is the current time period Total water consumption, is the total water consumption in the next time period, is the time window width; The comprehensive quality of data in each time period is evaluated by calculating the anomaly score. The calculation expression is as follows: , where is the anomaly score, is the integrity weight parameter, is the boundary change weight parameter; When the anomaly score Greater than the set threshold When It is the abnormality judgment threshold. The data in the current time period is judged as abnormal and automatically eliminated or alarmed to ensure the credibility of subsequent data aggregation.

2. The water meter data processing method based on data visualization according to claim 1, characterized in that: The data acquisition channel is an asynchronous parallel structure built based on the IoT communication module. It transmits the data collected by each water meter device to the cloud database in real time through the NB-IoT or LoRa communication protocol. The data acquisition channel is equipped with an independent buffer area to avoid data loss due to network delays, and a unique identifier and device number are attached to each data for subsequent data clustering identification and cross-validation.

3. The water meter data processing method based on data visualization according to claim 1, characterized in that: The timestamp normalization process performed further includes normalizing the data at minute-level time granularity and using a global clock synchronization mechanism for unified calibration; The global clock synchronization mechanism is based on the NTP protocol and combines dual synchronization with the local RTC redundancy calibration solution to ensure that the time error is controlled within ±1 second; At the same time, each data is set to be calibrated before and after normalization; During the calibration process, data with a time offset exceeding the set threshold will be automatically removed.

4. The water meter data processing method based on data visualization according to claim 1, characterized in that: The integrity check rules include a periodic data backtracking mechanism, which performs a window scan of the previous hour's data every 10 minutes; If the sampling frequency is abnormal, the supplementary sampling mechanism will be automatically triggered to fit the missing values ​​with the data of the adjacent time period; In addition, for time points that cross daily boundaries, the difference calculation and verification of the previous and subsequent sampling points are automatically performed. If the difference exceeds the maximum allowable flow change value of the water meter, it is reported as a boundary error and eliminated.

5. The water meter data processing method based on data visualization according to claim 1, characterized in that: The dynamic aggregation boundary control mechanism is based on a rolling window algorithm, which automatically determines whether the current time slice is in the data stability area when switching between each time window. If the data at the boundary changes dramatically, the aggregation window is expanded forward or backward to include the complete data segment to avoid deviations in the aggregation results caused by boundary truncation.

6. The water meter data processing method based on data visualization according to claim 1 is characterized in that: Multiple outlier rejection algorithms include: Use the moving median smoothing algorithm to model historical data; Real-time data is compared with the model prediction value through the deviation threshold mechanism. If the deviation exceeds the upper and lower limits, it is judged as abnormal; A seasonal cycle correction coefficient is introduced to adjust the threshold boundary according to historical seasonal fluctuations to enhance adaptability to periodic water use patterns.

7. The water meter data processing method based on data visualization according to claim 1 is characterized in that: The generated graphical results are built using a responsive dynamic graphics engine; A one-to-one mapping index is established for each graphic element and its corresponding original data. When the user clicks a node in the graphic on the front end, the back end will display the details of the original data associated with the node.

8. The water meter data processing method based on data visualization according to claim 1, characterized in that: During the visualization process, an index matching algorithm based on data structure similarity calculation is introduced to achieve accurate mapping between front-end graphic elements and back-end raw data. The specific steps are as follows: Construct a data scatter matrix to quantify the degree of difference between each graph node value and the candidate original data value. The calculation formula is as follows: , where Is the first node in the graph Display value, The first The actual sample value, is an element in the data scatter matrix, indicating the Graphic nodes display values With the True sample values The normalized squared difference between After constructing the divergence matrix, the optimal path search strategy is used to find a set of best-matching index paths. The optimization goal is to minimize the total divergence value. The formula is as follows: , where is the global optimal path, Is any set of legal index paths, containing several pairs , representing a graph node The actual sample value One-to-one mapping relationship, is the sum of the divergences between all graph nodes on the index path and their original data mapping points; In order to evaluate the overall matching quality of the index results, the index confidence score is introduced. The calculation expression is as follows: , where Is the optimal path The total divergence value on is used to measure the total deviation value between all matching pairs. is the number of matching pairs in the optimal path, is the index confidence score; When the rating value If the value is higher than the preset threshold, it means that the current graphic display has highly restored the original data structure and is traceable, allowing users to conduct behavioral analysis and trend analysis. Otherwise, the user will be prompted that there is a risk of mismatch in the graph and the visualization chart will be regenerated.

Images