Urban infrastructure-oriented missing data adaptive completion and quality evaluation method

Abstract

The invention relates to a missing data self-adaptive completion and quality evaluation method for urban infrastructures, and discloses the missing data self-adaptive completion and quality evaluation method for the urban infrastructures. The method comprises the following steps: firstly, acquiring monitoring data and identifying missing types, including short-time random missing, long-time continuous missing and static attribute missing; then adaptively matching an optimal completion strategy based on the missing features: adopting adaptive cubic spline interpolation for short-time missing, adopting STL periodic trend decomposition for long-time missing, and adopting topological neighborhood deduction for static attribute missing; constructing a confidence scoring model based on spatial adjacent measured values and historical data in the same period, and calculating a confidence score of complemented data; and finally, performing grading processing on the data according to a comparison result of the score and a threshold value. According to the method, the problem of data voids caused by equipment faults or network congestion in urban lifeline monitoring is effectively solved, and the false alarm rate of the system is effectively reduced through confidence evaluation while the data management accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of big data processing and industrial Internet of Things (IIoT) technology, specifically to a method for adaptive repair of missing values ​​and dynamic evaluation of data quality in multi-source monitoring data of urban lifeline projects (such as urban water supply networks, drainage networks, gas networks, bridge monitoring, etc.). Background Technology

[0002] In the construction of smart city infrastructure, a large number of sensors (such as level gauges, flow meters, and pressure transmitters) are widely deployed to achieve real-time monitoring. However, in actual operation, data gaps are extremely common, seriously affecting the accuracy of early warning systems.

[0003] The main reasons for the lack of existing data include:

[0004] 1. Equipment failure: The sensor battery is depleted, aged, or physically damaged, resulting in a long-term data interruption (manifested as a long-term continuous loss).

[0005] 2. Transmission congestion and packet loss: In city-level sensing networks, facing the high-concurrency reporting peaks of massive terminal devices, IoT gateways or message middleware often experience random data packet loss due to buffer overflow, resulting in discontinuous breaks in time series data (manifested as short-term point-like missing data).

[0006] 3. Missing static records: Historical existing pipeline network data (such as pipe diameter, material, and laying year) has blank fields during the digital archiving process.

[0007] The shortcomings of existing technologies lie in the fact that commonly used data completion methods (such as mean imputation, linear interpolation, and zero imputation) typically employ a "one-size-fits-all" static rule. These methods fail to fully utilize the spatiotemporal correlations of urban lifeline data (e.g., the periodic patterns of peak water usage in the morning and evening, and the flow direction correlations of upstream and downstream pipelines), resulting in low completion accuracy. Furthermore, existing technologies lack a quality assessment mechanism for the completed data. If the imputed values ​​deviate significantly (e.g., missing values ​​are mistakenly filled as high-risk values), it can easily trigger false alarms in the system, increasing operation and maintenance costs. Summary of the Invention

[0008] The present invention aims to solve the above-mentioned technical problems and provide a method that can adaptively select a completion strategy based on missing features and evaluate the credibility of the completion results.

[0009] This invention provides an adaptive completion and quality assessment method for missing data in urban infrastructure, comprising the following steps: Step S1: Acquire time-series data of urban infrastructure monitoring, and identify the type of missing data based on the time length and distribution characteristics of the missing window; the types of missing data include at least short-term random missing data, long-term continuous missing data, and static attribute missing data; Step S2: According to the type of missing data, adaptively match the corresponding completion strategy to fill the missing data, obtaining primary completed data; Step S3: Based on the measured values ​​of spatially adjacent sensors and historical data from the same period, use a confidence scoring model to assess the quality of the primary completed data and calculate the confidence score Q; Step S4: Compare the confidence score Q with a preset threshold, and perform graded processing on the primary completed data according to the comparison result; if the Q value is higher than the threshold, it is marked as valid data and stored in the database; if the Q value is lower than the threshold, it is marked as reference data and a verification prompt is generated.

[0010] Further, step S2 specifically includes: when a short-term random missing value is identified, an adaptive cubic spline interpolation algorithm is used to construct a smooth curve using normal data points before and after the missing point for fitting and completion; when a long-term continuous missing value is identified, historical periodic data is extracted, and the periodic term and trend term are separated using the STL decomposition algorithm, and the completion value is generated by combining the fluctuation coefficient at the current moment; when a static attribute is identified, upstream and downstream neighbor nodes are searched based on the pipeline topology relationship, and the missing attribute is deduced using the weighted voting method.

[0011] Furthermore, the confidence scoring model formula in step S3 is as follows: in, To complete the data for the primary stage, These are real-time measured values ​​from spatially adjacent sensors. To complete the Pearson correlation coefficient between the sequence and historical sequences from the same period, and These are the spatial consistency weight and the temporal continuity weight, respectively.

[0012] The beneficial effects of this invention are as follows:

[0013] 1. Improved data accuracy: It solves the problem of large errors in traditional single interpolation methods under long-term missing scenarios, and greatly improves the completion accuracy through an adaptive strategy.

[0014] 2. Reduced false alarm rate: The unique confidence scoring mechanism effectively filters out low-quality completed data, preventing false alarms caused by data error repair.

[0015] 3. Strong compatibility: A single architecture solves the problem of missing time-series data (IoT sensors) and static attribute data (ledgers). Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the method provided in the embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the adaptive strategy matching logic in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram illustrating the principle of the confidence scoring model in this embodiment of the invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: As Figure 1 As shown, this invention provides an adaptive completion and quality assessment method for missing data in urban infrastructure. This method runs on a server with a big data processing platform deployed, and the specific processing flow is as follows:

[0021] Step S1: Identification and Classification of Missing Data Features. The system receives real-time monitoring data from level gauges in the urban drainage network. A first threshold is set (e.g., 15 minutes, or 3 acquisition cycles). When data interruption is detected, the length of the continuous missing time window is calculated. . like If, within minutes, the system determines it to be a "short-term random missing" (label A) due to network congestion or packet loss; if If the system detects that the "pipe diameter" field in the accessed basic ledger data is empty, it determines that the data is "missing due to long-term continuous loss" (label B) because of equipment failure or maintenance. If the system detects that the "pipe diameter" field in the accessed basic ledger data is empty, it determines that the data is "missing due to static attribute loss" (label C).

[0022] Step S2: Adaptive algorithm matching based on missing type. The system automatically routes to the corresponding algorithm module according to the label generated in step S1: (1) For label A (short-term missing): Since the missing time is short, the pipeline liquid level usually does not change abruptly, so the system adopts the "adaptive cubic spline interpolation algorithm". The algorithm selects 3 effective data points before and after the missing time period as control points, constructs a cubic polynomial function, and calculates the smooth fitting value at the missing time. (2) For label B (long-term missing): Since the missing time is long, the interpolation method fails. The system extracts the historical data of the site over the past 30 days and uses the STL (Seasonal-Trend decomposition using Loess) algorithm to decompose the time series into seasonal terms (periodic fluctuations), trend terms, and residual terms. The system uses the seasonal terms and trend terms to predict the baseline value at the current time and fills it into the missing position. (3) For label C (attribute missing): The system searches for neighboring pipeline segments connected upstream and downstream of the pipeline segment based on the GIS topology network. If both upstream and downstream pipe sections are DN800, the missing attribute will be filled in directly as DN800; if the upstream and downstream sections are inconsistent, a weighted vote will be performed based on the distance.

[0023] Step S3: Construct a data quality confidence scoring model. After completion, the system uses the confidence scoring model to calculate the confidence score Q of the completion result. The formula is as follows: In this embodiment, considering the strong correlation between upstream and downstream liquid levels in the drainage pipe network scenario, spatial weights are set. Time weight Assume the algorithm calculates the completion value. The distance is 1.5 meters. The system reads the measured values ​​from the adjacent sensor 500 meters downstream. The value is 1.48 meters. Simultaneously, the system calculates the Pearson correlation coefficient between the completed sequence and historical sequences from the same period. The value is 0.95. Substitute this value into the formula to calculate: The final calculated confidence score Q is approximately 0.97.

[0024] Step S4: Graded processing. The system sets the confidence threshold to 0.8. Because the calculated confidence level in this embodiment... If the value (0.97) is greater than the threshold (0.8), the system marks the completed data as "high confidence valid data," stores it directly in the core production database, and allows the data to participate in subsequent liquid level over-limit alarm calculations. If, in another scenario, the calculated Q value is only 0.5, the system marks the data as "low confidence reference data," displays it as a gray dashed line on the front-end screen, does not trigger an automatic alarm, and automatically generates a maintenance work order, prompting maintenance personnel to go to the site to verify the equipment.

[0025] Those skilled in the art should understand that the above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications and substitutions made to the invention without departing from its spirit and scope should be included within the scope of the claims.

Claims

1. A method for adaptive completion and quality assessment of missing data for urban infrastructure, characterized in that, Includes the following steps: Step S1: Acquire time-series data of urban infrastructure monitoring, and identify the type of missing data based on the time length and distribution characteristics of the missing window; the types of missing data include at least short-term random missing data, long-term continuous missing data, and static attribute missing data; Step S2: According to the type of missing data, adaptively match the corresponding completion strategy to fill in the missing data to obtain primary completed data; Step S3: Based on the measured values ​​of spatially adjacent sensors and historical data from the same period, use a confidence scoring model to assess the quality of the primary completed data and calculate the confidence score Q; Step S4: Compare the confidence score Q with a preset threshold, and perform graded processing on the primary completed data according to the comparison result; if the Q value is higher than the threshold, it is marked as valid data and stored in the database; if the Q value is lower than the threshold, it is marked as reference data and a verification prompt is generated.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: when a short-term random missing value is identified, an adaptive cubic spline interpolation algorithm is used to construct a smooth curve using normal data points before and after the missing point for fitting and completion; when a long-term continuous missing value is identified, historical periodic data is extracted, and the periodic term and trend term are separated using the STL decomposition algorithm, and the completion value is generated by combining the fluctuation coefficient at the current moment; when a static attribute is identified, upstream and downstream neighbor nodes are searched based on the pipeline topology relationship, and the missing attribute is deduced using the weighted voting method.

3. The method according to claim 1, characterized in that, The confidence scoring model formula in step S3 is as follows: in, To complete the data for the primary stage, These are real-time measured values ​​from spatially adjacent sensors. To complete the Pearson correlation coefficient between the sequence and historical sequences from the same period, and These are the spatial consistency weight and the temporal continuity weight, respectively.

4. The method according to claim 1, characterized in that, In step S1, missing data with a continuous missing time less than a first threshold is determined to be short-term random missing data caused by network congestion or packet loss; the first threshold is set according to the sensor acquisition frequency.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

Images