A real-time quality control and processing method and device for acid rain automatic observation data
By using time-distance weighting and dynamic threshold adjustment, combined with physical consistency coupling coefficient, the problems of spatiotemporal alignment of multi-source data and adaptability of quality control thresholds in acid rain observation systems are solved, achieving high-accuracy and reliable quality control results.
Patent Information
- Application Number
- CN202611070910.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-11-03
AI Technical Summary
In existing acid rain monitoring systems, the inconsistent sampling frequency of sensors and clock sources lead to inconsistent data timing. Fixed quality control thresholds cannot adapt to sensor drift. Existing methods do not consider the physical correlation of multiple factors, which easily leads to false alarms and missed alarms.
Spatiotemporal alignment of multi-source data is achieved through time distance weight mapping, quality control thresholds are dynamically adjusted, multi-factor offsets are corrected by combining physical consistency coupling coefficients, and quality control results are quantified using comprehensive quality control confidence metrics.
It eliminates the quality control baseline error introduced by sensor time asynchrony, dynamically optimizes thresholds, reduces false alarm rate, and improves the reliability and accuracy of quality control results.
Smart Images
Figure CN123029178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of meteorological data quality inspection technology, specifically to a method and apparatus for real-time quality control and processing of automatic acid rain observation data. Background Technology
[0002] The automatic acid rain monitoring system collects observational data such as pH, conductivity, and temperature values from precipitation samples using pH, conductivity, and temperature sensors, and uploads this data to an information system via a communication network. Before uploading, the raw observation data undergoes basic quality control checks, including format checks, missing data checks, threshold range checks, sample contamination checks, and internal consistency checks, to ensure the validity and reliability of the uploaded data.
[0003] However, existing technologies still have shortcomings in the following aspects: pH sensors, conductivity sensors, and temperature sensors in acid rain monitoring systems have different sampling frequencies and independent clock sources, and the sampling times of each data source are inconsistent. Existing methods usually stitch the data together directly according to the receiving time, which leads to errors introduced by time deviations that affect the accuracy of quality control benchmarks; the detection thresholds of existing quality control methods mostly use fixed empirical values. As the sensors operate for a long time, the measurement characteristics will slowly drift, and fixed thresholds cannot adapt to dynamic changes; existing methods do not consider the inherent physical correlation between the various observation elements. When real acid rain events occur, pH, conductivity, and temperature elements exhibit a systematic coordinated shift that conforms to physical laws. Existing methods usually perform anomaly detection for each element independently, which is prone to false alarms in such physically expected coordinated shift scenarios.
[0004] Therefore, there is a need for a method and device for real-time quality control and processing of acid rain observation data that can achieve accurate spatiotemporal alignment of multi-source asynchronous data, adaptive adjustment of quality control thresholds, quantitative characterization of reliable quality control results, and coupling correction of physical consistency of multiple elements. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for real-time quality control and processing of automatic acid rain monitoring data, which addresses the shortcomings of the prior art.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for real-time quality control and processing of automatic acid rain monitoring data includes the following steps: Step S1: Obtain the original observation data of pH value, conductivity value and temperature value of acid rain, obtain the timestamp of each data source, calculate the time distance weight based on the time difference between the neighboring observation points of each data source and the target time, use the time distance weight to map each data source to a unified standard target time, generate a time-aligned standardized data message, and calculate the maximum time registration error. Step S2: For the three observation elements of pH value, conductivity value and temperature value in the standardized data message, calculate the mean and standard deviation of each element in the sliding window manner. Construct dynamic upper threshold and dynamic lower threshold based on the mean and standard deviation of each element and the adaptive threshold width coefficient. Calculate the single-element anomaly score of each element based on the dynamic upper threshold and dynamic lower threshold. Step S3: Based on the consistency of the deviation direction of the three observation elements relative to their respective sliding window mean and the overall level of the abnormal scores of each element, construct the physical consistency coupling coefficient. Multiply the weighted sum of the individual abnormal scores of each element by the physical consistency coupling coefficient to obtain the comprehensive abnormal score. Determine whether the current data packet is abnormal based on the comprehensive abnormal score. Step S4: Calculate the basic quality control detection weighted pass rate, spatiotemporal alignment quality, historical anomaly rate factor, and continuous anomaly penalty factor respectively, and fuse them to obtain the comprehensive quality control confidence. Determine whether the current data packet quality control has passed based on the comparison between the comprehensive quality control confidence and the preset threshold. Step S5: Generate hierarchical log records based on the comprehensive quality control confidence level and anomaly judgment results. The hierarchical log records include three levels: information level, warning level, and error level.
[0007] Furthermore, step S1 specifically includes the following steps: Step S1.1: For any neighboring observation point of any data source, divide the absolute value of the difference between the target time and the timestamp of the observation point by the time scale parameter of the data source, take the negative exponent, and then divide it by the sum of the negative exponent values of all neighboring observation points of the data source to obtain the time distance weight of the observation point. Step S1.2: For the mapping value of each data source at the target time, first add the observed value of each neighboring observation point, the product of the first time derivative of the observation point and the time difference, and the product of the second time derivative of the observation point and half of the square of the time difference, then multiply by the corresponding time distance weight, and finally add the calculation results of all neighboring observation points to obtain the mapping value of each data source at the target time. Step S1.3: After each data source completes the mapping, the maximum absolute value of the time difference between the nearest neighbor point of each data source and the target time is taken as the maximum time registration error.
[0008] Furthermore, step S2 specifically includes the following steps: Step S2.1: At any given moment, take a preset number of data points forward from the current moment, and calculate the mean and standard deviation of the sliding window for each of the three elements: pH value, conductivity value, and temperature value. Step S2.2: Calculate the dynamic upper threshold of each element, where the dynamic upper threshold is equal to the sliding window mean plus the product of the threshold width coefficient and the sliding window standard deviation; calculate the dynamic lower threshold of each element, where the dynamic lower threshold is equal to the sliding window mean minus the product of the threshold width coefficient and the sliding window standard deviation. Step S2.3: Update the threshold width coefficient. The threshold width coefficient at the current moment is equal to the threshold width coefficient at the previous moment plus the adjustment step size multiplied by the difference between the historical anomaly rate and the target anomaly rate. The result is then limited to between a preset minimum and maximum value. The historical anomaly rate is the ratio of the number of times an element was judged as an anomaly within a preset number of frames in the past to the total number of frames. Step S2.4: For the single-element anomaly score of each element at the stated time, first add the portion of the element's observed value that is higher than the dynamic upper threshold to the portion that is lower than the dynamic lower threshold, then divide by the difference between the dynamic upper threshold and the dynamic lower threshold, and add a positive number to prevent division by zero to obtain the single-element anomaly score of each element at the stated time.
[0009] Furthermore, step S3 specifically includes the following steps: Step S3.1: Calculate the deviation direction of the three observed elements, pH value, conductivity value and temperature value, relative to their respective sliding window mean values. When the observed value is greater than the sliding window mean value, the deviation direction value is 1; when it is less than the sliding window mean value, the value is -1; otherwise, the value is 0. Step S3.2: Take the minimum abnormal score of a single element among the three elements as the minimum abnormal score, and divide the absolute value of the sum of the deviation directions of the three elements by three as the deviation direction consistency measure. Step S3.3: Calculate the physical consistency coupling coefficient, which is equal to one minus the decoupling adjustment coefficient multiplied by the deviation direction consistency metric and then multiplied by the negative exponential decay value of the minimum anomaly score; Step S3.4: Calculate the comprehensive anomaly score. The comprehensive anomaly score is equal to the sum of the products of the physical consistency coupling coefficient and the individual anomaly scores of each element and their corresponding weights. When the comprehensive anomaly score is greater than or equal to the preset anomaly threshold, the current data packet is determined to be abnormal; otherwise, it is determined to be normal.
[0010] Further, in step S4, the weighted pass rate of the basic quality control test is calculated as follows: obtain the results of the six basic quality control tests: format check, missing test, pH threshold range check, conductivity threshold range check, sample contamination check, and internal consistency check. Each test is assigned a value of 1 when it passes and a value of 0 when it fails. Each test result is multiplied by its corresponding reliability weight and then summed. The sum is then compared with a preset positive number, and the larger value is taken to obtain the weighted pass rate of the basic quality control test. The sum of the reliability weights is 1. The spatiotemporal alignment quality is calculated as follows: the ratio of the maximum time registration error to the maximum permissible registration deviation is calculated, and the ratio is used as the independent variable to take the negative exponential function value to obtain the spatiotemporal alignment quality; The historical anomaly rate factor is calculated as follows: count the number of data frames that were judged to be abnormal within a preset number of frames in the past, divide the number of abnormal frames by the total number of frames to obtain the historical anomaly rate, and subtract the historical anomaly rate from 1 to obtain the historical anomaly rate factor. The continuous anomaly penalty factor is calculated as follows: obtain the number of data frames that have been continuously judged as abnormal before the current time, subtract the preset single penalty coefficient from 1, and then raise the result to the power of the number of consecutive abnormal frames to obtain the continuous anomaly penalty factor.
[0011] Further, in step S5, the generation method of the hierarchical log record is as follows: when the comprehensive quality control confidence level is greater than or equal to 0.8 and the current data packet is determined to be normal, an information-level log is generated; when the comprehensive quality control confidence level is greater than or equal to 0.5 and less than 0.8, a warning-level log is generated; when the comprehensive quality control confidence level is greater than or equal to 0.8 and the current data packet is determined to be abnormal, a warning-level log is generated; when the comprehensive quality control confidence level is less than 0.5, an error-level log is generated.
[0012] A real-time quality control and processing device for automatic acid rain monitoring data, used to implement any one of the methods for real-time quality control and processing of automatic acid rain monitoring data, comprising: The data acquisition module, including a pH sensor, a conductivity sensor, and a temperature sensor, is used to receive the raw observation data output by each sensor and obtain the timestamps of each data source. The chip module, connected to the data acquisition module, is used to receive the raw observation data sent by the data acquisition module, perform spatiotemporal alignment, dynamic threshold anomaly detection, physical consistency coupling correction, comprehensive quality control confidence assessment and hierarchical log recording on the raw observation data, and output standardized data messages with confidence labels. The communication module, connected to the chip module, is used to send standardized data messages with confidence tags output by the chip module to the receiving platform.
[0013] Furthermore, the chip module includes a module body and a module shielding cover disposed on the outside of the module body. The module body is provided with a memory, a clock unit, an intelligent processing module, a power supply unit, and peripheral interfaces. The module shielding cover is used to reduce electromagnetic interference and identify module information.
[0014] Furthermore, the intelligent processing module includes a data acquisition and protocol parsing unit, a dynamic threshold anomaly detection unit, a physical consistency coupling correction unit, a confidence assessment unit, and a hierarchical log recording unit.
[0015] Furthermore, the chip module supports two data output modes: active transmission and passive reading. The default mode is active transmission, in which the chip module listens to and receives raw observation data through the serial port, performs quality control processing, and then actively sends standardized data packets with confidence labels to the receiving platform. In passive reading mode, the receiving platform reads the most recent data frame in the memory by sending a read command.
[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves spatiotemporal alignment of multi-source data through asynchronous timestamp weighted mapping, incorporates first-order and second-order time derivatives into the mapping calculation, eliminates the quality control benchmark error introduced by the asynchronous time of sensors, and outputs the maximum time registration error as a quantitative indicator of alignment quality.
[0017] 2. This invention adaptively adjusts the threshold width coefficient based on closed-loop feedback of historical anomaly rate, enabling the dynamic threshold of each element to be dynamically optimized in response to sensor drift and environmental changes, thus solving the problem of false alarms and missed alarms caused by fixed thresholds.
[0018] 3. This invention corrects the multi-element collaborative offset by using a physical consistency coupling coefficient, reducing the anomaly score when all elements of an acid rain event deviate significantly and synchronously, thus avoiding false alarms; and maintaining sensitivity when a single element is abnormal in violation of physical laws, thereby optimizing the false alarm rate and the missed alarm rate.
[0019] 4. This invention achieves continuous quantitative characterization of the credibility of quality control judgments through a confidence index fused from multi-dimensional geometric averages, providing differentiated data quality labels for downstream systems. Attached Figure Description
[0020] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an embodiment of the present invention; Figure 2 This is a schematic diagram of the device structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the chip module dimensions according to an embodiment of the present invention; Figure 4 This is a graph showing pH observation data of constant value anomalies in an embodiment of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0022] like Figure 1As shown, a method for real-time quality control and processing of automatic acid rain monitoring data includes the following steps: Step S1: Obtain the original observation data of pH value, conductivity value and temperature value of acid rain, obtain the timestamp of each data source, calculate the time distance weight based on the time difference between the neighboring observation points of each data source and the target time, use the time distance weight to map each data source to a unified standard target time, generate a time-aligned standardized data message, and calculate the maximum time registration error. Step S2: For the three observation elements of pH value, conductivity value and temperature value in the standardized data message, calculate the mean and standard deviation of each element in the sliding window manner. Construct dynamic upper threshold and dynamic lower threshold based on the mean and standard deviation of each element and the adaptive threshold width coefficient. Calculate the single-element anomaly score of each element based on the dynamic upper threshold and dynamic lower threshold. Step S3: Based on the consistency of the deviation direction of the three observation elements relative to their respective sliding window mean and the overall level of the abnormal scores of each element, construct the physical consistency coupling coefficient. Multiply the weighted sum of the individual abnormal scores of each element by the physical consistency coupling coefficient to obtain the comprehensive abnormal score. Determine whether the current data packet is abnormal based on the comprehensive abnormal score. Step S4: Calculate the basic quality control detection weighted pass rate, spatiotemporal alignment quality, historical anomaly rate factor, and continuous anomaly penalty factor respectively, and fuse them to obtain the comprehensive quality control confidence. Determine whether the current data packet quality control has passed based on the comparison between the comprehensive quality control confidence and the preset threshold. Step S5: Generate hierarchical log records based on the comprehensive quality control confidence level and anomaly judgment results. The hierarchical log records include three levels: information level, warning level, and error level.
[0023] The pH sensor, conductivity sensor, and temperature sensor in the acid rain monitoring system have different sampling frequencies and independent clock sources, resulting in inconsistent sampling times for each data source. In order to align the multi-source data on a unified time base, each data source needs to be mapped to the same standard target time. For any neighboring observation point from any data source, the absolute value of the difference between the target time and the timestamp of the observation point is divided by the time scale parameter of the data source, and then the negative exponent is taken. This negative exponent is then divided by the sum of the negative exponent values of all neighboring observation points from the data source to obtain the temporal distance weight of the observation point. The formula for calculating the temporal distance weight is as follows: in, This represents the temporal distance weight of the k-th neighboring observation point of the i-th data source. This represents the i-th data source, where i=1 represents the pH data source, i=2 represents the conductivity data source, and i=3 represents the temperature data source. Indicates the standard target time. This represents the k-th nearest neighbor observation point. This represents the timestamp of the k-th nearest neighbor observation point of the i-th data source. This represents the time scale parameter for the i-th data source, used to control the rate at which the weights decay with distance. It is set according to the sampling frequency characteristics of each sensor, and its value ranges from 2 seconds to 10 seconds. This indicates the number of neighboring observation points, with a value ranging from 4 to 6. Represents the summation index variable, and They have the same meaning; they are all the serial numbers of nearby observation points, used to traverse all nearby observation points for normalization. After obtaining the time distance weights of each neighboring observation point, for the mapping value of each data source at the target time, first add the observed value of each neighboring observation point, the product of the first time derivative of the observation point and the time difference, and half of the square of the second time derivative of the observation point and the time difference, then multiply by the corresponding time distance weight, and finally add the calculation results of all neighboring observation points to obtain the mapping value of each data source at the target time. The calculation formula is as follows: in, This indicates that the i-th data source is mapped to the target time. The estimated value, This represents the observation value of the k-th neighboring observation point of the i-th data source. This represents the first-order time derivative of the i-th data source at its k-th neighbor. This represents the second time derivative of the i-th data source at its k-th neighbor. After each data source completes the mapping, the maximum absolute value of the time difference between the nearest neighbor point of each data source and the target time is taken as the maximum time registration error. The maximum time registration error is used to characterize the quality of spatiotemporal alignment. The larger the value, the less reliable the alignment.
[0024] A constant value anomaly detection step is also set before the dynamic threshold anomaly detection step: In actual observations, when the pipeline is blocked or the electrode performance is abnormal, the sensor's output value will remain unchanged for a long time or fluctuate only within a very small range. This manifests as the variance of the data sequence approaching zero, rather than exhibiting normal fluctuations with precipitation events. Figure 4 As shown, the pH value at a certain station remained stable between 7.01 and 7.24 over several consecutive precipitation sampling days, with extremely small fluctuations. This is clearly inconsistent with the physical law that the pH value of acid rain fluctuates normally with precipitation events, suggesting that the sensor output is rigid or the sampling pipeline is abnormal. Therefore, it is necessary to obtain the pH value sequence from the valid data packets acquired from multiple consecutive precipitation sampling days prior to the current moment, calculate the range of the sequence, i.e., the difference between the maximum and minimum values, and determine that the current data has a constant value anomaly when the range is less than the preset minimum fluctuation threshold. When the pH value is determined to be a constant value anomaly, the current data packet is directly marked as an anomaly, and the subsequent dynamic threshold anomaly detection and physical consistency coupling correction are no longer performed. The process is directly transferred to step S5 to generate the corresponding error level log. The number of consecutive precipitation sampling days is 20, and the minimum fluctuation threshold corresponding to the pH value is 0.25.
[0025] For the standardized data messages after spatiotemporal alignment and constant value anomaly detection, dynamic threshold anomaly detection is performed on the three observation elements of pH value, conductivity value and temperature value respectively. At any given moment, take a preset number of data points backward from the current moment, and calculate the sliding window mean and sliding window standard deviation of the three elements: pH value, conductivity value, and temperature value. The window length is 20 frames, corresponding to approximately 2 minutes of data. The dynamic upper threshold for each element is the product of the sliding window mean plus the threshold width coefficient and the sliding window standard deviation; the dynamic lower threshold for each element is the product of the sliding window mean minus the threshold width coefficient and the sliding window standard deviation. The specific formula is as follows: in, Let j represent the j-th element, where j=1 represents pH, j=2 represents conductivity, and j=3 represents temperature. Indicates the current moment. This represents the dynamic threshold of the j-th element at time t. This represents the threshold value of the j-th element under dynamic conditions at time t. This represents the sliding window mean of the j-th element at time t. This represents the sliding window standard deviation of the j-th element at time t. This represents the threshold width coefficient of the j-th element at time t, used to control the multiple of the threshold relative to the standard deviation; The threshold width coefficient is updated. The current threshold width coefficient is equal to the previous threshold width coefficient plus the adjustment step size multiplied by the difference between the historical anomaly rate and the target anomaly rate. The result is then limited to a preset minimum and maximum value. The historical anomaly rate is the ratio of the number of times an element was judged as an anomaly within a preset number of frames to the total number of frames. The specific formula is as follows: in, This represents the updated threshold width coefficient at the current time. This represents the threshold width coefficient at the previous time step. The initial time step uses a preset baseline value of 3.0. This indicates that the step size coefficient is adjusted, with a value of 0.1. This indicates the length of the historical statistics window, with a value of 50 frames. This represents the summation index variable. This represents the anomaly indicator function for the j-th element at time q. It is set to 1 if the observed value of the j-th element at that time exceeds the upper or lower threshold, and 0 otherwise. This represents the preset target anomaly rate, with a value of 0.02. This represents a thresholding function used to limit the threshold width coefficient between an upper and lower limit value. This represents the lower limit value, which is 1.5. This represents the upper limit value, which is 6.0. For the individual anomaly score of each element at the stated time, first add the portion of the element's observed value that is above the dynamic upper threshold to the portion that is below the dynamic lower threshold, then divide by the difference between the dynamic upper and lower thresholds and add a positive number to prevent division by zero to obtain the individual anomaly score of each element at the stated time. The specific formula is as follows: in, This represents the single-factor anomaly score of the j-th element at time t, and its value is a non-negative real number. This represents the real-time observation value of the j-th element at time t. It is represented as a very small positive number, with a value of 0.000001, and is used to prevent the denominator from being zero.
[0026] In real acid rain events, the pH value decreases, the conductivity value increases, and the temperature changes accordingly. These three observation elements will exhibit a systematic and coordinated shift that conforms to physical laws. This invention designs a physical consistency coupling coefficient to correct the coordinated shift of multiple elements. Calculate the deviation direction of the three observed parameters (pH, conductivity, and temperature) relative to their respective sliding window mean values. A positive deviation is assigned when the observed value is greater than the sliding window mean, a negative deviation when it is less than the mean, and a value of 0 otherwise. The specific formula is as follows: in, This indicates the normalization deviation direction of the j-th element at time t. This represents a sign function, which takes the value 1 when the independent variable is greater than 0, -1 when it is less than 0, and 0 otherwise; The minimum anomaly score for a single factor is taken as the minimum anomaly score. The absolute value of the sum of the deviations of the three factors, divided by 3, is used as the consistency measure of deviation direction. The specific formula is as follows: in, Indicates the minimum anomaly score. These represent the individual anomaly scores for pH, conductivity, and temperature at time t, respectively. This represents a measure of consistency in deviation direction. A value of 1 indicates that the three elements deviate in the same direction, while a value less than 1 indicates that they deviate in different directions. These represent the normalized deviation directions of the three factors, pH, conductivity, and temperature, at time t, respectively. The physical consistency coupling coefficient is equal to one minus the decoupling adjustment coefficient multiplied by the deviation direction consistency measure, and then multiplied by the negative exponential decay value of the minimum anomaly score. The specific formula is as follows: in, Represents the physical consistency coupling coefficient at time t. This represents the coupling adjustment coefficient, with a value of 0.4. This represents the sensitivity coefficient, with a value of 5.0. The overall anomaly score is equal to the sum of the products of the physical consistency coupling coefficient and the individual anomaly scores of each element and their corresponding weights. The specific formula is as follows: in, This represents the overall anomaly score at time t. This represents the weight coefficient of the j-th element. The sum of the weight coefficients is 1. Typically, the weights of pH, conductivity, and temperature are 0.4, 0.4, and 0.2, respectively. When the comprehensive anomaly score is greater than or equal to the preset anomaly threshold, the current data message is determined to be abnormal; otherwise, it is determined to be normal. The preset anomaly threshold is usually set to 0.5.
[0027] The overall quality control confidence score is a dimensionless real number used to quantify the reliability of a current data packet passing quality control. It is derived from the fusion of four dimensions: the weighted pass rate of basic quality control checks, spatiotemporal alignment quality, historical anomaly rate factor, and continuous anomaly penalty factor. Because of the bottleneck effect among these dimensions—a low quality in any one dimension will severely impact the overall reliability—a geometric mean is used instead of an arithmetic mean for fusion. This ensures that a low confidence score is significantly lowered if any factor is low. The formula for calculating the overall quality control confidence score is: in, This represents the overall quality control confidence level at time t. This indicates the sequence number of the basic quality control test item. p=1 to 6 represent format check, missing test, pH threshold range check, conductivity threshold range check, sample contamination check, and internal consistency check, respectively. This represents the reliability weight of the p-th basic quality control test item. The higher the reliability of the test item, the greater the weight. The weights of each item are obtained through experimental calibration. This represents the result of the basic quality control test of the p-th item at time t, with a value of 1 for passing and 0 for failing. This represents a very small positive number, with a value of 0.000001. It is used to add a very small cutoff value to the basic quality control pass rate, so that the penalty factor can still take effect. This indicates the maximum time registration error. This indicates the preset maximum allowable registration deviation, with a value of 10 seconds. This represents the alignment quality attenuation coefficient, dimensionless with a value of 2.0. This represents the number of data frames whose overall anomaly score within the past L frames is greater than or equal to a preset anomaly threshold. This indicates the length of the historical statistics window, with a value of 100 frames. This indicates the number of consecutive data frames that were judged as abnormal before the current frame. This represents the single-instance penalty coefficient, with a value of 0.2. These represent the exponential weights of each factor, all of which are dimensionless positive real numbers that sum to 1. They are typically 0.40, 0.15, 0.25, and 0.20, respectively. The first factor is the weighted pass rate of basic quality control tests, which integrates the results of six basic quality control tests and reflects the compliance of data packets in terms of basic format and physical quantity. The second factor is the spatiotemporal alignment quality, which reflects the alignment reliability of multi-source data in the time dimension. When the maximum time registration error is 0, this factor equals 1, and it decreases exponentially as the maximum time registration error increases. The third factor is the historical anomaly rate factor, which reflects the recent stability of the system. When there are many recent abnormal frames, this factor decreases, indicating that the system may be in an unstable state, and the confidence of the current data should be appropriately lowered. The fourth factor is the continuous anomaly penalty factor, which applies a time-series penalty to consecutively occurring abnormal events. The more consecutive abnormal frames there are, the smaller this factor becomes, to prevent the confidence of the equipment from being continuously overestimated when there is a systemic failure.
[0028] The final judgment rule is: when When the current data packet quality control is passed, it is determined that the current data packet has passed; when When this happens, the current data packet is determined to have failed quality control. The preset confidence threshold is set to 0.6. The hierarchical log recording is generated as follows: when And when the current data packet is determined to be normal, an information-level log is generated; when and When, generate a warning-level log; when And when the current data packet is determined to be abnormal, a warning-level log is generated; when When this happens, an error-level log is generated; Information-level logs are used to record normal observation data with good data quality; warning-level logs are used to record data with moderate confidence that is questionable or data with high confidence but abnormal, indicating that manual verification is required; error-level logs are used to record data with extremely low confidence, indicating that the data is seriously unreliable and that the equipment and data link need to be investigated immediately.
[0029] like Figure 2 As shown, a real-time quality control and processing device for automatic acid rain monitoring data is used to implement any of the methods for real-time quality control and processing of automatic acid rain monitoring data, comprising: The data acquisition module, including a pH sensor, a conductivity sensor, and a temperature sensor, is used to receive the raw observation data output by each sensor and obtain the timestamps of each data source. The chip module, connected to the data acquisition module, is used to receive the raw observation data sent by the data acquisition module, perform spatiotemporal alignment, dynamic threshold anomaly detection, physical consistency coupling correction, comprehensive quality control confidence assessment and hierarchical log recording on the raw observation data, and output standardized data messages with confidence labels. The communication module, connected to the chip module, is used to send standardized data messages with confidence tags output by the chip module to the receiving platform.
[0030] The chip module includes a module body and a module shielding cover disposed on the outside of the module body. The module body is provided with a memory, a clock unit, an intelligent processing module, a power supply unit, and peripheral interfaces. The module shielding cover is used to reduce electromagnetic interference and identify module information. The peripheral interfaces include an RS232 serial port, a TTL serial port, an Ethernet interface, a USB interface, and a debugging interface. The chip module is connected to external devices through a PCIe interface. The module body is also provided with a power indicator light and a running indicator light. The power indicator light is constantly lit when the power supply is normal, and the running indicator light flashes during normal operation.
[0031] The intelligent processing module includes a data acquisition and protocol parsing unit, a dynamic threshold anomaly detection unit, a physical consistency coupling correction unit, a confidence assessment unit, and a hierarchical log recording unit. The data acquisition and protocol parsing unit receives raw observation data frames and parses them according to the communication protocol to generate standardized data messages. The dynamic threshold anomaly detection unit calculates the mean and standard deviation of each observation element using a sliding window method and adaptively adjusts the threshold width coefficient based on the historical anomaly rate to construct dynamic upper and lower thresholds, calculating the single-element anomaly score for each element. The physical consistency coupling correction unit is used to adjust the deviation of the three observation elements according to their respective deviations. A physical consistency coupling coefficient is constructed based on the consistency level and the minimum anomaly score. The weighted sum of the individual anomaly scores of each element is multiplied by the physical consistency coupling coefficient to obtain a comprehensive anomaly score and determine whether the data is anomaly. The confidence assessment unit is used to calculate the comprehensive quality control confidence based on the weighted pass rate of basic quality control detection, spatiotemporal alignment quality, historical anomaly rate, and continuous anomaly penalty. The quality control confidence is compared with a preset threshold to output the judgment result of whether the quality control has passed. The hierarchical log recording unit is used to generate log records at three levels: information level, warning level, and error level, based on the comprehensive quality control confidence and the anomaly judgment result.
[0032] The intelligent processing module adopts a heterogeneous computing architecture of a central processing unit (CPU) and a digital signal processor (DSP). The CPU is a dual-core ARM Cortex-A7 processor used for system management, communication protocol parsing and framing, and data scheduling. The DSP is a single-core HiFi4 DSP used for spatiotemporal alignment calculation, real-time updating of sliding window statistical features, iterative calculation of dynamic thresholds, solving for physical consistency coupling coefficients, and parallel calculation of various factors of comprehensive confidence.
[0033] A communication module, connected to the chip module, is used to send standardized data packets with confidence labels output by the chip module to the receiving platform.
[0034] The chip module supports two data output modes: active transmission and passive reading. The default mode is active transmission, in which the chip module listens to and receives raw observation data through the serial port, performs quality control processing, and then actively sends standardized data packets with confidence labels to the receiving platform. In passive reading mode, the receiving platform reads the most recent data frame in the memory by sending a read command.
[0035] like Figure 3 As shown, the chip module has the following technical characteristics: With its small size, ease of installation, and powerful computing performance, this edge computing node can significantly enhance the performance of both cloud and station-side devices. Its dimensions are 53.15m in length, 40.02mm in width, and 6mm in thickness. The module features a high-precision clock circuit, memory, I / O interfaces (2 DI interfaces, 2 DO interfaces, 2 TTL serial ports, 2 RS232 serial ports, 1 Ethernet interface, 2 USB interfaces), a debugging interface, monitoring circuitry, a power supply interface, and indicator lights. The module uses a PCIe interface for easy connection to various devices.
[0036] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.
Claims
1. A method for real-time quality control and processing of automatic acid rain monitoring data, characterized in that, Includes the following steps: Step S1: Obtain the original observation data of pH value, conductivity value and temperature value of acid rain, obtain the timestamp of each data source, calculate the time distance weight based on the time difference between the neighboring observation points of each data source and the target time, use the time distance weight to map each data source to a unified standard target time, generate a time-aligned standardized data message, and calculate the maximum time registration error. Step S2: For the three observation elements of pH value, conductivity value and temperature value in the standardized data message, calculate the mean and standard deviation of each element in the sliding window manner. Construct dynamic upper threshold and dynamic lower threshold based on the mean and standard deviation of each element and the adaptive threshold width coefficient. Calculate the single-element anomaly score of each element based on the dynamic upper threshold and dynamic lower threshold. Step S3: Based on the consistency of the deviation direction of the three observation elements relative to their respective sliding window mean and the overall level of the abnormal scores of each element, construct the physical consistency coupling coefficient. Multiply the weighted sum of the individual abnormal scores of each element by the physical consistency coupling coefficient to obtain the comprehensive abnormal score. Determine whether the current data packet is abnormal based on the comprehensive abnormal score. Step S4: Calculate the basic quality control detection weighted pass rate, spatiotemporal alignment quality, historical anomaly rate factor, and continuous anomaly penalty factor respectively, and fuse them to obtain the comprehensive quality control confidence. Determine whether the current data packet quality control has passed based on the comparison between the comprehensive quality control confidence and the preset threshold. Step S5: Generate hierarchical log records based on the comprehensive quality control confidence level and anomaly judgment results. The hierarchical log records include three levels: information level, warning level, and error level.
2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S1.1: For any neighboring observation point of any data source, divide the absolute value of the difference between the target time and the timestamp of the observation point by the time scale parameter of the data source, take the negative exponent, and then divide it by the sum of the negative exponent values of all neighboring observation points of the data source to obtain the time distance weight of the observation point. Step S1.2: For the mapping value of each data source at the target time, first add the observed value of each neighboring observation point, the product of the first time derivative of the observation point and the time difference, and the product of the second time derivative of the observation point and half of the square of the time difference, then multiply by the corresponding time distance weight, and finally add the calculation results of all neighboring observation points to obtain the mapping value of each data source at the target time. Step S1.3: After each data source completes the mapping, the maximum absolute value of the time difference between the nearest neighbor point of each data source and the target time is taken as the maximum time registration error.
3. The method according to claim 2, characterized in that, Step S2 specifically includes the following steps: Step S2.1: At any given moment, take a preset number of data points forward from the current moment, and calculate the mean and standard deviation of the sliding window for each of the three elements: pH value, conductivity value, and temperature value. Step S2.2: Calculate the dynamic upper threshold of each element, where the dynamic upper threshold is equal to the sliding window mean plus the product of the threshold width coefficient and the sliding window standard deviation; calculate the dynamic lower threshold of each element, where the dynamic lower threshold is equal to the sliding window mean minus the product of the threshold width coefficient and the sliding window standard deviation. Step S2.3: Update the threshold width coefficient. The threshold width coefficient at the current moment is equal to the threshold width coefficient at the previous moment plus the adjustment step size multiplied by the difference between the historical anomaly rate and the target anomaly rate. The result is then limited to between a preset minimum and maximum value. The historical anomaly rate is the ratio of the number of times an element was judged as an anomaly within a preset number of frames in the past to the total number of frames. Step S2.4: For the single-element anomaly score of each element at the stated time, first add the portion of the element's observed value that is higher than the dynamic upper threshold to the portion that is lower than the dynamic lower threshold, then divide by the difference between the dynamic upper threshold and the dynamic lower threshold, and add a positive number to prevent division by zero to obtain the single-element anomaly score of each element at the stated time.
4. The method according to claim 3, characterized in that, Step S3 specifically includes the following steps: Step S3.1: Calculate the deviation direction of the three observed elements, pH value, conductivity value and temperature value, relative to their respective sliding window mean values. When the observed value is greater than the sliding window mean value, the deviation direction value is 1; when it is less than the sliding window mean value, the value is -1; otherwise, the value is 0. Step S3.2: Take the minimum abnormal score of a single element among the three elements as the minimum abnormal score, and divide the absolute value of the sum of the deviation directions of the three elements by three as the deviation direction consistency measure. Step S3.3: Calculate the physical consistency coupling coefficient, which is equal to one minus the decoupling adjustment coefficient multiplied by the deviation direction consistency metric and then multiplied by the negative exponential decay value of the minimum anomaly score; Step S3.4: Calculate the comprehensive anomaly score. The comprehensive anomaly score is equal to the sum of the products of the physical consistency coupling coefficient and the individual anomaly scores of each element and their corresponding weights. When the comprehensive anomaly score is greater than or equal to the preset anomaly threshold, the current data packet is determined to be abnormal; otherwise, it is determined to be normal.
5. The method according to claim 4, characterized in that, In step S4, the weighted pass rate of the basic quality control test is calculated as follows: obtain the results of the six basic quality control tests: format check, missing test, pH threshold range check, conductivity threshold range check, sample contamination check, and internal consistency check. Each test is assigned a value of 1 when it passes and a value of 0 when it fails. Multiply each test result by its corresponding reliability weight and sum them up. Then, take the larger value of the sum and a preset positive number to obtain the weighted pass rate of the basic quality control test. The sum of the reliability weights is 1. The spatiotemporal alignment quality is calculated as follows: the ratio of the maximum time registration error to the maximum permissible registration deviation is calculated, and the ratio is used as the independent variable to take the negative exponential function value to obtain the spatiotemporal alignment quality; The historical anomaly rate factor is calculated as follows: count the number of data frames that were judged to be abnormal within a preset number of frames in the past, divide the number of abnormal frames by the total number of frames to obtain the historical anomaly rate, and subtract the historical anomaly rate from 1 to obtain the historical anomaly rate factor. The continuous anomaly penalty factor is calculated as follows: obtain the number of data frames that have been continuously judged as abnormal before the current time, subtract the preset single penalty coefficient from 1, and then raise the result to the power of the number of consecutive abnormal frames to obtain the continuous anomaly penalty factor.
6. The method according to claim 5, characterized in that, In step S5, the generation method of the hierarchical log record is as follows: when the comprehensive quality control confidence level is greater than or equal to 0.8 and the current data packet is determined to be normal, an information-level log is generated; when the comprehensive quality control confidence level is greater than or equal to 0.5 and less than 0.8, a warning-level log is generated; when the comprehensive quality control confidence level is greater than or equal to 0.8 and the current data packet is determined to be abnormal, a warning-level log is generated; when the comprehensive quality control confidence level is less than 0.5, an error-level log is generated.
7. A real-time quality control and processing device for automatic acid rain monitoring data, used to implement the real-time quality control and processing method for automatic acid rain monitoring data as described in any one of claims 1-6, characterized in that, include: The data acquisition module, including a pH sensor, a conductivity sensor, and a temperature sensor, is used to receive the raw observation data output by each sensor and obtain the timestamps of each data source. The chip module, connected to the data acquisition module, is used to receive the raw observation data sent by the data acquisition module, perform spatiotemporal alignment, dynamic threshold anomaly detection, physical consistency coupling correction, comprehensive quality control confidence assessment and hierarchical log recording on the raw observation data, and output standardized data messages with confidence labels. The communication module, connected to the chip module, is used to send standardized data messages with confidence tags output by the chip module to the receiving platform.
8. The apparatus according to claim 7, characterized in that, The chip module includes a module body and a module shielding cover disposed on the outside of the module body. The module body is provided with a memory, a clock unit, an intelligent processing module, a power supply unit and peripheral interfaces. The module shielding cover is used to reduce electromagnetic interference and identify module information.
9. The apparatus according to claim 8, characterized in that, The intelligent processing module includes a data acquisition and protocol parsing unit, a dynamic threshold anomaly detection unit, a physical consistency coupling correction unit, a confidence assessment unit, and a hierarchical log recording unit.
10. The apparatus according to claim 9, characterized in that, The chip module supports two data output modes: active transmission and passive reading. The default mode is active transmission, which means that the chip module listens to and receives the raw observation data through the serial port, performs quality control processing, and then actively sends the standardized data message with confidence label to the receiving platform. In passive reading mode, the receiving platform reads the most recent data frame in the memory by sending a read command.