Quality control methods, devices, equipment, media, and procedures for meteorological observation data
By combining temporal consistency detection, spatial consistency detection, and reliability scoring into a quality control method, and dynamically adjusting the threshold, the problems of misjudgment and omission in the quality control of meteorological observation data in existing technologies are solved, achieving efficient and accurate data screening and reliability assurance.
Patent Information
- Application Number
- CN202611058190.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-07-16
AI Technical Summary
Existing meteorological observation data quality control methods rely on fixed thresholds or single rules, which cannot adapt to the data characteristics of different devices and time periods, leading to misjudgments or missed anomalies. Furthermore, they rely heavily on manual review, resulting in low efficiency.
We employ temporal consistency detection and spatial consistency detection combined with credibility scoring, dynamically adjust thresholds, identify anomalous data by comparing Pearson correlation coefficient with reanalysis data, and introduce multidimensional visualization to assist in verification.
It improves the accuracy and flexibility of meteorological observation data quality control, reduces the workload of manual review, ensures the stability and reliability of data, adapts to the historical performance of different equipment, and improves the timeliness and accuracy of quality control.
Smart Images

Figure CN122573282B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of meteorological quality assessment, specifically to a method, apparatus, equipment, medium, and program product for quality control of meteorological observation data. Background Technology
[0002] Based on meteorological data observations, operational planning is crucial in many fields, such as transportation and renewable energy power plants. For example, wind power and photovoltaic power plants typically deploy numerous high-frequency meteorological observation devices to monitor key meteorological parameters such as wind speed, wind direction, temperature, and solar radiation in real time. This data is not only essential for wind turbine power prediction, energy efficiency optimization, and safe operation, but also forms the basis for scientific management and decision-making at renewable energy power plants. However, due to the complex and variable meteorological environment, coupled with the potential for sensor aging, calibration deviations, or communication failures during long-term operation, observational data often suffers from incomplete measurements, abnormal fluctuations, or system drift. Therefore, quality control (QC) is necessary for the meteorological observation equipment used in the data collection.
[0003] Currently, quality control methods for meteorological observation data mostly rely on fixed thresholds or single rules, such as simple upper and lower limit elimination or logical relationship checks. These methods have significant limitations when dealing with high-frequency data: fixed thresholds cannot reflect the data characteristics of different devices or different time periods, which can easily lead to misjudgments or missed anomalies; single-dimensional rules are difficult to identify local anomalies or systematic drift; at the same time, these methods rely heavily on manual review, which is labor-intensive and inefficient. Summary of the Invention
[0004] In view of the above problems, embodiments of this application provide a method, apparatus, equipment, medium and program product for quality control of meteorological observation data.
[0005] A first aspect of this application provides a quality control method for meteorological observation data. The method includes: acquiring meteorological data collected by multiple observation devices in a target area to obtain a first dataset; acquiring a reliability score for each observation device; and performing classification quality control on the first dataset to divide it into three categories: reliable data, questionable data, and erroneous data. The classification quality control includes at least one of temporal consistency detection and spatial consistency detection. The temporal consistency detection includes: calculating a first mean of multiple first data collected by any first observation device within a sliding time window and a first deviation of each first data point from the first mean; classifying first data points with a first deviation greater than a first threshold as erroneous data, classifying data points with a first deviation not greater than the first threshold but greater than a first intermediate threshold as questionable data, and classifying the rest as reliable data. At least one of the first threshold and the first intermediate threshold is obtained based on the reliability score of the first observation device, such that the width of a first interval with the first threshold and the first intermediate threshold as upper and lower limits is negatively correlated with the reliability score of the first observation device. The spatial consistency detection includes: calculating a second mean of multiple second data points collected by the multiple observation devices at the same time and a second deviation of each second data point from the second mean. The second data collected by any second observation device is classified as follows: when the second deviation is greater than the second threshold, the second data is determined to be erroneous data; when the second deviation is not greater than the second threshold but greater than the second intermediate threshold, the second data is determined to be questionable data; and when the second deviation is not greater than the second intermediate threshold, the second data is determined to be reliable data. At least one of the second threshold and the second intermediate threshold is obtained based on the reliability score of the second observation device, so that the width of the second interval, with the second threshold and the second intermediate threshold as the upper and lower limits, is negatively correlated with the reliability score of the second observation device.
[0006] According to this embodiment, whether it's temporal consistency detection or spatial consistency detection, for each observation device's data (excluding erroneous data), the higher the confidence score, the smaller the interval width used to filter suspicious data, and the more likely the data from that observation device is to be classified as reliable data, and vice versa. This provides a more accurate filtering method for data other than obvious errors.
[0007] According to an embodiment of this application, after classifying and quality-controlling the first dataset to divide it into three categories—credible data, questionable data, and erroneous data—the method further includes: taking the credible data or the union of credible data and questionable data in the first dataset as a second dataset; performing reanalysis consistency detection on the data of each observation device in the second dataset to obtain a reanalysis consistency detection result corresponding to each observation device; and removing the data of the observation device to be excluded when the reanalysis consistency detection result indicates the existence of an observation device to be excluded. The reanalysis consistency detection includes: extracting data from any third observation device from the second dataset to form an observation sample sequence; extracting similar meteorological element values from the reanalysis data that are at the same location as the third observation device and have the same data collection time as the data in the observation sample sequence to obtain a reanalysis data sequence; calculating the correlation coefficient between the observation sample sequence and the reanalysis data sequence; and obtaining the reanalysis consistency detection result corresponding to the third observation device based on the correlation coefficient, including: determining the third observation device as an observation device to be excluded when the correlation coefficient meets a preset exclusion condition.
[0008] According to this embodiment, in the reanalysis consistency test, numerical weather prediction models or reanalysis data are used as a reference. By comparing the observed data with the model data, significant deviations can be found, and single-point or systematic deviations can be identified, thereby enhancing the quality control accuracy.
[0009] According to an embodiment of this application, the correlation coefficient is the Pearson correlation coefficient, and the exclusion condition is that the correlation coefficient is less than a preset third threshold. A Pearson correlation coefficient closer to 1 indicates a good match between the observation and reanalysis data. Using the Pearson correlation coefficient, observation equipment with significant deviations from the reanalysis data can be quickly screened out, allowing for timely detection of abnormalities such as equipment aging, and excluding data from that equipment during quality control.
[0010] According to an embodiment of this application, the time-series consistency detection further includes: calculating a first standard deviation of multiple first data collected by the first observation device within the sliding time window; and setting the first threshold as the first standard deviation. times, of which The first intermediate threshold is a constant; and the first intermediate threshold is set to the first standard deviation. Multiply by a factor of the credibility score, wherein the credibility score is greater than or equal to zero and less than or equal to 1.
[0011] According to this embodiment, in setting the dynamic threshold for time-series consistency detection, the first intermediate threshold is associated with the credibility score. The first intermediate threshold is essentially a threshold for distinguishing between credible data and questionable data and measures the first deviation. Thus, the larger the first intermediate threshold, the larger the deviation in time sequence of the data collected by the device can be, which means that the higher the credibility score of the device, the more lenient the screening conditions for credible data.
[0012] According to an embodiment of this application, the spatial consistency detection further includes: calculating the absolute median difference of multiple second data collected by the multiple observation devices at the same time; obtaining a robust estimate of the second standard deviation based on the absolute median difference and the assumption that the data follows a normal distribution; and setting the second threshold as the second standard deviation for the second observation device. The second intermediate threshold is set to the second standard deviation. Multiply by a factor of the confidence score of the second observation device, wherein the confidence score is greater than or equal to zero and less than or equal to 1; wherein, This is a constant applicable to all of the aforementioned observation devices.
[0013] According to this embodiment, in setting the dynamic threshold for spatial consistency detection, the second intermediate threshold is associated with the credibility score. The second intermediate threshold is essentially a threshold for distinguishing between credible data and questionable data and measures the second deviation. Therefore, the larger the second intermediate threshold, the larger the allowable spatial deviation of the data collected by the device can be, which means that the higher the credibility score of the device, the more relaxed the screening conditions for credible data.
[0014] According to an embodiment of this application, the classification quality control of the first dataset further includes: taking the intersection of the reliable data divided by the temporal consistency detection and the reliable data divided by the spatial consistency detection as the final reliable data; taking the union of the erroneous data divided by the temporal consistency detection and the erroneous data divided by the spatial consistency detection as the final erroneous data; and classifying all data other than reliable data and erroneous data into suspicious data.
[0015] According to this embodiment, when both temporal consistency detection and spatial consistency detection are included, the erroneous data from both methods are taken as the final erroneous data, the reliable data from both methods are taken as the final reliable data, and the rest are questionable data. By combining the detection results of the two methods, data that is reliable in both spatial and temporal dimensions can be filtered out.
[0016] According to an embodiment of this application, the calculation process of the reliability score for each observation device includes: obtaining the data quality control record of the observation device; based on the data quality control record, obtaining the anomaly rate and error rate of the observation device, wherein the anomaly rate represents the proportion of the amount of abnormal data removed from the historical observation data of the observation device, and the error rate represents the error range of the historical observation data relative to the reference data when the reanalysis data corresponding to the historical observation data of the observation device at the same observation location and the same observation time is used as the reference data; and the anomaly rate and error rate of the observation device are normalized and then weighted to obtain the reliability score.
[0017] According to this embodiment, the credibility score is obtained by weighted normalization of anomaly rate and error, which can provide a dynamic threshold basis for subsequent consistency detection, enabling data from different devices to adaptively determine anomalies based on their own historical performance, thereby improving the flexibility and accuracy of quality control.
[0018] According to embodiments of this application, the calculation of the reliability score for each observation device further includes: re-acquiring the data quality control record of the observation device in response to each update of the data quality control record of the observation device; and recalculating the reliability score in response to re-acquiring the data quality control record of the observation device. In this way, the reliability score is continuously updated, and the data judgment of different devices can follow changes in their historical performance, improving the timeliness of quality control.
[0019] According to an embodiment of this application, after classifying and quality controlling the first dataset to divide it into three categories—credible data, questionable data, and erroneous data—the method further includes: extracting data from each observation device and questionable data from each observation device from the first dataset; calculating the proportion of questionable data from any third observation device in the total data of the third observation device to obtain the proportion of questionable data from the third observation device; if the proportion of questionable data from the third observation device does not exceed a fourth threshold, directly removing the questionable data from the third observation device; if the proportion of questionable data from the third observation device exceeds the fourth threshold, instructing manual review of the questionable data from the third observation device.
[0020] According to this embodiment, questionable data with minor impact can be effectively eliminated, ensuring the stability and reliability of the output data. Manual review is only triggered for high-risk devices with a high proportion of questionable data, significantly reducing the manpower cost of quality control.
[0021] According to an embodiment of this application, the instruction to manually review the questionable data of the third observation device includes: extracting reliable data of the third observation device from the first dataset; generating a multi-dimensional comparison visualization result based on the reliable data and the questionable data of the third observation device; and displaying the multi-dimensional comparison visualization result to the user.
[0022] According to this embodiment, multi-dimensional comparison visualization results are generated, enhancing visualization and interpretation capabilities, making data issues clear at a glance, and improving the intuitiveness of manual review and judgment.
[0023] A second aspect of this application provides a quality control device for meteorological observation data. The device includes a first acquisition module, a second acquisition module, and a classification quality control module. The first acquisition module acquires meteorological data collected by multiple observation devices in a target area to obtain a first dataset. The second acquisition module acquires a reliability score for each observation device. The classification quality control module performs classification quality control on the first dataset to divide it into three categories: reliable data, questionable data, and erroneous data. The classification quality control module includes at least one of a temporal consistency detection submodule and a spatial consistency detection submodule.
[0024] The temporal consistency detection submodule is used to: calculate the first mean of multiple first data collected by any first observation device within a sliding time window and the first deviation of each first data compared to the first mean; and classify the first data with the first deviation greater than the first threshold as erroneous data, classify the data with the first deviation not greater than the first threshold but greater than the first intermediate threshold as questionable data, and classify the rest as reliable data, wherein at least one of the first threshold and the first intermediate threshold is obtained based on the reliability score of the first observation device, so that the width of the first interval with the first threshold and the first intermediate threshold as the upper and lower limits is negatively correlated with the reliability score of the first observation device.
[0025] The spatial consistency detection submodule is used to: calculate the second mean of multiple second data collected by the multiple observation devices at the same time and the second deviation of each second data compared with the second mean; and classify the second data collected by any second observation device in the following manner: when the second deviation is greater than the second threshold, the second data is determined to be erroneous data; when the second deviation is not greater than the second threshold but greater than the second intermediate threshold, the second data is determined to be questionable data; and when the second deviation is not greater than the second intermediate threshold, the second data is determined to be reliable data, wherein at least one of the second threshold and the second intermediate threshold is obtained based on the reliability score of the second observation device, so that the width of the second interval with the second threshold and the second intermediate threshold as the upper and lower limits is negatively correlated with the reliability score of the second observation device.
[0026] When performing classification quality control according to this embodiment, whether it is temporal consistency detection or spatial consistency detection, the dividing criterion between reliable data and doubtful data collected by each observation device is related to the reliability score of the observation device. The higher the reliability score of the observation device, the smaller the interval width used to filter doubtful data. Data other than erroneous data in the observation device is more likely to be classified as reliable data, and vice versa. In this way, a more accurate filtering method is provided for data other than obvious errors, and it can ensure that the data collected by observation devices with high reliability has a higher probability of being adopted reliably.
[0027] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.
[0028] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.
[0029] According to a fifth aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description
[0030] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0031] Figure 1 The illustration schematically depicts an application scenario of a quality control method, apparatus, device, medium, and program product for meteorological observation data according to embodiments of this application.
[0032] Figure 2 A flowchart illustrating a method for quality control of meteorological observation data according to an embodiment of this application is shown schematically.
[0033] Figure 3 A flowchart illustrating a method for quality control of meteorological observation data according to another embodiment of this application is shown schematically.
[0034] Figure 4 Schematic illustration Figure 3 The flowchart of the reanalysis consistency test in the quality control method shown is as follows;
[0035] Figure 5This illustration schematically shows a flowchart of obtaining a reliability score for each observation device in a quality control method for meteorological observation data according to an embodiment of this application;
[0036] Figure 6 A flowchart illustrating a method for quality control of meteorological observation data according to yet another embodiment of this application is shown.
[0037] Figure 7 A flowchart illustrating a method for quality control of meteorological observation data according to another embodiment of this application is shown schematically.
[0038] Figure 8 A block diagram schematically illustrates a quality control device for meteorological observation data according to an embodiment of this application; and
[0039] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a quality control method for meteorological observation data according to embodiments of this application. Detailed Implementation
[0040] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.
[0041] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0042] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0043] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0044] The key terms used in this article are explained below:
[0045] Quality control (QC) refers to the systematic methods and procedures used to monitor, inspect, and correct raw data to ensure that it meets established standards or requirements and maintains reliability and consistency.
[0046] Threshold quality control involves detecting and removing obvious outliers from observation data based on meteorological physical laws and climate thresholds of various meteorological elements.
[0047] Temporal consistency detection verifies the continuity of data collected by an observation device based on the time series and detects short-term anomalies.
[0048] Spatial consistency detection involves cross-checking data collected by multiple observation devices in spatial proximity based on the relationship between the stations to detect local anomalies.
[0049] Credibility Score (CS) is a credibility index of equipment data calculated from historical data quality control records. It is used to quantify the historical data quality level of each observation device.
[0050] Dynamic threshold consistency classification quality control refers to the introduction of an adaptive threshold based on credibility score. By incorporating credibility score into the threshold settings in temporal consistency detection and / or spatial consistency detection, the observed data is divided into three categories: credible data, questionable data, and erroneous data.
[0051] Reanalysis is a standard term in the field of meteorology and climate science. Its core meaning is to use a fixed numerical forecasting model and data assimilation system to "reprocess" all historical observation data (ground, satellite, radiosonde, etc.) and stitch them together to generate a long-term dataset that is continuous in time, three-dimensionally gridded in space, and physically coordinated and unified.
[0052] Figure 1 The illustration schematically depicts an application scenario of a quality control method, apparatus, device, medium, and program product for meteorological observation data according to embodiments of this application. For example... Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a primary observation device 101, a secondary observation device 102, a third observation device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the primary observation device 101, the secondary observation device 102, the third observation device 103, and the server 105. The network 104 may be a wired or fiber optic cable, etc.
[0053] Observation device 101, observation device 102, and observation device 103 can be any type of meteorological observation equipment (such as a thermometer, anemometer, barometer, etc.). Observation device 101, observation device 102, and observation device 103 can transmit the collected data to server 105 via network 104.
[0054] Server 105 can execute the meteorological observation data quality control method of this application embodiment, and classify and control the data collected by observation device 101, observation device 102, and observation device 103. Accordingly, the meteorological observation data quality control device, equipment, medium, and program product provided in this application embodiment can generally be set in server 105. It should be understood that... Figure 1 The number of observation devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of observation devices, networks, and servers can be included.
[0055] In one embodiment, observation device 101, observation device 102, and observation device 103 can be deployed at a renewable energy power station. Renewable energy power stations have high requirements for the reliability and traceability of meteorological observation data. For example, in wind power and photovoltaic scenarios, observation data directly affects wind turbine power prediction, wind farm scheduling, and equipment operation and maintenance decisions. If abnormal data is not detected or removed in a timely manner, it may lead to power prediction errors, misjudgments in operation and maintenance, and even affect the safe operation of the power station. Therefore, there is an urgent need for a meteorological data quality control scheme that can automatically identify anomalies and dynamically adjust judgment criteria to better assist the operation of renewable energy power stations.
[0056] This application proposes a quality control method for meteorological observation data, which can automate the quality control of high-frequency meteorological observation data in scenarios such as new energy power stations. According to various embodiments of this application, accurate identification and processing of abnormal data can be achieved through reliability scoring of various observation devices, dynamic threshold classification, multi-dimensional consistency detection, and / or visual assisted verification. This improves data quality and reliability while reducing the workload of manual verification, providing strong support for the operation optimization and scientific management of new energy power stations. It should be noted that the new energy power station mentioned above is merely an example of an application scenario of this application, and the application scenarios of this application are not limited.
[0057] The following will be through Figures 2-7 The quality control method for meteorological observation data in the embodiments of this application will be described in detail.
[0058] Figure 2 A flowchart illustrating a quality control method for meteorological observation data according to an embodiment of this application is shown schematically. Figure 2As shown, the method 200 according to this embodiment may include steps S210 to S230, wherein step S230 may include at least one of step S230A and step S230B.
[0059] In step S210, meteorological data collected by multiple observation devices in the target area are obtained to obtain the first dataset.
[0060] In step S220, the confidence score CS of each observation device is obtained.
[0061] In step S230, the first dataset is subjected to classification quality control to divide it into three categories: reliable data, questionable data, and erroneous data.
[0062] Specifically, the classification quality control in step S230 includes at least one of temporal consistency detection (step S230A) and spatial consistency detection (step S230B). That is, according to the embodiments of this application, the classification quality control in step S230 includes three methods: performing only temporal consistency detection, performing only spatial consistency detection, and performing both temporal consistency detection and spatial consistency detection simultaneously.
[0063] Continue to refer to Figure 2 The time series consistency detection step S230A may include steps S231A to S233A. Time series consistency detection utilizes the continuity and trend characteristics of meteorological elements in the time dimension to detect abrupt changes or unreasonable jumps. It can be used to discover short-term faults or transient anomalies in observation equipment. Its advantage lies in its ability to detect data abrupt changes and anomalies in local time.
[0064] In step S231A, calculate the first mean of multiple first data collected by any first observation device within a sliding time window and the first deviation of each first data from the first mean.
[0065] In step S232A, all first data with a first deviation greater than the first threshold are classified as erroneous data.
[0066] In step S233A, data with a first deviation amount not greater than a first threshold but greater than a first intermediate threshold are classified as questionable data, and the rest are classified as reliable data. The first intermediate threshold is less than the first threshold.
[0067] Wherein, at least one of the first threshold and the first intermediate threshold is obtained based on the reliability score of the first observation device, so that the width of the first interval, with the first threshold and the first intermediate threshold as upper and lower limits, is negatively correlated with the reliability score of the first observation device. In this way, the higher the reliability score of the first observation device, the less likely it is to be classified as questionable data and the greater the likelihood of it being classified as reliable data in the remaining data after obvious erroneous data is identified.
[0068] In one embodiment, step S230A may further include: calculating the first standard deviation of multiple first data collected by the first observation device within a sliding time window. Then set the first threshold to the first standard deviation. times (i.e.) ),in A constant and a positive number represents the time deviation threshold coefficient, which is used to filter out obviously erroneous data in time series consistency testing; correspondingly, the first intermediate threshold is set to the first standard deviation. of Multiply by the credibility score CS (i.e.) The credibility score is greater than or equal to zero and less than or equal to 1. Thus, the first interval, with the first threshold and the first intermediate threshold as its upper and lower limits, can be... The first data point whose deviation falls within this first interval will be classified as questionable data. The width of this first interval is... of The higher the credibility score (CS) of the first device, the smaller the width of the first interval.
[0069] In this embodiment, the dynamic threshold setting for temporal consistency detection associates a first intermediate threshold with the credibility score (CS). This first intermediate threshold is essentially the threshold for distinguishing between credible and questionable data and measures a first deviation. Therefore, the larger the first intermediate threshold, the more temporal deviation is allowed in the data collected by the device. In other words, the higher the device's credibility score, the more lenient the screening conditions for credible data, and correspondingly, the lower the probability of it being classified as questionable data.
[0070] Continue to refer to Figure 2 The spatial consistency detection step S230B may include steps S231B to S233B. Spatial consistency detection identifies single-point anomalies by comparing the differences in observation data from adjacent stations or multiple observation devices within the same field station, and can detect local observation point deviations or equipment malfunctions.
[0071] In step S231B, the second mean of multiple second data collected by multiple observation devices at the same time and the second deviation of each second data from the second mean are calculated.
[0072] In step S232B, for any second data collected by the second observation device, if the second deviation is greater than the second threshold, the second data is determined to be erroneous data.
[0073] In step S233B, for any second data collected by the second observation device, if the second deviation is not greater than the second threshold but greater than the second intermediate threshold, the second data is determined to be questionable data; and if the second deviation is not greater than the second intermediate threshold, the second data is determined to be reliable data, wherein the second intermediate threshold is less than the second threshold. At least one of the second threshold and the second intermediate threshold is obtained based on the reliability score of the second observation device, such that the width of the second interval, with the second threshold and the second intermediate threshold as upper and lower limits, is negatively correlated with the reliability score of the second observation device.
[0074] In this embodiment of the application, when classifying data from each observation device through spatial consistency detection, the higher the credibility of an observation device, the smaller the width of the second interval used to classify questionable data. This means that if the second data collected by the observation device is not obviously erroneous, it is more likely to be classified as credible data when divided according to the second deviation. This provides a more accurate screening method for data other than obviously erroneous data, and the probability of data collected by observation devices with high credibility being accepted is also higher.
[0075] In one embodiment, step S230B may further include: calculating the absolute median difference of multiple second data collected by multiple observation devices at the same time. Then, based on the assumptions of absolute median deviation and that the data follow a normal distribution, a robust estimate of the second standard deviation is obtained. Next, for the second observation device, the second threshold is set to the second standard deviation. times (i.e.) ), and set the second intermediate threshold to the second standard deviation. Multiply by the confidence score of the second observation device, i.e. The confidence score (CS) is greater than or equal to zero and less than or equal to 1. This is a constant applicable to multiple observation devices, representing the spatial deviation threshold coefficient, which is used to filter out obvious errors in spatial consistency testing.
[0076] In this embodiment, the second interval is defined as having the second threshold and the second intermediate threshold as its upper and lower limits. The width of the second interval is correspondingly... In other words, the higher the credibility score (CS) of the second observation device, the smaller the width of the second interval. This second intermediate threshold is essentially the threshold for distinguishing between credible and questionable data and measures the second degree of bias. Therefore, the larger this second intermediate threshold, the more spatially volatile the data collected by the device is allowed. Similarly, when filtering in the spatial dimension, that is, the higher the credibility score of the device, the more lenient the filtering conditions for credible data.
[0077] According to an embodiment of this application, when the classification quality control in step S230 includes both temporal consistency detection (step S230A) and spatial consistency detection (step S230B), the intersection of the reliable data partitioned by the temporal consistency detection in step S230A and the reliable data partitioned by the spatial consistency detection in step S230B is taken as the final reliable data; the union of the erroneous data partitioned by the temporal consistency detection in step S230A and the erroneous data partitioned by the spatial consistency detection in step S230B is taken as the final erroneous data; then, all data other than reliable data and erroneous data are classified as suspicious data. Specifically, when the classification quality control in step S230 includes both temporal consistency detection (step S230A) and spatial consistency detection (step S230B), the two detection methods can be implemented sequentially (e.g., one method detects and removes erroneous data, then marks reliable data and suspicious data and gives them to another detection method), or they can be executed independently, and the classification results are processed after execution. According to the embodiments of this application, when both temporal consistency detection and spatial consistency detection are included, the erroneous data from both methods are taken as the final erroneous data, the reliable data from both methods are taken as the final reliable data, and the rest are questionable data. By combining the detection results of the two methods, data that is reliable in both spatial and temporal dimensions can be filtered out.
[0078] According to the embodiments of this application, whether it is temporal consistency detection or spatial consistency detection, the criterion for classifying data (excluding erroneous data) between reliable data and questionable data in the data collected by each observation device is related to the reliability score of the observation device. The higher the reliability score of the observation device, the smaller the interval width used to filter questionable data, and the more likely the data of the observation device is to be classified as reliable data, and vice versa. In this way, a more accurate filtering method is provided for data other than obvious errors, and it can ensure that the data collected by observation devices with high reliability has a higher probability of being adopted reliably.
[0079] According to the embodiments of this application, a dynamic threshold adjustment mechanism can be introduced in temporal consistency detection and spatial consistency detection using the reliability score of each observation device. This allows the threshold for judging deviations to be adaptively adjusted according to the reliability of the device. For devices with high reliability, the division interval for doubtful data is smaller when separating reliable data from questionable data, resulting in a relatively looser selection range for reliable data. For devices with low reliability, the division interval for doubtful data is larger, thus tightening the selection range for reliable data. This approach ensures accurate classification of erroneous data while further refining classification quality control, thereby improving the overall stability and intelligence level of quality control.
[0080] Figure 3A flowchart illustrating a quality control method 300 for meteorological observation data according to another embodiment of this application is shown. Figure 3 As shown, in addition to steps S210 to S230, the method 300 according to this embodiment may also include steps S340 and S350.
[0081] First, through the classification quality control in steps S210 to S230, data is divided into three categories: reliable data, questionable data, and erroneous data. The specific implementation of steps S210 to S230 is described above and will not be repeated here.
[0082] Next, in step S340, the credible data or the union of credible data and questionable data in the first dataset is used as the second dataset. The data of each observation device in the second dataset is then subjected to reanalysis consistency detection to obtain the reanalysis consistency detection result corresponding to each observation device.
[0083] Among them, such as Figure 4 As shown, step S340 specifically includes steps S341 to S344.
[0084] In step S341, data from any third observation device is extracted from the second dataset to form an observation sample sequence.
[0085] In step S342, the values of the same meteorological elements in the reanalysis data that are at the same location as the third observation device and have the same data acquisition time as the observation sample sequence are extracted to obtain the reanalysis data sequence.
[0086] In step S343, the correlation coefficient between the observed sample sequence and the reanalysis data sequence is calculated.
[0087] In step S344, the reanalysis consistency test result corresponding to the third observation device is obtained based on the correlation coefficient, including: when the correlation coefficient meets the preset exclusion conditions, the third observation device is determined to be an observation device to be excluded.
[0088] In one embodiment, the correlation coefficient is the Pearson correlation coefficient, and the exclusion condition is that the correlation coefficient is less than a preset third threshold.
[0089] Continue to refer to Figure 3 In step S350, when the reanalysis consistency test results indicate the presence of observation devices to be excluded, the data of the observation devices to be excluded are removed.
[0090] In one embodiment, the confidence score of each observation device obtained in step S220 above can be obtained through... Figure 5 The calculation is performed according to the process shown. Specifically, as... Figure 5As shown, the credibility score of each observation device can be calculated through steps S501 to S503.
[0091] In step S501, the data quality control records of the observation equipment are obtained.
[0092] In step S502, based on the data quality control records, the anomaly rate and error rate of the observation equipment are obtained. The anomaly rate represents the proportion of abnormal data removed from the historical observation data of the observation equipment, and the error rate represents the error range of the historical observation data relative to the reference data when the reanalysis data corresponding to the same observation location and observation time of the historical observation data of the observation equipment is used as the reference data.
[0093] In step S503, the anomaly rate and error rate of the observation equipment are normalized and then weighted to obtain a reliability score.
[0094] If the data quality control records contain anomaly rates or error rates, the credibility score can be calculated based on the anomaly rates and error rates in the data quality control records.
[0095] If the data quality control record only records information about the removed outlier data (such as erroneous data, or a combination of erroneous and questionable data), such as data volume, total data volume, and data extraction time, then the anomaly rate can be calculated based on the data quality control record. Simultaneously, corresponding reanalysis data can be obtained from the data quality control record to calculate the error rate. Finally, a reliability score is calculated based on the anomaly rate and the error rate.
[0096] Quality control can be performed batch by batch. In this way, the reliability score can be calculated by separately calculating the anomaly rate and error rate of all batches and then taking the average. Alternatively, the reliability score can be calculated by summarizing the data of all batches and then calculating the anomaly rate and error rate from the summed results.
[0097] In one embodiment, the formula for the confidence score of each observation device is as follows:
[0098]
[0099] in, and These represent the weights of the anomaly rate and the error rate, respectively; A is the anomaly rate, which represents the proportion of abnormal data in the historical data quality control records; E is the error rate, which is the error range of the observed data calculated using the reanalysis data as reference data.
[0100] The closer CS is to 1, the higher the reliability of the device. Generally ,in, and The determination of the threshold can be based on terrain features (such as mountainous areas) and the abrupt changes in the measured meteorological elements, combined with experience. For example, if the data itself shows large abrupt changes, the threshold can be increased. , reduce .
[0101] One way to calculate the anomaly rate A is to calculate the anomaly rate of each batch of samples based on the quality control data of each batch. Then, the outlier rate of each batch of samples in the historical n batches is calculated. Take the average. The abnormality rate for each batch of samples is [not specified]. The number of abnormal samples removed in this batch is divided by the total number of samples originally collected in this batch.
[0102]
[0103]
[0104] As shown above, the abnormality rate A can be the average of the abnormality rates of all batches of quality control in history, with a value range of 0 to 1, and the closer to 0, the better.
[0105] Error E represents the range of error between historical observation data and reference data when reanalysis data at the same observation location and time corresponding to historical observation data of the observation equipment are used as reference data.
[0106] In one embodiment, the error E can be calculated as follows: The error of each batch of quality control samples is taken from the historical records. This is a normalized value for the root mean square error of the comparison between observed values and reanalysis data; the corresponding error E is the error of each batch of quality control samples in history. The mean.
[0107] Error of each batch of quality control samples The calculation is as follows:
[0108]
[0109] RMSE, or root mean square error, is one of the core metrics for evaluating the accuracy of reanalysis data. This represents the observations of the i-th batch. The root mean square error of the reanalysis data is calculated as the true value. This represents the maximum theoretical root mean square error calculated based on the theoretical thresholds of meteorological elements. Its main purpose is to... Normalization is performed to obtain a relative error index.
[0110] Error of each batch after quality control in n batches of samples Then, the average is taken to obtain the error E, as follows:
[0111] .
[0112] In this embodiment, the reliability score is obtained by weighted normalization of anomaly rate and error, which can provide a dynamic threshold basis for subsequent consistency detection, enabling data from different devices to adaptively determine anomalies based on their own historical performance, thereby improving the flexibility and accuracy of quality control.
[0113] According to embodiments of this application, as the data quality control records of the observation equipment are updated, the reliability score of the observation equipment can also be updated accordingly. Specifically, in response to each update of the data quality control records of the observation equipment, the data quality control records of the observation equipment are re-acquired; and in response to the re-acquisition of the data quality control records of the observation equipment, the reliability score is recalculated. With the reliability score continuously updated, data from different devices can adaptively determine anomalies based on their own historical performance, improving the dynamism and timeliness of quality control.
[0114] Figure 6 A flowchart illustrating a method for quality control of meteorological observation data according to yet another embodiment of this application is shown. Figure 6 As shown, in addition to steps S210 to S230, the method 600 according to this embodiment may also include steps S640 to S6100.
[0115] First, through the classification quality control in steps S210 to S230, data is divided into three categories: reliable data, questionable data, and erroneous data. The specific implementation of steps S210 to S230 is described above and will not be repeated here.
[0116] Next, in step S640, the data of each observation device and the questionable data of each observation device are extracted from the first dataset.
[0117] In step S650, the proportion of questionable data from any third observation device in the total data of the third observation device is calculated to obtain the proportion of questionable data from the third observation device.
[0118] In step S660, it is determined whether the proportion of questionable data from the third observation device does not exceed the fourth threshold (e.g., 5%). If yes, then proceed to step S670; if no, then proceed to steps S680 to S6100.
[0119] In step S670, if the proportion of questionable data from the third observation device does not exceed the fourth threshold, the questionable data from the third observation device is directly removed.
[0120] In step S680, if the proportion of questionable data from the third observation device exceeds the fourth threshold, reliable data from the third observation device is extracted from the first dataset.
[0121] In step S690, based on the reliable data and the questionable data from the third observation device, a multi-dimensional comparative visualization result is generated.
[0122] In step S6100, the multi-dimensional comparison visualization results are displayed to the user. This facilitates manual review.
[0123] The following example uses meteorological data quality control at a new energy power station. Figure 7 This paper presents a practical application example of a quality control method for meteorological observation data according to another embodiment of this application.
[0124] like Figure 7 As shown, the method in this embodiment includes three main steps: preliminary quality control, dynamic threshold consistency classification quality control, and visualization and manual review. The dynamic threshold consistency classification quality control step includes temporal consistency detection, spatial consistency detection, and reanalysis consistency detection.
[0125] Step 1: Preliminary quality control.
[0126] This step performs preliminary quality control on high-frequency meteorological observation data from new energy power plants through sample time screening, threshold checking, logical relationship checking, and mechanical failure checking. This step eliminates invalid data caused by incomplete formatting, physical or logical anomalies, and sensor malfunctions, ensuring a high-quality and reliable data foundation for subsequent dynamic threshold consistency classification quality control and machine learning comparative analysis. This lays the foundation for the accuracy and stability of the entire quality control system. The specific process is as follows.
[0127] First, the sample time was selected. Based on the characteristics of meteorological data usage in the new energy industry, high-frequency data from wind power, photovoltaic, and other power plants were screened. First, a format integrity check was performed to remove non-compliant data with incomplete timestamps or obvious errors. Based on this, data from the 0th, 15th, 30th, and 45th minutes of each hour were selected as representative samples.
[0128] Secondly, threshold checks are performed. Based on meteorological physical laws, the data is initially screened for reasonable ranges. For example: 1) Wind speed ≥ 0 and < 80 m / s; 2) Temperature range -60℃ ~ 55℃; 3) Solar radiation does not exceed the theoretical upper limit (1367). ).
[0129] Next, check the logical relationships. Perform a reasonableness test between different variables. For example, when the wind speed is 0, the wind direction should be marked as missing or calm; the nighttime radiation value should be close to 0.
[0130] Next, mechanical fault checks are performed. Observational data that show completely repeated values or only minor differences within a continuous time period are identified and discarded to eliminate invalid data caused by sensor malfunctions or equipment abnormalities, ensuring the reliability and authenticity of the observational data.
[0131] Finally, after completing the initial quality control, we proceed to the next step.
[0132] The second step is dynamic threshold consistency classification quality control.
[0133] First, obtain the credibility score for each device.
[0134] First, let's introduce the credibility scoring mechanism. For each observation device, a credibility score (CS) is calculated using historical data quality control records, and this score is updated after each subsequent quality control operation. The CS score is used to dynamically determine thresholds in subsequent dynamic threshold consistency classification quality control. Devices with higher credibility are considered to have more reliable data, a wider acceptable range for data fluctuations, and a more lenient threshold range.
[0135] Next, dynamic threshold consistency classification quality control is performed. This process divides the initial quality control sample data into three categories: reliable data, questionable data, and erroneous data. The classification results are then fed into the next stage.
[0136] In dynamic threshold consistency classification quality control, such as Figure 7 As shown, a temporal consistency check can be performed first, and then the erroneous data identified in the classification can be removed. The questionable data and the reliable data can be transferred to the spatial consistency check. After the spatial consistency check, the erroneous data can be removed, and the remaining data can be fully or partially (such as only reliable data) for further analysis and consistency check.
[0137] An example of timing consistency detection is as follows.
[0138] A sliding window (e.g., 15 minutes) can be used to compare the current observation with observations over a period of time before and after, identifying unreasonable abrupt changes. The calculation methods can include the following (1.1)~(1.3):
[0139] (1.1) Calculate the mean and standard deviation of the sliding window: For time t, take the observations of N consecutive times (the sliding window includes 2N+1 consecutive times) contained in a sliding window centered at time t, and calculate the first mean of the observations in the sliding window (i.e., the first data). and the first standard deviation .
[0140] (1.2) Calculate the observed value at time t First deviation: .
[0141] (1.3) In the temporal consistency detection, classification is performed based on a dynamic threshold. The classification method is as follows:
[0142] if ,,Record This is reliable data;
[0143] if ,Record Data is questionable;
[0144] if ,Record Remove erroneous data.
[0145] As mentioned earlier, The time deviation threshold coefficient is a factor used to measure obvious errors in time series consistency detection. It can be adjusted according to the temporal continuity characteristics of different elements. For example, for meteorological elements with small short-term fluctuations such as temperature and air pressure, it is typically... For factors such as precipitation and wind speed that may experience short-term abrupt changes, the time deviation threshold coefficient can be appropriately increased based on historical records.
[0146] It is understandable that the above only applies to the observations at the center of the sliding window (as described above). The classification method described in (1.3) is merely an example. According to the embodiments of this application, the observations at all times within the same sliding window can be classified using the classification method shown in (1.3).
[0147] In addition, the sliding step size of the sliding window in the timing consistency detection can be flexibly set as needed. For example, it can slide through only one time point at a time, or it can slide through multiple time points at a time point.
[0148] The duration of the sliding window can also be determined based on the real-time requirements of data quality control. For example, under the control of a new energy power station, the data acquisition frequency is once every 15 minutes, and the sliding window can be set to 30 minutes. This allows the sliding window to cover observations collected at three time points (e.g., 0 time, 15 minutes, and 30 minutes). In a new energy power station, for example, quality control can begin at 0 time, and the first time-series consistency check can be performed at 30 minutes. Each time-series consistency check can output the classification results of the data collected in the previous sliding window. When the sliding window duration is set appropriately (e.g., 30 minutes), it can effectively meet the real-time operational requirements of the new energy power station.
[0149] According to an embodiment of this application, the range of questionable data is a first interval. Thus, the larger the CS of the observation device, the smaller the width of the first interval. That is to say, when passing through... After removing obviously erroneous data, the data from that observation device is more likely to be classified as reliable data. Specifically, if an observation device has a high reliability score, obviously erroneous data is likely caused by abnormal fluctuations due to non-device reasons (such as data transmission network), while other data is more trustworthy. Conversely, if an observation device has a low reliability score, after excluding obviously erroneous data, the quality of other data cannot be assigned a high degree of reliability.
[0150] After temporal consistency testing, spatial consistency testing can be performed. An example of spatial consistency testing is as follows.
[0151] Spatial consistency testing involves comparing multiple devices within the same field station or in adjacent spaces to identify observation data with significant single-point deviations. An exemplary implementation process may include the following (2.1) to (2.4):
[0152] (2.1) Take the reliable data and the questionable data after the time sequence consistency test as input data;
[0153] (2.2) Using any observation device (e.g., observation device i) as the second observation device, take the observation values of the second observation device and multiple other observation devices in the space adjacent to the second observation device at time t, and calculate the second mean. , median and absolute median The absolute median here The process involves calculating the absolute deviation of each data point from the median after finding the median, forming an absolute deviation sequence, and then taking the median from the absolute deviation sequence.
[0154] (2.3) Calculate the observation value of observation device i at time t. The second deviation: .
[0155] (2.4) In spatial consistency detection, classification is performed based on a dynamic threshold. The classification method is as follows:
[0156] if ,Record This is reliable data; where 1.4826 is a mathematical constant derived from the theoretical properties of the normal distribution (Gaussian distribution), and its main function is as a scaling factor to convert the absolute median deviation into a robust estimate of the second standard deviation;
[0157] if ,Record Data is questionable;
[0158] if ,Record Remove erroneous data;
[0159] here This represents the spatial deviation threshold coefficient, a coefficient used to measure obvious errors in spatial consistency testing. It can be adjusted according to the spatial continuity characteristics of different elements. For example, for meteorological elements with high spatial continuity, such as air pressure and irradiance, a threshold value of [value missing] can be used. For meteorological elements such as precipitation that may have spatial discontinuities, the spatial deviation threshold coefficient can be appropriately increased based on historical records.
[0160] In spatial consistency testing, the second standard deviation is estimated using the absolute median, unlike in temporal consistency testing. This is because temporal consistency testing uses data from the same observation device within a sliding window. In spatial consistency testing, however, the data used to calculate the mean comes from different observation devices at the same time, and these devices have performance differences (different confidence scores). Therefore, estimating the second standard deviation using the absolute median and then setting the classification threshold is more robust.
[0161] Similarly, in spatial consistency testing, a second interval is defined for suspicious data. The width of the second interval is also negatively correlated with the size of CS; that is, the larger the CS, the smaller the width of the second interval. In other words, the higher the confidence score, the higher the quality of the data collected by the device, except for obvious errors caused by some abnormal factors, and the more likely it is to be classified as reliable data. Conversely, the lower the confidence score, the lower the likelihood of the data being classified as reliable data.
[0162] A reanalysis consistency check is performed after the spatial consistency check.
[0163] Reanalysis consistency detection involves extracting reanalysis data from the same locations of the observation equipment and performing correlation analysis with the data collected by the observation equipment to exclude data from observation equipment that meets the exclusion criteria. One embodiment of the process includes the following steps (3.1) and (3.2):
[0164] (3.1) Data extraction. An observation sample sequence is formed based on the observation values of observation device i at multiple observation times. The values of similar meteorological elements at the same location and time from observation equipment i are extracted to form a reanalysis data sequence. Specifically, for each sample data after the first two steps of removing erroneous data, reanalysis data from the same time point are selected.
[0165] (3.2) Correlation analysis. This embodiment specifically calculates the correlation of the observed sample sequences. With reanalysis data sequence Pearson correlation coefficient , to represent the observed sample sequence With reanalysis data sequence The correlation coefficients are as follows:
[0166]
[0167] in, Represents the observed sample sequence With reanalysis data sequence The correlation coefficient, It is an abbreviation for correlation coefficient. For the observation sample sequence With reanalysis data sequence covariance, For the observation sample sequence standard deviation For further analysis of data sequences The standard deviation. The closer to 1, the more likely it is to be an observed sample sequence. With reanalysis data sequence The better the fit. It can... Observation devices with values less than a preset third threshold (e.g., 0.7, for example only) are identified as observation devices to be excluded, and the data collected by such observation devices are discarded.
[0168] After sequentially completing the temporal consistency test, spatial consistency test, and reanalysis consistency test, the union of the erroneous data classified by each test method is taken as the final erroneous data and discarded. The intersection of the reliable data classified by each test method is taken as the final reliable data, and the remaining samples are taken as the final questionable data. Then, the aforementioned final reliable data is used as the reliable data for comprehensive consistency classification, and the aforementioned final questionable data is used as the questionable data for comprehensive consistency classification, and proceeds to subsequent analysis.
[0169] Step 3: Visualization and manual review.
[0170] First, statistical analysis is performed. Statistical analysis is conducted on the sample data sequences of each observation device to calculate the proportion of questionable data, thus obtaining the percentage of existing data.
[0171] When the percentage of questionable data from a certain observation device is below a preset fourth threshold (e.g., 5%), it indicates that the impact of these abnormal samples is relatively small. This questionable data can be directly removed, and only reliable data should be output and archived. The processing status should also be recorded as a quality control result. When the percentage of questionable data exceeds the fourth threshold, the data from that observation device can be marked as "requiring manual review." At this point, a multi-dimensional comparative visualization result can be generated based on the reliable and questionable data from the same observation device. This multi-dimensional comparative visualization result can include time series curves, statistical box plots, and spatial difference maps.
[0172] Time series graphs use time as the horizontal axis and observed values as the vertical axis. They can overlay labels for reliable data (e.g., blue), questionable data (e.g., orange), and erroneous data (e.g., red). They can also overlay reanalysis data (black) from the same time period as a baseline reference line to visually display the temporal distribution and concentration of outliers.
[0173] Statistical box plots can be used to show the differences in the numerical distribution of samples of different categories, and help determine whether anomalies are biased towards a specific range of values.
[0174] Spatial difference maps can identify potential local anomalies by overlaying the distribution of suspicious points on a base map of geographic coordinates when devices are spatially adjacent.
[0175] Users can view multi-dimensional comparative visualization results in the interactive interface. They can zoom in, filter, or hover to view specific time-specific values. The system can simultaneously output key statistical indicators (such as mean squared error, deviation rate, and duration of anomalies) to assist manual judgment in determining whether there is systematic data drift, sensor aging, or communication anomalies. After visualization, it becomes immediately clear whether questionable data needs to be considered. If the trend of questionable data is consistent with that of reliable data, more data will be available for visualization analysis; if the trends are inconsistent, removing or ignoring the questionable data will not have an impact. After manual review and confirmation, quality control is complete.
[0176] This approach achieves a closed-loop mechanism in the quality control process, encompassing automated screening, visual presentation, and manual review. This ensures data reliability while avoiding sample loss due to excessive rejection. After completing all quality control steps, the reliability score of each device is automatically updated.
[0177] This application's embodiments, by introducing a quality control method combining automatic detection, statistical grading, and visual verification, can achieve efficient identification and precise control of abnormal samples in meteorological observation data from new energy power stations. This significantly reduces the workload of manual verification while ensuring data credibility, and can visually present the distribution and trend changes of anomalies, thereby constructing a scientific and closed-loop data quality management system. The main effects include, but are not limited to: 1) Improving data credibility: Through automatic detection and grading screening, it effectively eliminates questionable data with minor impact, ensuring the stability and reliability of output data; 2) Reducing manual intervention: It can automatically complete anomaly identification and labeling, triggering manual verification only for equipment with a high proportion of questionable data, significantly reducing the manpower cost of quality control; 3) Enhancing visual interpretation capabilities: It generates multi-dimensional comparison charts, anomaly distribution charts, and statistical feature charts, making data problems clear at a glance and improving the intuitiveness of verification judgment; 4) Achieving a closed-loop quality control process: Through an automated link of "detection—analysis—verification—archiving," it forms a traceable and quantifiable quality control process.
[0178] Based on the quality control methods for meteorological observation data in the above embodiments, embodiments of this application also provide a quality control device for meteorological observation data. The following will be combined with... Figure 8 The device is described in detail.
[0179] Figure 8 A block diagram schematically illustrates a meteorological observation data quality control device 1000 according to an embodiment of this application.
[0180] like Figure 8 As shown, the meteorological observation data quality control device 1000 in this embodiment includes a first acquisition module 1010, a second acquisition module 1020, and a classification quality control module 1030.
[0181] The first acquisition module 1010 is used to acquire meteorological data collected by multiple observation devices in the target area to obtain a first dataset. In one embodiment, the first acquisition module 1010 can be used to perform step S210 described above, which will not be repeated here.
[0182] The second acquisition module 1020 is used to acquire the confidence score of each observation device. In one embodiment, the second acquisition module 1020 can be used to perform step S220 described above, which will not be repeated here.
[0183] The classification quality control module 1030 is used to perform classification quality control on the first dataset to divide it into three categories: reliable data, questionable data, and erroneous data. In one embodiment, the classification quality control module 1030 can be used to perform step S230 described above.
[0184] In one embodiment, the classification quality control module 1030 may include at least one of a temporal consistency detection submodule 1031 and a spatial consistency detection submodule 1032. In other embodiments, the classification quality control module 1030 may further include a reanalysis consistency detection submodule 1033.
[0185] The temporal consistency detection submodule 1031 is used to: calculate the first mean of multiple first data collected by any first observation device within a sliding time window and the first deviation of each first data point from the first mean; and classify first data points with a first deviation greater than a first threshold as erroneous data, classify data points with a first deviation not greater than the first threshold but greater than a first intermediate threshold as questionable data, and classify the rest as reliable data. At least one of the first threshold and the first intermediate threshold is obtained based on the reliability score of the first observation device, such that the width of the first interval, with the first threshold and the first intermediate threshold as upper and lower limits, is negatively correlated with the reliability score of the first observation device. In one embodiment, the temporal consistency detection submodule 1031 can be used to perform steps S231A to S233A as described above.
[0186] The spatial consistency detection submodule 1032 is used to: calculate the second mean of multiple second data collected by multiple observation devices at the same time and the second deviation of each second data compared to the second mean; and classify the second data collected by any second observation device as follows: when the second deviation is greater than a second threshold, the second data is determined to be erroneous data; when the second deviation is not greater than the second threshold but greater than a second intermediate threshold, the second data is determined to be questionable data; and when the second deviation is not greater than the second intermediate threshold, the second data is determined to be reliable data. At least one of the second threshold and the second intermediate threshold is obtained based on the reliability score of the second observation device, such that the width of the second interval, with the second threshold and the second intermediate threshold as upper and lower limits, is negatively correlated with the reliability score of the second observation device. In one embodiment, the spatial consistency detection submodule 1032 can be used to perform steps S231B to S233B described above.
[0187] The reanalysis consistency detection submodule 1033 is used to: extract data from any third observation device from the second dataset to form an observation sample sequence; extract similar meteorological element values from the reanalysis data that are at the same location as the third observation device and have the same data acquisition time as the data in the observation sample sequence to obtain a reanalysis data sequence; calculate the correlation coefficient between the observation sample sequence and the reanalysis data sequence; and obtain the reanalysis consistency detection result corresponding to the third observation device based on the correlation coefficient, including: determining the third observation device as an observation device to be excluded when the correlation coefficient meets the preset exclusion conditions. In one embodiment, the reanalysis consistency detection submodule 1033 can execute steps S341 to S344 described above.
[0188] In other embodiments, the quality control device 1000 may also include a rejection module and a manual review module.
[0189] The rejection module is used to reject erroneous data. In some embodiments, the rejection module is also used to directly reject the questionable data from the third observation device when the proportion of questionable data from the third observation device does not exceed a fourth threshold.
[0190] The manual review module is used to instruct manual review of the questionable data of the third observation device when the proportion of questionable data of the third observation device exceeds the fourth threshold. This includes extracting reliable data of the third observation device from the first dataset; generating multi-dimensional comparison visualization results based on the reliable data and questionable data of the third observation device; and displaying the multi-dimensional comparison visualization results to the user.
[0191] According to embodiments of this application, any multiple modules among the first acquisition module 1010, the second acquisition module 1020, the classification quality control module 1030, the temporal consistency detection submodule 1031, the spatial consistency detection submodule 1032, the reanalysis consistency detection submodule 1033, the elimination module, and the manual review module can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the first acquisition module 1010, the second acquisition module 1020, the classification quality control module 1030, the timing consistency detection submodule 1031, the spatial consistency detection submodule 1032, the reanalysis consistency detection submodule 1033, the rejection module, and the manual review module can be at least partially implemented as a hardware circuit, such as a field-programmable gate array, a programmable logic array, a system-on-a-chip, a system-on-a-substrate, a packaged system, an application-specific integrated circuit, or any other reasonable method of integrating or packaging the circuit, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of several of them. Alternatively, at least one of the first acquisition module 1010, the second acquisition module 1020, the classification quality control module 1030, the timing consistency detection submodule 1031, the spatial consistency detection submodule 1032, the reanalysis consistency detection submodule 1033, the rejection module, and the manual review module can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.
[0192] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a quality control method for meteorological observation data according to embodiments of this application.
[0193] like Figure 9 As shown, an electronic device 1200 according to an embodiment of this application includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory 1202 or a program loaded from a storage portion 1208 into a random access memory 1203. The processor 1201 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.
[0194] Random access memory 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, read-only memory 1202, and random access memory 1203 are interconnected via bus 1204. Processor 1201 executes various steps of the method flow according to embodiments of this application by executing programs stored in read-only memory 1202 and / or random access memory 1203. It should be noted that the program may also be stored in one or more memories other than read-only memory 1202 and random access memory 1203. Processor 1201 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in the aforementioned one or more memories.
[0195] According to embodiments of this application, the electronic device 1200 may further include an input / output interface 1205, which is also connected to the bus 1204. The electronic device 1200 may also include one or more of the following components connected to the input / output interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card, such as a local area network card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.
[0196] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0197] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 1202, and / or random access memory 1203, and / or one or more memories other than read-only memory 1202 and random access memory 1203 described above.
[0198] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.
[0199] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0200] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by processor 1201, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0201] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0203] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.
Claims
1. A quality control method for meteorological observation data, characterized in that, The method includes: Meteorological data collected by multiple observation devices in the target area are obtained to obtain the first dataset; The reliability score for each observation device is obtained, and the calculation process for the reliability score of each observation device includes: obtaining the data quality control record of the observation device; based on the data quality control record, obtaining the anomaly rate and error rate of the observation device, where the anomaly rate represents the proportion of anomaly data removed from the historical observation data of the observation device, and the error rate represents the error range of the historical observation data relative to the reanalysis data at the same observation location and time as the reference data; normalizing the anomaly rate and error rate of the observation device respectively, and then weighting them to obtain the reliability score; and The first dataset is subjected to classification quality control to divide it into three categories: reliable data, questionable data, and erroneous data. The classification quality control includes at least one of temporal consistency detection and spatial consistency detection. The timing consistency detection includes: Calculate the first mean of multiple first data points collected by any first observation device within a sliding time window, and the first deviation of each first data point from the first mean; and All data with a first deviation greater than a first threshold are classified as erroneous data, data with a first deviation not greater than the first threshold but greater than the first intermediate threshold are classified as questionable data, and the rest are classified as reliable data. At least one of the first threshold and the first intermediate threshold is obtained based on the reliability score of the first observation device, so that the width of the first interval with the first threshold and the first intermediate threshold as the upper and lower limits is negatively correlated with the reliability score of the first observation device. The spatial consistency detection includes: Calculate the second mean of multiple second data collected by the multiple observation devices at the same time and the second deviation of each second data compared to the second mean; The second data collected by any second observation device is classified as follows: when the second deviation is greater than the second threshold, the second data is determined to be erroneous data; when the second deviation is not greater than the second threshold but greater than the second intermediate threshold, the second data is determined to be questionable data; and when the second deviation is not greater than the second intermediate threshold, the second data is determined to be reliable data. At least one of the second threshold and the second intermediate threshold is obtained based on the reliability score of the second observation device, so that the width of the second interval, with the second threshold and the second intermediate threshold as the upper and lower limits, is negatively correlated with the reliability score of the second observation device.
2. The method of claim 1, wherein, After performing classification quality control on the first dataset to divide it into three categories: reliable data, questionable data, and erroneous data, the method further includes: Take the credible data or the union of credible data and questionable data in the first dataset as the second dataset, and perform reanalysis consistency detection on the data of each observation device in the second dataset to obtain the reanalysis consistency detection result corresponding to each observation device. When the reanalysis consistency test results indicate the presence of observation devices to be excluded, the data of the observation devices to be excluded are removed. The reanalysis consistency check includes: Extract data from any third observation device from the second dataset to form an observation sample sequence; Extract the values of the same meteorological elements from the reanalysis data that are at the same location as the third observation device and have the same data acquisition time as the observation sample sequence to obtain the reanalysis data sequence; Calculate the correlation coefficient between the observed sample sequence and the reanalysis data sequence; The reanalysis consistency test result corresponding to the third observation device is obtained based on the correlation coefficient, including: when the correlation coefficient meets the preset exclusion conditions, the third observation device is determined to be an observation device to be excluded.
3. The method of claim 2, wherein, The correlation coefficient is the Pearson correlation coefficient, and the exclusion condition is that the correlation coefficient is less than a preset third threshold.
4. The method of claim 1, wherein, The timing consistency detection also includes: Calculate the first standard deviation of multiple first data collected by the first observation device within the sliding time window; setting the first threshold to be a multiple of the first standard deviation wherein is a constant; and setting the first intermediate threshold to a multiple of the first standard deviation multiplying the credibility score by a factor, wherein the credibility score is greater than or equal to zero and less than or equal to one.
5. The method of claim 1, wherein, The spatial consistency detection also includes: Calculate the absolute median difference of multiple second data collected by the multiple observation devices at the same time; Based on the assumption that the absolute median and the data follow a normal distribution, a robust estimate of the second standard deviation is obtained; for the second observation device, setting the second threshold value to be a multiple of the second standard deviation and setting the second intermediate threshold value to be a multiple of the second standard deviation multiplied by a multiple of a reliability score of the second observation device, wherein the reliability score is greater than or equal to zero and less than or equal to 1; wherein, a constant applicable to the plurality of observation devices.
6. The method according to claim 1, characterized in that, The classification quality control includes the temporal consistency detection and the spatial consistency detection. The classification quality control of the first dataset also includes: The intersection of the trusted data partitioned by the temporal consistency detection and the trusted data partitioned by the spatial consistency detection is taken as the final trusted data. The union of the erroneous data identified by the temporal consistency detection and the erroneous data identified by the spatial consistency detection is taken as the final erroneous data; and All data other than reliable data and erroneous data are classified as questionable data.
7. The method according to claim 1, characterized in that, The calculation process for the reliability score of each observation device also includes: In response to each update of the data quality control record of the observation equipment by quality control, the data quality control record of the observation equipment is re-acquired; and In response to reacquiring the data quality control records of the observation equipment, the credibility score is recalculated.
8. The method according to claim 1, characterized in that, After performing classification quality control on the first dataset to divide it into three categories: reliable data, questionable data, and erroneous data, the method further includes: Extract the data from each observation device and the questionable data from each observation device from the first dataset; Calculate the proportion of questionable data from any third observation device in the total data of the third observation device to obtain the proportion of questionable data from the third observation device; If the proportion of questionable data from the third observation device does not exceed the fourth threshold, the questionable data from the third observation device will be directly removed. If the proportion of questionable data from the third observation device exceeds the fourth threshold, an instruction is given to manually review the questionable data from the third observation device.
9. The method according to claim 8, characterized in that, The instruction to manually review the questionable data from the third observation device includes: Extract reliable data from the third observation device from the first dataset; Based on the reliable and questionable data from the third observation device, a multi-dimensional comparative visualization result is generated; and The multidimensional comparison visualization results are displayed to the user.
Citation Information
Patent Citations
Method for controlling data quality of automatic meteorological station based on strong weather process correction
CN104635281A
