Real-time data quality control method and system

Abstract

The invention discloses a real-time data quality control method and system, and belongs to the technical field of new energy big data processing, and the method comprises the steps: obtaining wind power plant data in real time; performing quality monitoring on the wind power plant data acquired in real time; repairing the wind power plant data according to a quality monitoring result; performing quality scoring on the repaired data; performing data flow management according to a quality scoring result; the technical problems of low accuracy, timeliness and stability of data quality in a mass, high-frequency and dynamic change data environment in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy big data processing and relates to a real-time data quality control method and system. Background Art

[0002] With the increasing attention paid to renewable energy and the continuous advancement of technology, the development of new energy, especially wind and solar energy, has ushered in unprecedented opportunities. The rapid development of this field has not only promoted the optimization of energy structure, but also posed new challenges to data collection and processing. In new energy facilities such as wind farms and photovoltaic power stations, the collection and management of large-scale new energy data has become the core link to improve production efficiency and optimize operation and maintenance management.

[0003] New energy data has the characteristics of diverse sources, high frequency, and huge data volume, covering multiple aspects such as operation data, meteorological data, and equipment status. These data not only provide strong support for real-time monitoring, fault warning, and performance evaluation, but also put forward higher requirements for data processing technology due to its complexity and dynamics. However, existing data management and analysis platforms often seem to be unable to cope with these massive, heterogeneous, and rapidly changing data.

[0004] Especially in terms of real-time data quality control, traditional data monitoring platforms mainly rely on simple anomaly detection and offline processing methods, which not only fail to achieve real-time, automated data quality assessment and repair, but also often fail to cope with the complexity and dynamics of data streams. Current data quality control mechanisms are mostly focused on verification during data access and simple anomaly detection after collection. They lack real-time monitoring and intelligent repair capabilities for data streams, resulting in unstable data quality and affecting the accuracy and reliability of subsequent analysis.

[0005] In addition, the problems of missing, outliers and delays in real-time data are particularly prominent in the process of processing new energy data. These problems not only reduce the quality of data analysis results, but also seriously affect the formulation of important decisions such as wind turbine performance evaluation and fault warning. Therefore, how to achieve comprehensive monitoring, repair and quality control of real-time data in a massive, high-frequency and dynamically changing data environment has become a technical problem that needs to be solved urgently.

[0006] In summary, the existing technology has many deficiencies in the quality control of new energy data, especially in terms of real-time, intelligence and automation. Therefore, developing a more efficient, real-time and intelligent data quality control method to ensure the accuracy and reliability of new energy data has become an important technical requirement for the development of the new energy field. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problem of how to achieve comprehensive monitoring, repair and quality control of real-time data in the prior art, especially how to improve the accuracy, timeliness and stability of data quality in a massive, high-frequency and dynamically changing data environment, and to provide a real-time data quality control method and system.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: A first aspect of the present invention provides a real-time data quality control method, comprising the following steps: Get wind farm data in real time; Monitor the quality of wind farm data acquired in real time; Repair wind farm data based on quality monitoring results; Perform quality scoring on the repaired data; Data flow management is performed based on quality scoring results.

[0009] Furthermore, the quality monitoring of the wind farm data acquired in real time is specifically as follows: Monitor the integrity of wind farm data acquired in real time; Monitor the accuracy of wind farm data acquired in real time; Monitor the timeliness of wind farm data acquired in real time.

[0010] Furthermore, the monitoring of the integrity of the wind farm data acquired in real time is specifically as follows: Identify whether there are any missing wind farm data obtained in real time. Furthermore, the accuracy of the wind farm data acquired in real time is monitored by using a Z-score-based standard deviation method and a historical data threshold setting method to detect abnormal values.

[0011] Furthermore, the monitoring of the timeliness of the wind farm data acquired in real time is specifically as follows: By comparing the data collection time with the preset timestamp, it is determined whether the data has a delay. If the delay exceeds the set threshold, an early warning is triggered.

[0012] Further, repairing the wind farm data according to the quality monitoring result includes: Missing data are filled using an interpolation algorithm, which estimates reasonable values ​​of missing data based on historical data trends; The detected outliers are corrected using a smoothing algorithm to make them conform to the fluctuation range of normal data. Further, the quality scoring of the repaired data includes: Completeness score, which scores data completeness by the proportion of missing values; Accuracy score, which scores the data accuracy by the proportion of outliers; Timeliness scoring: Scoring the timeliness of data based on its delay; The completeness score, accuracy score, and timeliness score are weighted and summed to obtain the final quality score.

[0013] Further, the data flow management is performed according to the quality scoring result, specifically: Isolate data that fail to meet quality standards; Data with qualified quality scores will be used in subsequent data processing and analysis.

[0014] A second aspect of the present invention provides a real-time data quality control system, comprising: Data access module, used to obtain wind farm data in real time; Data quality monitoring module, used to monitor the quality of wind farm data acquired in real time; A data repair module is used to repair wind farm data according to quality monitoring results; Data quality scoring module, used to score the quality of repaired data; The data flow management module is used to manage the data flow according to the quality scoring results.

[0015] The system further comprises: The data standardization module is used to standardize the wind farm data acquired in real time to ensure the consistency of the data; the standardization operation converts data from different sources and in different formats into a unified format.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a real-time data quality control method, which collects various data of a wind farm, such as wind speed, wind direction, power generation, etc., in real time through sensors, measuring equipment, etc., monitors the collected data in real time, and promptly finds anomalies and errors in the data; uses corresponding algorithms and technologies to repair the data anomalies and errors found in the monitoring to ensure the accuracy, completeness and timeliness of the data; scores the repaired data according to its quality to quantify the quality level of the data; performs shunting processing on the data stream according to the quality score of the data to ensure that high-quality data is processed and utilized, and low-quality data is isolated; improves the accuracy and reliability of the data; high-quality data provides strong support for the operation and maintenance management of the wind farm, so that the operation and maintenance personnel can more accurately judge the operating status of the equipment, take measures in time, and improve the operation and maintenance efficiency; accurate data helps to optimize the power generation strategy of the wind farm and improve the power generation efficiency.

[0017] Furthermore, the present invention discloses a real-time data quality control system, wherein the data access module can obtain data generated by various sensors and measuring equipment in the wind farm in real time and accurately, and ensure the timeliness of the data. This provides a solid foundation for the subsequent data quality monitoring, repair and scoring; the data quality monitoring module can comprehensively and accurately identify anomalies and errors in the data by adopting a variety of monitoring means (such as integrity monitoring, accuracy monitoring and timeliness monitoring). This helps to timely discover and solve data quality problems and improve the reliability and accuracy of the data. The data repair module can automatically repair the wind farm data according to the quality monitoring results. By adopting intelligent technologies such as interpolation algorithms and smoothing algorithms, the module can accurately repair missing values ​​and abnormal values, thereby restoring the integrity and accuracy of the data; the data quality scoring module can objectively reflect the overall quality level of the data by scoring the quality of the repaired data. The scoring results comprehensively consider multiple aspects such as the integrity, accuracy and timeliness of the data, and are scientific and comprehensive; the data flow management module flexibly manages the data flow according to the quality scoring results. For data with unqualified quality scores, the module can isolate and process them to prevent them from affecting subsequent data processing and analysis. This helps ensure the purity and controllability of data flow and improves the efficiency and accuracy of data processing.

[0018] Furthermore, by standardizing the wind farm data acquired in real time, the differences between data from different sources and formats can be eliminated to ensure data consistency. This helps to simplify the data processing process and improve the efficiency and accuracy of data processing; standardization operations convert data from different sources and formats into a unified format, making data from different time periods, different equipment or different wind farms comparable. This helps to analyze and compare data across time periods, equipment or wind farms, and provides more valuable reference information for the operation and maintenance management of wind farms. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a flow chart of a real-time data quality control method according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a real-time data quality control system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention described and marked in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0024] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0025] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0026] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1An embodiment of the present invention discloses a real-time data quality control method, comprising the following steps: S1, real-time acquisition of wind farm data; first receive real-time data from different data sources. These data mainly include wind turbine operation data (such as power, speed, temperature, etc.), meteorological data (such as wind speed, wind pressure, etc.) and fault diagnosis data. After the data is accessed, it is first standardized, that is, the format is uniformly converted to ensure that data from different sources and different formats are converted into a unified format.

[0028] S2, quality monitoring of wind farm data acquired in real time; after receiving the data, real-time quality monitoring of the data is performed to ensure the integrity, accuracy and timeliness of the data. Specifically, integrity monitoring is used to check in real time whether there are missing data in the wind farm, and to mark and record the missing data. Accuracy monitoring uses the standard deviation method based on Z-score and the historical data threshold setting method to detect outliers. The system calculates the Z-score value of each data point and compares it with the preset threshold. If it exceeds the threshold, it is determined to be an outlier. At the same time, the system will also set a reasonable threshold range based on historical data, and mark the data that exceeds the range as abnormal. Timeliness monitoring determines whether there is a delay in the data by comparing the data collection time with the preset timestamp. If the delay exceeds the set threshold (such as 5 minutes), an early warning is triggered to remind relevant personnel to deal with it in time.

[0029] S3, repair the wind farm data according to the quality monitoring results; for the detected abnormal data, the present invention provides a corresponding data repair strategy. Missing data can be filled by an interpolation algorithm, and the interpolation method estimates the reasonable value of the missing data according to the trend of historical data. For the sudden abnormal value, it is corrected by a smoothing algorithm to make it conform to the fluctuation range of normal data. In addition, the correlation between the operation model of the wind turbine and the meteorological data is used to reconstruct the data, and the reconstructed data can more accurately reflect the actual operation situation. That is, on the basis of data repair, the system also uses the correlation between the operation model of the wind turbine and the meteorological data to reconstruct the data. By analyzing the operation model of the wind turbine and combining real-time meteorological data (such as wind speed, wind direction, temperature, etc.), the system can predict and infer the operation status of the wind turbine under different meteorological conditions. When there is a significant difference between the actual data and the predicted data, the actual data is reconstructed according to the correlation between the operation model of the wind turbine and the meteorological data to more accurately reflect the actual operation of the wind turbine.

[0030] S4, perform a quality score on the repaired data; perform a quality score on each batch of data after repair and reconstruction. The scoring dimensions mainly include data integrity, accuracy and timeliness. The data integrity score is measured by the proportion of missing values. The fewer missing values, the higher the score; the accuracy score is evaluated by the proportion of outliers and the repair effect. The fewer outliers, the higher the score; the timeliness score is evaluated based on the data delay. The smaller the delay, the higher the score. The final score of each data batch is the weighted sum of the scores of the three dimensions, and the scoring results are transmitted to the data flow management module. The weights can be set according to actual conditions.

[0031] S5, manage the data flow according to the quality scoring results; classify and manage the data according to the quality scoring results, and the data flow with qualified quality will be passed to the subsequent analysis and modeling modules. For the data flow with unqualified quality, it is isolated and processed to prevent the unqualified data from affecting the results of the overall data analysis. In this way, the present invention effectively ensures the quality of the data, and further provides high-quality data support for subsequent data analysis, fan efficiency evaluation, fault diagnosis, etc.

[0032] The real-time data quality control method of the present invention solves the problems of unstable data quality, insufficient data repair capability, low analysis accuracy, etc. in the prior art, especially for real-time data of large-scale distributed data sources such as wind farms in the field of new energy, ensuring its accuracy, completeness and timeliness. The technical solution provided by the present invention can effectively improve the quality of data through real-time monitoring, intelligent repair and quality scoring, thereby optimizing the efficiency evaluation and fault warning mechanism of wind turbines and improving the accuracy and reliability of data analysis.

[0033] Example 2: Real-time data quality control system This embodiment provides a real-time data quality control system, which is composed of multiple functional modules to ensure the efficient operation of the real-time data quality control method. Figure 2 shown.

[0034] Data access module, which is responsible for receiving real-time data from various data sources (such as fans, meteorological sensors, equipment diagnostic systems, etc.). Data sources may include various sensors and devices, and the data collection frequency is as high as every second or every minute, ensuring that the platform can obtain data in real time. The accessed data may exist in different formats and units, so the data access module is also responsible for data standardization to ensure the consistency of all data.

[0035] Data quality monitoring module, which monitors data in real time after data access. Specific functions include data integrity monitoring, data accuracy monitoring, and data timeliness monitoring. Integrity monitoring is used to detect whether there are missing data, accuracy monitoring uses anomaly detection algorithms to find outliers in the data, and timeliness monitoring determines whether there are data delays based on the comparison between the collection time and the actual processing time. Through these monitoring functions, the system can capture problems in the data in real time and respond.

[0036] The data repair module automatically repairs missing values ​​or outliers in the data once the data quality monitoring module identifies them. The missing values ​​are repaired using an interpolation algorithm, which interpolates the data before and after the time series. For outliers, the system uses a smoothing algorithm to adjust them to values ​​within the normal range. If a piece of data has a serious deviation, the system can also correct it through a wind turbine model and a meteorological data reconstruction algorithm to obtain a reasonable repair value.

[0037] The repaired data will enter the data quality scoring module. This module scores the data based on its completeness, accuracy, and timeliness, and quantitatively evaluates the quality of each batch of data. The scoring results reflect the quality level of the data and can help subsequent modules determine whether the data meets the analysis requirements.

[0038] Data flow management module: The data flow management module classifies and manages data flows according to quality scores. Data with qualified quality will enter the subsequent data analysis and modeling module for operations such as wind turbine efficiency evaluation and fault warning; data with unqualified quality will be isolated or marked as abnormal data to prevent it from affecting the analysis results. This module effectively improves the stability and reliability of the system by automatically classifying data flows.

[0039] Through the collaborative work of the above-mentioned functional modules, the system provided in this embodiment can achieve efficient and automated data quality control, ensure that the real-time data of new energy equipment such as wind farms is always in a high-quality state, and provide accurate data support for subsequent decision-making and analysis.

[0040] The real-time data quality control method and system provided by the present invention can effectively solve the problem of unstable real-time data quality in the new energy big data platform. Through real-time monitoring, intelligent repair and quality scoring, the present invention can ensure the high quality and reliability of data, thereby providing strong data support for wind turbine performance evaluation, fault diagnosis, early warning, etc. This solution can improve the stability and operating efficiency of the system while ensuring data accuracy, providing technical support for efficient management and optimization of the new energy industry.

[0041] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time data quality control method, characterized in that: The following steps are involved: Get wind farm data in real time; Monitor the quality of wind farm data acquired in real time; Repair wind farm data based on quality monitoring results; Perform quality scoring on the repaired data; Data flow management is performed based on quality scoring results.

2. The real-time data quality control method according to claim 1, characterized in that: The quality monitoring of the wind farm data acquired in real time is specifically as follows: Monitor the integrity of wind farm data acquired in real time; Monitor the accuracy of wind farm data acquired in real time; Monitor the timeliness of wind farm data acquired in real time.

3. The real-time data quality control method according to claim 1, characterized in that: The monitoring of the integrity of the wind farm data acquired in real time is specifically as follows: Identify whether there are any missing wind farm data obtained in real time.

4. The real-time data quality control method according to claim 1, characterized in that: The accuracy of the wind farm data acquired in real time is monitored by using a Z-score-based standard deviation method and a historical data threshold setting method for outlier detection.

5. The real-time data quality control method according to claim 1, characterized in that: The monitoring of the timeliness of the wind farm data acquired in real time is specifically as follows: By comparing the data collection time with the preset timestamp, it is determined whether the data has a delay. If the delay exceeds the set threshold, an early warning is triggered.

6. The real-time data quality control method according to claim 1, characterized in that: The repairing of wind farm data according to the quality monitoring result includes: Missing data are filled using an interpolation algorithm, which estimates reasonable values ​​of missing data based on historical data trends; The detected outliers are corrected using a smoothing algorithm to make them conform to the fluctuation range of normal data.

7. The real-time data quality control method according to claim 1, characterized in that: The quality scoring of the repaired data includes: Completeness score, which scores data completeness by the proportion of missing values; Accuracy score, which scores the data accuracy by the proportion of outliers; Timeliness scoring: Scoring the timeliness of data based on its delay; The completeness score, accuracy score, and timeliness score are weighted and summed to obtain the final quality score.

8. The real-time data quality control method according to claim 1, characterized in that: The data flow management according to the quality scoring result is specifically as follows: Isolate data that fail to meet quality standards; Data with qualified quality scores will be used in subsequent data processing and analysis.

9. A real-time data quality control system, based on the real-time data quality control method according to any one of claims 1 to 7, characterized in that: include: Data access module, used to obtain wind farm data in real time; Data quality monitoring module, used to monitor the quality of wind farm data acquired in real time; A data repair module is used to repair wind farm data according to quality monitoring results; Data quality scoring module, used to score the quality of repaired data; The data flow management module is used to manage the data flow according to the quality scoring results.

10. The real-time data quality control system according to claim 8, characterized in that: Also includes: The data standardization module is used to standardize the wind farm data acquired in real time to ensure the consistency of the data; the standardization operation converts data from different sources and in different formats into a unified format.

Images