AI-assisted data quality intelligent monitoring and analysis method

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an AI-assisted data quality intelligent monitoring and analysis method. According to the AI-assisted data quality intelligent monitoring and analysis method, through a multi-step processing flow, multi-dimensional power data can be acquired in real time, timestamp marking can be performed, missing values and abnormal values can be effectively identified, missing value complementation is performed by adopting an interpolation method, so that data deviation is reduced, and in the feature extraction stage, the data quality is improved. The average power and the standard deviation of the power data are calculated, the volatility of the data is quantified, the data features are richer and more comprehensive, the power data quality prediction model generated through training of the support vector machine can comprehensively consider multiple dimensions such as the integrity, the accuracy, the consistency and the timeliness of the data for evaluation, and the evaluation efficiency is improved. Excessive dependence of a traditional method on a single index is avoided, and the real situation of power data quality is comprehensively reflected.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to an AI-assisted data quality intelligent monitoring and analysis method. Background Art

[0002] With the development of smart grids, the technology of power monitoring and management is also constantly improving. In recent years, the widespread application of smart meters and power sensors has made it possible to obtain power data in real time. These devices can not only monitor basic parameters such as voltage, current and power, but also provide more power quality indicators such as power factor. These data provide important decision-making support for power companies and users, helping to optimize power consumption, improve energy efficiency and reduce operating costs. At the same time, technologies based on cloud computing and big data analysis are also gradually becoming popular in the field of power data processing, making the storage, analysis and visualization of massive power data a reality. The combination of these technologies has promoted the intelligent and digital development of smart power systems and created conditions for more efficient power management.

[0003] Traditional power data quality monitoring methods often ignore the multi-dimensional characteristics of data and rely only on a single indicator to judge data quality, which makes it difficult to fully reflect the true situation of power data. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides an AI-assisted intelligent monitoring and analysis method for data quality. Through a multi-step processing flow, it can not only acquire multi-dimensional power data in real time and perform timestamp marking, but also effectively identify missing values ​​and outliers, and use interpolation to fill in missing values, thereby reducing data deviation. In the feature extraction stage, the average power and standard deviation of the power data are calculated to quantify the volatility of the data, making the data features richer and more comprehensive. The power data quality prediction model generated by support vector machine training can comprehensively consider multiple dimensions such as data integrity, accuracy, consistency and timeliness for evaluation, avoiding the excessive reliance of traditional methods on a single indicator, comprehensively reflecting the true situation of power data quality, and solving the above problems.

[0005] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: an AI-assisted data quality intelligent monitoring and analysis method, comprising the following steps: S1. Obtain power data in real time through smart meters and power sensors, including voltage data, current data, power data, and power factor, and perform data time stamping; S2. Clean the power data acquired in real time, identify missing values ​​and remove outliers, use interpolation to complete missing values, and use statistical analysis methods to detect power data points that exceed the normal range; S3. After the power data is cleaned, feature extraction is performed to calculate the average power of the power data and the standard deviation of the power data to quantify the volatility of the power data; S4. After the power data features are extracted, data quality assessment is performed to calculate the power data quality integrity, power data accuracy, power data consistency and power data timeliness, and the power data is trained through a support vector machine to generate a power data quality prediction model; S5. Input the real-time power data into the trained power data quality prediction model, calculate the power data quality index and compare it with the power data quality expectation index to determine the power quality abnormality. Once the power abnormality is detected, the power data quality monitoring system automatically triggers the alarm mechanism and displays the power abnormality on the power data quality monitoring screen.

[0006] Preferably, the missing value completion formula is as follows: ; In the formula, Indicates the missing values ​​to be filled. Indicates the value of the previous valid power data point before the missing value. Indicates the value of a valid power data point after the missing value. Indicates the timestamp of the last valid electricity data point before the missing value. Indicates the timestamp of a valid power data point after the missing value. Indicates the timestamp corresponding to the missing value.

[0007] Preferably, the formula for removing outliers is as follows: ; In the formula, represents the outliers of the power data points, Indicates the power data point to be detected, represents the mean of the power data set, represents the standard deviation of the electricity data set; The mean calculation formula of the power data set is as follows: ; In the formula, represents the total number of data points in the power dataset, Represents the value of each power data point in the power data set, Indicates a counting subscript; The formula for calculating the standard deviation of the power data set is as follows: ; When the power data point is outlier , indicating that the power data point is an outlier.

[0008] Preferably, the calculation formula of the average power of the electric power data is as follows: ; In the formula, Indicates the average power of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, Indicates a counting subscript.

[0009] Preferably, the calculation formula of the standard deviation of the power data is as follows: ; In the formula, represents the standard deviation of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, represents a count subscript, Indicates the average power of electric power data.

[0010] Preferably, the calculation formula for the power data quality integrity is as follows: ; In the formula, Indicates the quality integrity of power data. represents the number of valid power data points, that is, the number of non-missing and non-abnormal power data points, Indicates the total number of power data points.

[0011] Preferably, the calculation formula for the accuracy of the power data is as follows: ; In the formula, Indicates the accuracy of power data, represents the total number of power data points, Indicates The power value of the data point, Indicates the real power value at the corresponding moment, Indicates a counting subscript.

[0012] Preferably, the calculation formula for the power data consistency is as follows: ; In the formula, Indicates power data consistency, , Represents power data points at different time periods, represents the total number of power data points, , Indicates a counting subscript.

[0013] Preferably, the calculation formula for the timeliness of the power data is as follows: ; In the formula, Indicates the timeliness of power data, Indicates the number of data points successfully recorded within the specified time. Indicates the total number of power data points.

[0014] Preferably, the calculation formula of the power data quality index is as follows: ; In the formula, represents the power data quality index, Represents the weight coefficients, corresponding to completeness, accuracy, consistency and timeliness respectively.

[0015] Compared with the prior art, the present invention provides an AI-assisted data quality intelligent monitoring and analysis method, which has the following beneficial effects: Through a multi-step processing flow, the present invention can not only acquire multi-dimensional power data in real time and perform time stamp marking, but also effectively identify missing values ​​and outliers, and use interpolation to fill in missing values, thereby reducing data deviation. In the feature extraction stage, the average power and standard deviation of the power data are calculated to quantify the volatility of the data, making the data features richer and more comprehensive. The power data quality prediction model generated by support vector machine training can comprehensively consider multiple dimensions such as data integrity, accuracy, consistency and timeliness for evaluation, avoiding the excessive reliance of traditional methods on a single indicator and fully reflecting the true situation of power data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the steps of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In view of the problem that traditional power data quality monitoring methods often ignore the multi-dimensional characteristics of data and rely only on a single indicator to judge data quality, it is difficult to fully reflect the true situation of power data. Therefore, an AI-assisted data quality intelligent monitoring and analysis method is proposed. Figure 1 , the method comprises the following steps: S1. Obtain power data in real time through smart meters and power sensors, including voltage data, current data, power data, and power factor, and perform data time stamping; As the core equipment of modern power monitoring, smart meters have high-precision measurement capabilities and can continuously monitor key power parameters such as voltage, current, power and power factor. The acquisition of voltage data is usually achieved through high-precision voltage sensors, which can provide stable voltage readings under different load conditions; The collection of current data relies on current sensors, which can use technologies such as Hall effect or current transformers to ensure safe and accurate measurement in high current environments. Power data is obtained by combining voltage and current data using real-time calculation algorithms. The calculation of power factor reflects the efficiency of the power system, which is usually achieved by measuring the phase difference between voltage and current. During the data collection process, each data point will be marked with a timestamp to ensure the timeliness and traceability of the data. The introduction of timestamps not only makes subsequent data analysis and trend monitoring more accurate, but also provides important time dimension information for fault diagnosis and load forecasting. In addition, data transmission uses the efficient communication protocol Modbus to ensure that real-time data can be quickly transmitted to the central monitoring system or cloud platform, facilitating big data analysis and intelligent decision-making; S2. Clean the power data acquired in real time, identify missing values ​​and remove outliers, use interpolation to complete missing values, and use statistical analysis methods to detect power data points that exceed the normal range; In the preliminary step of data cleaning, the system uses predefined rules and algorithms to identify missing values. By setting reasonable thresholds or ranges, the system can quickly locate the voltage, current or power data that is not accurately recorded. The identification of these missing values ​​usually depends on the timestamp of the data to ensure that the missing position can be accurately identified and located in the time series. Once the missing values ​​are identified, the next step is to use interpolation to complete them. The interpolation method uses linear interpolation to infer the missing values ​​based on the information of adjacent known data points, thereby maintaining the continuity and stability of the data series. The linear interpolation method estimates the missing power value by establishing a straight line relationship between the known data before and after the missing value. This method is simple and effective and is suitable for use in situations where data changes are relatively stable. On the other hand, outlier detection of data is equally important. In this link, statistical analysis methods are used to mine and analyze power data. The Z-score method calculates the standard deviation between each data point and the average value of the data set to determine whether there are data points that exceed the reasonable range. This method can effectively identify and remove abnormal data points caused by sensor failure or external interference, ensuring the overall quality of the data set. The formula for missing value completion is as follows: ; The existence of missing values ​​often leads to incompleteness in data analysis, statistics, and model building. Filling in missing values ​​helps provide a complete data set and improve the reliability of data analysis. In the formula, Indicates the missing values ​​to be filled. Indicates the value of the previous valid power data point before the missing value. Indicates the value of a valid power data point after the missing value. Indicates the timestamp of the last valid electricity data point before the missing value. Indicates the timestamp of a valid power data point after the missing value. Indicates the timestamp corresponding to the missing value and fills in the missing value, especially using interpolation method, which can maintain the timing characteristics and continuity of the data and reduce the impact of sudden jumps on data trend analysis; The formula for removing outliers is as follows: ; In the formula, represents the outliers of the power data points, Indicates the power data point to be detected, represents the mean of the power data set, represents the standard deviation of the electricity data set; The mean calculation formula for the power data set is as follows: ; In the formula, represents the total number of data points in the power dataset, Represents the value of each power data point in the power data set, Indicates a counting subscript; The standard deviation of the power data set is calculated as follows: ; When the power data point is outlier , indicating that the power data point is an outlier. Outliers often come from measurement errors, sensor failures and other problems. Removing these outliers can effectively improve the accuracy of the data and make the analysis results more credible. When using the algorithm model, outliers can significantly affect the parameter estimation of the model. By removing these values, the robustness of the model can be enhanced; S3. After the power data is cleaned, feature extraction is performed to calculate the average power of the power data and the standard deviation of the power data to quantify the volatility of the power data; The calculation formula for the average power of power data is as follows: ; The average power provides the power consumption level of the system in a specific period of time, which is convenient for analyzing the stability and energy efficiency of the power system load. In the formula, Indicates the average power of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, Represents the count subscript. By calculating the average power, load forecasting can be performed to provide data support for power dispatching and resource management, thereby optimizing the utilization of power resources. The calculation formula for the standard deviation of power data is as follows: ; The standard deviation can clearly show the degree of fluctuation of power data, help identify unstable factors in the power system, and facilitate the adjustment of operation strategies. In the formula, represents the standard deviation of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, represents a count subscript, Represents the average power of electricity data. By understanding volatility, we can better assess the risks of the electricity market and effectively manage supply chain and credit risks. S4. After the power data features are extracted, data quality assessment is performed to calculate the power data quality integrity, power data accuracy, power data consistency and power data timeliness, and the power data is trained through a support vector machine to generate a power data quality prediction model; The calculation formula for power data quality integrity is as follows: ; Data integrity testing helps confirm the proportion of valid data in the data set and ensures that the data used is reliable and can reflect the actual situation. In the formula, Indicates the quality integrity of power data. represents the number of valid power data points, that is, the number of non-missing and non-abnormal power data points, Indicates the total number of power data points. By ensuring integrity, it can improve the accuracy and effectiveness of subsequent data analysis and provide strong support for power system optimization; The calculation formula for the accuracy of power data is as follows: ; Through accuracy assessment, the credibility of the analysis results can be increased, relevant decisions can be based on more reliable data, and the decision-making risks can be reduced. In the formula, Indicates the accuracy of power data, represents the total number of power data points, Indicates The power value of the data point, Indicates the real power value at the corresponding moment, Indicates the count subscript. Highly accurate data helps improve the management of the power supply chain and ensure effective coordination of power generation, transmission and distribution systems. The calculation formula for power data consistency is as follows: ; Consistency checks help ensure that data from different systems or sources are compared and analyzed under the same standards, helping to unify data formats, formulas, Indicates power data consistency, , Represents power data points at different time periods, represents the total number of power data points, , Indicates the count subscript, ensuring the consistency between power data from different time periods and different sources, which helps to improve the overall data quality and reduce errors in data analysis; The calculation formula for the timeliness of power data is as follows: ; Timeliness indicators ensure that data is updated in real time, can quickly reflect market changes or system failures, and improve the power system's ability to respond to emergencies. In the formula, Indicates the timeliness of power data, Indicates the number of data points successfully recorded within the specified time. Indicates the total number of power data points. Timely data can help power managers monitor system status in real time, prevent potential problems, and improve the stability and reliability of power generation and distribution systems; S5. Input the real-time power data into the trained power data quality prediction model, calculate the power data quality index and compare it with the power data quality expectation index, and judge the power quality abnormality. Once the power abnormality is detected, the power data quality monitoring system automatically triggers the alarm mechanism and displays the power abnormality on the power data quality monitoring screen; The calculation formula of the power data quality index is as follows: ; By calculating the power quality index, data quality can be converted from a simple description to a quantifiable value. This quantification allows data quality to be presented in the form of specific indicators, which is convenient for comparison and analysis. In the formula, represents the power data quality index, Represents the weight coefficient, which corresponds to completeness, accuracy, consistency and timeliness respectively. By setting thresholds, it can help identify abnormal situations in power data. Once it is detected that the power quality index is lower than the expected index of power data quality, the system can automatically trigger the alarm mechanism and provide early warning to reduce potential risks.

[0019] Through the application of the above methods, the data is evaluated comprehensively based on multiple dimensions such as completeness, accuracy, consistency and timeliness, avoiding the excessive reliance of traditional methods on a single indicator and fully reflecting the true situation of power data quality.

[0020] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI-assisted data quality intelligent monitoring and analysis method, characterized in that: The following steps are involved: S1. Obtain power data in real time through smart meters and power sensors, including voltage data, current data, power data, and power factor, and perform data time stamping; S2. Clean the power data acquired in real time, identify missing values ​​and remove outliers, use interpolation to complete missing values, and use statistical analysis methods to detect power data points that exceed the normal range; S3. After the power data is cleaned, feature extraction is performed to calculate the average power of the power data and the standard deviation of the power data to quantify the volatility of the power data; S4. After the power data features are extracted, data quality assessment is performed to calculate the power data quality integrity, power data accuracy, power data consistency and power data timeliness, and the power data is trained through a support vector machine to generate a power data quality prediction model; S5. Input the real-time power data into the trained power data quality prediction model, calculate the power data quality index and compare it with the power data quality expectation index to determine the power quality abnormality. Once the power abnormality is detected, the power data quality monitoring system automatically triggers the alarm mechanism and displays the power abnormality on the power data quality monitoring screen.

2. According to claim 1, the AI-assisted data quality intelligent monitoring and analysis method is characterized by: The formula for missing value completion is as follows: ; In the formula, Indicates the missing values ​​to be filled. Indicates the value of the previous valid power data point before the missing value. Indicates the value of a valid power data point after the missing value. Indicates the timestamp of the last valid electricity data point before the missing value. Indicates the timestamp of a valid power data point after the missing value. Indicates the timestamp corresponding to the missing value.

3. According to claim 2, an AI-assisted data quality intelligent monitoring and analysis method is characterized by: The formula for removing outliers is as follows: ; In the formula, represents the outliers of the power data points, Indicates the power data point to be detected, represents the mean of the power data set, represents the standard deviation of the electricity data set; The mean calculation formula of the power data set is as follows: ; In the formula, represents the total number of data points in the power dataset, Represents the value of each power data point in the power data set, Indicates a counting subscript; The formula for calculating the standard deviation of the power data set is as follows: ; When the power data point is outlier , indicating that the power data point is an outlier.

4. According to claim 3, the AI-assisted data quality intelligent monitoring and analysis method is characterized by: The calculation formula of the average power of the power data is as follows: ; In the formula, Indicates the average power of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, Indicates a counting subscript.

5. According to claim 4, an AI-assisted data quality intelligent monitoring and analysis method is characterized by: The calculation formula of the standard deviation of the power data is as follows: ; In the formula, represents the standard deviation of power data, Indicates the total number of power data points involved in the calculation, Indicates The power value of the data point, represents a count subscript, Indicates the average power of electric power data.

6. The AI-assisted data quality intelligent monitoring and analysis method according to claim 5 is characterized by: The calculation formula for the power data quality integrity is as follows: ; In the formula, Indicates the quality integrity of power data, represents the number of valid power data points, that is, the number of non-missing and non-abnormal power data points, Indicates the total number of power data points.

7. The AI-assisted data quality intelligent monitoring and analysis method according to claim 6 is characterized by: The calculation formula for the accuracy of the power data is as follows: ; In the formula, Indicates the accuracy of power data, represents the total number of power data points, Indicates The power value of the data point, Indicates the real power value at the corresponding moment, Indicates a counting subscript.

8. The AI-assisted data quality intelligent monitoring and analysis method according to claim 7 is characterized by: The calculation formula for the power data consistency is as follows: ; In the formula, Indicates power data consistency, , Represents power data points at different time periods, represents the total number of power data points, , Indicates a counting subscript.

9. The AI-assisted data quality intelligent monitoring and analysis method according to claim 8 is characterized by: The calculation formula for the timeliness of the power data is as follows: ; In the formula, Indicates the timeliness of power data, Indicates the number of data points successfully recorded within the specified time. Indicates the total number of power data points.

10. The AI-assisted data quality intelligent monitoring and analysis method according to claim 9 is characterized in that: The calculation formula of the power data quality index is as follows: ; In the formula, represents the power data quality index, Represents the weight coefficients, corresponding to completeness, accuracy, consistency and timeliness respectively.

Images