Data cleaning method applied to thermal production

Abstract

The invention relates to the technical field of electrical digital data, in particular to a data cleaning method applied to thermal production, which comprises the following steps: acquiring a thermal parameter time sequence of a boiler, the thermal parameter time sequence comprising a plurality of dimension parameter time sequences at different moments, and taking the plurality of dimension parameter time sequences at the same moment as a data point; performing data cleaning on each data point by using an improved exponentially weighted moving average filtering algorithm; wherein the improved exponential weighted moving average filtering algorithm comprises a weighting coefficient, the weighting coefficient is a product of a preset value and an adjustment factor, the credibility is used for measuring the reliability of a single parameter, and the credibility level of the data is reflected by analyzing the deviation between an actual value and a predicted value of the parameter and the consistency of the parameter in surrounding data. The credible factor is used for representing the abnormal degree among the parameters in the data point. The problem that data cleaning is not high in integrity and accuracy is solved.

Description

Technical Field

[0001] The present invention relates to the field of electrical digital data technology, and more specifically, to a data cleaning method used in thermal production. Background Art

[0002] In the thermal production process, a large amount of real-time data generated by boilers and other equipment is crucial for the operation monitoring and optimization of the system. However, due to equipment measurement errors, sensor aging, environmental changes and interference during data transmission, the data in the thermal system often contains noise, missing values ​​or outliers. Traditional data cleaning methods usually rely on simple filtering techniques or empirical rules to remove noise and fill missing values. However, these methods often rely too much on manually set parameters, and have limited processing capabilities for complex multi-dimensional data, making it difficult to cope with dynamically changing system states. In addition, although the existing data analysis methods based on mechanism models can theoretically simulate the operation of thermal systems, the setting of model parameters is often based on simplified assumptions, resulting in certain limitations in their applicability and accuracy. Especially in the face of complex and changing operating environments, the accuracy of the model is not enough to effectively identify anomalies and missing problems in the data, thus affecting the integrity and accuracy of data cleaning.

[0003] The patent application document with application publication number CN112328590A discloses a method for deep cleaning of thermal equipment operation data. The patent application document is based on the equipment mechanism model, combines the mechanism model with the machine learning accompanying simulation model of healthy operation data and the mass and heat balance equation of the thermal system, and realizes deep cleaning of data in the intelligent operation and maintenance of thermal equipment through methods such as key parameter similarity analysis and different parameter correlation analysis.

[0004] However, the above technical solutions only achieve deep data cleaning based on existing models, without fully considering the applicability and limitations of existing models in specific application scenarios. Errors are prone to occur when processing complex data, resulting in low integrity and accuracy of data cleaning. Summary of the invention

[0005] In order to solve the problem of low integrity and accuracy of data cleaning proposed in the above background technology, the present invention provides the following solution.

[0006] In this scheme, the present invention provides a data cleaning method applied in thermal production, including: obtaining a thermal parameter time series of a boiler, wherein the thermal parameter time series includes a plurality of dimensional parameter time series at different times, and taking the plurality of dimensional parameters at the same time as a data point; performing data cleaning on each data point using an improved exponentially weighted moving average filtering algorithm to obtain a cleaned data point; wherein the improved exponentially weighted moving average filtering algorithm includes a weighting coefficient, wherein the weighting coefficient is the product of a preset value and an adjustment factor, and the first Of the data points The adjustment factor of the dimension parameter , where For the Of the data points The credibility of the dimension parameters, For the The credibility factor of each data point is is a natural constant; the credibility factor represents the abnormality of the data point; the credibility , where For the Of the data points The value of the dimension parameter, For the Of the data points The predicted value of the dimension parameter, For the The average value of the absolute value of the difference between the parameter value and the predicted value of each dimension in the data points, For the The data point The first of the neighboring data points The value of the dimension parameter, For the The standard deviation of the data points, is the total number of neighborhood data points.

[0007] The above technical solution introduces a weighting coefficient through an improved exponentially weighted moving average filtering algorithm, and combines the preset value and the dynamic calculation of the adjustment factor to effectively identify and suppress the influence of abnormal data points. The calculation of the adjustment factor fully considers the credibility and credibility factor of the data point, ensuring that the influence of the abnormal point on the filtering result is reasonably reduced, thereby improving the stability and reliability of the data. The calculation of credibility is based on the deviation between the parameter value and the predicted value, as well as the distribution characteristics of the neighborhood data, which can accurately capture the abnormal fluctuations of the data point and further enhance the accuracy of data cleaning. At the same time, the credibility factor reflects the degree of abnormality of the data point, enabling the algorithm to more accurately identify potential deviations or noise and make effective corrections, thereby solving the problem of low data quality integrity and low analysis accuracy.

[0008] Furthermore, the trust factor is: ; In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The average value of the credibility of each dimension parameter in the data points, For the The standard deviation of the reliability of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

[0009] The above technical solution further enhances the ability to accurately identify and correct the degree of abnormality of data points by defining and calculating credible factors. By calculating the deviation of the credibility of each dimensional parameter from its average value and normalizing it with the standard deviation, the credible factor can reflect the degree of deviation and consistency of the data points between dimensions. In particular, by comparing with the credibility difference of other dimensional parameters, it is possible to effectively identify abnormal fluctuations in a certain dimension in the data set, thereby providing accurate identification and control of abnormal data. Overall, this technical solution enables the data cleaning process to not only suppress noise, but also retain key information of the data, improve the accuracy and reliability of data processing, and provide more accurate and stable data support for subsequent analysis, modeling and optimization.

[0010] Furthermore, the trust factor is: ; In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The median of the credibility of each dimension parameter in the data points, For the The maximum value of the credibility of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

[0011] The above technical solution is different from the mean method. The use of the median can effectively avoid the interference of extreme values ​​and enhance the stability of the algorithm when facing asymmetric distribution or abnormal data. At the same time, the introduction of the maximum value helps to identify the upper limit deviation of the data point, further improving the sensitivity to abnormal points. By standardizing the difference between the credibility of the parameters of each dimension and the median, and combining empirical constants to avoid numerical instability, the algorithm can more effectively identify abnormal fluctuations in data points, ensure the retention of normal information during data cleaning, and reduce the impact of noise.

[0012] Furthermore, the time series model is used to obtain the predicted value of each dimensional parameter in each data point.

[0013] Furthermore, the time series model is an autoregressive moving average model or a long short-term memory network model.

[0014] The autoregressive moving average model in the above technical solution uses the linear relationship of historical data to provide more accurate predictions when the data fluctuations are small and the linear characteristics are obvious; while the long short-term memory network model can capture the long-term and short-term dependencies in the data through its deep learning mechanism. It is particularly suitable for processing complex, nonlinear and long-term memory time series data, and can better cope with the complex dynamic changes in the boiler system. By combining the advantages of these two models, not only the modeling ability of the boiler thermal parameter time series is enhanced, but also the stability and accuracy of the prediction are effectively improved, providing more reliable technical support for real-time monitoring of boiler operation status, fault prediction and optimization control.

[0015] Furthermore, it also includes data standardization and missing completion processing of the thermal parameter time series.

[0016] The above technical solution significantly improves the data quality and robustness of the model by standardizing and completing missing values ​​in the time series data of thermal parameters. Data standardization ensures that parameters of different dimensions are processed at the same scale, eliminating the interference between dimensions that may be caused by different dimensions, making subsequent analysis more accurate and consistent. Missing value completion solves the problem of vacancies caused by incomplete data collection, ensures the continuity and integrity of time series data through reasonable filling methods, and avoids the negative impact of missing values ​​on model training and prediction results.

[0017] Furthermore, the thermal parameter time series includes: boiler temperature time series, humidity time series, pressure time series and water flow time series.

[0018] Furthermore, a temperature sensor is used to collect the temperature time series of the boiler, a humidity sensor is used to collect the humidity time series of the boiler, a pressure sensor is used to collect the pressure time series of the boiler, and a flow sensor is used to collect the water flow time series of the boiler.

[0019] The beneficial effects of the present invention are: The present invention effectively improves the accuracy and robustness of boiler system data cleaning in thermal production by introducing an improved exponentially weighted moving average filtering algorithm and a multi-dimensional credibility assessment mechanism. By cleaning and standardizing the time series of multiple thermal parameters such as boiler temperature, humidity, pressure and water flow, noise and abnormal data can be effectively removed to ensure the quality and continuity of the data. The introduction of the credibility factor enables the algorithm to accurately capture abnormal fluctuations in data points and make intelligent adjustments based on the degree of deviation of parameters in each dimension, thereby improving the accuracy and stability of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 The figure schematically shows a flow chart of a data cleaning method applied in thermal production according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] An embodiment of a data cleaning method applied in thermal production.

[0024] like Figure 1 As shown, a flow chart of a data cleaning method applied in thermal production according to an embodiment of the present invention includes the following steps: S1: Obtain the time series of multiple dimensional thermal parameters of the boiler at different times, and take the multiple dimensional parameters at the same time as a data point.

[0025] In one embodiment, the thermal parameter time series includes: boiler temperature time series, humidity time series, pressure time series and water flow time series, and these parameters fully cover the key thermal characteristics of the boiler during operation. By using a high-precision temperature sensor to collect the temperature data of the boiler in real time, the thermal state of the boiler can be accurately reflected; the humidity time series data obtained by the humidity sensor is helpful to monitor the changes in boiler steam and humidity, and provide support for thermal performance analysis; the pressure time series data collected by the pressure sensor can quickly capture the pressure dynamics inside the boiler, thereby realizing the evaluation of operational safety; at the same time, the water flow time series data obtained by the flow sensor can reflect the boiler water supply status in real time, which helps to optimize the water supply efficiency and prevent abnormal operating conditions.

[0026] In order to ensure the reliability and integrity of the data, the thermal parameter time series is further processed for data standardization and missing completion. Data standardization can map parameters of different dimensions to the same dimension, so that each parameter has higher comparability and integration in subsequent analysis, especially in machine learning model training or thermal system anomaly detection, standardized data can significantly improve algorithm performance and stability. For possible missing data, repair processing is performed through missing completion technology, such as interpolation algorithms, historical data fitting or model prediction completion, which can effectively restore the complete thermal parameter time series and avoid analysis deviations caused by missing data.

[0027] This series of data processing measures not only improves the integrity and consistency of the thermal parameter time series, but also enhances the accuracy and reliability of subsequent analysis. By efficiently collecting and processing core parameters such as boiler temperature, humidity, pressure and water flow, we can have a deeper understanding of the boiler's operating status, identify potential risks, and provide a scientific basis for operation optimization, thereby achieving intelligent and efficient equipment management. This comprehensive and sophisticated technical solution helps to significantly improve the operating efficiency, reliability and safety of the boiler system, and provides strong technical support for intelligent monitoring and decision-making of industrial boiler operation.

[0028] S2: Using an improved exponentially weighted moving average filtering algorithm to clean each data point to obtain a cleaned data point.

[0029] In one embodiment, since the boiler thermal parameter time series is usually high-dimensional and dynamic, the traditional static filtering method is often difficult to cope with the complex relationship of multi-dimensional data, while the improved exponentially weighted moving average filtering algorithm can effectively adapt to this complex data characteristic, significantly improving the efficiency and effect of data cleaning. Ultimately, by generating high-quality data points after cleaning, the accuracy of thermal system operation status monitoring can be further improved, and the decision-making deviation caused by data anomalies can be significantly reduced, providing a strong guarantee for the efficiency, safety and intelligence of boiler operation.

[0030] Compared with the traditional filtering algorithm, the improved algorithm fully considers the credibility and abnormality of the data points by dynamically adjusting the weight coefficients, and can more effectively eliminate noise and smooth data while retaining important trends and feature changes. This process not only improves the overall consistency of the data, but also lays a solid foundation for subsequent analysis and modeling.

[0031] S3: The improved exponentially weighted moving average filtering algorithm includes a weighting coefficient, which is the product of a preset value and an adjustment factor.

[0032] In one embodiment, the preset value is: ; In the formula, For the The preset value of the data points, For the The standard deviation of each data point is calculated based on the standard deviation as the core measure of data fluctuation, which can dynamically reflect the stability of the data point. Data points with large fluctuations have higher standard deviation values, and the corresponding preset values ​​will be lower, thereby reducing their influence weight in the weighted calculation; while data points with small fluctuations and higher stability have higher preset values, which can play a more important role in the filtering process. In addition, the preset values ​​can also be set manually to more accurately meet personalized needs in specific scenarios. This diversified preset value generation method greatly enhances the adaptability and practicality of the algorithm.

[0033] For the adjustment factor, Of the data points The adjustment factor of the dimension parameter , where For the Of the data points The credibility of the dimension parameters, For the The credibility factor of each data point is is a natural constant.

[0034] By adjusting the weighting of parameters of different dimensions by the adjustment factor, the influence weight of abnormal parameters can be adaptively reduced, thereby further improving the accuracy of data cleaning. After the introduction of the above-mentioned improvement strategy, the technical effect is particularly significant. First, the dynamic adjustment mechanism of the preset value enables the filtering algorithm to adaptively adjust the weight distribution according to the volatility of the data points, thereby enhancing the ability to suppress noise data. Secondly, the adjustment factor integrates the complex characteristics of multi-dimensional data points into the weight calculation process by comprehensively evaluating the credibility and credibility factors, thereby realizing accurate detection and dynamic correction of abnormal data. In addition, this improved weight mechanism fully retains the true characteristics of the data points while ensuring the filtering effect, avoiding the loss of key information due to excessive smoothing.

[0035] S4: The adjustment factor is calculated based on the parameter credibility and the data point credibility factor.

[0036] In one embodiment, the credibility , where For the Of the data points The value of the dimension parameter, For the Of the data points The predicted value of the dimension parameter, For the The average value of the absolute value of the difference between the parameter value and the predicted value of each dimension in the data points, For the The data point The first of the neighboring data points The value of the dimension parameter, For the The standard deviation of the data points, is the total number of neighborhood data points.

[0037] The calculation of credibility comprehensively considers the deviation between the parameter value and the predicted value, the distribution characteristics of the neighborhood data, and the overall volatility of the data points, making the credibility assessment of each dimension parameter more targeted and robust. By standardizing the ratio of the deviation to the mean difference and combining it with the sum of squares of the relative deviations of the neighborhood data distribution, the credibility can not only reflect the degree of difference between the parameter value and the predicted value, but also reveal the consistency and rationality of the data points in their surrounding environment. This method effectively retains the key features and overall trends of the data while smoothing outliers and noise, thereby improving the reliability of data processing and the accuracy of subsequent analysis, and providing a scientific basis for anomaly identification and performance optimization of multidimensional dynamic systems.

[0038] The trust factor is: ; In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The average value of the credibility of each dimension parameter in the data points, For the The standard deviation of the reliability of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

[0039] The calculation of the trust factor comprehensively considers the degree of deviation of the trustworthiness of each dimensional parameter from its average value, and balances the data differences between different dimensions through standardization. Further, by comparing the deviation with other dimensional parameters, the balance and consistency of the internal trustworthiness distribution of the data points are fully revealed. The empirical constant is introduced to avoid the division by zero problem, while enhancing the robustness of the algorithm and adapting to different data scenarios. This solution can accurately capture potential local anomalies in data points and ensure that the weights of anomalies are reasonably reduced, thereby better protecting the characteristics of normal data during the filtering process, while eliminating the impact of noise and anomalies on the overall data quality. This quantitative design of trust factors provides strong technical support for the precise processing and reliability analysis of complex multidimensional data, and helps to achieve more efficient anomaly detection and intelligent optimization in dynamic systems.

[0040] In another embodiment, the trust factor is: ; In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The median of the credibility of each dimension parameter in the data points, For the The maximum value of the credibility of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

[0041] Compared with the calculation method based on mean and standard deviation, the introduction of median can more effectively reduce the interference of extreme values ​​on the calculation of credible factors and improve the robustness of asymmetric distribution data. The use of maximum value provides an upper limit reference for parameter credibility and enhances the sensitivity of anomaly detection. At the same time, the numerator can more accurately reflect the relative deviation between dimensional parameters through the standardization of difference and median, combined with the consistency comparison of global parameters. This design significantly improves the anomaly detection ability of the algorithm in complex data environments, ensuring that the filtering process can more accurately identify and suppress the influence of outliers, while retaining the overall characteristics and key trends of multidimensional data, providing more efficient and reliable technical support for data cleaning and analysis in dynamic systems.

[0042] The solution of the present invention can effectively clean the boiler thermal parameter time series data and remove abnormal data points by introducing an improved exponential weighted moving average filtering algorithm, combined with the dynamic calculation of the trust factor and the trustworthiness, thereby improving the accuracy and stability of the data. By weighted adjustment of the data points, the algorithm can automatically identify and correct potential noise and deviations according to the trustworthiness and abnormality of each data point, ensuring the high quality of the data. The use of time series models for data prediction further enhances the accurate capture of future trends, especially when processing multidimensional data such as temperature, humidity, pressure and water flow in boiler operation, it can effectively deal with data fluctuations and missing problems, and improve the reliability of data analysis and decision support. In addition, data standardization and missing completion processing effectively solve the problems of data inconsistency and missingness, ensure the stability of model training, and provide a more accurate and comprehensive data basis for subsequent boiler status monitoring, fault warning and optimization control, with significant technical advantages and application value.

[0043] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0044] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A data cleaning method used in thermal production, characterized in that: include: Obtain a thermal parameter time series of the boiler, wherein the thermal parameter time series includes a plurality of dimensional parameter time series at different times, and the plurality of dimensional parameters at the same time are taken as a data point; Using the improved exponentially weighted moving average filtering algorithm to clean each data point, the cleaned data point is obtained; The improved exponentially weighted moving average filtering algorithm includes a weighting coefficient, which is the product of a preset value and an adjustment factor. Of the data points The adjustment factor of the dimension parameter , where For the Of the data points The credibility of the dimension parameters, For the The credibility factor of each data point is is a natural constant; The credibility factor represents the abnormality degree of the data point; The credibility , where For the Of the data points The value of the dimension parameter, For the Of the data points The predicted value of the dimension parameter, For the The average value of the absolute value of the difference between the parameter value and the predicted value of each dimension in the data points, For the The data point The first of the neighboring data points The value of the dimension parameter, For the The standard deviation of the data points, is the total number of neighborhood data points.

2. A data cleaning method for use in thermal production according to claim 1, characterized in that: The trust factor is: ； In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The average value of the credibility of each dimension parameter in the data points, For the The standard deviation of the reliability of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

3. A data cleaning method for use in thermal production according to claim 1, characterized in that: The trust factor is: ； In the formula, For the The credibility factor of each data point is For the Of the data points The credibility of the dimension parameters, For the Of the data points The credibility of the dimension parameters, For the The median of the credibility of each dimension parameter in the data points, For the The maximum value of the credibility of each dimension parameter in the data points, is the total number of dimension parameters, is an empirical constant, and ≠0.

4. A data cleaning method for use in thermal production according to claim 1, characterized in that: The time series model is used to obtain the predicted value of each dimension parameter in each data point.

5. A data cleaning method for use in thermal production according to claim 4, characterized in that: The time series model is an autoregressive moving average model or a long short-term memory network model.

6. A data cleaning method for use in thermal production according to claim 1, characterized in that: It also includes data standardization and missing completion processing for the thermal parameter time series.

7. A data cleaning method for use in thermal production according to claim 1, characterized in that: The thermal parameter time series includes: boiler temperature time series, humidity time series, pressure time series and water flow time series.

8. A data cleaning method for use in thermal production according to claim 1, characterized in that: The temperature sensor is used to collect the temperature time series of the boiler, the humidity sensor is used to collect the humidity time series of the boiler, the pressure sensor is used to collect the pressure time series of the boiler, and the flow sensor is used to collect the water flow time series of the boiler.

Images