Data processing method, system and equipment based on giant hydraulic turbine set and storage medium

Abstract

The invention belongs to the technical field of data processing, and particularly provides a data processing method, system and equipment based on a giant hydraulic turbine set and a storage medium, which are used for performing big data processing on a state quantity based on giant hydraulic turbine set equipment, and comprises the following steps: obtaining state quantity data of the giant hydraulic turbine set equipment, performing preprocessing according to the state quantity data to obtain preprocessed data; carrying out sampling processing according to the preprocessed data to obtain sampling data; and performing cleaning processing according to the sampling data to obtain processed data. According to the method and the system, accurate sampling of samples in huge data is realized, a data cleaning model is constructed, outlier data, lost data, delayed data and invalid data are cleaned, and the authenticity of equipment operation data is restored.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular, relates to a data processing method, system, device and storage medium based on a giant water turbine unit. Background Art

[0002] The turbine unit is a complex electromechanical system with complex fault causes and high fault hazards. Therefore, in order to avoid serious consequences of the equipment, it is necessary to conduct real-time status monitoring and equipment health management of the turbine unit. The turbine unit system equipment is complex, with strong correlations between subsystems and between subsystems and systems. The functional components cooperate with each other, and any failure of any function will cause the entire system to fail.

[0003] The health management method of turbine equipment based on big data has the contradiction of rapid data expansion and relatively backward analysis and mining methods: ① Rapid data expansion: With the rapid development of computer network technologies such as sensor technology, data acquisition, processing and transmission technology, more and more production data with higher and higher accuracy are generated during the operation of hydropower stations. The amount of data stored in the historical database has expanded rapidly, and the analysis and use of such massive data has far exceeded the capabilities of operation and maintenance personnel. ② Relatively backward analysis and mining methods: At present, the automatic analysis methods and automatic analysis systems of the existing analysis platforms in the industry are relatively backward, and cannot accurately and automatically extract valuable analysis data and prediction information from a large amount of production data to maximize the value of data. The phenomenon of "very rich data and quite poor information" has emerged, resulting in functional defects in the analysis platform in terms of early warning, early prevention, and timely intervention, and the inability to timely discover equipment safety hazards such as oil mixing in the unit bearing oil tank and oil leakage in the governor pipeline.

[0004] In view of the increasing amount of production data during the operation of the power station and the rapid expansion of the amount of data stored in the historical database, the existing data processing methods have poor performance in terms of efficiency and effectiveness, and mainly rely on expert experience for processing and judgment. Due to the complex production environment of hydropower stations and the complex data cleaning tasks, it takes a lot of time to implement a business model. However, for the health diagnosis model of turbine equipment, it is common to establish thousands of business models, and the implementation cycle is long and difficult. Expert experience is limited, and it is impossible to exhaust all experience. At the same time, expert experience will change with working conditions and environmental changes. For example, after the C repair of the unit, the data fluctuates greatly and the feature space is greatly affected. The health management of turbine equipment is a very cross-disciplinary subject, involving electrical, mechanical, chemical, data, etc., and has extremely high requirements for personnel quality.

[0005] In this case, the data cannot be accurately sampled, and there are shortcomings such as large sampling deviation and incomplete sampling, which affect the accuracy of subsequent data analysis. At the same time, it is easy for invalid data to be unable to be completed, deleted, or corrected. Summary of the invention

[0006] The present invention provides a data processing method, system, device and storage medium based on a giant hydro-turbine unit. By analyzing the data properties and adopting a suitable algorithm, accurate sampling of samples in huge data is achieved. Further, by constructing a data cleaning model, outlier data, lost data, delayed data and invalid data are cleaned, thereby restoring the authenticity of equipment operation data.

[0007] The present invention provides a data processing method based on a giant water turbine unit, which is used for performing big data analysis, sampling and processing on the state quantity of the giant water turbine unit equipment, and comprises the following steps: S110, acquiring state quantity data of giant water turbine unit equipment, and performing preprocessing according to the state quantity data to obtain preprocessed data; S120, performing sampling processing according to the pre-processed data to obtain sampled data; S130, performing cleaning processing on the sampled data to obtain processed data.

[0008] In a preferred solution, in step S110, preprocessing is performed according to the state quantity data to obtain preprocessed data, including the following steps: S111, performing temporal sorting according to the state quantity data to obtain sorted data; S112, performing homologous compression processing on the sorted data to obtain the pre-processed data.

[0009] In a preferred solution, in step S111, the state quantity data is a collection of multiple turbine unit equipment data, and the time-series sorting based on the state quantity data includes: sorting each turbine unit equipment data in chronological order based on the timestamp in the state quantity data; if there is no state quantity with a timestamp, the turbine unit equipment data is deleted.

[0010] In a preferred solution, in step S112, performing homologous compression processing on the sorted data to obtain the pre-processed data includes: 1) Determine the repeated data in the sorted data according to the state, time and amplitude of each data in the sorted data; 2) Eliminate duplicate data in the sorted data to obtain the first data; 3) According to the state and amplitude of each data in the first data, adjacent data with the same amplitude and state in the first data list are determined, and the adjacent data with the same amplitude and state are compressed to obtain preprocessed data.

[0011] In a preferred solution, in step S120, performing sampling processing according to the preprocessed data to obtain the sampled data includes: S121, performing acquisition management according to the acquisition platform based on the pre-processed data to obtain the acquired data; S122: Perform sampling analysis based on the collected data to obtain the sampled data.

[0012] In a preferred solution, step S122 includes the following contents: 1) Determine the data properties of the collected data based on the collected data, wherein the data properties include time properties, abnormal properties and uneven properties; 2) The sampling data is obtained by matching the sampling method according to the data properties.

[0013] In a preferred solution, the operation method of obtaining sampling data by matching the sampling method according to the data property is: If the data property is a time property, the sampled data is obtained by generalized smoothing weighted sampling; If the data property is abnormal, obtaining the sampled data through spatial sampling; If the data property is non-uniform, the sampled data is obtained by irregular sampling.

[0014] In a preferred solution, in step S130, cleaning the sampled data to obtain processed data includes the following steps: S131, determining the data type of the sampled data according to the sampled data, the data type including outlier data, delayed data, incomplete data and invalid data; S132, performing cleaning according to the data type to obtain the processed data.

[0015] In a preferred solution, in step S132, cleaning is performed according to the data type to obtain processed data, including the following contents: If the data type is outlier data, obtaining the processed data by using an outlier algorithm; If the data type is delayed data, obtaining the processed data by a weighted average algorithm; If the data type is incomplete data, the processed data is obtained by correcting the preceding data; If the data type is invalid data, the processed data is obtained by correcting the preceding data.

[0016] The present invention also provides a data processing system based on a giant water turbine unit, which uses the above-mentioned data processing method based on a giant water turbine unit to perform big data processing on the state quantity based on the giant water turbine unit equipment, including: A preprocessing data module is used to obtain state quantity data of giant water turbine equipment and preprocess the state quantity data to obtain preprocessing data; A sampling module, used for performing sampling processing according to the pre-processed data to obtain sampling data; The data processing module is used to perform cleaning processing on the sampled data to obtain processed data.

[0017] The present invention also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data processing method based on a giant hydro turbine unit as described above are implemented.

[0018] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method based on a giant hydro turbine unit as described above are implemented.

[0019] The present invention provides a data processing method, system, device and storage medium based on a giant water turbine unit, which has the following beneficial effects: 1. Compared with the prior art that "data cannot be accurately sampled, there are shortcomings such as large sampling deviation and incomplete sampling, which affect the accuracy of subsequent data analysis; at the same time, invalid data is prone to problems such as being unable to be completed, deleted, and corrected", the present invention provides "through uniform sampling, stratified random sampling, OCSVM sampling, and non-uniform sampling methods, accurate sampling of samples in huge data is achieved to construct a positive sample space. By using algorithms such as Kmeans outlier algorithm, KNN outlier algorithm, LOF local outlier factor algorithm, isolation forest algorithm, quartile algorithm, etc., a data cleaning model is constructed to achieve the cleaning of outlier data, lost data, delayed data, and invalid data, and restore the authenticity of equipment operation data" solution.

[0020] 2. The present invention uses a newly developed data sampling method to sample positive sample data under various normal operating conditions, constructs a positive sample space, and is convenient for viewing data trends and calculating the deviation of runtime states. The sampled samples cover the state quantity mode under all working conditions, the sampling results can conform to the data trend of the original data, and the time series satisfies the unbiased estimation in probability statistics, while improving the operation efficiency and result accuracy of the early warning prediction algorithm.

[0021] 3. The present invention uses a newly developed data processing method to adaptively select a suitable algorithm, realize automatic deduplication, fill in delayed data, lost data, and invalid data, and automatically clean data such as mutations, outliers, dead points, and noise. It can denoise or smooth feature data containing high-frequency noise to avoid false alarms and improve the accuracy and robustness of the model. It can perform dimensionality reduction processing on multi-dimensional redundant signals to eliminate the uncertainty brought to the prediction results by redundant information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a flow chart of steps of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Figure 2 This is a structural block diagram of a data processing system based on a giant hydro turbine unit provided by an embodiment of the present invention; Figure 3 is a structural schematic diagram of a computer device provided by an embodiment of the present invention; Figure 4 A data source management diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Figure 5 A measuring point management diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Figure 6 It is a data set management diagram of a data processing method based on a giant water turbine unit provided by an embodiment of the present invention; Figure 7 This is a data example diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Figure 8 This is a generalized smoothing weighted sampling effect diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig. 9 It is an irregular sampling diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.10 It is a sample standard deviation principle diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.11 is a relationship diagram between a confidence level and a confidence interval of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.12 A confidence interval estimation of a data processing method based on a giant hydro turbine unit is provided in one embodiment of the present invention; Fig.13It is a raw data diagram of upper guide blade temperature of a data processing method based on a giant water turbine unit provided by an embodiment of the present invention; Fig.14 It is a normal sample upper guide tile temperature data diagram of a data processing method OCSVM sampling based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.15 It is a sample diagram after OCSVM sampling of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.16 It is a processing diagram of averaging sliced ​​data of a data processing method based on a giant water turbine unit provided by an embodiment of the present invention; Fig.17 A data processing method for cleaning simulation data based on a giant water turbine unit provided by an embodiment of the present invention; Fig.18 It is an outlier data cleaning diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.19 It is a delayed data cleaning diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig. 20 It is a diagram of incomplete data cleaning of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention; Fig.21 It is an invalid data cleaning diagram of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] Embodiment 1: Reference Figure 1 , shows a flow chart of the steps of a data processing method based on a giant hydro turbine unit provided by an embodiment of the present application, which specifically includes the following steps: S110, acquiring state quantity data of giant water turbine unit equipment, and performing preprocessing according to the state quantity data to obtain preprocessed data; S120, performing sampling processing according to the pre-processed data to obtain sampled data; S130, performing cleaning processing on the sampled data to obtain processed data.

[0025] Next, the data processing method based on the giant hydro turbine unit in this exemplary embodiment is further described.

[0026] As described in the above step S110, the state quantity data of the giant hydraulic turbine unit equipment is acquired, and preprocessing is performed according to the state quantity data to obtain preprocessed data.

[0027] In an embodiment of the present invention, the specific process of step S110 can be further described in combination with the following description, including the following steps: S111, performing temporal sorting according to the state quantity data to obtain sorted data; S112, performing homologous compression processing on the sorted data to obtain the pre-processed data.

[0028] In an embodiment of the present invention, the specific process of step S111 may be further explained in combination with the following description.

[0029] As an example, each hydro-turbine equipment data is sorted in chronological order according to the timestamp in the state quantity data; if there is no state quantity with a timestamp, the hydro-turbine equipment data is deleted. The state quantity is a collection of multiple hydro-turbine equipment data.

[0030] In an embodiment of the present invention, the specific process of step S112 may be further explained in combination with the following description.

[0031] In a specific embodiment, duplicate data in the sorted data is determined according to the state, time and amplitude of each data in the sorted data; the duplicate data in the sorted data is eliminated to obtain first data; adjacent data with the same amplitude and state in the first data list is determined according to the state and amplitude of each data in the first data, and the adjacent data with the same amplitude and state is compressed to obtain the preprocessed data. The compression here can be random elimination.

[0032] As described in the above step S120, the following steps are included: S121, performing acquisition management according to the acquisition platform based on the pre-processed data to obtain the acquired data; S122: Perform sampling analysis based on the collected data to obtain the sampled data.

[0033] In an embodiment of the present invention, the specific process of step S121 may be further explained in combination with the following description.

[0034] In a specific embodiment, the data collection management of the measuring point mainly comes from the industrial Internet platform of the hydropower station, and data access is achieved through the interface. The management of data sources, data measuring points, and data sets are completed. The specific functions of the entire application are as follows: through MQTT, Kafka, Restful and other interfaces, the interface interaction with the industrial Internet platform of the hydropower station is completed. At the same time, it has a data caching mechanism to ensure the integrity of the data interaction. Data source management such as Figure 4 As shown in the figure, the measurement point management is as follows Figure 5 As shown, dataset management is as follows Figure 6 shown.

[0035] As described in the following steps, the specific operation process of step S122 is as follows: 1) judging the data properties of the collected data according to the collected data; wherein the data properties include time properties, abnormal properties and uneven properties; 2) The sampling data is obtained by matching the sampling method according to the data properties.

[0036] In an embodiment of the present invention, the specific process of the step of "determining the data property of the collected data according to the collected data" can be further explained in combination with the following description.

[0037] As described in the following steps, the time weight, abnormal weight and uneven weight of the data are obtained based on the collected data; and the data properties are obtained based on the time weight, abnormal weight and uneven weight.

[0038] In a specific embodiment, among the time weight, abnormal weight and uneven weight, the numerical proportions of the three are compared, and the one with a higher proportion is the data property. For example, if the proportion of the time weight is greater than the abnormal weight and the uneven weight, the data property is the time property; similarly, the abnormal property and the uneven property can be obtained.

[0039] In a specific embodiment, the single property of each data in the collected data is used to obtain the property of the entire data. If the data in the collected data is abnormal, it is abnormal data; the abnormal property takes precedence over the uneven property, and the uneven property takes precedence over the time property. Then, based on the single property of each data, the time weight, abnormal weight and uneven weight are obtained.

[0040] During the operation of a hydropower station, a large amount of high-dimensional data is continuously generated every moment. Some of the data is generated over time and shows obvious time properties. After the data is standardized, if the data distribution range is wide and more than 99.5% of the data is normal data, it shows uneven properties; when more than 0.5% of the data is abnormal data, it shows abnormal properties. Based on the data properties, the corresponding weight is given the largest proportion, and the other two weights are smaller proportions; the more obvious the property performance, the greater the corresponding weight. For example, when 0.5% of the measured point data are abnormal, the abnormal weight of the measured point data is 0.5, and the time weight and uneven weight are 0.25 respectively; when 1% of the measured point data are abnormal, the abnormal weight is 0.6, and the time weight and uneven weight are 0.2 respectively; when all the measured point data are concentrated within 1 standard deviation, the time weight is 0.9, and the uneven weight and abnormal weight are 0.05 respectively; when 5% of the measured point data are within 1-2 standard deviations, and the rest are concentrated within 1 standard deviation, the time weight is 0.8, and the uneven weight and abnormal weight are 0.1 respectively. Perform statistical analysis on the weights of all measured point data to obtain the time weight, abnormal weight and uneven weight of the entire data, and compare the numerical proportions of the time weight, abnormal weight and uneven weight. The one with a higher proportion is the data nature of the entire data. In an embodiment of the present invention, the specific process of the step of "obtaining the sampling data according to the data property matching sampling method" can be further explained in combination with the following description.

[0041] As described in the following steps, if the data property is a time property, the sampled data is obtained through generalized smoothing weighted sampling; if the data property is an abnormal property, the sampled data is obtained through spatial sampling; if the data property is an uneven property, the sampled data is obtained through irregular sampling.

[0042] In a specific embodiment, for the health analysis of turbine equipment, it is necessary to use data to speak, and to realize the mining of hidden dangers through the combination of big data analysis technology and AI algorithm, to use as little expert experience as possible, to save the time of model implementation, and to truly realize the intelligent and smart application of hydropower stations. With the enhancement of the nonlinear fitting ability of AI algorithms, it provides a model possibility for mining the implicit laws of data. The improvement of big data computing power, storage capacity, and data compression capacity makes large-scale data calculation possible. According to technical experience, it can currently support PB-level data. In particular, data compression capabilities can increase data storage and backup cycles. Although big data storage has strong compression capabilities, data sampling is still essential in practical applications for the following reasons: On the one hand, from a business perspective, when visualizing the data trend of the status quantity, if sampling is not performed, only the trend within a short period of time can be queried, such as 1 hour. If the data of a day, a week, or a month is to be analyzed, the lag will be very obvious, resulting in a poor user experience. On the other hand, from the perspective of big data analysis, when constructing the positive sample space, since there are a large number of positive samples under different working conditions and the data similarity is high, if sampling is not performed, it will lead to a lot of redundant calculations.

[0043] Based on the above reasons, a reasonable sampling method must be adopted to construct the positive sample space. By analyzing the distribution of the positive sample space and combining the various working conditions of the turbine unit, a sampling method is designed. The following examples illustrate the difficulties and problems in the sampling process: Figure 7 It can be seen that because the number of data points is too dense and the fluctuations are very frequent, all data points will overlap after being connected in sequence by broken lines, and the visualization effect is poor. At this time, it is necessary to sample the original data points, for example, sampling 10,000 original data points into 200 data points. It is crucial to adopt a specific sampling method.

[0044] In a specific embodiment, if the data property is a time property, the sampled data is obtained by generalized smoothing weighted sampling; specifically, generalized smoothing weighted sampling (GSW), a hydropower station generates a large amount of high-dimensional data every day, and such data is naturally related to timestamps. Real-time predictive analysis can be carried out in two steps. First, the data portion to be focused on and the metric to be predicted are specified by slicing, segmenting and aggregating the data; second, a predictive model is trained based on the aggregation results to understand and predict the trend of the specified metric. The first step is to provide a performance bottleneck for real-time response. GSW sampling obtains an approximate set of original data as input for training the predictive model, and analyzes the error bound of estimating aggregation using GSW sampling.

[0045] The amount of time series data in the normal sample space is huge, and it is usually necessary to use several months of historical data to predict future data points. Secondly, time series data is very high-dimensional and may have hundreds of attribute combinations. Existing improvement methods mainly reduce the size of input data by sampling historical data, where the sampling algorithms are divided into uniform sampling and weighted sampling. Uniform sampling provides unbiased estimates for aggregation, but the estimation error may be proportional to the range of the measurement; the weighted sampling scheme provides optimal estimates with better error bounds, but a weighted sample must be drawn separately for each measurement. If there are multiple different measurements, the total space consumption may be huge. In order to generate compact samples that can take into account multiple measurements and still have accuracy guarantees, the generalized smoothing weighted GSW sampling method is used: the sampling probability of each sample that meets the constraints is , where △ is a positive constant and w is the sampling weight. The further optimization metric is: , while satisfying for The sampling effect of generalized smoothing weighted sampling (GSW) is as follows: Figure 8 shown.

[0046] In a specific embodiment, if the data property is abnormal, the sampled data is obtained by spatial sampling; spatial sampling (OCSVM), the OCSVM algorithm is a classic anomaly detection algorithm, and its basic principle is similar to that of SVM. Unlike the binary classification problem that SVM focuses on, OCSVM has only one classification, which is exactly what is needed for anomaly detection, and does not focus on abnormal data points. The basic idea of ​​OCSVM is to map data into a high-dimensional space so that normal samples have a higher density in the space, while abnormal samples are relatively isolated. Specifically, OCSVM divides normal samples and abnormal samples by finding a hyperplane, and this hyperplane should be as close to normal samples as possible. The goal of OCSVM is to maximize the margin from the support vector to the hyperplane, that is, to maximize the density of normal samples. In this way, OCSVM can effectively identify abnormal samples because abnormal samples are often far away from the normal sample area.

[0047] The OCSVM algorithm can be divided into the following steps: Data preprocessing: Standardize the input data set so that the data has the same scale in all dimensions to avoid a certain feature having a greater impact on the results. Model building: Based on the preprocessed data set, use the OCSVM algorithm to build a support vector machine model. The model training process is to find the optimal hyperplane so that normal samples are as close to the hyperplane as possible. Find abnormal samples: After training the model, by calculating the distance from each sample to the hyperplane, it is possible to determine which samples are abnormal samples. Samples that are far from the hyperplane are likely to be abnormal samples, while samples that are closer are normal samples. Threshold determination: Based on the distribution of distances, an appropriate threshold can be selected to determine whether a sample is abnormal. Generally, samples whose distance exceeds the threshold are considered abnormal. In the normal sample space, OCSVM uses uniform sampling; in the abnormal sample space, OCSVM uses weighted sampling.

[0048] In a specific embodiment, if the data property is non-uniform, the sampled data is obtained by irregular sampling. Specifically, irregular sampling is also called uneven sampling: a sampling method that combines sampling weights with the individual itself. When the sample is unbalanced: in some cases, the sample types that account for a smaller proportion may have more research value. For example: in terms of equipment health management, data samples caused by sensor detachment, loosening, etc. may account for the majority, but in fact, compared with other turbine unit defects, the value is not great. For irregular sampling, in addition to adopting business rules, clustering can also be used to achieve irregular sampling. First, the elements in the data set are grouped and clustered to construct a sample group that is representative of the whole, and then one or more sample groups are randomly selected as the sample set. The difference between cluster sampling and stratified sampling is that each group of categories distinguished by clustering can represent the whole to a certain extent. There is no need to sample and summarize each group separately. It is sufficient to directly select several groups. The weight distribution is achieved through clustering to achieve irregular sampling. Such as Fig. 9 shown.

[0049] As an example, the data after sampling needs further testing. For the evaluation of sample time series data, a preliminary test is needed for the sample time series data after sampling to determine whether the sample data and the overall data after sampling are statistically unbiased estimates. First of all, it should be determined that the estimator refers to any sample statistic that gives an estimated value for a population parameter, such as the sample mean. Estimated value: refers to a specific value of an estimator calculated from a sample. The population is estimated through a series of calculations on the sample, and this series of calculations is the calculation of statistics. Commonly used estimators are: sample mean, sample variance, and sample standard deviation. Schematics and commonly used methods such as Fig.10 The estimation methods include the following: Point estimate: For any random sample from a measurement population, if a specific value (the mean, variance or standard deviation of a sample) is calculated for a random quantity (e.g., the mean, variance or standard deviation of a sample) to estimate a population parameter (e.g., the mean, variance or standard deviation of a population), then this value is called a point estimate of the population parameter (e.g., the mean, variance or standard deviation of a population). When using point estimates to reflect population parameters, as much additional information as possible should be given to facilitate the evaluation of the accuracy and precision of the estimated value. Accuracy is affected by the measurement method and sampling design; precision is determined by the standard deviation of a sample of fixed size n. The smaller the standard deviation, the more precise it is.

[0050] Interval estimation: Although there is some information about the point estimate and its accuracy and precision, it still fails to jump from the sample to the population, that is, it fails to link the point estimate with the population parameter to be estimated, give the degree of closeness of the estimate to the parameter or determine how much possible error exists in the estimate. In order to infer the population parameter from the sample information, interval estimation is needed. Interval estimation is an inference from a sample to a population. Interval estimation places the population parameter on a real interval. The boundary values ​​of the interval are determined by three factors.

[0051] Sample point estimates; Sample statistics that relate population parameters to sample point estimates (such as the Z statistic, obtained by normal transformation); The sampling distribution of the statistic (for example, if the theoretical sampling distribution of the sample mean follows the normal distribution, then the sampling distribution of the Z statistic is the standard normal distribution).

[0052] Confidence level, significance level estimate By sampling the time series data of hydropower equipment, the confidence level and confidence interval of the time series sampling data are estimated, and the sampling positive sample space satisfies the distribution law of the sample population as an unbiased estimate. The general steps are as follows: Confidence interval estimation, the range of an estimated overall indicator; Confidence level estimation is the requirement for the probability of the estimated result being accurate, also known as confidence level; The significance level is set as 1-confidence level, usually The relationship between confidence level and confidence interval is as follows: Fig.11 shown.

[0053] Confidence interval estimation: Estimating the confidence interval, estimating the probability characteristics of the overall sample through sampling, can be achieved through the following 4 steps, such as Fig.12 shown.

[0054] As an example of sample space screening sampling in an embodiment, for the hydropower unit data sample sampling, the example part mainly adopts the following sampling methods, namely: OCSVM sampling and non-uniform sampling. The overall is as follows: the upper guide tile temperature raw data is as follows Fig.13 shown.

[0055] In a specific embodiment, OCSVM sampling is performed to select the upper guide temperature data of the normal sample under the power generation condition as follows: Fig.14 As shown, the samples after OCSVM sampling are as follows Fig.15 shown.

[0056] In a specific embodiment, the uneven sampling is performed by time under the power generation condition, starting from 0:00 am to 8:00 am, with a 2.5-hour interval as the slice, and then 3 hours, 4 hours, and 5 hours are used for the subsequent slicing. The average value of the sliced ​​data is taken, such as Fig.16 shown.

[0057] Finally, the sampling of positive sample space is achieved through various sampling methods, and the probability confidence is estimated at the same time.

[0058] As described in the above step S130, a cleaning process is performed on the sampled data to obtain processed data.

[0059] In an embodiment of the present invention, the specific process of step S130 can be further described in combination with the following description, as described in the following steps: S131, determining the data type of the sampled data according to the sampled data; wherein the data type includes outlier data, delayed data, incomplete data and invalid data; S132, performing cleaning according to the data type to obtain the processed data.

[0060] In an embodiment of the present invention, the specific process of step S132 may be further explained in combination with the following description.

[0061] As described in the following steps, if the data type is outlier data, the processed data is obtained by an outlier algorithm; if the data type is delayed data, the processed data is obtained by a weighted average algorithm; if the data type is incomplete data, the processed data is obtained by correcting the preceding data; if the data type is invalid data, the processed data is obtained by correcting the preceding data.

[0062] In a specific embodiment, the data cleaning algorithm is selected. For hydropower stations, data cleaning takes a lot of time, and the purpose is to restore the real operation process. For the processing of outlier data, lost data, delayed data, invalid data, etc. in hydropower stations, data cleaning is performed by combining historical data with data such as ledger defects to complete the real restoration of equipment operation data. The algorithm used at this time is as follows: The outlier data uses the following algorithms for comprehensive verification to adapt to different business scenarios: Kmeans outlier algorithm: continuously calculate the Euclidean distance between each sample point and the cluster center point, continuously converge to the optimal state, find the optimal K value, and use the cluster distance as the basis for outlier judgment; KNN outlier algorithm: use the sparsity or honeycomb degree of the adjacent K sample points to determine whether it is an outlier data; LOF local outlier factor algorithm: judge outlier data based on the local density sparsity of the sample; Isolation forest algorithm: The isolation forest will be divided similarly to a decision tree. The more divisions, the stronger the degree of compactness, and the fewer divisions, the higher the probability of being judged as outlier data. Quartile algorithm: remove outliers based on the 3sigma principle of data distribution. In this scenario, this method cannot completely remove outliers and is prone to deleting normal data.

[0063] The delayed data processing adopts the following algorithm: for a small amount of delayed data, based on the strong correlation of the data, the data sequence can be directly completed by using the rules of the previous data through simple conversion (such as weighted average method, etc.).

[0064] The following algorithms are used to handle data loss: For a large amount of missing data, based on the strong correlation of the data, the SVR and CONVLSTM algorithms are used to use the previous data to predict the subsequent data, thereby completing the data sequence.

[0065] The following algorithms are used to process invalid data: Based on the strong correlation of the data, the SVR and CONVLSTM algorithms are used to use the previous data to predict the subsequent data, thereby correcting the data sequence.

[0066] In a specific embodiment, the overall process of the SVR algorithm is explained as follows: The SVR algorithm is a machine learning algorithm with strong nonlinear feature fitting capabilities and high fitting accuracy. In practical applications, SVR is generally used to conduct model tests first. If during the test, it is found that the feature data is dependent on time and space, and the data volume is large, CONVLSTM is used for fitting. Currently, the two algorithms are generally used in combination, and the algorithm accuracy is higher.

[0067] The overall process of the CONLSTM algorithm is explained as follows: For the CONLSTM algorithm, it not only learns the feature correlation memory training of the time dimension and the space dimension, but also has the memory function of the fluctuation of the previous data. It is often used for fitting complex data and is generally used in conjunction with SVR during implementation.

[0068] In a specific embodiment, the entire data cleaning process determines the cleaning rules and algorithm parameters in the historical data. When the algorithm is stable, it is used for real-time data cleaning to update the cleaning of the entire data. This process is automated with little human involvement. The visualization of the entire process has strong business explanation and correctness judgment of cleaning, and users can customize the operation process control. The platform adopts Spark memory computing mode, with extremely high cleaning efficiency, meeting the requirements of data cleaning capacity of not less than 700,000 / time and algorithm duration of not more than 30 minutes.

[0069] As an example, after data cleaning, cleaned data is obtained, and non-stationary processing is performed on the cleaned data to obtain stationary data.

[0070] In a specific embodiment, data non-stationarity processing, for the non-stationarity problem in the measurement point data, uses moving average, weighted average, multi-order difference operator, etc. to stabilize the data, and selects the appropriate operator according to the actual situation. Generally, in the field of hydropower, multi-order difference and weighted average are often used to convert non-stationary data into stable data for intelligent algorithm analysis. Common smoothing algorithms are as follows: Moving average is the simplest data smoothing method used to smooth time series data. It reduces noise by calculating the average of data points within a certain window while retaining the trend of the data. Moving average can be a simple moving average or an exponentially weighted moving average.

[0071] Simple Moving Average (SMA): Simple moving average is a method of smoothing data by calculating the average of data points within a fixed window. The size of the window determines the degree of smoothing. A larger window will result in a smoother curve but will react slower to trends, while a smaller window will be more sensitive to fluctuations in the data.

[0072] Exponentially Weighted Moving Average: Exponentially weighted moving average is a method of smoothing data by applying exponential weights to data points. It gives higher weights to recent data points and lower weights to older data points. This makes EMA more suitable for tracking rapidly changing data.

[0073] Exponential smoothing is a commonly used time series data smoothing and forecasting method for processing data with trends and seasonality. It captures the changing trend of data by assigning different weights to historical data points, assigning higher weights to newer data. Exponential smoothing is often used to generate forecasts, especially when forecasts for future time points are required.

[0074] Local weighted scatter point smoothing, a nonparametric data smoothing method, is often used to fit or smooth scattered data to capture local trends and patterns in the data. Loess smoothing is a local regression method that uses different weights for data in different regions to ensure that a good fit is obtained in each local region of the data.

[0075] As an example, noise processing is performed based on the stationary data.

[0076] In a specific embodiment, for the noise situation of the measurement point, a filtering operator is generally used for processing, such as a moving average operator, an exponential average operator, and a Gaussian filter smoothing operator, to restore the real signal of the time series data and facilitate accurate prediction. Common noise processing techniques are as follows: A low-pass filter is a signal processing tool used to remove high-frequency components in a signal, thereby retaining low-frequency components. In the context of data smoothing, a low-pass filter is used to reduce or remove high-frequency noise or rapid changes in the data, so that the data tends to be smooth and retains slowly changing trends or periodic characteristics.

[0077] Kalman filter, Kalman filter is a powerful mathematical and statistical tool for state estimation and data fusion. Kalman filter is a recursive, optimal state estimation method, which is usually used to process noisy time series data to estimate the state of the data while providing the uncertainty of the estimate.

[0078] Savitzky-Golay filter,Savitzky-Golay filtering is a signal processing technique used to smooth and denoise discrete data sequences. It is a linear smoothing filter that estimates the mean of data points by fitting a polynomial to reduce noise and sudden fluctuations.

[0079] In a specific embodiment, a data cleaning example simulates real data such as Fig.17 As shown. For outlier data cleaning, take K-means outlier algorithm data cleaning as an example, Fig.18 Delayed data cleaning is shown in Fig.19 As shown; incomplete data cleaning, such as Fig. 20 As shown; invalid data cleaning, such as Fig.21 shown.

[0080] The above cleaning process is all run on the Spark platform with a fast running speed, processing 300,000 to 500,000 data points per minute.

[0081] In a specific embodiment, the beneficial effects and invention points of data sampling are as follows: the newly developed data sampling method is used to sample positive sample data under various normal operating conditions, and a positive sample space is constructed to facilitate viewing data trends and calculating the deviation of runtime states. The sampled samples cover the state quantity mode under all working conditions, the sampling results can conform to the original data trend, the time series satisfies the unbiased estimation in probability statistics, and can improve the accuracy of the algorithms related to the early warning prediction algorithm.

[0082] In a specific embodiment, the beneficial effects of data cleaning and the main points of the invention are as follows: the filling of delayed data, lost data, and invalid data is realized; the data repeated on the time axis of the measuring point can be automatically deduplicated; the data such as mutations, outliers, dead points, and noise can be cleaned, and different algorithms are used according to different scenarios to compare the cleaning effects; the feature data containing high-frequency noise can be denoised or smoothed to avoid false alarms and improve the accuracy and robustness of the model; the custom settings based on the adopted frequency can be realized to realize the downsampling task and complete the data conversion function; the multi-dimensional redundant signals can be reduced in dimension to eliminate the uncertainty brought by redundant information to the prediction results.

[0083] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0084] For the health analysis of turbine equipment, it is necessary to use data to speak, and to mine hidden dangers through big data analysis technology combined with AI algorithms, to use as little expert experience as possible, to save model implementation time, and to truly realize the intelligent and smart application of hydropower stations. With the improvement of big data computing power, storage capacity, data compression capacity, and the enhancement of AI algorithm nonlinear fitting ability, large-scale data computing and high-value data analysis and mining have become possible. Although big data storage and compression capabilities are strong, data sampling is still indispensable in practical applications for the following reasons: On the one hand, from a business perspective, when visualizing the data trend of the state quantity, if sampling is not performed, only the trend in a short period of time can be queried, such as 1 hour. If the data of one day, one week, or one month is to be analyzed, the jam is very obvious, which brings a bad user experience; on the other hand, from the perspective of big data analysis, when constructing the positive sample space, there are many positive samples under different working conditions and the data similarity is large. If sampling is not performed, it will bring a lot of redundant calculations. Based on the above reasons, a reasonable sampling method must be adopted to construct the positive sample space. The sampling method is designed by analyzing the distribution of the positive sample space and combining the various working conditions of the turbine unit.

[0085] At present, the commonly used methods for data sampling include generalized smoothing weighted sampling (GSW), reservoir sampling algorithm, irregular sampling, LTTB sampling, etc. However, these methods have shortcomings such as large sampling deviation and incomplete sampling, which affect the accuracy of subsequent data analysis. Therefore, it is necessary to explore and develop more advanced and more accurate sampling methods to achieve accurate sampling of data samples.

[0086] During the operation of power plants, more and more production data with higher and higher accuracy are generated, and the amount of data stored in historical databases has expanded dramatically. In such a large amount of production data, there are outlier data, missing data, delayed data, invalid data, etc. These data need to be cleaned to restore the real operation process of the equipment. At present, there are relatively few methods for cleaning data, and they mainly rely on expert experience for judgment. However, expert experience is limited and it is impossible to exhaust all experience. Moreover, relying solely on expert experience to judge is not only time-consuming and labor-intensive, but also ineffective. Therefore, it is necessary to explore and develop intelligent data cleaning methods to achieve the completion, deletion, and correction of outlier data, missing data, delayed data, invalid data, etc.

[0087] At present, big data sampling has the following problems: data sampling fails or sampling samples are not representative due to the instability of data sources; the large amount of data makes sampling laborious and slow; errors and incomplete low-quality data caused by various reasons lead to poor data sampling quality; data from different sources may have various format differences, making it difficult to process them uniformly, resulting in sampling difficulties; there is uncertainty in the source of data, and data sampling needs to be carried out from multiple channels. There are also huge differences in the data quality and effectiveness of these channels, which makes sampling difficult; large-scale data sampling requires a lot of manpower, material and financial resources, resulting in high sampling costs.

[0088] Currently, there are problems with data cleaning. There are many sources of data and application scenarios, and it takes a lot of time and effort to understand the source, purpose and significance of the data. It is also difficult to formulate standards and rules for data cleaning: how to determine which are outlier data, missing data, delayed data, invalid data, and how to delete invalid data, complete missing data, and restore the authenticity of the data is also a more complex and difficult problem to handle: due to outliers caused by errors or abnormal situations, how to determine a suitable standard to distinguish normal and abnormal data, how to identify and process outliers is also a challenge; for non-stationary data in the measurement point data, what method to use to convert non-stationary data into stationary data and conduct intelligent algorithm analysis is also a major problem; for the noise situation of the measurement point data, how to reduce noise, restore the real signal of the time series data, and facilitate accurate prediction is also a major challenge.

[0089] As more and more production data is generated during the operation of power plants, the amount of data stored in historical databases has expanded dramatically. In this case, how to accurately sample data and how to clean existing outlier data, missing data, delayed data, and invalid data requires advanced solutions. For data sampling, uniform sampling, stratified random sampling, OCSVM sampling, and non-uniform sampling methods are proposed. For data cleaning, it is proposed that the Kmeans outlier algorithm, KNN outlier algorithm, LOF local outlier factor algorithm, isolation forest algorithm, and quartile algorithm are used to clean outlier data; the delayed data is cleaned by using the strong correlation of the data and the law of the previous data through simple conversion (for example: weighted average method); the SVR and CONVLSTM algorithms are used to clean the missing data; the SVR and CONVLSTM algorithms are used to clean the invalid data.

[0090] Through uniform sampling, stratified random sampling, OCSVM sampling, and non-uniform sampling methods, accurate sampling of samples in huge data is achieved and a positive sample space is constructed. By using algorithms such as Kmeans outlier algorithm, KNN outlier algorithm, LOF local outlier factor algorithm, isolation forest algorithm, and quartile algorithm, a data cleaning model is constructed to clean outlier data, lost data, delayed data, and invalid data, and restore the authenticity of equipment operation data.

[0091] In the embodiments of the present application, compared with the prior art, which states that "data cannot be accurately sampled, there are large sampling deviations, sampling is not comprehensive enough, and the accuracy of later data analysis is affected; at the same time, invalid data is prone to be unable to be supplemented, deleted, or corrected, etc.", the present invention provides a solution of "accurate sampling of samples in huge data is achieved through uniform sampling, stratified random sampling, OCSVM sampling, and non-uniform sampling methods, and a positive sample space is constructed. By using algorithms such as Kmeans outlier algorithm, KNN outlier algorithm, LOF local outlier factor algorithm, isolated forest algorithm, and quartile algorithm, a data cleaning model is constructed to clean outlier data, lost data, delayed data, and invalid data, and restore the authenticity of equipment operation data", specifically: obtaining the state quantity data of giant hydro turbine equipment, and pre-processing the state quantity data to obtain pre-processed data; sampling the pre-processed data to obtain sampled data; and cleaning the sampled data to obtain processed data. The present invention samples positive sample data under various normal operating conditions by using a newly developed data sampling method, and constructs a positive sample space, which is convenient for viewing data trends and calculating the deviation of runtime states. The sampling samples cover the state quantity mode under all working conditions. The sampling results can conform to the original data trend, and the time series can meet the unbiased estimation in probability statistics, while improving the accuracy of the early warning prediction algorithm. It realizes the filling of delayed data, lost data, and invalid data; it can automatically remove the repeated data of the measuring points on the time axis; it can clean the data of mutations, outliers, dead points, noise, etc., and use different algorithms to compare the cleaning effects according to different scenarios; it can realize the denoising or smoothing of feature data containing high-frequency noise to avoid false alarms and improve the accuracy and robustness of the model; it realizes the custom setting based on the frequency of use, realizes the downsampling task, and completes the data conversion function; it can reduce the dimension of redundant signals in multiple dimensions to eliminate the uncertainty brought by redundant information to the prediction results.

[0092] Embodiment 2: like Figure 2 As shown, a data processing system based on a giant hydro turbine unit provided by an embodiment of the present application is shown, which specifically includes the following modules: The preprocessing data module 210 is used to obtain the state quantity data of the giant hydro turbine unit equipment, and preprocess the state quantity data to obtain preprocessing data; A sampling module 220, configured to perform sampling processing according to the pre-processed data to obtain sampled data; The data processing module 230 is used to perform cleaning processing on the sampled data to obtain processed data.

[0093] In one embodiment of the present invention, the data preprocessing module 210 includes: Sorting data submodule: used to perform temporal sorting according to the state quantity data to obtain sorted data; Compression processing submodule: used for performing homologous compression processing on the sorted data to obtain the pre-processed data.

[0094] In one embodiment of the present invention, the sampling module 220 includes: Data collection submodule: used to collect and manage the pre-processed data according to the acquisition platform to obtain the collected data; Sampling submodule: used to perform sampling analysis based on the collected data to obtain the sampled data.

[0095] In one embodiment of the present invention, the sampling submodule includes: Data property submodule: used to determine the data property of the collected data according to the collected data; wherein the data property includes time property, abnormal property and uneven property; Matching sampling method submodule: used to match the sampling method according to the data properties to obtain the sampling data.

[0096] In one embodiment of the present invention, the matching sampling method submodule includes: Time property submodule: used for obtaining the sampled data by generalized smoothing weighted sampling if the data property is a time property; Abnormal property submodule: used for obtaining the sampled data through spatial sampling if the data property is abnormal; Non-uniform property submodule: used for obtaining the sampled data by irregular sampling if the data property is non-uniform.

[0097] In one embodiment of the present invention, the data processing module 230 includes: Data type submodule: used to determine the data type of the sampled data according to the sampled data; wherein the data type includes outlier data, delayed data, incomplete data and invalid data; The data cleaning submodule is used to clean the data according to the data type to obtain the processed data.

[0098] In one embodiment of the present invention, the data cleaning submodule includes: Outlier data submodule: used for obtaining the processed data by using an outlier algorithm if the data type is outlier data; Delayed data submodule: used for obtaining the processed data by a weighted average algorithm if the data type is delayed data; Incomplete data submodule: used for obtaining the processed data by correcting the preceding data if the data type is incomplete data; Invalid data submodule: used for obtaining the processed data by correcting the preceding data if the data type is invalid data.

[0099] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0100] There are repeated operation steps in this specific embodiment and the above specific embodiment, and this specific embodiment is only briefly described. For other solutions, refer to the description of the above specific embodiment.

[0101] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0102] Embodiment 3: Reference Figure 3 , shows a computer device of the present application based on a data processing method of a giant hydro turbine unit, which may specifically include the following: The computer device 12 is in the form of a general-purpose computing device, and the components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 connecting different system components (including the memory 28 and the processing unit 16).

[0103] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, an Audio Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0104] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0105] Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 3 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of the various embodiments of the present application.

[0106] A program / utility 40 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0107] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, cameras, etc.), one or more devices that enable an operator to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an I / O interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs)), wide area networks (WANs), and / or public networks (e.g., the Internet) through a network adapter 20. Figure 3 As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.

[0108] The processing unit 16 executes various functional applications and data processing by running the programs stored in the memory 28, for example, implementing a data processing method based on a giant hydro turbine unit provided in an embodiment of the present application.

[0109] That is, when the processing unit 16 executes the program, it realizes: acquiring an image set, and obtaining a database based on the image set; wherein the image set includes inspection data, image data and report data; performing data cleaning based on the database to generate a standard database; and when receiving a scheduling data task, performing scheduling in the standard database based on the scheduling data task to generate a scheduling image.

[0110] Embodiment 4: In an embodiment of the present application, the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a data processing method based on a giant hydro turbine unit as provided in all embodiments of the present application is implemented.

[0111] That is, when the program is executed by the processor, it is implemented as follows: acquiring an image set, and obtaining a database based on the image set; wherein the image set includes inspection data, image data, and report data; performing data cleaning based on the database to generate a standard database; and when a scheduling data task is received, performing scheduling in the standard database based on the scheduling data task to generate a scheduling image.

[0112] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, system or device.

[0113] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, system, or device.

[0114] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as an independent software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the operator's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0115] Although the preferred embodiments of the present application have been described, once the basic creative concept is known to those skilled in the art, additional changes and modifications may be made to these embodiments. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application. Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or still includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0116] The above is a detailed introduction to the data processing method, system, device and storage medium based on a giant turbine unit provided by the present application. The present invention uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope, and the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method based on a giant hydro turbine unit, characterized in that: The method is used to process big data based on the state quantity of giant water turbine equipment, and comprises the following steps: S110, acquiring state quantity data of giant water turbine unit equipment, and performing preprocessing according to the state quantity data to obtain preprocessed data; S120, performing sampling processing according to the pre-processed data to obtain sampled data; S130, performing cleaning processing on the sampled data to obtain processed data.

2. A data processing method based on a giant hydro turbine unit according to claim 1, characterized in that: In the step S110, preprocessing is performed according to the state quantity data to obtain preprocessed data, including the following steps: S111, performing temporal sorting according to the state quantity data to obtain sorted data; S112, performing homologous compression processing on the sorted data to obtain the pre-processed data.

3. A data processing method based on a giant hydro turbine unit according to claim 2, characterized in that: In the step S111, the state quantity data is a collection of multiple turbine unit equipment data, and the time series sorting based on the state quantity data includes: sorting each turbine unit equipment data in chronological order based on the timestamp in the state quantity data; if there is no state quantity with a timestamp, the turbine unit equipment data is deleted.

4. A data processing method based on a giant hydro turbine unit according to claim 2, characterized in that: In the step S112, performing homologous compression processing on the sorted data to obtain the pre-processed data includes: 1) Determine the repeated data in the sorted data according to the state, time and amplitude of each data in the sorted data; 2) Eliminate duplicate data in the sorted data to obtain the first data; 3) According to the state and amplitude of each data in the first data, adjacent data with the same amplitude and state in the first data list are determined, and the adjacent data with the same amplitude and state are compressed to obtain preprocessed data.

5. The data processing method based on a giant hydro turbine unit according to claim 1 is characterized in that: In the step S120, performing sampling processing according to the pre-processed data to obtain the sampled data includes: S121, performing acquisition management according to the acquisition platform based on the pre-processed data to obtain the acquired data; S122: Perform sampling analysis based on the collected data to obtain the sampled data.

6. A data processing method based on a giant hydro turbine unit according to claim 5, characterized in that: The step S122 includes the following contents: 1) Determine the data properties of the collected data based on the collected data, wherein the data properties include time properties, abnormal properties and uneven properties; 2) The sampling method is matched according to the data properties to obtain the sampling data.

7. A data processing method based on a giant hydro turbine unit according to claim 6, characterized in that: The operation method of obtaining sampling data by matching the sampling method according to the data property is: If the data property is a time property, the sampled data is obtained by generalized smoothing weighted sampling; If the data property is abnormal, obtaining the sampled data through spatial sampling; If the data property is non-uniform, the sampled data is obtained by irregular sampling.

8. The data processing method based on a giant hydro turbine unit according to claim 1 is characterized in that: In the step S130, cleaning is performed on the sampled data to obtain processed data, including the following steps: S131, determining the data type of the sampled data according to the sampled data, the data type including outlier data, delayed data, incomplete data and invalid data; S132, performing cleaning according to the data type to obtain the processed data.

9. A data processing method based on a giant hydro turbine unit according to claim 8, characterized in that: In step S132, the processed data is cleaned according to the data type, including the following contents: If the data type is outlier data, obtaining the processed data by using an outlier algorithm; If the data type is delayed data, obtaining the processed data by a weighted average algorithm; If the data type is incomplete data, the processed data is obtained by correcting the preceding data; If the data type is invalid data, the processed data is obtained by correcting the preceding data.

10. A data processing system based on a giant hydro turbine unit, characterized in that: The data processing method based on a giant hydraulic turbine set according to any one of claims 1 to 9 is used to process big data on the state quantity of the giant hydraulic turbine set equipment, comprising: A preprocessing data module is used to obtain state quantity data of giant water turbine equipment and perform preprocessing according to the state quantity data to obtain preprocessing data; A sampling module, used for performing sampling processing according to the pre-processed data to obtain sampling data; The data processing module is used to perform cleaning processing on the sampled data to obtain processed data.

11. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data processing method based on a giant hydro turbine group are implemented as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method based on a giant hydro turbine set are implemented as described in any one of claims 1 to 9.

Images