Intelligent cleaning method and system for hydropower station operation data

Abstract

The present invention provides an intelligent cleaning method and system for operation data of a hydropower station, relates to the technical field of data cleaning, and solves the problem of inaccurate operation data of a hydropower station caused by repeated information, erroneous information and inconsistent information in a data collection process. The method comprises: obtaining original data in the operation process of the hydropower station, and preprocessing the data based on a feedforward neural network model and applying a wavelet transform method; by constructing a node heterogeneous model, encoding and converting the preprocessed original data to form a data format for data analysis and integration and temporarily storing it; integrating multiple original data obtained from multiple data sources into one data set, and applying a local similarity regression algorithm LSR and a random forest algorithm RF to implement data cleaning operations; verifying and storing the operation data obtained after cleaning. The present invention can effectively ensure the quality and accuracy of the data, and improve the practical value and management efficiency of the operation data of the hydropower station.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and is applied to the processing of hydropower station data, and in particular to an intelligent cleaning method and system for hydropower station operation data. Background Art

[0002] Intelligent hydropower stations are the result of upgrading traditional hydropower stations using advanced information technology and automation control technology, aiming to improve their operating efficiency, safety and reliability. By integrating sensors, intelligent instruments and monitoring systems, intelligent hydropower stations can monitor multiple key operating parameters such as water level, water pressure, water quality, voltage, current, etc. in real time, and realize remote control and adjustment of equipment.

[0003] Intelligent hydropower stations also use big data analysis and artificial intelligence technology to process massive amounts of operating data, predict potential problems and fault points, and provide optimized operating strategies to improve operating efficiency and reduce operating costs. Automated operation and maintenance is another important component, which not only reduces the manpower burden, but also improves equipment reliability and operation and maintenance efficiency. In addition, through intelligent scheduling and optimized control methods, hydropower stations can achieve energy conservation and emission reduction while ensuring power generation needs, reducing pressure on the environment.

[0004] However, in practical applications, intelligent hydropower stations also face a series of challenges. In particular, in the daily monitoring process, the need to collect data from a large number of intelligent devices will bring about problems of data complexity and diversity. Specifically, when the amount of collected data is large and diverse, duplicate information, erroneous information, and inconsistent information may appear. These problems will not only question the accuracy of the collected data, but also affect the effectiveness of decisions and operations based on these data.

[0005] Therefore, how to effectively solve the problems of duplicate information, erroneous information and inconsistency in the data collection process has become a technical problem that needs to be overcome in the development of intelligent hydropower stations. Solving this problem is crucial to improving the overall performance of intelligent hydropower stations and ensuring their long-term stable operation. With the continuous advancement of the Internet of Things, big data and artificial intelligence technologies, exploring new methods to meet the above challenges will be one of the key directions of future research on intelligent hydropower station technology. Summary of the invention

[0006] Based on the current status of the background technology, the purpose of the present invention is to solve the problem of inaccurate hydropower station operation data caused by repeated information, erroneous information and inconsistent information in the data collection process, so a method and system for intelligent cleaning of hydropower station operation data are proposed. After application, the method of the present invention can not only ensure the quality and accuracy of the data, but also realize subsequent data analysis and mining based on the data verification process, so as to better ensure the readability, modifiability and maintainability of the hydropower station operation data.

[0007] The present invention adopts the following technical solutions to achieve the purpose:

[0008] An intelligent cleaning method for hydropower station operation data comprises the following steps:

[0009] S1. Obtain the original data of the hydropower station operation process by deploying intelligent monitoring and collection equipment in the hydropower station;

[0010] S2, based on the feedforward neural network model and applying the wavelet transform method, the acquired raw data is preprocessed;

[0011] S3. By constructing a node heterogeneous model, the pre-processed raw data is encoded and converted to form a data format for data analysis and integration and temporarily stored;

[0012] S4, multiple original data sets obtained from multiple data sources are integrated into one data set, and a local similarity regression algorithm LSR and a random forest algorithm RF are applied to the data set to implement data cleaning operations;

[0013] S5. Verify the operation data obtained after cleaning, and perform statistical analysis, logic check, rule check and data comparison operations in a visual manner;

[0014] S6. After completing the verification of the operation data, a preset storage format is selected for the operation data, and the data is stored in the hydropower station operation database, waiting for the call of the business application.

[0015] Furthermore, in step S2, the data preprocessing process of the original data includes the following steps:

[0016] S21, performing a cyclic elimination operation, inputting the original data into the feedforward neural network model, and having the feedforward neural network model output a data sample in which the error of the original data exceeds a preset threshold, and cyclically eliminating the data sample;

[0017] S22, performing filtering and denoising operations, removing clutter signals and noise signals from the original data after cyclic elimination through filtering and denoising;

[0018] S23, performing a missing filling operation, for the original data that has undergone cyclic elimination and filtering denoising, filling the missing values ​​generated in the original data due to the cyclic elimination and filtering denoising.

[0019] Furthermore, in step S4, for the data set, the data cleaning operation includes the following steps:

[0020] S41, calculating the abnormal factor size of all data points corresponding to the data samples in the data set, obtaining the abnormal value of each data point and performing sorting processing;

[0021] S42, using a Gaussian mixture model, for the data points corresponding to the sorted outliers, according to the classification of normal data samples and outlier data samples, performing a clustering operation; using the outlier value corresponding to the dividing line between the normal data samples and the outlier data samples obtained after clustering as an outlier judgment threshold;

[0022] S43, cyclically compare the relationship between the outliers of all data points in the data sample and the outlier judgment threshold, judge the data samples corresponding to the data points whose outliers are smaller than the outlier judgment threshold as normal data samples, otherwise judge them as abnormal data samples, and eliminate all abnormal data samples to form blank missing data samples in the data set; perform a dynamic missing data filling operation based on the data set containing blank missing data samples and the data set in its original state.

[0023] Further, after step S43, the missing data dynamic filling operation includes the following steps:

[0024] S44, using the original data set as a reference, judging the time length of the missing data samples therein, and classifying the missing data samples into two types: local missing and long-term missing; the missing type of the blank missing data samples obtained after the abnormal identification and elimination operation is local missing;

[0025] S45. For the local missing type, the local similarity regression algorithm LSR is used for processing, the data points before and after the missing data sample are used as training data, a regression model is established and trained, and the regression model is used with the local similarity regression algorithm LSR to predict and fill the missing data samples of the local missing type;

[0026] S46. For long-term missing data types, use the random forest algorithm RF for processing, set the number of decision trees, use the data samples with similar lengths before and after the missing data samples as training data, build and train a regression model, and use the regression model equipped with the random forest algorithm RF to predict and fill in the missing data samples of the long-term missing type.

[0027] The present invention also provides an intelligent cleaning system for hydropower station operation data, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the aforementioned intelligent cleaning method for hydropower station operation data.

[0028] In summary, due to the adoption of this technical solution, the beneficial effects of the present invention are as follows:

[0029] The intelligent cleaning method of the present invention first ensures the basic quality of data by selecting reliable data sources, and takes measures to ensure the integrity and accuracy of the collected data. A variety of technologies and corresponding algorithms are applied in the method, so as to preferably perform a series of operations such as data deduplication, pruning, replacement, filling, normalization and missing value processing to improve data quality and ensure its correctness.

[0030] In addition, through the data verification step, the present invention performs field definition, type conversion and encoding conversion on the data, effectively solving the problem of data duplication and conflict, thereby facilitating subsequent data analysis and mining. The entire data cleaning process not only strengthens the data quality and integrity, but also can timely discover and correct anomalies and errors in the data.

[0031] Finally, the present invention selects a suitable storage format and method to save the cleaned data, ensuring that it has good readability, modifiability and maintainability, thereby greatly improving the practical value and management efficiency of the hydropower station operation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the intelligent cleaning method of the present invention;

[0033] Figure 2 A flowchart briefly describing the data preprocessing process of the raw data in the method of the present invention;

[0034] Figure 3 A flowchart briefly describing the data cleaning operation in the method of the present invention;

[0035] Figure 4 The figure is a flow chart of abnormal data early warning operation in the method of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example 1

[0039] An intelligent cleaning method for hydropower station operation data, Figure 1 This is a brief description of the overall process of the method, which can be viewed simultaneously. The steps of the method are summarized as follows:

[0040] S1. Obtain the original data of the hydropower station operation process by deploying intelligent monitoring and collection equipment in the hydropower station;

[0041] S2, based on the feedforward neural network model and applying the wavelet transform method, the acquired raw data is preprocessed;

[0042] S3. By constructing a node heterogeneous model, the pre-processed raw data is encoded and converted to form a data format for data analysis and integration and temporarily stored;

[0043] S4, multiple original data sets obtained from multiple data sources are integrated into one data set, and a local similarity regression algorithm LSR and a random forest algorithm RF are applied to the data set to implement data cleaning operations;

[0044] S5. Verify the operation data obtained after cleaning, and perform statistical analysis, logic check, rule check and data comparison operations in a visual manner;

[0045] S6. After completing the verification of the operation data, a preset storage format is selected for the operation data, and the data is stored in the hydropower station operation database, waiting for the call of the business application.

[0046] In this embodiment, the data collection in step S1 can collect various data generated during the operation of the hydropower station in real time, including but not limited to key indicators such as power generation, water flow speed, water level, etc. This is the first step and foundation of the entire data processing chain.

[0047] In the data preprocessing process of step S2, the feedforward neural network is an artificial neural network that can be used for pattern recognition and classification tasks. Wavelet transform is a signal processing technology that is often used to remove noise or extract useful information. The combination of the two can effectively reduce noise and smooth the data, improving the efficiency and accuracy of subsequent processing steps.

[0048] Step S3 aims to convert the preprocessed data into a data format suitable for in-depth analysis and integration, and temporarily store the data. Then, step S4 applies the local similarity regression algorithm LSR and the random forest algorithm RF to the integrated data set to perform data cleaning; LSR can help discover local regularities in the data, while RF is an integrated learning method that can identify outliers and improve data quality.

[0049] The cleaned data needs to be verified to ensure its accuracy and reliability. This step includes using visualization tools to perform statistical analysis, logic checks, rule checks, and data comparisons to confirm whether the data meets the expected quality standards. Finally, when the data is verified to be correct, it is stored in the hydropower station's operation database according to a pre-set format. The purpose of this is to facilitate the subsequent various business applications to call this data, such as for performance evaluation, fault prediction and other scenarios.

[0050] This method of the present embodiment aims to improve the processing efficiency and quality of hydropower station operation data by using modern data science technology, thereby supporting more efficient decision-making and technical optimization.

[0051] Example 2

[0052] Based on Example 1, this example provides a more detailed introduction and description of some key parts of the method process.

[0053] like Figure 2 As shown, in step S2 of the method, the data preprocessing process of the original data includes the following steps:

[0054] S21, performing a cyclic elimination operation, inputting the original data into the feedforward neural network model, and having the feedforward neural network model output a data sample in which the error of the original data exceeds a preset threshold, and cyclically eliminating the data sample;

[0055] S22, performing filtering and denoising operations, removing clutter signals and noise signals from the original data after cyclic elimination through filtering and denoising;

[0056] S23, performing a missing filling operation, for the original data that has undergone cyclic elimination and filtering denoising, filling the missing values ​​generated in the original data due to the cyclic elimination and filtering denoising.

[0057] In step S21 of this embodiment, the cyclic elimination operation includes a sample elimination process and a sample recall process performed sequentially.

[0058] In the sample elimination process, first use all the original data to train the feedforward neural network model and perform the first cycle; in the first cycle, the existence of abnormal data caused by errors will make the model's accuracy on the original data very low, so the feedforward neural network model will eliminate data samples whose errors exceed the preset threshold from the original data; then use the remaining original data to continue training the feedforward neural network model, and continue to eliminate corresponding data samples in the same elimination method; repeat the iterative cycle until the feedforward neural network model no longer eliminates data samples, then stop training, output the feedforward neural network model at this time, and use the feedforward neural network model to recall the data samples that were mistakenly eliminated in the iterative cycle.

[0059] During the sample recall process, first use the feedforward neural network model that stops training and outputs to test all the original data, and eliminate data samples whose error magnitude is greater than the average error magnitude; then use the remaining original data to continue training the current feedforward neural network model, and continue to eliminate data samples whose error magnitude is greater than the average error magnitude in the remaining original data; repeat this elimination process until the number of remaining data samples in the original data remains stable, calculate the corresponding average error, and obtain the original data that has been cyclically eliminated.

[0060] In step S22 of this embodiment, the filtering process in the filtering and denoising operation is: in the input original data, the continuously acquired data samples are taken as a fixed length of When the intelligent monitoring and acquisition equipment obtains new raw data in real time, the first data sample in the sequence is removed, and then the remaining data samples are moved forward, and the data samples corresponding to the new raw data are inserted into the end of the sequence, and arithmetic average operation is performed to realize filtering processing;

[0061] The denoising process in the filtering denoising operation is: perform wavelet transform on the data samples in the original data, and calculate the wavelet transform coefficients by the following formula: ,Right now:

[0062]

[0063] In the formula, represents the wavelet function, represents the complex conjugate of the wavelet function; represents the original data independent variable value, represents the stretch factor, represents the translation factor, Indicates the corresponding data after the original data is translated; Represents the useful data samples in the original data. Based on the useful data samples, the noisy data samples are converted It is expressed as the following formula:

[0064]

[0065] In the formula, ; represents the standard deviation of the noise figure, represents a random sample of noise data.

[0066] The larger the absolute value of the calculated wavelet transform coefficient is, the higher the probability that the corresponding data sample is a useful data sample is. Therefore, the distinction threshold is continued to be set to determine the noisy data samples in the original data. After the calculation of the wavelet transform coefficient is completed, the wavelet transform coefficients corresponding to the high-frequency levels in the data samples of the original data are processed through the distinction threshold. After the data samples of the original data are processed, the corresponding data samples are reconstructed through the inverse wavelet transform method to achieve denoising.

[0067] In step S3 of the method, firstly, an encoder is used to encode the preprocessed raw data, and then a node heterogeneous model of the raw data is constructed, and the encoded raw data is input into the node heterogeneous model for conversion; in the node heterogeneous model, a total of nodes, representing the size of each data sample in the original data; the number of data samples passed by each transmission task in the original data conversion process is recorded as After that, the data transmission channel model It is expressed as the following formula:

[0068]

[0069] In this model, Indicates the total number of nodes, Indicates the number of copies corresponding to the original data; Indicates that the branch task in the transmission task is at the node The number of data samples processed during the above operation; Indicates the efficiency of task execution; Representation sample The total number of corresponding copies; Indicates that the sample is being processed In the process, the node The number of data samples processed during the above operation; Indicates that at the node The total number of branch tasks processed;

[0070] After the conversion of the original data is completed, it is decoded by a decoder to form a data format for data analysis and integration, and stored in the corresponding memory.

[0071] like Figure 3As shown, in step S4 of the method, for the data set, the data cleaning operation includes the following steps:

[0072] S41, calculating the abnormal factor size of all data points corresponding to the data samples in the data set, obtaining the abnormal value of each data point and performing sorting processing;

[0073] S42, using a Gaussian mixture model, for the data points corresponding to the sorted outliers, according to the classification of normal data samples and outlier data samples, performing a clustering operation; using the outlier value corresponding to the dividing line between the normal data samples and the outlier data samples obtained after clustering as an outlier judgment threshold;

[0074] S43, cyclically compare the relationship between the outliers of all data points in the data sample and the outlier judgment threshold, judge the data samples corresponding to the data points whose outliers are smaller than the outlier judgment threshold as normal data samples, otherwise judge them as abnormal data samples, and eliminate all abnormal data samples to form blank missing data samples in the data set; perform a dynamic missing data filling operation based on the data set containing blank missing data samples and the data set in its original state.

[0075] Further, see also Figure 3 As shown in FIG. 4 , after step S43, the operation of dynamically filling missing data includes the following steps:

[0076] S44, using the original data set as a reference, judging the time length of the missing data samples therein, and classifying the missing data samples into two types: local missing and long-term missing; the missing type of the blank missing data samples obtained after the abnormal identification and elimination operation is local missing;

[0077] S45. For the local missing type, the local similarity regression algorithm LSR is used for processing, the data points before and after the missing data sample are used as training data, a regression model is established and trained, and the regression model is used with the local similarity regression algorithm LSR to predict and fill the missing data samples of the local missing type;

[0078] S46. For long-term missing data, use the random forest algorithm RF to process it, and reasonably set the number of decision trees to reduce the algorithm operation time; use the data samples with similar lengths before and after the missing data samples as training data, establish a regression model and train it, and use the regression model equipped with the random forest algorithm RF to predict and fill in the missing data samples of the long-term missing type.

[0079] At the same time, as a preferred content of this embodiment, before step S4, an abnormal data warning operation is also performed, using a CNN-LSTM neural network prediction model combined with the Laida criterion to warn of abnormal data in the obtained operating data.

[0080] In this embodiment, Figure 4 As shown in the figure, the specific process of abnormal data warning operation is as follows: obtain historical data during the operation of the hydropower station to form a historical sample set as the input data of the CNN-LSTM neural network prediction model; divide the historical sample set into a training set, a validation set, and a test set, and perform cyclic training on the CNN-LSTM neural network prediction model. After the training is completed, an intermediate model is obtained;

[0081] The changes in historical data are predicted through the intermediate model to obtain the corresponding predicted value. At the same time, the true value of the historical data corresponding to the predicted value is obtained. The predicted value and the true value are calculated through residual calculation to obtain the predicted residual value, and then the standard deviation and mean of the predicted residual value are calculated. The intermediate model is optimized, and the abnormal data early warning model is obtained after applying the Laida criterion. The Laida criterion is widely used to identify and eliminate abnormal data to ensure the purity of the data set and the effectiveness of the analysis results.

[0082] Determine the warning sample set that needs to be warned from the historical data, input it into the abnormal data warning model, and obtain the verification prediction set output by the model; compare the original data temporarily stored in step S3 with the data in the verification prediction set to determine whether the original data falls within the data range in the verification prediction set, thereby realizing the abnormal data warning operation. Specifically, the verification prediction set output by the model can be characterized as a data range that does not require warning, that is, if the original data does not fall within the data range in the verification prediction set, it means that an abnormal situation has occurred, and an early warning is required to inform relevant personnel to handle it. Otherwise, it means that the original data corresponds to a normal situation, and the subsequent method steps can be continued.

[0083] The overall method of Example 1 and the detailed processing and operation process of each part of the method in this embodiment can be implemented by a computer system, that is, an intelligent cleaning system for hydropower station operation data. This embodiment briefly introduces it as follows: the system includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the intelligent cleaning method for hydropower station operation data.

Claims

1. An intelligent cleaning method for hydropower station operation data, characterized in that: The steps include: S1. Obtain the original data of the hydropower station operation process by deploying intelligent monitoring and collection equipment in the hydropower station; S2, based on the feedforward neural network model and applying the wavelet transform method, the acquired raw data is preprocessed; S3. By constructing a node heterogeneous model, the pre-processed raw data is encoded and converted to form a data format for data analysis and integration and temporarily stored; S4, multiple original data sets obtained from multiple data sources are integrated into one data set, and a local similarity regression algorithm LSR and a random forest algorithm RF are applied to the data set to implement data cleaning operations; S5. Verify the operation data obtained after cleaning, and perform statistical analysis, logic check, rule check and data comparison operations in a visual manner; S6. After completing the verification of the operation data, a preset storage format is selected for the operation data, and the data is stored in the hydropower station operation database, waiting for the call of the business application; In step S2, the data preprocessing process of the original data includes the following steps: S21, performing a cyclic elimination operation, inputting the original data into the feedforward neural network model, and having the feedforward neural network model output a data sample in which the error of the original data exceeds a preset threshold, and cyclically eliminating the data sample; S22, performing filtering and denoising operations, removing clutter signals and noise signals from the original data after cyclic elimination through filtering and denoising; S23, performing a missing value filling operation, for the original data that has undergone cyclic elimination and filtering denoising, filling the missing values ​​generated by the cyclic elimination and filtering denoising in the original data; In step S21, the cyclic elimination operation includes a sample elimination process and a sample recall process performed in sequence; In the sample elimination process, firstly, all the original data are used to train the feedforward neural network model, and the first cycle is performed; in the first cycle, the feedforward neural network model eliminates the data samples whose errors exceed the preset threshold from the original data; then the remaining original data are used to continue to train the feedforward neural network model, and the corresponding data samples are continuously eliminated in the same elimination method; the iterative cycle is repeated until the feedforward neural network model no longer eliminates data samples, then the training is stopped, the feedforward neural network model at this time is output, and the feedforward neural network model is used to recall the data samples that were mistakenly eliminated in the iterative cycle; In the sample recall process, firstly, the feedforward neural network model that stops training and outputs is used to test all the original data, and the data samples whose error order is greater than the average error order are eliminated; then the remaining original data is used to continue to train the current feedforward neural network model, and the data samples whose error order is greater than the average error order are eliminated in the remaining original data; the elimination process is repeated until the number of remaining data samples in the original data remains stable, and the corresponding average error is calculated to obtain the original data after cyclic elimination; In step S22, the filtering process in the filtering and denoising operation is as follows: in the input raw data, the continuously acquired data samples are taken as a fixed length of When the intelligent monitoring and acquisition equipment obtains new raw data in real time, the first data sample in the sequence is removed, and then the remaining data samples are moved forward, and the data samples corresponding to the new raw data are inserted into the end of the sequence, and arithmetic average operation is performed to realize filtering processing; The denoising process in the filtering denoising operation is: perform wavelet transform on the data samples in the original data, and calculate the wavelet transform coefficients by the following formula: ,Right now: In the formula, represents the wavelet function, represents the complex conjugate of the wavelet function; represents the original data independent variable value, represents the stretch factor, represents the translation factor, Indicates the corresponding data after the original data is translated; Represents the useful data samples in the original data. Based on the useful data samples, the noisy data samples are converted It is expressed as the following formula: In the formula, ; represents the standard deviation of the noise figure, represents a random noise data sample; The larger the absolute value of the calculated wavelet transform coefficient is, the higher the probability that the corresponding data sample is a useful data sample is. Therefore, the distinction threshold is continued to be set to determine the noisy data samples in the original data. After the calculation of the wavelet transform coefficient is completed, the wavelet transform coefficients corresponding to the high-frequency level of the data samples of the original data are processed by the distinction threshold. After the data samples of the original data are processed, the corresponding data samples are reconstructed by inverse wavelet transform to achieve denoising. In step S3, firstly, an encoder is used to encode the preprocessed raw data, and then a node heterogeneous model of the raw data is constructed, and the encoded raw data is input into the node heterogeneous model for conversion; In the node heterogeneous model, a total of nodes, representing the size of each data sample in the original data; the number of data samples passed by each transmission task in the original data conversion process is recorded as After that, the data transmission channel model It is expressed as the following formula: In this model, Indicates the total number of nodes, Indicates the number of copies corresponding to the original data; Indicates that the branch task in the transmission task is at the node The number of data samples processed during the above operation; Indicates the efficiency of task execution; Representation sample The total number of corresponding copies; Indicates that the sample is being processed In the process, the node The number of data samples processed during the above operation; Indicates that at the node The total number of branch tasks processed; After the conversion of the original data is completed, it is decoded by a decoder to form a data format for data analysis and integration, and stored in the corresponding memory.

2. The intelligent cleaning method for hydropower station operation data according to claim 1 is characterized in that: In step S4, for the data set, the data cleaning operation includes the following steps: S41, calculating the abnormal factor size of all data points corresponding to the data samples in the data set, obtaining the abnormal value of each data point and performing sorting processing; S42, using a Gaussian mixture model, for the data points corresponding to the sorted outliers, according to the classification of normal data samples and outlier data samples, performing a clustering operation; using the outlier value corresponding to the dividing line between the normal data samples and the outlier data samples obtained after clustering as an outlier judgment threshold; S43, cyclically comparing the magnitude relationship between the abnormal values ​​of all data points in the data sample and the abnormal judgment threshold, judging the data samples corresponding to the data points whose abnormal values ​​are smaller than the abnormal judgment threshold as normal data samples, otherwise judging them as abnormal data samples, and eliminating all abnormal data samples to form blank missing data samples in the data set; Based on the data set that already contains blank missing data samples and the data set in its original state, the missing data are dynamically filled.

3. The intelligent cleaning method for hydropower station operation data according to claim 2 is characterized in that: After step S43, the missing data dynamic filling operation includes the following steps: S44, using the original data set as a reference, judging the time length of the missing data samples therein, and classifying the missing data samples into two types: local missing and long-term missing; the missing type of the blank missing data samples obtained after the abnormal identification and elimination operation is local missing; S45. For the local missing type, the local similarity regression algorithm LSR is used for processing, the data points before and after the missing data sample are used as training data, a regression model is established and trained, and the regression model is used with the local similarity regression algorithm LSR to predict and fill the missing data samples of the local missing type; S46. For long-term missing data types, use the random forest algorithm RF for processing, set the number of decision trees, use the data samples with similar lengths before and after the missing data samples as training data, build and train a regression model, and use the regression model equipped with the random forest algorithm RF to predict and fill in the missing data samples of the long-term missing type.

4. The intelligent cleaning method for hydropower station operation data according to claim 1 is characterized in that: Before step S4, an abnormal data warning operation is also performed, using a CNN-LSTM neural network prediction model in combination with the Laida criterion to issue a warning for abnormal data in the obtained operating data.

5. The intelligent cleaning method for hydropower station operation data according to claim 4 is characterized in that: The specific process of abnormal data warning operation is as follows: obtain historical data during the operation of the hydropower station to form a historical sample set as the input data of the CNN-LSTM neural network prediction model; divide the historical sample set into a training set, a validation set, and a test set, and perform cyclic training on the CNN-LSTM neural network prediction model. After the training is completed, an intermediate model is obtained. The changes in historical data are predicted through the intermediate model to obtain the corresponding predicted value. At the same time, the true value of the historical data corresponding to the predicted value is obtained. The predicted value and the true value are calculated through residual calculation to obtain the predicted residual value. Then, the standard deviation and mean of the predicted residual value are calculated, and the intermediate model is optimized to obtain the abnormal data warning model. Determine the warning sample set that needs to be warned from the historical data, input it into the abnormal data warning model, and obtain the verification prediction set output by the model; compare the original data temporarily stored in step S3 with the data in the verification prediction set to determine whether the original data falls within the data range in the verification prediction set, thereby realizing the abnormal data warning operation.

6. An intelligent cleaning system for hydropower station operation data, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the intelligent cleaning method for hydropower station operation data according to any one of claims 1 to 5.

Images