A method and system for cleaning ecological environment monitoring data

Abstract

The present invention discloses an ecological environment monitoring data cleaning method and system, which belongs to the field of data processing technology, including: collecting monitoring data, classifying all monitoring data into independent variable data and dependent variable data based on the label type of monitoring equipment; a data cleaning model cleans and corrects the monitoring data based on a machine learning algorithm, and generates corresponding corrected independent variable data and corrected dependent variable data respectively; extracts independent variable associated data and reference data respectively, and sets them as training sets, and constructs an environmental monitoring model based on the training sets and the machine learning algorithm; outputs the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjusts the type of monitoring equipment to generate standard monitoring equipment, and realizes the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence. Through the present invention, not only the diversity and accuracy of data cleaning can be improved, but also the necessity of monitoring equipment and the accuracy of ecological environment monitoring can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for cleaning ecological environment monitoring data. Background Art

[0002] With the increasing awareness of ecological environmental protection, the demand for monitoring and analysis of environmental data is growing. Existing monitoring data are often affected by factors such as sensor errors, transmission noise and environmental interference, which leads to a decrease in data quality and affects the accuracy of analysis results. Therefore, using an effective monitoring data cleaning method and system is of great significance to improving the quality of environmental monitoring.

[0003] A similar prior art includes a Chinese patent application with publication number CN114579548A, which discloses a data cleaning system for data collection based on the Internet of Things, relates to the field of data processing technology, and solves the technical problem that the prior art cannot perform data cleaning on the cleaned sensor data as required during the data cleaning process, resulting in poor data cleaning effect and low cleaning efficiency; the invention includes a data server, several edge processors and several types of acquisition sensors; the invention reasonably divides the sensor types and configures a reasonable number of edge processors, and then uses the edge processor or the data server to clean the sensor data according to the requirements and cleaning rules, so as to achieve the purpose of improving data cleaning efficiency and data cleaning effect; after the cleaning rules are updated, the invention can perform data cleaning uniformly through the data processor, and can also perform data cleaning separately through the edge processor, thereby improving the data cleaning efficiency while ensuring the data cleaning quality. In addition, there is a Chinese patent application with publication number CN116578842A, which discloses a data cleaning method and device for wireless sensing data sets, which groups the data in the wireless sensing data set to be cleaned into equal amounts; constructs a network model training set and a sample set to be cleaned based on the grouped data; uses the training set to train multiple classification networks with the same structure, and monitors the sample sets to be cleaned that match the training set one by one after the model converges, judges and calibrates the model to identify error samples based on the sample labels, and records the calibration frequency and sample index of all error samples to achieve error sample calibration; takes other uncleaned sample groups as data to be cleaned in turn, repeats the steps of error sample calibration until all grouped samples are cleaned; judges the samples in the data to be cleaned whose calibration times exceed the cleaning threshold as abnormal samples, and deletes the abnormal samples. The invention uses the network model trained by the consistency of most of the data in the data set to eliminate inconsistent data and achieve the effect of data cleaning.

[0004] However, the above-mentioned prior art only performs data cleaning and data elimination of abnormal samples. In actual situations, it is necessary not only to clean the monitoring data, but also to effectively analyze and predict the monitoring data after data cleaning, and to obtain effective monitoring equipment to realize data monitoring. Summary of the invention

[0005] To solve the above problems, the present invention provides an ecological environment monitoring data cleaning method and system to solve the problems in the prior art.

[0006] In order to achieve the above-mentioned purpose of the invention, the present invention proposes a method for cleaning ecological environment monitoring data, comprising:

[0007] Setting the monitoring type of the ecological environment, installing monitoring equipment based on the monitoring type, the monitoring equipment collecting monitoring data based on time series, and classifying all the monitoring data into independent variable data and dependent variable data based on the label type of the monitoring equipment;

[0008] Constructing a data cleaning model, inputting the monitoring data into the data cleaning model respectively, setting a machine learning algorithm, and the data cleaning model performs cleaning and correction on the monitoring data based on the machine learning algorithm, and generates corresponding correction independent variable data and correction dependent variable data respectively;

[0009] Extracting independent variable associated data from all the corrected independent variable data, extracting data associated with the independent variable associated data from all the corrected dependent variable data and setting them as reference data, setting all the independent variable associated data and all the reference data as a training set, and constructing an environmental monitoring model based on the training set and the machine learning algorithm;

[0010] Output the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjust the type of the monitoring equipment based on the label type corresponding to the independent variable associated data, generate standard monitoring equipment corresponding to the monitoring type, and implement the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence.

[0011] Furthermore, the data cleaning model performs cleaning and correction on the monitoring data based on the machine learning algorithm, including the following steps:

[0012] The monitoring data corresponding to any label type is divided into multiple sub-data based on a preset number, the data features corresponding to all the sub-data are extracted respectively based on the machine learning algorithm, all the data features are subjected to principal component analysis to extract the standard features of the monitoring data, the sub-data whose data features are different from the standard features are set as abnormal data, an error correction method is set based on the machine learning algorithm, the abnormal data is corrected based on the error correction method to generate standard monitoring data corresponding to the monitoring data, and this step is repeated until the monitoring data corresponding to all label types are corrected to generate corresponding standard monitoring data.

[0013] Furthermore, the error correction method is set based on the machine learning algorithm, including:

[0014] The error correction method includes a deletion method and a replacement method. If the machine learning algorithm extracts the corresponding data features based on the continuity of the sub-data, the error correction method is set to the replacement method, a time frame is set, and adjacent data is selected at the time point corresponding to the abnormal data based on the time frame, and data corresponding to other monitoring data at the time point is obtained and set as comparison data, and the fluctuation change rate of any comparison data is obtained based on the number of time points included in the time frame, and the average fluctuation change rate of all comparison data at the same time point is calculated, and the average value is set as the fluctuation change rate of the adjacent data, and the data at the time point corresponding to the abnormal data is generated based on the fluctuation change rate of the adjacent data, and set as replacement data, and the abnormal data is modified to the replacement data;

[0015] If the machine learning algorithm extracts the corresponding data feature based on the data frequency of the sub-data, the error correction method is set to the deletion method, and the abnormal data is deleted based on the deletion method.

[0016] Furthermore, the standard monitoring data corresponding to the independent variable data is set as the corrected independent variable data, and the standard monitoring data corresponding to the dependent variable data is set as the corrected dependent variable data.

[0017] Furthermore, the step of extracting the independent variable associated data from all the corrected independent variable data comprises the following steps:

[0018] Respectively obtain a correlation function between any correction independent variable data and any correction dependent variable data, output a corresponding element correlation based on the correlation function, and set the element correlation as a weight value;

[0019] Based on all the weight values, the corresponding correlation functions are trained to generate a correlation analysis model, and subsequence data corresponding to any time series is intercepted in any of the corrected independent variable data, and the subsequence data is input into the correlation function between two preset variable data in the correlation analysis model, and a predicted sequence is output, and the difference between the predicted sequence and the subsequence data at the same time point is compared. If the difference is greater than a first preset value, it is determined that the correlation function corresponding to the two preset variable data is invalid, otherwise, the corrected independent variable data in the two preset variable data is set as the independent variable associated data, and the corrected dependent variable data in the two preset variable data is set as the reference data, wherein the two preset variable data include any of the corrected independent variable data and any of the corrected dependent variable data.

[0020] Furthermore, constructing an environmental monitoring model based on the training set and the machine learning algorithm includes:

[0021] Based on the machine learning algorithm, attribute feature information of the training set is extracted; based on the attribute feature information, a variation range is calculated; based on the variation range, whether the training set exceeds a preset normal fluctuation threshold is determined; data in the training set whose variation range exceeds the normal fluctuation threshold is set as aggregated data; data whose variation range does not exceed the normal fluctuation threshold is set as training data; the machine learning algorithm generates a learning model based on the training data; the learning model adjusts the model accuracy of the learning model based on the aggregated data to generate the environmental monitoring model.

[0022] Further, the calculating the variation range based on the attribute feature information includes:

[0023] The data with the same attribute feature information in the training set is obtained and set as the first data. The mean and standard deviation of the first data are calculated based on the time series. The variation range T is calculated based on the first formula. The first formula is: , where M is the mean of the first data, N is the standard deviation of the first data, A precision parameter for extracting the attribute feature information for the machine learning algorithm;

[0024] This step is repeated to calculate the variation range corresponding to all the attribute feature information.

[0025] Furthermore, the environmental monitoring model sets the reference data with the same label type in the training data as a validation set, outputs a prediction value based on the validation set, combines all the prediction values ​​in sequence based on the prediction time, and generates the dependent variable change sequence corresponding to the label type.

[0026] The present invention also provides an ecological environment monitoring data cleaning system, which is used to implement the above-mentioned ecological environment monitoring data cleaning method, and the system mainly includes:

[0027] A collection module, which sets the monitoring type of the ecological environment, installs monitoring equipment based on the monitoring type, and the monitoring equipment collects monitoring data based on time series, and classifies all the monitoring data into independent variable data and dependent variable data based on the label type of the monitoring equipment;

[0028] A cleaning module constructs a data cleaning model, inputs the monitoring data into the data cleaning model respectively, sets a machine learning algorithm, and the data cleaning model cleans and corrects the monitoring data based on the machine learning algorithm to generate corresponding corrected independent variable data and corrected dependent variable data respectively;

[0029] A training module, extracting independent variable associated data from all the corrected independent variable data, extracting data associated with the independent variable associated data from all the corrected dependent variable data and setting them as reference data, setting all the independent variable associated data and all the reference data as a training set, and constructing an environmental monitoring model based on the training set and the machine learning algorithm;

[0030] The monitoring module outputs the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjusts the type of the monitoring equipment based on the label type corresponding to the independent variable associated data, generates standard monitoring equipment corresponding to the monitoring type, and realizes the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence.

[0031] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0032] The present invention first cleans and corrects the monitoring data corresponding to different label types through a data cleaning model, which can improve the accuracy of the monitoring data; then generates an environmental monitoring model by setting a training set and a machine learning algorithm, which can improve the accuracy of the monitoring type of the ecological environment; finally, the type of monitoring equipment is adjusted through independent variable correlation data, which can improve the effectiveness of the ecological environment monitoring equipment and reduce equipment loss.

[0033] The present invention also improves the integrity of the data cleaning model by setting different error correction modes to perform cleaning correction of abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for cleaning ecological environment monitoring data according to the present invention;

[0035] Figure 2It is a schematic diagram of the replacement method in the present invention;

[0036] Figure 3 This is a structural diagram of an ecological environment monitoring data cleaning system of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0039] like Figure 1 As shown, a method for cleaning ecological environment monitoring data includes:

[0040] S1: Set the monitoring type of the ecological environment, install monitoring equipment based on the monitoring type, and use the monitoring equipment to collect monitoring data based on time series. Classify all monitoring data into independent variable data and dependent variable data based on the label type of the monitoring equipment.

[0041] Specifically, in this embodiment, the ecological environment includes different environmental factors and biological types, etc. The monitoring type refers to the type used for ecological environment monitoring, including but not limited to atmospheric monitoring, water quality monitoring, soil monitoring and biological monitoring, etc. This embodiment takes water quality monitoring in the monitoring type as an example for detailed description. Monitoring equipment refers to instruments and tools used to monitor and evaluate various ecological and environmental factors in the natural environment, that is, sensor equipment corresponding to various elements, such as temperature sensors, pH meters, turbidity meters and chromatographs, etc. Time series refers to the sequence characteristics composed of the corresponding time points when the monitoring equipment collects data. The monitoring data is the data collected and generated by the monitoring equipment. The label type refers to the equipment type corresponding to the monitoring equipment. The independent variable data refers to the monitoring data corresponding to the independent variable factor with the label type, and the dependent variable data refers to the monitoring data corresponding to the dependent variable factor with the label type, wherein the independent variable factor refers to the independent variable, and the dependent variable factor refers to the dependent variable that changes due to the influence of the independent variable. For example, in water quality monitoring, the independent variable data includes but is not limited to the monitoring data corresponding to the temperature sensor, light intensity and water flow rate, etc., and the dependent variable data includes the monitoring data corresponding to the pH meter and turbidity meter, etc.

[0042] S2: Construct a data cleaning model, input the monitoring data into the data cleaning model respectively, set the machine learning algorithm, and the data cleaning model cleans and corrects the monitoring data based on the machine learning algorithm to generate corresponding corrected independent variable data and corrected dependent variable data respectively.

[0043] Specifically, in this embodiment, the data cleaning model is a model for processing monitoring data, wherein the monitoring data includes independent variable data and dependent variable data. The machine learning algorithm is an algorithm used to construct a data cleaning model during data processing, which can improve data quality. Correction of independent variable data refers to data after data cleaning of monitoring data classified as independent variable data, and correction of dependent variable data refers to data after data cleaning of monitoring data classified as dependent variable data. The data cleaning model can improve the accuracy of data cleaning of monitoring data.

[0044] S3: Extract independent variable associated data from all corrected independent variable data, extract data associated with the independent variable associated data from all corrected dependent variable data and set them as reference data, set all independent variable associated data and all reference data as training sets, and build an environmental monitoring model based on the training sets and machine learning algorithms.

[0045] Specifically, in this embodiment, the independent variable associated data refers to the data that has a greater correlation with the corrected dependent variable data among all the corrected independent variable data, the reference data refers to the data that has a greater correlation with the independent variable associated data among all the corrected dependent variable data, the training set is the data used for machine learning algorithm training to learn and generate the environmental monitoring model, wherein the machine learning algorithm can use the GRU (Gated Recurrent Unit) neural network algorithm, and the environmental monitoring model refers to a model generated by the machine learning algorithm training for processing sequence data. By extracting the independent variable associated data and the reference data, the label type of the monitoring data can be effectively screened, and the prediction accuracy of the environmental monitoring model can be improved.

[0046] S4: Output the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjust the type of monitoring equipment based on the label type corresponding to the independent variable associated data, generate standard monitoring equipment corresponding to the monitoring type, and realize ecological environment monitoring based on the standard monitoring equipment and the dependent variable change sequence.

[0047] Specifically, in this embodiment, the dependent variable change sequence refers to the prediction result sequence output by the environmental monitoring model based on the reference data. According to the dependent variable change sequence, the numerical change of the corrected dependent variable data in the ecological environment at the future prediction time can be obtained to achieve the effectiveness of ecological environment monitoring. Adjusting the type of monitoring equipment based on the label type of the independent variable associated data can reduce the unnecessary monitoring equipment used for the monitoring type of the ecological environment, thereby reducing the data analysis workload of the monitoring data, that is, the standard monitoring equipment refers to the necessary monitoring equipment required for the monitoring type. The standard monitoring equipment and the dependent variable change sequence can achieve the accuracy of ecological environment monitoring with less monitoring equipment load.

[0048] As a preferred technical solution of the present invention, the data cleaning model cleans and corrects the monitoring data based on the machine learning algorithm, including the following steps:

[0049] The monitoring data corresponding to any label type is divided into multiple sub-data based on a preset number, the data features corresponding to all the sub-data are extracted respectively based on the machine learning algorithm, all the data features are subjected to principal component analysis to extract the standard features of the monitoring data, the sub-data whose data features are different from the standard features are set as abnormal data, the error correction method is set based on the machine learning algorithm, the abnormal data is corrected based on the error correction method to generate the standard monitoring data corresponding to the monitoring data, and this step is repeated until the monitoring data corresponding to all label types are corrected to generate the corresponding standard monitoring data.

[0050] Specifically, in this embodiment, the monitoring data corresponding to the same tag type have a greater correlation. Therefore, the monitoring data corresponding to any tag type is divided into multiple sub-data, wherein the preset number refers to the total number of sub-data, and the value of the preset number can be set according to the collection time interval of the monitoring equipment.

[0051] Data features are information features that are useful for prediction tasks and are identified and constructed from sub-data by machine learning algorithms. Principal component analysis (PCA) is a statistical method that transforms a set of possibly related variables into a set of linearly unrelated variables through orthogonal transformation. These unrelated variables are called principal components. Therefore, principal component analysis can reduce the dimensions of all data features to reduce the number of features while retaining the original structure of the data. All data features can be reduced to generate standard features through principal component analysis, and then the classifiers in the machine learning algorithm compare the separability between all data features and standard features, such as support vector machine classifiers. Separability refers to the ability of a classifier to distinguish different categories or groups. If two sets of data features are separable, it means that there is a certain pattern or feature that enables a learning algorithm to find a decision boundary in the feature space to effectively separate the two sets of data features. If the data feature is separable from the standard feature, it is determined that the data feature is different from the standard feature. Therefore, whether there is abnormal data in all sub-data is screened by whether there is separability between the data feature and the standard feature, that is, the sub-data with data features different from the standard feature is set as abnormal data with outliers. The error correction method refers to the method of correcting abnormal values ​​in abnormal data. Standard monitoring data refers to the data after all abnormal data contained in the monitoring data are corrected by the error correction method. By repeating this step, all monitoring data can be corrected by abnormal data to generate corresponding standard monitoring data. Screening abnormal data in monitoring data by data features can avoid missing or incorrectly correcting abnormal values ​​in monitoring data, and can improve the comprehensiveness and effectiveness of data cleaning of monitoring data.

[0052] As a preferred technical solution of the present invention, the error correction method is set based on the machine learning algorithm, including:

[0053] Error correction methods include deletion and replacement. If the machine learning algorithm extracts corresponding data features based on the continuity of sub-data, the error correction method is set to replacement, and a time frame is set. Adjacent data is selected at the time point corresponding to the abnormal data based on the time frame, and data corresponding to other monitoring data at the time point is obtained and set as comparison data. The fluctuation change rate of any comparison data is obtained based on the number of time points included in the time frame, and the average fluctuation change rate of all comparison data at the same time point is calculated. The average value is set as the fluctuation change rate of adjacent data, and the data at the time point corresponding to the abnormal data is generated based on the fluctuation change rate of adjacent data, and set as replacement data, and the abnormal data is modified to replacement data.

[0054] If the machine learning algorithm extracts corresponding data features based on the data frequency of the sub-data, the error correction method is set to the deletion method, and the abnormal data is deleted based on the deletion method.

[0055] Specifically, in the present embodiment, the deletion method refers to a processing method for deleting abnormal data, and the replacement method refers to a processing method for modifying abnormal data. The continuity of sub-data refers to the time continuity feature, numerical continuity feature or trend continuity feature of sub-data that is relied on when extracting data features using a machine learning algorithm. If continuity is used to extract data features, the replacement method is used to process abnormal data. First, the time frame refers to a preset time length, and the time frame must be greater than the time length contained in the preset number. For example, the preset number of sub-data corresponding to abnormal data is 20, and the time length contained in the time frame must be greater than the time points corresponding to 20 abnormal data. Adjacent data refers to data corresponding to the time frame containing abnormal data, and comparison data refers to data in other monitoring data that have a time point belonging to the time frame with adjacent data, wherein other monitoring data refers to monitoring data in all monitoring data except for the abnormal data. For example, the monitoring data H1 is (1, 2, 3, 4, 5, 10, 7, 8, 9, 10, 11, 12). After being divided into multiple sub-data, the sub-data corresponding to the abnormal data h1 is (5, 10, 7, 8). Among them, the abnormal data h1 contains data corresponding to 4 time points. Then the time frame can be set to T1, and the start time and end time of T1 must be greater than the time range included in the 4 time points. Then, the data belonging to the time frame T1 is extracted from the monitoring data as adjacent data, and the data belonging to the time frame T1 is extracted from other monitoring data as comparison data. The fluctuation change rate of comparison data refers to the changing numerical sequence composed of the difference ratio between any two adjacent data in the comparison data. For example, Figure 2 As shown, the comparison data F1 is (20, 25, 30, 35, 40), then the fluctuation change rate P1 of the comparison data F1 is [(25-20) / 20, (30-25) / 25, (35-30) / 30, (40-35) / 35]. The fluctuation change rates of all comparison data are averaged at the same time point, and the adjacent data K1 containing the abnormal data h1 is averaged to generate the corresponding fluctuation change rate R1, and the replacement data G1 can be calculated.

[0056] The frequency of sub-data refers to the reliance on Fourier transform, cyclic statistics or frequency domain feature engineering of sub-data when extracting data features using machine learning algorithms. In this case, deletion is used to process abnormal data, which can effectively reduce outliers in sub-data.

[0057] As a preferred technical solution of the present invention, the standard monitoring data corresponding to the independent variable data is set as the corrected independent variable data, and the standard monitoring data corresponding to the dependent variable data is set as the corrected dependent variable data.

[0058] Specifically, in this embodiment, if the monitoring data is independent variable data, the corresponding standard monitoring data is set as the corrected independent variable data; if the monitoring data is dependent variable data, the corresponding standard monitoring data is set as the corrected dependent variable data.

[0059] As a preferred technical solution of the present invention, extracting independent variable associated data from all corrected independent variable data includes the following steps:

[0060] A correlation function between any correction independent variable data and any correction dependent variable data is obtained respectively, and the element correlation corresponding to the correlation function output is output, and the element correlation is set as a weight value.

[0061] Based on all weight values, the corresponding correlation function is trained to generate a correlation analysis model, subsequence data corresponding to any time series is intercepted in any correction independent variable data, the subsequence data is input into the correlation function between two preset variable data in the correlation analysis model, and a predicted sequence is output, and the difference between the predicted sequence and the subsequence data at the same time point is compared. If the difference is greater than a first preset value, it is determined that the correlation function corresponding to the two preset variable data is invalid, otherwise, the correction independent variable data in the two preset variable data are set as independent variable associated data, and the correction dependent variable data in the two preset variable data are set as reference data, wherein the two preset variable data include any correction independent variable data and any correction dependent variable data.

[0062] Specifically, in this embodiment, extracting independent variable correlation data can provide a data basis for establishing a model for subsequent ecological environment monitoring. The correlation function refers to a function that describes the degree of correlation between two variable data. The correlation function between the two variables can be learned through a neural network set by a machine learning algorithm, and the correlation value, i.e., the element correlation, can be output.

[0063] The correlation analysis model is an analysis model composed of a neural network, which is generated through training and learning of all correlation functions and corresponding weight values, and randomly intercepts subsequence data in any correction independent variable data, wherein the subsequence data is a data group continuously generated based on a time series in the correction independent variable data. The correlation analysis model predicts the subsequence data based on the correlation function between two preset variable data and outputs the corresponding prediction sequence. The first preset value refers to a preset threshold value. If the difference is greater than the first preset value, it means that there is a large data change between the prediction sequence and the subsequence data, and the correlation between the correction independent variable data corresponding to the subsequence data and any correction dependent variable data is low, so the corresponding correlation function is destroyed and invalid. If the difference is less than or equal to the first preset value, it means that the prediction sequence has a high similarity with the subsequence data, that is, the correlation between the correction independent variable data corresponding to the subsequence data and any correction dependent variable data is high, therefore, the correction independent variable data is set as the independent variable association data, and the correction dependent variable data in the two preset variable data is set as the reference data.

[0064] As a preferred technical solution of the present invention, building an environmental monitoring model based on a training set and a machine learning algorithm includes:

[0065] Based on the machine learning algorithm, the attribute feature information of the training set is extracted, the variation range is calculated based on the attribute feature information, and it is judged whether the training set exceeds the preset normal fluctuation threshold based on the variation range. The data in the training set whose variation range exceeds the normal fluctuation threshold is set as aggregated data, and the data whose variation range does not exceed the normal fluctuation threshold is set as training data. The machine learning algorithm generates a learning model based on the training data, and the learning model adjusts the model accuracy of the learning model based on the aggregated data to generate an environmental monitoring model.

[0066] Specifically, in this embodiment, the environmental monitoring model is generated by a machine learning algorithm through a training set, wherein the attribute feature information refers to the attribute information of the training set, including but not limited to the node information, timestamp, label type, metric value and data quality index of the data source, etc., and the variation range refers to the allowable range of variation calculated from the recent variation trend of the attribute feature information contained in the time series data in the training set, that is, the allowable range of deviation of the time series data in the training set. The normal fluctuation threshold refers to the preset variation threshold of the time series data. If the variation range exceeds the preset normal fluctuation threshold, it means that the data has a sudden fluctuation in the time series data, and the corresponding data in the training set can be set as aggregated data, otherwise, it can be set as training data.

[0067] The machine learning algorithm can be trained to generate a learning model through training data, and then the attribute feature information corresponding to the aggregated data is used as an additional learning feature to adjust the learning model to generate an environmental monitoring model. The attribute feature information corresponding to the aggregated data can increase the data dimension and depth of the learning model training, so that the learning model can learn and predict more accurately. The adjusted learning model can be set as the environmental monitoring model.

[0068] As a preferred technical solution of the present invention, calculating the variation range based on the attribute feature information includes:

[0069] The data with the same attribute feature information in the training set is obtained and set as the first data. The mean and standard deviation of the first data are calculated based on the time series. The variation range T is calculated based on the first formula. The first formula is: , where M is the mean of the first data, N is the standard deviation of the first data, The precision parameter for extracting attribute feature information for machine learning algorithms.

[0070] Repeat this step to calculate the variation range corresponding to all attribute feature information.

[0071] Specifically, in this embodiment, the training set includes multiple independent variable associated data and reference data, so there will be multiple attribute feature information, all attribute feature information are clustered to generate multiple clustering results, the attribute feature information included in each clustering result is set to the same attribute feature information, and then the data corresponding to the same attribute feature information are combined to form the first data. The first formula can be used to calculate the variation range of the attribute feature information corresponding to each first data.

[0072] As a preferred technical solution of the present invention, the environmental monitoring model sets the reference data with the same label type in the training data as the validation set, outputs the predicted value based on the validation set, combines all the predicted values ​​in sequence based on the prediction time, and generates a dependent variable change sequence corresponding to the label type.

[0073] Specifically, in this embodiment, the environmental monitoring models after adjusting the model accuracy have higher prediction accuracy. Therefore, the reference data of the same label type in the training data is used as the verification set, and the time characteristics of the verification set are obtained. The prediction value corresponding to the verification set in the future time can be output, and all the prediction values ​​are combined in sequence to generate a dependent variable change sequence with the same label type as the verification set.

[0074] like Figure 3 As shown, the present invention also provides an ecological environment monitoring data cleaning system, which is used to implement the above-mentioned ecological environment monitoring data cleaning method, and the system mainly includes:

[0075] The collection module sets the monitoring type of the ecological environment, installs monitoring equipment based on the monitoring type, and the monitoring equipment collects monitoring data based on time series. All monitoring data are classified into independent variable data and dependent variable data based on the label type of the monitoring equipment.

[0076] The cleaning module builds a data cleaning model, inputs the monitoring data into the data cleaning model respectively, sets the machine learning algorithm, and the data cleaning model cleans and corrects the monitoring data based on the machine learning algorithm to generate corresponding corrected independent variable data and corrected dependent variable data respectively.

[0077] The training module extracts independent variable associated data from all corrected independent variable data, extracts data associated with the independent variable associated data from all corrected dependent variable data and sets them as reference data, sets all independent variable associated data and all reference data as training sets, and builds an environmental monitoring model based on the training sets and machine learning algorithms.

[0078] The monitoring module outputs the dependent variable change sequence of the ecological environment within the predicted time based on the environmental monitoring model, adjusts the type of monitoring equipment based on the label type corresponding to the independent variable associated data, generates standard monitoring equipment corresponding to the monitoring type, and realizes the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence.

[0079] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0080] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0081] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0083] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for cleaning ecological environment monitoring data, characterized in that: The method comprises the following steps: Setting the monitoring type of the ecological environment, installing monitoring equipment based on the monitoring type, the monitoring equipment collecting monitoring data based on time series, and classifying all the monitoring data into independent variable data and dependent variable data based on the label type of the monitoring equipment; Constructing a data cleaning model, inputting the monitoring data into the data cleaning model respectively, setting a machine learning algorithm, and the data cleaning model performs cleaning and correction on the monitoring data based on the machine learning algorithm, and generates corresponding correction independent variable data and correction dependent variable data respectively; Extracting independent variable associated data from all the corrected independent variable data, extracting data associated with the independent variable associated data from all the corrected dependent variable data and setting them as reference data, setting all the independent variable associated data and all the reference data as a training set, and constructing an environmental monitoring model based on the training set and the machine learning algorithm; The extracting of independent variable associated data from all the corrected independent variable data comprises the following steps: Respectively obtain a correlation function between any correction independent variable data and any correction dependent variable data, output a corresponding element correlation based on the correlation function, and set the element correlation as a weight value; Based on all the weight values, the corresponding correlation function is trained to generate a correlation analysis model, subsequence data corresponding to any time series is intercepted in any of the correction independent variable data, the subsequence data is input into the correlation function between two preset variable data in the correlation analysis model, and a prediction sequence is output, and the difference between the prediction sequence and the subsequence data at the same time point is compared. If the difference is greater than a first preset value, it is determined that the correlation function corresponding to the two preset variable data is invalid, otherwise, the correction independent variable data in the two preset variable data is set as the independent variable associated data, and the correction dependent variable data in the two preset variable data is set as the reference data, wherein the two preset variable data include any of the correction independent variable data and any of the correction dependent variable data; Output the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjust the type of the monitoring equipment based on the label type corresponding to the independent variable associated data, generate standard monitoring equipment corresponding to the monitoring type, and implement the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence.

2. The method according to claim 1, characterized in that The data cleaning model performs cleaning and correction on the monitoring data based on the machine learning algorithm, comprising the following steps: The monitoring data corresponding to any label type is divided into multiple sub-data based on a preset number, the data features corresponding to all the sub-data are extracted respectively based on the machine learning algorithm, all the data features are subjected to principal component analysis to extract the standard features of the monitoring data, the sub-data whose data features are different from the standard features are set as abnormal data, an error correction method is set based on the machine learning algorithm, the abnormal data is corrected based on the error correction method to generate standard monitoring data corresponding to the monitoring data, and this step is repeated until the monitoring data corresponding to all label types are corrected to generate corresponding standard monitoring data.

3. The method according to claim 2, characterized in that The error correction method set based on the machine learning algorithm includes: The error correction method includes a deletion method and a replacement method. If the machine learning algorithm extracts the corresponding data features based on the continuity of the sub-data, the error correction method is set to the replacement method, a time frame is set, and adjacent data is selected at the time point corresponding to the abnormal data based on the time frame, and data corresponding to other monitoring data at the time point is obtained and set as comparison data, and the fluctuation change rate of any comparison data is obtained based on the number of time points included in the time frame, and the average fluctuation change rate of all comparison data at the same time point is calculated, and the average value is set as the fluctuation change rate of the adjacent data, and the data at the time point corresponding to the abnormal data is generated based on the fluctuation change rate of the adjacent data, and set as replacement data, and the abnormal data is modified to the replacement data; If the machine learning algorithm extracts the corresponding data feature based on the data frequency of the sub-data, the error correction method is set to the deletion method, and the abnormal data is deleted based on the deletion method.

4. The method according to claim 3, characterized in that The standard monitoring data corresponding to the independent variable data is set as the corrected independent variable data, and the standard monitoring data corresponding to the dependent variable data is set as the corrected dependent variable data.

5. The method according to claim 1, characterized in that The constructing of the environmental monitoring model based on the training set and the machine learning algorithm comprises: Based on the machine learning algorithm, attribute feature information of the training set is extracted; based on the attribute feature information, a variation range is calculated; based on the variation range, whether the training set exceeds a preset normal fluctuation threshold is determined; data in the training set whose variation range exceeds the normal fluctuation threshold is set as aggregated data; data whose variation range does not exceed the normal fluctuation threshold is set as training data; the machine learning algorithm generates a learning model based on the training data; the learning model adjusts the model accuracy of the learning model based on the aggregated data to generate the environmental monitoring model.

6. The method according to claim 5, characterized in that The calculating of the variation range based on the attribute feature information includes: The data with the same attribute feature information in the training set is obtained and set as the first data. The mean and standard deviation of the first data are calculated based on the time series. The variation range T is calculated based on the first formula. The first formula is: , where M is the mean of the first data, N is the standard deviation of the first data, A precision parameter for extracting the attribute feature information for the machine learning algorithm; This step is repeated to calculate the variation range corresponding to all the attribute feature information.

7. The method according to claim 5, characterized in that The environmental monitoring model sets the reference data with the same label type in the training data as a validation set, outputs a prediction value based on the validation set, combines all the prediction values ​​in sequence based on the prediction time, and generates the dependent variable change sequence corresponding to the label type.

8. An ecological environment monitoring data cleaning system, used to implement the method according to any one of claims 1 to 7, characterized in that: The system includes the following modules: A collection module, which sets the monitoring type of the ecological environment, installs monitoring equipment based on the monitoring type, and the monitoring equipment collects monitoring data based on time series, and classifies all the monitoring data into independent variable data and dependent variable data based on the label type of the monitoring equipment; A cleaning module constructs a data cleaning model, inputs the monitoring data into the data cleaning model respectively, sets a machine learning algorithm, and the data cleaning model cleans and corrects the monitoring data based on the machine learning algorithm to generate corresponding corrected independent variable data and corrected dependent variable data respectively; The training module extracts independent variable associated data from all the corrected independent variable data, extracts data associated with the independent variable associated data from all the corrected dependent variable data and sets them as reference data, sets all the independent variable associated data and all the reference data as training sets, and constructs an environmental monitoring model based on the training sets and the machine learning algorithm; respectively obtains the correlation function between any corrected independent variable data and any corrected dependent variable data, outputs the corresponding element correlation based on the correlation function, and sets the element correlation as a weight value; trains the corresponding correlation function based on all the weight values ​​to generate a correlation analysis model, intercepts any element in any of the corrected independent variable data, and outputs the element correlation based on the correlation function. subsequence data corresponding to the time series, inputting the subsequence data into the correlation function between two preset variable data in the correlation analysis model, and outputting a predicted sequence, comparing the difference between the predicted sequence and the subsequence data at the same time point, if the difference is greater than a first preset value, determining that the correlation function corresponding to the two preset variable data is invalid, otherwise, setting the corrected independent variable data in the two preset variable data as the independent variable associated data, and setting the corrected dependent variable data in the two preset variable data as the reference data, wherein the two preset variable data include any of the corrected independent variable data and any of the corrected dependent variable data; The monitoring module outputs the dependent variable change sequence of the ecological environment within the prediction time based on the environmental monitoring model, adjusts the type of the monitoring equipment based on the label type corresponding to the independent variable associated data, generates standard monitoring equipment corresponding to the monitoring type, and realizes the monitoring of the ecological environment based on the standard monitoring equipment and the dependent variable change sequence.

Images