Water environment quality evaluation model based on big data analysis

Abstract

The invention discloses a water environment quality evaluation model based on big data analysis. The water environment quality evaluation model comprises a database, a random forest model and a neural network model. The database is used for carrying out normalization processing on data of different seasons, time periods and point locations; random forest sub-models are constructed according to the data of the seasons, the time periods and the point locations, and then all the random forest sub-models are integrated into a multi-layer random forest model; according to the neural network model, model branches are constructed according to data of seasons, time periods and point locations as input features, and the neural network model combines an output result of the random forest model and other original features to carry out final result prediction; fine adjustment of the evaluation model is realized by using water flow velocity and flow data of different seasons, different time periods and different point locations, and the adaptability of the model to various environmental conditions can be improved, so that the generalization ability of the model is enhanced, and more accurate and comprehensive information is provided for evaluation of the health condition of the water environment.

Description

Technical Field

[0001] The present invention relates to the technical field of water ecological environment monitoring, and in particular to a water environment quality evaluation model based on big data analysis. Background Art

[0002] The health of the water ecological environment is affected by many factors, which are intertwined and jointly determine the stability and biodiversity of the water ecosystem. Among them, the velocity and flow of water flow are two key hydrological elements in the water ecological environment, which have a significant impact on the health of the water ecological environment. High flow rate helps to reoxygenate the water body, increase the dissolved oxygen content, and is conducive to the survival of aerobic organisms. Low flow rate may lead to a decrease in dissolved oxygen, affecting the respiration and growth of aquatic organisms; sufficient flow maintains the continuity of the river, which is conducive to fish migration and biological migration. Low flow may cause the river to dry up and destroy ecological connectivity.

[0003] Building a water ecological environment monitoring system is a complex and systematic project that requires comprehensive consideration of monitoring objectives, technical means, data management, quality control and other aspects. It is generally based on the random forest model, and the selected model is verified to ensure its applicability and accuracy, and the model output results are scientifically interpreted to provide support for decision-making. However:

[0004] 1. Although the random forest model has strong predictive ability and robustness, it still has its own limitations. For example, it may not be as good as some other models (such as neural networks) in capturing nonlinear relationships.

[0005] 2. Water environment quality often varies with spatial location and seasons, and the random forest model may not be able to fully capture the spatial and temporal variability of the water environment. Summary of the invention

[0006] The embodiment of the present application provides a water environment quality evaluation model based on big data analysis. By utilizing water flow velocity and flow rate data in different seasons, different time periods, and different locations, the evaluation model can be fine-tuned, thereby improving the model's adaptability to various environmental conditions, thereby enhancing the model's generalization ability and providing more accurate and comprehensive information for the evaluation of the water environment health status.

[0007] The embodiment of the present application provides a water environment quality evaluation model based on big data analysis, including a database, a random forest model and a neural network model;

[0008] Database: seasonal decomposition of data, identification and removal of seasonal effects, and then normalization of data of different seasons, time periods and locations, respectively introducing temporal and spatial features as inputs of the model to ensure data consistency and comparability;

[0009] Random forest model: Random forest sub-models are constructed according to the data of season, time period and location, and then all random forest sub-models are integrated into a multi-level random forest model. The prediction results of the multi-level random forest model are output as one of the input data of the neural network model;

[0010] Neural network model: Similarly, model branches are constructed based on the data of season, time period and location as input features. The neural network model combines the output results of the random forest model and other original features (input data) to predict the final results.

[0011] Database construction: collect velocity and flow data in real time through monitoring stations in water quality areas, and send the data to the data center through a remote transmission system; use hydrological models (such as SWAT, HYPE, etc.) to simulate water flow velocity and flow to supplement the insufficiency of field monitoring data; consult historical hydrological archives to obtain velocity and flow data for different seasons, time periods and points in the past. Clean and preprocess these data to obtain a water quality area database.

[0012] The data in the database are input with season, time period and location as input features, and the values ​​of water velocity and flow are extracted respectively. The values ​​are normalized and divided into training set, validation set and test set for the training of random forest model and neural network model respectively.

[0013] The process of constructing the random forest model based on the data characteristics of season, time period and point is as follows:

[0014] 1.1 Divide the data set according to season, time period and location, divide a random forest sub-model for each data set, set the parameters of each sub-model, the number of trees, the depth of the tree and the feature selection; use the corresponding divided training set data to train each random forest sub-model;

[0015] 1.2 Apply cross-validation to evaluate the performance and stability of each sub-model;

[0016] 1.3 Use stacking to train a meta-model to integrate the prediction results of the sub-models;

[0017] 1.4 Use the test set data to evaluate the performance of the integrated model and calculate evaluation indicators such as accuracy, mean square error (MSE), F1 score, etc.

[0018] 1.5 Use the trained multi-level random forest model for prediction, select the corresponding sub-model for prediction according to the season, time period or point of the input data, and integrate the results.

[0019] The process of constructing the neural network model based on the data characteristics of season, time period and location is as follows:

[0020] 2.1 Taking season, time period and location as part of the input features, the structure of the neural network model is constructed, including input layer, hidden layer and output layer;

[0021] 2.2 Set model parameters, such as learning rate, batch size, number of iterations, etc., and choose mean square error (MSE) or cross entropy loss as the loss function;

[0022] 2.3 Use the training set data to train the neural network model, and apply early stopping technology to prevent overfitting of the data; use the validation set data to adjust the model parameters to avoid overfitting

[0023] 2.4 Use the test set data to evaluate the performance of the model and calculate evaluation indicators such as accuracy, recall, F1 score, etc.;

[0024] 2.5 Select the corresponding model branch for prediction based on the season, time period or point of the input data and the prediction results of the random forest model.

[0025] The neural network model combines the output of the random forest and other original features to make the final prediction, and the prediction results are processed using weighted average.

[0026] The trained model is converted into a deployable format, such as ONNX, PMML, etc. The converted model is deployed to the selected environment (select the deployment environment according to actual needs, such as a local server, cloud platform, or edge computing device) to ensure that the model can run stably and receive real-time data.

[0027] The water flow velocity and flow data of different seasons, time periods and locations are collected in real time and input into the deployed model; the model outputs the evaluation results of the water environment health status in real time based on the input data; the random forest model first analyzes and predicts the data, and the neural network model combines the output results of the random forest model with the input data characteristics to make the final prediction using the weighted average method to obtain the final water environment health status evaluation results.

[0028] Regularly update and optimize the model based on newly collected data and model performance monitoring results; continuously monitor the performance of the model, such as prediction accuracy, response time, etc., to ensure stable operation of the model; collect user feedback on evaluation results to further optimize the model and deployment strategy.

[0029] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: by combining the random forest model and the neural network model, using the constructed water quality regional database, the values ​​are normalized, the data is divided into a training set, a validation set and a test set, which are used for the training of the random forest model and the neural network model respectively, the random forest model and the neural network model are pre-trained respectively, the neural network model combines the output of the random forest and other original features for the final prediction, and the prediction results are processed using weighted average. The training task allows the model to adjust the flow velocity and flow rate according to different seasons, different time periods, and different points, enhance the model's ability to identify time variation patterns and spatial variation patterns, and can be fine-tuned through continuous learning, which can improve the model's adaptability to various environmental conditions, thereby enhancing the model's generalization ability and providing more accurate and comprehensive information for the evaluation of the health status of the water environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart was constructed for the water environment quality assessment model for this application. DETAILED DESCRIPTION

[0031] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0033] Embodiment 1

[0034] See also Figure 1 , a water environment quality evaluation model based on big data analysis, including a database, a random forest model, and a neural network model;

[0035] Database: Collect velocity and flow data in real time through monitoring stations in water quality areas, and send the data to the data center through a remote transmission system; use hydrological models (such as SWAT, HYPE, etc.) to simulate water flow velocity and flow to supplement the insufficiency of field monitoring data; consult historical hydrological archives to obtain velocity and flow data of different seasons, time periods and points in the past, clean and pre-process these data to obtain a water quality regional database; seasonally decompose the data, identify and remove seasonal effects, so that the model can better capture non-seasonal changes; then normalize the data of different seasons, time periods and points, and introduce temporal features (such as seasons, months, time periods, etc.) and spatial features (such as point coordinates, watershed characteristics, etc.) as inputs to the model to ensure the consistency and comparability of the data, so that the model can identify temporal and spatial variation patterns;

[0036] Random forest model: Random forest sub-models are constructed according to the data of season, time period and location, and then all random forest sub-models are integrated into a multi-level random forest model. The prediction results of the multi-level random forest model are output as one of the input data of the neural network model;

[0037] Neural network model: Similarly, model branches are constructed based on the data of season, time period and location as input features. The neural network model combines the output results of the random forest model and other original features (input data) to predict the final results.

[0038] The data in the database are inputted according to season, time period and location, and the values ​​of water velocity and flow are extracted respectively. The values ​​are normalized and the data are divided into training set, validation set and test set.

[0039] Use the training set data to fit the normalizer, learn the maximum and minimum values ​​of the data, and apply the normalizer to transform the training set data to the target range (normalized target range, usually [0,1) or [-1,1));

[0040] The test set data is transformed using the normalizer fitted on the training set, ensuring that the training and test sets use the same scale.

[0041] Normalize different seasons, time periods and points separately according to the following process:

[0042] Divide the data by season and normalize the data of each season separately; divide the data by time period and normalize the data of each time period separately; divide the data by point and normalize the data of each point separately; save the normalizer of each season, time period and point to facilitate the subsequent inverse transformation or normalization of new data;

[0043] Use the normalized training set data to train the model; use the test set data to evaluate the model performance.

[0044]

[0045] The process of constructing the random forest model based on the data characteristics of season, time period and point is as follows:

[0046] 1.1 Divide the data set according to season, time period and location, divide a random forest sub-model for each data set, set the parameters of each sub-model, the number of trees, the depth of the tree and the feature selection; use the corresponding divided training set data to train each random forest sub-model;

[0047] 1.2 Apply cross-validation to evaluate the performance and stability of each sub-model;

[0048] 1.3 Use stacking to train a meta-model to integrate the prediction results of the sub-models;

[0049] 1.4 Use the test set data to evaluate the performance of the integrated model and calculate evaluation indicators such as accuracy, mean square error (MSE), F1 score, etc.

[0050] 1.5 Use the trained multi-level random forest model for prediction, select the corresponding sub-model for prediction according to the season, time period or point of the input data, and integrate the results.

[0051] The process of constructing the neural network model based on the data characteristics of season, time period and location is as follows:

[0052] 2.1 Taking season, time period and location as part of the input features, the structure of the neural network model is constructed, including input layer, hidden layer and output layer;

[0053] 2.2 Set model parameters, such as learning rate, batch size, number of iterations, etc., and choose mean square error (MSE) or cross entropy loss as the loss function;

[0054] 2.3 Use the training set data to train the neural network model, and apply early stopping technology to prevent overfitting of the data; use the validation set data to adjust the model parameters to avoid overfitting;

[0055] 2.4 Use the test set data to evaluate the performance of the model and calculate evaluation indicators such as accuracy, recall, F1 score, etc.;

[0056] 2.5 Select the corresponding model branch for prediction based on the season, time period or point of the input data and the prediction results of the random forest model.

[0057] The neural network model combines the output of the random forest and other original features to make the final prediction, and uses weighted average to process the prediction results. The processing process is as follows:

[0058] 3.1 Random Forest Model Prediction

[0059] Use the trained multi-level random forest model to predict the data and get the prediction result RF output ;

[0060] 3.2 Feature Merging:

[0061] The output RF of the random forest model output With the original feature X original Merge to form a new feature set X combined ;X combined =C ombine (X original ,RF output ), where C ombine The function combines the original features and the output of the random forest model into a new feature set;

[0062] 3.3 Neural Network Model Training

[0063] Use the merged feature set X combined And the true label Y to train the neural network model;

[0064] 3.4 Neural Network Model Prediction

[0065] For new data, first use the trained random forest model to generate RF output , the RF output The original features X of the new data original Merge to get X combined ;

[0066] Use the trained neural network model to combined Make predictions and get NN output .

[0067] The actual calling process of the trained model:

[0068] 4.1 Model Deployment

[0069] Convert the trained model into a deployable format, such as ONNX, PMML, etc.; deploy the converted model to the selected environment (select the deployment environment according to actual needs, such as local server, cloud platform or edge computing device) to ensure that the model can run stably and receive real-time data;

[0070] 4.2 Deploy API interfaces to facilitate external systems to call models for real-time prediction and output water environment health status evaluation results;

[0071] 4.3 Real-time prediction and result processing

[0072] The water flow velocity and flow data collected in real time in different seasons, time periods and locations are input into the deployed model; the model outputs the evaluation results of the water environment health status in real time based on the input data; the random forest model first analyzes and predicts the data, and the neural network model combines the output results of the random forest model with the input data characteristics to make the final prediction using the weighted average method to obtain the final water environment health status evaluation results.

[0073] Continuously monitor the performance of the model, such as prediction accuracy, response time, etc., to ensure stable operation of the model; regularly update and optimize the model based on newly collected data and model performance monitoring results, and collect user feedback on evaluation results to further optimize the model and deployment strategy.

[0074] The technical solution in the above-mentioned embodiment of the present application has at least the following technical effects or advantages: by combining the random forest model and the neural network model, using the constructed water quality regional database, the numerical values ​​are normalized, the data is divided into a training set, a validation set and a test set, which are used for the training of the random forest model and the neural network model respectively, and the random forest model and the neural network model are pre-trained respectively, and the neural network model combines the output of the random forest and other original features for the final prediction, and the prediction result is processed using weighted average. The training task allows the model to adjust the flow velocity and flow rate according to different seasons, different time periods, and different points, enhance the model's ability to identify time variation patterns and spatial variation patterns, and can be fine-tuned through continuous learning, which can improve the model's adaptability to various environmental conditions, thereby enhancing the model's generalization ability and providing more accurate and comprehensive information for the evaluation of the health status of the water environment.

[0075] Embodiment 2

[0076] A water environment quality evaluation model based on big data analysis, the specific process of outputting water environment quality evaluation results is as follows:

[0077] Model deployment: Convert the trained model into a deployable format. The converted model is deployed to the selected environment using the API interface to ensure that the model can run stably and receive real-time data.

[0078] Data collection: Input the real-time collected water velocity and flow data of different seasons, time periods and locations into the deployed model;

[0079] Random forest model prediction: select the corresponding sub-model for prediction according to the season, time period or point of the input data, and integrate the prediction results as one of the input data of the neural network model;

[0080] Neural network model: select the corresponding model branch for prediction according to the season, time period or point of the input data and the prediction results of the random forest model; use weighted average to process the prediction results and output the final predicted water environment health status evaluation results.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A water environment quality evaluation model based on big data analysis, including a database, a random forest model and a neural network model, characterized in that: Database: seasonal decomposition of data, identification and removal of seasonal effects, and then normalization of data of different seasons, time periods and locations, respectively introducing temporal and spatial features as inputs of the model to ensure data consistency and comparability; Random forest model: Random forest sub-models are constructed according to the data of season, time period and location, and then all random forest sub-models are integrated into a multi-level random forest model. The prediction results of the multi-level random forest model are output as one of the input data of the neural network model; Neural network model: Similarly, model branches are constructed based on the data of season, time period and location as input features. The neural network model combines the output results of the random forest model and other original features to predict the final results.

2. A water environment quality evaluation model based on big data analysis as claimed in claim 1, characterized in that: Through monitoring stations in water quality areas, real-time flow rate and flow data are collected and sent to the data center through a remote transmission system; hydrological models are used to simulate water flow rate and flow to supplement the deficiencies of field monitoring data; Historical hydrological archives were consulted to obtain velocity and flow data for different seasons, time periods and locations in the past. These data were cleaned and preprocessed to obtain a regional water quality database.

3. The water environment quality evaluation model based on big data analysis as claimed in claim 1, characterized in that: The data in the database are input with season, time period and location as input features, and the values ​​of water velocity and flow are extracted respectively. The values ​​are normalized and divided into training set, validation set and test set for the training of random forest model and neural network model respectively.

4. A water environment quality evaluation model based on big data analysis as claimed in claim 3, characterized in that: The data is divided by season, and the data of each season is normalized separately; the data is divided by time period, and the data of each time period is normalized separately; The data is divided by points, and the data of each point is normalized separately; Save the normalizer for each season, time period and point; Use the normalized training set data to train the model; use the test set data to evaluate the model performance.

5. The water environment quality evaluation model based on big data analysis as claimed in claim 1, characterized in that: The process of constructing the random forest model based on the data characteristics of season, time period and point is as follows: 1.1 Divide the data set according to season, time period and location, divide a random forest sub-model for each data set, set the parameters of each sub-model, the number of trees, the depth of the tree and the feature selection; use the corresponding divided training set data to train each random forest sub-model; 1.2 Apply cross-validation to evaluate the performance and stability of each sub-model; 1.3 Use stacking to train a meta-model to integrate the prediction results of the sub-models; 1.4 Use the test set data to evaluate the performance of the integrated model, calculate the evaluation indicators, and select accuracy, mean square error, and F1 score; 1.5 Use the trained multi-level random forest model for prediction, select the corresponding sub-model for prediction according to the season, time period or point of the input data, and integrate the results.

6. The water environment quality evaluation model based on big data analysis according to claim 1, characterized in that: The process of constructing the neural network model based on the data characteristics of season, time period and location is as follows: 2.1 Taking season, time period and location as part of the input features, the structure of the neural network model is constructed, including input layer, hidden layer and output layer; 2.2 Set model parameters, such as learning rate, batch size, number of iterations, etc., and choose mean square error or cross entropy loss as the loss function; 2.3 Use the training set data to train the neural network model, and apply early stopping technology to prevent overfitting of the data; use the validation set data to adjust the model parameters to avoid overfitting 2.4 Use the test set data to evaluate the performance of the model, calculate the evaluation indicators, and select accuracy, recall rate, and F1 score; 2.5 Select the corresponding model branch for prediction based on the season, time period or point of the input data and the prediction results of the random forest model.

7. A water environment quality evaluation model based on big data analysis as claimed in claim 6, characterized in that: The neural network model combines the output of the random forest and the original features to make the final prediction, and the prediction results are processed using weighted average.

8. The water environment quality evaluation model based on big data analysis as claimed in claim 1, characterized in that: The trained model is actually called: 4.1 Model Deployment Convert the trained model into a deployable format and deploy it to the selected environment to ensure that the model can run stably and receive real-time data; 4.2 Deploy API interfaces to facilitate external systems to call models for real-time prediction and output water environment health status evaluation results; 4.3 Real-time prediction and result processing The water flow velocity and flow data collected in real time in different seasons, time periods and locations are input into the deployed model; the model outputs the evaluation results of the water environment health status in real time based on the input data; the random forest model first analyzes and predicts the data, and the neural network model combines the output results of the random forest model with the input data characteristics to make the final prediction using the weighted average method to obtain the final water environment health status evaluation results.

9. The water environment quality evaluation model based on big data analysis according to claim 1, characterized in that: The specific process of outputting the water environment quality evaluation results is as follows: Model deployment: Convert the trained model into a deployable format. The converted model is deployed to the selected environment using the API interface to ensure that the model can run stably and receive real-time data. Data collection: Input the real-time collected water velocity and flow data of different seasons, time periods and locations into the deployed model; Random forest model prediction: select the corresponding sub-model for prediction according to the season, time period or point of the input data, and integrate the prediction results as one of the input data of the neural network model; Neural network model: selects the corresponding model branch for prediction based on the season, time period or location of the input data and the prediction results of the random forest model; The prediction results are processed using weighted average to output the final predicted water environment health status evaluation results.

Images