Unsupervised clustering watershed partitioning method based on graph embedding
Through the unsupervised clustering method based on graph embedding, the spatiotemporal embedding feature representation of the basin network is obtained, which solves the problem that the spatiotemporal characteristics in water environment monitoring is not fully considered, and more accurate basin partitioning and management is achieved.
Patent Information
- Application Number
- CN202311108639.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-31
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art fails to fully consider the spatiotemporal characteristics of water quality data in water environment monitoring, resulting in inaccurate basin partitioning results.
The unsupervised clustering method based on graph embedding is adopted to obtain the spatial and temporal embedding feature representation of the basin network, and combine the standardized processing and cluster analysis of water quality monitoring data to realize the spatiotemporal partitioning of the basin.
It improves the accuracy of basin division, can manage the basin water environment more scientifically, and provides scientific basis for basin ecological management and emergency warning.
Smart Images
Figure CN119537988A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning, graph embedding learning and network representation, and in particular to an unsupervised clustering watershed partitioning method based on graph embedding. Background Art
[0002] Monitoring river conditions is crucial to river basin management, because human life cannot be separated from water resources, which will directly affect human health and safety. Environmental pollution discharges various pollutants into nature, exceeding the natural self-purification capacity, causing changes in the water environment, destroying the ecological balance, and causing harm to human health, production and life. Therefore, scientific management of the water ecological environment is an important part of ecological environmental development. An important part of water ecological management is to manage and govern water basins in different regions, realize grid division of water ecological basins, and divide different ecological basins into relatively independent regions, so as to manage the water flow sections in each region more scientifically.
[0003] The water quality data in the watershed water environment monitoring and management system are all time series data. In order to determine the similarity of time series, the commonly used method is to cluster the monitoring sites. The methods currently used in research are mostly the Euclidean algorithm, which compares two time series data by calculating the sum of linear distances at the same time. Some use the dynamic time warping (DTW) algorithm to compare two time series by changing the time point. However, there are very few studies that apply the spatial distribution of sites to the analysis of water environment monitoring data, and the specific conditions of the water environment are not comprehensively considered, which may lead to inaccurate results. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides an unsupervised clustering watershed partitioning method based on graph embedding, which analyzes the spatial correlation between water quality monitoring stations in the watershed, and combines the spatial representation of the watershed with the characteristics of water quality indicators, thereby improving the embedded expression of the watershed network, thereby providing reliable data support for watershed zoning management and providing a scientific basis for zoning management.
[0005] The technical solution adopted by the present invention is as follows: an unsupervised clustering watershed partitioning method based on graph embedding, comprising the following steps:
[0006] Step 1: Obtain water quality monitoring data and preprocess the water quality monitoring data;
[0007] Step 2: Standardize the preprocessed data;
[0008] Step 3: Use the graph embedding method to obtain the spatiotemporal embedding feature representation of the watershed network.
[0009] Step 4: Reduce the dimension of the obtained spatiotemporal embedding feature representation;
[0010] Step 5: Use the reduced-dimensional spatiotemporal embedding feature representation of the watershed network to obtain clustering results through cluster analysis method;
[0011] Step 6: Conduct comprehensive evaluation and analysis on the clustering results to obtain the spatiotemporal seasonal division of the watershed.
[0012] The pre-processing comprises the following steps:
[0013] 1) Use the mean method to supplement the water quality data monitoring value: For the missing water quality values, calculate the daily average of the water quality monitoring data and use it as the missing value to supplement the water quality monitoring data;
[0014] 2) For the supplemented water quality monitoring data, use the box plot to process abnormal data:
[0015] The box plot method is used to calculate the upper limit and lower limit of each water quality monitoring indicator, and the abnormal data is processed:
[0016] IQR=Q3-Q1
[0017] Upper limit=Q3+1.5*IQR
[0018] Lower limit=Q1-1.5*IQR
[0019] Among them, Q1 is the first quartile, Q3 is the third quartile, and IQR is the interquartile range;
[0020] Delete the data that exceeds the upper limit or is less than the lower limit.
[0021] The pre-processed data is standardized, and the water quality data of different indicators are processed into the same scale to obtain standardized data, specifically:
[0022] X_scaled=(XX.mean()) / X.std()
[0023] Among them, X_scaled is the standardized data, X is the feature matrix to be calculated, that is, the water quality monitoring data of the water quality monitoring indicators of the monitoring site after preprocessing, X.mean() is the mean value of the X data, and X.std() is the standard deviation of X.
[0024] The graph embedding method is the TADW method, whose input is the standardized water quality monitoring data, and obtains the node embedding representation of the watershed network, that is, the spatiotemporal embedding feature representation;
[0025] The monitoring sites are regarded as nodes in the watershed network, and the rivers are regarded as the edges between the monitoring sites. The propagation mode of the TADW method is used to express the co-occurrence probability between a node and other nodes within one step:
[0026] M=(A+A 2 +A 3 +…+A t ) / t
[0027] Among them, A is the adjacency matrix of the node, that is, node V i With node V j There is an edge e between ij , A i，j =1, otherwise 0; the probability of co-occurrence within t steps is A t , M is the probability matrix of co-occurrence within t steps.
[0028] The TADW method adopts the square loss function to solve the matrices W and H to obtain the node embedding representation of the watershed network, and its propagation is as follows:
[0029]
[0030] Among them, λ represents the factor of the balance component, which is set to 2000 by default, and F represents the standardized water quality monitoring data;
[0031] During the training process, W and H are optimized alternately, and T is a fixed part; the obtained W and HT are used to represent the embedding representation vector of the node, and the spatiotemporal embedding feature representation is obtained by concatenating the W matrix and the HT matrix.
[0032] The standardized water quality monitoring data obtained in step 2 is concatenated with the spatiotemporal embedding feature representation to obtain the structural representation of the entire watershed network for representing the spatiotemporal embedding feature representation and the content representation for representing the water quality monitoring data, which are used as the input vector of the clustering analysis method:
[0033] F_embeddings=W+HT+F
[0034] Among them, W and HT are two matrix vectors, representing the embedded representation vectors of the watershed nodes, and F is the standardized water quality monitoring data.
[0035] The cluster analysis method comprises the following steps:
[0036] The clustering results of the two clustering methods are obtained by using the k-means method and the DBSCAN method respectively;
[0037] The spatiotemporal clustering results of the watershed obtained by the two clustering methods were analyzed, and the results were sorted and classified according to seasonality to obtain the spatiotemporal seasonal zoning results of the watershed.
[0038] An unsupervised clustering watershed partitioning system based on graph embedding, including:
[0039] The data processing module is used to obtain water quality monitoring data and pre-process the water quality monitoring data; and standardize the pre-processed data;
[0040] The feature representation module is used to obtain the spatiotemporal embedding feature representation of the watershed network through the graph embedding method for the standardized data; and to reduce the dimension of the obtained spatiotemporal embedding feature representation;
[0041] The clustering and partitioning module is used to obtain clustering results by clustering analysis methods from the spatiotemporal embedding features of the watershed network after dimensionality reduction; the clustering results are comprehensively evaluated and analyzed to obtain the spatiotemporal seasonal partitioning of the watershed.
[0042] A device for unsupervised clustering watershed partitioning based on graph embedding comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the unsupervised clustering watershed partitioning method based on graph embedding when executing the computer program.
[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the unsupervised clustering watershed partitioning method based on graph embedding is implemented.
[0044] The present invention has the following beneficial effects and advantages:
[0045] 1. The present invention uses deep learning technology and graph embedding technology to obtain the embedded representation of the watershed network, acquires the spatial features of the watershed network, compensates for the randomness of traditional time series data, and improves the accuracy of watershed division.
[0046] 2. The present invention uses the proposed graph embedding unsupervised clustering method. The graph embedding part obtains the spatiotemporal feature extraction of the watershed network, and the clustering analysis part clusters sites with similar spatiotemporal distribution into a cluster to obtain the spatiotemporal partitioning of the watershed.
[0047] 3. The present invention can be applied to the actual watershed zoning system, providing a scientific basis for scientific management of watershed ecology and emergency warning, and providing useful information for zoning management. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a flow chart of the method of the present invention.
[0049] Figure 2 It is a model architecture diagram of the present invention. DETAILED DESCRIPTION
[0050] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.
[0052] An unsupervised clustering watershed partitioning method based on graph embedding includes the following steps:
[0053] Obtain water quality monitoring data and pre-process the water quality monitoring data;
[0054] Standardize the preprocessed data, process the data of different indicators to the same scale, and obtain normalized data;
[0055] The obtained watershed network embedding representation vector is combined with the original features to obtain the spatiotemporal feature representation of the watershed network;
[0056] Reduce the dimension of the obtained feature representation;
[0057] The reduced-dimensional watershed network representation is input into the cluster analysis method and visualized simultaneously.
[0058] The preprocessing includes using the mean method to supplement the water quality data monitoring value, specifically: for the missing values of water quality, the daily average value of the water quality index data is calculated, and it is used as the missing value to supplement the water quality monitoring data.
[0059] The preprocessing includes: for the supplemented water quality monitoring data, using a box plot to process abnormal monitoring data, including the following steps: using a box plot method to calculate the upper limit and lower limit of each water quality monitoring index, and processing the abnormal data:
[0060] IQR=Q3-Q1
[0061] Upper limit=Q3+1.5*IQR
[0062] Lower limit=Q1-1.5*IQR
[0063] Among them, Q1 is the first quartile, Q3 is the third quartile, and IQR is the interquartile range.
[0064] The standardization processing method is X_scaled = (XX.mean()) / X.std(), which removes the mean value of the preprocessed data by attribute and scales it to unit variance to standardize the features so that each indicator is on the same scale.
[0065] The graph embedding method is the TADW method, which takes standardized water quality index data as input to obtain an embedded representation of the watershed network.
[0066] The spatiotemporal correlation characteristics of the water quality monitoring stations include the spatial embedding representation vector of the watershed network and the water quality index data monitored by the monitoring stations.
[0067] The unsupervised clustering watershed partitioning method based on graph embedding, wherein the dimensionality reduction method is T-SNE dimensionality reduction.
[0068] The clustering methods are k-means and DBSCAN methods, and the clustering results are visualized to obtain a partition clustering map of the watershed.
[0069] The TADW method is used to obtain the representation learning of the water quality monitoring network, that is, to obtain the embedded representation of the watershed network, and its propagation mode is: M = (A + A 2 +A 3 +…+A t ) / t
[0070] Where A is the adjacency matrix of the vertex, that is, the vertex V i With vertex V j There is an edge e between ij , then A i，j =1, otherwise 0. It represents the probability of co-occurrence between a node and other nodes within one step. Then, the probability of co-occurrence within t steps is A t , M is the probability matrix of co-occurrence within t steps.
[0071] The TADW method solves the matrices W and H by using a square loss function, and its propagation is as follows:
[0072]
[0073] During the training process, W and H are optimized alternately, and T is kept as a fixed part. The obtained W and HT can be regarded as the representation of the vertex, so by splicing the W matrix and the HT matrix, a feature representation with attribute information (watershed network monitoring data information) can be obtained.
[0074] The spatiotemporal characteristics of the watershed, because the water quality monitoring data has a specific point of time nature, the monitoring data and the watershed network embedding representation are matrix spliced to obtain the structural representation and content representation of the entire network as the input vector of cluster analysis, and the final expression is as follows:
[0075] F_embeddings=W+HT+F
[0076] The dimensionality reduction method is T-SNE, and the value of the perplexity parameter is determined by observing the performance of different parameter values.
[0077] The clustering methods include k-means method and DBSCAN method. For k-means method, the elbow method is used to determine the K value. The DBSCAN method determines the parameter values by observing the performance of different parameters.
[0078] The present invention first obtains the correlation between water quality monitoring indicators, and then performs standardization processing on the water quality monitoring data. The standardized data is input into the graph embedding model to obtain the spatiotemporal feature vector of the watershed network, and then the spatiotemporal feature vector is reduced in dimension using the T-SNE method to obtain the reduced-dimensional data. The reduced-dimensional data is input into the clustering methods k-means and DBSCAN to obtain the spatiotemporal cluster distribution of the watershed. The present invention proves the accuracy of the method through relevant experiments.
[0079] like Figure 1 , Figure 2 As shown in Figure 1, the unsupervised clustering method based on graph embedding includes the following steps:
[0080] Step 1: Input the sorted water quality data of the monitoring stations, a total of ten columns of data, including the name of the monitoring station, monitoring time, and eight water quality monitoring index data, including pH, ammonia nitrogen, conductivity, permanganate index, dissolved oxygen, water temperature, total phosphorus, and total nitrogen;
[0081] Step 2: Use the mean method to supplement the missing values of water quality index data;
[0082] Step 3: Based on the data values obtained in step 2, use the box plot method to calculate the upper and lower limits, delete the outliers, and fill them with the mean;
[0083] Step 4: Use the z-value normalization method to standardize the data obtained in step 3 and convert all eigenvalues to the same dimension.
[0084] Step 5: Segment the data obtained in step 4 according to time slices and input them into the graph embedding model to obtain the spatiotemporal graph embedding feature representation of the watershed network;
[0085] Step 6: Use the T-SNE dimensionality reduction method to reduce the dimension of the spatiotemporal representation vector obtained in step 5;
[0086] Step 7: Input the data obtained in step 6 into the clustering method and adjust the clustering parameters;
[0087] Step 8: Input the watershed network representation vector obtained in step 6 after dimensionality reduction into the clustering analysis methods k-means and DBSCAN with adjusted parameters, respectively, to obtain the clustering results of the two clustering methods.
[0088] Step 9: Analyze and integrate the spatiotemporal clustering results of the watershed obtained by the two clustering methods to obtain the seasonal spatiotemporal partitioning results of the watershed.
[0089] The standardization method in step 4 can process data of different dimensions so that each indicator is in the same dimension. This method standardizes the features by deleting the mean value and scaling the unit variance according to the attribute. The formula is as follows:
[0090] X_scaled=(XX.mean()) / X.std()
[0091] Among them, X_scaled is the standardized data, X is the feature matrix to be calculated, that is, the eight water quality monitoring indicator data of the monitoring site, X.mean() is the mean value of the X data, and X.std() is the standard deviation of X.
[0092] The spatiotemporal characteristics of the watershed network in step 5 include two parts: one is the spatial embedding representation vector of the watershed network after the TADW model; the other is the time series data of the watershed water quality index after standardization. The reconstruction formula in TADW is as follows:
[0093] M=(A+A 2 +A 3 +…+A t ) / t
[0094] Where A is the adjacency matrix, vertex V i With vertex V j There is an edge e between ij , then A i，j =1, otherwise 0. M represents the probability of co-occurrence between a node and other nodes within one step. Here, it is assumed that neighboring nodes move to their neighboring nodes with the same probability. Then, the probability of co-occurrence in t steps is A t , M is the probability matrix of co-occurrence within t steps.
[0095] The representation vector of the watershed network spatial node embedding is solved by minimizing the loss function. In TADW, the square loss function is used to train and optimize the matrices W and H. The formula is as follows:
[0096]
[0097] By minimizing the formula, we can obtain the spatial node representation vectors W and HT of the watershed network. By concatenating the W matrix with the HT matrix, we can obtain the embedded representation of the watershed network with attribute information.
[0098] The final spatiotemporal representation vector of the watershed is the spatial node embedding representation of the watershed network and the watershed monitoring water quality data, and the formula is as follows:
[0099] F_embeddings=W+HT+F
[0100] The dimensionality reduction method in step 6 is T-SNE, which can make the distance between points with greater similarity smaller in the t-distribution, while the distance between points with lower similarity is farther in the t-distribution. The perplexity parameter value of this method is set for experimental visualization.
[0101] In step 7, the clustering methods include the k-means method and the DBSCAN method. The k-means method sets the k value through the elbow method, and the DBSCAN method adjusts the effect of clustering analysis by setting the two parameters of min_sample and eps.
[0102] In step 9, the results of the two different clustering methods are integrated according to seasonality, and the results are classified into time categories according to seasonality to form seasonal spatiotemporal partitions of the watershed.
Claims
1. An unsupervised clustering watershed partitioning method based on graph embedding, characterized in that: The following steps are involved: Step 1: Obtain water quality monitoring data and preprocess the water quality monitoring data; Step 2: Standardize the preprocessed data; Step 3: Use the graph embedding method to obtain the spatiotemporal embedding feature representation of the watershed network. Step 4: Reduce the dimension of the obtained spatiotemporal embedding feature representation; Step 5: Use the reduced-dimensional spatiotemporal embedding feature representation of the watershed network to obtain clustering results through cluster analysis method; Step 6: Conduct comprehensive evaluation and analysis on the clustering results to obtain the spatiotemporal seasonal division of the watershed.
2. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 1 is characterized in that: The pre-processing comprises the following steps: 1) Use the mean method to supplement the water quality data monitoring value: For the missing water quality values, calculate the daily average of the water quality monitoring data and use it as the missing value to supplement the water quality monitoring data; 2) For the supplemented water quality monitoring data, use the box plot to process abnormal data: The box plot method is used to calculate the upper limit and lower limit of each water quality monitoring indicator, and the abnormal data is processed: IQR=Q3-Q1 Upper limit=Q3+1.5*IQR Lower limit=Q1-1.5*IQR Among them, Q1 is the first quartile, Q3 is the third quartile, and IQR is the interquartile range; Delete the data that exceeds the upper limit or is less than the lower limit.
3. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 1, characterized in that: The pre-processed data is standardized, and the water quality data of different indicators are processed into the same scale to obtain standardized data, specifically: X_scaled=(XX.mean()) / X.std() Among them, X_scaled is the standardized data, X is the feature matrix to be calculated, that is, the water quality monitoring data of the water quality monitoring indicators of the monitoring site after preprocessing, X.mean() is the mean value of the X data, and X.std() is the standard deviation of X.
4. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 1, characterized in that: The graph embedding method is the TADW method, whose input is the standardized water quality monitoring data, and obtains the node embedding representation of the watershed network, that is, the spatiotemporal embedding feature representation; The monitoring sites are regarded as nodes in the watershed network, and the rivers are regarded as the edges between the monitoring sites. The propagation mode of the TADW method is used to express the co-occurrence probability between a node and other nodes within one step: M=(A+A 2 +A 3 +…+A t ) / t Among them, A is the adjacency matrix of the node, that is, node V i With node V j There is an edge e between ij , A i，j =1, otherwise 0; the probability of co-occurrence within t steps is A t , M is the probability matrix of co-occurrence within t steps.
5. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 1 or 4, characterized in that: The TADW method adopts the square loss function to solve the matrices W and H to obtain the node embedding representation of the watershed network, and its propagation is as follows: Among them, λ represents the factor of the balance component, which is set to 2000 by default, and F represents the standardized water quality monitoring data; During the training process, W and H are optimized alternately, and T is a fixed part; the obtained W and HT are used to represent the embedding representation vector of the node, and the spatiotemporal embedding feature representation is obtained by concatenating the W matrix and the HT matrix.
6. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 5, characterized in that: The standardized water quality monitoring data obtained in step 2 is concatenated with the spatiotemporal embedding feature representation to obtain the structural representation of the entire watershed network for representing the spatiotemporal embedding feature representation and the content representation for representing the water quality monitoring data, which are used as the input vector of the clustering analysis method: F_embeddings=W+HT+F Among them, W and HT are two matrix vectors, representing the embedded representation vectors of the watershed nodes, and F is the standardized water quality monitoring data.
7. The unsupervised clustering watershed partitioning method based on graph embedding according to claim 1, characterized in that: The cluster analysis method comprises the following steps: The clustering results of the two clustering methods are obtained by using the k-means method and the DBSCAN method respectively; The spatiotemporal clustering results of the watershed obtained by the two clustering methods were analyzed, and the results were sorted and classified according to seasonality to obtain the spatiotemporal seasonal zoning results of the watershed.
8. Unsupervised clustering watershed partitioning system based on graph embedding, characterized by: include: A data processing module is used to obtain water quality monitoring data and pre-process the water quality monitoring data; Standardize the preprocessed data; The feature representation module is used to obtain the spatiotemporal embedding feature representation of the watershed network by using the graph embedding method to process the standardized data; Perform dimensionality reduction on the obtained spatiotemporal embedding feature representation; The clustering partitioning module is used to obtain clustering results by clustering analysis method based on the spatiotemporal embedding feature representation of the watershed network after dimensionality reduction; The clustering results are comprehensively evaluated and analyzed to obtain the spatiotemporal seasonal divisions of the watershed.
9. An unsupervised clustering watershed partitioning device based on graph embedding, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the graph embedding-based unsupervised clustering watershed partitioning method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the unsupervised clustering watershed partitioning method based on graph embedding as described in any one of claims 1 to 7 is implemented.