An adaptive clustering method and system for industrial water quality detection

Abstract

The application discloses an adaptive clustering method and system for industrial water quality detection, and the method comprises the following steps: pre-processing industrial water quality detection data; calculating the divergence between data samples and converting the divergence into distance, and performing matrixization on the data based on distance measurement; using a histogram algorithm to count the distance frequency distribution of the samples, and determining the highest frequency interval of the samples; setting the upper limit of the distance interval as an adaptive search radius of a mean shift algorithm; using the mean shift algorithm to perform clustering analysis on the industrial water quality detection data; and visualizing the water quality samples and marking abnormal samples. The application does not need to set the clustering number in advance through experiments and experience, can adaptively process various data by obtaining the clustering characteristics in the data set, can more accurately describe the clustering structure of the data, and can make the clustering result closer to the actual needs of users.

Description

Technical Field

[0001] This invention relates to the field of computer and data analysis technology, and in particular to an adaptive clustering method and system for industrial water quality testing. Background Technology

[0002] In the context of global environmental protection and sustainable development, industrial wastewater discharge poses a threat to water resources and the ecological environment. Accurate monitoring of industrial water quality is crucial for preventing water pollution and achieving water recycling. Therefore, researching adaptive clustering methods for industrial water quality monitoring is of great significance. This method not only helps improve the accuracy and real-time performance of water quality monitoring but also provides technical support for environmental protection and sustainable water resource utilization, promoting industrial production towards a more environmentally friendly and sustainable direction.

[0003] Clustering algorithms are generally classified into several types, including hierarchical clustering, partitioning clustering, density clustering, grid clustering, and model clustering. Hierarchical clustering methods gradually merge or split data points by constructing a hierarchical cluster structure, but their computational complexity is high, making them suitable for small datasets. Partitioning clustering methods attempt to divide the dataset into K disjoint subsets, such as the K-means algorithm, but this method requires pre-determining the number of clusters K and is sensitive to initialization and noise. Density clustering algorithms, such as DBSCAN, discover clusters of arbitrary shapes based on the density of data points, but parameter selection significantly impacts the results. Grid clustering methods divide the data space into grid cells and then perform clustering on the grid, suitable for handling high-dimensional data, but may fail to capture irregularly shaped clusters. Model clustering is based on fitting a probabilistic model to the data, such as the Gaussian mixture model, but model selection and parameter estimation can be complex.

[0004] Traditional clustering algorithms suffer from several common drawbacks, which limit their performance and application scope to some extent. First, clustering algorithms typically require pre-setting parameters such as the number of clusters and distance thresholds. The selection of these parameters significantly impacts the clustering results, and choosing suitable parameters often relies on experience and understanding of the data. For different datasets and problems, multiple trials and parameter adjustments may be necessary to obtain satisfactory clustering results, increasing the algorithm's complexity and uncertainty. Second, clustering algorithms are sensitive to data initialization and noise. Some clustering algorithms, such as K-means, randomly select cluster centers during initialization, which may lead the algorithm to get trapped in local optima rather than the global optimum. Furthermore, noise and outliers in the data can also negatively affect the clustering results, causing instability or inaccuracy. Therefore, adaptively determining various parameters in the clustering process to achieve high clustering performance is crucial for the successful application of the proposed method. Summary of the Invention

[0005] To overcome the above shortcomings, this invention provides an Industrial Water Quality Adaptive Clustering (IWQAC) method and system for industrial water quality testing. It uses JS divergence to estimate the distance between samples, utilizes histogram statistical algorithms to analyze the distance frequency distribution, and uses high-frequency distances for mean-shift clustering.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention proposes an adaptive clustering method for industrial water quality testing, comprising:

[0008] Step 1: Perform data preprocessing on industrial water quality testing data;

[0009] Step 2: Calculate the distance between the pretreated industrial water quality test data, and then matrix the calculated distance between the data.

[0010] Step 3: Use a histogram algorithm to count the distances between data points and determine the distance interval with the highest data distribution.

[0011] Step 4: Use the upper limit r of the distance interval as the adaptive search radius of the mean shift algorithm;

[0012] Step 5: Based on the adaptive search radius, use the mean shift algorithm to divide the data within the adaptive search range into the corresponding cluster sets.

[0013] Furthermore, the preprocessing in step 1 includes removing null and erroneous data.

[0014] Furthermore, in step 2, the JS divergence between the preprocessed industrial water quality test data is calculated, and the calculated JS divergence is used as the distance between the data.

[0015] Further, step 3 includes:

[0016] The maximum and minimum values ​​in the distance matrix are used to form the sample value interval. The sample value interval is divided into multiple distance intervals. The frequency of distance data appearing in each distance interval is counted. The distance data is used as the horizontal axis and the frequency or proportion of appearing in each distance interval is used as the vertical axis to count the distribution of distance between the data in the intervals. Then, the distance interval with the highest frequency or proportion is selected.

[0017] Further, step 5 includes:

[0018] Step 5.1: Select an unvisited initial sample point p in the industrial water quality testing data sample space. i Use it as the search center and find all sample points inside a circle with radius r;

[0019] Step 5.2: Calculate the density center m(p) of the sample points inside the circle. i ), and calculate p i With m(p) i If the distance between p is less than a threshold, the iteration stops; otherwise, p is stopped. i Move to m(p) i ), and repeat step 5.2;

[0020] Step 5.3: After obtaining the maximum density center point, determine whether the distance between the maximum density center point and the existing density center point is less than r. If it is greater than r, retain the obtained maximum density center point and its corresponding sample point; otherwise, merge the maximum density center point with the existing density center point.

[0021] Step 5.4: Repeat steps 5.1 to 5.3 until all sample points in the sample space have been visited;

[0022] Step 5.5: Organize the obtained cluster count and clustering results.

[0023] Furthermore, following step 5, the following steps are also included:

[0024] Visualize industrial water quality testing data and label abnormal data samples.

[0025] Another aspect of this invention proposes an adaptive clustering system for industrial water quality testing, comprising:

[0026] The data preprocessing module is used to preprocess industrial water quality testing data.

[0027] The distance matrix generation module is used to calculate the distance between pre-processed industrial water quality test data and to matrix the calculated distance between the data.

[0028] The distance interval derivation module is used to use a histogram algorithm to statistically analyze the distance between data points and determine the distance interval with the largest distribution of distance data.

[0029] An adaptive search radius determination module is used to take the upper limit r of the distance interval as the adaptive search radius of the mean drift algorithm;

[0030] The clustering module is used to divide data within the adaptive search range into corresponding cluster sets based on the mean shift algorithm and the adaptive search radius.

[0031] Furthermore, the system also includes:

[0032] The visualization module is used to visualize industrial water quality testing data and annotate abnormal data samples.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] This invention captures subtle differences in data distributions using JS divergence, providing a more accurate measure of data similarity, and uses the calculated JS divergence as the distance between data points. Then, based on histogram statistics, the number of distance intervals is determined, and the interval with the highest number of data points is identified, thereby determining the adaptive search radius. This determined adaptive search radius is then applied to the mean shift algorithm.

[0035] This invention does not require pre-setting the number of clusters through experiments and experience. Instead, it obtains the clustering characteristics within the dataset through statistics and adaptively processes various types of data. At the same time, this invention can more accurately describe the clustering structure of the data, making the clustering results closer to the user's actual needs.

[0036] This invention features parameter-free operation, wide applicability, and strong robustness, and can obtain a good clustering result in a relatively short time. Attached Figure Description

[0037] Figure 1 This is one of the flowcharts for an adaptive clustering method for industrial water quality testing according to an embodiment of the present invention;

[0038] Figure 2 This is a second flowchart of an adaptive clustering method for industrial water quality testing according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of an adaptive clustering system architecture for industrial water quality testing according to an embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0041] like Figure 1 , Figure 2 As shown, an adaptive clustering method for industrial water quality testing includes:

[0042] S101: Preprocess the industrial water quality testing dataset. Null and erroneous values ​​in the data can lead to inaccurate analysis results. Therefore, before performing data analysis, clean up these invalid data to ensure the accuracy and reliability of the results.

[0043] S102: Calculate the similarity between data sample points in the preprocessed industrial water quality testing dataset and generate a distance matrix. As one possible implementation, this invention uses JS divergence to calculate the similarity between samples. JS divergence and distance similarity have consistent measurement characteristics, therefore JS divergence can be used instead of distance measurement.

[0044] Specifically, the JS divergence is calculated as follows:

[0045]

[0046] Where P1 and P2 represent the probability distributions of two different data samples, denoted by J. ik This represents the value in the i-th row and k-th column of matrix J, taking values ​​[0, 1]. When J... ik When J equals 0, it means that the probability distributions of the two data samples are exactly the same. ik When the value is close to 1, it indicates that the difference between the probability distributions of the two data samples is relatively large. Similarly, the distance similarity measure is consistent with the JS divergence characteristic between samples. Therefore, in this embodiment, the distance matrix D = J is set.

[0047] S103: Select the distance interval with the highest frequency based on the interval distance distribution. As one possible implementation, this invention uses a histogram statistical algorithm to transform numerical values ​​into a distribution, including:

[0048] The maximum and minimum values ​​in the distance matrix are used to form the sample value interval. The sample value interval is divided into multiple distance intervals. The frequency of distance data appearing in each distance interval is counted. The distance data is used as the horizontal axis and the frequency or proportion of appearing in each distance interval is used as the vertical axis to count the distribution of distance between the data in the intervals. Then, the distance interval with the highest frequency or proportion is selected.

[0049] S104: Determine the search radius for the mean shift algorithm. Select the distance interval with the highest frequency based on the interval distance distribution. (j and r are the lower and upper bounds of the distance interval, respectively), select the distance interval. The upper bound r is the search radius of the mean-shift algorithm. It is worth noting that the combination of S102, S103, and S104 is for adaptively determining the search radius using statistical methods.

[0050] S105: Cluster the data using the mean shift algorithm. This includes:

[0051] S105.1: Based on the statistical frequency results, the upper limit r of the distance interval with the highest frequency is used as the search radius;

[0052] S105.2: Select an unvisited initial point p in the sample space. i Use it as the search center and find all sample points inside a circle with radius r;

[0053] S105.3: Calculate the density center m(p) of the sample points inside the circle. i ), and measure p i With m(p) iIf the distance between p and p is less than a threshold, stop the iteration; otherwise, set p to p. i Move to m(p) i ), and repeat step S105.3;

[0054] S105.4: After obtaining the maximum density center point, determine whether the distance between the maximum density center point and the existing density center point is less than r; if it is greater than r, retain the obtained maximum density center point and its corresponding sample point; otherwise, merge the maximum density center point with the existing density center point.

[0055] S105.5: Repeat steps S105.2 to S105.4 until all sample points in the sample space have been visited;

[0056] S105.6: Organize to obtain the number of clusters and clustering results.

[0057] Furthermore, following S105, the following is also included:

[0058] S106: Visualize industrial water quality testing data and label abnormal data samples.

[0059] Based on the above embodiments, such as Figure 3 As shown, this invention also proposes an adaptive clustering system for industrial water quality testing, comprising:

[0060] The data preprocessing module is used to preprocess industrial water quality testing data.

[0061] The distance matrix generation module is used to calculate the distance between pre-processed industrial water quality test data and to matrix the calculated distance between the data.

[0062] The distance interval derivation module is used to use a histogram algorithm to statistically analyze the distance between data points and determine the distance interval with the largest distribution of distance data.

[0063] An adaptive search radius determination module is used to take the upper limit r of the distance interval as the adaptive search radius of the mean drift algorithm;

[0064] The clustering module is used to divide data within the adaptive search range into corresponding cluster sets based on the mean shift algorithm and the adaptive search radius.

[0065] Furthermore, the system also includes:

[0066] The visualization module is used to visualize industrial water quality testing data and annotate abnormal data samples.

[0067] In summary, the entire clustering method of this invention does not require manual parameter setting. It adopts statistical and density-based mean shift algorithms, which has better practicality and meets the needs of real-world applications. At the same time, it can resist the influence of noise or outliers in the data to a certain extent.

[0068] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An adaptive clustering method for industrial water quality testing, characterized in that, include: Step 1: Perform data preprocessing on industrial water quality testing data; Step 2: Calculate the distance between the pretreated industrial water quality test data, and then matrix the calculated distances between the data. Step 3: Use a histogram algorithm to count the distances between data points and determine the distance interval with the highest data distribution. Step 4: Set the upper limit of the distance interval. As the adaptive search radius of the mean-shift algorithm; Step 5: Based on the adaptive search radius, use the mean shift algorithm to divide the data within the adaptive search range into the corresponding cluster sets; In step 2, the JS divergence between the pre-processed industrial water quality test data is calculated, and the calculated JS divergence is used as the distance between the data. Step 3 includes: The maximum and minimum values ​​in the distance matrix are used to form the sample value interval. The sample value interval is divided into multiple distance intervals. The frequency of distance data appearing in each distance interval is counted. The distance data is used as the horizontal axis and the frequency or proportion of appearing in each distance interval is used as the vertical axis to count the distribution of distance between the data in the intervals. Then, the distance interval with the highest frequency or proportion is selected.

2. The adaptive clustering method for industrial water quality testing according to claim 1, characterized in that, The preprocessing in step 1 includes removing null and erroneous data.

3. The adaptive clustering method for industrial water quality testing according to claim 1, characterized in that, Step 5 includes: Step 5.1: Select an unvisited initial sample point in the industrial water quality testing data sample space. As the search center, and to find a radius of All sample points within the circle; Step 5.2: Calculate the density center of the sample points inside the circle. and calculate and If the distance between them is less than a threshold, the iteration stops; otherwise, it stops. Move to And repeat step 5.2; Step 5.3: After obtaining the maximum density center point, determine whether the distance between the maximum density center point and the existing density center point is less than [the specified value]. If greater than If the maximum density center point is obtained, its corresponding sample point is retained; otherwise, the maximum density center point is merged with the existing density center point. Step 5.4: Repeat steps 5.1 to 5.3 until all sample points in the sample space have been visited; Step 5.5: Organize the obtained cluster count and clustering results.

4. The adaptive clustering method for industrial water quality detection according to claim 1, characterized in that, Following step 5, the following is also included: Visualize industrial water quality testing data and label abnormal data samples.

5. An adaptive clustering system for industrial water quality testing, characterized in that, include: The data preprocessing module is used to preprocess industrial water quality testing data. The distance matrix generation module is used to calculate the distance between pre-processed industrial water quality test data and to matrix the calculated distance between the data. The distance interval derivation module is used to use a histogram algorithm to statistically analyze the distance between data points and determine the distance interval with the largest distribution of distance data. An adaptive search radius calculation module is used to determine the upper limit of the distance interval. As the adaptive search radius of the mean-shift algorithm; The clustering module is used to divide data within the adaptive search range into corresponding cluster sets based on the mean shift algorithm and the adaptive search radius; In the distance matrix generation module, the JS divergence between the preprocessed industrial water quality test data is calculated, and the calculated JS divergence is used as the distance between the data. The distance interval calculation module is specifically used for: The maximum and minimum values ​​in the distance matrix are used to form the sample value interval. The sample value interval is divided into multiple distance intervals. The frequency of distance data appearing in each distance interval is counted. The distance data is used as the horizontal axis and the frequency or proportion of appearing in each distance interval is used as the vertical axis to count the distribution of distance between the data in the intervals. Then, the distance interval with the highest frequency or proportion is selected.

6. The adaptive clustering system for industrial water quality testing according to claim 5, characterized in that, The system also includes: The visualization module is used to visualize industrial water quality testing data and annotate abnormal data samples.

Images