Data identification method, system, device and storage medium based on K-means clustering

By using the genetic algorithm to determine the initial clustering center and dynamically adjusting the number of clusters K in the K-mean clustering algorithm, combined with the evaluation index evaluation results, the sensitivity problem of the K-mean clustering algorithm in automatic data recognition is solved, and the data is improved The accuracy and efficiency of identification enhance the stability and generalization capabilities of the model.

CN118503768BActive Publication Date: 2024-11-05XIAMEN YUANTING INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410961615.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-05
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing K-mean clustering algorithm has problems such as initial clustering center sensitivity, difficulty in determining K value and sensitivity to noise and outliers in automatic data identification, which affects the accuracy of data classification and clustering.

Method used

Using a data recognition method based on K-mean clustering, the initial cluster center is determined through the genetic algorithm, and the cluster number K is dynamically adjusted during the clustering process, and the cluster results are evaluated based on internal, external and relative evaluation indicators to remove the non-existent and non-existent The results of the standard are met, and the data points that have the greatest impact on the training quality of deep learning models are identified and marked.

Benefits of technology

It improves the automation level and accuracy of data screening, enhances the robustness and stability of clustering, significantly improves the efficiency of data processing and the generalization ability of the model, and reduces the need for manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118503768B_ABST
    Figure CN118503768B_ABST
Patent Text Reader

Abstract

The present invention provides a data identification method, system, device and storage medium based on K-means clustering, including: cleaning and standardizing the input data set to obtain pre-processed data; determining the initial center of K-means clustering by genetic algorithm; performing K-means clustering on the pre-processed data, and dynamically adjusting the number of clusters K during clustering to obtain clustering results; evaluating the clustering results by using internal evaluation indicators, external evaluation indicators and relative evaluation indicators, removing the substandard clustering results, and obtaining the clustering results to be processed; based on the clustering results to be processed, identifying and marking the data points that have the greatest impact on the training quality of the deep learning model, the data points are located in the cluster center or dense area; the identified data points are combined into a training data set for deep learning model training. Using advanced clustering algorithms and automated processing technologies, the efficiency of processing large data sets and the accuracy of data identification are significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data identification method, system, device and storage medium based on K-means clustering. Background Art

[0002] In the fields of data analysis, machine learning, and artificial intelligence, the quality of data directly affects the training effect and prediction accuracy of the model. Traditional data processing methods often rely on manual data screening and quality assessment, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, leading to inconsistency and repeatability problems in data selection. With the rapid increase in the amount of data, especially in big data application scenarios, complex, heterogeneous, unbalanced data with high-dimensional feature space are usually used, which makes manual processing more difficult. Therefore, there is an urgent need for an automated method to identify and select high-quality data for building efficient prediction models. Automatic data identification methods can not only improve the efficiency of data processing, but also enhance the generalization ability and reliability of the model.

[0003] The K-means clustering algorithm is a widely used unsupervised learning algorithm that divides data into multiple categories by calculating the distance between data points, each category is represented by its center point. However, the K-means clustering algorithm has a wide range of applications in data classification and clustering, but it has some limitations in automatic data identification: (1) Sensitive to initial cluster centers: The clustering results of the K-means algorithm are affected by the selection of initial cluster centers, and different initial values ​​may lead to different clustering results. (2) Difficulty in determining the K value: In automatic data identification, selecting a suitable K value is a challenge because the optimal K value may be different for different data sets. (3) Sensitive to noise and outliers: The K-means algorithm is sensitive to noise and outliers, which may affect the accuracy of the clustering results.

[0004] In view of this, the present application proposes a method, system, device and storage medium for automatic identification of high-quality data for large models based on K-means clustering, which can effectively identify and screen out high-quality data points that are most beneficial to deep learning model training. Summary of the invention

[0005] In order to solve the problem that the existing K-means clustering algorithm is sensitive to the initial cluster center in automatic data identification, the present application provides a data identification method, system, device and storage medium based on K-means clustering to solve the above-mentioned technical defects.

[0006] According to the first aspect of the present application, a data identification method based on K-means clustering is proposed, and the method comprises the following steps:

[0007] S1. Clean and standardize the input data set to obtain preprocessed data;

[0008] S2, determine the initial center of K-means clustering through genetic algorithm;

[0009] S3, performing K-means clustering on the preprocessed data, and dynamically adjusting the number of clusters K during clustering to obtain clustering results;

[0010] S4. Evaluate the clustering results using internal evaluation indicators, external evaluation indicators, and relative evaluation indicators, remove the clustering results that do not meet the standards, and obtain the clustering results to be processed;

[0011] The expression of the internal evaluation index is:

[0012]

[0013] In the formula represents the internal evaluation index, is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster;

[0014] The expression of external evaluation index is:

[0015]

[0016] In the formula represents external evaluation indicators, is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively;

[0017] The expression of relative evaluation index is:

[0018]

[0019] In the formula Represents a relative evaluation index, is the number of clusters, It is The average distance within a cluster, It is The average distance within a cluster, It is The center of the cluster, It is The center of the cluster, is the distance between different cluster centers;

[0020] S5. Based on the clustering results to be processed, identify and mark the data points that have the greatest impact on the training quality of the deep learning model, and the data points are located in the cluster center or dense area;

[0021] S6. The identified data points are combined into a training data set for use in deep learning model training.

[0022] In the above technical solution, the present application improves the automation level and accuracy of data screening by optimizing the evaluation criteria and data processing strategies in the clustering process.

[0023] Preferably, in step S1, the input data set is cleaned and standardized, including the following:

[0024] Automatically detect outliers in input data sets through machine learning algorithms, obtain abnormal data, and identify and process abnormal data based on dynamic thresholds of statistical methods;

[0025] Use K-nearest neighbor interpolation, multiple interpolation or prediction model-based interpolation to handle missing values ​​for different types of input data sets;

[0026] Normalize the input data set by min-max normalization or Z-score normalization;

[0027] For data with skewed distribution, logarithmic transformation or Box-Cox transformation method was used for transformation;

[0028] Perform data quality scoring and determine whether to re-clean the data based on the data quality score.

[0029] In the above technical solution, outliers are automatically detected through statistical methods and processed using dynamic thresholds, which can reduce the impact of outliers on model training and prediction results and improve the reliability and accuracy of data; normalization processing can help machine learning algorithms to better process data and improve the prediction performance of the model; based on the data quality score, it can be decided whether to re-clean the data, forming a closed-loop process for data cleaning to ensure that data quality is continuously improved.

[0030] Preferably, in step S2, determining the initial center of K-means clustering by genetic algorithm includes the following sub-steps:

[0031] S21, encoding the cluster centers as chromosomes of the genetic algorithm, each chromosome representing a group of cluster centers, wherein the cluster center is represented by a data point in the preprocessed data, randomly selecting a data point as an initial cluster center, and generating a plurality of chromosomes, wherein the size of the population is set by a parameter of the genetic algorithm;

[0032] S22, using the clustering fitness function to evaluate the fitness of each chromosome, and selecting the chromosome to enter the next generation according to the fitness;

[0033] S23, randomly pairing the chromosomes entering the next generation in step S22, and performing a crossover operation with a first preset probability to generate a new chromosome;

[0034] S24, randomly modifying one or more genes in the new chromosome in step S23 with a second preset probability to obtain a mutant chromosome;

[0035] S25, repeating steps S22 to S24 until the maximum number of iterations is reached;

[0036] S26. After the iteration, the chromosome with the highest fitness is selected from the final population as the initial center of the final K-means clustering.

[0037] The above technical solution can significantly improve the quality and efficiency of clustering by determining the initial center of K-means clustering through genetic algorithm, enhance the robustness and stability of the algorithm, and has good implementation and scalability. This has important application value for processing complex and uncertain data sets.

[0038] Preferably, in step S22, the clustering fitness function includes: a total internal square function, a fitness function of comprehensive distance and density, a fitness function considering outliers, a fuzzy clustering fitness function, and an energy function fitness function;

[0039] The total internal square function measures the sum of the squares of the distances from each data point to its cluster center. It is applicable to situations where the data points within the cluster are closely clustered together. Its expression is:

[0040]

[0041] In the formula represents the total internal square function, is the number of clusters, It is clusters, yes data points, yes The cluster center of

[0042] The fitness function of comprehensive distance and density is suitable for the case where the size, shape and density of clusters in the data set are uneven, and its expression is:

[0043]

[0044] In the formula Fitness function representing comprehensive distance and density is the adjustment parameter; It is clusters, yes data points, It is The center of the cluster, It is The center of the cluster, It is a data point The local density of

[0045] The fitness function that considers outliers is applicable to the situation where there are outliers in the data set and the influence of outliers needs to be processed. Its expression is:

[0046]

[0047] In the formula represents the fitness function considering outliers, It is clusters, yes The cluster center of It is a data point The outlier weight is determined by the data point Determined by the distance to the nearest cluster center;

[0048] The fuzzy clustering fitness function is applicable to the case where a data point belongs to multiple clusters, and its expression is:

[0049]

[0050] In the formula represents the fuzzy clustering fitness function, is the dataset containing all data points, It is a data point Belongs to cluster The membership degree of is the fuzziness coefficient, It is clustering The center point of

[0051] The energy function fitness function is suitable for situations where you want to discover data structures by simulating physical processes. Its expression is:

[0052]

[0053] In the formula represents the energy function fitness function, It is The center of the cluster, It is The center of the cluster, It is clusters, yes data points.

[0054] The above fitness function can be used to more comprehensively evaluate the effect of clustering, adapt to different data characteristics and distributions, and make the genetic algorithm more efficient and accurate in selecting cluster centers.

[0055] Preferably, in step S4, evaluating the clustering result includes the following sub-steps:

[0056] S41. Analyze the key characteristics of the preprocessed data, including dimension size, sample size, correlation between features and distribution uniformity, and use statistical test methods to evaluate the distribution characteristics of the preprocessed data;

[0057] S42. Select appropriate clustering evaluation indicators based on key characteristics using a decision tree or rule engine;

[0058] S43. Collect clustering performance feedback in the clustering process in real time, and adjust clustering evaluation indicators according to the clustering performance feedback.

[0059] In the above technical solutions, for specific data characteristics, such as high-dimensional data or non-uniformly distributed data, traditional clustering evaluation indicators may not be sufficient to provide effective performance feedback. The dynamic adjustment strategy can select the most suitable evaluation criteria according to the specific characteristics of the data, such as using a density-based evaluation method for high-dimensional data, or adjusting the calculation method of the silhouette coefficient to solve the problem of uneven data distribution, making it more sensitive to subtle changes in the data set. The method of dynamically adjusting the evaluation criteria increases the adaptability of clustering analysis. The most appropriate evaluation criteria are automatically selected for different data characteristics (such as dimension, distribution, density) and clustering results (such as cluster size, shape, and degree of dispersion). This allows the evaluation process to more accurately reflect the actual effect of clustering, thereby better guiding the adjustment and optimization of the clustering algorithm.

[0060] Further preferably, in step S42, a decision tree is constructed based on the key characteristics, which specifically includes the following sub-steps:

[0061] S421. Define key characteristics;

[0062] S422, constructing rules: if the key characteristic is determined to be high-dimensional data, the silhouette coefficient evaluation index is used; if the key characteristic is uneven data distribution or the presence of outliers, the fitness function considering outliers or the fuzzy clustering fitness function is used;

[0063] S423, implementing a decision tree or rule engine;

[0064] S424: Verify and adjust, and adjust the rules based on the feedback.

[0065] Preferably, a data identification method based on K-means clustering provided in the present application also includes: using data visualization tools to automatically generate data distribution graphics from preprocessed data; integrating the adaptive clustering evaluation index selection system into the clustering analysis software or platform, and configuring the corresponding API to realize the interaction between clustering evaluation index selection and clustering algorithm.

[0066] In a second aspect, the present application provides a data identification system based on K-means clustering, the system comprising:

[0067] A preprocessing module is configured to clean and standardize the input data set to obtain preprocessed data;

[0068] An initial center determination module, configured to determine the initial center of K-means clustering through a genetic algorithm;

[0069] A clustering result acquisition module is configured to perform K-means clustering on the preprocessed data and dynamically adjust the number of clusters K during clustering to obtain clustering results;

[0070] An evaluation module is configured to evaluate the clustering results using internal evaluation indicators, external evaluation indicators and relative evaluation indicators, and obtain the clustering results to be processed after removing the clustering results that do not meet the standards;

[0071] The expression of the internal evaluation index is:

[0072]

[0073] In the formula represents the internal evaluation index, is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster;

[0074] The expression of external evaluation index is:

[0075]

[0076] In the formula represents external evaluation indicators, is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively;

[0077] The expression of relative evaluation index is:

[0078]

[0079] In the formula Represents a relative evaluation index. is the number of clusters, It is The average distance within a cluster, It is The average distance within a cluster, It is The centers of the clusters, It is The centers of the clusters, is the distance between different cluster centers;

[0080] The recognition module identifies and marks the data points that have the greatest impact on the training quality of the deep learning model based on the clustering results to be processed. The data points are located in the cluster center or dense area;

[0081] The data set integration module is configured to form a training data set from the identified data points for use in deep learning model training.

[0082] In a third aspect, the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement a data identification method based on K-means clustering as described in any one of the above.

[0083] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor, the data identification method based on K-means clustering as described in any one of the above is implemented.

[0084] Compared with the prior art, the beneficial results of the present invention are:

[0085] (1) By introducing an improved K-means clustering method, the accuracy and efficiency of clustering are significantly improved. By adopting innovative initial cluster center selection techniques (such as K-means++ and genetic algorithms), automated data preprocessing, and optimized fitness functions, this method can effectively process large-scale data sets, improve the degree of automation of data processing, and reduce the need for manual intervention.

[0086] (2) The introduced clustering quality assessment technology enhances the applicability of the algorithm, enabling it to be applied in different industries. The combination of these technologies not only improves the performance of the algorithm in practical applications, but also expands its effectiveness and application scope in a variety of data environments.

[0087] (3) By cleaning and standardizing the input data set, outliers are effectively removed, missing values ​​are processed, and the data is normalized and transformed, thereby improving the quality and reliability of the data and providing more accurate input for subsequent data analysis and clustering.

[0088] (4) Dynamically adjusting the number of clusters K during the K-means clustering process can more flexibly adapt to the actual distribution of data and avoid poor clustering results caused by an inappropriate number of clusters.

[0089] (5) Using data visualization tools to automatically generate data distribution graphs helps to intuitively understand the structure and distribution of data; integrating the adaptive clustering evaluation index selection system into the clustering analysis software or platform makes it easier for users to use and configure, thereby improving the system's ease of use and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0091] Figure 1 is a flow chart of a data identification method based on K-means clustering according to the present application;

[0092] Figure 2 is a structural diagram of a data identification system based on K-means clustering according to the present application;

[0093] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0094] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0095] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0096] Figure 1 A flow chart of the data identification method based on K-means clustering of the present application is shown as follows: Figure 1 As shown, the method comprises the following steps:

[0097] S1. Clean and standardize the input data set to obtain preprocessed data.

[0098] In this embodiment, the data set includes, but is not limited to, data in the fields of finance, medical care, network security, social media analysis, and natural language processing. Preferably, massive user behavior data is collected from multiple social media platforms and online interactive systems as input data sets.

[0099] The input data set is cleaned and standardized, which includes the following sub-steps:

[0100] S11. Automated outlier detection and processing: Introduce machine learning algorithms for automatic outlier detection; implement dynamic threshold determination based on statistical methods, such as using standard deviation or interquartile range (IQR) to identify and process outliers;

[0101] S12. Advanced missing value processing strategies: Use a variety of missing value interpolation techniques, such as K-nearest neighbor interpolation (KNN), multiple interpolation (MICE), and interpolation based on prediction models (such as decision trees and regression models); use different processing strategies for different types of data (numerical, categorical);

[0102] S13. Data normalization and standardization: Implement adaptive normalization methods, such as minimum-maximum normalization and Z-score standardization, and automatically select the most appropriate method based on the distribution of the data; for data with obvious skewed distribution, use logarithmic transformation, Box-Cox transformation and other methods to transform the data distribution to make it closer to normal;

[0103] S14. Introduction of data quality evaluation indicators: Data quality scores are given based on the completeness, consistency, accuracy and latestness of the data, and the data quality scores are used to automatically decide whether to enter the subsequent processing flow or re-clean the data;

[0104] S15. Automated data exploration: For pre-processed data, use data visualization tools to automatically generate data distribution graphics to help analysts quickly identify data features and potential problems; use exploratory data analysis (EDA) before clustering analysis, such as calculating the correlation and distribution characteristics of each feature, to guide the correct application of clustering algorithms.

[0105] Continue to refer Figure 1 The data identification method based on K-means clustering provided in this application also includes the following steps:

[0106] S2. Determine the initial center of K-means clustering by genetic algorithm, which specifically includes the following sub-steps:

[0107] S21. Initialize the population: First, define how to encode cluster centers into chromosomes of the genetic algorithm. Each chromosome represents a set of possible cluster centers, where each center is represented by a data point in the preprocessed data; randomly select data points as initial cluster centers to generate multiple such chromosomes. The size of the population is set by the algorithm parameters.

[0108] S22. Evaluate fitness: Use the clustering fitness function to evaluate the fitness of each chromosome (i.e., cluster center combination). The fitness function is an indicator based on clustering quality.

[0109] S23, selection operation: select chromosomes to enter the next generation according to fitness. Specifically, methods such as roulette selection and tournament selection can be used to ensure that chromosomes with high fitness have a higher probability of being selected.

[0110] S24, crossover operation: randomly pair the chromosomes that enter the next generation in step S23, and perform a crossover operation with a first preset probability to generate a new chromosome. The first preset probability is generally set at 60% to 95%. When the solution space of the problem is large or the problem is complex, a larger value can be preset. The crossover can be a single-point crossover, a multi-point crossover, or a uniform crossover.

[0111] S25, mutation operation: mutate the new chromosome generated in step S24, and randomly modify one or some genes (i.e., cluster centers) in the chromosome with a second preset probability. The second preset probability is usually set at 0.5% to 10%. When the population diversity is low or the algorithm falls into a local optimum, a larger value can be preset. This is achieved by randomly selecting a new cluster center.

[0112] S26, iterative operation: repeat steps S22 to S25 until a stop condition is met (such as reaching a maximum number of iterations or the fitness is no longer significantly improved);

[0113] S27, select the final cluster center: After the iteration, select the chromosome with the highest fitness from the final population as the initial center of the final K-means clustering.

[0114] In step S22, optional fitness functions include but are not limited to:

[0115] (1) The common total internal square (WCSS) measures the sum of the squares of the distances from each data point to its cluster center. It is applicable when the data points within the cluster are closely clustered together. The calculation formula is:

[0116]

[0117] in, is the number of clusters; It is clusters; yes data points; yes The cluster center of .

[0118] (2) Fitness function that combines distance and density. This function combines the compactness within the cluster, the separation between clusters, and the local density of data points. It is suitable for situations where the size, shape, and density of clusters in the data set are uneven. Its calculation formula is:

[0119]

[0120] is the adjustment parameter; It is clusters, yes data points, It is The center of the cluster, It is The center of the cluster, It is a data point The local density of

[0121] (3) Consider the fitness function of outliers, which introduces a weight for each data point , which reduces the impact of outliers. It is suitable for when there are outliers in the data set and the impact of outliers needs to be explicitly handled. It should be understood that outliers, also known as escape values, refer to one or more values ​​in the data that are significantly different from other values. The situations that need to be explicitly handled include: business or research purposes require special attention to outliers, such as anomaly detection and other fields; there is a significant impact on clustering results, outliers will cause the cluster center to shift, affecting the quality and interpretation of clustering.

[0122] The fitness function is calculated as follows:

[0123]

[0124] in, It is clusters, yes The cluster center of It is a data point The outlier weight is determined by the data point It is determined by the distance to the nearest cluster center. The farther away, the smaller the weight.

[0125] (4) Fuzzy clustering fitness function: This function is applicable to the case where the clustering affiliation of a data point is not absolute, that is, a data point can belong to multiple clusters, where each data point has a certain degree of belonging to each cluster. Its calculation formula is:

[0126]

[0127] in, is the dataset containing all data points, It is a data point Belongs to cluster The membership degree of is the fuzziness coefficient (usually greater than 1), which is used to control the fuzziness of clustering. It is clustering The center point of .

[0128] (5) Energy function fitness. This fitness function introduces the concept of energy in physics and defines a composite index that includes attraction and repulsion. It is suitable for situations where one hopes to discover data structures by simulating physical processes (such as interparticle forces). Its calculation formula is:

[0129]

[0130] in, It is The center of the cluster, It is The center of the cluster, It is clusters, yes data points.

[0131] The above fitness function can more comprehensively evaluate the effect of clustering, adapt to different data characteristics and distributions, and make the genetic algorithm more efficient and accurate in selecting cluster centers.

[0132] Continue to refer Figure 1 The data identification method based on K-means clustering provided in this application also includes the following steps:

[0133] S3. Perform K-means clustering on the preprocessed data, and dynamically adjust the number of clusters K during clustering to obtain clustering results.

[0134] In this embodiment, the dynamic adjustment of the number of clusters K is achieved by the following method:

[0135] Set the possible values ​​of K, calculate the clustering index for each possible K, and then select the K with the best index as the final number of clusters.

[0136] S4. Use internal evaluation indicators, external evaluation indicators and relative evaluation indicators (such as silhouette coefficient and adjusted Rand index) to evaluate the clustering results, remove the clustering results that do not meet the standards, and obtain the clustering results to be processed.

[0137] In this embodiment, the internal evaluation index adjusts the silhouette coefficient in combination with the characteristics of time series data so that it can handle dynamically changing data. Its expression is:

[0138]

[0139] In the formula is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster;

[0140] External evaluation indicators are evaluated by comparing the clustering results with pre-known labels. The normalized mutual information indicator is used to adjust the mutual information calculation and consider the nonlinear relationship and structural characteristics of the data to obtain deeper insights. Its expression is:

[0141]

[0142] In the formula is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively;

[0143] Relative evaluation index, using different clustering configurations to evaluate the stability and effect of the clustering algorithm, using the Davies-Bouldin Index for evaluation, its expression is:

[0144]

[0145] In the formula, is the number of clusters, It is The average distance within a cluster, It is The average distance within a cluster; It is The centers of the clusters, It is The centers of the clusters, is the distance between different cluster centers.

[0146] For specific data characteristics, such as high-dimensional data or unevenly distributed data, traditional clustering evaluation indicators may not be sufficient to provide effective performance feedback. Dynamic adjustment strategies can select the most suitable evaluation criteria based on the specific characteristics of the data, such as using density-based evaluation methods for high-dimensional data, or adjusting the calculation method of the silhouette coefficient to solve the problem of uneven data distribution, making it more sensitive to subtle changes in the data set. The method of dynamically adjusting the evaluation criteria increases the adaptability of clustering analysis. The most appropriate evaluation criteria are automatically selected for different data characteristics (such as dimension, distribution, density) and clustering results (such as cluster size, shape, and degree of dispersion). This allows the evaluation process to more accurately reflect the actual effect of clustering, thereby better guiding the adjustment and optimization of clustering algorithms.

[0147] In a specific embodiment, evaluating the clustering result in step S4 includes the following sub-steps:

[0148] S41. Data characteristic analysis: Automatically analyze the key characteristics of preprocessed data, including dimension size, sample size, correlation between features, and distribution uniformity. Use statistical test methods such as KS test and chi-square test to evaluate the distribution characteristics of preprocessed data.

[0149] S42. Design a dynamic selection mechanism: Use a decision tree or rule engine to automatically select appropriate clustering evaluation criteria based on key features. For situations that require more complex decisions, train a machine learning model to predict the most appropriate evaluation criteria.

[0150] The characteristics that affect the selection of clustering evaluation indicators include data type, clustering target, data distribution characteristics, type of clustering method and processing speed. The decision tree is constructed based on key characteristics. The steps are as follows:

[0151] 1. Define key characteristics, such as data dimension (low dimensionality / high dimensionality), data distribution (uniform, uneven, presence of outliers), clustering objectives (finding tightly clustered groups, identifying outliers), etc.

[0152] Rule construction: if the data is high-dimensional, use evaluation indicators suitable for high-dimensional data, such as silhouette coefficient; if the data distribution is uneven or there are outliers, use the fitness function that takes outliers into consideration or the fuzzy clustering fitness function; implement a decision tree or rule engine, where each node of the decision tree represents a decision point, such as an attribute of the data, and each branch represents a response to the decision; 4. Verification and adjustment: make necessary adjustments based on feedback.

[0153] S43, real-time adjustment of evaluation indicators: In the clustering process, clustering performance feedback is collected in real time, and the evaluation indicators are adjusted according to the feedback. If the preliminary evaluation results show that the current evaluation indicator is not suitable, the system will automatically switch to another evaluation indicator.

[0154] S44. Integration and implementation: Integrate the adaptive evaluation index selection system into the existing clustering analysis software or platform, and develop corresponding APIs to realize the interaction between evaluation index selection and clustering algorithm.

[0155] S45. Testing and optimization: Test the effectiveness of the adaptive evaluation index selection system on a variety of simulated data sets, and adjust the parameters of the decision support system or machine learning model based on the test results to optimize the overall performance and response speed of the system.

[0156] Continue to refer Figure 1 The data identification method based on K-means clustering provided in this application also includes the following steps:

[0157] S5. Based on the clustering results to be processed, identify and mark the data points that have the greatest impact on the training quality of the deep learning model, and the data points are located in the cluster center or dense area. In a specific embodiment, based on the clustering results, user behavior can be segmented, including specific topics and content preferences, active time periods, etc., so as to optimize target advertising and content recommendations.

[0158] S6. The identified data points are combined into a training data set for use in deep learning model training.

[0159] Further references Figure 2 As an implementation of the above method, in the second aspect, the present application provides an embodiment of a structure diagram of a data recognition system 200 based on K-means clustering, which can be specifically applied to various electronic devices. The system 200 includes the following modules:

[0160] The preprocessing module 210 is configured to clean and standardize the input data set to obtain preprocessed data;

[0161] An initial center determination module 220, configured to determine the initial center of K-means clustering by genetic algorithm;

[0162] The clustering result acquisition module 230 is configured to perform K-means clustering on the preprocessed data and dynamically adjust the number of clusters K during clustering to obtain clustering results;

[0163] The evaluation module 240 is configured to evaluate the clustering results using internal evaluation indicators, external evaluation indicators and relative evaluation indicators, and obtain the clustering results to be processed after removing the clustering results that do not meet the standards;

[0164] The expression of the internal evaluation index is:

[0165]

[0166] In the formula represents the internal evaluation index, is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster;

[0167] The expression of external evaluation index is:

[0168]

[0169] In the formula represents external evaluation indicators, is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively;

[0170] The expression of relative evaluation index is:

[0171]

[0172] In the formula Represents a relative evaluation index, is the number of clusters, It is The average distance within a cluster, It is The average distance within a cluster, It is The center of the cluster, It is The center of the cluster, is the distance between different cluster centers;

[0173] Identification module 250, based on the clustering results to be processed, identifies and marks the data points that have the greatest impact on the training quality of the deep learning model, and the data points are located in the cluster center or dense area;

[0174] The data set integration module 260 is configured to combine the identified data points into a training data set for use in deep learning model training.

[0175] In a third aspect, the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement a data identification method based on K-means clustering as described in any one of the above.

[0176] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor, the data identification method based on K-means clustering as described in any one of the above is implemented.

[0177] Reference below Figure 3 , which shows a schematic diagram of the structure of a computer system 300 of a terminal device or server suitable for implementing an embodiment of the present application. Figure 3 The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0178] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the computer system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0179] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0180] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0181] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data identification method based on K-means clustering, characterized in that: The method comprises the following steps: S1. Clean and standardize the input data set to obtain preprocessed data; S2, determining the initial center of K-means clustering by genetic algorithm; including the following sub-steps: S21, encoding the cluster centers as chromosomes of a genetic algorithm, each of the chromosomes representing a group of cluster centers, wherein the cluster centers are represented by a data point in the preprocessed data, randomly selecting a data point as an initial cluster center, and generating a plurality of chromosomes, wherein the size of the population is set by a parameter of the genetic algorithm; S22, using a clustering fitness function to evaluate the fitness of each of the chromosomes, and selecting a chromosome to enter the next generation according to the fitness; S23, randomly pairing the chromosomes entering the next generation in step S22, and performing a crossover operation with a first preset probability to generate a new chromosome; S24, randomly modifying one or more genes in the new chromosome in step S23 with a second preset probability to obtain a mutant chromosome; S25, repeating steps S22 to S24 until the maximum number of iterations is reached; S26, after the iteration is completed, the chromosome with the highest fitness is selected from the final population as the initial center of the final K-means clustering; In step S22, the clustering fitness function includes: a total internal square function, a fitness function of comprehensive distance and density, a fitness function considering outliers, a fuzzy clustering fitness function, and an energy function fitness function; The total internal square function measures the sum of the squares of the distances from each data point to its cluster center, and is applicable to the case where the data points within the cluster are closely clustered together. Its expression is: In the formula represents the total internal square function, is the number of clusters, It is clusters, yes data points, yes The cluster center of The fitness function of the comprehensive distance and density is applicable to the case where the size, shape and density of clusters in the data set are uneven, and its expression is: In the formula Represents the fitness function of comprehensive distance and density; is the adjustment parameter; yes The cluster center of yes The cluster center of It is a data point The local density of The fitness function considering outliers is applicable to the situation where there are outliers in the data set and the influence of the outliers needs to be processed, and its expression is: In the formula represents the fitness function considering outliers, yes The cluster center of It is a data point The outlier weight is determined by the data point Determined by the distance to the nearest cluster center; The fuzzy clustering fitness function is applicable to the case where a data point belongs to multiple clusters, and its expression is: In the formula Represents the fuzzy clustering fitness function It is a data point Belongs to cluster The membership degree of is the fuzziness coefficient, yes The cluster center of The energy function fitness function is suitable for situations where you want to discover data structures by simulating physical processes, and its expression is: In the formula represents the energy function fitness function, yes The cluster center of yes The cluster center of S3, performing K-means clustering on the preprocessed data, and dynamically adjusting the number of clusters K during clustering to obtain a clustering result; S4, evaluating the clustering results using internal evaluation indicators, external evaluation indicators, and relative evaluation indicators, removing clustering results that do not meet the standards, and obtaining clustering results to be processed; The expression of the internal evaluation index is: In the formula represents the internal evaluation index, is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster; The expression of the external evaluation index is: In the formula represents external evaluation indicators, is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively; The expression of the relative evaluation index is: In the formula Represents a relative evaluation index, It is The average distance within a cluster, It is The average distance within a cluster, It is The center of the cluster, is the distance between different cluster centers; Evaluating the clustering results includes the following sub-steps: S41, analyzing the key characteristics of the preprocessed data, the key characteristics including: dimension size, sample size, correlation between features and distribution uniformity, and using statistical testing methods to evaluate the distribution characteristics of the preprocessed data; S42, selecting appropriate clustering evaluation indicators based on the key characteristics using a decision tree or rule engine; S43, collecting clustering performance feedback in the clustering process in real time, and adjusting the clustering evaluation index according to the clustering performance feedback; S5. Based on the clustering results to be processed, identify and mark the data points that have the greatest impact on the training quality of the deep learning model, where the data points are located in the cluster center or dense area; S6. The identified data points are combined into a training data set for use in deep learning model training.

2. The data identification method based on K-means clustering according to claim 1, characterized in that: In step S1, the input data set is cleaned and standardized, including the following: Automatically detect outliers on input data sets through machine learning algorithms, obtain abnormal data, and identify and process the abnormal data based on dynamic thresholds of statistical methods; Use K-nearest neighbor interpolation, multiple interpolation or prediction model-based interpolation to handle missing values ​​for different types of input data sets; Normalize the input data set by min-max normalization or Z-score normalization; For data with skewed distribution, logarithmic transformation or Box-Cox transformation method was used for transformation; Perform data quality scoring, and determine whether to perform data cleaning again based on the data quality scoring.

3. The data identification method based on K-means clustering according to claim 1, characterized in that: In step S42, a decision tree is constructed based on the key characteristics, which specifically includes the following sub-steps: S421. Define key characteristics; S422, constructing rules: if it is determined that the key characteristic is high-dimensional data, then using the silhouette coefficient evaluation index; if the key characteristic is uneven data distribution or the presence of outliers, then using a fitness function that considers outliers or a fuzzy clustering fitness function; S423, implementing a decision tree or rule engine; S424: Verify and adjust, and adjust the rules based on the feedback.

4. The data identification method based on K-means clustering according to claim 1, characterized in that: Also includes: Automatically generate a data distribution graph from the preprocessed data using a data visualization tool; Integrate the adaptive clustering evaluation index selection system into the clustering analysis software or platform, and configure the corresponding API to realize the interaction between clustering evaluation index selection and clustering algorithm.

5. A data recognition system based on K-means clustering, applied to the data recognition method according to any one of claims 1 to 4, characterized in that: The system comprises: A preprocessing module is configured to clean and standardize the input data set to obtain preprocessed data; An initial center determination module, configured to determine the initial center of K-means clustering through a genetic algorithm; A clustering result acquisition module is configured to perform K-means clustering on the preprocessed data and dynamically adjust the number of clusters K during clustering to obtain a clustering result; An evaluation module is configured to evaluate the clustering results using internal evaluation indicators, external evaluation indicators, and relative evaluation indicators, and obtain the clustering results to be processed after removing the clustering results that do not meet the standards; The expression of the internal evaluation index is: In the formula represents the internal evaluation index, is the average distance between all data points in the same cluster, is the average distance of all data points to the nearest cluster; The expression of the external evaluation index is: In the formula represents external evaluation indicators, is the mutual information between the clustering result and the true class label, and are the entropy of the true class label and the clustering result respectively; The expression of the relative evaluation index is: In the formula Represents a relative evaluation index, It is The average distance within a cluster, It is The average distance within a cluster, It is The center of the cluster, is the distance between different cluster centers; An identification module, based on the to-be-processed clustering results, identifies and marks the data points that have the greatest impact on the training quality of the deep learning model, wherein the data points are located in the cluster center or dense area; The data set integration module is configured to form a training data set by combining the identified data points for use in deep learning model training.

6. A terminal device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the data identification method based on K-means clustering as claimed in any one of claims 1 to 4.

7. A computer-readable storage medium, wherein a computer program is stored in the medium, and when the computer program is executed by a processor, the data identification method based on K-means clustering as described in any one of claims 1 to 4 is implemented.