Method for cleaning and removing abnormal points of industrial key point data and related device

Abstract

The application discloses a kind of industrial key point data cleaning and abnormal point elimination method and related device, wherein, the method includes: obtaining the industrial key point coordinate data to be processed, and the coordinate data is preprocessed;The optimal cluster number of the coordinate data after preprocessing is dynamically determined;Based on the optimal cluster number, the first clustering algorithm is used to cluster the coordinate data, and according to the distance distribution statistical characteristics in each cluster, the first type of abnormal point is identified;After the data of the first type of abnormal point is removed, the second clustering algorithm based on density is used for secondary cleaning to identify the second type of abnormal point judged as noise point;Integrate the first type of abnormal point and the second type of abnormal point, generate cleaned data set.The application combines dynamic clustering, cluster-based statistics-based anomaly detection and density secondary cleaning, enhances the adaptive ability, accuracy and robustness of data cleaning process, and can output more pure and reliable data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial data processing, and in particular to a method for cleaning and removing abnormal points of industrial key point data and a related device. BACKGROUND

[0002] In the field of data-driven artificial intelligence and big data analysis, data quality directly affects the performance and reasoning accuracy of the model. However, in the actual collection and annotation process, the data often has a certain degree of abnormality, noise or error annotation phenomenon. For example, in image key point annotation, sensor sampling, user behavior log and other applications, there may be problems such as offset points, repeated points, systematic errors or human annotation errors. Such "dirty data" not only affects the model training process, but also causes the model to make wrong judgments in the deployment stage.

[0003] In order to solve the above problems, some data cleaning techniques have been proposed in related technologies. Common methods include rule-based anomaly detection (such as setting range thresholds), statistical analysis (such as mean / variance outlier detection) or machine learning methods (such as isolation forest, principal component analysis, etc.). However, these methods usually rely on global statistical distribution or assume that the data follows a certain distribution, and the processing effect on complex data structures (especially local cluster structures) is limited, and misjudgment or omission may occur. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the purpose of the present application is to propose a method for cleaning and removing abnormal points of industrial key point data and a related device.

[0005] To achieve the above-mentioned purpose, in a first aspect, a method for cleaning and removing abnormal points of industrial key point data according to an embodiment of the present application comprises:

[0006] Obtaining industrial key point coordinate data to be processed, and preprocessing the coordinate data;

[0007] Dynamically determining the optimal cluster number of the preprocessed coordinate data;

[0008] Based on the optimal cluster number, a first clustering algorithm is used to cluster the coordinate data, and based on the distance distribution statistical characteristics within each cluster, a first type of abnormal point whose distance from the cluster center of the cluster exceeds a first dynamic threshold is identified;

[0009] After removing the first type of abnormal point, a second clustering algorithm based on density is used for secondary cleaning to identify a second type of abnormal point that is judged to be a noise point based on the local density of the data point;

[0010] Integrate the first type of abnormal points and the second type of abnormal points to generate a cleaned data set.

[0011] According to one embodiment of the present application, the dynamic determination of the optimal cluster number of the preprocessed coordinate data comprises:

[0012] Within a preset initial cluster number interval, for each cluster number value, the sum of squared Euclidean distances of all points in the corresponding cluster to the cluster center is calculated;

[0013] Based on the rate of change of the sum of squared Euclidean distances, an elbow point is determined using the elbow rule, and a candidate cluster number range is determined with the elbow point as the center;

[0014] Within the candidate cluster number range, for each cluster number value, the corresponding average silhouette coefficient is calculated;

[0015] The cluster number value that maximizes the average silhouette coefficient is selected as the optimal clustering cluster number.

[0016] According to one embodiment of the present application, the identification of the first type of abnormal points according to the distance distribution statistical characteristics in each cluster comprises:

[0017] After the first clustering algorithm is completed, the centroid of each cluster is calculated;

[0018] The distance of each data point in the cluster to the corresponding centroid is calculated to form an intra-cluster distance set;

[0019] Based on the mean and standard deviation of the intra-cluster distance set, an anomaly detection threshold is set;

[0020] Data points with a distance to the corresponding centroid exceeding the anomaly detection threshold are identified as the first type of abnormal points.

[0021] According to one embodiment of the present application, the setting of the anomaly detection threshold further comprises:

[0022] The skewness of the intra-cluster distance set is calculated, and the weight factor used to calculate the anomaly detection threshold is adjusted according to the absolute value of the skewness, thereby dynamically adjusting the anomaly detection threshold.

[0023] According to one embodiment of the present application, the identification of the first type of abnormal points further comprises:

[0024] The local outlier factor (LOF) value of each data point is calculated;

[0025] Data points with a distance to the corresponding centroid exceeding the anomaly detection threshold and an LOF value exceeding a preset LOF threshold are finally determined as the first type of abnormal points.

[0026] According to one embodiment of the present application, the first clustering algorithm is a K-Means algorithm, specifically comprising:

[0027] Based on the optimal cluster number, a plurality of cluster centers are initialized;

[0028] Each data point is assigned to a cluster corresponding to the cluster center closest to it in Euclidean distance;

[0029] The center of each cluster is recalculated and updated as a new cluster center;

[0030] The above steps are repeated until the cluster assignment no longer changes or the maximum number of iterations is reached.

[0031] According to one embodiment of the present application, the second clustering algorithm is a DBSCAN algorithm, the parameters of which are dynamically determined, specifically comprising:

[0032] Based on the data dimension, a minimum point number parameter is set;

[0033] By calculating the k-distance ordering curve and identifying the inflection point of the curve, the neighborhood radius parameter is adaptively determined;

[0034] According to the dynamically determined neighborhood radius and minimum point number parameters, all data points after the first type of outlier points are removed are traversed;

[0035] Data points containing a number of data points in the neighborhood that is not less than the minimum point number are identified as core points;

[0036] Starting from any core point, a cluster is formed through density reachable relationship;

[0037] Data points that do not belong to any cluster, neither core points nor boundary points, are identified as noise points as the second type of outlier points.

[0038] According to one embodiment of the present application, the preprocessing step comprises:

[0039] Extreme outliers are preliminarily screened out using an IQR-based method;

[0040] The coordinate data is normalized using a z-score standardization method;

[0041] The data is format-verified and converted to generate a structured numerical array;

[0042] In addition, the data is de-duplicated to remove duplicate coordinate points.

[0043] According to one embodiment of the present application, the generation of the cleaned data set comprises:

[0044] According to the preset mode, all the identified abnormal points are removed from the original data or are marked as abnormal on the basis of the original data.

[0045] In a second aspect, the industrial key point data cleaning and abnormal point removing device according to the embodiments of the present application comprises:

[0046] The data processing module is configured to acquire the industrial key point coordinate data to be processed and pre-process the coordinate data.

[0047] The cluster number determination module is configured to dynamically determine the optimal clustering cluster number of the pre-processed coordinate data.

[0048] The first cleaning module is configured to cluster the coordinate data by using a first clustering algorithm based on the optimal clustering cluster number, and identify a first type of abnormal point whose distance to the cluster center of the cluster is beyond a first dynamic threshold according to the distance distribution statistical characteristics in each cluster.

[0049] The second cleaning module is configured to clean the data after removing the first type of abnormal point by using a second clustering algorithm based on density for secondary cleaning, so as to identify a second type of abnormal point which is determined to be a noise point based on the local density of the data point.

[0050] The result generation module is configured to integrate the first type of abnormal point and the second type of abnormal point, and generate a cleaned data set.

[0051] In a third aspect, the computer device according to the embodiments of the present application comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the industrial key point data cleaning and abnormal point removing method as described above when executing the computer program.

[0052] In a fourth aspect, the computer storage medium according to the embodiments of the present application has a computer program stored thereon, and the program is executed by the processor to implement the industrial key point data cleaning and abnormal point removing method as described above.

[0053] According to the method and related device for industrial key point data cleaning and outlier elimination provided by the embodiment of the present application, firstly, the optimal cluster number is dynamically determined, so that the defect of poor adaptability to different data distribution caused by the dependence on fixed parameters is overcome, and it is ensured that the subsequent cluster analysis can more accurately reflect the internal structure of the data. After the preliminary clustering based on the optimal cluster number, the threshold is dynamically set according to the distance distribution statistical characteristics of each cluster to identify the first type of outliers, so that the fine and localized judgment of the outliers is realized, and the misjudgment and omission caused by the global unified threshold when processing the uneven data are avoided. Finally, the second clustering algorithm based on density is introduced for secondary cleaning, so that the second type of outliers in the form of noise that may be missed in the previous stage can be effectively identified and eliminated, and the thoroughness of the cleaning is further improved. Therefore, by combining the dynamic clustering, the outlier detection based on the cluster internal statistics and the density secondary cleaning, the adaptability, accuracy and robustness of the data cleaning process are significantly enhanced, so that a more pure and reliable high-quality data set can be output.

[0054] Additional aspects and advantages of the present application will be made apparent by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained according to the structures shown in the drawings without any creative labor.

[0056] Figure 1 is a flow chart of an embodiment of the method for industrial key point data cleaning and outlier elimination of the present application;

[0057] Figure 2 is a flow chart of step S102 in the method for industrial key point data cleaning and outlier elimination of the present application;

[0058] Figure 3 is a flow chart of the first clustering algorithm in the method for industrial key point data cleaning and outlier elimination of the present application;

[0059] Figure 4 is a flow chart of identifying the first type of outliers in the method for industrial key point data cleaning and outlier elimination of the present application;

[0060] Figure 5 is a flow chart of the second clustering algorithm in the method for industrial key point data cleaning and outlier elimination of the present application;

[0061] Figure 6 is a flow chart of step S101 in the method for cleaning and removing outliers of industrial key point data of the present application;

[0062] Figure 7 is a workpiece picture marked with data points obtained in step S101 in the method for cleaning and removing outliers of industrial key point data of the present application;

[0063] Figure 8 is a WCSS curve chart of the sum of squared Euclidean distances within clusters in step S201 in the method for cleaning and removing outliers of industrial key point data of the present application;

[0064] Figure 9 is a K-distance sorting curve chart in step S502 in the method for cleaning and removing outliers of industrial key point data of the present application;

[0065] Figure 10 is a two-dimensional plane scatter plot of KMeans clustering cleaning result in the method for cleaning and removing outliers of industrial key point data of the present application;

[0066] Figure 11 is a text file chart saved after KMeans clustering cleaning in the method for cleaning and removing outliers of industrial key point data of the present application;

[0067] Figure 12 is a certain workpiece picture marked with outliers corresponding to the KMeans clustering cleaning result in the method for cleaning and removing outliers of industrial key point data of the present application;

[0068] Figure 13 is a two-dimensional plane scatter plot of DBSCAN clustering cleaning result in the method for cleaning and removing outliers of industrial key point data of the present application;

[0069] Figure 14 is a text file chart saved after DBSCAN clustering cleaning in the method for cleaning and removing outliers of industrial key point data of the present application;

[0070] Figure 15 is a certain workpiece picture marked with outliers corresponding to the DBSCAN clustering cleaning result in the method for cleaning and removing outliers of industrial key point data of the present application;

[0071] Figure 16 is a structural schematic diagram of an embodiment of the device for cleaning and removing outliers of industrial key point data of the present application;

[0072] Figure 17 is a structural schematic diagram of an embodiment of the computer device of the present application.

[0073] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0074] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application.

[0075] Reference Figure 1 As shown, Figure 1 A flowchart of one embodiment of the method for industrial key point data cleaning and outlier elimination provided by the embodiments of the present application is shown. For ease of description, only the part related to the embodiments of the present application is shown. Specifically, the method for industrial key point data cleaning and outlier elimination can be executed by a computer, and specifically includes:

[0076] S101, obtaining industrial key point coordinate data to be processed, and preprocessing the coordinate data. This step aims to provide a reliable data basis for subsequent accurate analysis.

[0077] In an industrial automation production scene, for example, on a visual quality inspection line of parts, multiple industrial cameras deployed will continuously shoot part images, and automatically label the coordinates of several key points such as screw hole center, edge corner, and welding point position through a preliminary image recognition model. These original coordinate data are used as the data set to be processed. Figure 7 An example is a workpiece picture marked with data points, and there are a total of 16 points in the image. The four points from 0 to 3 are the four vertices of a rectangular frame, which identifies the area range of the workpiece. The points from 4 to 15 are key points inside the workpiece, such as marked position points used in the automated dispensing process. During data acquisition, all key point coordinate information including detection area coordinates can be read from a pre-prepared CSV file to ensure data integrity and correctness.

[0078] However, due to the complex and changeable production environment, such as sudden changes in light, workpiece surface reflection, conveyor belt vibration, or camera lens contamination, the collected coordinate data will inevitably be mixed with noise and outliers. If not handled, these "dirty data" will seriously distort the true distribution of the data, directly affecting the accuracy of subsequent analysis. Therefore, the preprocessing step is performed to eliminate these original noise and structural bias, ensuring that the data provided to subsequent analysis is preliminarily purified and standardized. The preprocessing process can eliminate extreme outliers that deviate too much from the normal range, such as invalid data whose coordinate values are far beyond the physical boundary of the workpiece due to program errors or hardware failures. In addition, the coordinate data is standardized, data format is checked and de-duplicated, etc., to ensure the accuracy and robustness of subsequent analysis.

[0079] S102, dynamically determine the optimal cluster number of the preprocessed coordinate data.

[0080] In industrial applications, the sources of data have high heterogeneity. Due to the influence of various uncontrollable factors such as lighting conditions, shooting angles, process fluctuations, equipment parameters, and operation methods, there are significant differences in the distribution of data collected at different time periods and different batches. This uncertainty makes it difficult to maintain stability and effectiveness in all scenarios by pre-setting a fixed cluster number. For example, a batch of data may naturally cluster into 3 categories, while another batch of data is more suitable for being divided into 5 categories or even more, which leads to a sharp decline in algorithm performance. To solve this problem, this step uses a dynamic and adaptive cluster number selection mechanism to consider the clustering compactness and separation degree to determine the optimal cluster number more stably, greatly improving the adaptive ability and robustness in variable industrial scenarios.

[0081] S103, based on the optimal cluster number, using a first clustering algorithm to cluster the coordinate data, and according to the distance distribution statistical characteristics of each cluster, identifying a first type of abnormal points whose distance to the cluster center of the cluster exceeds a first dynamic threshold.

[0082] This step is used to perform the first cleaning. After clustering is completed, each cluster can be understood as an aggregation area of a certain type of normal data points, defining the "normal" range and distribution characteristics of this type of data. Traditional global threshold method cannot handle unevenly distributed data. A point that seems normal in a sparse cluster may be a significant anomaly if placed in a tight cluster. This step gives each cluster an abnormality judgment boundary. By analyzing the dispersion degree of each cluster, a threshold value that matches its own statistical characteristics can be dynamically generated. In this way, whether a point is abnormal is no longer determined by its absolute coordinates, but by the degree of deviation from its own group behavior. This discrimination method can accurately identify those abnormal points that are mixed in the normal group but have slight deviations, greatly improving the sensitivity and accuracy of detection.

[0083] S104, using a second clustering algorithm based on density to perform secondary cleaning on the data after removing the first type of abnormal points, to identify a second type of abnormal points that are judged to be noise points based on the local density of data points.

[0084] The purpose of this step is to make up for the possible blind spots in the first cleaning, further improving the thoroughness of cleaning. The first cleaning is based on the detection method of cluster centroid, which is good at detecting abnormal points deviating from the center of the group, but it may be difficult to effectively identify certain types of abnormalities, such as isolated points that do not belong to any clear cluster and are sparsely distributed. That is, after the first clustering analysis is completed, there may still be a small number of isolated points or boundary abnormal points that have not been accurately identified. This step introduces a density-based clustering analysis by analyzing the "neighborhood" density around each data point. Those points that cannot form a density core regardless of their absolute position are identified as noise, thereby further improving the purity of the final data set.

[0085] S105, integrating the first type of abnormal points and the second type of abnormal points to generate a cleaned data set.

[0086] This step summarizes the abnormal results detected in the previous two cleaning steps and generates a high-quality final data set that can be directly used by downstream tasks. In specific applications, according to application requirements, it can generate a "pure" data that excludes all abnormal points for training more stable and accurate artificial intelligence models; or it can also choose to retain the original data and only mark the abnormal points, and deliver a report with detailed information for saving, in order to trace the root cause of the problem, such as diagnosing which device or process has a systematic deviation, etc. Thus, the method can stably output a highly reliable and pure data set, providing data support for improving the level of industrial intelligence.

[0087] According to the industrial key point data cleaning and abnormal point elimination method provided by the embodiment of the application, first, the optimal clustering cluster number is dynamically determined, which overcomes the defect of poor adaptability to different data distributions caused by relying on fixed parameters, ensuring that the subsequent clustering analysis can more accurately reflect the internal structure of the data. After preliminary clustering based on the optimal cluster number, the threshold is dynamically set according to the distance distribution statistical characteristics of each cluster to identify the first type of abnormal points, realizing the fine and localized judgment of the abnormal points and avoiding the misjudgment and omission that may be caused by the global unified threshold when processing unevenly dense data. Finally, by introducing a second clustering algorithm based on density for secondary cleaning, the second type of abnormal points that may be missed in the previous stage and exist in the form of noise can be effectively identified and eliminated, further improving the thoroughness of cleaning. Thus, the application combines dynamic clustering, abnormal detection based on cluster statistics, and density secondary cleaning, significantly enhancing the adaptability, accuracy, and robustness of the data cleaning process, so as to output a more pure and reliable high-quality data set.

[0088] Reference Figure 2As shown, in one embodiment of the present application, the dynamic determination of the optimal cluster number of the preprocessed coordinate data includes:

[0089] S201, in a preset initial cluster number interval, for each cluster number value, calculate the Euclidean distance square sum (WCSS) of all points in the corresponding cluster to the corresponding cluster center. For example, the preset initial cluster number interval , calculate the WCSS curve under different ; for a given cluster number K, the clustering result is , and the centroid of each cluster is μ k , and the calculation formula of WCSS is:

[0090] .

[0091] Where C k represents the kth cluster, X i is a data point in cluster C k ; μ k is the centroid of the cluster; is the Euclidean distance square of the data point and the centroid. The WCSS index measures the sum of the distance squares of all data points to their cluster centroids, and the smaller the value, the more compact the data points in the cluster, and the higher the aggregation degree. By calculating the WCSS under different K values, a WCSS curve that monotonically decreases with the increase of K value can be obtained (as shown in Figure 8 ).

[0092] S202, based on the change rate of the Euclidean distance square sum in the cluster, the elbow rule is used to determine an elbow point, and a candidate cluster number range is determined based on the elbow point.

[0093] When the K value increases from too small to a suitable value, WCSS will decrease rapidly; but when the K value exceeds the real optimal cluster number, even if the K value is increased, the decline amplitude of WCSS will be sharply slowed down, forming an elbow-like inflection point. In order to accurately locate the "elbow point", the differential slope method is adopted, for example, by calculating the second order derivative approximation value of the WCSS curve, the calculation formula is as follows:

[0094] .

[0095] Select the point with the largest slope change as the elbow preliminary K elbow , and set a small offset δ (for example, for a data set with a key point number between 10 and 20, δ can be taken as 2) based on the K elbow , so as to determine a candidate cluster number range , for example K elbow= 5, then the candidate cluster number range is This way effectively reduces the search space, avoids complex calculations for all possible K values, and improves efficiency.

[0096] S203、In the candidate cluster number range, for each cluster number value, calculate its corresponding average silhouette coefficient.

[0097] To more accurately evaluate the clustering quality, this step introduces the silhouette coefficient for fine selection. Unlike WCSS, which only focuses on the tightness of the cluster, the silhouette coefficient takes into account both the cohesion and separation of the cluster. For any data point i in the data set, first calculate the average distance a(i) from all other data points in its own cluster, i.e. "intra-class distance", which represents cohesion. Then, calculate the average distance b(i) from all data points in the nearest other cluster, i.e. "inter-class distance", which represents separation. Finally, by calculating the average silhouette coefficient S(K) of the entire data set corresponding to each K value in the candidate cluster number range, i.e. the mean of all data point silhouette coefficients s(i).

[0098] The silhouette coefficient calculation formula for a single sample is:

[0099] .

[0100] The silhouette coefficient value range is between [-1, 1], and the closer the value is to 1, the better the clustering effect of the data point.

[0101] The average silhouette coefficient calculation formula for the entire data set is:

[0102] .

[0103] S204、Select the cluster number value that maximizes the average silhouette coefficient as the optimal clustering cluster number K*.

[0104] .

[0105] That is, by comparing the average silhouette coefficients corresponding to all K values in the candidate cluster number range, select the cluster number value that maximizes the coefficient value as the final optimal clustering cluster number K*. This step ensures that the selected cluster number not only makes each cluster as compact as possible, but also makes different clusters as far apart as possible, thereby finding a clustering effect that balances cohesion and separation.

[0106] By the dynamic cluster number determination method combining the elbow rule with the contour coefficient, the adaptive ability of the algorithm is greatly improved, so that it can automatically adapt to industrial data of different sources, different batches and different distribution characteristics without manual intervention and prior knowledge, and the problem of poor generalization ability caused by relying on fixed parameters is solved. In addition, the calculation efficiency and selection accuracy are considered, the high-potential candidate interval is quickly locked by the elbow rule, and then the contour coefficient is used for fine evaluation, so that the accuracy and stability of the optimal cluster number selection are ensured, the clustering result can more truly reflect the internal structure of the data, the subsequent anomaly point detection based on cluster statistics is ensured, and the robustness and accuracy of the entire data cleaning are improved.

[0107] Referring to Figure 3 In one embodiment of the application, the first clustering algorithm is a K-Means algorithm, which specifically includes:

[0108] S301, initializing a plurality of cluster centers based on the optimal cluster number.

[0109] S302, assigning each data point to the cluster corresponding to the cluster center closest to it in Euclidean distance.

[0110] S303, recalculating the center of each cluster and updating it to a new cluster center.

[0111] S304, repeat the above steps until the cluster assignment no longer changes or the maximum number of iterations is reached.

[0112] In this embodiment, based on the dynamically selected optimal cluster number, clustering analysis is performed on the structured point set data, and each data point is classified to its nearest cluster center, thereby achieving automatic grouping. The K-Means clustering algorithm is used, and the goal is to minimize the sum of squared Euclidean distances of all points to their corresponding cluster centers, i.e.

[0113] .

[0114] wherein, is the i th point; C k represents the k th cluster;

[0115] .

[0116] is the center of the K th cluster.

[0117] The whole clustering process is iterated, that is, first, K* cluster centers are initialized; then each data point is assigned to the cluster corresponding to the cluster center with the closest Euclidean distance; then, the center of each cluster is recalculated and updated as the new cluster center; repeat the above steps until the cluster assignment no longer changes or the maximum number of iterations is reached.

[0118] Referring to Figure 4 In one embodiment of the present application, the identification of the first type of abnormal points according to the distance distribution statistical characteristics in each cluster comprises:

[0119] S401, after the first clustering algorithm is completed, the center of each cluster is calculated.

[0120] For the Kth cluster, its center (center point) is defined as the mean vector of all data points in the cluster:

[0121] .

[0122] wherein, denotes the coordinates of the cluster center; is the number of data points in cluster K; X j is the data point in the cluster. This center represents the "center position" of the cluster in geometry, which is the reference point for subsequent distance calculation.

[0123] S402, the distance of each data point in each cluster to its corresponding center is calculated to form a distance set in the cluster.

[0124] The Euclidean distance of each data point in the cluster to the center:

[0125] .

[0126] The distance set in the cluster:

[0127] .

[0128] That is, after the center is calculated, the Euclidean distance of each data point in the cluster to its corresponding center is calculated. By traversing all points in the cluster, a distance set belonging to the cluster is formed, which completely reflects the dispersion degree and distribution form of the data points in the cluster.

[0129] S403, based on the mean and standard deviation of the distance set in the cluster, an anomaly detection threshold is set.

[0130] This step specifically includes: first, the distance set D k The mean and standard deviation are calculated. The setting of the anomaly detection threshold is based on the statistical distribution of the distance set, especially the mean and standard deviation, so the mean and standard deviation of the distance set D kCompute its mean and standard deviation.

[0131] Mean:

[0132] .

[0133] Standard deviation:

[0134] .

[0135] Then, for each cluster C k , compute its set of distances of data points to the centroid:

[0136] .

[0137] Finally, based on the mean μ k and standard deviation σ k of the distances, construct the anomaly detection threshold:

[0138] .

[0139] where parameter α controls the looseness of the threshold, according to experience, when α takes the value of 2, it can achieve good preliminary screening effect for most industrial scenes.

[0140] Since the complexity of the structure inside each cluster and data fluctuation are taken into account, a dynamic confidence interval adjustment strategy can also be introduced to dynamically adjust the anomaly detection threshold, and the dynamic confidence interval adjustment strategy includes:

[0141] Compute the skewness of the set of distances within the cluster, and according to the absolute value of the skewness, adjust the weight factor used to calculate the anomaly detection threshold, so as to dynamically adjust the anomaly detection threshold.

[0142] More specifically, the skewness and kurtosis of the distance D k are calculated to determine the degree of deviation of the distribution, if the distance distribution is approximately normal, the anomaly detection threshold T k remains unchanged; otherwise, the threshold weight β k is adjusted in combination with the skewness, for example:

[0143] .

[0144] where γ is an adjustment coefficient, used to enhance the anomaly discrimination sensitivity of the skew cluster, in practice, a slightly strong γ=1.5 is adopted, and the effect is better. skew(D k ) refers to the skewness of D k , that is, the degree of asymmetry of the data distribution.

[0145] .

[0146] where μ is the mean, σ is the standard deviation, and E denotes the mathematical expectation operator.

[0147] At this time, the anomaly detection threshold is updated as:

[0148] .

[0149] S404, identifying data points with a distance from the corresponding centroid exceeding the anomaly detection threshold as first-class abnormal points.

[0150] To avoid misjudgment caused by the difference in the scale between clusters, in combination with the local outlier factor concept, the identification of the first-class abnormal points also includes:

[0151] Calculate the local outlier factor LOF value of each data point.

[0152] The data points with a distance from the corresponding centroid exceeding the anomaly detection threshold and their LOF values exceeding the preset LOF threshold are finally determined as first-class abnormal points.

[0153] Specifically, first, calculate the relative density anomaly degree of each data point:

[0154] .

[0155] where N k (X i ) is the k-distance neighborhood set of point X i ;

[0156] And reach-dist and lrd are the kth reachable distance and local reachable density respectively, and their mathematical definitions are as follows:

[0157] .

[0158] where d(X i， X j ) is the Euclidean distance between data points X i ， X j ; d k (X i ) is the K-distance of data point X i .

[0159] .

[0160] lrd k (X i ), essentially the reciprocal of the "average reachable distance". If the reachable distance around X i is very small, it means that it is in a high-density area, so lrd k (X i) will be large; if X i is isolated, the average distance to the nearest neighbor is large, so lrd k (X i ) is small.

[0161] For LOF k (X i ), if X i has about the same density as its neighbors, then LOF≈1 (normal point); if X i has much lower density than its neighbors, then LOF≥1 (outlier); if LOF<1, then X i is even denser than its neighbors.

[0162] Then, the threshold is integrated to determine the abnormal point:

[0163] If the distance of the point is , and , then the point X i is determined as an abnormal point.

[0164] Among them, according to experience, the value of τ is 1.5, the detection is more stable, and part of the normal points that may be excluded are protected.

[0165] Through the above method of the embodiment, first, the localization and refinement of the abnormality determination are realized, and by calculating the statistical characteristics and dynamic abnormality detection threshold for each cluster separately, the defect that the global threshold is easy to produce misjudgment and omission when processing uneven data is overcome. Secondly, by introducing skewness to dynamically adjust the threshold, and combining the local outlier factor (LOF) for double verification, the robustness and accuracy of the detection process are greatly enhanced, and the normal data points located at the edge of the sparse cluster can be effectively protected from being wrongly excluded. The comprehensive discrimination strategy based on the distance distribution within the cluster and the local density ensures that the first cleaning can efficiently and reliably identify the abnormal data that deviates from the behavior mode of the group to which it belongs, and lays a foundation for subsequent secondary cleaning and finally generating a high-quality data set.

[0166] Referring to Figure 5 , in an embodiment of the application, the second clustering algorithm is a DBSCAN algorithm, and the parameters thereof are dynamically determined, and specifically include:

[0167] S501, setting a minimum point number parameter based on data dimensions.

[0168] S502, adaptively determining a neighborhood radius parameter by calculating a k-distance ordering curve (as shown in Figure 9 ) and identifying the inflection point of the curve.

[0169] S503, traversing and removing all data points after the first type of abnormal points according to the dynamically determined neighborhood radius and the minimum point number parameter.

[0170] S504, identifying the data points contained in the neighborhood as core points, the number of which is not less than the minimum point number.

[0171] S505, starting from any core point, a cluster is formed through density reachable relationship.

[0172] S506, identifying the data points that do not belong to any cluster, neither core points nor boundary points, as noise points of the second type of abnormal points.

[0173] In this embodiment, after the preliminary clustering analysis is completed, there may still be a small amount of isolated points or boundary abnormal points that are not accurately identified. In order to further improve the data quality, the DBSCAN algorithm based on density clustering method is introduced for secondary cleaning.

[0174] The DBSCAN algorithm does not depend on the number of clusters, but divides the data based on density. When a low-density area is found, noise points can be naturally identified, which is suitable for abnormal cleaning after Kmeans algorithm.

[0175] The core content of the DBSCAN algorithm is as follows:

[0176] 1. Core point: a point X i contains at least minPts points in its ε neighborhood.

[0177] 2. Boundary point: a point in the ε neighborhood of a core point, but the points contained in its own neighborhood are less than minPts.

[0178] 3. Noise point: neither a core point nor a boundary point.

[0179] For minPts, take minPts=sample data dimension*2. In this project, each key point has information of 2 dimensions of x coordinate and y coordinate, so minPts=4.

[0180] For ε, after determining minPts, calculate the distance from each point X i to its minPts-1th nearest neighbor:

[0181] d i =d(X i , X i k ), where k=minPts-1.

[0182] After sorting all d i in ascending order, a vector D is formed:

[0183] .

[0184] Where n is the total number of points in the sample data. Then, the K-distance ranking curve of D is plotted (e.g., ...). Figure 9 As shown in the figure, the X-axis represents the sorted point index, and the Y-axis represents the corresponding d. i .

[0185] like Figure 9 The image shows the K-distance sorting curve for keypoint 6, and the calculation of ε = 9.866. ε is calculated by finding the position with the largest slope change in the graph, i.e., the ordinate d corresponding to the elbow point. i That is, the value of ε.

[0186] The neighborhood is defined as follows:

[0187] .

[0188] Core point determination criteria:

[0189] .

[0190] Density can be expressed as:

[0191] like And X i If it is the core point, then X j The density can reach X i。

[0192] Regarding outlier identification methods:

[0193] After DBSCAN clustering, points that are not assigned to any cluster will be marked as noise points. These points can be considered as secondary outliers after K-means clustering analysis.

[0194] .

[0195] Reference Figure 6 As shown, in one embodiment of the present invention, the preprocessing step includes:

[0196] S601. An interquartile range (IQR) method is used to initially screen out extreme outliers.

[0197] In the initial screening, outliers that significantly deviate from the normal range are removed using simple statistical indicators to reduce noise interference with the model. Specifically, the interquartile range (IQR) method can be used to calculate the first quartile (Q1) and third quartile (Q3) of the feature, and define the interquartile range:

[0198] .

[0199] Wherein, the first quartile Q1 refers to the 25th data assuming 100 data arranged in ascending order, and the third quartile refers to the 75th data.

[0200] At the same time, the upper and lower boundaries are set:

[0201] .

[0202] In one example, the weight coefficient κ can be preset according to experience or data distribution characteristics, for example, κ = 2 can be taken, and if it exceeds this upper and lower limit, it is considered as a very extreme outlier value, which does not need subsequent clustering analysis to be eliminated.

[0203] S602, the coordinate data is normalized by using the z-score standardization method.

[0204] Since the coordinate data can come from different image resolutions, shooting scales or coordinate systems, in order to avoid affecting the learning effect of the model due to the difference in the value range, the original data needs to be standardized or normalized to unify the scale of each dimension and ensure that each feature has equal importance in calculation.

[0205] In clustering, distance measurement or regression modeling, if there is a difference in the order of magnitude of the feature dimension, the model is likely to be overly sensitive to high-amplitude features, thereby masking important information in other dimensions. Therefore, standardization is a key preprocessing step to improve the stability, convergence speed and interpretability of the model.

[0206] Specifically, the z-score standardization method can be used to process all coordinate features. This method assumes that the data approximately follows a Gaussian distribution, and for each coordinate feature x, the standardization calculation is as follows:

[0207] .

[0208] Wherein, x is the original coordinate value; μ is the sample mean of the dimension; σ is the sample standard deviation of the dimension; z is the standardized value. After z-score processing, all coordinate features will have a distribution form with a mean of 0 and a standard deviation of 1, so that they have equivalent weights in Euclidean space, thereby avoiding the dominance of a dimension in distance calculation.

[0209] S603, format checking and conversion are performed on the data to generate a structured numerical array.

[0210] In order to ensure that data can smoothly enter the clustering model and get accurate and effective output, it is necessary to systematically check and convert the structure and content of the original data in the preprocessing stage. The main goal is to unify the data format, correct coding errors, and construct a structured expression form suitable for mathematical modeling.

[0211] 1) Format unification and structured conversion:

[0212] The original data may be stored in text, CSV, JSON or other semi-structured forms, containing image names, coordinate pairs, missing markers, etc. In actual operation, it should be converted into a standardized two-dimensional numerical array (such as a matrix) for subsequent analysis.

[0213] Specifically, suppose a data sample contains n key points, each with two-dimensional coordinates (x i ,y i ), then the sample can be represented as:

[0214] .

[0215] At this point, when processing multiple image samples, a three-dimensional tensor can be constructed:

[0216] .

[0217] Where N represents the number of samples (images), and each page matrix is a set of key points for a sample.

[0218] When further expanded into a "flattened" feature vector for each row of samples, each sample can be represented as:

[0219] .

[0220] 2) Format checking and exception correction:

[0221] Since data may come from multiple channels, consistency checks need to be performed on the original file, including:

[0222] Field number check: Each row of data should meet the preset fixed dimension requirement, for example, if a row of data should contain an image identifier and n two-dimensional coordinate points (2n values in total), the total field number should be 2n+1, any data row that does not meet this structure can be considered as format exception.

[0223] Value type check: Non-numeric fields (such as strings, null values) should be removed or interpolated.

[0224] Illegal character processing: Regular cleaning of non-numeric characters mixed in coordinate fields (such as Chinese commas, spaces, unit symbols, etc.).

[0225] Encoding consistency: Ensure the consistency of character set encoding (such as UTF-8 / GBK) during file reading process, avoid misalignment of fields caused by garbled characters.

[0226] For format abnormalities that cannot be automatically repaired, the line number and error type should be recorded to prompt manual intervention. The verification mechanism can be formally represented as:

[0227] .

[0228] S604, the data is de-duplicated to eliminate duplicate coordinate points.

[0229] Through the above preprocessing operations, the integrity and consistency of the input data can be guaranteed to a greater extent, and the misjudgment and noise caused by data quality problems in the subsequent clustering process can be reduced, thereby providing protection for accurate identification of abnormal points.

[0230] In an embodiment of the present application, the generating the cleaned data set comprises:

[0231] According to the preset mode, all identified abnormal points are removed from the original data, or the original data is marked as abnormal.

[0232] That is, after completing the aforementioned multiple rounds of outlier identification and cleaning operations (including preliminary screening based on statistical rules, distance anomaly detection in Kmeans clustering, and density anomaly identification of DBSCAN), a more representative and higher quality data set can be obtained. Therefore, the cleaning results are integrated to generate the final version of the "data set" for subsequent analysis, model training or engineering application deployment.

[0233] For abnormal point processing, the following two processing methods can be selected according to business requirements and task sensitivity:

[0234] Elimination mode: all samples determined to be abnormal are completely removed, suitable for tasks with extremely high data quality requirements, such as supervised model training.

[0235] Labeling mode: On the basis of preserving the original data, a new field is added to mark whether it is abnormal (set "is_outlier" to 0 or 1), which is convenient for subsequent strategic processing, weighted analysis or visualization.

[0236] Specifically, which mode to provide to the operator ensures universality.

[0237] Referring to Figure 10 to Figure 14 In other embodiments of the present application, the method of industrial key point data cleaning and outlier removal can further include:

[0238] Visualize and save the generated data set.

[0239] That is, after completing the clustering analysis and anomaly detection, the results are visualized and standardized to facilitate subsequent verification, review, and model training.

[0240] 1) Cluster result visualization.

[0241] As shown in Figure 10 , to facilitate understanding of the clustering effect and abnormal distribution, the results are visually displayed in a graphical manner. Figure 10 For two-dimensional scatter plots, different colors are used to indicate different clusters; abnormal points are marked with special shapes (such as "X" or red circles); if the sample has an image background, the point image can be superimposed to enhance readability. Figure 10 The clustering and cleaning results for key point 10 are shown, which are divided into 3 clusters, and the crossed points are the abnormal points found. The output cleaning results can be opened to find a certain picture marked as abnormal, as shown in Figure 11 The KMeans clustering and cleaning processing text file graph is shown, and opening one of the text files, such as opening the 2023-03-15-14-49-35-909.bmp file, can see Figure 12 The workpiece graph, as shown in Figure 12 , the red key point is point 10, which deviates greatly from the normal workpiece position, and the corresponding point of the corresponding picture is found through data cleaning and displayed to the user.

[0242] 2) Abnormal detection result visualization.

[0243] As shown in Figure 13 , the core points, boundary points, and noise points in density detection are displayed through DBSCAN clustering analysis. Specifically, the DBSCAN clustering analysis of point 11 is divided into 7 clusters, and the crossed points are the points that are judged to be abnormal in density.

[0244] As shown in Figure 14 , the DBSCAN clustering and cleaning processing text file graph is shown, and opening one of the text files, such as opening the DR8GTG000Y300006C7+5.bmp file, can see Figure 15 The workpiece graph. As shown in Figure 15 , it is a certain workpiece graph that is filtered as abnormal, and the red point is point 11, which is the abnormal point, and it can be found that it deviates from the normal position.

[0245] In other embodiments of the present application, the industrial key point data cleaning and abnormal point elimination method can further include:

[0246] Output a report according to the data set.

[0247] After completing data clustering and anomaly cleaning, detailed analysis and cleaning reports can be generated, ensuring the transparency and reproducibility of the entire process, and facilitating team collaboration, superior review, or model filing.

[0248] For example, the report structure may include an overview of the project and data, threshold setting logic, outlier detection statistics, and instructions on how to save the results. The report can be in PDF or Word format, including charts and formulas for easy sharing and archiving. The report may include basic metadata such as timestamps, parameter configurations, and processor identifiers; it may also include parameter configuration records in .json format and cluster analysis tables in .csv format.

[0249] Reference Figure 16 As shown, Figure 16 This diagram illustrates a structural schematic of one embodiment of the industrial critical point data cleaning and outlier removal apparatus provided by the present invention. For ease of description, only the parts relevant to the embodiment of the present invention are shown. Specifically, the industrial critical point data cleaning and outlier removal apparatus includes:

[0250] The data processing module 701 is used to acquire the coordinate data of the industrial key points to be processed and to preprocess the coordinate data.

[0251] The cluster number determination module 702 is used to dynamically determine the optimal number of clusters for the preprocessed coordinate data.

[0252] The first cleaning module 703 is used to cluster the coordinate data based on the optimal number of clusters using a first clustering algorithm, and to identify first-type anomalies whose distance from the centroid of their respective cluster exceeds a first dynamic threshold based on the statistical characteristics of the distance distribution within each cluster.

[0253] The second cleaning module 704 is used to perform secondary cleaning on the data after removing the first type of outliers using a density-based second clustering algorithm, so as to identify the second type of outliers that are judged as noise points based on the local density of data points.

[0254] The result generation module 705 is used to integrate the first type of outliers and the second type of outliers to generate a cleaned dataset.

[0255] The apparatus for cleaning and removing outlier data of industrial key points according to embodiments of the present invention first overcomes the shortcomings of relying on fixed parameters and resulting in poor adaptability to different data distributions by dynamically determining the optimal number of clusters, ensuring that subsequent clustering analysis can more accurately reflect the internal structure of the data. After preliminary clustering based on the optimal number of clusters, a threshold is dynamically set according to the distance distribution statistical characteristics of each cluster to identify the first type of outlier, realizing refined and localized judgment of anomalies and avoiding misjudgment and omission that may occur when processing data with uneven density using a globally uniform threshold. Finally, by introducing a density-based second clustering algorithm for secondary cleaning, the second type of outlier that may have been missed in the previous stage and exists in the form of noise can be effectively identified and removed, further improving the thoroughness of cleaning. Thus, the present invention, by combining dynamic clustering, anomaly detection based on intra-cluster statistics, and density-based secondary cleaning, significantly enhances the adaptability, accuracy, and robustness of the data cleaning process, thereby producing a cleaner, more reliable, and high-quality dataset.

[0256] In one embodiment of the present invention, the cluster number determination module includes:

[0257] The sum of squares calculation module is used to calculate the sum of squares of the Euclidean distances from all points within a given initial cluster number range to the center of the corresponding cluster for each cluster value.

[0258] The candidate range determination module is used to determine an elbow point based on the rate of change of the sum of squared Euclidean distances within the cluster, using the elbow rule, and to determine a range of candidate cluster numbers centered on the elbow point.

[0259] The contour coefficient calculation module is used to calculate the average contour coefficient for each cluster value within the range of the number of candidate clusters.

[0260] The optimal cluster number selection module is used to select the cluster number that maximizes the average profile coefficient as the optimal cluster number.

[0261] In one embodiment of the present invention, the first cleaning module includes a KMeans clustering execution module and a first type of outlier identification module, wherein the first outlier identification module includes:

[0262] The centroid calculation module is used to calculate the centroid of each cluster after the first clustering algorithm is completed;

[0263] The intra-cluster distance calculation module is used to calculate the distance from each data point within each cluster to its corresponding centroid, forming an intra-cluster distance set;

[0264] The threshold setting module is used to set an anomaly detection threshold based on the mean and standard deviation of the intra-cluster distance set;

[0265] The first anomaly point identification module is configured to identify a data point as a first type of anomaly point if the distance between the data point and the corresponding centroid exceeds the anomaly detection threshold.

[0266] In an embodiment of the present application, the threshold setting module is specifically configured to:

[0267] calculate the skewness of the distance set in the cluster, and adjust a weight factor used to calculate the anomaly detection threshold according to the absolute value of the skewness, so as to dynamically adjust the anomaly detection threshold.

[0268] In an embodiment of the present application, the first anomaly point identification module further includes:

[0269] The LOF value calculation module is configured to calculate a local outlier factor (LOF) value of each data point.

[0270] The final determination module is configured to finally determine a data point as a first type of anomaly point if the distance between the data point and the corresponding centroid exceeds the anomaly detection threshold and the LOF value of the data point exceeds a preset LOF threshold.

[0271] In an embodiment of the present application, the KMeans clustering execution module specifically includes:

[0272] The centroid initialization module is configured to initialize a plurality of cluster centroids based on the optimal cluster number.

[0273] The data point assignment module is configured to assign each data point to a cluster corresponding to a cluster centroid closest to the data point in terms of Euclidean distance.

[0274] The centroid updating module is configured to recalculate the centroid of each cluster and update the centroid as a new cluster centroid.

[0275] The iteration control module is configured to repeat the above steps until the cluster assignment no longer changes or a preset maximum iteration number is reached.

[0276] In an embodiment of the present application, the second cleaning module specifically includes:

[0277] The minimum point number setting module is configured to set a minimum point number parameter based on the data dimension.

[0278] The neighborhood radius determination module is configured to adaptively determine a neighborhood radius parameter by calculating a k-distance ranking curve and identifying an inflection point of the curve.

[0279] The traversal elimination module is configured to traverse and eliminate all data points after the first type of anomaly points according to the dynamically determined neighborhood radius and minimum point number parameters.

[0280] The core point recognition module is configured to recognize a point with a number of data points contained in a neighborhood of the point being not less than the minimum number of points as a core point.

[0281] The cluster forming module is configured to form a cluster from any core point by density reachable relationship.

[0282] The second-type abnormal point recognition module is configured to recognize a data point that does not belong to any cluster and is neither a core point nor a boundary point as a noise point of the second-type abnormal point.

[0283] In an embodiment of the present application, the data processing module comprises:

[0284] The preliminary screening module is configured to preliminarily screen extreme outliers by using an interquartile range (IQR) based method.

[0285] The data standardization module is configured to normalize coordinate data by using a z-score standardization method.

[0286] The format checking and converting module is configured to check and convert data formats to generate a structured numerical array.

[0287] The deduplication processing module is configured to perform deduplication processing on data to eliminate duplicate coordinate points.

[0288] In an embodiment of the present application, the result generation module is specifically configured to:

[0289] According to a preset mode, all the identified abnormal points are eliminated from original data or are marked as abnormal on the basis of retaining the original data.

[0290] The above-mentioned modules in the device for industrial key point data cleaning and abnormal point elimination can be all or partially realized by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above-mentioned modules.

[0291] Referring to Figure 17 The present application also provides a computer device, which comprises a memory 802, a processor 801 and a computer program 8021 stored in the memory 802 and executable on the processor 801, and the processor 801 implements the method for industrial key point data cleaning and abnormal point elimination as described above when executing the computer program 8021.

[0292] The present application also provides a computer storage medium, which stores the computer program 8021, and the program is executed by a processor to implement the method for industrial key point data cleaning and abnormal point elimination as described above.

[0293] It should be noted that each of the embodiments described herein is merely exemplary of the inventive concepts and that numerous modifications can be made to each of the embodiments without departing from the spirit and scope of the present disclosure. As such, the scope of the present disclosure is not to be limited to the embodiments described herein but is only to be limited by the scope of the claims that follow.

[0294] It should also be noted that the foregoing description is merely illustrative of various embodiments of the application, and that, therefore, numerous changes and modifications can be made to the illustrated embodiments and variations thereon without departing from the spirit and scope of the present application. It will be apparent to those skilled in the art that aspects of the present application can be embodied in a variety of ways, including as a system, method or device and that the application can be implemented using hardware components, software components, and / or combinations thereof. It is therefore contemplated to this end that certain features set forth herein could be provided in a combination of software and / or firmware instructions and if desired, programmed onto a computer readable medium, which can include non-transitory computer readable media, for execution by a processing unit. As will be realized, the application is capable of great adaptability and can be practiced in numerous different environments with various components substituted for those described, used with different systems, and / or in different architectures.

[0295] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM or EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC.

[0296] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An industrial key point data cleaning and outlier elimination method, characterized in that, The method comprises the following steps: acquiring industrial key point coordinate data to be processed, and preprocessing the coordinate data; dynamically determining the optimal cluster number of the preprocessed coordinate data; based on the optimal cluster number, using a first clustering algorithm to cluster the coordinate data, and identifying first type of abnormal points with a distance to the cluster center of the corresponding cluster exceeding a first dynamic threshold according to the distance distribution statistical characteristics in each cluster; using a second clustering algorithm based on density to perform secondary cleaning on the data after removing the first type of abnormal points, to identify second type of abnormal points judged as noise points based on the local density of the data points; integrating the first type of abnormal points and the second type of abnormal points to generate a cleaned data set; wherein the dynamic determination of the optimal cluster number of the preprocessed coordinate data comprises: in a preset initial cluster number interval, for each cluster number value, calculating the sum of squared Euclidean distances of all points in the corresponding cluster to the cluster center; based on the change rate of the sum of squared Euclidean distances in the cluster, using the elbow rule to determine an elbow point, and determining a candidate cluster number range centered on the elbow point; in the candidate cluster number range, for each cluster number value, calculating the average silhouette coefficient corresponding thereto; selecting the cluster number value that maximizes the average silhouette coefficient as the optimal cluster number.

2. The method of claim 1, wherein, The identification of the first type of abnormal points according to the distance distribution statistical characteristics in each cluster comprises: after the first clustering algorithm is completed, calculating the center of each cluster; calculating the distance of each data point in each cluster to the corresponding center, to form a cluster distance set; based on the mean and standard deviation of the cluster distance set, setting an anomaly detection threshold; identifying data points with a distance to the corresponding center exceeding the anomaly detection threshold as the first type of abnormal points.

3. The method of claim 2, wherein, The setting of the anomaly detection threshold further comprises: calculating the skewness of the cluster distance set, and adjusting the weight factor used to calculate the anomaly detection threshold according to the absolute value of the skewness, to dynamically adjust the anomaly detection threshold.

4. The method according to claim 2 or 3, characterized in that, The identification of the first type of abnormal points further comprises: calculating the local outlier factor (LOF) value of each data point; identifying data points with a distance to the corresponding center exceeding the anomaly detection threshold and an LOF value exceeding a preset LOF threshold as the first type of abnormal points.

5. The method of claim 1, wherein, The first clustering algorithm is a K-Means algorithm, which specifically comprises: initializing multiple cluster centers based on the optimal cluster number; assigning each data point to the cluster corresponding to the cluster center with the closest Euclidean distance; recalculating the center of each cluster and updating it as a new cluster center; repeating the above steps until the cluster assignment no longer changes or a preset maximum iteration number is reached.

6. The method of claim 1, wherein, The second clustering algorithm is a DBSCAN algorithm, and its parameters are dynamically determined, specifically comprising: setting a minimum point number parameter based on the data dimension; adaptively determining a neighborhood radius parameter by calculating a k-distance ordering curve and identifying the inflection point of the curve; traversing all data points after removing the first type of abnormal points according to the dynamically determined neighborhood radius and minimum point number parameter; identifying points with a number of data points contained in the neighborhood not less than the minimum point number as core points; A cluster is formed by density reachable relationship from any core point; Data points that do not belong to any cluster and are neither core points nor boundary points are identified as noise points as the second type of abnormal points.

7. The method of claim 1, wherein, The preprocessing step includes: Extreme outliers are preliminarily screened out by using an IQR-based method; The coordinate data is normalized by using a z-score standardization method; The data is format-verified and converted to generate a structured numerical array; And the data is de-duplicated to eliminate duplicate coordinate points.

8. The method of claim 1, wherein, The generating of the cleaned data set includes: All the identified abnormal points are removed from the original data or are marked as abnormal on the basis of retaining the original data according to a preset mode.

9. An apparatus for cleaning and removing outlier data of key industrial points, characterized in that, It includes: A data processing module for obtaining industrial key point coordinate data to be processed and preprocessing the coordinate data; A cluster number determination module for dynamically determining the optimal clustering cluster number of the preprocessed coordinate data; A first cleaning module for clustering the coordinate data based on the optimal clustering cluster number by using a first clustering algorithm and identifying a first type of abnormal points with a distance from the cluster center of the cluster exceeding a first dynamic threshold according to the distance distribution statistical characteristics within each cluster; A second cleaning module for performing secondary cleaning on the data after removing the first type of abnormal points by using a second clustering algorithm based on density to identify a second type of abnormal points that are determined to be noise points based on local density of data points; A result generation module for integrating the first type of abnormal points and the second type of abnormal points to generate a cleaned data set; The cluster number determination module includes: A sum of squares calculation module for calculating the sum of squares of Euclidean distances of all points within a cluster from the cluster center corresponding to each cluster number value in a preset initial cluster number interval; A candidate range determination module for determining an elbow point by using an elbow rule based on the change rate of the sum of squares of Euclidean distances within the cluster and determining a candidate cluster number range centered on the elbow point; A contour coefficient calculation module for calculating the average contour coefficient corresponding to each cluster number value in the candidate cluster number range; An optimal cluster number selection module for selecting the cluster number value that maximizes the average contour coefficient as the optimal clustering cluster number.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method for industrial key point data cleaning and abnormal point removal according to any one of claims 1 to 8.

11. A computer storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method for industrial key point data cleaning and abnormal point removal according to any one of claims 1 to 8.

Images