Industrial key point data cleaning and abnormal point eliminating method and related device

Abstract

The invention discloses an industrial key point data cleaning and abnormal point removing method and a related device, and the method comprises the steps: obtaining to-be-processed industrial key point coordinate data, and carrying out the preprocessing of the coordinate data; dynamically determining the optimal clustering number of the preprocessed coordinate data; based on the optimal clustering number, clustering the coordinate data by adopting a first clustering algorithm, and identifying a first type of abnormal points according to distance distribution statistical characteristics in each cluster; performing secondary cleaning on the data from which the first type of abnormal points are removed by adopting a density-based second clustering algorithm so as to identify a second type of abnormal points which are judged as noise points; and integrating the first type of abnormal points and the second type of abnormal points to generate a cleaned data set. According to the method, dynamic clustering, anomaly detection based on intra-cluster statistics and density secondary cleaning are combined, so that the adaptive capacity, accuracy and robustness of the data cleaning process are enhanced, and a purer and more reliable data set can be produced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial data processing, and in particular to a method for cleaning and removing abnormal points of industrial key point data and a related device. BACKGROUND

[0002] In the field of data-driven artificial intelligence and big data analysis, data quality directly affects the performance and reasoning accuracy of the model. However, in the actual collection and annotation process, the data often has a certain degree of abnormality, noise or incorrect annotation phenomenon. For example, in image key point annotation, sensor sampling, user behavior log and other applications, there may be problems such as offset points, repeated points, systematic errors or human annotation errors. Such "dirty data" not only affects the model training process, but also causes the model to make incorrect judgments in the deployment stage.

[0003] In order to solve the above problems, some data cleaning techniques have been proposed in related technologies. Common methods include rule-based anomaly detection (such as setting range thresholds), statistical analysis (such as mean / variance outlier detection) or machine learning methods (such as isolation forest, principal component analysis, etc.). However, these methods usually rely on global statistical distribution or assume that the data follows a certain distribution, and the processing effect on complex data structures (especially local cluster structures) is limited, and misjudgment or omission may occur. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the purpose of the present application is to propose a method for cleaning and removing abnormal points of industrial key point data and a related device.

[0005] To achieve the above-mentioned purpose, in a first aspect, a method for cleaning and removing abnormal points of industrial key point data according to an embodiment of the present application comprises: obtaining industrial key point coordinate data to be processed, and preprocessing the coordinate data; dynamically determining the optimal cluster number of the preprocessed coordinate data; based on the optimal cluster number, using a first clustering algorithm to cluster the coordinate data, and according to the distance distribution statistical characteristics in each cluster, identifying a first type of abnormal point whose distance to the cluster center of the cluster exceeds a first dynamic threshold; using a second clustering algorithm based on density to perform secondary cleaning on the data after removing the first type of abnormal point, to identify a second type of abnormal point that is judged to be a noise point based on the local density of the data point; integrating the first type of abnormal point and the second type of abnormal point to generate a cleaned data set.

[0006] According to one embodiment of the present application, the dynamically determining the optimal cluster number of the pre-processed coordinate data comprises: Within a preset initial cluster number interval, for each cluster number value, the Euclidean distance square sum of all points in the corresponding cluster to the cluster center is calculated; Based on the change rate of the Euclidean distance square sum in the cluster, an elbow point is determined using the elbow rule, and a candidate cluster number range is determined with the elbow point as the center; Within the candidate cluster number range, for each cluster number value, the corresponding average silhouette coefficient is calculated; The cluster number value that maximizes the average silhouette coefficient is selected as the optimal clustering cluster number.

[0007] According to one embodiment of the present application, the identifying the first type of abnormal points according to the distance distribution statistical characteristics in each cluster comprises: After the first clustering algorithm is completed, the centroid of each cluster is calculated; The distance of each data point in the cluster to the corresponding centroid is calculated to form an intra-cluster distance set; Based on the mean and standard deviation of the intra-cluster distance set, an anomaly detection threshold is set; Data points with a distance to the corresponding centroid exceeding the anomaly detection threshold are identified as the first type of abnormal points.

[0008] According to one embodiment of the present application, the setting of the anomaly detection threshold further comprises: The skewness of the intra-cluster distance set is calculated, and the weight factor used to calculate the anomaly detection threshold is adjusted according to the absolute value of the skewness, thereby dynamically adjusting the anomaly detection threshold.

[0009] According to one embodiment of the present application, the identifying the first type of abnormal points further comprises: The local outlier factor (LOF) value of each data point is calculated; Data points with a distance to the corresponding centroid exceeding the anomaly detection threshold and with an LOF value exceeding a preset LOF threshold are finally determined as the first type of abnormal points.

[0010] According to one embodiment of the present application, the first clustering algorithm is the K-Means algorithm, specifically comprising: Based on the optimal clustering cluster number, multiple cluster centroids are initialized; Each data point is assigned to the cluster corresponding to the cluster centroid with the closest Euclidean distance; The centroid of each cluster is recalculated and updated as a new cluster centroid; The above steps are repeated until the cluster assignment no longer changes or a preset maximum iteration number is reached.

[0011] According to one embodiment of the present application, the second clustering algorithm is a DBSCAN algorithm, parameters of which are dynamically determined, and specifically include: Setting a minimum point number parameter based on data dimension; Adaptively determining a neighborhood radius parameter by calculating a k-distance order curve and identifying an inflection point of the curve; Traversing and removing all data points after the first type of abnormal points according to the dynamically determined neighborhood radius and minimum point number parameters; Identifying points with a number of data points contained in the neighborhood no less than the minimum point number as core points; Forming a cluster by density reachable relationship from any core point; Identifying data points that are neither core points nor boundary points and do not belong to any cluster as noise points of the second type of abnormal points.

[0012] According to one embodiment of the present application, the preprocessing step includes: Preliminarily screening extreme outliers by using an IQR-based method; Normalizing coordinate data by using a z-score standardization method; Performing format checking and conversion on data to generate a structured numerical array; And, performing a deduplication process on data to remove duplicate coordinate points.

[0013] According to one embodiment of the present application, the generating of the cleaned data set includes: According to a preset mode, removing all identified abnormal points from the original data or marking the original data as abnormal based on the original data.

[0014] In a second aspect, the device for industrial key point data cleaning and abnormal point removal according to an embodiment of the present application includes: A data processing module configured to obtain industrial key point coordinate data to be processed and preprocess the coordinate data; A cluster number determination module configured to dynamically determine an optimal clustering cluster number of the preprocessed coordinate data; A first cleaning module configured to cluster the coordinate data by using a first clustering algorithm based on the optimal clustering cluster number, and identify a first type of abnormal points with a distance from a cluster center of a cluster exceeding a first dynamic threshold based on distance distribution statistical characteristics in the cluster; A second cleaning module configured to clean data after removing the first type of abnormal points by using a second clustering algorithm based on density to identify a second type of abnormal points determined as noise points based on local density of data points; A result generation module is configured to integrate the first type of abnormal points and the second type of abnormal points to generate a cleaned data set.

[0015] In a third aspect, a computer device according to an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for industrial key point data cleaning and abnormal point elimination as described above when executing the computer program.

[0016] In a fourth aspect, a computer storage medium according to an embodiment of the present application has a computer program stored thereon, and the program is executable on a processor to implement the method for industrial key point data cleaning and abnormal point elimination as described above.

[0017] The method for industrial key point data cleaning and abnormal point elimination and the related device according to the embodiments of the present application first overcome the defect of poor adaptability to different data distributions caused by relying on fixed parameters by dynamically determining the optimal cluster number, and ensure that the subsequent clustering analysis can more accurately reflect the internal structure of the data. After preliminary clustering based on the optimal cluster number, a threshold is dynamically set according to the distance distribution statistical characteristics of each cluster to identify the first type of abnormal points, which realizes the fine and localized judgment of the abnormal points and avoids the misjudgment and omission that may be caused by the global unified threshold when processing uneven data. Finally, the second clustering algorithm based on density is introduced for secondary cleaning, which can effectively identify and eliminate the second type of abnormal points in the form of noise that may be missed in the previous stage, and further improves the thoroughness of cleaning. Therefore, the present application combines dynamic clustering, abnormal detection based on cluster statistics and density secondary cleaning, significantly enhances the adaptive ability, accuracy and robustness of the data cleaning process, and thus can output more pure and reliable high-quality data set.

[0018] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0020] Figure 1 is a flowchart of an embodiment of the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 2is a flow chart of step S102 in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 3 is a flow chart of the first clustering algorithm in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 4 is a flow chart of identifying the first type of abnormal point in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 5 is a flow chart of the second clustering algorithm in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 6 is a flow chart of step S101 in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 7 is a workpiece picture marked with data points obtained in step S101 in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 8 is a WCSS curve graph of the intra-cluster Euclidean distance square sum in step S201 in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 9 is a K-distance ordering curve graph in step S502 in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 10 is a two-dimensional plane scatter plot of the KMeans clustering cleaning result in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 11 is a text file graph saved after KMeans clustering cleaning processing in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 12 is a certain workpiece picture marked with abnormalities corresponding to the KMeans clustering cleaning result in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 13 is a two-dimensional plane scatter plot of the DBSCAN clustering cleaning result in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 14 is a text file graph saved after DBSCAN clustering cleaning processing in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 15 is a certain workpiece picture marked with abnormalities corresponding to the DBSCAN clustering cleaning result in the method for industrial key point data cleaning and abnormal point elimination of the present application; Figure 16 is a structural schematic diagram of one embodiment of the device for industrial key point data cleaning and abnormal point elimination of the present application; Figure 17is a structural schematic diagram of an embodiment of the computer device of the present application.

[0021] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0022] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application.

[0023] Referring to Figure 1 As shown, Figure 1 A flowchart of one embodiment of the method for industrial key point data cleaning and abnormal point elimination provided by the embodiment of the present application is shown, only the part related to the embodiment of the present application is shown for convenience of description. Specifically, the method for industrial key point data cleaning and abnormal point elimination can be executed by a computer, specifically including: S101, obtaining the industrial key point coordinate data to be processed, and pre-processing the coordinate data. This step aims to provide a reliable data basis for subsequent accurate analysis.

[0024] In an industrial automation production scene, for example, on a visual quality inspection line of parts, a plurality of industrial cameras deployed will continuously shoot part images, and automatically label the coordinates of a plurality of key points such as screw hole center, edge corner, welding point position, etc. through a preliminary image recognition model. These original coordinate data are used as the data set to be processed. Figure 7 An example is a workpiece picture marked with data points, there are a total of 16 points in the image, the four points from 0 to 3 are the four vertices of a rectangular frame, which identifies the area range of the workpiece. Points 4 to 15 are key points inside the workpiece, such as marked position points used in the automated dispensing process. In the data acquisition process, all key point coordinate information including detection area coordinates can be read from a pre-prepared CSV file to ensure data integrity and correctness.

[0025] However, due to the complex and changeable production environment, such as sudden changes in light, workpiece surface reflection, conveyor belt vibration or camera lens contamination, the collected coordinate data will inevitably be mixed with noise and abnormalities. If not processed, these "dirty data" will seriously distort the true distribution of the data, directly affecting the accuracy of subsequent analysis. Therefore, a preprocessing step is performed to eliminate these original noise and structural bias, ensuring that the data provided to subsequent analysis is preliminarily purified and standardized. The preprocessing process can eliminate extreme outliers that deviate from the normal range, such as invalid data caused by program errors or hardware failures, whose coordinate values are far beyond the physical boundaries of the workpiece. In addition, standardization of coordinate data, data format verification, and deduplication, etc. ensure the accuracy and robustness of subsequent analysis.

[0026] S102, dynamically determining the optimal cluster number of the preprocessed coordinate data.

[0027] In industrial applications, the source of data has high heterogeneity. Due to the influence of factors such as lighting conditions, shooting angles, process fluctuations, equipment parameters, and operation methods, there are significant differences in the distribution of data collected at different times and in different batches. This uncertainty makes it difficult to maintain stability and effectiveness in all scenarios by pre-setting a fixed cluster number. For example, a batch of data may naturally cluster into 3 categories, while another batch of data is more suitable for being divided into 5 categories or even more, which leads to a sharp decline in algorithm performance. To solve this problem, this step uses a dynamically adaptive cluster number selection mechanism that considers clustering compactness and separation degree to more robustly determine the optimal cluster number, greatly improving the adaptive ability and robustness in variable industrial scenarios.

[0028] S103, based on the optimal cluster number, using a first clustering algorithm to cluster the coordinate data, and according to the distance distribution statistical characteristics within each cluster, identifying a first type of abnormal point whose distance from the cluster center of the cluster exceeds a first dynamic threshold.

[0029] This step is used to perform the first cleaning. After clustering is completed, each cluster can be understood as an aggregation area of a certain type of normal data points, defining the "normal" range and distribution characteristics of that type of data. Traditional global threshold methods cannot handle unevenly distributed data. A point that may be normal in a sparse cluster may be a significant anomaly in a tight cluster. This step gives each cluster an abnormality judgment boundary. By analyzing the dispersion degree of data points within each cluster, a threshold value that matches its own statistical characteristics can be dynamically generated. In this way, whether a point is abnormal is no longer determined by its absolute coordinates, but by its deviation from the behavior of its own group. This discrimination method can accurately identify abnormal points that are mixed in the normal group but slightly deviate, greatly improving the sensitivity and accuracy of detection.

[0030] S104, on the data after the first type of outlier is removed, a second clustering algorithm based on density is used for secondary cleaning to identify the second type of outlier based on the local density of the data points being judged as noise points.

[0031] The purpose of this step is to make up for the possible blind spots in the first cleaning and further improve the thoroughness of the cleaning. The first cleaning is based on the detection method of cluster centroid, which is good at detecting abnormal points deviating from the center of the group, but it may be difficult to effectively identify certain types of abnormalities, such as isolated points that do not belong to any clear cluster and are sparsely distributed. That is, after the first clustering analysis is completed, there may still be a small number of isolated points or boundary abnormal points that have not been accurately identified. This step introduces a density-based clustering analysis by analyzing the "neighborhood" density around each data point. Those points that are located in sparse areas and cannot form a density core are identified as noise regardless of their absolute position, thereby further improving the purity of the final data set.

[0032] S105, integrate the first type of abnormal point and the second type of abnormal point, and generate a cleaned data set.

[0033] This step summarizes the abnormal results detected by the two previous cleanings and generates a high-quality final data set that can be directly used by downstream tasks. In specific applications, according to application requirements, it can generate a "pure" data that excludes all abnormal points, which can be used to train more stable and accurate artificial intelligence models. Alternatively, it can choose to retain the original data and only mark the abnormal points, and deliver a detailed report with information saved to trace the root cause of the problem, such as diagnosing which device or process has a systematic deviation, etc. Thus, this method can stably output a highly reliable and pure data set, providing data support for improving the level of industrial intelligence.

[0034] According to the method for cleaning and removing outliers of industrial key point data provided by the embodiment of the present invention, first, by dynamically determining the optimal number of clusters, it overcomes the defect of poor adaptability to different data distributions caused by relying on fixed parameters, ensuring that subsequent cluster analysis can more accurately reflect the internal structure of the data. After preliminary clustering based on the optimal number of clusters, it dynamically sets the threshold to identify the first type of outliers according to the statistical characteristics of the distance distribution of each cluster itself, achieving refined and localized judgment of anomalies, and avoiding the misjudgment and omission that may occur when processing data with uneven density using a global unified threshold. Finally, by introducing a second density-based clustering algorithm for secondary cleaning, it can effectively identify and remove the second type of outliers that may have been missed in the previous stage and exist in the form of noise, further improving the thoroughness of cleaning. Therefore, the present invention significantly enhances the adaptability, accuracy and robustness of the data cleaning process by combining dynamic clustering, anomaly detection based on intra-cluster statistics and density secondary cleaning, thereby producing a purer, more reliable and high-quality data set.

[0035] Reference Figure 2 As shown, in one embodiment of the present invention, the dynamically determining the optimal number of clusters of the pre-processed coordinate data includes: S201, within a preset initial cluster number interval, for each cluster value, calculate the sum of squares of the Euclidean distances (WCSS) of all points in the corresponding cluster to the corresponding cluster center. For example, the preset initial cluster number interval , calculate different WCSS curve under ; for a given number of clusters K, the clustering result is , the centroid of each cluster is μ k , the calculation formula of WCSS is: .

[0036] Among them, C k represents the kth cluster, X i It is cluster C k A data point in μ k is the centroid of the cluster; is the square of the Euclidean distance between the data point and the centroid. The WCSS metric measures the sum of the squared distances of all data points to the centroid of their cluster. The smaller the value, the more compact the data points within the cluster and the higher the degree of aggregation. By calculating the WCSS under different K values, we can obtain a WCSS curve that decreases monotonically as the K value increases (e.g. Figure 8 shown).

[0037] S202: Based on the change rate of the sum of squares of the Euclidean distances within the cluster, an elbow point is determined using the elbow rule, and a range of candidate cluster numbers is determined with the elbow point as the center.

[0038] When the K value increases from an insufficient value to a suitable value, the WCSS decreases rapidly; but when the K value exceeds the true optimal cluster number, even if the K value is increased, the decrease of the WCSS will be sharply slowed down, forming an elbow-like inflection point. In order to accurately locate the elbow point, the differential slope method is adopted, for example, by calculating the second-order derivative approximation value of the WCSS curve, the formula is as follows: .

[0039] Select the point with the largest slope change as the elbow initial K elbow , and set a small offset δ (for example, for a data set with a key point number between 10 and 20, δ can be 2) centered on the K elbow , so as to determine a candidate cluster number range , for example, K elbow = 5, then the candidate cluster number range is . This way effectively reduces the search space and avoids complex calculations for all possible K values, improving efficiency.

[0040] S203, for each cluster number value in the candidate cluster number range, calculate the corresponding average silhouette coefficient.

[0041] In order to more accurately evaluate the clustering quality, the silhouette coefficient is introduced for fine selection. Unlike the WCSS which only focuses on the tightness within the cluster, the silhouette coefficient takes into account both the cohesion and separation of the cluster. For any data point i in the data set, first calculate the average distance a(i) from all other data points within its own cluster, i.e. "intra-class distance", which represents cohesion. Then, calculate the average distance b(i) from all data points in the nearest other cluster, i.e. "inter-class distance", which represents separation. Finally, calculate the average silhouette coefficient S(K) of the entire data set corresponding to each K value in the candidate cluster number range, i.e. the mean of all data point silhouette coefficients s(i).

[0042] The silhouette coefficient calculation formula of a single sample: .

[0043] The value range of the silhouette coefficient is between [-1, 1], and the closer the value is to 1, the better the clustering effect of the data point.

[0044] The average silhouette coefficient calculation formula of the entire data set: .

[0045] S204, select the cluster number value that maximizes the average silhouette coefficient as the optimal clustering cluster number K*

[0046] .

[0047] That is, by comparing the average silhouette coefficient corresponding to all K values in the candidate cluster number range, the cluster number value that makes the coefficient value maximum is selected as the final optimal clustering cluster number K*. This step ensures that the selected cluster number not only makes the clusters as compact as possible, but also makes the different clusters as far away from each other as possible, thereby finding a clustering effect that balances the cohesion and separation.

[0048] Through the dynamic cluster number determination method combining the elbow rule and the silhouette coefficient described above, the embodiment greatly improves the adaptive ability of the algorithm, enabling it to automatically adapt to industrial data of different sources, different batches, and different distribution characteristics without human intervention and prior knowledge, thereby solving the problem of poor generalization caused by reliance on fixed parameters. In addition, both the calculation efficiency and the selection accuracy are taken into account, the elbow rule is used to quickly lock the high-potential candidate interval, and then the silhouette coefficient is used for fine evaluation, thereby ensuring the accuracy and stability of the optimal cluster number selection, making the clustering result more truly reflect the internal structure of the data, providing a guarantee for subsequent anomaly point detection based on cluster statistics, and improving the robustness and accuracy of the entire data cleaning.

[0049] Referring to Figure 3 Fig. 1, in an embodiment of the present application, the first clustering algorithm is a K-Means algorithm, which specifically includes: S301, initializing a plurality of cluster centroids based on the optimal clustering cluster number.

[0050] S302, assigning each data point to the cluster corresponding to the cluster centroid closest to it in Euclidean distance.

[0051] S303, recalculating the centroid of each cluster and updating it as a new cluster centroid.

[0052] S304, repeating the above steps until the cluster assignment no longer changes or the maximum number of iterations is reached.

[0053] In the embodiment, based on the dynamically selected optimal cluster number, clustering analysis is performed on the structured point set data, and each data point is classified to its nearest cluster centroid, thereby achieving automated grouping. The K-Means clustering algorithm is adopted, and the objective is to minimize the sum of squared Euclidean distances of all points to their corresponding cluster centroids, i.e.: .

[0054] wherein, is the i th point; C k is the k th cluster. .

[0055] is the Kth cluster's centroid.

[0056] The entire clustering process is iterated, that is: first, initialize K* cluster centroids; then, assign each data point to the cluster corresponding to the nearest cluster centroid in terms of Euclidean distance; then, recalculate the centroid of each cluster and update it as the new cluster centroid; repeat the above steps until the cluster assignment no longer changes or the maximum number of iterations is reached.

[0057] Referring to FIG. 1 Figure 4 In one embodiment of the present application, the identification of the first type of abnormal points according to the distance distribution statistical characteristics in each cluster comprises: S401, after the first clustering algorithm is completed, the centroid of each cluster is calculated.

[0058] For the Kth cluster, its centroid (center point) is defined as the mean vector of all data points in the cluster: .

[0059] wherein, denotes the coordinates of the cluster centroid; is the number of data points in cluster K; X j is the data point in the cluster. This centroid represents the "center position" of the cluster in geometry, which is the reference point for subsequent distance calculation.

[0060] S402, the distance of each data point in each cluster to its corresponding centroid is calculated to form a distance set in the cluster.

[0061] The Euclidean distance of each data point in a cluster to the centroid: .

[0062] The distance set in the cluster: .

[0063] That is, after the centroid is calculated, the Euclidean distance of each data point in the cluster to its corresponding centroid is calculated. By traversing all points in the cluster, a distance set belonging to the cluster is formed, which completely reflects the dispersion degree and distribution pattern of the data points in the cluster.

[0064] S403, based on the mean and standard deviation of the distance set in the cluster, an anomaly detection threshold is set.

[0065] This step specifically includes: first, for the distance set D kThe mean and standard deviation thereof are calculated. The setting of the anomaly detection threshold is based on the statistical distribution of the distance set, especially the mean and standard deviation, so the distance set D k The mean and standard deviation thereof are calculated.

[0066] The mean is: .

[0067] The standard deviation is: .

[0068] Next, for each cluster C k , the distance set of its data points to the centroid is calculated: .

[0069] Finally, based on the mean μ k and the standard deviation σ k of the distance, the anomaly detection threshold is constructed: .

[0070] Wherein, the parameter α controls the looseness of the threshold, according to experience, when α is 2, for most industrial scenes, good preliminary screening effect can be achieved.

[0071] Since the structural complexity and data fluctuations in each cluster are taken into account, a dynamic confidence interval adjustment strategy can also be introduced to dynamically adjust the anomaly detection threshold, and the dynamic confidence interval adjustment strategy includes: The skewness of the distance set in the cluster is calculated, and according to the absolute value of the skewness, the weight factor for calculating the anomaly detection threshold is adjusted, so as to dynamically adjust the anomaly detection threshold.

[0072] More specifically, the skewness and kurtosis of the distance D k are calculated to determine the degree of deviation of the distribution, if the distance distribution is approximately normal, the anomaly detection threshold T' k remains unchanged; otherwise, the threshold weight β k is adjusted in combination with the skewness, for example: .

[0073] Wherein, γ is an adjustment coefficient, used to enhance the anomaly discrimination sensitivity of the skew cluster, in practice, a slightly strong γ=1.5 is adopted, and the effect is better. skew(D k ) refers to the skewness of D k , that is, the degree of asymmetry of the data distribution.

[0074] .

[0075] where μ is the mean, σ is the standard deviation, and E denotes the mathematical expectation operator.

[0076] At this time, the anomaly detection threshold is updated as: .

[0077] S404, identifying the data points with a distance from the corresponding centroid exceeding the anomaly detection threshold as first-class abnormal points.

[0078] To avoid misjudgment caused by the scale difference between clusters, in combination with the local outlier factor concept, the identification of the first-class abnormal points further includes: Calculating the local outlier factor LOF value of each data point.

[0079] The data points with a distance from the corresponding centroid exceeding the anomaly detection threshold and their LOF values exceeding the preset LOF threshold are finally determined as first-class abnormal points.

[0080] Specifically, first, the relative density anomaly degree of each data point is calculated: .

[0081] where N k (X i ) is the k-distance neighborhood set of point X i ; And reach-dist and lrd are the kth reachable distance and local reachable density, respectively, which are mathematically defined as follows: .

[0082] where d(X i， X j ) is the Euclidean distance between data points X i ， X j ; d k (X i ) is the Kth distance of data point X i .

[0083] .

[0084] lrd k (X i ), in essence, is the reciprocal of the "average reachable distance". If X i is surrounded by small reachable distances, it means that it is in a high-density area, so lrd k (X i ) will be large; if X i is isolated, the average reachable distance is large, so lrd k (X i ) is small.

[0085] For LOF k (X i ), if the density of X i is similar to its neighbors, then LOF≈1 (normal point); if the density of X i is much smaller than its neighbors, then LOF≥1 (outlier); if LOF<1, then X i is even more "dense" than its neighbors.

[0086] Then, the threshold is integrated to determine the abnormal point: If the distance of a point X is greater than τ*σ, and , then the point X i is determined as an abnormal point.

[0087] Wherein, according to experience, the value of τ is 1.5, the detection is more robust, and part of the normal points that may be excluded are protected.

[0088] Through the above method of the embodiment, firstly, the "localization" and "refinement" of the abnormality determination are realized, the statistical characteristics and the dynamic abnormality detection threshold are calculated for each cluster, and the defects that the global threshold is prone to produce misjudgment and omission when processing unevenly dense data are overcome. Secondly, the threshold is dynamically adjusted by introducing skewness, and double verification is performed in combination with the local outlier factor (LOF), which greatly enhances the robustness and accuracy of the detection process, and can effectively protect the normal data points located at the edge of the sparse cluster from being incorrectly excluded. The comprehensive discrimination strategy based on the distance distribution within the cluster and the local density ensures that the first cleaning can efficiently and reliably identify abnormal data that deviates from the behavior mode of the group to which it belongs, and lays a foundation for subsequent secondary cleaning and finally generating a high-quality data set.

[0089] Referring to FIG. 7, in an embodiment of the present application, the second clustering algorithm is a DBSCAN algorithm, parameters of which are dynamically determined, and specifically include: Figure 5 S501, setting a minimum point number parameter based on data dimension. S502, adaptively determining a neighborhood radius parameter by calculating a k-distance ordering curve (as shown in FIG. 8) and identifying the inflection point of the curve.

[0090] Figure 9 S503, traversing all data points after the first type of abnormal points are removed according to the dynamically determined neighborhood radius and minimum point number parameters.

[0091] S504, identifying a point as a core point if the number of data points contained in the neighborhood is not less than the minimum point number.

[0092] S505, identifying a point as a border point if the number of data points contained in the neighborhood is less than the minimum point number. ​

[0093] S505、From any core point, a cluster is formed by density reachable relationship.

[0094] S506、Data points that do not belong to any cluster, neither core points nor boundary points, are identified as noise points as the second type of abnormal points.

[0095] In this embodiment, after the preliminary clustering analysis is completed, there may still be a small number of isolated points or boundary abnormal points that are not accurately identified. In order to further improve the data quality, the DBSCAN algorithm based on density clustering method is introduced for secondary cleaning.

[0096] The DBSCAN algorithm does not depend on the number of clusters, but divides the data based on density. When a low-density area is found, noise points can be naturally identified, which is suitable for abnormal cleaning after the Kmeans algorithm.

[0097] The core content of the DBSCAN algorithm is as follows: 1. Core point: A point X i has at least minPts points in its ε neighborhood.

[0098] 2. Boundary point: A point in the ε neighborhood of a core point, but the points contained in its own neighborhood are less than minPts.

[0099] 3. Noise point: Neither a core point nor a boundary point.

[0100] For minPts, take minPts = sample data dimension * 2. In this project, each key point has information of 2 dimensions of x coordinate and y coordinate. Therefore, minPts = 4.

[0101] For ε, after determining minPts, calculate the distance from each point X i to its minPts-1th nearest neighbor: d i = d(X i , X i k ), where k = minPts-1.

[0102] After sorting all d i in ascending order, a vector D is formed: .

[0103] Where n is the total number of points in the sample data. Then, draw the K-distance ordering curve of D (as shown in Figure 9 ), where the X-axis is the sorted point index and the Y-axis is the corresponding d i .

[0104] As shown in Figure 9 , it is the K-distance order curve drawn for the 6th key point, and the calculation of epsilon = 9.866. The calculation method of epsilon is to find the position of the elbow point with the largest slope change in the figure, that is, the vertical coordinate d i is the value of epsilon.

[0105] The neighborhood is defined as follows: .

[0106] The core point determination condition is: .

[0107] The density reachable representation is: If and X i is a core point, then the density of X j is reachable to X i。

[0108] For the abnormal point identification method: After DBSCAN clustering, the points not belonging to any cluster will be marked as noise points, which can be regarded as secondary abnormal points after Kmeans clustering analysis: .

[0109] Referring to Figure 6 , in an embodiment of the present application, the preprocessing step includes: S601, using the method based on the interquartile range IQR to preliminarily screen out extreme outliers.

[0110] In the preliminary screening, combined with simple statistical indicators, abnormal points obviously deviating from the normal range are removed to reduce the interference of noise on the model. Specifically, the first quartile Q1 and the third quartile Q3 of the feature can be calculated based on the interquartile range IQR method, and the interquartile range is defined: .

[0111] Among them, the first quartile Q1 refers to assuming that there are 100 data arranged from small to large, the 25th data is the first quartile, and similarly, the third quartile refers to the 75th data.

[0112] At the same time, the upper and lower boundaries are set: .

[0113] In one example, the weight coefficient κ can be preset according to experience or data distribution characteristics, for example, κ = 2 can be taken, and if it exceeds this upper and lower limit, it is considered to be a very extreme abnormal value and does not need subsequent clustering analysis to be eliminated.

[0114] S602, the coordinate data is normalized by using a z-score standardization method.

[0115] Since the coordinate data can come from different image resolutions, shooting scales or coordinate systems, in order to avoid affecting the learning effect of the model due to the difference in the value range, the original data needs to be standardized or normalized to unify the scale of each dimension and ensure that each feature has equal importance in calculation.

[0116] In clustering, distance measurement or regression modeling, if there is a difference in the order of magnitude of the feature dimension, the model is likely to be overly sensitive to high-amplitude features, thereby masking important information in other dimensions. Therefore, standardization is a key preprocessing step to improve the stability, convergence speed and interpretability of the model.

[0117] Specifically, the z-score standardization method can be used to process all coordinate features. This method assumes that the data approximately follows a Gaussian distribution, and for each coordinate feature x, the standardization calculation is as follows: .

[0118] Where x is the original coordinate value; μ is the sample mean of the dimension; σ is the sample standard deviation of the dimension; and z is the standardized value. After z-score processing, all coordinate features will have a distribution with a mean of 0 and a standard deviation of 1, so that they have equivalent weights in Euclidean space, thereby avoiding the dominance of a dimension in distance calculation.

[0119] S603, format checking and conversion of data to generate structured numerical arrays.

[0120] In order to ensure that the data can smoothly enter the clustering model and obtain accurate and effective output, the structure and content of the original data need to be systematically checked and converted in the preprocessing stage. The main goal is to unify the data format, correct coding errors, and construct a structured expression form suitable for mathematical modeling.

[0121] 1) Format unification and structured conversion: The original data can be stored in text, CSV, JSON or other semi-structured forms, containing image names, coordinate pairs, missing markers and other information. In actual operation, it should be converted into a standardized two-dimensional numerical array (such as a matrix) for subsequent analysis.

[0122] Specifically, suppose a data sample contains n key points, each with two-dimensional coordinates (x i ,y i ), the sample can be represented as: .

[0123] At this time, when processing multiple image samples, a three-dimensional tensor can be constructed: .

[0124] where N represents the number of samples (images), and each page matrix is a key point set of a sample.

[0125] When further expanded into a "flattened" feature vector for each row of samples, each sample can be represented as: .

[0126] 2) Format verification and exception correction: Since data may come from multiple channels, consistency verification of the original file is required, including: Field number check: Each line of data should meet the preset fixed dimension requirement, for example, if a line of data should contain an image identifier and n two-dimensional coordinate points (total of 2n values), the total field number should be 2n+1, any data line that does not meet this structure can be considered as format exception.

[0127] Value type verification: Non-numeric fields (such as strings, null values) should be removed or interpolated.

[0128] Illegal character processing: Regular cleaning of non-numeric characters mixed in coordinate fields (such as Chinese commas, spaces, unit symbols, etc.).

[0129] Encoding consistency: Ensure that the character set encoding (such as UTF-8 / GBK) is consistent during file reading to avoid field misplacement due to garbled characters.

[0130] For format exceptions that cannot be automatically repaired, record the line number and error type, and prompt manual intervention. The verification mechanism can be formally represented as: .

[0131] S604, de-duplicate the data to remove duplicate coordinate points.

[0132] Through the above preprocessing operations, the integrity and consistency of the input data can be guaranteed to the greatest extent, reducing the misjudgment and noise influence caused by data quality problems in the subsequent clustering process, thereby providing protection for accurate identification of abnormal points.

[0133] In an embodiment of the present application, the generating the cleaned dataset comprises: According to the preset mode, all the identified abnormal points are removed from the original data, or are marked as abnormal on the basis of the original data.

[0134] That is, after the foregoing multiple rounds of outlier identification and cleaning operations (including preliminary screening based on statistical rules, distance anomaly detection in Kmeans clustering, and density anomaly identification of DBSCAN), a more representative and higher quality dataset can be obtained. Therefore, the cleaning results are integrated to generate the final version of the "dataset" for subsequent analysis, model training or engineering application deployment.

[0135] For abnormal point processing, the following two processing methods can be selected according to business requirements and task sensitivity: Elimination mode: all samples determined to be abnormal are completely removed, which is suitable for tasks with extremely high data quality requirements, such as supervised model training.

[0136] Labeling mode: on the basis of retaining the original data, a new field is added to mark whether it is abnormal (set "is_outlier" to 0 or 1), which facilitates subsequent strategic processing, weighted analysis or visualization.

[0137] Specifically, which mode to provide to the operator ensures universality.

[0138] Referring to Figure 10 to Figure 14 In other embodiments of the present application, the method of industrial key point data cleaning and abnormal point elimination can further comprise: Visualizing and saving the generated dataset.

[0139] That is, after completing the clustering analysis and anomaly detection, the results are visualized and standardized for subsequent verification, review and model training.

[0140] 1) Cluster result visualization.

[0141] As Figure 10 shown, in order to facilitate understanding of the clustering effect and abnormal distribution, the results are intuitively displayed in a graphical manner. Figure 10 For a two-dimensional plane scatter plot, different colors are used to indicate different clusters; abnormal points are marked with special shapes (such as "X" or red circles); if the sample has an image background, the point image can be superimposed to enhance readability. Figure 10 The clustering and cleaning results for key point No. 10 are shown, which are divided into 3 clusters, and the crossed points in the figure are the abnormal points. The output cleaning results can be opened to find a certain picture marked as abnormal, as Figure 11The KMeans clustering cleaning processing is shown in the saved text file graph, opening one of the text files, for example, opening the 2023-03-15-14-49-35-909.bmp file can see Figure 12 The workpiece graph, through Figure 12 It can be seen that the red key point is point 10, which deviates greatly from the normal workpiece position, and the corresponding point of the corresponding picture is found out through data cleaning and displayed to the user.

[0142] 2) Abnormal detection result visualization.

[0143] As shown in Figure 13 , the core points, boundary points and noise points in the density detection are displayed through DBSCAN clustering analysis. Specifically, the DBSCAN clustering analysis of point 11 is divided into 7 clusters, and the crossed points are judged as abnormal points in density.

[0144] As shown in Figure 14 , the DBSCAN clustering cleaning processing is shown in the saved text file graph, opening one of the text files, for example, opening the DR8GTG000Y300006C7+5.bmp file can see Figure 15 The workpiece graph. As shown in Figure 15 , it is a certain workpiece graph screened as an abnormal point, and the red point is point 11, that is, the abnormal point, which can be found to deviate from the normal position.

[0145] In other embodiments of the present application, the industrial key point data cleaning and abnormal point elimination method can further include: Output a report according to the data set.

[0146] After completing data clustering and abnormal cleaning, a detailed analysis and cleaning report can be generated to ensure the transparency, reproducibility of the entire processing process, and facilitate team collaboration, superior review or model filing.

[0147] Exemplarily, the report content structure can include project and data overview, threshold setting logic, abnormal point detection statistics, result saving path description, etc. The report format can be PDF or Word format, containing charts and formulas, which is convenient for sharing and archiving. The report can contain timestamp, parameter configuration, processing person identification and other basic meta information; it can also be accompanied by.json format parameter configuration record and.csv format cluster analysis table.

[0148] Referring to Figure 16 , Figure 16A structural schematic diagram of one embodiment of an industrial key point data cleaning and outlier elimination device provided by the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown. Specifically, the industrial key point data cleaning and outlier elimination device comprises: The data processing module 701 is configured to acquire the industrial key point coordinate data to be processed and pre-process the coordinate data.

[0149] The cluster number determination module 702 is configured to dynamically determine the optimal clustering cluster number of the pre-processed coordinate data.

[0150] The first cleaning module 703 is configured to cluster the coordinate data by using a first clustering algorithm based on the optimal clustering cluster number, and identify a first type of outlier whose distance to the cluster center of the corresponding cluster exceeds a first dynamic threshold according to the distance distribution statistical characteristics in each cluster.

[0151] The second cleaning module 704 is configured to clean the data after eliminating the first type of outlier by using a second clustering algorithm based on density to identify a second type of outlier which is determined to be a noise point based on the local density of the data point.

[0152] The result generation module 705 is configured to integrate the first type of outlier and the second type of outlier to generate a cleaned data set.

[0153] According to the industrial key point data cleaning and outlier elimination device provided by the embodiment of the present application, first, the optimal clustering cluster number is dynamically determined, which overcomes the defect of poor adaptability to different data distributions caused by relying on fixed parameters, and ensures that the subsequent clustering analysis can more accurately reflect the internal structure of the data. After preliminary clustering based on the optimal cluster number, the threshold is dynamically set according to the distance distribution statistical characteristics of each cluster to identify the first type of outlier, which realizes the fine and localized judgment of the outlier, and avoids the misjudgment and omission of the global unified threshold when processing uneven data. Finally, the second cleaning is performed by introducing the second clustering algorithm based on density, which can effectively identify and eliminate the second type of outlier in the form of noise that may be missed in the previous stage, and further improves the thoroughness of cleaning. Therefore, by combining dynamic clustering, outlier detection based on cluster statistics and density secondary cleaning, the present application significantly enhances the adaptive ability, accuracy and robustness of the data cleaning process, so as to output a more pure and reliable high-quality data set.

[0154] In one embodiment of the present application, the cluster number determination module comprises: The sum of squares calculation module is configured to calculate the sum of squares of the Euclidean distances of all points in each cluster to the corresponding cluster center for each cluster number value in a preset initial cluster number interval. a candidate range determination module configured to determine an elbow point by using an elbow rule based on a rate of change of the sum of squared Euclidean distances in the cluster, and determine a candidate cluster number range centered on the elbow point; a silhouette coefficient calculation module configured to calculate, for each cluster number value in the candidate cluster number range, a corresponding average silhouette coefficient; an optimal cluster number selection module configured to select a cluster number value that maximizes the average silhouette coefficient as the optimal clustering cluster number.

[0155] In an embodiment of the present application, the first cleaning module includes a KMeans clustering execution module and a first type of outlier identification module, wherein the first outlier identification module includes: a centroid calculation module configured to calculate a centroid of each cluster after the first clustering algorithm is completed; an intra-cluster distance calculation module configured to calculate distances from each data point in a cluster to a corresponding centroid of the cluster to form an intra-cluster distance set; a threshold setting module configured to set an anomaly detection threshold based on a mean and a standard deviation of the intra-cluster distance set; a first outlier identification module configured to identify a data point as a first type of outlier if a distance from the data point to a corresponding centroid exceeds the anomaly detection threshold.

[0156] In an embodiment of the present application, the threshold setting module is specifically configured to: calculate a skewness of the intra-cluster distance set, and adjust a weight factor used to calculate the anomaly detection threshold according to an absolute value of the skewness, thereby dynamically adjusting the anomaly detection threshold.

[0157] In an embodiment of the present application, the first outlier identification module further includes: a LOF value calculation module configured to calculate a local outlier factor (LOF) value for each data point; a final determination module configured to finally determine a data point as a first type of outlier if the distance from the data point to a corresponding centroid exceeds the anomaly detection threshold and the LOF value of the data point exceeds a preset LOF threshold.

[0158] In an embodiment of the present application, the KMeans clustering execution module specifically includes: a centroid initialization module configured to initialize a plurality of cluster centroids based on the optimal clustering cluster number; a data point assignment module configured to assign each data point to a cluster corresponding to a cluster centroid closest to the data point in terms of Euclidean distance; a centroid update module configured to recalculate a centroid of each cluster and update the centroid as a new cluster centroid; An iteration control module is configured to repeat the above steps until the allocation of the clusters no longer changes or a preset maximum number of iterations is reached.

[0159] In an embodiment of the present application, the second cleaning module specifically includes: A minimum point number setting module is configured to set a minimum point number parameter based on the data dimension; A neighborhood radius determination module is configured to adaptively determine a neighborhood radius parameter by calculating a k-distance ranking curve and identifying an inflection point of the curve; A traversal elimination module is configured to eliminate all data points after the first type of abnormal points based on the dynamically determined neighborhood radius and minimum point number parameters; A core point identification module is configured to identify a point as a core point if the number of data points contained in the neighborhood of the point is not less than the minimum point number; A cluster formation module is configured to form a cluster from any core point through density reachable relationship; A second type of abnormal point identification module is configured to identify a data point that does not belong to any cluster and is neither a core point nor a boundary point as a noise point of the second type of abnormal point.

[0160] In an embodiment of the present application, the data processing module includes: A preliminary screening module is configured to preliminarily screen extreme outliers by using an IQR-based method; A data standardization module is configured to normalize coordinate data by using a z-score standardization method; A format verification and conversion module is configured to perform format verification and conversion on the data to generate a structured numerical array; A deduplication processing module is configured to perform deduplication processing on the data to eliminate duplicate coordinate points.

[0161] In an embodiment of the present application, the result generation module is specifically configured to: According to a preset mode, all identified abnormal points are eliminated from the original data or are marked as abnormal on the basis of retaining the original data.

[0162] Each module in the above-described industrial key point data cleaning and abnormal point elimination device can be implemented wholly or partially by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0163] Reference Figure 17As shown, the embodiment of the present application further provides a computer device, which comprises a memory 802, a processor 801, and a computer program 8021 stored in the memory 802 and capable of running on the processor 801, and the processor 801 implements the method for industrial key point data cleaning and abnormal point elimination as described above when running the computer program 8021.

[0164] The embodiment of the present application further provides a computer storage medium, which stores the computer program 8021, and the program is executed by the processor to implement the method for industrial key point data cleaning and abnormal point elimination as described above.

[0165] It should be noted that each embodiment in the present specification is described in a progressive manner, and each embodiment mainly describes the difference from other embodiments, and the same parts between each embodiment can be referred to each other. For the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0166] It should also be noted that the terms such as first and second in the present specification are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0167] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0168] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments described herein but is intended to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for cleaning industrial key point data and eliminating abnormal points, characterized in that: include: Acquiring coordinate data of industrial key points to be processed and preprocessing the coordinate data; Dynamically determine the optimal number of clusters for preprocessed coordinate data; Based on the optimal number of clusters, clustering the coordinate data using a first clustering algorithm, and identifying first-type outliers whose distance from the centroid of the cluster to which they belong exceeds a first dynamic threshold according to statistical characteristics of distance distribution within each cluster; After removing the first type of outliers, the data is cleaned again using a second density-based clustering algorithm to identify the second type of outliers that are determined to be noise points based on the local density of the data points. The first and second types of outliers are integrated to generate a cleaned data set.

2. The method according to claim 1, characterized in that The dynamically determining the optimal number of clusters of the pre-processed coordinate data includes: Within the preset initial cluster number range, for each cluster value, calculate the sum of the squares of the Euclidean distances of all points in the corresponding cluster to the center of the corresponding cluster; Based on the change rate of the sum of squares of the Euclidean distances within the cluster, an elbow point is determined using the elbow rule, and a range of candidate cluster numbers is determined with the elbow point as the center; Within the candidate cluster number range, for each cluster value, calculate the corresponding average silhouette coefficient; The cluster value that maximizes the average silhouette coefficient is selected as the optimal cluster number.

3. The method according to claim 1, characterized in that Identifying the first type of outliers based on the statistical characteristics of the distance distribution within each cluster includes: After the first clustering algorithm is completed, the centroid of each cluster is calculated; Calculate the distance between each data point in each cluster and its corresponding centroid to form an intra-cluster distance set; Setting an anomaly detection threshold based on the mean and standard deviation of the intra-cluster distance set; Data points whose distance from the corresponding centroid exceeds the anomaly detection threshold are identified as first-category anomalies.

4. The method according to claim 3, characterized in that The setting of the anomaly detection threshold further includes: The skewness of the intra-cluster distance set is calculated, and according to the absolute value of the skewness, a weight factor used to calculate the anomaly detection threshold is adjusted, thereby dynamically adjusting the anomaly detection threshold.

5. The method according to claim 3 or 4, characterized in that The identifying of the first type of abnormal points further includes: Calculate the local outlier factor LOF value for each data point; The data points whose distance from the corresponding centroid exceeds the anomaly detection threshold and whose LOF value exceeds the preset LOF threshold at the same time are finally determined to be the first type of anomaly points.

6. The method according to claim 1, characterized in that The first clustering algorithm is the K-Means algorithm, which specifically includes: Initializing a plurality of cluster centroids based on the optimal number of clusters; Assign each data point to the cluster corresponding to the centroid of the cluster with the closest Euclidean distance; Recalculate the centroid of each cluster and update it to the new cluster centroid; Repeat the above steps until the cluster assignment no longer changes or the preset maximum number of iterations is reached.

7. The method according to claim 1, characterized in that The second clustering algorithm is the DBSCAN algorithm, whose parameters are dynamically determined, specifically including: Set the minimum number of points parameter based on the data dimension; By calculating the k-distance sorting curve and identifying the inflection point of the curve, the neighborhood radius parameter is adaptively determined; Based on the dynamically determined neighborhood radius and minimum number of points parameters, traverse all data points after removing the first type of outliers; Identify points in the neighborhood that contain no less than the minimum number of data points as core points; Starting from any core point, a cluster is formed through density reachability relations; Data points that do not belong to any cluster and are neither core points nor boundary points are identified as noise points serving as the second type of abnormal points.

8. The method according to claim 1, characterized in that The pre-processing step comprises: The interquartile range (IQR)-based method was used to preliminarily screen out extreme outliers; The coordinate data were normalized using the z-score standardization method; Perform format verification and conversion on the data to generate a structured numerical array; Also, deduplication is performed on the data to remove duplicate coordinate points.

9. The method according to claim 1, characterized in that Generating a cleaned data set includes: According to the preset mode, all identified abnormal points are removed from the original data, or marked as abnormal while retaining the original data.

10. A device for cleaning industrial key point data and eliminating abnormal points, characterized in that: include: The data processing module is used to obtain the coordinate data of the industrial key points to be processed and pre-process the coordinate data; A cluster number determination module is used to dynamically determine the optimal cluster number of the preprocessed coordinate data; a first cleaning module, configured to cluster the coordinate data using a first clustering algorithm based on the optimal number of clusters, and identify first-type outliers whose distance from the centroid of the cluster to which they belong exceeds a first dynamic threshold according to statistical characteristics of distance distribution within each cluster; a second cleaning module, configured to perform a secondary cleaning on the data after removing the first type of outliers, using a second density-based clustering algorithm, to identify the second type of outliers that are determined to be noise points based on the local density of the data points; The result generation module is used to integrate the first type of outliers and the second type of outliers to generate a cleaned data set.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for cleaning industrial key point data and eliminating abnormal points as described in any one of claims 1 to 9 is implemented.

12. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for cleaning industrial key point data and eliminating abnormal points as described in any one of claims 1 to 9 is implemented.

Images