Noise data identification method, apparatus and device, and computer readable storage medium
By removing training data exceeding the threshold in the HDBSCAN data clustering method and verifying it using the isolated forest algorithm, the problem of misjudgment of noise points and boundary points is solved, and the accuracy of data clustering is improved.
Patent Information
- Application Number
- CN202511588588.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-01-20
AI Technical Summary
Existing HDBSCAN-based data clustering methods suffer from accuracy issues when dealing with noisy points and boundary points, leading to distorted cluster structures and poor prediction results.
By obtaining the k-th nearest neighbor distance of the initial training data, data exceeding a preset threshold are removed. The initial noise data is then trained and identified using a clustering model, and a secondary verification is performed using the isolated forest algorithm to ensure that the identified noise data is real noise.
It achieves more accurate and effective noise data identification, improves the accuracy of clustering models and subsequent data clustering processing, and avoids misjudgment of boundary points.
Smart Images
Figure CN121365249A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, in particular to a noise data identification method and device, electronic equipment and computer readable storage medium. BACKGROUND
[0002] HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is a density-based hierarchical clustering algorithm that combines the advantages of hierarchical clustering and HDBSCAN density clustering to achieve automatic cluster number discovery and noise point identification. Since HDBSCAN is unsupervised learning, it does not require manual labeling, which can greatly save labor costs. Therefore, in the field of automobile diagnosis or industrial control, HDBSCAN clustering algorithm is often used for unsupervised learning to classify data, such as classifying the features of the paired relationship of CAN ID.
[0003] However, the single HDBSCAN method has the following disadvantages: on the one hand, the training data used to implement HDBSCAN model training will inevitably have outliers, i.e. noise points (noise data), and the existence of outliers will inevitably distort the cluster structure (such as affecting core distance calculation), thereby affecting the subsequent prediction effect; on the other hand, for boundary points, i.e. low-density points located at the edge of the cluster, there is also a possibility of being misjudged as noise points, which will also affect the subsequent prediction effect. As can be seen, the existing HDBSCAN-based data clustering processing method cannot effectively guarantee its accuracy.
[0004] Therefore, how to achieve more accurate and effective noise data identification in clustered data and further guarantee the accuracy of subsequent data clustering processing is a problem that needs to be solved by those skilled in the art. SUMMARY
[0005] The purpose of the present application is to provide a noise data identification method that can achieve more accurate and effective noise data identification in clustered data and further guarantee the accuracy of subsequent data clustering processing. Another purpose of the present application is to provide a noise data identification device, electronic equipment and computer readable storage medium, all of which have the above-mentioned beneficial effects.
[0006] In a first aspect, the present application discloses a noise data identification method, comprising:
[0007] Obtaining initial training data and determining the kth nearest neighbor distance corresponding to each of the initial training data;
[0008] The initial training data with the kth nearest neighbor distance exceeding the preset threshold is removed to obtain real training data;
[0009] A clustering model is trained using the real training data to obtain the clustering model;
[0010] The target data is identified using the clustering model to obtain initial noise data;
[0011] The initial noise data is verified using an isolation forest algorithm to determine whether the initial noise data is real noise data.
[0012] Optionally, removing the initial training data with the kth nearest neighbor distance exceeding the preset threshold to obtain real training data includes:
[0013] A kth nearest neighbor distance statistic is calculated according to the kth nearest neighbor distance corresponding to all the initial training data; wherein the kth nearest neighbor distance statistic includes a kth nearest neighbor distance median and a kth nearest neighbor distance standard deviation;
[0014] The preset threshold is calculated according to the kth nearest neighbor distance statistic;
[0015] The initial training data with the kth nearest neighbor distance exceeding the preset threshold is removed to obtain the real training data.
[0016] Optionally, before identifying the target data using the clustering model to obtain initial noise data, the method further includes:
[0017] A noise knowledge base is constructed based on the initial training data with the kth nearest neighbor distance exceeding the preset threshold;
[0018] When the target data hits the noise knowledge base, the target data is determined to be the real noise data;
[0019] When the target data does not hit the noise knowledge base, the step of identifying the target data using the clustering model to obtain initial noise data is performed.
[0020] Optionally, the noise data identification method further includes:
[0021] The first distance between the target data and each noise data in the noise knowledge base is calculated;
[0022] The first distance minimum value is determined among all the first distances;
[0023] The target noise data corresponding to the first distance minimum value is determined;
[0024] determine whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data.
[0025] Optionally, the determining whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data comprises:
[0026] calculating an average distance of all the first distances, and determining a first threshold according to the average distance;
[0027] when the first distance minimum value is lower than the first threshold, determining that the target data is similar to the target noise data, and determining that the target data hits the noise knowledge base;
[0028] when the first distance minimum value is not lower than the first threshold, determining that the target data is not similar to the target noise data, and determining that the target data does not hit the noise knowledge base.
[0029] Optionally, the verifying the initial noise data by using the isolation forest algorithm to determine whether the initial noise data is real noise data comprises:
[0030] verifying the initial noise data by using the isolation forest algorithm to obtain an anomaly score of the initial noise data;
[0031] when the anomaly score is greater than a preset score, determining that the initial noise data is the real noise data;
[0032] when the anomaly score is not greater than the preset score, determining that the initial noise data is not the real noise data.
[0033] Optionally, when the anomaly score is not greater than the preset score, determining that the initial noise data is not the real noise data comprises:
[0034] when the anomaly score is not greater than the preset score, calculating a second distance between the initial noise data and each cluster center point in the clustering model;
[0035] determining a second distance minimum value in all the second distances;
[0036] when the second distance minimum value is less than a second threshold, determining that the initial noise data is not the real noise data;
[0037] when the second distance minimum value is not less than the second threshold, determining that the initial noise data is the real noise data.
[0038] In a second aspect, the present application discloses a noise data identification device, comprising:
[0039] determining a kth nearest neighbor distance corresponding to each of the initial training data;
[0040] rejecting the initial training data whose kth nearest neighbor distance exceeds a preset threshold, to obtain real training data;
[0041] training a model using the real training data, to obtain a clustering model;
[0042] identifying target data using the clustering model, to obtain initial noise data;
[0043] verifying the initial noise data using an isolation forest algorithm, to determine whether the initial noise data is real noise data.
[0044] In a third aspect, the present application discloses an electronic device, comprising:
[0045] a memory for storing a computer program;
[0046] a processor for executing the computer program to implement the steps of any of the noise data identification methods described above.
[0047] In a fourth aspect, the present application discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any of the noise data identification methods described above.
[0048] The present application provides a noise data identification method, comprising: obtaining initial training data, and determining a kth nearest neighbor distance corresponding to each of the initial training data; rejecting the initial training data whose kth nearest neighbor distance exceeds a preset threshold, to obtain real training data; training a model using the real training data, to obtain a clustering model; identifying target data using the clustering model, to obtain initial noise data; and verifying the initial noise data using an isolation forest algorithm, to determine whether the initial noise data is real noise data.
[0049] With the technical scheme provided in the application, in the forward model training process, the initial training data is subjected to potential noise identification, so as to eliminate the noise data in the initial training data, obtain real training data not containing noise data, and use the real training data to implement clustering model training, so as to effectively ensure the model precision of the clustering model; in the backward model prediction process, the initial noise data identified based on the clustering model is further verified by using the isolation forest algorithm, so as to determine whether the initial noise data is real noise data, and effectively avoid the problem of misjudgment of boundary points. As can be seen, the technical scheme combines forward noise filtering with backward noise verification, realizes a bidirectional noise processing mechanism, can realize more accurate and effective noise data identification in clustering data, and further ensures the accuracy of subsequent data clustering processing.
[0050] The noise data identification device, the electronic device and the computer readable storage medium provided in the application also have the technical effects described above, and the application will not be described here. BRIEF DESCRIPTION OF DRAWINGS
[0051] In order to more clearly illustrate the technical schemes in the prior art and the embodiments of the application, the drawings needed to be used in the description of the prior art and the embodiments of the application will be briefly introduced below. Of course, the following description of the drawings of the embodiments of the application is only a part of the embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor, and the obtained other drawings also belong to the protection scope of the application.
[0052] Figure 1 A flowchart of a noise data identification method provided by an embodiment of the application;
[0053] Figure 2 A flowchart of a forward noise filtering method provided by an embodiment of the application;
[0054] Figure 3 A flowchart of a backward noise correction method provided by an embodiment of the application;
[0055] Figure 4 A structure diagram of a noise data identification device provided by an embodiment of the application;
[0056] Figure 5 A structure diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION
[0057] The core of the present application is to provide a noise data identification method which can achieve more accurate and effective noise data identification in clustered data, further ensuring the accuracy of subsequent data clustering processing. Another core of the present application is to provide a noise data identification device, an electronic device and a computer readable storage medium, all of which have the above beneficial effects.
[0058] In order to more clearly and completely describe the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0059] Please refer to Figure 1 , Figure 1 A flowchart of a noise data identification method provided by the embodiments of the present application can include but is not limited to S101-S105.
[0060] S101: Obtain initial training data and determine the kth nearest neighbor distance corresponding to each initial training data.
[0061] This step aims to achieve the acquisition of initial training data and the determination of the kth nearest neighbor distance corresponding to each initial training data. The initial training data is the original training data used to realize the training of the clustering model. After obtaining the initial training data, the kth nearest neighbor distance corresponding to each initial training data can be determined respectively. In the specific implementation process, determining the kth nearest neighbor distance corresponding to each initial training data can include: determining the k nearest neighbors corresponding to each initial training data; for each initial training data, calculating the distance between the initial training data and the k nearest neighbors to obtain k nearest neighbor distances; taking the maximum nearest neighbor distance in the k nearest neighbor distances as the kth nearest neighbor distance corresponding to the initial training data.
[0062] S102: Remove the initial training data whose kth nearest neighbor distance exceeds the preset threshold to obtain the real training data.
[0063] The step is to effectively eliminate the noise data in the initial training data, so as to effectively ensure the accuracy of the subsequent model. Specifically, when the kth nearest neighbor distance corresponding to the initial noise data exceeds the preset threshold, it indicates that the initial noise data is far away from the cluster, that is, the initial noise data is noise data, which can be eliminated; when the kth nearest neighbor distance corresponding to the initial noise data does not exceed the preset threshold, it indicates that the initial noise data is close to the cluster, that is, the initial noise data is not noise data, which can be retained. Thus, the noise data in the initial training data is effectively eliminated, thereby effectively avoiding the adverse effects on model training due to the existence of noise data in the model training data, improving the model accuracy, and thereby ensuring the accuracy of the model processing result.
[0064] In an embodiment of the present application, the initial training data with the kth nearest neighbor distance exceeding the preset threshold is eliminated to obtain the real training data, which can include: calculating the kth nearest neighbor distance statistics from the kth nearest neighbor distance corresponding to all initial training data; wherein the kth nearest neighbor distance statistics include the kth nearest neighbor distance median and the kth nearest neighbor distance standard deviation; calculating the preset threshold from the kth nearest neighbor distance statistics; eliminating the initial training data with the kth nearest neighbor distance exceeding the preset threshold to obtain the real training data.
[0065] In order to achieve more accurate noise data recognition in the initial training data, the present application provides a method for calculating the preset threshold, so as to find the most suitable preset threshold to filter the initial training data. Specifically, the median and standard deviation of all kth nearest neighbor distances can be calculated first, then the preset threshold can be calculated according to the median and standard deviation, and finally the initial training data with the kth nearest neighbor distance exceeding the preset threshold can be eliminated, that is, the real training data can be obtained.
[0066] In a possible implementation, the preset threshold = median + 3 x standard deviation. It can be understood that the median represents the "center" or "typical" level of the data set, and the standard deviation represents the "average degree" of all data points deviating from the average value. Therefore, the normal points (non-noise points) should generally be around the median and fluctuate within the range of the standard deviation. On this basis, when the kth nearest neighbor distance corresponding to the initial training data is much larger than "median + standard deviation", the initial training data is likely to be noise data. It should be pointed out that "3" in the formula is the standard deviation multiple, which is only an example here, and its specific value can be customized according to actual needs.
[0067] S103: training the model using the real training data to obtain a clustering model.
[0068] S104: identifying the target data using the clustering model to obtain the initial noise data.
[0069] The above steps are aimed at realizing the training of the clustering model and the initial noise recognition based on the clustering model. Specifically, based on S101 and S102, the model training data not containing noise data, i.e., the real training data, can be obtained, and the clustering model trained based on the real training data can effectively guarantee the model precision; further, the target data requiring noise recognition can be identified by using the clustering model, i.e., the initial noise data determined as "noise" by the clustering model can be obtained.
[0070] As described above, for the boundary points, i.e., the low-density points located at the edge of the cluster, there is also a possibility of being misjudged as noise points by the clustering model, which will also affect the subsequent prediction effect. Based on this, for the initial noise data determined by the clustering model, subsequent S105 can be further performed to verify it again, so as to effectively guarantee the accuracy of the noise recognition result.
[0071] S105: verifying the initial noise data by using the Isolation Forest algorithm to determine whether the initial noise data is real noise data.
[0072] The above steps are aimed at realizing the training of the clustering model and the initial noise recognition based on the clustering model. Specifically, based on S101 and S102, the model training data not containing noise data, i.e., the real training data, can be obtained, and the clustering model trained based on the real training data can effectively guarantee the model precision; further, the target data requiring noise recognition can be identified by using the clustering model, i.e., the initial noise data determined as "noise" by the clustering model can be obtained.
[0073] Further, when the anomaly score is not greater than the preset score, it can be determined that the initial noise data is not real noise data, which can include: when the anomaly score is not greater than the preset score, the second distance between the initial noise data and each cluster center point in the clustering model is calculated; the minimum value of all second distances is determined; when the minimum value of the second distance is less than the second threshold, it is determined that the initial noise data is not real noise data; and when the minimum value of the second distance is not less than the second threshold, it is determined that the initial noise data is real noise data.
[0074] In the embodiments of the present application, in order to further improve the accuracy of the noise recognition result, the initial noise data whose abnormal score does not exceed the preset score can also be verified three times. Specifically, for each initial noise data whose abnormal score does not exceed the preset score, the distance between the initial noise data and each cluster center point in the clustering model, i.e., the second distance, can be calculated, so as to obtain a second distance minimum value from all the second distances. Obviously, the second distance minimum value indicates that the current initial noise data is closest to the cluster corresponding to the second distance minimum value. Then, if the second distance minimum value is less than a second threshold, it can be determined that the current initial noise data is not real noise data; if the second distance minimum value is not less than the second threshold, it can be determined that the current initial noise data is real noise data.
[0075] In a possible implementation, the second threshold = cluster diameter of the cluster corresponding to the second distance minimum value × preset multiple, wherein the specific value of the preset multiple can be customized according to actual requirements, and the present application does not limit this, for example, the value can be 0.6. On this basis, when the second distance minimum value is less than the second threshold, it indicates that the current initial noise data is located inside the cluster corresponding to the second distance minimum value, that is, the current initial noise data is normal data rather than noise data. At this time, the cluster corresponding to the second distance minimum value can also be assigned a label for the current initial noise data, so as to realize clustering recognition.
[0076] It can be seen that the noise data recognition method provided in the embodiments of the present application performs potential noise recognition on the initial training data in the forward model training process, so as to eliminate the noise data in the initial training data and obtain real training data not containing noise data for implementing clustering model training, so as to effectively ensure the model precision of the clustering model. In the backward model prediction process, the initial noise data recognized based on the clustering model is further verified by using the isolation forest algorithm, so as to determine whether the initial noise data is real noise data, thereby effectively avoiding the misjudgment problem of the boundary point. It can be seen that the present technical solution combines forward noise filtering and backward noise verification, realizes a bidirectional noise processing mechanism, can realize more accurate and effective noise data recognition in the clustering data, and further ensures the accuracy of subsequent data clustering processing.
[0077] On the basis of the above-mentioned embodiments, in an embodiment of the present application, before the target data is identified by using the clustering model to obtain the initial noise data, the following steps can also be included:
[0078] Constructing a noise knowledge base based on the initial training data whose kth nearest neighbor distance exceeds the preset threshold;
[0079] When the target data hits the noise knowledge base, determining that the target data is real noise data;
[0080] When the target data does not hit the noise knowledge base, a step of identifying the target data by using a clustering model to obtain initial noise data is performed.
[0081] To effectively improve the noise recognition efficiency, an embodiment of the present application proposes a noise data initial filtering scheme based on a noise knowledge base. Specifically, before calling a clustering model to identify target data, the target data can be matched with the noise knowledge base. If the target data hits the noise knowledge base, subsequent model identification is not needed, and the target data can be directly determined as real noise data. If the target data does not hit the noise knowledge base, subsequent model identification is continued. The noise knowledge base can be obtained based on the initial training data whose kth nearest neighbor distance exceeds a preset threshold, that is, the noise knowledge base is constructed directly by using the model training data determined as noise data.
[0082] The process of determining whether the target data hits the noise knowledge base can include: calculating a first distance between the target data and each noise data in the noise knowledge base; determining a first distance minimum value among all first distances; determining a target noise data corresponding to the first distance minimum value; and determining whether the target data hits the noise knowledge base according to a similarity between the target data and the target noise data.
[0083] In an embodiment of the present application, how to determine whether the target data hits the noise knowledge base can be realized by distance calculation and similarity calculation between data. First, a target noise data closest to the target data (i.e., the first distance minimum value) is found in the noise knowledge base, because the two are closest, they are most similar. Further, whether the target data hits the noise knowledge base is determined by similarity calculation between the two.
[0084] In the specific implementation process, whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data can include: calculating an average distance of all first distances, and determining a first threshold according to the average distance; when the first distance minimum value is lower than the first threshold, it is determined that the target data is similar to the target noise data, and it is determined that the target data hits the noise knowledge base; when the first distance minimum value is not lower than the first threshold, it is determined that the target data is not similar to the target noise data, and it is determined that the target data does not hit the noise knowledge base. Obviously, the smaller the distance between the target data and the target noise data, the more similar they are. The first threshold can be a preset multiple of the average distance. Similarly, the specific value of the preset multiple can be customized according to actual needs, and the present application does not limit it, for example, it can be 1.5.
[0085] Exemplarily, an embodiment of the present application provides a noise data recognition method.
[0086] First, please refer to Figure 2 , Figure 2 The flowchart of a forward noise filtering method provided by the embodiments of the present application is shown in the figure, and the implementation process is as follows:
[0087] 1. Obtain the original training data input pre_noise_filter (noise filter, the core idea of which is that noise points are usually far away from dense areas, and the distance to the kth nearest neighbor will be significantly greater than that of normal points) to perform global outlier identification:
[0088] (1) Calculate the distance of each point (original training data) to its k nearest neighbors, obtain the nearest neighbor list of each point, and sort the k nearest neighbor distances in the nearest neighbor list in ascending order;
[0089] (2) Extract the kth nearest neighbor distance of each point (i.e., the last distance in the nearest neighbor list);
[0090] (3) Calculate the statistics of the kth nearest neighbor distances of all points, including the median and the standard deviation;
[0091] (4) Calculate the preset threshold: threshold = median + 3 × std, median is the median, std is the standard deviation, and points exceeding the preset threshold are regarded as noise points;
[0092] (5) Return the noise mask.
[0093] 2. Use non-noise points to enter HDBSCAN training to obtain an HDBSCAN clustering model and save it.
[0094] 3. Isolate and store the noise points independently to obtain a noise knowledge base.
[0095] Further, please refer to Figure 3 , Figure 3 The flowchart of a backward noise correction method provided by the embodiments of the present application is shown in the figure, and the implementation process is as follows:
[0096] 1. Newly obtain target data and start matching known global noise patterns, the core idea of which is to calculate the distance between the measured data and the known noise (noise data in the noise knowledge base), and find the nearest noise point, if the distance is less than the threshold, it is determined as noise:
[0097] (1) Calculate the distance of the target noise to all known noise points in the noise knowledge base;
[0098] (2) Select the minimum distance;
[0099] (3) judging the similarity of the target data and the known noise point corresponding to the minimum distance, the judgment condition being that the minimum distance is less than 1.5 times of the average distance of the noise;
[0100] (4) if the similarity is determined as noise data, that is, the target data hits the noise knowledge base, otherwise, the target data does not hit the noise knowledge base.
[0101] 2) for the target data that does not hit the noise knowledge base, the HDBSCAN clustering model is used to identify the target data to obtain the determination result of the target data; for the target data that hits the noise knowledge base, the target data can be directly determined as noise data.
[0102] 3) for the target data determined as non-noise data by the HDBSCAN clustering model, the corresponding clustering label of the target data can be directly outputted.
[0103] 4) for the target data determined as noise data by the HDBSCAN clustering model, the isolated forest algorithm is used to verify the target data.
[0104] 5) for the target data verified by the isolated forest algorithm and having an abnormal score greater than a threshold value, the target data can be directly determined as noise data.
[0105] 6) for the target data verified by the isolated forest algorithm and having an abnormal score not greater than a threshold value, the distance between the target data and each clustering center point is further calculated and threshold value judgment is performed, so as to determine whether the target data is noise data.
[0106] It can be seen that the noise data recognition method provided by the embodiment of the present application performs potential noise recognition on the initial training data in the forward model training process, so as to eliminate the noise data in the initial training data and obtain real training data not containing noise data for implementing clustering model training, so as to effectively ensure the model precision of the clustering model; in the backward model prediction process, the isolated forest algorithm is used to further verify the initial noise data recognized based on the clustering model, so as to determine whether the initial noise data is real noise data, effectively avoiding the misjudgment problem of the boundary point. It can be seen that the technical solution combines forward noise filtering and backward noise verification, realizes a bidirectional noise processing mechanism, can realize more accurate and effective noise data recognition in the clustering data, and further ensures the accuracy of subsequent data clustering processing.
[0107] The embodiment of the present application provides a noise data recognition device.
[0108] Please refer to Figure 4 , Figure 4 The structure diagram of the noise data recognition device provided by the embodiment of the present application can include:
[0109] A determining module 1 is configured to obtain initial training data and determine a kth nearest neighbor distance corresponding to each initial training data.
[0110] A removing module 2 is configured to remove the initial training data whose kth nearest neighbor distance exceeds a preset threshold to obtain real training data.
[0111] A training module 3 is configured to train a model by using the real training data to obtain a clustering model.
[0112] A recognizing module 4 is configured to recognize target data by using the clustering model to obtain initial noise data.
[0113] A verifying module 5 is configured to verify the initial noise data by using an isolation forest algorithm to determine whether the initial noise data is real noise data.
[0114] It can be seen that the noise data recognition device provided in the embodiments of the present application performs potential noise recognition on the initial training data in the forward model training process, so as to remove the noise data in the initial training data to obtain real training data that does not contain noise data for implementing clustering model training, thereby effectively ensuring the model precision of the clustering model. In the backward model prediction process, the initial noise data recognized based on the clustering model is further verified by using the isolation forest algorithm to determine whether the initial noise data is real noise data, thereby effectively avoiding the misjudgment problem of the boundary point. It can be seen that the present technical solution combines forward noise filtering with backward noise verification to implement a bidirectional noise processing mechanism, which can achieve more accurate and effective noise data recognition in the clustering data and further ensure the accuracy of subsequent data clustering processing.
[0115] In an embodiment of the present application, the removing module 2 can be specifically configured to calculate a kth nearest neighbor distance statistic quantity based on the kth nearest neighbor distances corresponding to all initial training data, wherein the kth nearest neighbor distance statistic quantity includes a kth nearest neighbor distance median and a kth nearest neighbor distance standard deviation; calculate the preset threshold based on the kth nearest neighbor distance statistic quantity; and remove the initial training data whose kth nearest neighbor distance exceeds the preset threshold to obtain the real training data.
[0116] In an embodiment of the present application, the noise data recognition device can further include an initial recognizing module configured to, before recognizing the target data by using the clustering model to obtain the initial noise data, construct a noise knowledge base based on the initial training data whose kth nearest neighbor distance exceeds the preset threshold; when the target data hits the noise knowledge base, determine that the target data is real noise data; and when the target data does not hit the noise knowledge base, perform the step of recognizing the target data by using the clustering model to obtain the initial noise data.
[0117] In an embodiment of the present application, the noise data identification apparatus can further comprise a judging module configured to calculate the first distance between the target data and each noise data in the noise knowledge base; determine the minimum value of the first distance among all the first distances; determine the target noise data corresponding to the minimum value of the first distance; and determine whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data.
[0118] In an embodiment of the present application, the judging module can be specifically configured to calculate the average distance of all the first distances, and determine the first threshold value according to the average distance; when the minimum value of the first distance is lower than the first threshold value, determine that the target data is similar to the target noise data, and determine that the target data hits the noise knowledge base; and when the minimum value of the first distance is not lower than the first threshold value, determine that the target data is not similar to the target noise data, and determine that the target data does not hit the noise knowledge base.
[0119] In an embodiment of the present application, the verifying module 5 can be specifically configured to verify the initial noise data by using the isolation forest algorithm to obtain the anomaly score of the initial noise data; when the anomaly score is greater than a preset score, determine that the initial noise data is the real noise data; and when the anomaly score is not greater than the preset score, determine that the initial noise data is not the real noise data.
[0120] In an embodiment of the present application, the verifying module 5 can be specifically configured to, when the anomaly score is not greater than the preset score, calculate the second distance between the initial noise data and each cluster center point in the clustering model; determine the minimum value of the second distance among all the second distances; when the minimum value of the second distance is less than a second threshold value, determine that the initial noise data is not the real noise data; and when the minimum value of the second distance is not less than the second threshold value, determine that the initial noise data is the real noise data.
[0121] The apparatus provided in the embodiments of the present application can refer to the above method embodiments, and the present application will not be described here in detail.
[0122] The present application further provides an electronic device, which can refer to Figure 5 , Figure 5 The present application further provides an electronic device, which can refer to
[0123] The present application further provides an electronic device, which can refer to
[0124] The present application further provides an electronic device, which can refer to
[0125] The present application further provides an electronic device, which can refer to Figure 5As shown in the figure, it is a schematic diagram of a structural composition of the electronic device, and the electronic device can include a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 can complete communication with each other through the communication bus 13.
[0126] In the embodiment of the present application, the processor 10 can be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
[0127] The processor 10 can call a program stored in the memory 11, and specifically, the processor 10 can perform the operations in the embodiments of the noise data identification method.
[0128] The memory 11 is used to store one or more programs, and the program can include program code including computer operation instructions. In the embodiment of the present application, the memory 11 at least stores a program for implementing the following functions:
[0129] Obtaining initial training data and determining the kth nearest neighbor distance corresponding to each initial training data; removing the initial training data whose kth nearest neighbor distance exceeds a preset threshold to obtain real training data; training a model using the real training data to obtain a clustering model; identifying target data using the clustering model to obtain initial noise data; and verifying the initial noise data using an isolated forest algorithm to determine whether the initial noise data is real noise data.
[0130] In a possible implementation, the memory 11 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function; and the data storage area can store data created during use.
[0131] In addition, the memory 11 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device or other volatile solid-state storage device.
[0132] The communication interface 12 can be an interface of a communication module, used to connect with other devices or systems.
[0133] Of course, it needs to be explained that, Figure 5 The structure shown does not constitute a limitation on the electronic device in the embodiments of the present application, and the electronic device can include more or fewer components than those shown in the figure, or combine some components in actual application. Figure 5 The structure shown does not constitute a limitation on the electronic device in the embodiments of the present application, and the electronic device can include more or fewer components than those shown in the figure, or combine some components in actual application.
[0134] The application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of any one of the noise data identification methods.
[0135] The computer readable storage medium can include a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.
[0136] The computer readable storage medium provided by the application is described above. Here, the application does not make redundant description.
[0137] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.
[0138] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example are described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.
[0139] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a Random Access Memory (RAM), a memory, a Read-Only Memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art.
[0140] The technical solutions provided by the application are described in detail above. The principles and implementation manners of the application are described by using specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the application. It should be pointed out that, for those skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application. These improvements and modifications also fall within the protection scope of the application.
Claims
1. A method of identifying noisy data, characterized by, The method comprises the following steps: acquiring initial training data and determining the kth nearest neighbor distance corresponding to each of the initial training data; eliminating the initial training data whose kth nearest neighbor distance exceeds a preset threshold to obtain real training data; training a model using the real training data to obtain a clustering model; identifying target data using the clustering model to obtain initial noise data; verifying the initial noise data using an isolation forest algorithm to determine whether the initial noise data is real noise data.
2. The noise data identification method according to claim 1, characterized by, The method of eliminating the initial training data whose kth nearest neighbor distance exceeds a preset threshold to obtain real training data comprises the following steps: calculating a kth nearest neighbor distance statistic from the kth nearest neighbor distances corresponding to all the initial training data; wherein the kth nearest neighbor distance statistic comprises a kth nearest neighbor distance median and a kth nearest neighbor distance standard deviation; calculating the preset threshold from the kth nearest neighbor distance statistic; eliminating the initial training data whose kth nearest neighbor distance exceeds the preset threshold to obtain the real training data.
3. The noise data identification method according to claim 1 or 2, characterized by, Before the step of identifying target data using the clustering model to obtain initial noise data, the method further comprises the following steps: constructing a noise knowledge base based on the initial training data whose kth nearest neighbor distance exceeds the preset threshold; when the target data hits the noise knowledge base, determining that the target data is the real noise data; when the target data does not hit the noise knowledge base, performing the step of identifying target data using the clustering model to obtain initial noise data.
4. The noise data identification method according to claim 3, characterized by, The method further comprises the following steps: calculating a first distance between the target data and each noise data in the noise knowledge base; determining a first distance minimum value among all the first distances; determining a target noise data corresponding to the first distance minimum value; determining whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data.
5. The noise data identification method according to claim 4, characterized by, The method of determining whether the target data hits the noise knowledge base according to the similarity between the target data and the target noise data comprises the following steps: calculating an average distance of all the first distances and determining a first threshold according to the average distance; when the first distance minimum value is lower than the first threshold, determining that the target data is similar to the target noise data and that the target data hits the noise knowledge base; when the first distance minimum value is not lower than the first threshold, determining that the target data is not similar to the target noise data and that the target data does not hit the noise knowledge base.
6. The method of claim 1, wherein The method of verifying the initial noise data using an isolation forest algorithm to determine whether the initial noise data is real noise data comprises the following steps: verifying the initial noise data using an isolation forest algorithm to obtain an anomaly score of the initial noise data; when the anomaly score is greater than a preset score, determining that the initial noise data is the real noise data; when the anomaly score is not greater than the preset score, determining that the initial noise data is not the real noise data.
7. The noise data identification method according to claim 6, wherein When the abnormal score is not greater than the preset score, it is determined that the initial noise data is not the real noise data, comprising: When the abnormal score is not greater than the preset score, a second distance between the initial noise data and each cluster center point in the clustering model is calculated; A second distance minimum value is determined from all the second distances; When the second distance minimum value is less than a second threshold value, it is determined that the initial noise data is not the real noise data; When the second distance minimum value is not less than the second threshold value, it is determined that the initial noise data is the real noise data.
8. A noise data identifying apparatus characterized by comprising: Comprising: A determination module is configured to obtain initial training data and determine a kth nearest neighbor distance corresponding to each of the initial training data; An elimination module is configured to eliminate the initial training data whose kth nearest neighbor distance exceeds a preset threshold value, to obtain real training data; A training module is configured to train a model using the real training data, to obtain a clustering model; An identification module is configured to identify target data using the clustering model, to obtain initial noise data; A verification module is configured to verify the initial noise data using an isolation forest algorithm, to determine whether the initial noise data is real noise data.
9. An electronic device, comprising: Comprising: A memory is configured to store a computer program; A processor is configured to execute the computer program to implement the steps of the noise data identification method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and when executed by the processor, the computer program implements the steps of the noise data identification method according to any one of claims 1 to 7.