Abnormal data detection method, system, device, and medium based on K-nearest neighbor algorithm

By applying the K nearest neighbor algorithm in abnormal data detection, calculating the similarity between data and determining the categories of K neighbors, the complex and cumbersome verification rules in the prior art are solved, and efficient and general abnormal data detection is achieved.

CN114417962BActive Publication Date: 2025-04-08AEROSPACE SCI & ENG NETWORK INFORMATION DEV CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111491483.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-08
Publication Date
2025-04-08
Estimated Expiration
2041-12-08

AI Technical Summary

Technical Problem

The prior art requires the formulation of specific and complex verification rules in abnormal data detection, resulting in strong targeted rules, poor reusability, and inability to exhaust all abnormal situations, and the operation of configuring verification rules is cumbersome.

Method used

An abnormal data detection method based on K nearest neighbor algorithm is adopted, by randomly selecting unclassified data in the original data set to be detected, the similarity between it and the classified data is calculated, the classified data of the first K similarity is selected as the K neighbors of the unclassified data, and the categories of unclassified data are determined based on the categories of the K neighbors.

Benefits of technology

A general abnormal data detection method is implemented, which can be applied to abnormal data detection in all scenarios, improve detection efficiency, simplify verification rules, and eliminate the need to formulate specific and complex verification rules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114417962B_ABST
    Figure CN114417962B_ABST
Patent Text Reader

Abstract

The present invention discloses an abnormal data detection method, system, device, and medium based on a K-nearest neighbor algorithm, and relates to the technical field of abnormal data detection to solve the problem of poor reusability of existing detection methods. The detection method includes: calculating the similarity between unclassified data and each classified data in a classified data set based on a comprehensive verification rule, and determining the K neighbors of the unclassified data according to the similarity, determining the category of the unclassified data according to the category of the K neighbors, and then determining the category of each data in the original data set to be detected based on the similarity to determine all abnormal data and realize abnormal data detection. The method used in the present invention can be applied to abnormal data detection in all scenarios, and has strong versatility. The abnormal data detection method, system, device, and medium based on the K-nearest neighbor algorithm provided by the present invention are used to optimize the abnormal data detection method, and have strong reusability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal data detection, and in particular to an abnormal data detection method, system, device and medium based on a K-nearest neighbor algorithm. Background Art

[0002] With the advent of the data age, the demand for all industries to manage their own enterprise data through information technology is increasing day by day. After the early exploratory business system construction, many enterprises are increasingly aware that the construction of informatization itself is the construction and management of enterprise data. Data quality management, abnormal data detection and data standardization are the top priorities of enterprise data governance. Without good data quality management methods and technologies, it is difficult to promote production through data.

[0003] The existing data quality detection methods require the formulation of specific and complex verification rules for abnormal data detection. The rules are highly targeted, have poor reusability, and cannot exhaust all abnormal situations, which has defects. Based on this, a universal abnormal data detection method is urgently needed. Summary of the invention

[0004] The object of the present invention is to provide an abnormal data detection method, system, device and medium based on a K-nearest neighbor algorithm, which are used to determine abnormal data and have strong versatility.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for detecting abnormal data based on a K-nearest neighbor algorithm, the method comprising:

[0007] Determine a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute a classified data set;

[0008] Randomly select an unclassified data in the original data set to be detected, and calculate the similarity between the unclassified data and each classified data in the classified data set based on the comprehensive verification rule;

[0009] Sort the similarities in descending order, and select the classified data corresponding to the first K similarities as the K neighbors of the unclassified data;

[0010] Determine the category of the unclassified data according to the categories of the K neighbors to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data;

[0011] Determine whether there is unclassified data in the original data set to be detected;

[0012] If yes, return to the step of “randomly selecting an unclassified data in the original data set to be detected” until every unclassified data in the data set to be detected has been detected;

[0013] If not, abnormal data is determined based on the classified data set.

[0014] Compared with the prior art, the present invention provides an abnormal data detection method based on a K-nearest neighbor algorithm, which calculates the similarity between unclassified data and each classified data in a classified data set based on a comprehensive verification rule, and determines the K neighbors of the unclassified data according to the similarity, determines the category of the unclassified data according to the category of the K neighbors, and then determines the category of each data in the original data set to be detected based on the similarity, so as to determine all abnormal data and realize abnormal data detection. The method used in the present invention can be applied to abnormal data detection in all scenarios, and has strong versatility. In addition, the inspection rules used are simple, and there is no need to formulate specific and complex verification rules for abnormal data detection, and the detection efficiency is high.

[0015] An abnormal data detection system based on a K-nearest neighbor algorithm, the detection system comprising:

[0016] A classified data set determination module is used to determine a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute the classified data set;

[0017] A similarity calculation module, used for randomly selecting an unclassified data in the original data set to be detected, and calculating the similarity between the unclassified data and each classified data in the classified data set based on a comprehensive verification rule;

[0018] A neighbor determination module is used to sort the similarities from large to small, and select the classified data corresponding to the first K similarities as the K neighbors of the unclassified data;

[0019] A category determination module, used to determine the category of the unclassified data according to the categories of the K neighbors, to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data;

[0020] A judgment module, used to judge whether there is unclassified data in the original data set to be detected;

[0021] a return module, configured to, if yes, return to the step of "randomly selecting an unclassified data in the original data set to be detected" until each unclassified data in the data set to be detected has been detected;

[0022] The abnormal data determination module is used to determine the abnormal data according to the classified data set if no.

[0023] An abnormal data detection device based on a K-nearest neighbor algorithm, comprising:

[0024] Processor; and

[0025] a memory storing computer readable program instructions,

[0026] Wherein, the above detection method is executed when the computer-readable program instructions are executed by the processor.

[0027] A computer-readable storage medium stores a computer program, which implements the steps of the above detection method when executed by a processor.

[0028] Compared with the prior art, the beneficial effects of the detection system, equipment, and medium provided by the present invention are the same as the beneficial effects of the detection method described in the above technical solution, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 This is a schematic diagram of the process of the detection method provided in Example 1 of the present invention;

[0031] Figure 2 A schematic diagram of the K nearest neighbor algorithm provided in Example 1 of the present invention;

[0032] Figure 3 This is a system block diagram of the detection system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0033] In order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] Embodiment 1:

[0035] Many companies have followed the trend and developed their own data quality software products. For example, Neusoft Group's "DataQuality" provides nearly 100 field-level checks, 60 table-level checks, 40 inter-table rule checks and other rich verification models, as well as auxiliary functions such as automatic storage of erroneous data. Another example is the "Data Quality Management System" developed by Shenzhou Information, which improves the accuracy, timeliness and integrity of data by establishing a unified and standardized data quality operation mechanism, a comprehensive and complete abnormal data monitoring mechanism, a data monitoring mechanism based on process management and a complete data quality assessment mechanism. However, the abundance of data verification rules cannot completely identify abnormal data. This is because the real abnormal data situation is always more complicated than the verification rules, and no matter how many verification rules there are, it is impossible to exhaust all kinds of abnormal data. Moreover, the data quality effect of using many verification rules is not good. This is because each field is configured with many rules, which is a boring and tedious task. Once the configuration is insufficient, the verification will be inaccurate.

[0036] In view of the problem that the prior art must formulate specific and complex verification rules for abnormal data detection, the rules are highly targeted, have poor reusability and cannot cover all abnormal situations, and configure verification rules for many fields, which is cumbersome, time-consuming and labor-intensive. This embodiment provides a universal abnormal data detection method, which uses a k-nearest neighbor algorithm to solve the abnormal data detection problem. The method has strong versatility, and the verification rules used are simple, which can improve data quality.

[0037] This embodiment is used to provide an abnormal data detection method based on the K nearest neighbor algorithm. Figure 1 As shown, the detection method comprises:

[0038] S1: determining a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute a classified data set;

[0039] First, the original data set to be tested whose data quality is to be analyzed is loaded into the database for storage, and one standardized data and one non-standard data are selected from the original data set to be tested based on the preset standard, and the categories are set as normal data and abnormal data respectively, that is, one normal data and one abnormal data are determined from the original data set to be tested, and the similarities are 1 and 0 respectively, to form a classified data set. For the data whose categories are already known in the original data set to be tested, they are classified as classified data and stored in the classified data set.

[0040] S2: randomly selecting an unclassified data in the original data set to be detected, and calculating the similarity between the unclassified data and each classified data in the classified data set based on a comprehensive verification rule;

[0041] In the original data set to be detected, a data without classification is randomly selected, that is, a data with unknown category is randomly selected as unclassified data.

[0042] Wherein, calculating the similarity between the unclassified data and each classified data in the classified data set based on the comprehensive verification rule may include:

[0043] 1) For each classified data in the classified data set, calculate the character length matching degree, special character matching degree and data type matching degree between the unclassified data and the classified data;

[0044] 2) Perform weighted summation of character length matching, special character matching, and data type matching to obtain the similarity between unclassified data and classified data. The greater the similarity, the closer the distance between the two.

[0045] Specifically, similarity = character length matching degree*A+special character matching degree*B+data type matching degree*C;

[0046] Where A+B+C=1, A, B, C are the weights of character length matching, special character matching and data type matching respectively, and the specific values ​​of A, B, C are adjusted according to the experimental accuracy.

[0047] Character length matching degree: equal to the length of the short character / the length of the long character, the value is <= 1.

[0048] Special character matching degree: Set characters other than English letters, numbers, Chinese characters, and punctuation marks as special characters (such as!, @, #, ¥, %, etc.). If only one of the unclassified data and the classified data contains special characters, the special character matching degree is 0, and the others are 1.

[0049] Data type matching degree: If the data types of the unclassified data and the classified data are both Chinese characters, numbers or English letters, the data type matching degree is 1, otherwise it is 0.

[0050] More specifically, calculating the character length matching degree between the unclassified data and the classified data includes: calculating the first character length of the unclassified data and the second character length of the classified data respectively. Determining whether the first character length is less than the second character length; if so, taking the ratio of the first character length to the second character length as the character length matching degree between the unclassified data and the classified data, that is, taking the quotient of the first character length divided by the second character length as the character length matching degree; if not, taking the ratio of the second character length to the first character length as the character length matching degree between the unclassified data and the classified data, that is, taking the quotient of the second character length divided by the first character length as the character length matching degree.

[0051] Calculating the special character matching degree between the unclassified data and the classified data includes: determining whether there are special characters in the unclassified data and the classified data; the special characters are characters other than English letters, numbers, Chinese characters, and punctuation marks; if the special characters exist in both the unclassified data and the classified data, or if the special characters do not exist in both the unclassified data and the classified data, the special character matching degree between the unclassified data and the classified data is 1; otherwise, the special character matching degree between the unclassified data and the classified data is 0.

[0052] Calculating the data type matching degree between the unclassified data and the classified data includes: determining whether the data types of the unclassified data and the classified data are the same; the data types include Chinese characters, numbers, and English letters; if so, the data type matching degree between the unclassified data and the classified data is 1; otherwise, the data type matching degree between the unclassified data and the classified data is 0.

[0053] The comprehensive verification rule proposed in this embodiment can obtain the similarity between unclassified data and classified data by calculating the character length matching, special character matching and data type matching. The verification rule is simple and can significantly improve the efficiency of abnormal data detection compared with the existing complex verification rules.

[0054] S3: Sort the similarities in descending order, and select the classified data corresponding to the first K similarities as K neighbors of the unclassified data;

[0055] The K value is initialized to 2 and remains unchanged. Experiments have shown that if k is all classified samples, it is easy to cause the early sample classification results to have too great an impact on the later sample classification results, so K is a fixed value in this embodiment.

[0056] S4: Determine the category of the unclassified data according to the categories of the K neighbors to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data;

[0057] Wherein, determining the category of the unclassified data according to the categories of the K neighbors includes: taking the category of the majority of the K neighbors as the category of the unclassified data, that is, when the categories of more than K / 2 neighbors are the same, the category is the category of the unclassified data. For example, K is 5, when the categories of 3 neighbors are normal data, then the category of the unclassified data at this time is normal data. When among the K neighbors, K / 2 neighbors are normal data and K / 2 neighbors are abnormal data, then the category of the unclassified data is the same as the category of the neighbor with the highest similarity among the K neighbors. For example, K is 6, where the categories of 3 neighbors are normal data and 3 neighbors are abnormal data, and the category of the neighbor with the highest similarity is normal data, then the category of the unclassified data is normal data.

[0058] It should be noted that after the category of the unclassified data is determined, the unclassified data becomes classified data, and the classified data is added to the classified dataset. When determining the category of the next unclassified data, the new classified dataset is used as the classified dataset.

[0059] S5: Determine whether there is unclassified data in the original dataset to be detected;

[0060] S6: If so, return to the step of "randomly select an unclassified data from the original dataset to be detected" until each unclassified data in the dataset to be detected has been detected;

[0061] S7: If not, determine the abnormal data according to the classified dataset.

[0062] Next, through a specific experiment, the abnormal data detection method used in this embodiment is further described:

[0063] 1. Initialize the data: Select the original dataset to be detected, and select normal data and abnormal data from it, as shown in Table 1.

[0064] Table 1

[0065]

[0066]

[0067] 2. Take [Guo Moumou] as the unclassified data, and preset the initial K = 2.

[0068] 3. Calculate the similarity between the unclassified data and the classified data, that is, calculate the similarity between [Guo Moumou] and [Zhang Mou], [281927] respectively.

[0069] The similarity between [Guo Moumou] and [Zhang Mou] = 0.67 * 0.2 + 0 * 0.4 + 1 * 0.4 = 0.534

[0070] The similarity between [Guo Moumou] and [281927] = 0.5 * 0.2 + 1 * 0.4 + 0 * 0.4 = 0.5

[0071] 4. Determine the K neighbors and the category of the unclassified data

[0072] Since the number of neighbors K is equal to 2, the neighbors are [Zhang Mou], [281927]. At this time, K / 2 neighbors are normal data and K / 2 neighbors are abnormal data. Also, because the maximum similarity is 0.534, which is the similarity corresponding to the neighbor [Zhang Mou], the category of the unclassified data is the same as that of [Zhang Mou], and the category is normal data.

[0073] 5. Calculate and classify other unclassified data according to the above method, and the abnormal data detection situation is shown in Table 2.

[0074] Table 2

[0075]

[0076]

[0077] As shown in Table 2, there are 5 unclassified data in this experiment. One type of abnormal data is detected by the algorithm. In addition, the sample [$8you are] is abnormal data but not detected. Therefore, the accuracy of detecting abnormal data by the method of this embodiment is 75%. The factor affecting the accuracy is mainly the semantic difference of the data. The semantic matching rule can be added to the subsequent similarity calculation to further improve the accuracy of abnormal data detection. However, since there is no suitable semantic matching algorithm support at present, it is not considered to be added to this embodiment for the time being.

[0078] This example uses the K nearest neighbor algorithm to detect abnormal data. Here we introduce the principle of the K nearest neighbor algorithm.

[0079] 1. Concept introduction:

[0080] The so-called K-nearest neighbor algorithm is to find the K instances closest to the new input instance in the training data set (the K neighbors mentioned above) given a training data set. If most of these K instances belong to a certain class, the input instance is classified into this class.

[0081] 2. Case introduction:

[0082] like Figure 2 As shown in the figure, there are two different types of sample data, represented by squares and triangles respectively, and the data marked by the circle in the figure is the data to be classified. In other words, we don’t know which category the circle belongs to (square or triangle), so we need to solve this problem: classify the circle.

[0083] If K=3, the three nearest neighbors of the circle are two triangles and one square. The minority is subordinate to the majority, and based on statistical methods, it is determined that the circle belongs to the triangle category.

[0084] If K=5, the five nearest neighbors of the circle are 2 triangles and 3 squares. The minority is subordinate to the majority, and based on statistical methods, it is determined that the circle belongs to the category of squares.

[0085] It can be seen that when it is impossible to determine which category the current point to be classified belongs to among the known categories, we can look at its location characteristics based on statistical theory, measure the weights of its neighbors, and classify (or assign) it to the category with a larger weight. This is the core idea of ​​the K-nearest neighbor algorithm. In the KNN algorithm (K-nearest neighbor algorithm), the selected neighbors are all objects that have been correctly classified. This method only determines the category of the sample to be classified based on the category of one or several nearest samples in the classification decision. The KNN algorithm itself is simple and effective. It is a lazy-learning algorithm. The classifier does not need to be trained using a training set, and the training time complexity is 0. The computational complexity of KNN classification is proportional to the number of documents in the training set. That is, if the total number of documents in the training set is n, then the classification time complexity of KNN is O(n). Although the KNN method also relies on the limit theorem in principle, it is only related to a very small number of adjacent samples when making category decisions. Since the KNN method mainly relies on the limited number of neighboring samples rather than the method of distinguishing the class domain to determine the category, the KNN method is more suitable than other methods for the sample sets to be classified with more cross-domain or overlap. The model used by the K nearest neighbor algorithm actually corresponds to the division of the feature space. The selection of K value, distance measurement and classification decision rule are the three basic elements of the algorithm: When implementing the K nearest neighbor algorithm, the main consideration is how to perform fast K nearest neighbor search on the training data, which is very necessary when the feature space dimension is large and the training data capacity is large.

[0086] The present embodiment provides an abnormal data detection method based on a K-nearest neighbor algorithm. Based on the principle of the above-mentioned KNN algorithm, the similarity between the unclassified data and each classified data in the classified data set is calculated based on a comprehensive verification rule, and the K neighbors of the unclassified data are determined according to the similarity, and the category of the unclassified data is determined according to the category of the K neighbors, and then the category of each data in the original data set to be detected is determined based on the similarity to determine all abnormal data, thereby providing a general method for identifying abnormal data, making the verification operation simple and highly reusable, and at the same time reducing the operation of configuring verification rules for each field in data quality verification, reducing human workload, and improving verification efficiency.

[0087] Embodiment 2:

[0088] This embodiment is used to provide an abnormal data detection system based on the K nearest neighbor algorithm. Figure 3 As shown, the detection system comprises:

[0089] The classified data set determination module M1 is used to determine a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute the classified data set;

[0090] A similarity calculation module M2 is used to randomly select an unclassified data in the original data set to be detected, and calculate the similarity between the unclassified data and each classified data in the classified data set based on a comprehensive verification rule;

[0091] A neighbor determination module M3 is used to sort the similarities in descending order and select the classified data corresponding to the first K similarities as the K neighbors of the unclassified data;

[0092] A category determination module M4 is used to determine the category of the unclassified data according to the categories of the K neighbors, to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data;

[0093] A judging module M5 is used to judge whether there is unclassified data in the original data set to be detected;

[0094] Return module M6, for, if yes, returning to the step of "randomly selecting an unclassified data in the original data set to be detected" until each unclassified data in the data set to be detected has been detected;

[0095] The abnormal data determination module M7 is used to determine the abnormal data according to the classified data set if no.

[0096] Embodiment 3:

[0097] This embodiment is used to provide an abnormal data detection device based on a K-nearest neighbor algorithm, including:

[0098] Processor; and

[0099] a memory storing computer readable program instructions,

[0100] Wherein, when the computer-readable program instructions are executed by the processor, the detection method as described in Example 1 is executed.

[0101] Embodiment 4:

[0102] This embodiment is used to provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the detection method described in Example 1 are implemented.

[0103] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0104] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the claims of the present invention and their equivalents.

Claims

1. A method for detecting abnormal data based on K-nearest neighbor algorithm, characterized in that: The detection method comprises: Determine a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute a classified data set; Randomly select an unclassified data in the original data set to be detected, and calculate the similarity between the unclassified data and each classified data in the classified data set based on the comprehensive verification rule; Sort the data in descending order according to the similarity, select the classified data corresponding to the first K similarities as the K neighbors of the unclassified data, and initialize the K value to 2 and keep it fixed; Determine the category of the unclassified data according to the categories of the K neighbors to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data; Determine whether there is unclassified data in the original data set to be detected; If yes, return to the step of "randomly selecting an unclassified data in the original data set to be detected" until every unclassified data in the data set to be detected has been detected; If not, determining abnormal data based on the classified data set; in, The calculating of the similarity between the unclassified data and each classified data in the classified data set based on the comprehensive verification rule specifically includes: For each classified data in the classified data set, calculating the character length matching degree, special character matching degree and data type matching degree between the unclassified data and the classified data; Performing a weighted summation on the character length matching degree, the special character matching degree and the data type matching degree to obtain the similarity between the unclassified data and the classified data, Similarity = character length matching degree*A+special character matching degree*B+data type matching degree*C, A+B+C=1, A, B, C are the weights of character length matching degree, special character matching degree and data type matching degree respectively.

2. The detection method according to claim 1, characterized in that: The calculating of the character length matching degree between the unclassified data and the classified data specifically includes: respectively calculating a first character length of the unclassified data and a second character length of the classified data; Determining whether the first character length is less than the second character length; If yes, taking the ratio of the first character length to the second character length as the character length matching degree between the unclassified data and the classified data; If not, the ratio of the second character length to the first character length is used as the character length matching degree between the unclassified data and the classified data.

3. The detection method according to claim 1, characterized in that: The calculating of the special character matching degree between the unclassified data and the classified data specifically includes: Determine whether there are special characters in the unclassified data and the classified data; the special characters are characters other than English letters, numbers, Chinese characters, and punctuation marks; If the special characters exist in both the unclassified data and the classified data, or if the special characters do not exist in both the unclassified data and the classified data, the special character matching degree between the unclassified data and the classified data is 1; Otherwise, the special character matching degree between the unclassified data and the classified data is 0.

4. The detection method according to claim 1, characterized in that: The calculating of the data type matching degree between the unclassified data and the classified data specifically includes: Determine whether the data types of the unclassified data and the classified data are the same; the data types include Chinese characters, numbers and English letters; If yes, then the data type matching degree between the unclassified data and the classified data is 1; Otherwise, the data type matching degree between the unclassified data and the classified data is 0.

5. The detection method according to claim 1, characterized in that: Determining the category of the unclassified data according to the categories of the K neighbors specifically includes: The categories of the majority of the K neighbors are used as the categories of the unclassified data.

6. The detection method according to claim 5, characterized in that: When K / 2 of the K neighbors are normal data and K / 2 of the neighbors are abnormal data, the category of the unclassified data is the same as the category of the neighbor with the highest similarity among the K neighbors.

7. An abnormal data detection system based on K-nearest neighbor algorithm, characterized in that: The detection system comprises: A classified data set determination module is used to determine a normal data and an abnormal data in the original data set to be detected based on a preset standard; the normal data and the abnormal data constitute the classified data set; a similarity calculation module, configured to randomly select an unclassified data in the original data set to be detected, and calculate the similarity between the unclassified data and each classified data in the classified data set based on a comprehensive verification rule, in, The calculating of the similarity between the unclassified data and each classified data in the classified data set based on the comprehensive verification rule specifically includes: For each classified data in the classified data set, calculating the character length matching degree, special character matching degree and data type matching degree between the unclassified data and the classified data; Performing a weighted summation on the character length matching degree, the special character matching degree and the data type matching degree to obtain the similarity between the unclassified data and the classified data, Similarity = character length matching degree*A+special character matching degree*B+data type matching degree*C, A+B+C=1, A, B, C are the weights of character length matching degree, special character matching degree and data type matching degree respectively. A neighbor determination module is used to sort the similarities from large to small, select the classified data corresponding to the first K similarities as the K neighbors of the unclassified data, and initialize the K value to 2 and keep it fixed; A category determination module, used to determine the category of the unclassified data according to the categories of the K neighbors, to make it classified data, and put the classified data into the classified data set; the categories include normal data and abnormal data; A judgment module, used to judge whether there is unclassified data in the original data set to be detected; a return module, for, if yes, returning to the step of "randomly selecting an unclassified data in the original data set to be detected" until each unclassified data in the data set to be detected has been detected; The abnormal data determination module is used to determine the abnormal data according to the classified data set if no.

8. An abnormal data detection device based on K-nearest neighbor algorithm, characterized in that: include: processor; as well as a memory storing computer readable program instructions, Wherein, when the computer-readable program instructions are executed by the processor, the detection method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the detection method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • text classification method and device based on K nearest neighbor KNN

    CN109993216A