A method for quickly detecting and processing dirty data in log collection

Abstract

The present invention relates to a method for quickly detecting and processing dirty data collected by logs. The method comprises: S1, collecting logs of various types of network devices and aggregating them into a message middleware; S2, matching the logs in the message middleware with parsing rules according to the parsing rules in the parsing library, extracting fields that are successfully matched, and executing step S3; considering that the fields that are unsuccessfully matched are dirty data, and executing step S5; S3, classifying the extracted fields according to the field classification rules in the field classification library; S4, for the classified fields, according to their corresponding classification values, finding the probability of this classification value through breadth-first traversal, and similarly finding the probability of the classification values ​​of all fields under the corresponding log, and judging the probability of the classification value of each field in turn, as long as the probability of the classification value of one of the fields is greater than the set threshold, it is judged as dirty data; S5, storing the dirty data in a dirty database, and the maintenance personnel readjust the rules according to the dirty data in the dirty database.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for quickly detecting and processing dirty data in log collection. Background Art

[0002] In the process of collecting logs, dirty data is often generated in the system from time to time due to incorrect rule configuration, defective rule configuration, or incomplete configuration. These dirty data will affect the normal data display of our system and are time-consuming and laborious to clean up.

[0003] In the existing collection and cleaning solutions, the general process is to first extract fields according to the parsing rules. Then classify based on the classification rules to achieve log standardization. The existing method can avoid dirty data to the greatest extent. If the correct field is not extracted during the parsing process, it is judged as dirty data without further judgment.

[0004] The existing solutions for handling dirty data during the collection process have the following main disadvantages:

[0005] 1: Simply determine whether the log parsing is successful. If it fails, it is considered dirty data.

[0006] 2: Some dirty data that is hidden deeper cannot be detected successfully, for example, the classification of some fields is not the result we want.

[0007] 3: Some systems have dirty data judgment rules, which are complex to match, inefficient, and have low judgment accuracy. Summary of the invention

[0008] The purpose of the present invention is to solve the problem of detecting and avoiding various dirty data from contaminating normal data in the current network security log collection process, and thus provide a method for quickly detecting and processing dirty data in log collection.

[0009] To achieve the above object, the technical solution of the present invention is: a method for quickly detecting and processing dirty data in log collection, comprising the following steps:

[0010] S1. Collect logs of various network device types through the network and finally aggregate them to the message middleware;

[0011] S2, according to the parsing rules in the parsing library, the logs in the message middleware are parsed and matched, and the fields are extracted if the match is successful, and step S3 is executed; if the match is unsuccessful, it is considered to be dirty data, and step S5 is executed;

[0012] S3, classifying some of the extracted classification fields according to the field classification rules in the field classification library;

[0013] S4. For the classified fields, find the probability of the classification value through breadth-first traversal according to the corresponding classification value. Similarly, find the probability of the classification value of all fields in the corresponding log, and judge the probability of the classification value of each field in turn. As long as the probability of the classification value of one field is greater than the set threshold, it is determined to be dirty data.

[0014] S5. The dirty data is stored in a dirty database, and the maintenance personnel readjust the rules according to the dirty data in the dirty database.

[0015] In one embodiment of the present invention, the field classification type in the segment classification rule is stored by row, and the segment classification rule is a classification rule tree constructed starting from the root node, and each leaf node is composed of the classification value of the corresponding field, that is, the digital code and the corresponding probability value.

[0016] Compared with the prior art, the present invention has the following beneficial effects: because the method of the present invention performs a breadth-first traversal of the tree in memory, it can quickly improve the search efficiency, and the determination of the probability value is more accurate. The determination of each field has higher accuracy, and the entire process can quickly and accurately determine dirty data.

[0017] In addition, if it is judged to be dirty data, it will be stored in the corresponding library and then manually corrected, so that the dirty data will not be imported into the normal warehouse and affect the normal process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention.

[0019] Figure 2 This is a schematic diagram of the classification rule tree structure of the present invention.

[0020] Figure 3 It is a schematic diagram of the field judgment flow of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0022] The present invention provides a method for quickly detecting and processing dirty data in log collection, comprising the following steps:

[0023] S1. Collect logs of various network device types through the network and finally aggregate them to the message middleware;

[0024] S2, according to the parsing rules in the parsing library, the logs in the message middleware are parsed and matched, and the fields are extracted if the match is successful, and step S3 is executed; if the match is unsuccessful, it is considered to be dirty data, and step S5 is executed;

[0025] S3, classifying some of the extracted classification fields according to the field classification rules in the field classification library;

[0026] S4. For the classified fields, find the probability of the classification value according to its corresponding classification value through breadth-first traversal. Similarly, find the probability of the classification value of all the fields that need to be classified in the corresponding log, and judge the probability of the classification value of each field in turn. As long as the probability of the classification value of one field is greater than the set threshold, it is determined to be dirty data.

[0027] S5. The dirty data is stored in a dirty database, and the maintenance personnel readjust the rules according to the dirty data in the dirty database.

[0028] The following is the specific implementation process of the present invention.

[0029] The present invention provides a method for quickly detecting and processing dirty data in log collection, which includes the following modules:

[0030] The collection module collects logs of various types of network devices through the network and finally aggregates them to the message middleware.

[0031] The parsing library and classification library respectively store the rules of the parsing module and the classification module, which are generally regular expressions and keywords.

[0032] The parsing module consumes the message middleware data and extracts fields from the original log according to various algorithms.

[0033] The classification module categorizes fields.

[0034] The dirty data processing module mainly stores dirty data in the corresponding warehouse, which can be viewed manually.

[0035] In the manual correction module, maintenance personnel readjust the rules based on dirty data.

[0036] The target library is the warehouse where normal events are recorded.

[0037] The dirty data detection module is the most important module. Many secret and hard-to-find dirty data are quickly and accurately identified by this module. The detailed process is as follows:

[0038] 1. Set the threshold for the classification library of each classification field respectively.

[0039] 2. Various classification libraries are loaded into the memory (mainly various classification values ​​and judgment probabilities. For example, 1 represents the root node, which basically means that the classification is not successful, that is, dirty data, and the probability can be set to 0.99).

[0040] 3. Build a classification rule tree in memory, which has higher processing efficiency.

[0041] 4. Based on the classification value of the event, perform a breadth-first search on the tree to find the classification value of the event and find out the probability value. Determine whether the data is dirty

[0042] 5. Judge each field in turn. As long as one of them is greater than the threshold we set, it is considered dirty data.

[0043] like Figure 1 As shown, it is the overall processing flow of a method for quickly detecting and processing dirty data in log collection of the present invention.

[0044] In the process of log normalization, the probability judgment of the classification rules of the present invention is based on each field that needs to be classified. The judgment is relatively strict. As long as one of the items does not meet the requirements, it is judged as dirty data. Such result judgment is more accurate. The probability judgment of a certain field requires fast and accurate results, and the efficiency requirement is relatively high.

[0045] Classification rules for multiple classification fields are loaded in the memory, such as device type, event classification, etc. The probability judgment for each field must also have a corresponding accurate value. The present invention sets the probability based on the classification value.

[0046] The classification types defined in the library table of the present invention are stored in rows, for example, 10 represents network security events, 1002 represents scanning behavior, 100200 represents host scanning, and 11 represents data security events. The present invention constructs the entire table data into a tree, and each leaf node consists of a digital code and a probability value, such as Figure 2 shown.

[0047] The entire table data is a complete tree in memory, and then the entire tree is traversed breadth-first according to the classification value of the field (such as event) to find the probability p1 of this classification value.

[0048] Then, the probabilities p2...pn of other fields are calculated respectively according to the breadth-first traversal algorithm.

[0049] Finally, a comprehensive judgment is made. As long as any probability value is greater than the predetermined threshold, it is considered dirty data. Figure 3 shown.

[0050] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for quickly detecting and processing dirty data in log collection, characterized in that: The steps include: S1. Collect logs of various network device types through the network and finally aggregate them to the message middleware; S2, according to the parsing rules in the parsing library, the logs in the message middleware are parsed and matched, and the fields are extracted if the match is successful, and step S3 is executed; if the match is unsuccessful, it is considered to be dirty data, and step S5 is executed; S3, classifying the extracted fields according to the field classification rules in the field classification library; S4. For the classified fields, find the probability of the classification value through breadth-first traversal according to the corresponding classification value. Similarly, find the probability of the classification value of all fields in the corresponding log, and judge the probability of the classification value of each field in turn. As long as the probability of the classification value of one field is greater than the set threshold, it is determined to be dirty data. S5. The dirty data is stored in a dirty database, and the maintenance personnel readjust the parsing rules according to the dirty data in the dirty database; The field classification type in the field classification rule is stored by row. The field classification rule is a classification rule tree constructed starting from the root node. Each node is composed of the classification value of the corresponding field and the corresponding probability. The classification value of the corresponding field is a digital code; the probability of the classification value decreases from the root node to the leaf node.

Images