A normalized log generation method based on entropy increase principle

Abstract

This invention discloses a normalized log generation method based on the entropy increase principle, belonging to the field of database log auditing technology. The method includes: determining the entropy value of each field in the current log based on the entropy increase principle; determining whether a field is an important or unimportant field by using the relationship between the entropy value of the field and a set threshold; setting a related primary key for the current log, wherein the related primary key can represent the unique mapping relationship between all important fields and all unimportant fields in the current log; and converting the current log into a normalized log structure including important fields, related primary keys, and unimportant fields. This solution, using the entropy increase principle normalization method, can effectively handle original logs from multiple sources with different structures and has strong generalization ability. In addition, the final normalized log structure containing important and unimportant fields and related primary keys has a rigorous and unified format and concise rule definitions, and can be rapidly iterated and upgraded as the log structure changes.

Description

Technical Field

[0001] This invention belongs to the field of database log auditing technology, and more specifically, relates to a standardized log generation method based on the principle of entropy increase. Background Technology

[0002] With the widespread adoption of big data, cloud computing, the Internet of Things, edge computing, and artificial intelligence, contemporary information technology typically incorporates a large amount of distributed computing resources. These computing resources generate hundreds of millions of data points every day, presenting an increasingly complex and difficult-to-manage situation.

[0003] Logs, serving as crucial evidence for system maintenance, iteration, intrusion detection, and anomaly detection, are an indispensable component of distributed information systems. However, the complexity and uncertainty of system operating environments, along with differences between protocols, lead to significant variations in log structures generated by different systems. Without an effective log normalization method, subsequent log parsing, evidence analysis, and tracing processes will face numerous inconveniences. However, manually normalizing log data is clearly impractical in the face of massive datasets. Most existing log normalization and management tools follow classic expert system methods and heavily rely on custom rules for filtering under specific formats. This not only has weak generalization capabilities but also requires continuous maintenance as the computing system upgrades. This approach incurs additional maintenance costs and necessitates constant combination and fine-tuning of rules.

[0004] With the rise of artificial intelligence and the increase in AIOps activities, using AI technology to generate log normalization structures has gradually become a hot topic. Although AI technology has stronger generalization capabilities than traditional log normalization and management tools, it still cannot achieve a high degree of normalization due to the uncertainty of its output. Therefore, most existing AI technologies still process logs in natural text form or semi-structured logs, which suffer from problems such as chaotic log format structure and weak generalization ability. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a standardized log generation method based on the principle of entropy increase. The purpose is to determine the entropy value of each field in the current log based on the principle of entropy increase; to determine whether a field is important or unimportant by using the relationship between its entropy value and a set threshold; to set a primary key for the current log; and to transform the current log into a standardized log structure including important fields, the primary key, and unimportant fields. This solves the technical problems of chaotic log format structure and weak generalization ability in existing log standardization methods.

[0006] To achieve the above objectives, according to one aspect of the present invention, a standardized log generation method based on the principle of entropy increase is provided, comprising:

[0007] S1: Preprocess the current log to obtain a preliminary normalized structure;

[0008] S2: Calculate the entropy value of each field in the preliminary normalized structure based on the principle of entropy increase; determine whether each field is an important field or an unimportant field by using the relationship between the entropy value of each field and a set threshold.

[0009] S3: Set the associated primary key corresponding to the current log, wherein the associated primary key can represent the unique mapping relationship between all the important fields and all the non-important fields in the current log;

[0010] S4: Transform the preliminary normalized structure corresponding to the current log into a log normalized structure that includes important fields, related primary keys, and non-important fields.

[0011] In one embodiment, S1 includes:

[0012] S11: Divide the original current log into multiple segments to obtain the first log format;

[0013] S12: Remove redundancy from the first log format to obtain the second log format;

[0014] S13: Parse the second log format into a tree structure, traverse the tree structure, and obtain the preliminary normalized structure based on the content of each node.

[0015] In one embodiment, S11 includes:

[0016] The original current log is divided into the first log format by using spaces as delimiters and dividing it according to the token format of each word;

[0017] The "Word" is determined semantically and can be one or more of the following: English letters, English words, English phrases, Chinese characters, and Chinese phrases.

[0018] In one embodiment, S12 includes:

[0019] Redundancy is removed by deleting irrelevant characters from the first log format using regular expressions, resulting in the second log format.

[0020] In one embodiment, S13 includes:

[0021] The Drain3 log parsing tool is used to parse the English text in the second log format in a mixed manner, so as to parse the current log into a tree structure; the tree structure is traversed and the preliminary normalized structure is obtained based on the content of each node.

[0022] In one embodiment, S2 includes:

[0023] S21: Calculate the difference score for each field in the preliminary normalized structure corresponding to the current log, and calculate the entropy value for each field based on the difference score;

[0024] S22: Determine whether a field is important or unimportant by using the relationship between the entropy value corresponding to each field and a set threshold.

[0025] In one embodiment, S21 includes:

[0026] S211: Calculate the cumulative number of different values ​​for each field in the current log, starting from the first log entry;

[0027] S212: Calculate the Euclidean distance between different values ​​in each field using the cumulative number of different values ​​in the current log. ;Utilize the Euclidean distance between different values ​​under each field Calculate the differential score corresponding to the field;

[0028] S213: Divide the differential scores of each field by the cumulative number n of different values ​​of the field to obtain the corresponding entropy value.

[0029] In one embodiment, S212 includes:

[0030] Using formula Calculate the Euclidean distance between different values ​​in each field of the current log. ; , For different values ​​under a field, and n represents the cumulative number of different values ​​in the field. There are p in total. n* ( n-1 ) / 2;

[0031] Using formula Calculate the differential score corresponding to the t-th field. ; This represents the Euclidean distance corresponding to the k-th combination under the t-th field.

[0032] In one embodiment, S22 includes:

[0033] Sort the entropy values ​​corresponding to each field in the current log.

[0034] Fields with entropy values ​​greater than the set threshold are considered important fields; fields with entropy values ​​less than or equal to the set threshold are considered unimportant fields.

[0035] In one embodiment, the associated primary key includes one or more of the following: event ID, event reception time, time name, device name, and generalization policy name.

[0036] To achieve the above objectives, according to another aspect of the present invention, a standardized log generation apparatus based on the principle of entropy increase is provided, comprising:

[0037] The preprocessing module is used to preprocess the current log to obtain a preliminary normalized structure;

[0038] The determination module is used to calculate the entropy value of each field in the preliminary normalized structure based on the principle of entropy increase; and to determine whether each field is an important field or an unimportant field by using the relationship between the entropy value of each field and a set threshold.

[0039] The setting module is used to set the associated primary key corresponding to the current log, wherein the associated primary key can represent the unique mapping relationship between all the important fields and all the non-important fields in the current log;

[0040] The transformation module is used to transform the preliminary normalized structure corresponding to the current log into a log normalized structure that includes important fields, related primary keys, and non-important fields.

[0041] To achieve the above objectives, according to another aspect of the present invention, a database system is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a normalized log generation method based on the principle of entropy increase.

[0042] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a standardized log generation method based on the principle of entropy increase.

[0043] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0044] (1) This invention provides a normalized log generation method based on the principle of entropy increase. The entropy value of each field in the current log is determined based on the principle of entropy increase; the relationship between the entropy value of the field and a set threshold is used to determine whether it is an important field or an unimportant field; the associated primary key of the current log is set; and the current log is transformed into a normalized log structure including important fields, associated primary keys and unimportant fields. The normalization method based on the principle of entropy increase can effectively handle original logs from multiple sources with different structures. As the cumulative number of values ​​of each field changes, the corresponding entropy value is also dynamically updated. The division method of important fields and unimportant fields also changes accordingly. The final target normalized log structure can be dynamically updated and has strong generalization ability. In addition, the final normalized log structure containing important fields, unimportant fields and associated primary keys has a strict and unified format and simple rule definition, and can be rapidly iterated and upgraded as the log structure changes.

[0045] (2) This scheme divides the original log format, removes redundancy and parses it into a tree structure to obtain a preliminary standardized structure, so as to facilitate the subsequent differential score calculation, accurately extract key log information and achieve efficient log information filtering.

[0046] (3) This scheme divides the original current log according to the token format of each word, which facilitates the subsequent neural network vectorization process, enhances the generalization ability of the method, and realizes an automated log segmentation scheme.

[0047] (4) This solution uses regular expressions to remove irrelevant semantic characters in the first log format to achieve redundancy removal, while retaining core key information such as log templates and values, thus realizing an automated noise reduction solution for logs.

[0048] (5) This solution uses the Drain3 log parsing tool to perform mixed parsing of English in the second log format in order to parse the current log into a tree structure. The Drain3 log parsing tool runs fast and can complete the running inference in linear polynomial time, which is significantly faster than AI-based methods.

[0049] (6) This scheme calculates the difference score of each field in the preliminary normalized structure corresponding to the current log one by one, and calculates the entropy value of each field based on the difference score. This calculation method is based on the principle of entropy increase, which can effectively reflect the importance of different fields and realize efficient log field division.

[0050] (7) This scheme uses the cumulative number of different values ​​of each field in the current log to calculate the Euclidean distance between different values ​​of each field. ;Utilize the Euclidean distance between different values ​​under each field Calculate the difference score corresponding to the field; divide the difference score of each field by the cumulative number n of different values ​​of the field to obtain the corresponding entropy value. This calculation method takes into account the differences of different logs under the same field, and realizes efficient and automated calculation of entropy value.

[0051] (8) This scheme utilizes the formula and The calculation method for the differential score corresponding to the t-th field is simple and has low algorithm complexity, which allows for the rapid acquisition of the entropy value of each field, ultimately improving the execution efficiency of the entire algorithm.

[0052] (9) In this scheme, fields with entropy values ​​greater than the set threshold are considered important fields; otherwise, they are considered unimportant fields. The classification of important and unimportant fields is updated as the cumulative number of each field changes, and the target specification structure of the final log can also be dynamically updated. This criterion is simple, the algorithm complexity is extremely low, and thus the importance of each field can be quickly determined, ultimately improving the execution efficiency of the entire algorithm.

[0053] (10) The primary key associated with this scheme includes one or more of the following: event ID, event receiving time, time name, device name and generalization strategy name; this setting takes into account the possibility of repetition of different fields, can adapt to different database system requirements, and realizes efficient and accurate association lookup. Attached Figure Description

[0054] Figure 1 This is a flowchart of the standardized log generation method based on the principle of entropy increase provided in Embodiment 1 of the present invention.

[0055] Figure 2 This is a schematic diagram of the log normalization structure provided in Embodiment 1 of the present invention.

[0056] Figure 3 This is a flowchart of the preprocessing of the current log in S1 provided in Embodiment 2 of the present invention.

[0057] Figure 4 This is a schematic diagram of the process of using the Drain3 log parsing tool to perform mixed parsing of English in the second log format in S13 of Embodiment 5 of the present invention.

[0058] Figure 5 This is a flowchart illustrating the process of determining whether each field is an important field or an unimportant field based on the principle of entropy increase in S2 provided in Embodiment 6 of the present invention.

[0059] Figure 6 This is a schematic diagram of S21 in Embodiment 7 of the present invention, which calculates the differential scores corresponding to each field and calculates the corresponding entropy value based on the differential scores. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0061] Example 1

[0062] like Figure 1 As shown, this solution provides a normalized log generation method based on the entropy increase principle, including: S1: Preprocessing the current log to obtain a preliminary normalized structure; S2: Calculating the entropy value of each field in the preliminary normalized structure based on the entropy increase principle; determining whether a field is important or unimportant by using the relationship between the entropy value of each field and a set threshold; S3: Setting a primary key for the current log, which represents the unique mapping relationship between all important fields and all unimportant fields in the current log; S4: Transforming the preliminary normalized structure of the current log into a log normalized structure including important fields, the primary key, and unimportant fields, as shown in the figure. Figure 2 As shown.

[0063] Example 2

[0064] In this embodiment, as Figure 3 As shown, S1 includes: S11: Divide the original current log into multiple segments to obtain the first log format; S12: Remove redundancy from the first log format to obtain the second log format; S13: Parse the second log format into a tree structure, traverse the tree structure, and obtain a preliminary normalized structure based on the content of each node.

[0065] Example 3

[0066] In this embodiment, S11 includes: dividing the original current log into segments using spaces as delimiters according to the token format of each Word, and saving each segmented log as a whole to form a first log format; wherein, the Word is determined according to semantics and is one or more of English letters, English words, English phrases, Chinese characters, and Chinese phrases.

[0067] Example 4

[0068] In this embodiment, S12 includes: using regular expressions to delete characters with irrelevant semantics (such as quotation marks, colons, semicolons, etc.) in the first log format to remove redundancy and obtain the second log format.

[0069] Example 5

[0070] In this embodiment, as Figure 4 As shown, S13 includes: using the Drain3 log parsing tool to perform mixed parsing of English in the second log format to parse the current log into a tree structure; traversing the tree structure and obtaining a preliminary normalized structure based on the content of each node.

[0071] Example 6

[0072] In this embodiment, as Figure 5 As shown, S2 includes: S21: Calculate the difference score of each field in the preliminary normalized structure corresponding to the current log one by one, and calculate the entropy value of each field based on the difference score; S22: Determine whether a field is important or unimportant by using the relationship between the entropy value of each field and the set threshold.

[0073] Example 7

[0074] In this embodiment, as Figure 6 As shown, S21 includes: S211: Calculate the cumulative number n of distinct values ​​n for each field in the current log, starting from the first log entry; S212: Calculate the Euclidean distance between distinct values ​​in each field using the cumulative number n of distinct values ​​n in the current log. ;Utilize the Euclidean distance between different values ​​under each field Calculate the differential score corresponding to the field. , m is the number of fields in the current database; S213: Utilize the differential scores of each field Divide by the cumulative number n of different values ​​in the field to obtain the corresponding entropy value. .

[0075] Example 8

[0076] In this embodiment, S212 includes: using the formula Calculate the Euclidean distance between different values ​​in each field of the current log. ; , For different values ​​under a field, and n represents the cumulative number of different values ​​in the field. There are p in total. n* ( n-1 ) / 2; using the formula Calculate the differential score corresponding to the t-th field. ; This represents the Euclidean distance corresponding to the k-th combination under the t-th field.

[0077] Example 9

[0078] In this embodiment, S22 includes: sorting the entropy values ​​corresponding to each field in the current log to obtain a list of entropy values ​​of different fields arranged in descending order: ; Set the field whose entropy value is greater than the set threshold All of these are considered important fields:

[0079] ;

[0080] ;

[0081] Fields whose entropy values ​​are less than or equal to a set threshold are considered unimportant fields.

[0082] Example 10

[0083] In this embodiment, the associated primary key includes one or more of the following: event ID, event reception time, time name, device name, and generalization strategy name.

[0084] Example 11

[0085] This embodiment provides a normalized log generation device based on the principle of entropy increase, including: a preprocessing module, a judgment module, a setting module, and a transformation module; the preprocessing module is used to preprocess the current log to obtain a preliminary normalized structure; the judgment module is used to calculate the entropy value of each field in the preliminary normalized structure based on the principle of entropy increase; and to determine whether a field is an important field or an unimportant field by using the relationship between the entropy value of each field and a set threshold; the setting module is used to set the associated primary key corresponding to the current log, which can represent the unique mapping relationship between all important fields and all unimportant fields in the current log; the transformation module is used to transform the preliminary normalized structure of the current log into a log normalized structure including important fields, the associated primary key, and unimportant fields.

[0086] The division of modules in the above-described normalized log generation device based on the principle of entropy increase is only for illustrative purposes. In other embodiments, the normalized log generation device based on the principle of entropy increase can be divided into different units as needed to complete all or part of the functions of the above-described normalized log generation device based on the principle of entropy increase.

[0087] Specific limitations regarding the normalized log generation device based on the entropy increase principle can be found in the limitations of the normalized log generation method based on the entropy increase principle above, and will not be repeated here. Each module in the aforementioned normalized log generation device based on the entropy increase principle can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each unit.

[0088] Example 12

[0089] This embodiment provides a database system, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a normalized log generation method based on the principle of entropy increase.

[0090] The database system includes a processor and memory connected via a system bus. The processor provides computational and control capabilities to support the operation of the entire database system. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. These computer programs can be executed by the processor to implement the normalized log generation method based on the entropy increase principle provided in the various embodiments described above. The internal memory provides a cached runtime environment for the operating system computer programs in the non-volatile storage media.

[0091] Example 13

[0092] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a standardized log generation method based on the principle of entropy increase.

[0093] Any references to memory, storage, databases, or other media used in this embodiment may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0094] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A standardized log generation method based on the principle of entropy increase, characterized in that, include: S1: Preprocess the collected current logs to obtain a preliminary normalized structure; S2: Calculate the entropy value of each field in the preliminary normalized structure based on the principle of entropy increase; By using the relationship between the entropy value corresponding to each field and a set threshold, it is determined whether it is an important field or an unimportant field; S3: Set the associated primary key corresponding to the current log, wherein the associated primary key can represent the unique mapping relationship between all the important fields and all the non-important fields in the current log; S4: Transform the preliminary normalized structure corresponding to the current log into a log normalized structure that includes important fields, related primary keys, and non-important fields.

2. The standardized log generation method based on the entropy increase principle as described in claim 1, characterized in that, S1 includes: S11: Divide the original current log into multiple segments to obtain the first log format; S12: Remove redundancy from the first log format to obtain the second log format; S13: Parse the second log format into a tree structure, traverse the tree structure, and obtain the preliminary normalized structure based on the content of each node.

3. The standardized log generation method based on the principle of entropy increase as described in claim 2, characterized in that, S11 includes: The original current log is divided into the first log format by using spaces as delimiters and dividing it according to the token format of each word; The "Word" is determined semantically and can be one or more of the following: English letters, English words, English phrases, Chinese characters, and Chinese phrases.

4. The standardized log generation method based on the entropy increase principle as described in claim 2, characterized in that, S12 includes: Redundancy is removed by deleting irrelevant characters from the first log format using regular expressions, resulting in the second log format.

5. The standardized log generation method based on the entropy increase principle as described in claim 2, characterized in that, S13 includes: The Drain3 log parsing tool is used to parse the English text in the second log format in a mixed manner, so as to parse the current log into a tree structure; the tree structure is traversed and the preliminary normalized structure is obtained based on the content of each node.

6. The standardized log generation method based on the entropy increase principle as described in claim 1, characterized in that, S2 includes: S21: Calculate the difference score for each field in the preliminary normalized structure corresponding to the current log, and calculate the entropy value for each field based on the difference score; S22: Determine whether a field is important or unimportant by using the relationship between the entropy value corresponding to each field and a set threshold.

7. The standardized log generation method based on the entropy increase principle as described in claim 6, characterized in that, S21 includes: S211: Calculate the cumulative number of different values ​​for each field in the current log, starting from the first log entry; S212: Calculate the Euclidean distance between different values ​​in each field using the cumulative number of different values ​​in the current log. ;Utilize the Euclidean distance between different values ​​under each field Calculate the differential score corresponding to the field; S213: The corresponding entropy value is obtained by dividing the differential score of each field by the cumulative number n of different values ​​of the field.

8. The standardized log generation method based on the entropy increase principle as described in claim 7, characterized in that, S212 includes: Using formula Calculate the Euclidean distance between different values ​​in each field of the current log. ; , For different values ​​under a field, and n represents the cumulative number of occurrences of different values ​​in the field. There are p in total. n* ( n-1 ) / 2; Using formula Calculate the differential score corresponding to the t-th field. ; This represents the Euclidean distance corresponding to the k-th combination under the t-th field.

9. The standardized log generation method based on the entropy increase principle as described in claim 6, characterized in that, S22 includes: Sort the entropy values ​​corresponding to each field in the current log. Fields with entropy values ​​greater than the set threshold are considered important fields; fields with entropy values ​​less than or equal to the set threshold are considered unimportant fields.

10. The standardized log generation method based on the entropy increase principle as described in any one of claims 1-9, characterized in that, The associated primary key includes one or more of the following: event ID, event reception time, time name, device name, and generalization strategy name.

Images