Log analysis method and device, electronic device and storage medium

Abstract

The application relates to a log analysis method and device, an electronic device and a storage medium, wherein the log analysis method comprises the following steps: obtaining original log data; processing a preset analysis requirement into an analysis rule for the original log data; based on a preset database syntax of a ClickHouse database, converting the analysis rule into a materialized view, and using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data. The application uses the ClickHouse database to realize analysis on the original log data, so that real-time analysis on the original log data can be completed without relying on an additional analysis system, and the utilization rate of the performance of the ClickHouse database can be improved, thereby reducing the development cost and maintenance cost of a current log audit system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of log analysis, and in particular to a log analysis method and device, an electronic device and a storage medium. BACKGROUND

[0002] As a high-performance, easy-to-expand relational database management system, ClickHouse can realize instant query of massive data, and thus has been widely used in OLAP (on-Line Analytic Processing) solutions since it was open-sourced.

[0003] At present, in the application of ClickHouse, the data analysis system with ClickHouse as the main body lacks the ability of data conversion, cannot control the data quality before storage and analysis, and cannot realize the analysis of log data. Therefore, the original log data collected is often sent to an additional log analysis system, which parses the logs of different data sources according to different rules to structured data that can distinguish different fields. Then the parsed structured data is inserted into the ClickHouse database table. At present, this way of using an additional log analysis system to analyze the original log data results in high development and operation costs of the log audit system.

[0004] At present, there is no effective solution to the problem of high technical and operation costs in the related art using an additional log analysis system to analyze the original log data. SUMMARY

[0005] A log analysis method, device, electronic device and storage medium are provided in the present embodiment to solve the problem of high technical and operation costs in the related art using an additional log analysis system to analyze the original log data.

[0006] In a first aspect, a log analysis method is provided in the present embodiment for a ClickHouse database, comprising:

[0007] obtaining original log data;

[0008] processing a preset analysis requirement into an analysis rule for the original log data;

[0009] based on a preset database syntax of the ClickHouse database, converting the analysis rule into a materialized view, and using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data.

[0010] In some embodiments, the obtaining the original log data comprises:

[0011] The original log data received by a message component of the ClickHouse database from a preset data source is obtained.

[0012] In some embodiments, the parsing rule is a tree structure composed of type nodes; and the processing of the preset parsing requirement into a parsing rule for the original log data comprises:

[0013] According to the preset parsing requirement, the type nodes and the combination relationship between the type nodes are determined; wherein the type nodes indicate at least one processing logic of matching replacement, matching extraction, and conditional judgment on the original log data;

[0014] The type nodes are combined into a tree structure according to the determined combination relationship to obtain the parsing rule.

[0015] In some embodiments, the determining the type nodes and the combination relationship between the type nodes according to the preset parsing requirement comprises:

[0016] According to the preset parsing requirement, a regular expression and / or a logical expression for data processing of the original log data are determined, and a type node is generated based on the regular expression and / or the logical expression;

[0017] The combination relationship between the type nodes is determined based on the preset parsing requirement.

[0018] In some embodiments, the converting the parsing rule into a materialized view based on the database syntax preset by the ClickHouse database comprises:

[0019] The parsing rule is traversed according to the tree structure of the parsing rule, and in the traversal process, the type nodes in the parsing rule are converted into materialized view segments based on the database syntax preset by the ClickHouse database in turn;

[0020] After the traversal ends, all the materialized view segments are spliced to obtain the materialized view.

[0021] In some embodiments, the real-time parsing of the original log data using the materialized view to obtain the parsing result of the original log data comprises:

[0022] The original log data is filtered and parsed in real time based on the database statements in the materialized view to obtain the parsing result of the original log data.

[0023] In some embodiments, after obtaining the parsing result of the original log data, the method further comprises:

[0024] writing the parsing result into a pre-established parsing result table of the ClickHouse database for storage.

[0025] In a second aspect, a log parsing device for a ClickHouse database is provided in the embodiments, comprising an acquisition module, a processing module, and a parsing module, wherein:

[0026] The acquisition module is configured to acquire original log data.

[0027] The processing module is configured to process a preset parsing requirement into a parsing rule for the original log data.

[0028] The parsing module is configured to convert the parsing rule into a materialized view based on a preset database syntax of the ClickHouse database, and perform real-time parsing on the original log data by using the materialized view to obtain a parsing result of the original log data.

[0029] In a third aspect, an electronic device is provided in the embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the log parsing method of the first aspect when executing the computer program.

[0030] In a fourth aspect, a storage medium is provided in the embodiments, and the storage medium stores a computer program executable by a processor, and the computer program implements the log parsing method of the first aspect when executed by the processor.

[0031] Compared with related technologies, the log parsing method, device, electronic device, and storage medium provided in the embodiments acquire original log data, process a preset parsing requirement into a parsing rule for the original log data, convert the parsing rule into a materialized view based on a preset database syntax of a ClickHouse database, and perform real-time parsing on the original log data by using the materialized view to obtain a parsing result of the original log data. The log parsing method provided in the embodiments utilizes the ClickHouse database to realize the parsing of the original log data, thereby being able to complete the real-time parsing of the original log data without relying on an additional parsing system, and further being able to improve the utilization rate of the performance of the ClickHouse database, thereby reducing the development cost and maintenance cost of the current log audit system.

[0032] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application, illustrate embodiments of the application, and together with the description serve to explain the application. In the drawings:

[0034] Figure 1 is a hardware structure block diagram of a terminal of the log analysis method of the present embodiment;

[0035] Figure 2 is a flow chart of the log analysis method of the present embodiment;

[0036] Figure 3 is a flow chart of the log analysis method of the present preferred embodiment;

[0037] Figure 4 is a data flow direction schematic diagram in the log analysis process;

[0038] Figure 5 is a structure block diagram of the log analysis device of the present embodiment. DETAILED DESCRIPTION

[0039] In order to more clearly understand the objects, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0040] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the same meaning as those commonly understood by a person of ordinary skill in the art to which the present application belongs. The terms "one", "a", "an", "the", "these", and similar terms in the present application do not indicate quantity of limitation, and they can be singular or plural. The terms "include", "contain", "have", and any variants thereof in the present application are intended to cover non-exclusive inclusion; for example, a process, method, and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. The terms "connect", "connected", "couple" and similar terms in the present application are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. The term "multiple" in the present application refers to two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. Generally, the character " / " represents an "or" relationship between the associated objects. The terms "first", "second", "third" and the like in the present application are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0041] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure block diagram of the terminal of the log analysis method of the present embodiment. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processor 102 and memory 104 for storing data, wherein the processor 102 can include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA. The above terminal can also include a transmission device 106 for communication function and an input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above terminal. For example, the terminal can include more or less components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0042] The memory 104 can be configured to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the log analysis method in the present embodiment. The processor 102 can execute various functional applications and data processing, i.e., implement the method described above, by running the computer programs stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely disposed relative to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0043] The transmission device 106 is configured to receive or send data via a network. The network described above includes a wireless network provided by a communication provider of the terminal. In an example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module, which is configured to communicate with the Internet in a wireless manner.

[0044] In the present embodiment, a log analysis method is provided, Figure 2 A flowchart of the log analysis method of the present embodiment is shown in FIG. 2, which includes the following steps: Figure 2

[0045] In step S210, raw log data is obtained.

[0046] The raw log data can be data received in real time from a data source system by a kafka (distributed publish / subscribe message system) engine table of a ClickHouse database. Specifically, the kafka engine table can be created by the following database table creation statement, and the raw log data sent by other systems or devices through kafka can be received. The statement for creating the kafka engine table can be as follows:

[0047]

[0048] ENGINE=Kafka('kafka address list', 'kafka topic', 'consumer group', 'data format');

[0049] ​Wherein, after filling in the correct Kafka-related parameters, such as the list of Kafka addresses, Kafka topics, consumer groups, and data formats, the above statement can be executed in the ClickHouse database to establish a Kafka engine table, and the table name is rawEvent_Kafka.

[0050] Exemplarily, when the original log data is the log data of an online management system, the log format can be: "event time principal action: behavior type [key1: value1 key2: value2...]". The online management system sends 2 pieces of original log data at time "2020-02-02 02:00:00", which are recorded as E1 and E2 respectively, wherein the original log data E1 is:

[0051] 2020-02-02 01:01:01 admin action:mail[from:a@mail.com to:b@mail.com subj:mail subject];

[0052] Wherein the original log data E2 is:

[0053] 2020-02-02 01:10:01 Tom action:web[url:www.foo.com ip:123.123.123.123].

[0054] The existence form of the above two pieces of original log data in the kafka engine table is shown in the following table:

[0055]

[0056]

[0057] Additionally, after receiving the original log data, a local table for storing the original log data can be established, and the original log data in the kafka engine table can be written into the local table by using the materialized view for writing the original log data. Exemplarily, the local table can be established based on the following database table creation statement:

[0058]

[0059] ENGINE=MergeTree()

[0060] PARTITION BY toYYYYMMDD(collectTime)

[0061] ORDER BY(sourceIP,collectTime)

[0062] SETTINGS index_granularity = 8192;

[0063] Afterwards, the above materialized view for writing original log data is established by the following database statement, and the data in the kafka engine table is inserted into the local table in real time:

[0064] CREATE MATERIALIZED VIEW IF NOT EXISTS view_rawEvent_local

[0065] TO rawEvent_local AS

[0066] SELECT sourceIP, toDateTime(collectTime), rawEvent

[0067] FROM rawEvent_Kafka;

[0068] Wherein the name of the materialized view for writing original log data is view_rawEvent_local, and the materialized view will monitor the original log data in the kafka engine table in real time, and insert the incremental data therein into the local table rawEvent_local.

[0069] So far, the embodiment has completed the insertion of the original log data from the kafka engine table to the local table based on the above operation. In addition, before the original log data is parsed, the embodiment can also establish a parsing result table based on the preset parsing requirement. The parsing result table is used to store the parsing results of the original log data. Wherein, according to the parsing requirement, for example, the target of the parsing task, the fields that need to be parsed can be determined, and in the parsing result table, each field corresponds to a column in the parsing result table. Taking the original log data of the above network management system as an example, the fields in the corresponding parsing result table can include: event time, principal, behavior type, mail sender, mail recipient, mail subject, website domain name, website IP, etc. Exemplarily, the SQL (Structured Query Language, structured query language) table creation statement of the parsing result table can be:

[0070]

[0071] ENGINE=MergeTree()

[0072] PARTITION BY toYYYYMMDD(collectTime)

[0073] ORDER BY (sourceIP, collectTime)

[0074] SETTINGS index_granularity=8192;

[0075] Those skilled in the art can understand that the above table building statement, original log data, log format and other information are examples provided for describing the method of the embodiment and do not constitute a specific limitation on the embodiment.

[0076] In step S220, the preset analysis requirement is processed into an analysis rule for the original log data.

[0077] The corresponding analysis rule can be determined based on the preset analysis requirement. The specific processing logic for analyzing and processing the original log data in the analysis rule can be determined based on the preset analysis requirement, and the processing logic can be represented by one or more of a regular expression and a logical expression. Preferably, a regular expression for string replacement of the original log data can be set in the analysis rule to replace a specific form of string therein; a regular expression for string matching of the original log data can also be set in the analysis rule to filter out the matched string; and a logical expression can be set in the analysis rule to conditionally judge the string in the original log data. In addition, the order of the above different processing logics can also be set to obtain the analysis rule that meets the preset analysis requirement.

[0078] Further, the analysis rule can be a tree structure with several types of nodes. For the sake of convenience of description, the analysis rule of the tree structure is referred to as an analysis tree in the following. The analysis tree can have multiple different types of type nodes. Preferably, the analysis tree can include the following four types of nodes: a root node, a matching and replacement node, a matching and extraction node, and a conditional judgment node. The root node is the starting node of the analysis process, takes the original log data as input, does not perform any transformation on the original log data, and outputs the original log data. The matching and replacement node takes the parent node output as node input, uses a regular expression to match a specific form of string in the input content, replaces it with a specified string, and outputs the result of the entire input string after replacement. The matching and extraction node takes the parent node output as node input, uses a regular expression to match a specific form of string in the input content, and outputs the matched string result as node output. The conditional judgment node takes the parent node output as node input, uses a logical expression to operate on the input. When the operation result is "true", the input content is taken as output, otherwise an empty string is taken as output.

[0079] Additionally, in the present embodiment, the user can also be provided with a setting operation of the parsing rule in the form of a user interaction interface. In the interaction interface, the user can select the required type node from the above type nodes and set the input and output relationship thereof, so as to generate the parsing tree. For example, the parsing requirement can be processed into the parsing rule in response to the following operation of the user in the above interaction interface: selecting the output of a certain type node as the input of the current type node, selecting the type of the type node, inputting the name of the type node, setting the corresponding regular expression or logical expression in the type node, and the like. After the parsing is completed, the user can also be shown the structure of the parsing tree.

[0080] In step S230, based on the database syntax preset in the ClickHouse database, the parsing rule is converted into a materialized view, and the original log data is parsed in real time by using the materialized view to obtain the parsing result of the original log data.

[0081] Specifically, based on the database syntax in the ClickHouse database, each processing logic in the parsing rule can be converted into a corresponding materialized view segment in sequence according to the order of the processing logic, and then the materialized view segments are spliced to obtain the materialized view. Then, the original log data in the local table is filtered and parsed based on the materialized view, and the parsing result is inserted into the pre-established parsing result table. It can be understood that the materialized view converted based on the parsing rule in the present step is a materialized view for parsing and filtering the original log data, which is different from the materialized view for writing the original log data in the above step S210.

[0082] Preferably, based on the tree structure of the parsing tree, each type node of the above parsing tree is traversed, and each type node is converted into a corresponding SQL statement segment in sequence. Then, the SQL statement segments converted from the type nodes are combined to generate a complete ClickHouse materialized view definition statement, that is, the above materialized view for parsing the original log data. Next, based on the ClickHouse materialized view definition statement, the original log data in the above local table is filtered and parsed in real time, and the parsing result is inserted into the parsing result table.

[0083] In the related art, the original log data needs to be parsed in advance by using an additional parsing system or component, and the parsed fields are inserted into the ClickHouse database. It depends on high-quality original log data, or needs to be matched with an additional parsing system, thereby causing a high development and maintenance cost of the log auditing system, and the performance of the ClickHouse database cannot be fully utilized. The log parsing method provided in the embodiment can be implemented based on the materialized view of the ClickHouse in the entire parsing process, without depending on other big data components or parsing systems, thereby being capable of utilizing the performance of the ClickHouse database. Compared with the related art, the log auditing system development and maintenance cost can be reduced.

[0084] The steps S210 to S230 described above, obtaining the original log data; processing the preset parsing requirement as the parsing rule for the original log data; converting the parsing rule into the materialized view based on the preset database syntax of the ClickHouse database, and utilizing the materialized view to perform real-time parsing on the original log data to obtain the parsing result of the original log data. The ClickHouse database is utilized to implement the parsing of the original log data, thereby being capable of completing the real-time parsing of the original log data without depending on an additional parsing system, and thereby being capable of improving the utilization of the performance of the ClickHouse database, thereby reducing the development cost and maintenance cost of the current log auditing system.

[0085] In one embodiment, based on the step S210 described above, obtaining the original log data, specifically can include the following steps:

[0086] Step S211, obtaining the original log data received by the message component of the ClickHouse database from the preset data source.

[0087] Specifically, the message component can be a kafka engine table of the ClickHouse database. In the embodiment, the kafka engine table is utilized to receive the original log data sent from the data source system in real time, and the original log data in the kafka engine table is written into the local table by using the materialized view written for the original log data.

[0088] In addition, in one embodiment, based on the step S220 described above, the parsing rule is a tree structure composed of type nodes; processing the preset parsing requirement as the parsing rule for the original log data, specifically can include the following steps:

[0089] Step S221, determining the type nodes and the combination relationship between the type nodes according to the preset parsing requirement; wherein the type node indicates at least one processing logic of matching replacement, matching extraction, and condition judgment on the original log data.

[0090] Specifically, the type nodes can be as follows: root node, match-replace node, match-extract node, and condition-judge node. The root node is a starting node of the parsing process, takes the original log data as input, does not transform the original log data, and outputs the original log data.

[0091] The match-replace node matches a specific form of string in the input content by using a regular expression, replaces the specific form of string with a specified string, and outputs the result of replacing the entire input string. The match-replace node can be denoted as R1, and the match-replace node R1 can be in the form of: output = REPLACE(input, pattern, target). The input and the output represent the node input and the node output, respectively. The pattern represents a template for matching, and can be a regular expression. The target represents a string to be replaced, and can be an empty string.

[0092] The match-extract node matches a specific form of string in the input content by using a regular expression, and outputs the matched string as the node output. The match-extract node can be denoted as R2, and the match-extract node R2 can be in the form of: output = EXTRACT(input, pattern). The pattern represents a template for extraction, and can be a regular expression.

[0093] The condition-judge node performs an operation on the input by using a logical expression. When the operation result is "true", the input content is taken as the output, otherwise an empty string is taken as the output. The condition-judge node can be denoted as R3, and the condition-judge node R3 can be in the form of: output = IF(input, condition). The condition represents a logical expression about any established type node.

[0094] In addition, the combination relationship can be an input-output relationship between different type nodes. According to the predetermined parsing requirement, the embodiment determines the required type nodes and the combination relationship between the type nodes, so as to generate a parsing rule for the original log data that meets the parsing requirement.

[0095] In step S222, the type nodes are combined into a tree structure according to the determined combination relationship, to obtain a parsing rule. After the type nodes and the combination relationship between the type nodes are determined, the required type nodes can be combined according to the determined combination relationship, to form a parsing rule in a tree structure.

[0096] Further, in an embodiment, based on the step S221, determining the type nodes and the combination relationship between the type nodes according to the preset parsing requirement can include: determining the regular expression and / or the logical expression for data processing of the original log according to the preset parsing requirement, and generating the type nodes based on the regular expression and / or the logical expression; determining the combination relationship between the type nodes based on the preset parsing requirement.

[0097] For example, an interactive interface can be provided for the user to select the type nodes, set the regular expression and / or the logical expression, and combine the input / output relationship between the type nodes on the interactive interface. Wherein, the user can determine the input / output relationship by setting one type node as the parent node of another type node. The following table is the parsing rule obtained based on the user interactive interface. As shown in the following table, the content of the parsing rule can include:

[0098]

[0099] Wherein, different type nodes can be combined based on the node number and the parent node of each node, so as to obtain the parsing rule.

[0100] In addition, in an embodiment, based on the step S230, the parsing rule is converted into a materialized view based on the database syntax preset by the ClickHouse database, which can include the following steps:

[0101] Step S231, the parsing rule is traversed according to the tree structure of the parsing rule, and in the traversal process, the type nodes in the parsing rule are converted into the materialized view fragments based on the database syntax preset by the ClickHouse database.

[0102] Wherein, based on the tree structure of the parsing rule, each type node in the parsing rule can be traversed in turn, and in the traversal process, the corresponding SQL statement fragment is generated based on the regular expression or the logical expression in the corresponding type node and the database syntax preset by the ClickHouse database. The SQL statement fragment is the materialized view fragment corresponding to the type node. After the traversal is completed, the above-mentioned SQL statement fragments are combined based on the combination relationship between the type nodes, so as to obtain the complete materialized view. The complete materialized view is the materialized view for parsing the original log data.

[0103] For example, after obtaining the parsing rule in the step S222, each type node in the parsing rule can be converted into the SQL statement fragment as shown in the following table:

[0104] For example, after obtaining the parsing rule in the step S222, each type node in the parsing rule can be converted into the SQL statement fragment as shown in the following table:

[0105]

[0106] Step S232, after the end of the traversal, all the materialized view picture segments are spliced to obtain the materialized view.

[0107] That is, the SQL statement fragments of each type of node in step S231 can be combined to generate a complete ClickHouse materialized view definition statement. The specific ClickHouse materialized view definition statement can be:

[0108]

[0109]

[0110] Based on the above materialized view definition statement, the above materialized view can filter and parse the original log data from the local table rawEvent_local in real time, and then insert it into the parsed result table onlineBehavior. Thus, the filtering and parsing of the original log data in the ClickHouse database are realized.

[0111] In addition, in one embodiment, based on the above step S230, the real-time parsing of the original log data by the materialized view to obtain the parsed result of the original log data can further include the following steps:

[0112] Step S233, based on the database statement in the materialized view, the original log data is filtered and parsed in real time to obtain the parsed result of the original log data.

[0113] In addition, in one embodiment, after obtaining the parsed result of the original log data, the log parsing method can further include the following steps:

[0114] Step S240, the parsed result is written into the parsed result table of the ClickHouse database to be stored.

[0115] The preferred embodiment will be described and explained below.

[0116] Figure 3 is a flowchart of the log parsing method of the preferred embodiment. As shown in Figure 3 , the log parsing method includes the following steps:

[0117] Step S301, receiving the original log data through the ClickHouse kafka engine table;

[0118] Step S302, using the materialized view for data writing to transfer the original log data of the Kafka engine table to the local table;

[0119] Step S303, a parsing result table is established, which is used to store the parsing result of the original log data;

[0120] Step S304, the parsing rule is designed by using user interaction decoding;

[0121] Step S305, the designed parsing rule is converted into a ClickHouse materialized view;

[0122] Step S306, the original log data in the local table is parsed based on the ClickHouse materialized view converted from the parsing rule, and the parsing result is stored in the parsing result table.

[0123] Further, Figure 4 For the data flow direction in the log parsing process. Based on the above steps S301 to S306, the data flow direction of the original log data in the above preferred embodiment can be obtained as shown in Figure 4 . Wherein, the original log data is written into the local table by the kafka engine table through the first materialized view for writing the original log data. Then, the original log data in the local table is filtered and parsed in real time based on the second materialized view for parsing the original log data, and the parsing result is stored in the corresponding parsing result table.

[0124] In this embodiment, a log parsing device is also provided, which is used to implement the above embodiments and preferred embodiments, and the description of which has been made. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that can realize the predetermined function. Although the device described in the following embodiments is preferably realized in software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0125] Figure 5 is a structure block diagram of the log parsing device 50 of the present embodiment, as shown in Figure 5 , the log parsing device 50 comprises an acquisition module 52, a processing module 54, and a parsing module 56; wherein:

[0126] The acquisition module 52 is used to acquire the original log data;

[0127] The processing module 54 is used to process the preset parsing requirement into the parsing rule for the original log data;

[0128] The parsing module 56 is used to convert the parsing rule into a materialized view based on the database syntax of the ClickHouse database, and to parse the original log data in real time by using the materialized view to obtain the parsing result of the original log data.

[0129] The log analysis device 50 obtains original log data, processes a preset analysis requirement into an analysis rule for the original log data, converts the analysis rule into a materialized view based on a preset database syntax of a ClickHouse database, and performs real-time analysis on the original log data by using the materialized view to obtain an analysis result of the original log data. The ClickHouse database is used to analyze the original log data, so that real-time analysis of the original log data can be completed without relying on an additional analysis system, and the utilization rate of the ClickHouse database performance can be improved, thereby reducing the development cost and maintenance cost of a current log audit system.

[0130] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor, or each of the above modules can be located in different processors in any combination.

[0131] In the embodiment, an electronic device is also provided, including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0132] Optionally, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] Optionally, in the embodiment, the processor can be configured to execute the following steps by the computer program:

[0134] S1, obtaining original log data;

[0135] S2, processing a preset analysis requirement into an analysis rule for the original log data;

[0136] S3, converting the analysis rule into a materialized view based on a preset database syntax of a ClickHouse database, and performing real-time analysis on the original log data by using the materialized view to obtain an analysis result of the original log data.

[0137] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, which will not be described herein again.

[0138] In addition, in combination with the log analysis method provided in the above-mentioned embodiments, a storage medium can also be provided to implement the log analysis method in the present embodiment. The storage medium has a computer program stored thereon. The computer program is executed by a processor to implement any one of the log analysis methods in the above-mentioned embodiments.

[0139] It should be understood that the specific embodiments described herein are merely exemplary and are not intended to limit the application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0140] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0141] Obviously, the drawings are only some examples or embodiments of the present application, and those of ordinary skill in the art can also apply the present application to other similar situations according to the drawings without creative efforts. In addition, it can be understood that although the work done in the development process may be complex and long, some design, manufacture or production changes made by those of ordinary skill in the art according to the technical content disclosed in the present application are only routine technical means and should not be regarded as insufficient disclosure of the present application.

[0142] The term "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiments can be included in at least one embodiment of the present application. The presence of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. Those of ordinary skill in the art can clearly or implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0143] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of patent protection. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all fall within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A log parsing method for a ClickHouse database, characterized in that, The method comprises: obtaining original log data; processing a preset analysis requirement into an analysis rule for the original log data, the analysis rule being a tree structure composed of type nodes; based on a preset database syntax of the ClickHouse database, converting the analysis rule into a materialized view, comprising: based on the tree structure of the analysis rule, sequentially traversing each type node in the analysis rule, and in the traversal process, based on the regular expression or logical expression in the corresponding type node and the preset database syntax of the ClickHouse database, generating a corresponding SQL statement fragment; the SQL statement fragment is a materialized view fragment corresponding to the type node; after the traversal is completed, combining the SQL statement fragments based on the combination relationship between the type nodes to obtain a complete materialized view; using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data.

2. The log analysis method according to claim 1, characterized in that, The method comprises: obtaining original log data received by a message component of the ClickHouse database from a preset data source.

3. The log parsing method of claim 1, wherein: The method comprises: determining the type nodes and the combination relationship between the type nodes according to the preset analysis requirement; wherein the type nodes indicate at least one processing logic of matching and replacing, matching and extracting, and condition judging on the original log data; combining the type nodes into a tree structure according to the determined combination relationship to obtain the analysis rule.

4. The log analysis method according to claim 3, characterized in that, The method comprises: determining regular expressions and / or logical expressions for data processing on the original log data according to the preset analysis requirement, and generating type nodes based on the regular expressions and / or logical expressions; determining the combination relationship between the type nodes based on the preset analysis requirement.

5. The log analysis method of claim 1, wherein, The method comprises: based on the database statements in the materialized view, performing real-time filtering and real-time analysis on the original log data to obtain an analysis result of the original log data.

6. The log parsing method according to any one of claims 1 to 5, characterized in that, After obtaining the analysis result of the original log data, the method further comprises: writing the analysis result into a pre-established analysis result table in the ClickHouse database for storage.

7. A log parsing device for a ClickHouse database, characterized in that, The method comprises: an obtaining module, a processing module, and an analysis module; wherein: the obtaining module is configured to obtain original log data; the processing module is configured to process a preset analysis requirement into an analysis rule for the original log data, the analysis rule being a tree structure composed of type nodes; the analysis module is configured to convert the analysis rule into a materialized view based on a preset database syntax of the ClickHouse database, comprising: Based on the tree structure of the parsing rule, each type node in the parsing rule is sequentially traversed, and in the traversal process, based on the regular expression or the logical expression in the corresponding type node and the database syntax preset by the ClickHouse database, a corresponding SQL statement fragment is generated; the SQL statement fragment is a materialized view fragment corresponding to the type node; After the traversal is completed, the SQL statement fragments are combined based on the combination relationship between the type nodes to obtain a complete materialized view; The original log data is parsed in real time by using the materialized view to obtain a parsing result of the original log data. 8.An electronic device comprising a memory and a processor, the electronic device comprising: The memory stores a computer program, and the processor is configured to run the computer program to execute the log parsing method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the log parsing method in any one of claims 1 to 6.

Images