Log template extraction method and device

Abstract

The present application discloses a log template extraction method and device, which belongs to the field of computer technology. The method comprises: deleting specified characters in each line of log records in multiple lines of log records of a log to obtain each line of log records after preprocessing; determining at least one first log record group based on each line of log records after preprocessing, and different first log record groups include different lines of log records in the log; extracting log templates in each of the at least one first log record group, and determining the log template of the log based on the extracted log template. The present application solves the problem of low accuracy of the currently determined log templates, and the present application is applied to log template extraction of logs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log template extraction method and device. Background Art

[0002] By adding some specific codes to the software source code, the real-time status of the software operation can be recorded in a text file, which is called a log or log file. Software developers (or operation and maintenance staff) can understand the real-time status of the software operation by reading the log.

[0003] A log includes multiple lines of log records (also called log statements). Each line of log records is used to record an event during software runtime. Log records in a log usually have an implicit log template (schema), that is, the pattern or format of the record itself. Based on the difference in log templates, logs can be divided into two categories: homogeneous logs and heterogeneous logs. Homogeneous logs means that the log templates of all log records in the log are the same; heterogeneous logs means that there is no unified log template for each line of log records in the log. Identifying the log template of a log can help users quickly understand the overall picture of system operation.

[0004] At present, for heterogeneous logs, the method of extracting log templates is as follows: tokenize the line of log records to obtain multiple entries (tokens); based on the tokenization results of each line of log records, calculate the similarity between each log record, and hierarchically cluster the log records in the log based on the similarity to obtain multiple types of log records; extract templates for each type of log record, and use the log templates of the obtained multiple types of log records as log templates for heterogeneous logs.

[0005] However, since the log contains a lot of content, it is easy to affect the accuracy of the final log template. Summary of the invention

[0006] The embodiment of the present application provides a log template extraction method and device, which can solve the problem of high computational cost of the current log template extraction method. The technical solution is as follows:

[0007] In a first aspect, a log template extraction method is provided, the method comprising:

[0008] Delete specified characters in each line of log records in multiple lines of log records of a log to obtain each line of log records after preprocessing; determine at least one first log record group based on each line of log records after preprocessing, and different first log record groups include different lines of log records in the log; extract log templates in each of the at least one first log record group respectively, and determine the log template of the log based on the extracted log templates.

[0009] In an embodiment of the present application, by deleting the specified characters in each line of log records in the multiple lines of log records, the computational complexity can be effectively reduced, and the interference of the specified characters in the log template extraction process can be avoided, thereby improving the progress of extracting the log template.

[0010] Optionally, deleting the specified characters in each line of log records in the multiple lines of log records to obtain each line of log records after preprocessing includes:

[0011] In an optional example, at least one entry of each line of log records in the log is obtained, and the specified character in each entry of each line of log records is deleted to obtain each line of log records after preprocessing;

[0012] In another optional example, designated characters in each line of the multiple lines of log records are deleted to obtain simplified log records, and at least one entry of each simplified log record of the multiple lines of log records is obtained to obtain the preprocessed log records.

[0013] The number of different first simplified hash codes is usually smaller than the number of different first hash codes, and the number of first log record groups obtained by grouping is correspondingly reduced. Therefore, by adopting the aforementioned another optional example, the number of hash buckets can be reduced to avoid overly dispersed distribution of hash buckets.

[0014] Optionally, the designated characters are numbers and / or punctuation marks. Since variable symbols in log records, such as timestamps and IP addresses, include a large number of numbers and punctuation marks, when the designated characters are numbers and / or punctuation marks, the interference of these variable symbols on log template extraction can be effectively reduced, thereby improving the accuracy of log template extraction.

[0015] Optionally, determining at least one first log record group based on each line of the preprocessed log record includes:

[0016] Concatenate m consecutive non-empty entries at the first specified position of each line of log record after preprocessing to obtain a first character signature, where m is a positive integer; determine the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after preprocessing; and group multiple lines of log records in the log based on the first digital signature of each line of log record to obtain at least one first log record group, wherein the first digital signatures corresponding to all log records included in each first log record group are the same.

[0017] In an embodiment of the present application, the first designated position is usually a pre-designated (usually user-designated) position, and the probability that the character content corresponding to it is a constant is greater than a preset threshold, that is, the character content corresponding to the first designated position has a greater probability of being a constant. Since the content of the front position in the log record is more likely to be a constant, optionally, the first designated position is a front position, and the m non-empty entries are the first m non-empty entries of each line of log record. However, in different application scenarios, the first designated position may be different in the log records of different logs, and the embodiment of the present application does not limit this.

[0018] Optionally, based on the first digital signature of each line of log records, multiple lines of log records in the log are grouped to obtain the at least one first log record group, including: concatenating n consecutive non-empty entries at the second designated position of each line of log records after preprocessing to obtain a second character signature, and the first designated position is different from the second designated position, and n is a positive integer; based on the first digital signature of each line of log records, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; based on the second digital signature of each line of log records, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

[0019] In this way, the grouping basis can be supplemented on the basis of the first digital signature to perform grouping, so that the hash collisions in the obtained first log record group are less and the grouping result is more accurate.

[0020] In the embodiment of the present application, the second designated position is usually a pre-designated (usually user-designated) position, and the probability that the character content corresponding to it is a constant is greater than a preset threshold, that is, the character content corresponding to the second designated position is probably a constant. As mentioned above, since the possibility that the content of the front position in the log record is a constant is high, the second designated position is optionally a front position, and the n non-empty entries are the adjacent n non-empty entries after the first m non-empty entries in each line of log record. However, in different application scenarios, the second designated position may be different in the log records of different logs, and the embodiment of the present application does not limit this.

[0021] Optionally, determining the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after the preprocessing includes:

[0022] The first hash code of the first character signature corresponding to each line of log record after the preprocessing is determined as the first digital signature of each line of log record; or, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

[0023] The simplification process may include remainder processing, modulo processing, or calculating a new hash code based on the obtained first hash code (the number of bits of the new hash code is less than the number of bits of the original hash code, thereby playing a simplification role). By simplifying the first hash code, the number of bits of the first simplified hash code obtained by processing can be reduced, thereby reducing the number of subsequent groupings and reducing the complexity of grouping.

[0024] Optionally, respectively extracting the log templates in each of the at least one first log record group, and determining the log template of the log based on the extracted log templates, includes:

[0025] For each line of log record in each of the first log record groups, an alternative log template is generated based on the log record and the historical log template of the first log record group; when the log record matches any of the generated alternative log templates, any of the alternative log templates is determined as the new log template of the first log record group; when the log record does not match all of the alternative log templates, all of the alternative log templates are rolled back to the corresponding historical log templates and the log record, and the log template extracted from the log record is added as the new log template of the first log record group.

[0026] In a second aspect, a log template extraction device is provided. The device may include at least one module. The at least one module may be used to implement the log template extraction method provided by the first aspect or various possible implementations of the first aspect.

[0027] In a third aspect, the present application provides a computer device, the computer device comprising a processor and a memory. The memory stores computer instructions; when the processor executes the computer instructions stored in the memory, the computer device executes the method provided by the first aspect or various possible implementations of the first aspect, so that the computer device deploys the log template extraction device provided by the second aspect or various possible implementations of the second aspect.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions instruct the computer device to execute the method provided by the above-mentioned first aspect or various possible implementations of the first aspect, or the computer instructions instruct the computer device to deploy the log template extraction device provided by the above-mentioned second aspect or various possible implementations of the second aspect.

[0029] In a fifth aspect, the present application provides a computer program product, the computer program product including computer instructions, the computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided by the first aspect or various possible implementations of the first aspect, so that the computer device deploys the log template extraction device provided by the second aspect or various possible implementations of the second aspect.

[0030] In a sixth aspect, an analysis system is provided, comprising: a terminal and an analysis device, wherein the analysis device comprises the log template extraction device described in the second aspect or various possible implementations of the second aspect or the computer device described in the third aspect.

[0031] In a seventh aspect, a chip is provided, which may include a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the template extraction method as described in any one of the first aspects.

[0032] In an embodiment of the present application, by deleting the specified characters in each line of log records in the multiple lines of log records, the computational complexity can be effectively reduced, and the interference of the specified characters in the log template extraction process can be avoided, thereby improving the progress of extracting the log template.

[0033] For example, since variable symbols in log records, such as timestamps and IP addresses, include a large number of numbers and punctuation marks, when the specified characters are numbers and / or punctuation marks, the interference of these variable symbols on log template extraction can be effectively reduced, thereby improving the accuracy of log template extraction.

[0034] When there are multiple first log record groups, the processing processes for each first log record group can be executed in parallel, thereby reducing computing delays. The amount of data required to calculate each time the processing process is executed is much smaller than the amount of data in the entire log, effectively reducing the computing cost and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of part of the log content in a log provided in an embodiment of the present application;

[0036] Figure 2 It is a schematic diagram of an application environment involved in a log template extraction method provided in an embodiment of the present application;

[0037] Figure 3 It is a schematic diagram of an application environment involved in another log template extraction method provided in an embodiment of the present application;

[0038] Figure 4 It is a flow chart of a log template extraction method provided in an embodiment of the present application;

[0039] Figure 5 This is a schematic diagram of a method of using space word segmentation provided in an embodiment of the present application;

[0040] Figure 6 This is a schematic diagram of a word segmentation result obtained by performing word segmentation using a space word segmentation method provided in an embodiment of the present application;

[0041] Figure 7 This is a schematic diagram of a word segmentation result obtained by performing word segmentation using special characters provided in an embodiment of the present application;

[0042] Figure 8 This is a schematic diagram of the result of deleting the specified characters after word segmentation in the log record X1 provided in the embodiment of the present application;

[0043] Fig. 9 is Figure 8 The log record X1 shown is a schematic diagram of the result of character concatenation based on deleting the specified characters after word segmentation;

[0044] Fig.10 This is a schematic diagram of a process for determining a first digital signature provided in an embodiment of the present application;

[0045] Fig.11is a schematic diagram of the grouping results of log records X3 to X6 provided in an embodiment of the present application;

[0046] Fig.12 is a schematic diagram of the grouping results of log records X7 and X8 provided in an embodiment of the present application;

[0047] Figures 13 to 16 A schematic diagram of a log template extraction process in a first log record group provided in an embodiment of the present application;

[0048] Fig.17 is a schematic diagram of a grouping result of a first log record group provided in an embodiment of the present application;

[0049] Fig.18 This is a log template schematic diagram provided in an embodiment of the present application;

[0050] Fig.19 is a schematic diagram of a determination module provided in an embodiment of the present application;

[0051] Fig. 20 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] The log is used to record the real-time status of the software operation. By analyzing the log, the real-time situation of the software operation can be grasped, or the log abnormality detection can be performed. In the embodiment of the present application, the scenarios of log analysis include offline analysis scenarios and online analysis scenarios. In the offline analysis scenario, the log data to be analyzed can be batch log data, such as log files, or log data obtained by querying in the log database; in the online analysis scenario, the log data to be analyzed can be real-time log data, also known as log stream data. Among them, the log file is usually a file downloaded by a user, software developer or operation and maintenance staff, or a file obtained by keyword search.

[0054] like Figure 1 As shown, Figure 1It is a schematic diagram of part of the log content in a log. The log includes multiple lines of log records (also called log text). Each line (also called each log record) is used to record an event when the software is running. Each line of log record consists of multiple characters, which may include letters and / or symbols. A line of log record includes a constant part and a variable part. The constant part includes at least one character, and the variable part includes at least one character (or, in some special cases, the variable part is an empty character, that is, there is no content). For example, suppose the pseudo code corresponding to a log record is defined as recording user login information: "logging.info('User%d login at%s',$uid,$time)", the "$" in the pseudo code corresponding to the log record is used to define the variable part (also called variable name). For example, "$IP" means the Internet Protocol (IP) address. It can be seen from the pseudo code corresponding to the log record that the log record contains two sets of variable parts, which are used to record the user name (uid) and login time (time) of each login.

[0055] Log records in a log usually have an implicit log template (also called a log pattern), which refers to a standard style or fixed format for generating log records in a log. For example, after the pseudo code corresponding to the aforementioned log record is actually run, it outputs a multi-line log record in the log for recording user login information. In the embodiment of the present application, the log where the multi-line log record is located is called the first log:

[0056] "User 025862 login at 2018-12-03 02:03:00

[0057] User 045210 login at 2018-12-04 02:03:15

[0058] User 033658 login at 2018-12-05 02:03:38

[0059] User 010100 login at 2018-12-06 02:04:06

[0060] User 023025 login at 2018-12-07 02:04:51

[0061] User 046523 login at 2018-12-08 02:05:22".

[0062] The log template of a log is the log template of the log records in the log. Usually, when extracting the log template of a log record, if the variable part of the log record is identified, the variable part will be marked with a preset variable identifier. The marking method is essentially to replace the variable part with the variable identifier. The variable identifier is usually a wildcard "*". For example, when extracting the log template of multiple lines of log records in the first log above, the variable part can be replaced with a wildcard "*", and the log template of each line of log record obtained is "User*login at*", then the log template of the first log is "User*login at*". It is worth noting that in the matching process in the following text, the variable identifier can be judged to be the same as any character or entry. For example, when matching "*" with "046523", it can be determined that "*" is the same as "046523".

[0063] Logs can be divided into two categories: homogeneous logs and heterogeneous logs. In the first log of the aforementioned embodiment, a homogeneous log is used as an example for explanation. The log records in the homogeneous log have a unified log template. The extraction process of the log template can be implemented by regular expressions, while the multi-line log records of the heterogeneous log do not have a unified log template. Usually, business logs are heterogeneous logs. For example, part of the content of the heterogeneous log is shown below (the "#" in the heterogeneous log is used to mark the line number. In actual applications, the content of the log may not include the line number mark and the specific line number. This example is mainly for the convenience of readers to understand. Therefore, the line number mark and the specific line number are ignored in the subsequent processing process). In the embodiment of the present application, the log containing the multi-line log records shown below is called the second log:

[0064] "#0mod_jk child workerEnv in error state 6

[0065] #1jk2_init()Found child 6740 in scoreboard slot 7

[0066] #2jk2_init()Found child 6741 in scoreboard slot 8

[0067] #3workerEnv.init()ok / etc / httpd / conf / workers2.properties

[0068] #4mod_jk child workerEnv in error state 7

[0069] #5workerEnv.init()ok / etc / httpd / conf / workers2.properties”.

[0070] Since there is no unified log template for each line of log record, the extraction process is more complicated. In addition, since there are many contents recorded in the log, it is easy to affect the accuracy of the final log template.

[0071] The present application embodiment provides a log template extraction method, which can reduce the computational cost in the log template extraction process. Figure 2 , Figure 2 1 is a schematic diagram of an application environment involved in a log template extraction method provided in an embodiment of the present application. The application environment includes a terminal 110, an analysis device 120 and a network device 130.

[0072] The terminal 110 can be a device that can interact with a user, such as a display, a computer, a smart phone, a tablet computer, and a laptop computer. The analysis device 120 can be a server, or a server cluster composed of several servers, etc., which can perform data analysis. Optionally, the analysis device 120 can be a cloud server (also known as a cloud computing server), for example, a deep learning server for a deep learning service (DLS) provided. The terminal 110 establishes a wired or wireless communication connection with the analysis device 120 through a communication network. Among them, the network device 130 can be a device such as a sensor or a terminal that can run software and generate log data. The network device 130 is used to provide the data to be analyzed to the analysis device 120, the analysis device 120 is used to analyze the log data, and the terminal 110 is used to present the analysis results to the user. The communication network involved in the embodiment of the present application is a second generation (2-Generation, 2G) communication network, a third generation (3rd Generation, 3G) communication network, a long term evolution (Long Term Evolution, LTE) communication network or a fifth generation (5rd Generation, 5G) communication network, etc.

[0073] Optionally, the aforementioned application environment may also include a storage device, which is used to store the data required to be stored by the terminal 110, the analysis device 120 and / or the network device 130. The storage device may be a distributed storage device, and the terminal 110, the analysis device 120 and / or the network device 130 may read and write the data stored in the storage device. In this way, when there is a lot of data in the application scenario, the data is stored in the storage device, which can reduce the load of the analysis device and improve the data analysis efficiency of the analysis device. It should be noted that when the amount of data in the application environment is small, the storage device may not be set. In this case, the functions of the terminal 110 and the analysis device 120 may also be implemented by the same device, such as a computer.

[0074] like Figure 3 As shown, the application environment includes two parts, the foreground 201 and the background 202. The foreground 201 is used to present data to the user, receive data input by the user, and realize interaction with the user; the background 202 is used to interact with the foreground 201 for data, and perform management operations and / or data processing, etc. Among them, the foreground 201 can be deployed in the aforementioned terminal 110. The background 202 can be deployed in the aforementioned analysis device 120. For example, a client, a script or a browser can be installed in the terminal 110 to realize the deployment of the foreground 201. In this way, the terminal 110 can present the user interface in the form of a client interface, a terminal interface or a web page corresponding to the browser.

[0075] The log template extraction method provided in the embodiment of the present application can be used in scenarios such as software debugging, performance optimization or business analysis. Specifically, it can be applied in the anomaly detection scenario in these scenarios. Anomaly detection refers to the detection of patterns that do not meet expectations. In the embodiment of the present application, the data source for anomaly detection is the log data generated by the operation of software in an application, process, operating system, device or network, which can be stored in a database, local file or message queue (such as a streaming scenario). For example, the aforementioned analysis device 120 can use a deep learning algorithm to perform anomaly detection on log data. It is worth noting that the log template extraction method provided in the embodiment of the present application can also be used in other scenarios such as log compression and keyword retrieval, and the embodiment of the present application does not limit this.

[0076] like Figure 4 As shown, the present application embodiment provides a log template extraction method, which is applied to Figure 2 or Figure 3 In the application environment shown. Figure 4 Taking the application of this method in an anomaly detection scenario as an example, the method includes:

[0077] Step 301: Analyze the device to obtain a log, which includes multiple lines of log records.

[0078] As mentioned above, logs are in the form of batch log data and real-time log data. In an embodiment of the present application, the analysis device supports the analysis of these two forms of logs. In an optional example, the analysis device periodically obtains log files, or obtains log files during a specified period to obtain batch log data, and the specified period can be a low power consumption period of the terminal and / or server (i.e., a period when the power consumption is less than a specified power consumption threshold), so that the impact of log file acquisition and subsequent log analysis on other functions of the terminal and / or server can be reduced; in another optional example, the analysis device continuously obtains real-time log data; in another optional example, the analysis device obtains batch log data or real-time log data after receiving an analysis instruction. The analysis instruction can be generated by a user triggering the terminal and sent to the analysis device by the terminal.

[0079] When the analysis device obtains the log stream in real time and analyzes it, since the log stream can be monitored in time, if an anomaly occurs in the log stream, it can be discovered and reported in time, thereby improving the effectiveness of anomaly detection, avoiding the occurrence of large-scale anomalies, and thus improving user experience.

[0080] The log template extraction method provided in the embodiment of the present application can be used for the log template extraction of the aforementioned homogeneous log, and can also be used for the template extraction of the aforementioned heterogeneous log. In one optional manner, after executing step 301, the analysis device can directly execute step 302; in another optional manner, since the log template extraction method provided in the embodiment of the present application has higher computational efficiency when applied to template extraction of heterogeneous logs, the type of the log can be detected first. If the type of the log is a homogeneous log, a regular expression is used to extract the template of the log. If the type of the log is a heterogeneous log, the subsequent step 302 is executed.

[0081] Step 302: The analysis device deletes the specified characters in each line of the log record in the multiple lines of the log record to obtain each line of the log record after preprocessing.

[0082] Since some designated characters in log records are variable characters in most cases, these characters affect the subsequent computational complexity and the accuracy of the log template finally extracted. Therefore, deleting these designated characters can reduce the computational complexity and the accuracy of the extracted log template. For example, the designated characters are numbers and / or punctuation marks. For example, "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "-", "[", "(", ".", ":", " / ", ")" and "]", etc. In this way, each line of log records after preprocessing usually only includes az, and one or more characters in AZ.

[0083] There are many ways to delete the specified characters in each line of log records in the multi-line log records of the log. The following two ways are used as examples in the embodiments of the present application:

[0084] In the first method, you can first segment the words and then delete the specified characters. The process includes:

[0085] Step A1: The analysis device obtains at least one token for each line of log records in the log.

[0086] Optionally, the analysis device may segment each line of log records in the log by word segmentation technology to obtain at least one entry for each line of log records after word segmentation. An entry includes at least one minimum semantic unit, and usually only includes one minimum semantic unit. A semantic unit is a word, a phrase, or a symbol, and the symbol may be a digital symbol, abbreviated as a number, such as 1 or 2, or other symbols, such as " / " or ":".

[0087] In actual implementation of the embodiment of the present application, a line of log records can usually be divided into at least two entries; in a few cases, a line of log records can be divided into one entry, and the embodiment of the present application does not limit the number of entries obtained by division.

[0088] The purpose of word segmentation is to cut each line of log records into a set of entries. Word segmentation can reduce the processing complexity of log records, reduce the computational cost of subsequent character signature acquisition, and improve computational efficiency.

[0089] In the embodiment of the present application, different ways of word segmentation can be used, for example, using space segmentation; or using special characters for word segmentation; or using designated segmentation marks, which include spaces or special characters; or using natural language for word segmentation.

[0090] like Figure 5 As shown, Figure 5 The figure is a schematic diagram of a space-based word segmentation method. Space-based word segmentation refers to segmenting a line of log records into multiple entries based on spaces. The segmentation process is simple and has high segmentation efficiency.

[0091] When using special characters for word segmentation, the special characters are usually user-specified characters, such as "\n", "[", ":", "=", "{", "(", "\t", "\r", "|", or "##", etc. If a word entry only includes one minimum semantic unit, compared with using space word segmentation, using special characters for word segmentation can make the semantic units included in the segmented word entry more accurate and the segmentation accuracy higher. For example, for the following log record: "20171223-22:15:35:11|Step_SPUtils|30002312|getTodayTotalDetailSteps=1514038440000##7015##548365##8661##13026##27177962", the word segmentation result obtained by using space word segmentation is as follows Figure 6 As shown. It can be seen that the first entry and the last entry both contain a lot of information. For example, the first entry contains time, class name, and numeric information. In the last entry, there are many numbers separated by special characters "##". Each number represents a certain meaning and should be cut out separately. Therefore, each entry cannot be effectively cut into a minimum semantic unit. Assuming that the special characters include: "|", "##" or "=", the segmentation results obtained by using special character segmentation are as follows Figure 7 As shown in the figure, each segmented word is a minimum semantic unit, so the segmentation accuracy is higher.

[0092] The word segmentation using the specified segmentation character is a combination of a space and a sampled special character word segmentation method. For example, the specified segmentation character may include one of: "\n", "[", ":", "=", "{", "(", "\t", "\r", "|", "##", and a space.

[0093] The natural language word segmentation method is more commonly used. In this method, log records can be directly input into a natural language-based word segmenter, such as Word_Tokenizer, TreeBank_Tokenizer, S-Expression_tokenizer, etc. in NLTK (Natural Language Toolkit). Compared with the aforementioned method of using specified segmentation symbols or special characters for word segmentation, the natural language word segmentation method does not require the user to pre-specify the symbols used for segmentation, thereby simplifying user operations.

[0094] Step A2: The analysis device deletes the specified characters in each entry of each line of log record to obtain each line of log record after preprocessing.

[0095] Optionally, the analysis device may traverse each word entry obtained by word segmentation and delete the detected designated characters to obtain each line of log record after preprocessing.

[0096] like Figure 8 As shown, assuming that the aforementioned specified characters are numbers and punctuation marks, for the log record X1: "2019-04-3007:00:31[INFO](com.navercorp.pinpoint.bootstrap.AgentDirBaseClassPathResolver)Agent original-path: / paas-apm / collectors / pinpoint", 9 entries are obtained after word segmentation. After deleting the specified characters, the first 4 entries of the log record X1 become empty entries, that is, there is no actual content.

[0097] In the second method, the analysis device may delete the specified characters before segmenting the words. The process includes:

[0098] Step B1: The analysis device deletes the specified characters in each line of log records in the multiple lines of log records to obtain simplified log records in each line.

[0099] Optionally, the analysis device may traverse each line of log records in the log, and delete the detected specified characters to obtain a simplified log record for each line.

[0100] For example, assuming that the specified character can be a number, log record X2 is: "Connected to 10.110.12.01 at 2019-11-04 15:40:00", then the simplified log record X2 obtained by this method is: "Connected to...at--::".

[0101] It is worth noting that if a certain designated character is the same as the word segmentation symbol used in the subsequent step B2, the designated character is retained to ensure that step B2 can achieve effective word segmentation.

[0102] Step B2: The analysis device obtains at least one entry of each simplified log record of the multiple log lines to obtain each preprocessed log record.

[0103] Step B2 can refer to the aforementioned step A1, and the word segmentation process can also adopt the aforementioned space word segmentation, special character word segmentation, designated segmentation mark word segmentation or natural language word segmentation method, which is not elaborated in the embodiment of the present application.

[0104] Still taking the simplified log record X2 as an example, assuming that the specified segmentation characters are sampled and the specified segmentation characters include: ".", "-", ":" and space, then the simplified log record X2: "Connected to...at--::" is segmented to obtain "Connected", "to" and "at" for a total of 3 entries.

[0105] Step 303: The analysis device determines at least one first log record group based on each line of the preprocessed log record, where different first log record groups include different lines of log records in the log.

[0106] There are multiple ways to determine at least one first log record group based on each line of preprocessed log records. The following ways are used as examples to illustrate the embodiments of the present application:

[0107] In a first possible implementation, based on each line of preprocessed log records, the process of determining at least one first log record group may include:

[0108] Step C1: The analysis device concatenates m consecutive non-empty terms at the first designated position of each line of the preprocessed log record to obtain a first character signature, where m is a positive integer, for example, 2 or 3.

[0109] In an embodiment of the present application, the first designated position is usually a pre-designated (usually user-designated) position, and the probability that the character content corresponding to it is a constant is greater than a preset threshold, that is, the character content corresponding to the first designated position has a greater probability of being a constant. Since the content of the front position in the log record is more likely to be a constant, optionally, the first designated position is a front position, and the m non-empty entries are the first m non-empty entries of each line of log record. However, in different application scenarios, the first designated position may be different in the log records of different logs, and the embodiment of the present application does not limit this.

[0110] For each line of log record after preprocessing, the m consecutive non-empty entries at the first specified position of the log record are concatenated in order of arrangement to obtain a long string, which is called the first character signature.

[0111] Assuming m = 3, still with Figure 8 Take the preprocessed log record X2 as an example, Fig. 9 As shown, the first character signature obtained by concatenation is: "INFOcom.navercorppinpointbootstrapAgentDirBaseClassPathResolverAgent".

[0112] It is worth noting that in step 302, the analysis device can also directly delete the specified characters in each line of the log record in the multi-line log record of the log, obtain each line of the log record after preprocessing, and do not perform word segmentation. In this way, step C1 can be replaced by: the analysis device concatenates the continuous w non-blank characters at the first specified position of each line of the log record after preprocessing to obtain the first character signature. For example, the continuous w non-blank characters at the first specified position are the continuous first w non-blank characters, and w is a positive integer, such as the value range of w is 20-50.

[0113] Step C2: The analysis device determines the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after preprocessing.

[0114] In an optional example, the analysis device may calculate a first hash code of a first character signature corresponding to each preprocessed line of log records, and then determine the first hash code of the first character signature corresponding to each preprocessed line of log records as a first digital signature for each line of log records.

[0115] For example, the aforementioned hash algorithm can be Message Digest Algorithm 4 (MD4) or Secure Hash Algorithm-2 (SHA-2). The embodiments of the present application do not limit the type of hash algorithm.

[0116] In another optional example, the analysis device can calculate the first hash code of the first character signature corresponding to each line of log records after preprocessing, and then simplify the first hash code of each line of log records after preprocessing to obtain the first simplified hash code of each line of log records after preprocessing, and determine the first simplified hash code of each line of log records after preprocessing as the first digital signature of each line of log records.

[0117] The simplification process may include remainder processing, modulo processing, or calculating a new hash code based on the obtained first hash code (the number of bits of the new hash code is less than the number of bits of the original hash code, thereby playing a simplification role). By simplifying the first hash code, the number of bits of the first simplified hash code obtained by processing can be reduced, thereby reducing the number of subsequent groupings and reducing the complexity of grouping.

[0118] like Fig.10 As shown, Fig.10This other optional method is explained by taking the third log record X3 to the sixth log record X6 as an example. After hash operation, the first character signature X31 to the first character signature X61 corresponding to the third log record X3 to the sixth log record X6 have hash codes of 20090, 23327, 10036 and 20090 respectively; after simplification, they are 90, 327, 326 and 90 respectively. Fig.10 The simplified processing is a modulo operation with 1000 as the divisor, but the type of simplified processing adopted in the embodiment of the present application is not limited.

[0119] Step C3: based on the first digital signature of each line of log record, group multiple lines of log records in the log to obtain at least one first log record group, wherein the first digital signatures corresponding to all log records included in each first log record group are the same.

[0120] In a first optional manner, multiple lines of log records in a log may be grouped directly based on the first digital signature of each line of log records to obtain at least one first log record group.

[0121] Generally, there are multiple first log record groups obtained by grouping. The analysis device can group the log records with the same first digital signature in the multiple lines of log records into the same first log record group to obtain the at least one first log record group.

[0122] like Fig.11 As shown, still Fig.10 Taking the four log records shown as an example, the third log record X3 and the sixth log record X6 are very similar, and by deleting the specified characters, the first character signatures obtained are the same, so their hash codes are also the same, and the final first digital signatures are also the same, and they are finally divided into the same first log record group. Fig.11 Each first log record group is only illustratively identified by the corresponding first digital signature, and the embodiment of the present application does not limit the identification method of the first log record group.

[0123] In the second optional manner, a grouping basis can be added based on the first digital signature to perform grouping, so that the hash collision in the obtained first log record group is less and the grouping result is more accurate. Then, based on the first digital signature of each line of log record, the process of grouping multiple lines of log records in the log to obtain at least one first log record group may include:

[0124] Step C31, concatenate the n consecutive non-empty entries at the second designated position of each line of the preprocessed log record to obtain a second character signature, where n is a positive integer, and the first designated position is different from the second designated position.

[0125] Wherein, n may be equal to or different from m, for example, n is 1, 2 or 3. In an embodiment of the present application, the second designated position is generally a pre-designated (usually user-designated) position, and the probability that the character content corresponding to it is a constant is greater than a preset threshold value, that is, the character content corresponding to the second designated position is probably a constant. As previously mentioned, since the possibility that the content of the front position in the log record is a constant is high, the second designated position is optionally a front position, and the n non-empty entries are the adjacent n non-empty entries after the first m non-empty entries of each line of log record. However, in different application scenarios, the second designated position may be different in the log records of different logs, and the embodiment of the present application does not limit this.

[0126] It is worth noting that, similar to the aforementioned step C2, in an optional example, the analysis device can calculate the second hash code of the second character signature corresponding to each line of log record after preprocessing, and then determine the second hash code of the second character signature corresponding to each line of log record after preprocessing as the second digital signature of each line of log record.

[0127] In two optional examples, the analysis device can calculate the second hash code of the second character signature corresponding to each line of log record after preprocessing, and then simplify the second hash code of each line of log record after preprocessing to obtain the second simplified hash code of each line of log record after preprocessing, and determine the second simplified hash code of each line of log record after preprocessing as the second digital signature of each line of log record. Among them, the simplification process can include remainder processing, modulus processing, or calculating a new hash code based on the obtained second hash code (the number of bits of the new hash code is less than the number of bits of the original hash code, thereby playing a simplifying role). By simplifying the second hash code, the number of bits of the second simplified hash code obtained by processing can be reduced, thereby reducing the number of groups in the second log record group and reducing the complexity of grouping.

[0128] Step C32: based on the first digital signature of each line of log record, group multiple lines of log records in the log to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same.

[0129] Usually, there are multiple second log record groups obtained by grouping. The analysis device can group the log records with the same first digital signature in the multiple lines of log records into the same second log record group to obtain the at least one second log record group.

[0130] Step C33: based on the second digital signature of each line of log record, group each second log record group in at least one second log record group to obtain at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

[0131] The analysis device may classify the log records with the same second digital signature in each second log record group into the same first log record group to obtain the at least one first log record group.

[0132] As can be seen from the first optional method of the aforementioned step C3, since this grouping method may place different log records in the same first log record group, it is easy to make it difficult to extract the log template in the first log record group. The second optional method can reduce the probability of placing different log records in the same first log record group, thereby reducing the difficulty of extracting the log template.

[0133] like Fig.12 As shown, assuming m=2, n=2, the seventh log record X7 is: "Order 123 has been created", and the eighth log record X8 is: "Good morning Bill Gates", then for each line of log records, the analysis device can execute step C1 to extract its first character signature, and execute step C31 to extract its second character signature. For example, the first character signature of the seventh log record X7 is the concatenation of the first and second non-empty entries, that is, "Order has". The second character signature is the concatenation of the third and fourth non-empty entries, that is, "been created". Similarly, the first character signature and the second character signature of the eighth log record X8 are: "Good morning" and "Bill Gates", respectively. For example, a hash code is calculated based on the first character signatures of the seventh log record X7 and the eighth log record X8 to obtain the corresponding first digital signatures. The first digital signatures of the two log records are both 90, and both are divided into the same second log record group 90; then a hash code is calculated based on the second character signatures of the seventh log record X7 and the eighth log record X8 to obtain the corresponding second digital signatures: 123 and 6. Based on the two second digital signatures, the two lines of log records are divided into different first log record groups, then log record X7 falls into the first log record group 123, and log record X8 falls into the first log record group 6.

[0134] It can be seen from this that for two different log records, the probability of the first digital signature and the second digital signature colliding at the same time is very low. The second optional method of grouping can effectively distinguish different log records in the same log record group. Fig.12Each second log record group is only schematically identified by the corresponding first digital signature, and each first log record group is identified by the second digital signature. The embodiment of the present application does not limit the identification method of the first log record group and the second log record group. The grouping algorithm adopted by the second optional method is also called a multi-layer hash algorithm, and the first log record group and the second log record group belong to different layers.

[0135] Step 304: The analysis device extracts the log templates in each of the at least one first log record group, and determines the log template of the log based on the extracted log templates.

[0136] Optionally, the process in which the analyzing device extracts the log templates in each of the at least one first log record group, and determines the log template of the log based on the extracted log templates may include:

[0137] Step D1: The analysis device extracts log templates from each first log record group respectively.

[0138] The purpose of the aforementioned step 303 is to classify similar log records into the same first log record group. Although in step 303, one or more first log record groups have been obtained based on the first digital signature, there may still be hash collisions in each first log record group. For example, if the first digital signature is a first simplified hash code, assuming that the first hash code of one log record is 20090, and the first hash code of another log record is 30090, the two are essentially different, but the corresponding simplified hash codes are both 90, and they are ultimately classified into the same first log record group. In this way, a hash collision occurs. On the one hand, the aforementioned steps C31 to C33 can be used to reduce the hash collision; on the other hand, the log template extraction method provided in the subsequent embodiments of the present application can be used to effectively extract one or more log templates in the first log record group.

[0139] There are many ways to implement the analysis device to extract the log templates in each first log record group. The present application embodiment takes the following two implementations as examples for illustration:

[0140] In the first implementation mode, the analysis device first merges the historical log template with the log record, and then determines the new log template.

[0141] Typically, there are one or more historical log templates in a first log record group. The following steps D11 to D14 take the existence of a historical log template in the first log record as an example to illustrate the first implementation method, and the process includes:

[0142] Step D11: For each line of log record in each first log record group, the analysis device generates an alternative log template based on the log record and the historical log template of the first log record group.

[0143] The process of generating an alternative log template based on the log record and the historical log template of the first log record group by the analysis device can be called a merging process. Optionally, assuming that the log record currently being compared is log record R, the analysis device can replace the part of the historical log template that is different from the log record R with a variable indicator, and keep the same part unchanged (equivalent to merging the same part as the log record R) to obtain an alternative log template.

[0144] Alternatively, the analysis device first determines whether the part of the historical log template that is different from the log record R is only the part where the variable indicator of the historical log template is located. If the different part also includes other parts besides the part where the variable indicator is located, the other parts are replaced by the variable indicator to obtain an alternative log template. If the different part only includes other parts besides the part where the variable indicator is located, the historical log template can be directly used as an alternative log template.

[0145] It is worth noting that, in the aforementioned comparison process, when the length of the log record R is less than the length of the historical log template, the part of the historical log template that extends beyond the log record R can be determined as the same part, and the part of the historical log template is retained; when the length of the log record R is greater than the length of the historical log template, the part of the log record R that extends beyond the historical log template can be determined as an invalid part, and the part is ignored, that is, the part is not included in the final alternative log template; or, the part of the log record R that extends beyond the historical log template can be determined as a different part, and the part is replaced with a variable indicator and added to the corresponding position (i.e., the end) of the historical log template. Usually, the length of a log record is represented by the number of entries included in the log record.

[0146] The analysis device may compare the log record R with the historical log template of the first log record group by means of a bitwise comparison to determine the same and different parts. The bitwise comparison is to compare the entries at the same position of the log record with the historical log template.

[0147] For example, the variable indicator is "*", assuming that the historical log template is: "Good morning, Bill Gates.", and the log record X9 is: Good morning, Ma Hua Teng. Through the comparison, the first two entries of the historical log template and the log record X9 are the same, so the two entries of the historical log template are kept unchanged (equivalent to merging with the log record X9), and the third and fourth entries of the historical log template and the log record X9 are different, so the corresponding position in the historical log template is replaced by a "*". Since the length of the log record X9 is greater than the historical log template, its fifth entry "Teng" can be ignored. Therefore, the final candidate log template is: "Good morning**".

[0148] Step D12: The analysis device determines whether the log record matches the generated candidate log template.

[0149] Optionally, the analysis device may calculate a matching degree between the log record and the generated candidate log template, and determine whether the log record matches the candidate log template based on the calculated matching degree.

[0150] For example, the matching degree between the log record and the candidate log template may be determined based on a matching degree calculation formula, which is:

[0151] P = f1 / (f-f2);

[0152] Among them, P represents the matching degree between the log record and the alternative log template, f1 represents the number of non-variable indicators in the alternative log template, f represents the maximum number of entries between the log record and the alternative log template, that is, the maximum number of entries between the two, and f2 represents the number of variable indicators in the alternative log template.

[0153] Still taking the example of the aforementioned step D11, the number of entries in the historical log template that are not "*" is 2 ("Good" and "Morning" respectively), so f1=2, the number of entries in the historical log template and log record X9 are 4 and 5 respectively, taking the maximum value, which is 5, then f=5, there are 2 "*" in the historical log template, f2=2, so the matching degree P=2 / (max(4,5)-2)=0.67.

[0154] Optionally, the analysis device may compare the calculated matching degree with a matching degree threshold to determine whether the log record matches the candidate log template. For example, when the matching degree is greater than the matching degree threshold, it is determined that the log record matches the candidate log template, and when the matching degree is not greater than the matching degree threshold, it is determined that the log record does not match the candidate log template.

[0155] Still taking the example of the aforementioned step D11 as an example, the matching degree threshold = 0.5. The matching degree obtained by the aforementioned calculation is 0.67, which is greater than 0.5, so it is determined that the log record matches the candidate log template.

[0156] Step D13: When the log record matches the candidate log template, the analysis device determines the candidate log template as a new log template for the first log record group.

[0157] When the log record matches the candidate log template, it means that the log record is similar to the candidate log template. Based on the log record and the historical log template, the generated candidate log template is an acceptable log template. Therefore, the analysis device can determine the candidate log template as a new log template for the first log record group. In an optional example, the analysis device can delete the original historical log template in the first log record group.

[0158] Step D14: When the log record does not match the candidate log template, the analysis device rolls back the candidate log template to the historical log template and log record.

[0159] When the log record does not match the alternative log template, it means that the log record is not similar to the alternative log template. The alternative log template generated based on the log record and the historical log template is an unacceptable log template. Therefore, the log record and the historical log template can be rejected to be merged into the alternative log template. The analysis device then rolls back the alternative log template to the historical log template and log record.

[0160] When there are multiple historical log templates in the first log record group, in an optional manner, for any log record, the analysis device may merge the log record with each of the multiple log templates to obtain an alternative log template, and the merging process refers to the process of the aforementioned step D11. The analysis device matches the any log record with each alternative log template, and the matching process refers to the process of the aforementioned step D12. Among them, the merging process of any log record with each log template can be performed in parallel or in series; similarly, the matching process can be performed in parallel or in series. In another optional manner, for any log record, the analysis device may scan each historical log template in turn, and for each scanned historical log template, perform the processes of the aforementioned steps D11 to D14. When any log record matches any alternative log template, after obtaining the first alternative log template as the new log template of the first log record group, or after scanning all historical log templates, stop the action.

[0161] Among them, when any log record matches only one alternative log template, the one alternative log template is determined as the new log template for the first log record group; when any log record matches multiple alternative log templates, a target log template can be determined from the multiple matched alternative log templates as the new log template for the first log record group, and the target log template is the template with the highest matching degree among the multiple matched alternative log templates or a randomly selected template; when any log record does not match all alternative log templates, the analysis device can add the log template extracted from the log record as the new log template for the first log record group after rolling back all alternative log templates to historical log templates and log records.

[0162] It should be noted that, when there is no historical log template in the first log record group, the log template extracted from the log record can be added as a new log template for the first log record group.

[0163] Optionally, the process of the aforementioned analysis device extracting a log template from a log record can be implemented in a variety of ways. In the first example, the analysis device can directly use the log record as a log template, that is, the extracted log template is the log record itself; in the second example, since some specified characters in the log record are variables in most cases, these specified characters can be replaced with variable indicators to generate a log template extracted from the log record. For example, the specified character can be a number, and the fixed character can be a number or other symbols. For example, 1 or 2, *, etc.

[0164] The second implementation method is to analyze the device to match the historical log template with the log record, and then determine the log template based on the matching result. This process includes:

[0165] Step D15: For each line of log records in each first log record group, the analysis device may compare the line of log records with the historical log template of the first log record group.

[0166] The lengths of the log records in the same first log record group finally obtained by the analysis device may be the same or different. This application provides two comparison methods for these two situations:

[0167] In the first case, the lengths of the log records in the same first log record group are different. In this way, the log record and the historical log template are regarded as word sequences respectively, and the log record can be compared with the historical log template of the first log record group by finding the longest common subsequence (Longest Common Subsequence, LCS) of the two. In the first example, if the proportion of the determined longest common subsequence length (i.e., the total number of words in the longest common subsequence) in the log record length (i.e., the total number of words in the log record) is greater than the third ratio threshold, it is determined that the log record matches the historical log template; if the proportion of the determined longest common subsequence length in the log record length is not greater than the third ratio threshold, it is determined that the log record does not match the historical log template. In the second example, if the length of other sequences other than the longest common subsequence in the log record (i.e., the total number of words in the log record minus the total number of words in the longest common subsequence) in the log record length is less than the fourth ratio threshold, it is determined that the log record matches the historical log template; if the proportion of the length of other sequences in the log record length is not less than the fourth ratio threshold, it is determined that the log record does not match the historical log template. In the third example, if the length of the longest common subsequence is greater than the first length threshold, it is determined that the log record matches the historical log template; if the length of the longest common subsequence is not greater than the first length threshold, it is determined that the log record does not match the historical log template. There are other ways to determine whether the log record matches the historical log template of the first log record group, which is not limited in the embodiments of the present application.

[0168] Finding the longest common subsequence of the log record and the historical log template is to find the longest common subsequence of the two. For example, the longest common subsequence can be found by recursion or dynamic programming. Taking each entry as an example, it is assumed that each word includes a semantic unit. Suppose in a first log record group, log record X10 is: "User Yang XiaoYu has been logged in"; the first historical log template is: "User*has been logged in". Then the longest common subsequence of the two is: "User has been logged in". Suppose in another first log record group, log record X11 is: "User Yang Xiao Yu has been logged in"; the second historical log template is: "User**registered successfully". Then the longest common subsequence of the two is: "User".

[0169] Taking the determination method of the third example as an example, assuming that the first length threshold is 3, the length of the longest common subsequence of log record X10 is 5, and it is determined that log record X10 matches the first historical log template. The length of the longest common subsequence of log record X11 is 1, and it is determined that log record X11 does not match the second historical log template.

[0170] In the second case, the lengths of the log records in the same first log record group are the same. In this way, the log record can be compared with the historical log template of the first log record group by means of a bitwise comparison. In the first example, if the proportion of the number of identical terms in the log record length (i.e., the total number of terms in the log record) is greater than the first ratio threshold, it is determined that the log record matches the historical log template; if the proportion of the number of identical terms in the log record length is not greater than the first ratio threshold, it is determined that the log record does not match the historical log template. In the second example, if the proportion of the number of different terms (the different terms refer to terms in a line of log records that are different from another log record) in the log record length is less than the second ratio threshold, it is determined that the log record matches the historical log template; if the proportion of the number of different terms in the log record length is not less than the second ratio threshold, it is determined that the log record does not match the historical log template; in the third example, if the proportion of the number of identical terms in the log record length is greater than the first ratio threshold, and the number of different terms is less than the first number threshold, it is determined that the log record matches the historical log template; if the proportion of the number of identical terms in the log record length is not greater than the first ratio threshold, or the number of different terms is not less than the first number threshold, it is determined that the log record does not match the historical log template. In the fourth example, if the number of identical terms is greater than the second number threshold, it is determined that the log record matches the historical log template; if the number of identical terms is not greater than the second number threshold, it is determined that the log record does not match the historical log template. In the fifth example, if the number of different entries is less than the third numerical threshold, it is determined that the log record matches the historical log template; if the number of different entries is not less than the third numerical threshold, it is determined that the log record does not match the historical log template. There are other ways to determine whether the log record matches the historical log template of the first log record group, which are not limited in the embodiments of the present application.

[0171] Assume that log record X10 is: "User Yang Xiao Yu has been logged in"; the historical log template is: "User***has been logged in". Assuming that each entry includes a semantic unit, "User", "Yang", "Xiao", "Yu", "has", "been", "logged" and "in" are compared one by one with "User", "*", "*", "*", "has", "been", "logged" and "in". Taking the judgment method of the first example mentioned above as an example, the number of identical entries is 8, the length of the log record is 8, and assuming that the first ratio threshold is 1 / 2, 8 / 8 is greater than 1 / 2, and it is determined that the log record matches the historical log template.

[0172] It should be noted that, usually, there are one or more log templates in a first log record group. When there are multiple log templates in the first log record group, for any log record, the log record can be compared with each of the multiple log templates, and the comparison process refers to the process of the above two cases; or, the distance between the log record and each of the multiple log templates is calculated (such as using the Jaccard distance function to calculate the distance), and the log record is compared with the log template closest to it, which can reduce the computational cost.

[0173] Step D16: When the log record matches the historical log template, a new log template for the first log record group is determined based on the log record and the historical log template.

[0174] The process of determining a new log template for the first log record group based on the log record and the historical log template may be referred to as a process of merging the log record and the historical log template.

[0175] In the first example, since the historical log template is an extracted template and the log record matches the historical log template, the historical log template can be directly used as a new log template for the first log record group.

[0176] In the second example, since there may be some different parts between the historical log template and the log record, these different parts can be considered as variable parts. The analysis device can replace the part of the historical log template that is different from the log record with a variable indicator; or, the analysis device first determines whether the part of the historical log template that is different from the log record is only the part where the variable indicator is located. If the different part also includes other parts besides the part where the variable indicator is located, the other parts are replaced with the variable indicator to obtain a new log template. If the different part only includes other parts besides the part where the variable indicator is located, the historical log template can be directly used as the new log template. Such processing can obtain a more accurate log template.

[0177] In the above two examples, the corresponding historical log template can be updated after adopting the new log template, for example, the historical log template can be deleted, or the corresponding historical log template can be overwritten after adopting the new log template, so as to ensure that there is no duplicate log template in the first log record group.

[0178] It is worth noting that, corresponding to the first case mentioned above, since there is no need to ensure the consistency of the length of the log template and the log record, when performing a replacement operation, a variable indicator can replace one or more consecutive entries.

[0179] Corresponding to the second case mentioned above, in order to ensure the consistency of the length of the log template and the log record, when performing the replacement operation, a variable indicator only replaces one entry, and the length of a variable indicator in the log template is regarded as 1.

[0180] Step D17: When the log record does not match the historical log template, the log template extracted from the log record is added as a new log template of the first log record group.

[0181] When the log record does not match the historical log template, it means that there is no log template matching the log record in the first log record group, and a new log template corresponding to the log record needs to be generated. The process of generating a new log template can refer to the aforementioned step D14, and this embodiment of the application will not be repeated here.

[0182] In step 303, the analysis device usually determines at least one first log record group by traversing each line of log records in the log. Then the aforementioned process of extracting the log templates in each first log record group can be performed after all log record groupings are completed, or it can be performed in real time during the log record grouping process. Among them, extracting the log templates in each first log record group in real time during the log record grouping process can reduce the delay of template extraction and improve the overall timeliness of the template extraction process.

[0183] For example, when the log template in the first log record group is determined by adopting the aforementioned first implementation method, taking the real-time extraction of the log template in each first log record group during the grouping process of the log records as an example, corresponding to the aforementioned step 303, for each first log record group, the process of extracting the log template in each first log record group separately may include: after receiving a line of log records, merging the received log records with the historical log template to obtain an alternative log template; when the received log record matches the alternative log template, determining the alternative log template as the new log template for the first log record group; when the received log record does not match the alternative log template, performing template rollback, and adding the log template extracted from the received log record as the new log template for the first log record group.

[0184] Assuming that the matching degree threshold is 50%, the first log record group is identified by the corresponding first digital signature, the variable indicator is a wildcard, and the eleventh log record X11 to the fourteenth log record X14 are sequentially divided into the same log record group. Figures 13 to 16 The corresponding log template extraction processes are schematically introduced respectively.

[0185] like Fig.13 As shown, assume that the eleventh log record X11 is: "Good morning, Bill Gaga". Its first digital signature is 90. Since there is no log record group corresponding to the first digital signature of 90 in the log record group, the analysis device creates a log record group 90 to store log records with the first digital signature equal to 90. Since there is no historical log template in the log record group 90 before, the first log record, that is, the alternative log template 1 obtained by merging the eleventh log record X11 with the historical log template can be assumed to be the eleventh log record X11 itself, and the matching degree between the two is 100%. And the alternative log template 1 is determined as a new log template, that is, log template 1, which corresponds to the eleventh log record X11.

[0186] like Fig.14 As shown, assuming that the twelfth log record X12 is: "Good morning, Ma Xiao Ming". Its first digital signature is 90, and the analysis device divides the twelfth log record X12 into the log record group 90. At this time, the historical log template in the log record group 90 is log template 1, and the twelfth log record X12 is combined with log template 1 to obtain the alternative log template 2: "Good morning, **", and the matching degree between the twelfth log record X12 and the alternative log template 2 is calculated. The matching degree is greater than the matching degree threshold, so the alternative log template 2 is used as the new log template, that is, log template 2, which corresponds to the eleventh log record X11 and the twelfth log record X12.

[0187] like Fig.15 As shown, assume that the thirteenth log record X13 is: "Order 123 has been created". Its first digital signature is 90, and the analysis device divides the thirteenth log record X13 into the log record group 90. At this time, the historical log template in the log record group 90 is log template 2, and the thirteenth log record X13 is merged with log template 2 to obtain the alternative log template 3: "****", and the matching degree between the thirteenth log record X13 and the alternative log template 3 is calculated. The matching degree is less than the matching degree threshold, so the alternative log template 3 is rolled back to log template 2 and the thirteenth log record X13, and a new log template is generated based on the thirteenth log record X13, namely log template 3: "Order 123 has been created", which corresponds to the thirteenth log record X13. At this time, there are two log templates in the log record group 90, namely log template 2 and log template 3.

[0188] like Fig.15 As shown, assuming that the fifteenth log record X15 is: "Order 456 has been created". Its first digital signature is 90, and the analysis device divides the fifteenth log record X15 into the log record group 90. At this time, the historical log templates in the log record group 90 are log template 2 and log template 3. The fifteenth log record X15 is merged with log template 2 to obtain the alternative log template 4: "****", and the matching degree between the fifteenth log record X15 and the alternative log template 3 is calculated. The matching degree is not greater than the matching degree threshold, so the alternative log template 4 is rolled back to log template 2 and the fifteenth log record X15; the fifteenth log record X15 is merged with log template 3 to obtain the alternative log template 5: "Order* has been created", and the matching degree between the fifteenth log record X15 and the alternative log template 5 is calculated. The matching degree is greater than the matching degree threshold, and the alternative log template 5 is used as the new log template, that is, log template 4, which corresponds to the fourteenth log record X14 and the fifteenth log record X15. At this time, there are two log templates in the log record group 90, namely log template 2 and log template 4.

[0189] For example, when the second implementation method described above is used to determine the log template in the first log record group, taking the real-time extraction of the log template in each first log record group during the grouping process of the log records as an example, corresponding to the aforementioned step 303, for each first log record group, the process of extracting the log template in each first log record group respectively may include: after receiving a line of log records, comparing the received log records with the historical log templates, when the received log records match the historical log templates, determining a new log template for the first log record group based on the received log records and the historical log templates, and when the received log records do not match the historical log templates, adding the log template extracted from the received log records as the new log template for the first log record group.

[0190] Taking the second log as an example, assuming the grouping result is Fig.17 The grouping result shown is as follows. For the first log record group 0, after receiving the 0th line of log record, the historical log template of the first log record group 0 is empty, and the 0th line of log record does not match the historical log record. The log template of the 0th line of log record is extracted and added as the new log template for the first log record group: "mod_jk child workerEnv in error state6"; after receiving the 4th line of log record, the received log record is compared with the historical log template: "mod_jk child workerEnv in error state 6". Since the received 4th line of log record matches the historical log template, the historical log template "mod_jk child workerEnv in error state*" of the first log record group 0 can be used as the new log template. The template extraction method for the first log record group 1 and the first log record group 2 is the same as this template extraction method. The templates of each first log record group finally extracted are as follows. Fig.17 As shown, the embodiments of the present application will not be described in detail.

[0191] It should be noted that the aforementioned historical log template and new log template are relative concepts. The historical log template refers to the log template that already exists at the current moment, and the new log template refers to the log template that is newly generated at the current moment.

[0192] Step D2: The analysis device determines a log template for the log based on the log template of each first log record group.

[0193] Optionally, based on the log template of each first log record group, the process of determining the log template of the log can be implemented in the following three ways:

[0194] In a first manner, clustering is performed on a log template of at least one first log record group to obtain a log template of the log.

[0195] Clustering is essentially a grouping method used to group similar processing objects into one category and dissimilar processing objects into different categories. In an embodiment of the present application, after obtaining one or more log templates of at least one first log record group, the analysis device can classify the one or more log templates through clustering. In particular, when there are multiple log templates obtained, there may be situations where the log templates of different first log record groups are similar. Through classification, similar log templates can be divided into one category of log templates, so that the one or more categories of log templates obtained by the division are used as log templates for the log. When subsequently presented to the user, the one or more categories of log templates can be presented to the user so that the user can intuitively see how many categories of log templates exist in the log. For example, the clustering process can be hierarchical clustering, and its processing process refers to the aforementioned hierarchical clustering process, which is not described in detail in the embodiment of the present application.

[0196] The log templates obtained through hierarchical clustering have a hierarchical relationship, and the user can adjust the clustering accuracy (also called granularity) to obtain different clustering results.

[0197] In a second manner, a merging process is performed on the log templates of at least one first log record group to obtain a log template of the log.

[0198] Merge processing refers to the process of integrating the same or similar processing objects into one object, and its processing effect is similar to the effect of deduplication processing. In an embodiment of the present application, the log template of at least one first log record group is merged to obtain the log template of the log, including: when the log template of at least one first log record group includes at least two log templates, for each two log templates, detect whether the similarity of the constant part of the two log templates is 1; when the similarity of the constant part of the two log templates is 1, the variable part of one log template in the two log templates is replaced with a variable identifier, and the other log template is deleted (equivalent to retaining the constant part of any log template, and inserting the variable identifier into the position of the original variable part between the constant parts). For example, the variable identifier can be a wildcard "*". Among them, the similarity of the constant part of the two log templates can be determined by calculating the distance between the constant parts of the two log templates. The similarity can be calculated using the Jaccard similarity (Jaccard similarity, also known as the Jaccard coefficient) algorithm. It should be noted that when the similarity of the constant part of the two log templates is not 1, the two log templates are not processed.

[0199] For example, two templates are: "User**has logged in" and "User***has logged in", and the constant part of both contains four entries: {User, has, logged, in}. The similarity between the two is 1. Therefore, the variable part "**" of "User**has logged in" can be replaced with "*", and "User***has logged in" can be deleted, and the merged log template is obtained: "User*has logged in".

[0200] In a third manner, at least one log template of the first log record group is used as the log template of the log.

[0201] In most cases, each first log record group usually has only one log template, and if the grouping method of the aforementioned first log record groups is appropriate and there are fewer identical or similar log templates, clustering or merging processing is not required, and the log templates of each log record group obtained can be directly used as the log template of the log.

[0202] In the embodiment of the present application, the analysis device can support one or more of the above three methods. The terminal can present all the trigger buttons (or icons) of the multiple methods in the user interface, or present the trigger buttons of the multiple methods in a scrolling manner, and can also present the trigger buttons of one or more methods with higher frequency of use in the trigger buttons of the multiple methods (the trigger buttons of other methods can be displayed by the user triggering other buttons again, and the other buttons can be drop-down buttons), etc., and the embodiment of the present application does not limit this. When the user wants to view the log template of the log corresponding to a certain method, the trigger button corresponding to the certain method is triggered by clicking, etc., and accordingly, the terminal receives the user's selection instruction, which carries the identification of the certain method, and the terminal sends the selection instruction to the analysis device. The analysis device obtains the log template of the log in the corresponding manner based on the obtained selection instruction, and the terminal presents it to the user in the user interface. Among them, when the log template is presented in the first way, the log template can be presented in the form of a multi-layer file directory structure or a tree structure (such as a binary tree); when the log template is presented in the second or third way, if there are multiple log templates, the multiple log templates can be presented in a list.

[0203] It is worth noting that the aforementioned step 304 may also be implemented by a clustering algorithm, such as a hierarchical clustering algorithm, which will not be elaborated in the embodiment of the present application.

[0204] Step 305: The analysis device performs anomaly detection on the log based on the log template of the log.

[0205] In an optional manner, the analysis device extracts features from the log based on a log template of the log; and performs anomaly detection based on the extracted features of the log.

[0206] The characteristics of a log refer to the characteristics of the log records contained in the log. For example, it may include: the number of occurrences of a log template, the frequency of occurrence of a log template, and / or the period of occurrence of a log template. Among them, the number of occurrences of a log template refers to the number of log records corresponding to the log template in the log; the frequency of occurrence of a log template refers to the ratio of the number of log records corresponding to the log template to the total number of log records contained in the log; the period of occurrence of a log template refers to the period to which the log record corresponding to the log template belongs, either at the time of occurrence or at the time of collection.

[0207] For example, assuming that the first log pattern is any log pattern in the log, for the first log pattern, the analysis device can divide the log into multiple time windows, count each line of log records included in each time window to detect log records that match the first log pattern, and count the features of the logs required to be determined in the time window, such as the number of occurrences of the first log template. The analysis device compares the features of the logs in multiple time windows, and determines the time window whose feature difference with other time windows is greater than the specified difference threshold as an abnormal time window, then the log record of the first log pattern in the abnormal time window is the log record with the abnormality. The aforementioned multiple time windows can be time windows of fixed size and non-overlapping, or time windows determined by a sliding window algorithm.

[0208] For example, when the feature to be analyzed is the number of occurrences of the first log template, the analysis device can mark a hot event and issue an alarm message when it finds that the number of occurrences of the first log template is significantly higher (for example, the difference with the mean of the number of occurrences of the first log template in other time windows or all time windows is positive, and the difference is greater than the specified difference threshold). The analysis device can mark a cold event and issue an alarm message when it finds that the number of occurrences of the first log template is significantly lower (for example, the difference with the mean of the number of occurrences of the first log template in other time windows or all time windows is negative, and the absolute value of the difference is greater than the specified difference threshold).

[0209] It is worth noting that the terminal can display the log template of the log determined by the analysis device, and the user can specify the target log template. Accordingly, the terminal receives the template selection instruction and sends the template selection instruction carrying the identifier of the target log template to the analysis device. The analysis device performs anomaly detection on the target log template. The detection process can refer to the anomaly detection process of the first log template mentioned above. In this way, the analysis device can perform anomaly detection on a specific log template according to the user's instructions, improve the pertinence of anomaly detection, and ensure user experience.

[0210] In two optional ways, the analysis device detects unknown events based on the log template of the log. The analysis device can use each log template to match the log records in the log respectively. When there is a log record that does not match all log templates, the log record is determined to be an unknown log record, and the event corresponding to the unknown log record is an unknown event. It may be an abnormal event.

[0211] In the anomaly detection scenario, the analysis device may also detect anomalies in the log in other ways, which is not limited in the embodiments of the present application.

[0212] It should be noted that in the aforementioned step 302, a hash code is used when grouping log records in the log. Therefore, the distribution rule of the log records follows the hash distribution rule, that is, the key-value distribution rule, so that load balancing can be achieved.

[0213] To facilitate the reader's understanding, the embodiment of the present application briefly introduces the principle of hash distribution. Hash distribution is a data distribution method based on a hash function, which can also be called a hash function. A hash function is a function that obtains a value (also called a hash code) based on the key (key, also called a key value, also called a distribution key value in a distributed system) of the data. That is, value = f (key), and function f is the hash function. Taking Table 1 as an example, assuming that the hash function is f (key) = key mod5, "mod" means modulus, that is, the hash function is a module operation function. Assuming that the keys are 1, 2, 3, 4, 5, 6, 7, 8 and 9 respectively, the corresponding values ​​are 1, 2, 3, 4, 0, 1, 2, 3 and 4 respectively.

[0214] Table 1

[0215] key 1 2 3 4 5 6 7 8 9 value 1 2 3 4 0 1 2 3 4

[0216] As can be seen from the above, when the key is 1 and 6, the value is 1. Therefore, when using a hash function to determine the value, there may be different keys corresponding to the same value. This situation is called a hash conflict. The hash bucket algorithm is a special hash algorithm that can resolve hash conflicts. A hash bucket is a container for placing different key lists (also called hash tables). The hash bucket is also called an f(key) set, or a value set. The hash codes corresponding to the same hash bucket are the same. In an embodiment of the present application, referring to the aforementioned step C2, in an optional example corresponding to step C2, the number of hash buckets can be set to the number of different first hash codes obtained. Multiple first hash codes correspond one-to-one to multiple hash buckets. For example, the first hash code value can be used as an identifier of a hash bucket, such as an index or number. Each hash bucket stores keys with the same first hash code, and conflicting keys in the same hash bucket are stored using a one-way linked list, thereby resolving hash conflicts. When searching for data corresponding to a key, you only need to use the key to index the hash bucket corresponding to the first hash code, and then start searching from the node corresponding to the first address of the hash bucket, that is, search in the order of the linked list, compare the key value (that is, the first hash code), until the corresponding key is found, and then index the corresponding data based on the found key. As shown in Table 1, assuming that the hash code in Table 1 is the first hash code, there are 5 different first hash codes, so there are 5 hash buckets. When the key is 1 and 6, it is stored in hash bucket 1, when the key is 2 and 7, it is stored in hash bucket 2; when the key is 3 and 8, it is stored in hash bucket 3; when the key is 4 and 9, it is stored in hash bucket 4; when the key is 5, it is stored in hash bucket 0.

[0217] In another optional example corresponding to step C2, the number of hash buckets can be set to the number of different first simplified hash codes obtained. Multiple first simplified hash codes correspond one to one with multiple hash buckets. For example, the first simplified hash code can be used as an identifier of the hash bucket, such as an index or number, and each hash bucket stores keys with the same first simplified hash code. Conflicting keys in the same hash bucket are stored in a one-way linked list, which solves the hash conflict. When searching for data corresponding to the key, it is only necessary to index the hash bucket corresponding to the first simplified hash code through the key, and then start searching from the node corresponding to the first address of the hash bucket, that is, search in the order of the linked list, compare the key value, until the corresponding key is found, and then index the corresponding data based on the found key.

[0218] As can be seen from the above, the number of different first simplified hash codes is usually less than the number of different first hash codes, and the number of hash buckets is reduced accordingly. Therefore, by adopting the aforementioned alternative example, the number of hash buckets can be reduced to avoid the hash buckets being too dispersed.

[0219] It should be noted that the aforementioned embodiment only uses the hash function as a modulus function as an example for explanation. In fact, the hash function can also be a remainder function (in this case, the hash function is a complementation function, and the number of hash buckets is the value of the modulus), or other functions. The embodiments of the present application do not limit this.

[0220] With reference to the foregoing introduction, the embodiment of the present application can introduce a hash bucket algorithm to distribute log records, thereby avoiding hash conflicts. In this case, the distributed data is usually identified in units of hash buckets. Therefore, each of the aforementioned first log record groups can be represented by a hash bucket, and each hash bucket is identified by a corresponding first digital signature (i.e., the aforementioned first hash code or the first simplified hash code). If there is a second log record group, each second log record group can also be represented by a hash bucket. Each hash bucket is identified by a corresponding second digital signature (i.e., the aforementioned second hash code or the second simplified hash code).

[0221] The order of steps of the log template extraction method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased according to the situation. For example, in other application scenarios, such as log compression or keyword retrieval, the aforementioned step 305 may not be executed. Any technician familiar with the technical field can easily think of a change within the technical scope disclosed in the present application, and the change should be included in the protection scope of the present application, so it will not be repeated.

[0222] To sum up, in the embodiment of the present application, by deleting the specified characters in each line of log records in the multiple lines of log records, the computational complexity can be effectively reduced, and the interference of the specified characters in the log template extraction process can be avoided, thereby improving the progress of extracting the log template.

[0223] For example, since variable symbols in log records, such as timestamps and IP addresses, include a large number of numbers and punctuation marks, when the specified characters are numbers and / or punctuation marks, the interference of these variable symbols on log template extraction can be effectively reduced, thereby improving the accuracy of log template extraction.

[0224] When there are multiple first log record groups, the processing processes for each first log record group can be executed in parallel, thereby reducing computing delays. The amount of data required to calculate each time the processing process is executed is much smaller than the amount of data in the entire log, effectively reducing the computing cost and improving computing efficiency.

[0225] The present application embodiment provides a log template extraction device 40, such as Fig.18 As shown, the device comprises:

[0226] A preprocessing module 40 is used to delete specified characters in each line of log records in a log having multiple lines of log records, so as to obtain each line of log records after preprocessing; a determination module 402 is used to determine at least one first log record group based on each line of log records after preprocessing, and different first log record groups include different lines of log records in the log; a processing module 403 is used to respectively extract log templates in each of the at least one first log record group, and determine the log template of the log based on the extracted log templates.

[0227] To sum up, in the embodiment of the present application, by deleting the specified characters in each line of log records in the multiple lines of log records, the computational complexity can be effectively reduced, and the interference of the specified characters in the log template extraction process can be avoided, thereby improving the progress of extracting the log template.

[0228] Optionally, the preprocessing module 401 is used to: obtain at least one entry for each line of log records in the log, delete the specified characters in each entry in each line of log records, and obtain each line of log records after preprocessing; or, delete the specified characters in each line of log records in the multiple lines of log records to obtain each simplified line of log records, obtain at least one entry for each simplified line of log records in the multiple lines of log, and obtain each line of log records after preprocessing.

[0229] Optionally, the designated characters are numbers and / or punctuation marks.

[0230] Alternatively, if Fig.19 As shown, the determination module 402 includes:

[0231] The splicing submodule 4021 is used to splice m consecutive non-empty entries at the first specified position of each line of log record after preprocessing to obtain a first character signature, where m is a positive integer; the determination submodule 4022 is used to determine the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after preprocessing; the grouping submodule 4023 is used to group multiple lines of log records in the log based on the first digital signature of each line of log record to obtain at least one first log record group, and the first digital signatures corresponding to all log records included in each first log record group are the same.

[0232] Optionally, the grouping submodule 4023 is used to:

[0233] The second character signature is obtained by concatenating the consecutive n non-empty entries at the second designated position of each line of log record after preprocessing, and the first designated position is different from the second designated position, and n is a positive integer; based on the first digital signature of each line of log record, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; based on the second digital signature of each line of log record, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

[0234] Optionally, the m non-empty entries are the first m non-empty entries in each line of log records.

[0235] Optionally, the determining submodule 4022 is used to:

[0236] The first hash code of the first character signature corresponding to each line of log record after the preprocessing is determined as the first digital signature of each line of log record; or, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

[0237] Optionally, the simplification process includes a remainder process or a modulo process.

[0238] Optionally, the processing module 403 is used to:

[0239] For each line of log record in each of the first log record groups, an alternative log template is generated based on the log record and the historical log template of the first log record group; when the log record matches any of the generated alternative log templates, any of the alternative log templates is determined as the new log template of the first log record group; when the log record does not match all of the alternative log templates, all of the alternative log templates are rolled back to the corresponding historical log templates and the log record, and the log template extracted from the log record is added as the new log template of the first log record group.

[0240] Optionally, Fig. 20 A possible basic hardware architecture of the computer device described in the present application is schematically provided. The computer device may be a server.

[0241] See also Fig. 20, the computer device 500 includes a processor 501 , a memory 502 , a communication interface 503 and a bus 504 .

[0242] In the computer device 500, the number of processors 501 may be one or more. Fig. 20 Only one of the processors 501 is shown. Optionally, the processor 501 may be a central processing unit (CPU). If the computer device 500 has multiple processors 501, the types of the multiple processors 501 may be different or the same. Optionally, the multiple processors 501 of the computer device 500 may also be integrated into a multi-core processor.

[0243] The memory 502 stores computer instructions and data; the memory 502 may store computer instructions and data required to implement the log template extraction method provided in the present application, for example, the memory 502 stores instructions for implementing the steps of the log template extraction method. The memory 502 may be any one or any combination of the following storage media: non-volatile memory (e.g., read-only memory (ROM), solid-state drive (SSD), hard disk (HDD), optical disk), volatile memory.

[0244] The communication interface 503 may be any one or any combination of the following devices: a network interface (eg, an Ethernet interface), a wireless network card, or other device with a network access function.

[0245] The communication interface 503 is used for the computer device 500 to perform data communication with other computer devices or terminals.

[0246] The bus 504 can connect the processor 501 to the memory 502 and the communication interface 503. In this way, the processor 501 can access the memory 502 through the bus 504, and can also use the communication interface 503 to exchange data with other computer devices or terminals.

[0247] In the present application, the computer device 500 executes the computer instructions in the memory 502, so that the computer device 500 implements the log template extraction method provided in the present application, or enables the computer device 500 to deploy a log template extraction device.

[0248] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of a server to complete the log template extraction method shown in various embodiments of the present application. For example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0249] An embodiment of the present application provides an analysis system, including: a terminal and an analysis device, wherein the analysis device includes any of the aforementioned log template extraction devices.

[0250] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, and the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0251] It should be noted that: the log template extraction device provided in the above embodiment only uses the division of the above functional modules as an example when performing log template extraction. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the log template extraction device provided in the above embodiment and the log template extraction method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0252] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0253] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise expressly defined. A refers to B, which means that A is the same as B or A is a simple variation of B.

[0254] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0255] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A log template extraction method, characterized in that: The method comprises: Delete the specified characters in each line of log records in the multi-line log records to obtain each line of log records after preprocessing; Based on each line of log records after preprocessing, determining at least one first log record group, where different first log record groups include different lines of log records in the log; For each line of log record in each of the first log record groups, merge the log record with the historical log template of the first log record group to obtain an alternative log template; When the log record matches any of the generated candidate log templates, determining any of the candidate log templates as a new log template for the first log record group; When the log record does not match any of the candidate log templates, all of the candidate log templates are rolled back to the corresponding historical log templates and the log record, and the log template extracted from the log record is added as a new log template of the first log record group; Based on the new log template for each first log record group, a log template for the log is determined.

2. The method according to claim 1, characterized in that: The step of deleting the specified characters in each line of log records in the multiple lines of log records to obtain each line of log records after preprocessing includes: Obtain at least one entry of each line of log records in the log, delete the specified character in each entry of each line of log records, and obtain each line of log records after preprocessing; Alternatively, designated characters in each line of the multiple lines of log records are deleted to obtain simplified log records, and at least one entry of each simplified log record of the multiple lines of log records is obtained to obtain the preprocessed log records.

3. The method according to claim 1 or 2, characterized in that: The designated characters are numbers and / or punctuation marks.

4. The method according to claim 1 or 2, characterized in that: The step of determining at least one first log record group based on each line of the preprocessed log record comprises: splicing m consecutive non-empty entries at the first designated position of each line of the preprocessed log record to obtain a first character signature, where m is a positive integer; Determine a first digital signature for each line of log record based on a first hash code of a first character signature corresponding to each line of log record after the preprocessing; Based on the first digital signature of each line of log records, multiple lines of log records in the log are grouped to obtain the at least one first log record group, and the first digital signatures corresponding to all log records included in each first log record group are the same.

5. The method according to claim 3, characterized in that: The step of determining at least one first log record group based on each line of the preprocessed log record comprises: splicing m consecutive non-empty entries at the first designated position of each line of the preprocessed log record to obtain a first character signature, where m is a positive integer; Determine a first digital signature for each line of log record based on a first hash code of a first character signature corresponding to each line of log record after the preprocessing; Based on the first digital signature of each line of log records, multiple lines of log records in the log are grouped to obtain the at least one first log record group, and the first digital signatures corresponding to all log records included in each first log record group are the same.

6. The method according to claim 4, characterized in that The step of grouping multiple lines of log records in the log based on the first digital signature of each line of log records to obtain the at least one first log record group includes: splicing n consecutive non-empty entries at the second designated position of each line of the preprocessed log record to obtain a second character signature, wherein the first designated position is different from the second designated position, and n is a positive integer; Based on the first digital signature of each line of log record, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; Based on the second digital signature of each line of log records, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

7. The method according to claim 5, characterized in that The step of grouping multiple lines of log records in the log based on the first digital signature of each line of log records to obtain the at least one first log record group includes: splicing n consecutive non-empty entries at the second designated position of each line of the preprocessed log record to obtain a second character signature, wherein the first designated position is different from the second designated position, and n is a positive integer; Based on the first digital signature of each line of log record, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; Based on the second digital signature of each line of log records, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

8. The method according to claim 4, characterized in that The m non-empty entries are the first m non-empty entries of each line of log records.

9. The method according to any one of claims 5 to 7, characterized in that: The m non-empty entries are the first m non-empty entries of each line of log records.

10. The method according to claim 4, characterized in that The determining the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after the preprocessing includes: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

11. The method according to any one of claims 5 to 8, characterized in that: The determining the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after the preprocessing includes: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

12. The method according to claim 9, characterized in that The determining the first digital signature of each line of log record based on the first hash code of the first character signature corresponding to each line of log record after the preprocessing includes: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

13. The method according to claim 10 or 12, characterized in that: The simplification process includes a remainder process or a modulus process.

14. The method according to claim 11, characterized in that The simplification process includes a remainder process or a modulus process.

15. A log template extraction device, characterized in that: The device comprises: A preprocessing module is used to delete specified characters in each line of log records in multiple lines of log records to obtain each line of log records after preprocessing; A determination module, configured to determine at least one first log record group based on each line of the preprocessed log record, wherein different first log record groups include different lines of log records in the log; A processing module, configured to merge, for each line of log records in each of the first log record groups, the log records with a historical log template of the first log record group to obtain an alternative log template; The processing module is further configured to, when the log record matches any of the generated candidate log templates, determine the any of the candidate log templates as a new log template for the first log record group; The processing module is further configured to, when the log record does not match any of the candidate log templates, roll back all of the candidate log templates to corresponding historical log templates and the log record, and add the log template extracted from the log record as a new log template for the first log record group; The processing module is further used to determine the log template of the log based on the new log template of each first log record group.

16. The device according to claim 15, characterized in that The preprocessing module is used to: Obtain at least one entry of each line of log records in the log, delete the specified character in each entry of each line of log records, and obtain each line of log records after preprocessing; Alternatively, designated characters in each line of the multiple lines of log records are deleted to obtain simplified log records, and at least one entry of each simplified log record of the multiple lines of log records is obtained to obtain the preprocessed log records.

17. The device according to claim 15 or 16, characterized in that The designated characters are numbers and / or punctuation marks.

18. The device according to claim 15 or 16, characterized in that The determining module comprises: A concatenation submodule, used for concatenating m consecutive non-empty entries at the first designated position of each line of the preprocessed log record to obtain a first character signature, where m is a positive integer; A determination submodule, configured to determine a first digital signature for each line of log record based on a first hash code of a first character signature corresponding to each line of log record after the preprocessing; The grouping submodule is used to group multiple lines of log records in the log based on the first digital signature of each line of log records to obtain at least one first log record group, and the first digital signatures corresponding to all log records included in each first log record group are the same.

19. The device according to claim 17, characterized in that The determining module comprises: A concatenation submodule, used for concatenating m consecutive non-empty entries at the first designated position of each line of the preprocessed log record to obtain a first character signature, where m is a positive integer; A determination submodule, configured to determine a first digital signature for each line of log record based on a first hash code of a first character signature corresponding to each line of log record after the preprocessing; The grouping submodule is used to group multiple lines of log records in the log based on the first digital signature of each line of log records to obtain at least one first log record group, and the first digital signatures corresponding to all log records included in each first log record group are the same.

20. The device according to claim 18, characterized in that The grouping submodule is used to: splicing n consecutive non-empty entries at the second designated position of each line of the preprocessed log record to obtain a second character signature, wherein the first designated position is different from the second designated position, and n is a positive integer; Based on the first digital signature of each line of log record, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; Based on the second digital signature of each line of log records, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

21. The device according to claim 19, characterized in that The grouping submodule is used to: splicing n consecutive non-empty entries at the second designated position of each line of the preprocessed log record to obtain a second character signature, wherein the first designated position is different from the second designated position, and n is a positive integer; Based on the first digital signature of each line of log record, multiple lines of log records in the log are grouped to obtain at least one second log record group; the first digital signatures corresponding to all log records included in each second log record group are the same; Based on the second digital signature of each line of log records, each second log record group in the at least one second log record group is grouped to obtain the at least one first log record group; the second digital signatures corresponding to all log records included in each first log record group are the same.

22. The device according to claim 18, characterized in that The m non-empty entries are the first m non-empty entries of each line of log records.

23. The device according to any one of claims 19 to 21, characterized in that: The m non-empty entries are the first m non-empty entries of each line of log records.

24. The device according to claim 18, characterized in that The determining submodule is used to: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

25. The device according to any one of claims 19 to 22, characterized in that: The determining submodule is used to: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

26. The device according to claim 23, characterized in that The determining submodule is used to: Determine the first hash code of the first character signature corresponding to each line of log record after the preprocessing as the first digital signature of each line of log record; Alternatively, the first hash code of each line of log record after the preprocessing is simplified to obtain the first simplified hash code of each line of log record after the preprocessing, and the first simplified hash code of each line of log record after the preprocessing is determined as the first digital signature of each line of log record.

27. The device according to claim 24 or 26, characterized in that The simplification process includes a remainder process or a modulus process.

28. The device according to claim 25, characterized in that The simplification process includes a remainder process or a modulus process.

29. A computer device, characterized in that: including a processor and a memory; When the processor executes the computer instructions stored in the memory, the computer device executes the log template extraction method according to any one of claims 1 to 14.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and the computer instructions instruct a computer device to execute the log template extraction method according to any one of claims 1 to 14.

Images