Log analysis method and device, equipment and storage medium
By building a dual-tree model based on historical logs, analyzing test logs with low structure, the problem of inaccurate analysis in the existing technology is solved, and the accuracy and efficiency of log analysis are improved.
Patent Information
- Application Number
- CN202411187702.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-08
AI Technical Summary
When the prior art performs intelligent analysis of test logs with low structural degree, it is easy to lead to inaccurate analysis results, affecting the accuracy and efficiency of the analysis.
By determining multiple character sets based on historical logs, grouping multiple times to identify constants and variable parts, a two-tree model is built to generate log templates and perform structured analysis of the original log to be processed.
Improve the intelligence and efficiency of log analysis, ensuring accurate analysis and analysis of logs with low structural degree.
Smart Images

Figure CN118916528A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of testing technology, and in particular to a method, device, equipment and storage medium for analyzing a test log. Background Art
[0002] With the rapid development of information technology and the continuous expansion of business scale, various systems, applications and services generate massive amounts of log data during operation. Therefore, intelligent review and analysis of logs are becoming increasingly important.
[0003] At present, for the intelligent analysis of logs, clustering, frequent item mining and other methods are usually used to parse the patterns of raw log data, and then artificial intelligence is used to detect anomalies and analyze the test conclusions.
[0004] However, these methods are suitable for highly structured logs. For logs with low structure, such as logs generated during the test process, using the above methods to parse the test logs can easily lead to inaccurate parsing results, thereby affecting the accuracy and efficiency of log intelligent analysis. Summary of the invention
[0005] Embodiments of the present application provide a log analysis method, apparatus, device, and storage medium.
[0006] The technical solution of the embodiment of the present application is implemented as follows:
[0007] In a first aspect, an embodiment of the present application provides a log analysis method, comprising: determining multiple character sets based on historical logs; wherein the characters in each character set have the same frequency of occurrence; based on the multiple character sets, grouping the historical logs multiple times, and determining at least one first node with a constant part and at least one second node with a variable part in each group log; based on at least one first node and at least one second node, determining a dual-tree model corresponding to each group of logs; based on the dual-tree model, determining a log template; the log template is used when analyzing the original log to be processed.
[0008] In a second aspect, an embodiment of the present application provides a log analysis device, comprising: a determination device for determining multiple character sets based on historical logs; and, based on the multiple character sets, grouping the historical logs multiple times to determine at least one first node with a constant part and at least one second node with a variable part in each group log; and, based on at least one first node and at least one second node, determining a dual-tree model corresponding to each group of logs; and, based on the dual-tree model, determining a log template; the log template is used when analyzing the original log to be processed. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0010] Figure 1a One of the schematic diagrams of a log provided for related art;
[0011] Figure 1b A second schematic diagram of a log provided for related technology;
[0012] Figure 2 One of the flow charts of a log analysis method provided in an embodiment of the present application;
[0013] Figure 3 A second flow chart of a log analysis method provided in an embodiment of the present application;
[0014] Figure 4 A schematic diagram of determining a first string and a first group log provided in an embodiment of the present application;
[0015] Figure 5 A schematic diagram of building a dual-tree model provided in an embodiment of the present application;
[0016] Figure 6 A schematic diagram of word vectorization provided in an embodiment of the present application;
[0017] Figure 7 A schematic diagram of log analysis using a deep learning model provided in an embodiment of the present application;
[0018] Figure 8 A flowchart of an exemplary log analysis method provided in an embodiment of the present application;
[0019] Fig. 9 A schematic diagram of the structure of a log analysis device provided in an embodiment of the present application;
[0020] Fig.10 A hardware entity diagram of a log analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0022] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0023] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0024] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0025] In the related art, the log of B2C business is as follows: Figure 1a As shown in the figure, it mainly contains information such as timestamp, thread, IP, port, etc., which is basically composed of variables and related parameters of the backend code. However, different from the logs of B2C business, the test logs of the server of Cloud Service Provider (CSP) are as follows: Figure 1b As shown, the degree of structuring is lower than that of logs in the B2C business model. The data is essentially a description of the test process, with high data repetitiveness and weak time series.
[0026] The following is an illustrative introduction to the log analysis method, device, electronic device, and storage medium provided in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.
[0027] In the embodiments of the present application, Figure 2 This is one of the flow charts of a log analysis method provided in the application embodiment. The method may include S101 to S104:
[0028] S101. Determine multiple character sets based on historical logs; wherein the characters in each character set have the same frequency of occurrence.
[0029] In the embodiment of the present application, there is no limitation on the type of historical logs. The historical logs may be logs generated based on business to consumer (Business to Customers, B2C) software services, or test logs based on (Cloud Service Provider, CSP) testing processes, such as BMC test logs, BIOS test logs, system logs, etc.
[0030] In some embodiments, the historical log can record the operating status of the software or system in detail, including normal operation, abnormal conditions, user operations, etc., to provide data support for subsequent problem analysis and system optimization.
[0031] In some embodiments, the history log includes a large number of characters, character strings, numbers, Boolean values, structured data, etc., which are used to record various information.
[0032] In some embodiments, a dual-tree model is obtained by training a large number of historical logs, and the logs to be processed are automatically analyzed based on the dual-tree model.
[0033] In some embodiments, based on the historical log, the method of determining multiple character sets may include:
[0034] S201, standardize the historical logs to obtain historical standard logs;
[0035] S202: Determine multiple character sets based on historical standard logs.
[0036] In the embodiment of the present application, the method of standardization processing is not limited, and it can be a unified conversion of uppercase and lowercase characters in the historical log, or it can be the deletion of special characters in the historical log.
[0037] In some embodiments, the standardization process may also be: first, uniformly convert the case of characters in the historical logs and delete special characters, and then perform regular expression prediction processing on the historical logs, that is, add a highest priority judgment, replace the characters that meet the highest priority with a constant part, and obtain a standard log with a constant part, wherein the highest priority characters include: at least one of: timestamp, unit of measurement, address, and number.
[0038] In the embodiment of the present application, the historical standard log is obtained after the historical log is standardized. The historical standard log includes multiple different characters. According to the determination principle, different characters are determined in different character sets to obtain multiple character sets.
[0039] It is understandable that the historical logs are standardized to obtain historical standard logs, which are more standardized and more conducive to operating them, realizing the division of character sets, and then determining multiple character sets.
[0040] In some embodiments, based on the historical standard log, the method for determining the multiple character sets may be: counting the frequency of characters in the historical standard log, classifying characters with the same frequency into the same character set, and obtaining the multiple character sets.
[0041] In some embodiments, according to the frequency of occurrence of the characters in the historical standard log, characters with the same frequency are divided into the same group to form a character set. Based on this, multiple character sets can be determined in the historical standard log.
[0042] For example, in character set 1 (A, B, C), A, B, and C each appear 3 times in the historical log; in character set 2 (D, E, F), D, E, and F each appear 4 times in the historical log.
[0043] It can be understood that according to the frequency of occurrence of characters in historical standard logs, characters with the same frequency are divided into the same group to form a character set, and the grouping principle of the character set is determined to facilitate identification of the constant part and the variable part in the historical standard log, thereby realizing structured parsing of the historical standard log.
[0044] The historical logs are standardized to obtain historical standard logs, which are more standardized and more convenient to operate, so as to realize the division of character sets and further determine multiple character sets.
[0045] S102: Grouping historical logs multiple times based on multiple character sets, and determining at least one first node having a constant part and at least one second node having a variable part in each group log.
[0046] In the embodiment of the present application, each character set includes at least one character or string, and the frequency of occurrence of each character or string in the historical log is the same.
[0047] In an embodiment of the present application, characters in multiple character sets are used to traverse historical logs in sequence, and all historical logs containing a certain character or string are divided into a group, based on which multiple groups of divided logs are obtained.
[0048] In the embodiment of the present application, multiple logs in each group of logs contain the same characters or strings. Based on this, the same characters or strings are used as the constant part of the group; and the remaining different characters or strings are used as the variable part of the group.
[0049] In some embodiments, both the constant part and the variable part may include multiple characters or strings, the first node is the constant part of the dual-tree model, ie, a root node or a leaf node, and the second node is the variable part of the dual-tree model.
[0050] S103: Determine a dual-tree model corresponding to each group of logs based on at least one first node and at least one second node.
[0051] In an embodiment of the present application, building a dual-tree model requires at least one first node and at least one second node, and based on the at least one first node and the at least one second node, the dual-tree model corresponding to each group of logs is determined.
[0052] It should be noted that the first node may include multiple character strings of the constant part, and the second node may include multiple character strings of the variable part.
[0053] In the embodiment of the present application, the connection relationship between the first node and the second node is not limited. In short, a dual-tree model can be formed by multiple first nodes and multiple second nodes.
[0054] In some embodiments, the dual-tree model is used to realize automatic analysis of logs. That is, the dual-tree model is first obtained through training, and when the logs need to be analyzed later, the constructed dual-tree model can be used to realize automatic analysis of the original logs to be processed.
[0055] S104. Determine a log template based on the dual-tree model; the log template is used when analyzing the original log to be processed.
[0056] In the embodiment of the present application, the dual-tree model is constructed after continuous training using a large number of historical logs. The purpose of constructing the dual-tree model is to be able to automatically analyze the subsequent original logs to be processed. Based on this, after the dual-tree model is constructed and trained, a log template is obtained, and structured data processing can be performed on the original logs to be processed through the log template, thereby realizing the conversion of the original logs into logs with structured data.
[0057] In some embodiments, after converting the original log into a log with structured data, the log is subjected to word vectorization processing to obtain a vectorized log, and automatic analysis of the log is achieved through analysis of the vectorized log to obtain an analysis result of the log.
[0058] It can be understood that, based on historical logs, multiple character sets are determined; wherein, the characters in each character set have the same frequency of occurrence, and based on multiple character sets, the historical logs are grouped multiple times, and at least one first node with a constant part and at least one second node with a variable part in each grouped log are determined, and based on at least one first node and at least one second node, the dual-tree model corresponding to each group of logs is determined. Based on this, a large number of historical logs are structuredly divided by constant parts and variable parts to obtain a trained dual-tree model. The dual-number model obtains the analysis results of the original logs to be processed by structurally dividing and automatically processing and analyzing the original logs to be processed, thereby improving the intelligence of log analysis and, thereby, improving the efficiency of log analysis.
[0059] In some embodiments, Figure 3 As shown, S102 may include S301 and S303 as follows:
[0060] S301. Based on multiple character sets, traverse historical standard logs to perform string matching to obtain at least one first matching string; the first string is the longest continuous character in the historical standard log that belongs to the same character set and has an appearance frequency greater than a preset threshold.
[0061] In the embodiment of the present application, the preset threshold is a threshold defined when determining the first character string, in order to filter out high-frequency continuous characters, that is, continuous characters with a frequency less than the preset threshold cannot be determined as the first character string. The present application does not limit the preset threshold, and it is determined based on factors such as the number of multiple character sets in the historical standard log.
[0062] In an embodiment of the present application, string matching refers to a first string related to multiple character sets found when traversing historical standard logs, and any character in the first string is a character in multiple character sets, that is, the characters in the first string are composed of characters in multiple character sets, and the characters in the first string may come from different multiple character sets.
[0063] In the embodiment of the present application, string matching is performed multiple times on the historical standard logs. Multiple first strings can be matched by traversing the historical standard logs for string matching multiple times. The number of consecutive characters of different first strings may be different. The one with a frequency of occurrence greater than a preset threshold and the largest number of consecutive characters, that is, the longest consecutive characters, is taken as the first string.
[0064] S302 . Divide multiple logs having first character strings into a group from historical standard logs to determine multiple first group logs; a first character string in a first group log is a root node.
[0065] In the embodiment of the present application, multiple logs having the first strings are grouped into a group through the determined multiple first strings, and multiple first group logs are determined. That is, the first group log is a collection of multiple logs containing the first string.
[0066] For example, Figure 4 As shown, the historical logs are standardized to obtain the historical standard log 11, and the historical standard log 11 is grouped to obtain multiple character sets. The characters in each character set are used to traverse the historical standard log 11 to match the first string BMC FW Install Results. The logs with the BMC FW Install Results string are divided into Group 1, where Group 1 is the first group log.
[0067] In the embodiment of the present application, the root node is the root node of the dual tree model and is also the first node to be determined when constructing the dual tree model. A first string in the first group log is a root node, that is, the first string is the root node when constructing the dual tree model.
[0068] S303. For each first grouping log, continue to perform string matching and grouping using at least one character set other than the matched character set in the multiple character sets until no character string with a frequency of occurrence greater than a preset threshold is matched, then stop grouping, and obtain at least one second character string that matches in the grouping log of each level in the multi-level grouping, and at least one third character string that does not match; wherein, a second character string in each level of the grouping log is a first leaf node; a third character string in each level of the grouping log is a second leaf node; the root node and the first leaf node are first nodes with a constant part, and the second leaf node is a second node with a variable part.
[0069] In an embodiment of the present application, string matching is continued using at least one character set other than the matched character set in the multiple character sets to obtain a second character string, and any character in the second character string is a character in at least one character set other than the matched character set in the multiple character sets, that is, the characters in the second character string are all composed of characters in at least one character set other than the matched character set in the multiple character sets.
[0070] In the embodiment of the present application, at least one character set other than the matched character set in the multiple character sets is used to continue string matching, wherein, in addition to obtaining a matched second character string, an unmatched third character string can also be obtained. None of the characters in the third character string appear in at least one character set other than the matched character set in the multiple character sets.
[0071] For example, Figure 4 As shown, after the first match, it is determined that the first string is BMC FW InstallResults, and then a second match is performed to divide the log containing the Update Config string into Group 2, which is also the first group log.
[0072] In some embodiments, multiple character sets are divided according to the frequency of occurrence of characters. Among the multiple character sets, the character set with the highest frequency is first selected to traverse the historical standard log, and the first string and the first group log are matched. Secondly, the character set with the second highest frequency is selected from the multiple character sets to traverse the historical standard log, and the second string and the unmatched third string are matched. According to the frequencies corresponding to the character sets, starting from the character set with the highest frequency, to the character set with the same frequency as the preset threshold, the historical standard log is traversed in sequence, and multiple second strings and third strings are matched.
[0073] In the embodiment of the present application, multiple character sets are used to match and group the historical standard logs multiple times, and a group log of one level is obtained after each match. Therefore, after the matching is completed, a multi-level group log is obtained. A second string in each level of group log is a first leaf node; a third string in each level of group log is a second leaf node, wherein the first leaf node is a constant part and the second leaf node is a variable part.
[0074] It can be understood that based on multiple character sets, historical standard logs are traversed for string matching to obtain at least one first string that matches, and multiple logs with each first string are divided into a group from the historical standard logs to determine multiple first group logs; for each first group log, at least one character set other than the matched character set in multiple character sets is used to continue string matching and grouping until no string with a frequency greater than a preset threshold is matched, and grouping is stopped to obtain at least one second string that matches in each level of the group log in the multi-level grouping, and at least one third string that does not match. Based on this, the constant part and the variable part of the log are distinguished level by level, which improves the convenience of determining the constant part and the variable part.
[0075] In some embodiments, the first node includes a root node and a first leaf node; the second node includes a second leaf node; the root node and the first leaf node are first nodes having a constant part, and the second leaf node is a second node having a variable part; based on at least one first node and at least one second node, determining a dual-tree model corresponding to each group of logs includes:
[0076] S401, constructing a tree structure with a root node according to a first direction for the second leaf nodes obtained after the first grouping;
[0077] S402: The first leaf node of the first group log forms a tree structure with the first node and the second node of the previous level in the second direction according to the order of multi-level grouping until all nodes are constructed to obtain a double tree model corresponding to each first group log.
[0078] In an embodiment of the present application, the first node includes a root node and a first leaf node; the second node includes a second leaf node; the root node and the first leaf node are first nodes with a constant part, and the second leaf node is a second node with a variable part.
[0079] In the embodiment of the present application, the historical logs are grouped multiple times based on multiple character sets, and after the first grouping, a first group log is obtained, and a second leaf node, that is, a second node of the variable part, is obtained in the first group log. Based on this, the second leaf node is constructed in a tree structure with the root node according to the first direction.
[0080] In an embodiment of the present application, after the first grouping, a first-level tree structure related to the root node can be constructed. In the case of multiple groupings, each grouping can obtain a group of first leaf nodes and second leaf nodes. The first leaf nodes and second leaf nodes obtained in the same grouping are regarded as the same level. Therefore, multiple levels of first leaf nodes and second leaf nodes can be obtained. According to the order of multi-level grouping, the first leaf nodes of each level are arranged in the second direction to form a tree structure with the first nodes and second nodes of the previous level until all nodes are constructed, and a dual tree model corresponding to each first group log is obtained.
[0081] In the embodiments of the present application, the first direction and the second direction are not limited. In short, the first direction and the second direction are two different directions. For example, the first direction is an upward direction, and the second direction is a downward direction.
[0082] For example, Figure 5As shown, after the first grouping, when the first string is determined to be BMC FW InstallResults as the root node, the second string is FAIL and PASS, that is, FAIL and PASS are the first leaf nodes of the dual-tree model. When BMC FW Install Results is the root node, and PASS and FAIL are the first leaf nodes, the root node and the first leaf node (the root node and the first leaf node are the constant part) are represented by white balls; the unmatched string is used as the second leaf node (variable part) for grid ball representation. Among them, the variable part related to the root node is connected to the root node in the upward direction, and the second leaf nodes related to the first leaf nodes (PASS and FAIL) are connected to the first leaf nodes in the downward direction.
[0083] It can be understood that the root node of the dual-tree model is determined through the first grouping; the first leaf node and the second leaf node of the dual-tree model are determined through multiple groupings, the root node and the first leaf node are used to represent the constant part, and the second leaf node represents the variable part, and the constant part and the variable part are displayed in order of frequency. The construction of the dual-tree model is realized, and then, after the dual-tree model is trained, the automatic analysis of the original log can be realized, which improves the efficiency of log analysis.
[0084] In some embodiments, the method further includes S501 to S503:
[0085] S501: Standardize the acquired original log to obtain a standard log.
[0086] In the embodiments of the present application, there is no limitation on the method of standardization processing. In short, the processing method of the standard log can be obtained.
[0087] In some embodiments, based on a regular expression, the characters in the original log or the preprocessed log are judged with the highest priority, and the characters that meet the highest priority are replaced with a constant part to obtain a standard log with a constant part; the highest priority characters include: at least one of: a timestamp, a unit of measurement, an address, and a number; the preprocessed log is a log after the original log is case-converted.
[0088] S502, parsing the standard log using a log template to determine a templated log having a constant part with a consistent format;
[0089] S503: Analyze the templated log using a log analysis network to determine the analysis result.
[0090] In the embodiment of the present application, the log template is a reusable template file obtained by training the historical standard file. The log template is used to parse the standard log to determine the templated log, that is, the standard log is converted into the templated log, and the templated log has a constant part with the same format.
[0091] In the embodiments of the present application, there is no limitation on the method of log network analysis. In short, the templated log can be analyzed. For example, the templated log is analyzed using a word vectorization model and a deep learning network, and based on this, the analysis result of the templated log is obtained.
[0092] It can be understood that the standard log is parsed through the log template to determine the templated log. The templated log is a log with a constant part with a consistent format. Therefore, it is helpful to use the log analysis network to perform automatic analysis on the templated log, and realize the automatic conversion and automatic analysis of the standard log to the templated log, thereby improving the analysis efficiency of the standard log.
[0093] In some embodiments, S502 may include S601 to S602:
[0094] S601, parsing the log template using the script file to determine the template format;
[0095] S602: Using the template format, convert the standard log into a templated log having a constant part with a consistent format.
[0096] In the embodiment of the present application, there is no limitation on the script file. For example, the script file can be a program written in a programming language such as Python, BashPower Shell, etc.
[0097] In some embodiments, the script file generally contains code for reading, parsing, and processing a log template, as well as logic for determining the format of the log template.
[0098] In the embodiment of the present application, the template format can cover the logs in the log template, and the logs under the template format have a constant part with a consistent format.
[0099] In some embodiments, the template format may be a string format that contains placeholders for inserting variable values.
[0100] In the embodiment of the present application, a template format is used to convert a standard log into a templated log having a constant part with a consistent format.
[0101] It can be understood that the script file parses the log template, determines the template format, and uses the template format to convert the standard log into a templated log with a constant part with a consistent format. Through the log template, the automatic conversion of the standard log to the templated log is realized, wherein the template format can cover the logs in the log template, and the logs under the template format have a constant part with a consistent format, making it easier to use and analyze. Based on this, the analysis efficiency of the standard log is improved.
[0102] In some embodiments, S503 may include S701 to S702:
[0103] S701, using a word vectorization model, performing a word vectorization operation on the templated log to obtain a vectorized log;
[0104] S702: Use a deep learning model to perform deep learning processing on the vectorized logs, and determine the analysis results of the logs.
[0105] In the embodiments of the present application, the type of vectorization model is not limited, and in short, it is a model that can perform word vectorization operations on templated logs, such as FastText model, Word2Vec model and GloVe model.
[0106] In some embodiments, a word vectorization model is used to perform word vectorization operations on templated logs, and text data in the templated logs are converted into vector data. The purpose of converting into vector data is to make it easier for a deep learning model to deeply learn the vectorized logs.
[0107] In some embodiments, a word vectorization model is used to convert each word in the templated log into a corresponding vector representation, so that each templated log is converted into a vector sequence.
[0108] For example, Figure 6 As shown, the templated log “*BMC FW CycleResults PASS*” is vectorized using the FastText model to obtain a vectorized array 12 with a length of 100.
[0109] In the embodiments of the present application, there is no limitation on the type of deep learning model. In short, it is a model that can determine the analysis results of the log.
[0110] Exemplarily, the deep learning model can be a recurrent neural network, a long short-term memory network, or a convolutional neural network, etc.
[0111] In some embodiments, a deep learning model is used to perform deep learning processing on the vectorized logs, and the vectorized log data is input into the deep learning model. The deep learning model performs a series of complex calculations and operations on these vectors to extract features and patterns in the vectorized logs.
[0112] For example, the neural network model Transformer20 is used to perform deep learning processing and analysis on vectorized logs. The analysis diagram of the neural network model Transformer20 is shown in FIG. Figure 7 As shown, the encoder 21 of the neural network model Transformer 20 performs deep learning processing on the vectorized log "-9.59068420e-05,…,3.61223763e-04.", and decodes and analyzes "The BMC firmware upgrade is successful" through the encoder 22.
[0113] It is understandable that the word vectorization model converts the log text into a vector form to facilitate the efficient processing of the subsequent deep learning model. The deep learning model can perform complex learning and analysis on the vectorized log data, more accurately determine the analysis results of the log, and improve the reliability of the analysis results of the log.
[0114] The above log analysis method is described below in conjunction with an application example. However, it should be noted that this example is only for better illustrating the present application and does not constitute an improper limitation on the present application.
[0115] Next, taking the test log of the server during the test process as an example, the exemplary application of the embodiment of the present application in this application scenario will be explained.
[0116] The present application embodiment provides a log analysis method. Figure 8 The schematic diagram of the operation flow and data flow of the method is shown in FIG. 1 . The method may include: S1001 to S1004:
[0117] S1001. Historical log preprocessing.
[0118] In the embodiment of the present application, the data in the historical log is subjected to the standardization of letter case, the deletion of special characters, and the addition of the highest priority word mask processing operation (part of the data is uniformly replaced with a standardized constant part through a regular expression), for example, the word mask processing is performed on the timestamp, the measurement unit, etc. After preprocessing the historical log, the historical standard log is obtained.
[0119] S1002. Historical standard log analysis.
[0120] In an embodiment of the present application, logs can be grouped by the longest common pattern (first character string), and historical standard logs with the same longest common pattern can be grouped to obtain a first grouped log. Furthermore, considering that characters with higher and lower frequencies than the longest common pattern may be constant parts, a threshold (preset threshold) is set for the number of different characters in the first grouped log to further divide the group. When the new constant part (first leaf node) is divided into the longest common pattern, the group will be further divided until the grouping of the historical standard log is accurate and all constant parts are found. Among them, the constant part (first node) and the variable part (second node) serve as the root node or the first leaf node for constructing a dual-tree model, and the variable part serves as the second leaf node for constructing a dual-tree model. The dual-tree model is composed of a root point, a first leaf node, and a second leaf node.
[0121] S1003: word vectorization operation.
[0122] In an embodiment of the present application, after the dual-tree model is constructed and trained, a log template is obtained. The log template can be used to complete the conversion of original log data into structured data, and a templated log with a constant part with a consistent format can be obtained. The FastText model is then used to perform word vectorization operations on the templated log to convert the text data in the templated log into vector data.
[0123] S1004. Analyze the results using a neural network model.
[0124] In the embodiment of the present application, after the word vectorization operation is completed and the vectorized log is obtained, the vectorized log is analyzed by the neural network model Transformer, wherein the fixed-length vectorized log data is sent to the constructed neural network model Transformer, and the learning rate Ir is set to 0.01, the number of iterations is 100, the optimizer is set to Adam, and the loss function uses a combination loss function of the cross entropy loss function and the dice function. The specific calculation formula (1) of the combined loss function is as follows;
[0125]
[0126] Among them, α and β are different constant weights, N is the number of samples, X i is the prediction result of the i-th sample, Y i is the actual result of the i-th sample, |X| is the set of predicted results, |Y| is the set of actual results, |X∩Y| is the intersection of the predicted results and the actual results, |X i ∩Y i | is the intersection of the predicted result and the actual result of the i-th sample.
[0127] In the embodiment of the present application, the historical logs are preprocessed to obtain the historical standard logs, which improves the standardization of the historical logs, and the construction of the double-tree model is realized by parsing the historical standard logs. The log template is obtained by training the double-tree model, and the log template is used to convert the original log to be processed into a structured templated log, which improves the feasibility of intelligent analysis of the original log. The analysis results of the original logs are obtained by performing word vectorization operations on the templated logs and analyzing them with a neural network model, which improves the intelligence of the original log analysis, and then improves the efficiency of the original log analysis.
[0128] The present application embodiment provides a log analysis device 900, such as Fig. 9 As shown, the log analysis device 900 includes:
[0129] Determination module 901 is used to determine multiple character sets based on historical logs; wherein the characters in each character set have the same frequency of occurrence; and, based on at least one first node and at least one second node, determine the dual-tree model corresponding to each group of logs; and, based on the dual-tree model, determine the log template; the log template is used when analyzing the original log to be processed.
[0130] In some embodiments, the determination module 901 is further used to perform standardization processing on the historical log to obtain a historical standard log; and determine multiple character sets based on the historical standard log.
[0131] In some embodiments, the log analysis device 900 further includes a matching module 902 and a grouping module 903.
[0132] The matching module 902 is used to group multiple logs having first character strings into one group from historical standard logs, and determine multiple first group logs.
[0133] The grouping module 903 is used to continue to perform string matching and grouping for each first grouping log using at least one character set other than the matched character set in multiple character sets until no character string with an occurrence frequency greater than a preset threshold is matched, and then stop grouping to obtain at least one second character string that matches in the grouping log of each level in the multi-level grouping, and at least one third character string that does not match.
[0134] In some embodiments, the log analysis device 900 also includes a construction module 904, which is used to construct a tree structure with the root node according to the first direction for the second leaf node obtained after the first grouping; and to form a tree structure with the first node and the second node of the previous level in the second direction according to the order of multi-level grouping, until all nodes are constructed, thereby obtaining a double tree model corresponding to each first group log.
[0135] In some embodiments, the determination module 901 is further used to count the frequency of characters in the historical standard logs, and to classify characters with the same frequency into the same character set to obtain multiple character sets.
[0136] In some embodiments, the log analysis device 900 further includes a parsing module 905.
[0137] The determination module 901 is further used to perform standardization processing on the acquired original log to obtain a standard log; and to use a log analysis network to analyze the templated log to determine the analysis result.
[0138] The parsing module 905 is used to parse the standard log using the log template to determine a templated log having a constant part with a consistent format.
[0139] In some embodiments, the log analysis device 900 further includes a conversion module 906.
[0140] The determination module 901 is further used to parse the log template using the script file to determine the template format.
[0141] The conversion module 906 is used to convert the standard log into a templated log having a constant part with a consistent format by using the template format.
[0142] In some embodiments, the conversion module 906 is further used to determine the highest priority of characters in the original log or preprocessed log according to the regular expression, and replace the characters with the highest priority with the constant part to obtain a standard log with the constant part.
[0143] In some embodiments, the conversion module 906 is further used to utilize a word vectorization model to perform a word vectorization operation on the templated log to obtain a vectorized log.
[0144] The parsing module 905 is also used to use a deep learning model to perform deep learning processing on the vectorized log and determine the analysis result of the log.
[0145] It can be understood that, based on historical logs, multiple character sets are determined; wherein, the characters in each character set have the same frequency of occurrence, and based on multiple character sets, the historical logs are grouped multiple times, and at least one first node with a constant part and at least one second node with a variable part in each grouped log are determined, and based on at least one first node and at least one second node, the dual-tree model corresponding to each group of logs is determined. Based on this, a large number of historical logs are structuredly divided by constant parts and variable parts to obtain a trained dual-tree model. The dual-number model obtains the analysis results of the original logs to be processed by structurally dividing and automatically processing and analyzing the original logs to be processed, thereby improving the intelligence of log analysis and, thereby, improving the efficiency of log analysis.
[0146] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0147] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.
[0148] The present application embodiment provides a log analysis device, such as Fig.10 As shown, the hardware entity of the log analysis device 1000 includes: a processor 1001 and a memory 1003, the memory stores a computer program that can be run on the processor, and the processor implements the steps in the log analysis method when executing the program.
[0149] It should be noted that the processor 1001 generally controls the overall operation of the log analysis device 1000. The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and each module in the log analysis device 1000 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM). The log analysis device may also include: a communication interface 1002; the communication interface 1002 enables the log analysis device to communicate with other terminals or servers through the network. Data transmission can be carried out between the processor 1001, the communication interface 1002 and the memory 1003 through the bus 1004.
[0150] Based on the foregoing embodiments, the modules included in the log analysis device provided in the embodiments of the present application, as well as the units included in each module, etc., can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0151] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, all steps in the log analysis method are implemented. The computer-readable storage medium can be transient or non-transient.
[0152] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device and storage medium embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device and storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0153] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.
[0154] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0155] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. The above are only implementation methods of this application, but the protection scope of this application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered within the protection scope of this application.
Claims
1. A log analysis method, comprising: Based on the historical log, multiple character sets are determined; wherein the characters in each character set have the same frequency of occurrence; Based on the multiple character sets, the historical logs are grouped multiple times, and at least one first node having a constant part and at least one second node having a variable part in each grouped log is determined; Determine the dual-tree model corresponding to each group of logs based on at least one first node and at least one second node; Based on the dual-tree model, the log template is determined; the log template is used when analyzing the original log to be processed.
2. The analysis method according to claim 1, wherein determining a plurality of character sets based on historical logs comprises: Performing standardization processing on the historical log to obtain a historical standard log; Based on the historical standard log, the plurality of character sets are determined.
3. The analysis method according to claim 2, wherein the historical standard logs are grouped multiple times based on the multiple character sets, and at least one first node having a constant part and at least one second node having a variable part in each grouped log is determined, comprising: Based on the multiple character sets, traverse the historical standard logs to perform string matching to obtain at least one first matching string; The first character string is the longest continuous character in the historical standard log that belongs to the same character set and has an appearance frequency greater than a preset threshold; From the historical standard log, a plurality of logs having respective first character strings are divided into a group to determine a plurality of first group logs; a first character string in the first group log is a root node; For each of the first grouping logs, continue to perform string matching and grouping using at least one character set other than the matched character set in the multiple character sets until no character string with an appearance frequency greater than a preset threshold is matched, then stop grouping, and obtain at least one second character string that matches in the grouping log of each level in the multi-level grouping, and at least one third character string that does not match; Among them, a second string in each level of group log is a first leaf node; a third string in each level of group log is a second leaf node; the root node and the first leaf node are first nodes with a constant part, and the second leaf node is a second node with a variable part.
4. According to the analysis method of claim 2 or 3, the first node includes a root node and a first leaf node; the second node includes a second leaf node; the root node and the first leaf node are first nodes having a constant part, and the second leaf node is a second node having a variable part; The determining, based on at least one first node and at least one second node, the dual-tree model corresponding to each group of logs includes: After the first grouping, the second leaf nodes obtained are constructed into a tree structure with the root node according to the first direction; The first leaf node of the first group log forms a tree structure with the first node and the second node of the previous level in the second direction according to the order of multi-level grouping until all nodes are constructed, thereby obtaining the dual-tree model corresponding to each first group log.
5. The analysis method according to claim 2, wherein determining the plurality of character sets based on the historical standard log comprises: The frequency of characters in the historical standard log is counted, and characters with the same frequency are classified into the same character set to obtain the multiple character sets.
6. The analysis method according to any one of claims 1 to 5, further comprising: Standardize the obtained original logs to obtain standard logs; Parsing the standard log using the log template to determine a templated log having a constant part with a consistent format; The templated log is analyzed using a log analysis network to determine an analysis result.
7. The analysis method according to claim 6, wherein the step of parsing the standard log using the log template to determine a templated log having a constant portion with a consistent format comprises: Parsing the log template using a script file to determine the template format; The standard log is converted into the templated log having a constant part with a consistent format by using the template format.
8. According to the analysis method of claim 6 or 7, the step of standardizing the acquired original log to obtain a standard log comprises: According to the regular expression, characters in the original log or the preprocessed log are judged with the highest priority, and characters meeting the highest priority are replaced with constant parts to obtain the standard log with the constant part; The characters with the highest priority include: at least one of a timestamp, a measurement unit, an address, and a number; The preprocessed log is a log after case conversion is performed on the original log.
9. According to any one of claims 1 to 8, the log analysis network comprises a word vectorization model and a deep learning model; the step of using the log analysis network to analyze the templated log and determine the analysis result comprises: Using the word vectorization model, the templated log is subjected to a word vectorization operation to obtain a vectorized log; The deep learning model is used to perform deep learning processing on the vectorized log to determine the analysis result of the log.
10. A log analysis device, the device comprising: A determination module is used to determine multiple character sets based on historical logs; wherein the characters in each character set have the same frequency of occurrence; and, based on at least one first node and at least one second node, determine the dual-tree model corresponding to each group of logs; and, based on the dual-tree model, determine the log template; the log template is used when analyzing the original log to be processed. The grouping module is used to group the historical logs multiple times based on the multiple character sets, and determine at least one first node with a constant part and at least one second node with a variable part in each group log.
Citation Information
Patent Citations
Log template extraction method and device
CN113128213A
Log processing method and device and electronic equipment
CN116303875A
Vehicle log anomaly detection method and device, electronic equipment and storage medium
CN117093443A
Online log analysis method, system, equipment and medium
CN117520289A
