A log analysis method, device, storage medium and electronic equipment

Abstract

This application discloses a method, apparatus, storage medium, and electronic device for log analysis. The method includes: reading the log file to be analyzed line by line; for each line of log text, sequentially matching keyword rules, including error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules; marking the operation name and error attribute of the current log line according to the matched keyword rule type; counting the total number of calls for the operation name of each log line and the number of calls for each time point, and outputting the statistical results. The log analysis method and apparatus provided in this application ensure the clarity and uniqueness of the analysis results through clear log attribute marking rules, helping users quickly distinguish between error logs and normal operation logs, and improving the efficiency of log fault diagnosis and status analysis.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for log analysis. Background Technology

[0002] Software typically has logs that record its runtime status, critical operations, error messages, and other information. Software maintenance personnel can analyze these logs to assess the software's operational status and intervene proactively when problems or potential issues arise. Software developers can analyze logs to understand the actual execution logic of the code and resolve bugs. Therefore, from operating systems to application software, logging is generally essential. Because natural language logs are the easiest to record and analyze and the most readily understandable, writing logs in natural language in text files is the most common form of logging.

[0003] A large software system, comprising various modules, each with its own logs, can easily accumulate over 10 million log lines per day. If a software operations engineer were to analyze these lines one by one, it would take months. On the one hand, detailed log recording is essential for troubleshooting; on the other hand, the sheer volume of daily log entries results in insufficient manpower for analysis. Software developers and operations personnel urgently need a technology for rapid log analysis to significantly improve efficiency. Summary of the Invention

[0004] This application provides a log analysis method, apparatus, storage medium, and electronic device. The clear log attribute marking rules ensure the clarity and uniqueness of the analysis results, helping users quickly distinguish between error logs and normal operation logs, and improving the efficiency of log fault diagnosis and status analysis. Firstly, this application provides a log analysis method, comprising the steps of: reading the log file to be analyzed line by line; for each line of log text, sequentially matching keyword rules, the keyword rules including: error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules; marking the operation name and error attribute of the current log line according to the matched keyword rule type; counting the total number of calls for the operation name of each log line in all log lines and the number of calls for the operation at each time point, and outputting the statistical results.

[0005] In one optional implementation, each log line is sequentially matched against the keyword rules according to a preset priority rule. Each log line is matched only with the rule of the highest priority, and no further rules are matched after a successful match. Specifically, this includes: prioritizing the matching of error code rules: if the log contains a specified error code keyword and the error code value is greater than zero, it is marked as an error log and a corresponding operation name is generated; if the error code rule is not matched, the error keyword rule is matched: if the log contains a preset error keyword, it is directly marked as an error log and a corresponding operation name is generated; if the error keyword rule is not matched, the special operation rule is matched: if a match is found, it is uniformly classified as a specified special operation name; if the special operation rule is not matched, the key operation rule is matched: if a match is found, a precise key operation name is generated; if the key operation rule is not matched, the ordinary operation rule is matched: the operation name is generated by truncating the symbols of the ordinary operation rule; if all five types of rules are not matched, the entire log line text is directly used as the operation name, and a rule supplement prompt is triggered.

[0006] In one optional implementation, the step of marking the operation name and error attribute of the current log line according to the matched keyword rule type includes: logs that match error code rules or error keyword rules are uniformly marked as error logs; logs that match special operation rules, key operation rules, or normal operation rules are marked as normal operation logs.

[0007] In one optional implementation, the total number of calls for the operation name in each log line and the number of calls for each time point in all log lines includes: traversing all operation name rules in the keyword rules and analyzing the operation names in each log line; incrementing the count by 1 for each matched operation name according to the operation name counting dictionary; parsing the timestamps in the log lines, maintaining a mapping table between time points and operation count dictionaries, and counting the total number of calls for all operations at the same time point; outputting the operation name, number of calls, percentage, error flag; and the percentage of time point, number of operations, and time dimension.

[0008] Secondly, embodiments of this application also provide a log analysis apparatus, comprising: The analysis unit is used to read the log file to be analyzed line by line. For each line of log text, it matches keyword rules in sequence. The keyword rules include: error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules. The tagging unit allows the user to tag the operation name and error attribute of the current log line based on the matched keyword rule type. The statistics unit is used to count the total number of calls for the operation name of each log line in all log lines and the number of operation calls at each time point, and output the statistical results.

[0009] In one optional implementation, the analysis unit sequentially matches the keyword rules according to a preset priority rule. Each log line is matched only with the rule of the highest priority. Once a match is successful, subsequent rules are not matched. Specifically, this includes: prioritizing the matching of error code rules: if the log contains a specified error code keyword and the error code value is greater than zero, it is marked as an error log and a corresponding operation name is generated; if the error code rule is not matched, the error keyword rule is matched: if the log contains a preset error keyword, it is directly marked as an error log and a corresponding operation name is generated; if the error keyword rule is not matched, the special operation rule is matched: if a match is found, it is uniformly classified as a specified special operation name; if the special operation rule is not matched, the key operation rule is matched: if a match is found, a precise key operation name is generated; if the key operation rule is not matched, the ordinary operation rule is matched: the operation name is generated by truncating the symbols of the ordinary operation rule; if all five types of rules are not matched, the entire log line text is directly used as the operation name, and a rule supplement prompt is triggered.

[0010] In one optional implementation, the marking unit is used to mark the operation name and error attribute of the current log line according to the matched keyword rule type. Specifically, logs that match error code rules or error keyword rules are uniformly marked as error logs; logs that match special operation rules, key operation rules, or ordinary operation rules are marked as normal operation logs.

[0011] In one optional implementation, the statistics unit is used to count the total number of calls for the operation name in each log line and the number of calls for each time point. Specifically, it iterates through all operation name rules in the keyword rules and analyzes the operation names in each log line; according to the operation name counting dictionary, the count is incremented by 1 for each matched operation name; it parses the timestamps in the log lines, maintains a mapping table between time points and operation count dictionaries, and counts the number of calls for all operations at the same time point; it outputs the operation name, number of calls, percentage, error flag, and the percentage of time point, number of operations, and time dimension.

[0012] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the log analysis method as described in any of the first aspects.

[0013] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the processor executes a program in the memory to implement the log analysis method as described in any of the first aspects.

[0014] This application provides a log analysis method, apparatus, storage medium, and electronic device. The clear log attribute marking rules ensure the clarity and uniqueness of the analysis results, helping users quickly distinguish between error logs and normal operation logs, and improving the efficiency of log fault diagnosis and status analysis. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a log analysis method provided in this application embodiment; Figure 2 This is a flowchart of a log analysis method provided in an embodiment of this application; Figure 3 This is a flowchart of the log analysis method in the embodiments of this application; Figure 4 This is a schematic diagram of the log analysis device in the embodiments of this application; Figure 5 This is a schematic diagram of an electronic device used to perform a log analysis method, as provided in an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. According to one aspect of the embodiments of this application, a log analysis method is provided. Optionally, in this embodiment, the above-described log analysis method may be applied to, but is not limited to, [examples of applications such as...]. Figure 1The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., log analysis services) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to store log files to be analyzed, log keyword rule XML files, and analysis result data.

[0018] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), a laptop computer, an industrial control computer, etc. Server 104 may be, but is not limited to, a cloud server, a server cluster, or a dedicated log analysis server, etc. The log analysis method of this application embodiment can be executed by server 104, by terminal device 102, or by both server 104 and terminal device 102. Alternatively, the log analysis method of this application embodiment can be executed by a client installed on terminal device 102. Taking the log analysis method in this embodiment being executed by terminal device 102 and / or server 104 as an example, Figure 2 This is a flowchart illustrating an optional log analysis method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps: Step S202: Read the log file to be analyzed line by line, and match the keyword rules for each line of log text.

[0019] Optionally, in this embodiment, the log file to be analyzed may include, but is not limited to, log data generated during the operation of an optical transmission network management system, server, application, etc., covering system operations, status changes, error messages, service execution, etc., and its format may be structured logs, unstructured text logs, etc. For example, logs generated by the network management system that include timestamps, process IDs, and service operation descriptions, and text logs containing error information output during the operation of the application, etc.

[0020] Before reading the log file to be analyzed line by line as described above, load the log keyword rule XML file and parse it to obtain the preset keyword rules.

[0021] The keyword rules mentioned above include, but are not limited to: error code rules, error keyword rules, special operation rules, key operation rules, and general operation rules. The core definitions and functions of each type of rule are as follows: Error code rules: Used to identify error code information in logs, including configurations such as error code keywords and error code value character markers. It can match logs containing specified keywords and error code values ​​> 0 and mark them as error logs; Error keyword rules: Used to identify logs containing preset error keywords; if a match is found, the log will be marked as an error log. Special operation rules: used to classify logs containing specific keywords into the same operation without parsing other log content; Key operation rules: Used to accurately identify key business operation logs containing specific keywords, and extract the content from the keyword to the end of the log line as the operation name; Normal operation rules: Used to process ordinary logs that do not match the above four types of rules, and generate operation names by truncating the log content according to preset symbols.

[0022] Loading the log keyword rule XML file and parsing it to obtain the preset keyword rules can be understood as follows: After the main program starts, it reads the log keyword rule XML file through an XML parser and extracts the default keyword rules from the file. <lineobject>The tags are parsed into rule objects, and configuration information such as keywords and operation names for each rule is extracted. The rules are then prioritized according to the default order of "error code rules → error keyword rules → special operation rules → key operation rules → general operation rules" and stored in a rule list for subsequent log matching. For example, error code rules containing the keyword "Fail", error keyword rules containing the keyword "error", and special operation rules containing the keyword "Notify" are parsed to form an ordered set of rules.

[0023] Optionally, in this embodiment of the application, the aforementioned log keyword rule XML file may include, but is not limited to, a pre-configured rule file for log classification and operation name extraction, the core structure of which consists of ` <lineobject>The tag defines each keyword rule, and each rule contains... <stringstart>`(keyword definition) and` <functionend>The `(Operation Name Rule Definition)` tag has two core sub-tags, one for regular operation rules and the other for general operation rules. <stringstart>The tag content is empty. This XML file supports flexible configuration, and the rule content and sorting can be adjusted according to changes in log format and analysis needs. The rule configuration meets specific constraints: multiple configurations are supported for error code rules, error keyword rules, special operation rules, and key operation rules, while only one ordinary operation rule is allowed.

[0024] Step S204: Based on the matched keyword rule type, mark the operation name and error attribute of the current log line. For example, marking the operation name and error attribute of the current log line according to the matched keyword rule type can be understood as follows: read the log to be analyzed line by line through the file stream, extract the timestamp and core content of each log line, and then match them in order according to the rule priority. If a match is successful, the operation name and error attribute are generated. If not all rules are matched, the entire log line is used as the operation name. For example, a log line "2019-12-16T15:23:30.527422>[4] [0][0x0083e780] Fail Auto PV_FinishCircuit: 55082" is extracted. The timestamp "2019-12-16T15:23:30" and the core content "Fail Auto PV_FinishCircuit: 55082" are extracted. The error code rules are matched first. The keyword "Fail" is hit. The error code "55082" (value > 0) is extracted and the operation name "Fail:55082" is generated and marked as an error log. Optionally, in the embodiments of this application, the above-mentioned error attribute may include, but is not limited to, tagging information used to identify whether a log is an error log. Only logs that match the error code rule (error code value > 0) or the error keyword rule are tagged as "error logs", and logs that match the other rules are tagged as "normal operation logs". This attribute provides a core reference for log fault diagnosis.

[0025] Optionally, in the embodiments of this application, the above-mentioned operation name may include, but is not limited to, a name generated according to the rule type of log matching to identify the core business content or error information of the log, such as "Fail:55082" generated by error code rule matching, "Notify" generated by special operation rule matching, "MSMPMain::CNetworkCross::GetMutiObj" generated by key operation rule matching, etc., and each operation name uniquely corresponds to a type of log content.

[0026] Step S206: Count the total number of calls for the operation name of each log line in all log lines and the number of calls for the operation at each time point, and output the statistical results. Optionally, in the embodiments of this application, the above statistical results may include, but are not limited to, two types of core data: first, the total number of calls for each operation name, including the operation name, the number of calls, the proportion of the total number of log lines, and the error identifier; second, the number of operation calls for each time point, including the time point accurate to the second, the total number of operations at that time point, and the proportion of the total number of log lines. The statistical results are output in a standardized format such as an Excel spreadsheet.

[0027] For example, the above-mentioned statistics of all log lines, generating two types of statistical results, can be understood as follows: maintaining an operation name count dictionary and a time point count dictionary, traversing all log lines, incrementing the count in the corresponding dictionary for each matched operation name, parsing the timestamp to update the time point count, and finally calculating the percentage of calls for each operation name and the percentage of operations for each time point. For example, the statistics show that the "Notify" operation was called 850 times, accounting for 22.97% of the total log lines, and the operation at the time point "2019-12-16T15:23:30" was called 156 times, accounting for 4.22% of the total log lines.

[0028] For example, the above output and display of statistical results can be understood as follows: writing the two types of statistical results into an Excel spreadsheet, generating an operation name statistics worksheet and a time point statistics worksheet, supporting user-defined output paths and filenames. Simultaneously, if there are a large number of logs that do not match any rules, the output rules will be supplemented and optimized with prompts. For example, the output might include a table containing operation name, call count, percentage, and error identifier, and a table containing time point, operation count, and time dimension percentage, with the prompt "Currently, there are XX logs that do not match any rules; it is recommended to add corresponding rules in the XML keyword rules." Through the embodiments of this application, different log formats can be flexibly adapted by loading XML rules, the uniqueness and accuracy of log classification can be ensured by priority matching, and operation frequency and time distribution information can be quickly output through global statistics. This solves the problems of low efficiency, easy omission, and inability to have a global control in traditional manual analysis of massive logs, improves the level of intelligence of log analysis and the efficiency of operation and maintenance fault diagnosis, and provides an efficient solution for log processing. It should be noted that it is not limited to analyzing logs of optical transmission network management systems. Modifying XML rules can adapt to log analysis of other systems. As an optional solution, the keyword rules are configured as follows: (1) Error code rules, error keyword rules, special operation rules, and key operation rules all support the configuration of multiple rules; (2) Only one rule is allowed to be configured in a normal operation rule; (3) All rules are prioritized according to the order in which they are written in the XML file. The rules that appear earlier have higher priority than the rules that appear later. A log line only matches the rule with the highest priority. Once a match is successful, no further rules will be matched. Optionally, in this embodiment, the multiple configurations of the above-mentioned error code rules may include, but are not limited to, rule settings for different error code keywords. For example, error code rules containing keywords such as "Fail", "ErrorCode", and "Ret" can be configured respectively, and each rule can be set with a corresponding error code value character marker (such as ":" or "=") to meet the recognition requirements of error logs in different formats. The multiple configurations of the above-mentioned error keyword rules may include, but are not limited to, rule settings for different error-related terms. For example, rules containing keywords such as "error", "failed", "timeout", and "exception" can be configured respectively, comprehensively covering the recognition scenarios of various error logs. The multiple configurations of the above-mentioned special operation rules may include, but are not limited to, rule settings for different general operations. For example, rules containing keywords such as "Notify", "EndTransaction", and "StartTransaction" can be configured respectively, unifying the logs of similar general operations and simplifying statistical results. The multiple configurations of the above-mentioned key operation rules may include, but are not limited to, rule settings for different core business operations. For example, rules containing keywords such as "MSMPMain::", "CNetworkCross::", and "GetProjectSNCData" can be configured respectively, accurately identifying key business operation logs. The configuration of a single general operation rule can include, but is not limited to, setting multiple operation name ending markers (such as "", "[" "(")) to extract the core operation content of various general logs, ensuring that logs that do not match other rules can be effectively processed. For example, the configuration and priority mechanism of the above keyword rules can be understood as follows: First, configure two error code rules (keywords "Fail" and "ErrorCode") in the XML file; then configure two error keyword rules (keywords "error" and "failed"); next, configure two special operation rules (keywords "Notify" and "EndTransaction"); then configure one priority operation rule (keyword "MSMPMain::"); and finally configure one normal operation rule. The rule priorities decrease in the above order. When a log line contains both the keywords "error" and "Notify", because the error keyword rules have a higher priority than the special operation rules, the error keyword rules will be matched first, generating the corresponding operation name and terminating subsequent matching.

[0029] Through the embodiments of this application, the flexible configuration of keyword rules meets the log analysis needs of multiple scenarios, the priority mechanism ensures the uniqueness and accuracy of log classification, avoids analysis chaos caused by multiple rule conflicts, and improves the reliability and consistency of log analysis.

[0030] As an optional approach, the keyword matching rules according to default priority include: Error code matching priority rule: If the log contains the specified error code keyword and the error code value is greater than 0, it is marked as an error log and a corresponding operation name is generated; If no match is found, continue matching the error keyword rules: if the log contains a preset error keyword, it will be directly marked as an error log and a corresponding operation name will be generated; If no match is found, continue matching the special case operation rules: if a match is found, all will be classified as the specified special case operation name; If no match is found, continue matching the key operation rules: if a match is found, generate a precise key operation name; If no match is found, the final match will be the normal operation rule: the operation name will be generated by truncating the symbols according to the normal operation rule. If none of the rules are matched, the entire log line text will be used as the operation name, and a rule supplement prompt will be triggered.

[0031] Figure 3 This is a flowchart illustrating the log analysis method in this optional example, such as... Figure 3 As shown, the log analysis method's process can include: First, reading the operation name rule; then checking if Keysymbol is empty. If not empty, proceeding to the error code handling branch, first retrieving the error code value and determining if the retrieval was successful. If unsuccessful, directly checking if a next operation name rule exists. If successful, recording the error identifier and generating the error code operation name; then checking if a next operation name rule exists. If it exists, returning to the operation name rule reading step; otherwise, the process ends. If Keysymbol is empty, continuing to check if ErrorTag is empty. If ErrorTag is not empty, recording the error identifier and generating the error keyword operation name; then checking if a next operation name rule exists. If it exists, returning to the operation name rule reading step; otherwise, the process ends. If ErrorTag is empty, continuing to check if Symbol is empty. If Symbol is empty, directly generating the key operation name; then checking if a next operation name rule exists. If it exists, returning to the operation name rule reading step; otherwise, the process ends. If the log line is not empty, check if it contains a Symbol. If it does not, check if there is a next operation name rule. If it does, generate a normal operation name, and then check if there is a next operation name rule. If it does, return to the step of reading the operation name rule; otherwise, the process ends.

[0032] Optionally, in this embodiment, the matching logic of the above error code rule may include, but is not limited to: first determining whether the core content of the log contains the keyword of the current error code rule; if it does, then extracting the subsequent number as the error code value according to the error code value character marker (such as ":"=") configured in the rule; if the value is > 0, then marking it as an error log, and generating an operation name according to "keyword + value character + error code value"; otherwise, generating an operation name in the same format but not marking it as an error log. For example, the log line "GetLAGPort() Failed : NeId = 134219456 Ret = 129003" matches the keyword "Ret", extracts the error code value "129003" (> 0), generates the operation name "Ret=129003", and marks it as an error log. Optionally, in this embodiment, the matching logic of the above-mentioned error keyword rule may include, but is not limited to: determining whether the core content of the log contains the keyword of the current error keyword rule; if it does, it is directly marked as an error log, and the entire core content of the log line is used as the operation name. For example, the log line "error:GetLAGPort() failed" matches the keyword "error", generates the operation name "error:GetLAGPort() failed", and is marked as an error log.

[0033] Optionally, in this embodiment, the matching logic of the special case operation rule may include, but is not limited to: determining whether the core content of the log contains the keyword of the current special case operation rule; if it does, it is marked as a normal operation log, the keyword is used as the operation name, and other log content is ignored. For example, the log line "Notify SID = 1 EventType =Object State Update(5)" matches the keyword "Notify", generates the operation name "Notify", and is marked as a normal operation log.

[0034] Optionally, in this embodiment, the matching logic for the aforementioned key operation rule may include, but is not limited to: determining whether the core content of the log contains the keyword of the current key operation rule; if so, marking it as a normal operation log, and extracting the log line from the keyword to the end as the operation name. For example, the log line "xxx.x.xx.x (IP address) ZHANGSANLEAVE MSMPMain::CNetworkCross::GetMutiObj" matches the keyword "MSMPMain::", generates the operation name "MSMPMain::CNetworkCross::GetMutiObj", and marks it as a normal operation log.

[0035] Optionally, in the embodiments of this application, the matching logic of the above-mentioned ordinary operation rules may include, but is not limited to: determining whether the core content of the log contains the marker according to the order of the operation name ending markers configured in the rules (such as "", "[", "("). If it does, the log line is truncated from the beginning to the character before the marker as the operation name and marked as a normal operation log. For example, the log line "sn:115990007<==>VC[circuit_id:504497784" matches the marker "":", generating the operation name "sn" and marking it as a normal operation log.

[0036] Optionally, in this embodiment, the above-mentioned rule supplementary prompts may include, but are not limited to: counting the number of logs that do not match any rules, and when the number exceeds a preset threshold (such as 5% of the total number of log lines), outputting a prompt message suggesting that the user add corresponding rules in the XML keyword rules to improve the accuracy and efficiency of log analysis. For example, if the number of logs that do not match any rules reaches 8,000 and the total number of log lines is 100,000, exceeding the 5% threshold, the prompt "There are currently 8,000 logs that do not match any rules, and it is recommended to add corresponding rules in the XML keyword rules" will be output. For example, the above process of matching keyword rules according to default priority can be understood as follows: For a log line "Board is not found in ne,board id=134221734:Ne Id = 134221734", the error code rule is first matched, and no relevant keyword is found; then the error keyword rule is matched, and no preset error keyword is found; next, the special operation rule is matched, and no match is found; then the key operation rule is matched, and no match is found; finally, the normal operation rule is matched, which includes the comma, and the operation name "Board is not found in ne" is generated, and it is marked as a normal operation log. Through the embodiments of this application, various rules are matched sequentially according to priority, ensuring the logic and accuracy of log classification. The targeted matching logic of different rules enables the accurate extraction of operation names, and the rule supplementary prompts help users continuously optimize rule configurations and improve the comprehensiveness and reliability of log analysis.

[0037] As an optional approach, the operation name and error attribute of the current log line are marked according to the matched rule type, including: logs that match error code rules or error keyword rules are uniformly marked as error logs; logs that match special operation rules, key operation rules, or normal operation rules are marked as normal operation logs; each log line belongs to only one rule type, one operation name, and one error tag. Optionally, in this embodiment, the marking of the error logs may include, but is not limited to, explicit marking in the statistical results using specific identifiers (such as "Yes" or "Y"), to facilitate users in quickly filtering fault-related logs and focusing on fault troubleshooting. For example, "Fail:55082" matching the error code rule and "error: GetLAGPort() failed" matching the error keyword rule are both marked as "Yes" in the "Error Identifier" column of the statistical table. Optionally, in this embodiment, the marking of the normal operation log may include, but is not limited to, explicit marking in the statistical results using specific identifiers (such as "No" or "N") to distinguish between normal business operations and fault-related operations, helping users to fully understand the system's operating status. For example, "Notify" matching special operation rules and "MSMPMain::CNetworkCross::GetMutiObj" matching key operation rules are both marked as "No" in the "Error Identifier" column of the statistical table. For example, the above log attribute marking logic can be understood as follows: after each log line is matched by priority, it belongs to only one rule type, generating a unique operation name and an error flag. There is no cross-rule classification or multiple operation names or multiple error flags. For example, after a log line matches an error code rule, it will not be matched by other rules, and only an error code-related operation name and a "yes" error flag will be generated.

[0038] Through the embodiments of this application, the clear log attribute marking rules ensure the clarity and uniqueness of the analysis results, help users quickly distinguish between error logs and normal operation logs, and improve the efficiency of log fault diagnosis and status analysis. As an optional approach, the above statistics of all logs also include: maintaining a time point-operation name count dictionary, incrementing the count by 1 for each matched operation name; parsing the timestamps in the log lines, maintaining the time point-operation count dictionary, and counting the number of all operation calls at the same time point; the final output includes: operation name, number of calls, percentage, error flag; as well as time point, number of operations, and percentage of time dimension.

[0039] Optionally, in this embodiment, the operation name counting dictionary may include, but is not limited to, a key-value pair structure with operation name as the key and statistical information (number of calls, error flag) as the value. When traversing all log lines, for each generated operation name, if the operation name already exists in the dictionary, the call count is incremented by 1; if it does not exist, a new key-value pair is added and the call count is initialized to 1, while the error flag is recorded. For example, when the "Notify" operation is matched for the first time, "Notify" is added to the dictionary: call count: 1, error flag: no}. When matched again, the call count is updated to 2.

[0040] Optionally, in this embodiment, the aforementioned time point-operation count dictionary may include, but is not limited to, a key-value pair structure with time points accurate to the second as keys and operation counts as values. The timestamp of each log line is parsed and converted into a standard time point. If the time point already exists in the dictionary, the operation count is incremented by 1; otherwise, a new key-value pair is created and the operation count is initialized to 1. For example, the log timestamp "2019-12-16T15:23:30.527422" is converted into the time point "2019-12-16T15:23:30". If the time point already exists in the dictionary, the operation count is incremented by 1; otherwise, the operation count is initialized to 1. Optionally, in this embodiment, the above-mentioned percentage calculation may include, but is not limited to, the percentage of the number of times the operation name is called out to the total number of log lines. The calculation formula is (number of times the operation name is called / total number of log lines) × 100%, rounded to two decimal places. Similarly, the percentage based on the time dimension is (number of operations at a given time point / total number of log lines) × 100%, rounded to two decimal places. For example, if the total number of log lines is 3600, and the "Notify" operation is called 850 times, the percentage is (850 / 3600) × 100% ≈ 23.61%; the number of operations at the time point "2019-12-16T15:23:30" is 156, and the percentage based on the time dimension is (156 / 3600) × 100% ≈ 4.33%. For example, the above global statistical process can be understood as follows: traversing 3600 lines of logs, maintaining an operation name count dictionary and a time point-count dictionary, and finally obtaining the call count, percentage and error flag of 236 operation names, the operation count and time dimension percentage of 1286 time points, forming a complete statistical result set. Through the embodiments of this application, the dictionary-based statistical method achieves efficient operation frequency and time distribution statistics. The standardized output format facilitates users to quickly obtain key information, providing data support for system operation and maintenance, fault diagnosis and resource optimization.

[0041] As an optional approach, the method further includes: if the number of logs whose entire log line is used as the operation name exceeds a preset threshold, then output an optimization suggestion: it is recommended to add a corresponding rule for this type of log in the XML keyword rules. Optionally, in this embodiment, the aforementioned preset threshold may include, but is not limited to, a percentage threshold (e.g., 0.01% of the total log lines) or a fixed number threshold (e.g., 100 log lines) set by the user according to actual analysis needs, used to determine whether the number of logs without matching rules reaches the level requiring supplementary rules. For example, if the percentage threshold is set to 0.01% and the total number of log lines is 100,000, and the fixed number threshold is 100, an optimization prompt is triggered when the number of logs without matching rules exceeds 100. Optionally, in this embodiment, the aforementioned optimization prompts may include, but are not limited to, pop-up prompts that appear synchronously when outputting statistical results, prompts written to log files, or notes added to statistical tables, clearly informing users of the number of logs that do not match the rules and suggestions for supplementing the rules. For example, a pop-up prompt may state, "Currently, there are 200 logs that do not match any rules (accounting for 0.02%). It is recommended to add corresponding rules in the XML keyword rules to improve the accuracy and readability of the analysis."

[0042] For example, the triggering and output of the above rule optimization prompt can be understood as follows: the number of logs that do not match any rule is 600, the total number of log lines is 100,000, accounting for 0.6%, which exceeds the preset threshold of 0.01%. While outputting an Excel statistical table, the system pops up an optimization prompt window to guide the user to analyze the common characteristics of this type of log and add corresponding keyword rules to the XML file. Through the embodiments of this application, the rule optimization prompt mechanism helps users continuously improve keyword rule configuration, reduce the number of logs with unmatched rules, improve the comprehensiveness and accuracy of log analysis, and make the log analysis method more adaptable and practical. The log analysis method in the embodiments of this application will be explained below with reference to optional examples.

[0043] Example scenario: Taking 1 million lines of logs generated by the network management system service as an example, the XML keyword rule configuration includes 2 error code rules (keywords "Fail" "ErrorCode"), 2 error keyword rules (keywords "error" "failed"), 2 special operation rules (keywords "Notify" "EndTransaction"), 1 key operation rule (keyword "MSMPMain::"), and 1 normal operation rule (marked with "", ""[""(").

[0044] Log preprocessing: Read a log line "2019-12-16T15:23:30.527422>[4] [0][0x0083e780] Fail Auto PV_FinishCircuit: 55082", extract the time point "2019-12-16T15:23:30", and extract the core content "Fail Auto PV_FinishCircuit: 55082". Priority matching: Prioritize matching error code rules, if the keyword "Fail" is matched, extract the error code value "55082" (>0), generate the operation name "Fail:55082", and mark it as an error log.

[0045] Global statistics: Increment the number of calls for "Fail:55082" in the operation name count dictionary by 1, and increment the number of operations for "2019-12-16T15:23:30" in the time point count dictionary by 1; after traversal, calculate the proportion of this operation name as 0.012% and the proportion of the time point as 0.015%.

[0046] Output: Generate an Excel spreadsheet containing the operation name and corresponding statistics. The number of logs that do not match the rule is 30,000 (3%), which does not exceed the 5% threshold. No rule optimization prompts are output. Through the embodiments of this application, the analysis of 1 million lines of logs can be completed in 3.5 minutes, and standardized statistical results can be output. Compared with manual analysis, the efficiency is improved by about 5,000-10,000 times, which effectively supports the work of troubleshooting and system status analysis. It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. According to another aspect of the embodiments of this application, a log analysis apparatus is also provided, which can be used to implement the log analysis method provided in the above embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. Figure 4 This is a structural block diagram of an optional log analysis device according to an embodiment of this application, such as... Figure 4 As shown, the log analysis device includes: Analysis unit 402 is used to read the log file to be analyzed line by line, and for each line of log text, it sequentially matches keyword rules, including: error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules; Tagging unit 404: The user tags the operation name and error attribute of the current log line according to the matched keyword rule type; The statistics unit 406 is used to count the total number of calls for the operation name of each log line in all log lines and the number of operation calls at each time point, and output the statistical results.

[0047] As an optional approach, the analysis unit 402 reads the XML file and extracts configuration information such as keywords and operation names for each rule before analysis. The XML file arranges the rules in the order of "error code rules → error keyword rules → special operation rules → key operation rules → ordinary operation rules," forming an ordered rule list. For example, parsing the `...` in the XML file... <lineobject>The tag is used to extract the keyword "Fail" and the corresponding operation name rule, which is then categorized as an error code rule and placed at the top of the rule list. As an optional solution, the aforementioned log line-by-line matching module 404 may include, but is not limited to, a log preprocessing unit, a priority execution unit, and a unique matching unit. The log preprocessing unit is used to extract the timestamp and the core content of the log. The priority execution unit is used to match the logs sequentially according to the rule list. The unique matching unit is used to ensure that the subsequent matching of the current line is terminated immediately after a successful match. For example, the preprocessing unit extracts the core content of the log "Notify SID = 1 EventType = Object State Update", the priority execution unit matches the special operation rule in sequence, and the unique matching unit terminates the subsequent matching process. As an optional approach, the marking unit 406 is used to generate a corresponding operation name based on the type of the matching rule, and only marks logs that match the error code rule (error code value > 0) and the error keyword rule as error logs. For example, the marking unit 406 generates "ErrorCode=54033" for error code rule matching and marks the operation name as an error log.

[0048] As an optional approach, the aforementioned statistics unit 406 is used to maintain an operation name count dictionary and update the call count, maintain a time point-operation count dictionary and update the operation count, and calculate the proportion of each operation name and the proportion of each time point in the time dimension. For example, updating the call count of "Notify" to 850 times calculates its proportion of the total log lines to 22.97%. Statistics unit 406 integrates statistical data to form a standardized result set, writes the result set to an Excel spreadsheet and outputs it, and outputs optimization prompts when the number of logs with unmatched rules exceeds a threshold. For example, statistics unit 406 generates an Excel spreadsheet containing operation names, call counts, proportions, and error identifiers, and pops up an optimization prompt window when the proportion of logs with unmatched rules exceeds the threshold. As an optional approach, the analysis unit 402 strictly follows the XML order: error code, error keyword, special operation, key operation, and normal operation. Optionally, in this embodiment, the above priority execution logic may include, but is not limited to, implementing it by looping through the rule list, matching each rule sequentially from index 0 until a match is successful or all rules have been traversed, ensuring strict adherence to the rule order configured in the XML file. For example, if rule list index 0 is the error code rule, index 1 is the error keyword rule, and index 2 is the special operation rule, matching is performed in the order of index 0→1→2. For example, the above priority execution logic can be understood as follows: For example, for a log line "MSMPMain::CNetworkCross::GetMutiObj", the error code rule is matched first (missed), then the error keyword rule is matched (missed), then the special operation rule is matched (missed), and then the key operation rule is matched (hit), terminating the subsequent matching and generating the corresponding operation name. Through the embodiments of this application, the strict priority execution logic of the analysis unit 402 ensures the consistency and accuracy of log classification and avoids deviations in analysis results caused by disordered rule order.

[0049] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination. According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0050] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks. According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor. Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0051] According to another aspect of the embodiments of this application, a computer program product is also provided, which includes a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0052] The following reference Figure 5 To describe an electronic device 500 according to this embodiment of the present invention. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0053] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including storage unit 520 and processing unit 510).

[0054] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Embodiment Methods" section of this specification according to various exemplary embodiments of the present invention.

[0055] Storage unit 520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0056] Storage unit 520 may also include a program / utility 524 having a set (at least one) program module 525, such program module 525 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0057] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0058] Electronic device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 500, and / or any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0059] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0060] In particular, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.

[0061] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software. The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

[0062] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / lineobject> < / stringstart> < / functionend> < / stringstart> < / lineobject> < / lineobject>

Claims

1. A method for log analysis, characterized in that, include: The log file to be analyzed is read line by line. For each line of log text, keyword rules are matched sequentially. The keyword rules include: error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules. Based on the keyword rule type that is matched, mark the operation name and error attribute of the current log line; Count the total number of calls for the operation name in each log line and the number of calls for the operation at each time point, and output the statistical results.

2. The method according to claim 1, characterized in that, The keyword rules are matched sequentially according to a preset priority rule. Each log line is matched only with the rule of the highest priority. Once a match is successful, subsequent rules are not matched. Specifically, this includes: Prioritize matching the error code rules: if the log contains the specified error code keyword and the error code value is greater than zero, then mark it as an error log and generate the corresponding operation name; If the error code rule is not matched, the error keyword rule will continue to be matched: if the log contains a preset error keyword, it will be directly marked as an error log and a corresponding operation name will be generated; If the aforementioned error keyword rule is not matched, the special operation rule will continue to be matched: if a match is found, it will be uniformly classified as the specified special operation name; If the special operation rule is not matched, the key operation rule will continue to be matched: if a match is found, a precise key operation name will be generated; If the key operation rule is not matched, then the ordinary operation rule is matched: the operation name is generated by truncating the symbols of the ordinary operation rule; If none of the five rules are matched, the entire log text will be used as the operation name, and a rule supplement prompt will be triggered.

3. The method according to claim 1, characterized in that, The step of marking the operation name and error attribute of the current log line according to the matched keyword rule type includes: Logs that match error code rules or error keyword rules are uniformly marked as error logs; Logs that match special operation rules, key operation rules, or ordinary operation rules are marked as normal operation logs.

4. The method according to claim 1, characterized in that, The statistics include the total number of calls for the operation name in each log line and the number of calls for each time point, including: Iterate through all operation name rules in the keyword rules and analyze the operation names in each line of the log. Based on the operation name counting dictionary, the corresponding count is incremented by 1 for each matched operation name; Parse the timestamps in the log lines, maintain a time-operation count dictionary, and count the number of all operation calls at the same time point; Output the operation name, number of calls, percentage, and error flag; as well as the time point, number of operations, and percentage of time dimension.

5. A log analysis apparatus, characterized in that, include: The analysis unit is used to read the log file to be analyzed line by line. For each line of log text, it matches keyword rules in sequence. The keyword rules include: error code rules, error keyword rules, special operation rules, key operation rules, and ordinary operation rules. The tagging unit allows the user to tag the operation name and error attribute of the current log line based on the matched keyword rule type. The statistics unit is used to count the total number of calls for the operation name of each log line in all log lines and the number of operation calls at each time point, and output the statistical results.

6. The apparatus according to claim 5, characterized in that, The analysis unit matches the keyword rules sequentially according to a preset priority rule. Each log line is matched only with the rule of the highest priority. Once a match is successful, subsequent rules are not matched. Specifically, this includes: Prioritize matching the error code rules: if the log contains the specified error code keyword and the error code value is greater than zero, then mark it as an error log and generate the corresponding operation name; If the error code rule is not matched, the error keyword rule will continue to be matched: if the log contains a preset error keyword, it will be directly marked as an error log and a corresponding operation name will be generated; If the aforementioned error keyword rule is not matched, the special operation rule will continue to be matched: if a match is found, it will be uniformly classified as the specified special operation name; If the special operation rule is not matched, the key operation rule will continue to be matched: if a match is found, a precise key operation name will be generated; If the key operation rule is not matched, then the ordinary operation rule is matched: the operation name is generated by truncating the symbols of the ordinary operation rule; If none of the five rules are matched, the entire log text will be used as the operation name, and a rule supplement prompt will be triggered.

7. The apparatus according to claim 5, characterized in that, The marking unit is used to mark the operation name and error attribute of the current log line according to the matched keyword rule type, specifically: Logs that match error code rules or error keyword rules will be uniformly marked as error logs; Logs that match special operation rules, key operation rules, or ordinary operation rules will be marked as normal operation logs.

8. The apparatus according to claim 5, characterized in that, The statistics unit is used to count the total number of calls for the operation name in each log line and the number of calls for the operation at each time point in all log lines. Specifically: Iterate through all operation name rules in the keyword rules and analyze the operation name in each log line; based on the operation name count dictionary, increment the count by 1 for each matched operation name; parse the timestamps in the log lines, maintain a mapping table between time points and operation count dictionaries, and count the number of all operation calls at the same time point; Output the operation name, number of calls, percentage, error flag, time point, number of operations, and percentage of time dimension.

9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the log analysis method according to any one of claims 1-4.

10. An electronic device, comprising a memory and a processor, characterized in that, When the processor executes the program in the memory, it implements the log analysis method according to any one of claims 1-4.

Images