Log analysis method, device and equipment and storage medium

Abstract

The application discloses a log analysis method, device and equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: determining a log type corresponding to a to-be-analyzed alarm log based on a log analysis requirement; determining an alarm log analysis template corresponding to the to-be-analyzed alarm log according to the log type; determining a to-be-updated operation of the alarm log analysis template based on the log analysis requirement; generating an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template; and analyzing the to-be-analyzed alarm log through the alarm log analysis model to obtain an alarm log analysis result. The alarm log analysis model is generated based on the to-be-updated operation in the alarm log analysis template and the alarm log analysis template, so that the alarm log analysis is performed, and the technical problem that the rule-based alarm log analysis method in the prior art needs to constantly manually update and maintain the analysis rule, resulting in high labor and time costs, is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a log analysis method and device, equipment and a storage medium. BACKGROUND

[0002] In modern information technology systems, log data is an important record of system operation status, especially for alarm logs, which contain key information such as system failure, abnormal behavior, and security events. Therefore, accurately and efficiently analyzing alarm logs is crucial for system monitoring, troubleshooting, security analysis, and other tasks. However, with the rapid development of information technology, the scale and complexity of log data are constantly increasing, and therefore traditional log analysis methods also face many challenges.

[0003] Currently, traditional log analysis methods mainly rely on manually written rules for log analysis. These rules are stored in a rule library, and when new alarm logs arrive, the system takes rules from the rule library one by one and compares them with the logs. If a match is found, the rule is used for analysis. However, due to the continuous updating of alarm log formats and content, the rules in the rule library also need to be manually updated and maintained, resulting in a large amount of manpower and time. SUMMARY

[0004] The main purpose of the present application is to provide a log analysis method, device, equipment and storage medium, which aims to solve the technical problem that the alarm log analysis method based on rules in the prior art needs to be constantly manually updated and maintained, resulting in high labor and time costs.

[0005] To achieve the above purpose, the present application provides a log analysis method, which comprises:

[0006] Obtaining log analysis requirements of a to-be-analyzed alarm log, and determining a log type corresponding to the to-be-analyzed alarm log based on the log analysis requirements;

[0007] Determining an alarm log analysis template corresponding to the to-be-analyzed alarm log according to the log type;

[0008] Determining a to-be-updated operation of the alarm log analysis template based on the log analysis requirements;

[0009] Generating an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template;

[0010] Analyzing the to-be-analyzed alarm log through the alarm log analysis model to obtain an alarm log analysis result.

[0011] In an embodiment, the step of determining the log type corresponding to the to-be-analyzed alarm log based on the log analysis requirements comprises:

[0012] standardize the to-be-parsed alarm log to obtain a standardized to-be-parsed alarm log;

[0013] extract features from the standardized to-be-parsed alarm log based on the log parsing requirement through a preset log feature extraction model to obtain alarm log feature information;

[0014] determine a text mode in the to-be-parsed alarm log according to the alarm log feature information;

[0015] generate a regular expression corresponding to the to-be-parsed alarm log according to the text mode;

[0016] find a log type corresponding to the to-be-parsed alarm log from a preset regular expression library based on the regular expression.

[0017] In an embodiment, the step of determining the to-be-updated operation of the alarm log parsing template based on the log parsing requirement comprises:

[0018] perform a tentative parsing on the to-be-parsed alarm log through the alarm log parsing template to obtain a log tentative parsing result;

[0019] determine whether the log tentative parsing result meets the log parsing requirement;

[0020] If not, determine the to-be-updated operation of the alarm log parsing template based on the log tentative parsing result and the log parsing requirement.

[0021] In an embodiment, the step of determining the to-be-updated operation of the alarm log parsing template based on the log tentative parsing result and the log parsing requirement comprises:

[0022] determine a log filtering condition in the log tentative parsing result and a target log filtering condition in the log parsing requirement;

[0023] determine whether the log filtering condition and the target log filtering condition are consistent;

[0024] If not consistent, determine the to-be-updated operation of the alarm log parsing template as a log filtering condition updating operation.

[0025] In an embodiment, the step of generating an alarm log parsing model based on the to-be-updated operation and the alarm log parsing template comprises:

[0026] determine a target parsing strategy of the alarm log parsing template based on the to-be-updated operation and the log parsing requirement;

[0027] The alarm log parsing template is updated according to the target parsing strategy to obtain the updated alarm log parsing template;

[0028] An alarm log parsing model is generated based on the updated alarm log parsing template.

[0029] In one embodiment, the step of determining the target parsing strategy for the alarm log parsing template based on the operation to be updated and the log parsing requirements includes:

[0030] If the operation to be updated is the log filtering condition update operation, then the target log filtering condition is extracted from the log parsing requirements;

[0031] Determine the new log filtering conditions to be added in the target log filtering conditions;

[0032] The target parsing strategy for generating the alarm log parsing template based on the new log filtering conditions and the existing log filtering conditions.

[0033] In one embodiment, the step of generating an alarm log parsing model based on the updated alarm log parsing template includes:

[0034] Obtain the log parsing category and log parsing quantity from the updated alarm log parsing template;

[0035] Determine the log execution logic corresponding to the log parsing category;

[0036] An alarm log parsing model is generated based on the log parsing category, the number of log parsing operations, and the log execution logic.

[0037] Furthermore, to achieve the above objectives, this application also proposes a log parsing apparatus, the apparatus comprising:

[0038] The requirements analysis module is used to obtain the log parsing requirements of the alarm logs to be parsed, and to determine the log type corresponding to the alarm logs to be parsed based on the log parsing requirements.

[0039] The template determination module is used to determine the alarm log parsing template corresponding to the alarm log to be parsed based on the log type.

[0040] The template update module is used to determine the update operation to be performed on the alarm log parsing template based on the log parsing requirements.

[0041] The model generation module is used to generate an alarm log parsing model based on the operation to be updated and the alarm log parsing template;

[0042] The log analysis module is configured to analyze the to-be-analyzed alarm log based on the alarm log analysis model to obtain an alarm log analysis result.

[0043] In addition, to achieve the above-mentioned object, the present application further provides a log analysis device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the log analysis method as described above.

[0044] In addition, to achieve the above-mentioned object, the present application further provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the log analysis method as described above.

[0045] The present application provides a log analysis method, which discloses obtaining log analysis requirements of a to-be-analyzed alarm log, determining a log type corresponding to the to-be-analyzed alarm log based on the log analysis requirements, determining an alarm log analysis template corresponding to the to-be-analyzed alarm log according to the log type, determining a to-be-updated operation of the alarm log analysis template based on the log analysis requirements, generating an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template, and analyzing the to-be-analyzed alarm log based on the alarm log analysis model to obtain an alarm log analysis result. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0048] Figure 1 A flowchart provided by the log analysis method embodiment one of the present application;

[0049] Figure 2 A flowchart provided for the log analysis method embodiment two of the present application;

[0050] Figure 3 A flowchart provided for the log analysis method embodiment three of the present application;

[0051] Figure 4 A module structure diagram of the log analysis device of the present application embodiment;

[0052] Figure 5 A device structure diagram of the hardware running environment involved in the log analysis method of the present application embodiment.

[0053] The object implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0055] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The main solution of the present application embodiment is: obtaining log analysis requirements of the to-be-analyzed alarm log, and determining the log type corresponding to the to-be-analyzed alarm log based on the log analysis requirements; determining the alarm log analysis template corresponding to the to-be-analyzed alarm log according to the log type; determining the to-be-updated operation of the alarm log analysis template based on the log analysis requirements; generating an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template; and analyzing the to-be-analyzed alarm log through the alarm log analysis model to obtain an alarm log analysis result.

[0057] Because the prior art uses the analysis rules in the rule library to analyze the alarm log, it is necessary to constantly manually update and maintain the rules, resulting in a large amount of manpower and time consumption.

[0058] The present application provides a solution, which can determine the to-be-updated operation in the log analysis template corresponding to the alarm log based on the analysis requirements of the alarm log, and generate an alarm log analysis model based on the to-be-updated operation and the log analysis template to analyze the alarm log, thereby solving the technical problem that the rule-based alarm log analysis method in the prior art needs to constantly manually update and maintain the analysis rules, resulting in high labor and time cost, and thereby improving the alarm log analysis efficiency.

[0059] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, or an electronic device capable of realizing the above functions, a log analysis device, and a log analysis system containing the log analysis device. The log analysis system is taken as an example (hereinafter referred to as the system) to illustrate the embodiment and the following embodiments.

[0060] Based on this, the log analysis method provided in the embodiment of the present application is provided with reference to Figure 1 , Figure 1 The flowchart of the log analysis method provided in the first embodiment of the present application is shown in the figure.

[0061] In the embodiment, the log analysis method includes steps S10-S50:

[0062] Step S10: Obtain the log analysis requirement of the alarm log to be analyzed, and determine the log type corresponding to the alarm log to be analyzed based on the log analysis requirement.

[0063] It can be understood that the alarm log to be analyzed can be original alarm log data generated during system operation and requiring analysis processing. These logs usually contain important information such as system operation status, error information, and user behavior, and their corresponding categories can include but are not limited to Apache logs, system logs, and custom logs.

[0064] It should be understood that the log analysis requirement can be the specific requirements and goals when analyzing the alarm log to be analyzed, such as the fields to be extracted (such as timestamp, log level, IP address, error code, etc.), the format of the fields (such as the format of the timestamp is YYYY-MM-DD HH:MM:SS or other formats), the naming of the fields (such as the timestamp field is named timestamp, and the IP address field is named ip_address), the filtering conditions (such as only extracting logs of a specific log level (such as ERROR)), and other special processing requirements (such as additional processing of certain fields, such as time zone conversion of the timestamp), etc. In the embodiment, the system can receive the analysis requirement of the alarm log input by the user through an interface or a configuration file, for example, the user can specify the fields to be extracted and their formats when analyzing the alarm log through a configuration file.

[0065] It should be noted that the above log type can be a set of attributes such as the source, format, system or service to which the to-be-parsed alarm log belongs, and in actual application, different log types usually have different structures and contents, for example, Apache logs are used to record detailed information of HTTP requests; system logs are used to record system events and error information; custom logs are logs of a specific format generated by an application.

[0066] In this embodiment, after obtaining the log parsing requirement of the to-be-parsed alarm log, the system can analyze the log parsing requirement to extract key information in the to-be-parsed alarm log, and determine the log type of the to-be-parsed alarm log through the key information, for example, by extracting keywords such as error codes and service names in the to-be-parsed alarm log to determine the log type of the to-be-parsed alarm log, or by the log header information of the to-be-parsed alarm log to determine the log type of the to-be-parsed alarm log, etc. This embodiment does not limit this.

[0067] Further, the step S10 includes: performing standardization processing on the to-be-parsed alarm log to obtain a standardized to-be-parsed alarm log; performing feature extraction on the standardized to-be-parsed alarm log based on the log parsing requirement through a preset log feature extraction model to obtain alarm log feature information; determining a text mode in the to-be-parsed alarm log according to the alarm log feature information; generating a regular expression corresponding to the to-be-parsed alarm log according to the text mode; and searching for a log type corresponding to the to-be-parsed alarm log from a preset regular expression library based on the regular expression.

[0068] It should be noted that the standardization processing on the to-be-parsed alarm log can be a process of converting the original to-be-parsed alarm log into an alarm log with uniform format. After the standardization processing on the to-be-parsed alarm log, a standardized to-be-parsed alarm log can be obtained, which can include a uniform timestamp format (such as YYYY-MM-DD HH:MM:SS), uniform field naming, uniform data type, etc. In actual application, the system can first perform data cleaning on the to-be-parsed alarm log to remove invalid characters, duplicate records and irrelevant information in the to-be-parsed alarm log, to obtain a cleaned alarm log, and then perform normalization processing on the cleaned alarm log to convert the cleaned alarm log into a uniform format, and finally obtain the standardized to-be-parsed alarm log.

[0069] It should be noted that the above preset log feature extraction model can be a model for extracting key features from the alarm log, which can be constructed based on machine learning or deep learning algorithms, and can identify important information in the alarm log, such as convolutional neural network (CNN), recurrent neural network (RNN), etc., which is not limited in the embodiment. Correspondingly, the above alarm log feature information can be key information extracted from the standardized to-be-analyzed alarm log, for example, timestamp, log level (such as ERROR, INFO, WARN, etc.), IP address of the generated log, error information, etc. In actual application, the system can standardize the to-be-analyzed alarm log, process it into a format suitable for the input of the preset log feature extraction model, and then input the standardized to-be-analyzed alarm log into the preset log feature extraction model. At this time, the preset log feature extraction model can extract features from the standardized to-be-analyzed alarm log, and select key features from all extracted features according to the log analysis requirements, and finally obtain the alarm log feature information.

[0070] It should be understood that the above text pattern can be a rule for matching and extracting specific format text in the to-be-analyzed alarm log, such as a regular expression, which is a powerful text matching tool and can define complex text patterns. In log analysis, the embodiment can identify and extract key fields in the to-be-analyzed alarm log through regular expressions. For example, for the timestamp of the to-be-analyzed alarm log, its format can be YYYY-MM-DD HH:MM:SS, and the corresponding regular expression can be r"(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})"; for the log level, its format can be [ERROR], [INFO], [WARN], and the corresponding regular expression can be r"\[(\w+)\]". In the embodiment, the system can analyze the alarm log feature information extracted from the to-be-analyzed alarm log, determine the fields and their formats that need to be matched, and define the corresponding text pattern according to these feature information, and then generate the corresponding regular expression.

[0071] It should be noted that the above preset regular expression library can be a database for storing various log types and their corresponding regular expressions. The library usually exists in the form of a dictionary or a database table. Each regular expression in the library is used to match a log of a specific format, thereby determining the type of the log. Each log type has a unique identifier and a corresponding regular expression. In this embodiment, the system can first define a preset regular expression library containing various log types and their corresponding regular expressions. After determining the regular expression corresponding to the alarm log to be parsed, the system can match the regular expression with the regular expressions stored in the preset regular expression library one by one to determine the log type corresponding to the alarm log to be parsed according to the regular expression that matches successfully.

[0072] Step S20: determining an alarm log parsing template corresponding to the alarm log to be parsed according to the log type.

[0073] It should be noted that the above alarm log parsing template can be a template for guiding how to extract key information from a specific type of alarm log. The template can include field extraction rules, field formats, field naming, filtering conditions, etc. The field extraction rules are used to define how to extract key fields from the alarm log. The field format is used to define the format of each field. The field naming is used to define the name of each field. The filtering condition is used to define the alarm log to be parsed.

[0074] In actual application, after determining the log type corresponding to the alarm log to be parsed, the system can select the parsing template corresponding to the alarm log to be parsed from the predefined parsing template library according to the log type. The predefined parsing template library can be a database storing various log types and their corresponding parsing templates. Each parsing template in the database defines how to parse a specific type of alarm log.

[0075] Step S30: determining a to-be-updated operation of the alarm log parsing template based on the log parsing requirement.

[0076] It should be noted that the above to-be-updated operation can be an operation of modifying, supplementing or adjusting the template in order to meet the parsing requirement of the to-be-updated operation, for example, adding a new field extraction rule, adding a new field extraction rule, deleting a field, updating a filtering condition, etc. The present embodiment does not limit this.

[0077] In this embodiment, the system can compare the log parsing requirement of the alarm log to be parsed with the alarm log parsing template, and determine the to-be-updated operation in the alarm log parsing template according to the comparison result. In addition, the present embodiment can first parse the alarm log to be parsed through the alarm log parsing template, and compare the parsing result with the log parsing requirement, and finally determine the to-be-updated operation in the alarm log parsing template.

[0078] Step S40: generating an alarm log parsing model based on the to-be-updated operation and the alarm log parsing template.

[0079] It can be understood that the above-mentioned alarm log parsing model can be an executable degree or script for extracting key information from the alarm log. In this embodiment, the system can update the alarm log parsing template through the to-be-updated operation, and then generate the alarm log parsing model based on the updated alarm log parsing template.

[0080] Step S50: parsing the to-be-parsed alarm log through the alarm log parsing model to obtain an alarm log parsing result.

[0081] It should be understood that the above-mentioned alarm log parsing result can be the key information extracted from the to-be-parsed alarm log, including timestamp, log level, IP address, error information, etc. These parsing results can be output in dictionary or JSON format for subsequent analysis and processing. In actual application, the system can directly input the to-be-parsed alarm log into the alarm log parsing model, so that the alarm log parsing model can parse the to-be-parsed alarm log according to the log parsing rules defined therein to obtain the alarm log parsing result.

[0082] The embodiment provides a log parsing method. The method discloses obtaining a log parsing requirement of a to-be-parsed alarm log, and determining a log type corresponding to the to-be-parsed alarm log based on the log parsing requirement; determining an alarm log parsing template corresponding to the to-be-parsed alarm log according to the log type; determining a to-be-updated operation of the alarm log parsing template based on the log parsing requirement; generating an alarm log parsing model based on the to-be-updated operation and the alarm log parsing template; parsing the to-be-parsed alarm log through the alarm log parsing model to obtain an alarm log parsing result; compared with the prior art, when the alarm log is parsed using the parsing rules in the rule library, the rules need to be manually updated and maintained constantly, resulting in a large amount of manpower and time. Since the embodiment can determine the to-be-updated operation in the log parsing template corresponding to the alarm log based on the parsing requirement of the alarm log, and generate the alarm log parsing model based on the to-be-updated operation and the log parsing template to parse the alarm log, the technical problem that the log parsing method based on rules in the prior art needs to manually update and maintain the parsing rules constantly, resulting in high cost of manpower and time, is solved, and the alarm log parsing efficiency is improved.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 2 , Figure 2A flowchart is provided for the second log analysis method embodiment of the present application.

[0084] In this embodiment, step S30 includes steps S301-S303:

[0085] Step S301: The tentative analysis of the to-be-analyzed alarm log is performed by using the alarm log analysis template to obtain a log tentative analysis result.

[0086] It should be noted that the tentative analysis of the to-be-analyzed alarm log can be a process of analyzing the to-be-analyzed alarm log by using the alarm log analysis template without determining whether the log format in the alarm log analysis template meets the log analysis requirement. Correspondingly, the above log tentative analysis result can be preliminary analysis data obtained by the tentative analysis process, which can include successfully extracted fields (such as timestamp, log level, IP address, error information, etc.), fields that cannot be successfully extracted and reasons (such as regular expression mismatch or field missing), analysis errors or warnings, etc. This embodiment does not limit this.

[0087] In this embodiment, the tentative analysis of the to-be-analyzed alarm log by using the alarm log analysis template can verify the effectiveness of the alarm log analysis template, identify the deficiencies in the alarm log analysis template or the changes in the log format, and further dynamically adjust the alarm log analysis template according to the tentative analysis result, so that the alarm log analysis template dynamically adapts to new log formats, thereby facilitating to improve the accuracy of log analysis.

[0088] Step S302: It is determined whether the log tentative analysis result meets the log analysis requirement.

[0089] In actual application, the system can first determine the fields and formats thereof to be extracted and log analysis rules such as filtering conditions according to the log analysis requirement of the to-be-analyzed alarm log, and then detect whether the extracted information in the log tentative analysis result meets these analysis rules, for example, check whether the required fields are all analyzed, whether the format of the analyzed fields meets the requirement, whether the field values meet the expectation, etc. If yes, it is determined that the log tentative analysis result meets the log analysis requirement.

[0090] Step S303: If no, the to-be-updated operation of the alarm log analysis template is determined based on the log tentative analysis result and the log analysis requirement.

[0091] In this embodiment, if the log heuristic analysis result does not meet the log analysis requirement, the system can analyze the heuristic analysis result to determine which fields in the to-be-analyzed alarm log are not successfully extracted, which fields have incorrect formats, and whether there are other problems, and then determine the update operation to be performed on the analysis template according to the analysis result. For example, if some fields are not successfully extracted, the to-be-updated operation can be determined as a field extraction rule addition operation to add new field extraction rules to the alarm log analysis template, so that the subsequent alarm log analysis template can perform field extraction through the newly added log extraction rules. If some fields are no longer needed in the new analysis requirement, the to-be-updated operation can be determined as a field extraction rule deletion operation to delete the extraction rules of these fields from the alarm log analysis template.

[0092] Specifically, the step S303 includes: determining a log filtering condition in the log heuristic analysis result and a target log filtering condition in the log analysis requirement; judging whether the log filtering condition and the target log filtering condition are consistent; and if not, determining the to-be-updated operation of the alarm log analysis template as a log filtering condition update operation.

[0093] It should be noted that the target log filtering condition, i.e., the specific rule or standard defined in the log analysis requirement for filtering log entries when analyzing alarm logs, is used to determine which log entries need to be analyzed and which can be ignored, thereby improving the efficiency and pertinence of the analysis. In this embodiment, the types of target filtering conditions can include log level filtering, time range filtering, keyword filtering, IP address filtering, and the like.

[0094] In actual application, the system can compare the log filtering condition in the log heuristic analysis result with the target log filtering condition in the log analysis requirement to determine whether they are consistent. If not, it indicates that the filtering condition in the alarm log analysis template needs to be updated, and in this case, the to-be-updated operation of the alarm log analysis template can be determined as a log filtering condition update operation. For example, if the target log filtering condition of the log analysis requirement defines that only log data in the past 24 hours is analyzed, but through analysis of the log heuristic analysis result, it is found that the log data in the log heuristic analysis result is log data in the past 48 hours, then it is determined that the log filtering condition and the target log filtering condition are inconsistent. Or, if the target log filtering condition of the log analysis requirement defines that only ERROR level logs are analyzed and other level logs are ignored, but through analysis of the log heuristic analysis result, it is found that the log data in the log heuristic analysis result not only includes ERROR level logs, but also includes WARN level logs, then it is determined that the log filtering condition and the target log filtering condition are inconsistent.

[0095] In this embodiment, the log exploratory analysis result is obtained by using the alarm log analysis template to perform exploratory analysis on the alarm log to be analyzed; it is determined whether the log exploratory analysis result meets the log analysis requirement; if not, the operation to be updated of the alarm log analysis template is determined based on the log exploratory analysis result and the log analysis requirement; since the operation to be updated of the alarm log analysis template can be determined when the log exploratory analysis result does not meet the log analysis requirement in this embodiment, the alarm log analysis template can be updated according to the operation to be updated in the subsequent process, so that the analysis result can more accurately reflect the log content, thereby improving the reliability and accuracy of the alarm log analysis.

[0096] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiments can be referred to the above description, and the subsequent description will not be repeated. On this basis, please refer to Figure 3 , Figure 3 The flowchart provided by the third log analysis method embodiment of the present application is shown in the figure.

[0097] In this embodiment, step S40 includes steps S401-S403:

[0098] Step S401: determining a target analysis strategy of the alarm log analysis template based on the operation to be updated and the log analysis requirement.

[0099] It should be understood that the target analysis strategy can be a rule and process for analyzing logs that is newly formulated for the alarm log analysis template according to the operation to be updated and the log analysis requirement.

[0100] Specifically, step S401 includes: if the operation to be updated is the log filtering condition update operation, extracting the target log filtering condition from the log analysis requirement; determining the log filtering condition to be added in the target log filtering condition; and generating the target analysis strategy of the alarm log analysis template based on the log filtering condition to be added and the log filtering condition.

[0101] It should be noted that the to-be-added log filtering condition can be a filtering condition that needs to be added to the alarm log parsing template. In this embodiment, if the to-be-updated operation of the alarm log parsing template is a log filtering condition updating operation, the target log filtering condition can be extracted from the log parsing requirement, and then the target log filtering condition is compared with the log filtering condition in the log trial parsing result, and the filtering condition that does not exist in the target log filtering condition is determined as the to-be-added log filtering condition. Finally, the system can generate a target parsing strategy of the alarm log parsing template based on the to-be-added log filtering condition and the log filtering condition, so as to update the alarm log parsing template through the target parsing strategy. In addition, the filtering condition that is more than the target log filtering condition can be determined as a to-be-deleted log filtering condition, and at this time, the target parsing strategy of the alarm log parsing template can be generated based on the to-be-deleted log filtering condition, the to-be-added log filtering condition and the log filtering condition.

[0102] Step S402: updating the alarm log parsing template according to the target parsing strategy, to obtain an updated alarm log parsing template.

[0103] In this embodiment, since the target parsing strategy is generated based on the to-be-added log filtering condition and the log filtering condition, the to-be-added log filtering condition in the target parsing strategy can be extracted at this time, and the to-be-added log filtering condition is added to the alarm log parsing template, so as to update the alarm log parsing template and obtain an updated alarm log parsing template. In addition, if the target parsing strategy is generated based on the to-be-deleted log filtering condition, the to-be-added log filtering condition and the log filtering condition, the to-be-deleted log filtering condition and the to-be-added log filtering condition in the target parsing strategy can be extracted at this time, and the to-be-added log filtering condition is added to the alarm log parsing template, while the to-be-deleted log filtering condition in the alarm log parsing template is deleted, so as to update the alarm log parsing template.

[0104] Step S403: generating an alarm log parsing model based on the updated alarm log parsing template.

[0105] Further, the step S403 includes: obtaining a log parsing category and a log parsing quantity in the updated alarm log parsing template; determining a log execution logic corresponding to the log parsing category; generating an alarm log parsing model based on the log parsing category, the log parsing quantity and the log execution logic.

[0106] It should be understood that the above log analysis category can be the type or kind of log that needs to be parsed in the updated alarm log analysis template, for example, Apache, Syslog, Custom Log, etc.; the above log analysis quantity can be the number of log entries that need to be parsed in each log category defined in the updated alarm log analysis template. In this embodiment, the system can parse the updated alarm log analysis template to extract the log analysis category and the log analysis quantity in the updated alarm log analysis template.

[0107] It should be noted that the above log execution logic can be the parsing rule and processing flow set for each log category in the updated alarm log analysis template. In actual application, the system can obtain the corresponding execution logic of the log analysis category from the pre-defined logic library, and generate the final alarm log analysis model based on the log analysis category, the log analysis quantity and the log execution logic.

[0108] In this embodiment, the target parsing strategy of the alarm log analysis template is determined based on the to-be-updated operation and the log analysis requirement; the alarm log analysis template is updated according to the target parsing strategy to obtain an updated alarm log analysis template; and the alarm log analysis model is generated based on the updated alarm log analysis template, so that the alarm log analysis model generated by dynamically updating the alarm log analysis template can accurately extract the key information in the log, thereby facilitating the reduction of alarm log analysis errors and the improvement of the reliability of the alarm log analysis result.

[0109] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the log analysis method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0110] The present application also provides a log analysis device, please refer to Figure 4 , the log analysis device comprises:

[0111] The requirement analysis module 10 is configured to obtain the log analysis requirement of the to-be-analyzed alarm log, and determine the log type corresponding to the to-be-analyzed alarm log based on the log analysis requirement;

[0112] The template determination module 20 is configured to determine the alarm log analysis template corresponding to the to-be-analyzed alarm log according to the log type;

[0113] The template update module 30 is configured to determine the to-be-updated operation of the alarm log analysis template based on the log analysis requirement;

[0114] The model generation module 40 is configured to generate an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template.

[0115] The log analysis module 50 is configured to analyze the to-be-analyzed alarm log by using the alarm log analysis model to obtain an alarm log analysis result.

[0116] The log analysis device provided in the present application adopts the log analysis method in the above embodiments, and can solve the technical problem in the prior art that the rule-based alarm log analysis method needs to constantly update and maintain the analysis rules manually, resulting in high labor and time costs. Compared with the prior art, the log analysis device provided in the present application has the same beneficial effects as the log analysis method provided in the above embodiments, and other technical features in the log analysis device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0117] The present application provides a log analysis device, which comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the log analysis method in the above embodiment one.

[0118] Reference will be made to the following Figure 5 which shows a structural diagram of a log analysis device suitable for implementing the embodiments of the present application. The log analysis device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The log analysis device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0119] As Figure 5As shown, the log analysis device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. Various programs and data required for the operation of the log analysis device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch panel, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the log analysis device to communicate wirelessly or by wire with other devices to exchange data. Although the log analysis device having various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0120] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication apparatus, or installed from the storage apparatus 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0121] The log analysis device provided by the present disclosure adopts the log analysis method in the above-mentioned embodiments, and can solve the technical problem of log analysis. Compared with the prior art, the log analysis device provided by the present disclosure has the same beneficial effects as the log analysis method provided by the above-mentioned embodiments, and other technical features in the log analysis device are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0122] It should be understood that parts of the present disclosure can be realized by hardware, software, firmware, or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which shall be encompassed in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0124] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the log analysis method in the above embodiments.

[0125] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination thereof.

[0126] The above computer readable storage medium can be contained in the log analysis device; or can exist separately without being assembled into the log analysis device.

[0127] The above computer readable storage medium carries one or more programs, which, when executed by the log analysis device, cause the log analysis device to: obtain a log analysis requirement of an alarm log to be analyzed, and determine a log type corresponding to the alarm log to be analyzed based on the log analysis requirement; determine an alarm log analysis template corresponding to the alarm log to be analyzed according to the log type; determine a to-be-updated operation of the alarm log analysis template based on the log analysis requirement; generate an alarm log analysis model based on the to-be-updated operation and the alarm log analysis template; and analyze the alarm log to be analyzed through the alarm log analysis model to obtain an alarm log analysis result.

[0128] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0129] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0130] The modules involved in the embodiments of the present application can be implemented in the form of software or hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0131] The readable storage medium provided by the application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above log analysis method, and can solve the technical problem that the rule-based alarm log analysis method in the prior art needs to constantly manually update and maintain the analysis rules, resulting in high labor and time cost. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the application are the same as those of the log analysis method provided by the above embodiments, and will not be repeated here.

[0132] The above is only part of the embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields under the technical concept of the application, including in the patent protection scope of the application.

Claims

1. A log parsing method, characterized in that, The method includes: Obtain the log parsing requirements of the alarm logs to be parsed, and determine the log type corresponding to the alarm logs to be parsed based on the log parsing requirements; Determine the alarm log parsing template corresponding to the alarm log to be parsed based on the log type; Based on the log parsing requirements, determine the update operations to be performed on the alarm log parsing template; An alarm log parsing model is generated based on the operation to be updated and the alarm log parsing template; The alarm log to be parsed is parsed using the alarm log parsing model to obtain the alarm log parsing result; The step of determining the update operation of the alarm log parsing template based on the log parsing requirements includes: The alarm log parsing template is used to perform exploratory parsing of the alarm log to be parsed in order to verify the effectiveness of the alarm log parsing template, identify any deficiencies or changes in the log format in the alarm log parsing template, and obtain the log exploratory parsing results. Determine the log parsing rules for the alarm logs to be parsed based on the log parsing requirements; Determine whether the log probing and parsing results meet the log parsing requirements; If not, then determine the log filtering conditions in the log probing and parsing results, and the target log filtering conditions in the log parsing requirements. The target log filtering conditions include log level filtering, time range filtering, keyword filtering, and IP address filtering. Determine whether the log filtering conditions and the target log filtering conditions are consistent; If there is a discrepancy, the operation to be updated in the alarm log parsing template will be determined as a log filter condition update operation.

2. The method as described in claim 1, characterized in that, The step of determining the log type corresponding to the alarm log to be parsed based on the log parsing requirements includes: The alarm logs to be parsed are standardized to obtain standardized alarm logs to be parsed; Based on the log parsing requirements, the standardized alarm log to be parsed is subjected to feature extraction by a preset log feature extraction model to obtain alarm log feature information. The text pattern in the alarm log to be parsed is determined based on the alarm log feature information. Generate a regular expression corresponding to the alarm log to be parsed based on the text pattern; Based on the regular expression, the system searches for the log type corresponding to the alarm log to be parsed from a preset regular expression library.

3. The method as described in claim 1, characterized in that, The step of generating an alarm log parsing model based on the operation to be updated and the alarm log parsing template includes: Determine the target parsing strategy for the alarm log parsing template based on the operation to be updated and the log parsing requirements; The alarm log parsing template is updated according to the target parsing strategy to obtain the updated alarm log parsing template; An alarm log parsing model is generated based on the updated alarm log parsing template.

4. The method as described in claim 3, characterized in that, The step of determining the target parsing strategy for the alarm log parsing template based on the operation to be updated and the log parsing requirements includes: If the operation to be updated is the log filtering condition update operation, then the target log filtering condition is extracted from the log parsing requirements; Determine the new log filtering conditions to be added in the target log filtering conditions; The target parsing strategy for generating the alarm log parsing template based on the new log filtering conditions and the existing log filtering conditions.

5. The method as described in claim 3, characterized in that, The step of generating an alarm log parsing model based on the updated alarm log parsing template includes: Obtain the log parsing category and log parsing quantity from the updated alarm log parsing template; Determine the log execution logic corresponding to the log parsing category; An alarm log parsing model is generated based on the log parsing category, the number of log parsing operations, and the log execution logic.

6. A log parsing device, characterized in that, The device includes: The requirements analysis module is used to obtain the log parsing requirements of the alarm logs to be parsed, and to determine the log type corresponding to the alarm logs to be parsed based on the log parsing requirements. The template determination module is used to determine the alarm log parsing template corresponding to the alarm log to be parsed based on the log type. The template update module is used to determine the update operation to be performed on the alarm log parsing template based on the log parsing requirements. The model generation module is used to generate an alarm log parsing model based on the operation to be updated and the alarm log parsing template; The log parsing module is used to parse the alarm log to be parsed using the alarm log parsing model to obtain the alarm log parsing result; The template update module is further configured to perform exploratory parsing of the alarm log to be parsed using the alarm log parsing template to verify the validity of the alarm log parsing template, identify deficiencies or log format changes in the alarm log parsing template, and obtain log exploratory parsing results; determine the log parsing rules for the alarm log to be parsed according to the log parsing requirements; determine whether the log exploratory parsing results meet the log parsing requirements; if not, determine the log filtering conditions in the log exploratory parsing results and the target log filtering conditions in the log parsing requirements, wherein the target log filtering conditions include log level filtering, time range filtering, keyword filtering, and IP address filtering; determine whether the log filtering conditions and the target log filtering conditions are consistent; if they are inconsistent, determine the update operation of the alarm log parsing template as a log filtering condition update operation.

7. A log parsing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the log parsing method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the log parsing method as described in any one of claims 1 to 5.

Images