Log analysis method and device, equipment and storage medium

Abstract

The invention discloses a log analysis method, and relates to the technical field of computers. A first feature of the log is extracted, the log type of the log is determined according to the first feature, and the first feature represents the attribute of the log; according to the log type, the adaptation degree of the log and at least one basic analysis tool in an analysis tool library is calculated, at least one basic analysis tool with the adaptation degree larger than or equal to a preset threshold value is obtained, and in response to existence of multiple basic analysis tools, a target analysis tool is determined according to analysis benefits of the multiple basic analysis tools; and analyzing the log by adopting the target analysis tool to obtain the analysis result, thereby solving the technical problem that logs with different sources and structures can only be analyzed manually, and achieving the effects of improving the log analysis efficiency, reducing the labor cost, enhancing the readability and practicability of log data and improving the user experience. And meanwhile, the compatibility and the maintainability of the system are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log parsing method, apparatus, device, and storage medium. Background Art

[0002] In existing technology, the analysis of large amounts of logs related to BMCs (Baseboard Management Controllers) is primarily accomplished through manual interpretation and analysis. Furthermore, servers are equipped with different configurations, such as monitoring logs for smart network cards and DPUs (Data Processing Units), resulting in inefficient and cumbersome server log analysis. The analysis results lack a uniform and visual presentation, and manual tabular statistics cannot ensure the accuracy and reliability of BMC system log data analysis. Furthermore, due to the complexity and specificity of machine design, the BMCs of most servers are involved in monitoring and alarming various components of the entire machine. Improving the speed and accuracy of log analysis results will also improve the efficiency of server manufacturers in repairing and maintaining the entire machine.

[0003] Therefore, in view of the shortcomings of the existing technical solutions, the present invention provides a log parsing method. Summary of the Invention

[0004] The present application provides a log parsing method, apparatus, device, and storage medium to at least solve the problem in related technologies that logs with different sources and structures can only be parsed manually.

[0005] The present application provides a log parsing method, which includes: obtaining a log; extracting a first feature of the log, and determining a log type of the log based on the first feature, wherein the first feature represents an attribute of the log; calculating, based on the log type, the compatibility of the log with at least one basic parsing tool in a parsing tool library, obtaining at least one basic parsing tool whose compatibility is greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; and parsing the log using the target parsing tool to obtain a parsing result.

[0006] The present application also provides a log parsing device, which includes: a first processing module for obtaining a log, wherein a first feature represents an attribute of the log; a second processing module for extracting the first feature of the log and determining the log type of the log based on the first feature; a third processing module for calculating the compatibility of the log with at least one basic parsing tool in the parsing tool library based on the log type, obtaining at least one basic parsing tool with a compatibility greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; and a fourth processing module for parsing the log using the target parsing tool to obtain a parsing result.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; a processor for implementing the following steps when executing the computer program: obtaining a log; extracting a first feature of the log, and determining the log type of the log based on the first feature, wherein the first feature represents an attribute of the log; calculating the compatibility of the log with at least one basic parsing tool in the parsing tool library based on the log type, obtaining at least one basic parsing tool with a compatibility greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; and parsing the log using the target parsing tool to obtain a parsing result.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the following steps are implemented: obtaining a log; extracting a first feature of the log, and determining the log type of the log based on the first feature, wherein the first feature represents an attribute of the log; calculating the fitness of the log with at least one basic parsing tool in the parsing tool library based on the log type, obtaining at least one basic parsing tool with a fitness greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; and parsing the log using the target parsing tool to obtain a parsing result.

[0009] The present application also provides a computer program product, including a computer program, which implements the following steps when executed by a processor: obtaining a log; extracting a first feature of the log, and determining the log type of the log based on the first feature, wherein the first feature represents an attribute of the log; calculating the compatibility of the log with at least one basic parsing tool in the parsing tool library based on the log type, obtaining at least one basic parsing tool with a compatibility greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; and parsing the log using the target parsing tool to obtain a parsing result.

[0010] Through the present application, a log is obtained; a first feature of the log is extracted, and based on the first feature, the log type of the log is determined, wherein the first feature characterizes the attribute of the log; based on the log type, the fitness of the log and at least one basic parsing tool in the parsing tool library is calculated, and at least one basic parsing tool with a fitness greater than or equal to a preset threshold is obtained. In response to the existence of multiple basic parsing tools, a target parsing tool is determined based on the parsing benefits of the multiple basic parsing tools; the log is parsed using the target parsing tool to obtain a parsing result. Therefore, the efficiency of log analysis can be improved, labor costs can be reduced, the readability and practicality of log data can be enhanced, and at the same time, the system compatibility and maintainability can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 A flow chart of a log parsing method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of the parsing process of multiple basic parsing tools of a log parsing method provided in an embodiment of the present application;

[0014] Figure 3 A flowchart illustrating target parsing tool selection for a log parsing method provided in an embodiment of the present application;

[0015] Figure 4 A system diagram of a log parsing method provided in an embodiment of the present application;

[0016] Figure 5 A structural block diagram of a log parsing device provided in an embodiment of the present application;

[0017] Figure 6 This is a diagram of the internal structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] It should be noted that the terms "S1", "S2", etc. are used only for the purpose of describing the steps and do not specifically refer to the order or sequence, nor are they used to limit this application. They are merely for the convenience of describing the method of this application and should not be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0022] The embodiments of the present application provide a log parsing method, and the method is described in detail in conjunction with the execution process of the log parsing method.

[0023] S101: Obtain logs.

[0024] Here, logs can be logs with diverse sources and different structures. For example, different sources can include hardware underlying signals, firmware operation traces, and user interaction instructions, and different structures can include binary (such as SEL), structured text (IPMI command record), and streaming data (sensor list). Logs can include: system event log (SEL), sensor data record (SDR), firmware operation log (FW Trace), hardware error log (H / W Error), and user operation audit log (IPMI Command).

[0025] Among them, different logs can be obtained through collection instructions. Different collection instructions correspond to different transmission protocols. The transmission protocols may include IPMI (Intelligent Platform Management Interface) protocol, Redfish protocol, syslog protocol, HTTP (Hypertext Transfer Protocol), SNMP (Simple Network Management Protocol), etc.

[0026] Specifically, log files of network devices, security devices, etc. may also be obtained.

[0027] In one embodiment, when multiple logs are acquired through a collection instruction, the priority of the logs can be determined based on their impact scope and urgency. The impact scope primarily considers the impact of the logs on the system, users, and business, with specific factors including the number of affected users, user experience, financial impact, operational impact, system importance, likelihood of fault propagation, recovery difficulty, and data leakage risk. The urgency refers to the degree to which the logs require immediate attention and processing, with specific factors including: immediate response requirements, time-sensitive operations, error levels, abnormal behavior detection, and attack type and level.

[0028] Logs can be collected in real time or periodically. Specifically, machine learning algorithms can be used to adjust the collection strategy in real time based on system operating status, log volume changes, and other factors. For example, the collection frequency can be increased when the system is busy.

[0029] In one embodiment, assuming that the collected log is a BMC log, the execution subject may be a device corresponding to the BMC, or may be a device other than the device corresponding to the BMC.

[0030] In one embodiment, digital signatures or hash checks are added during the collection process, and encrypted transmission is performed using protocols such as HTTPS and TLS. This ensures that the log data is not tampered with and guarantees the authenticity and integrity of the log.

[0031] S102: Extract a first feature of the log, and determine the log type of the log based on the first feature, wherein the first feature represents an attribute of the log.

[0032] Here, the first feature may include features such as log format, log importance, log corresponding parser, log generation frequency, log dependency, log source, metadata, etc.

[0033] Among them, the log type can be determined through methods such as rule-based classification, classification models, and deep learning models.

[0034] S103: Calculate the compatibility of the log with at least one basic parsing tool in the parsing tool library according to the log type, obtain at least one basic parsing tool whose compatibility is greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determine the target parsing tool according to the parsing benefits of the multiple basic parsing tools.

[0035] Here, basic parsing tools may include regular expression parsing, structured data parsing, natural language processing, etc.

[0036] For example, assuming that the log type is an SEL log, the corresponding basic parsing tool may be an ipmitool parser; assuming that the log type is a Redfish log, the corresponding basic parsing tool may be a redfish parser.

[0037] The target parsing tool may include one basic parsing tool or multiple basic parsing tools.

[0038] Among them, fitness can be calculated through rule-based scoring systems, exact matching, machine learning models and other methods.

[0039] Specifically, the compatibility between the current log and each basic parsing tool in the parsing tool library is calculated.

[0040] In one embodiment, all types of logs can be displayed on the human-computer interaction interface. The user can select the category of the collected logs by clicking on the human-computer interaction interface, and the background determines the corresponding target parsing tool based on the category.

[0041] In one embodiment, after determining the target parsing tool, a lightweight virtual execution environment is created to verify the feasibility of the target parsing tool. For example, the code is as follows:

[0042]

[0043]

[0044] S104: Analyze the log using a target analysis tool to obtain analysis results.

[0045] Here, the analysis results may include hardware failures, temperature alarms, power status, system events, fan speeds, sensor data, firmware update records, etc.

[0046] In one embodiment, assuming that multiple logs need to be parsed, the priorities of the multiple logs can be determined through a directed acyclic graph scheduling algorithm; the processing order of the multiple logs can also be determined by setting the QoS (Quality of Service) level of the logs, specifically including real-time, high, medium and low levels.

[0047] It should be noted that this application improves log analysis efficiency, reduces labor costs, enhances the readability and practicality of log data, and improves system compatibility and maintainability by collecting and parsing logs.

[0048] In some specific implementations, after obtaining the log, the method further includes:

[0049] Detect the log encoding and log format of the log;

[0050] In response to detecting that there is garbled code in the log code, decoding the garbled code and converting the garbled code into a recognizable code;

[0051] In response to detecting that the log format is an unrecognizable format, converting the unrecognizable format into a recognizable format;

[0052] In response to detecting that the log encoding and the log format are all recognizable, the format of the log is converted into a preset format, wherein the preset format is at least any one of a key-value pair format, a JSON format, a CSV format, and an XML format.

[0053] The encoding format of the log can be detected through file header feature analysis, automatic detection using tools or libraries, rule matching, and pattern recognition.

[0054] In one embodiment, the garbled code can be decoded by using methods such as heuristic decoding, binary analysis, single-byte decoding, and using context information.

[0055] Specifically, first, it detects whether there is a sequence mark (BOM) at the head of the garbled code. When the sequence mark is detected, it is decoded using the corresponding encoding format. When the sequence mark is not present, the distribution range of the encoded byte value is detected. If the distribution of the byte value is detected to be within a specific range, the corresponding encoding format is determined (for example, the byte value of an ASCII character is between 0x00 and 0x7F), and decoding is performed using the encoding format. When the garbled code format cannot be identified, the source system of the log is obtained and decoding is performed using the default encoding format of the source system, or decoding is performed using the encoding format of an adjacent log. In this way, the integrity and accuracy of the data can be guaranteed, and the availability of the data can be enhanced.

[0056] Specifically, the unrecognized format may be that some log fragments in format B appear in a log in format A, and format B cannot be recognized by existing detection tools. At this time, the following is executed: analyze the log fragments in format B, analyze the structure, field separators, time format, etc. of format B, and formulate corresponding conversion rules or scripts based on the analysis results to convert format B into format A.

[0057] In this way, the efficiency of log integration and analysis can be improved.

[0058] In some specific implementations, extracting a first feature of a log and determining a log type of the log based on the first feature includes:

[0059] Extracting at least one first feature of the log using a convolutional neural network;

[0060] Converting at least one first feature into a vector, and matching the vector with a multivariate recognition model using a single instruction multiple data instruction to obtain a second feature corresponding to the log type, wherein the second feature represents multiple attributes of the log;

[0061] The second feature is input into the multivariate recognition model to determine the log type of the log.

[0062] Here, convolutional neural network is a deep learning model suitable for processing data with grid structure, including convolutional layers, activation functions, pooling layers, fully connected layers, regularization techniques, etc.

[0063] Here, the single instruction multiple data instruction is a parallel computing technology that allows corresponding operations to be performed on multiple data points at the same time under the control of one instruction.

[0064] Here, the multivariate recognition model identifies the type of log based on multiple parameters.

[0065] Here, the second feature is a feature composed of vectors transformed from multiple first features.

[0066] Among them, the multivariate recognition model can be trained through supervised learning or unsupervised learning algorithms.

[0067] In one embodiment, multiple models can be obtained by training on a labeled data set. The classification accuracy, recall rate, and F-index of each model are calculated. Based on the classification accuracy, recall rate, and F-index, the model with the highest overall score is determined as the multivariate recognition model. Simultaneously, during operation, user feedback and new data are collected to perform feedback learning on the multivariate recognition model, thereby improving the multivariate recognition model and enhancing its accuracy.

[0068] Among them, key features can be converted into vectors through methods such as word embedding, one-hot encoding, digitization, feature hashing, and custom feature engineering.

[0069] In one embodiment, in response to being unable to classify the log, an unknown log format may be identified through an LSTM (Long Short-Term Memory) network.

[0070] Here, different log types correspond to different second features.

[0071] Specifically, assuming that the first features are A, B, C, and D, the corresponding second features may be features composed of vectors converted from ABC, or may be features composed of vectors converted from BCD.

[0072] In this way, the use of SIMD instructions can accelerate the feature matching process and improve the classification accuracy through the multivariate recognition model.

[0073] In some specific embodiments, the compatibility between the log and at least one basic parsing tool in the parsing tool library is calculated based on the log type, and at least one basic parsing tool having a compatibility greater than or equal to a preset threshold is obtained. In response to the existence of multiple basic parsing tools, a target parsing tool is determined based on the parsing benefits of the multiple basic parsing tools, including:

[0074] Calculate, based on the log type, the ratio of the intersection of the log features and the parsing capability of at least one basic parsing tool in the parsing tool library to the union;

[0075] In response to the ratio being greater than or equal to a preset threshold, determining that the parsing tool matches the log;

[0076] The number of basic parsing tools in the statistical tool library that match the logs;

[0077] In response to a matching base parsing tool existing, using the base parsing tool as a target parsing tool;

[0078] In response to the presence of multiple matching basic parsing tools, establishing a parsing tool collaboration group;

[0079] Construct a synergy benefit matrix based on the analytical tool synergy group;

[0080] Calculate the combined analytical benefits of multiple basic analytical tools based on the synergistic benefit matrix and determine the equilibrium point where the analytical benefits are maximized;

[0081] In response to the equilibrium point corresponding to a basic parsing tool, the basic parsing tool corresponding to the equilibrium point is used as a target parsing tool;

[0082] In response to the equilibrium point corresponding to a plurality of basic analysis tools, the plurality of basic analysis tools corresponding to the equilibrium point are used as target analysis tools.

[0083] Here, a synergy payoff matrix is ​​a tool used to analyze the payoffs of multiple participants under different strategy combinations, particularly when considering the trade-off between cooperation and competition. The synergy payoff matrix includes participants, strategy options, and beneficiaries. For example, assuming there are basic analytical tools A, B, C, and D, the synergy payoff matrix calculates the analytical payoffs of these basic analytical tools or combinations of these basic analytical tools under various combinations.

[0084] Here, the parsing capabilities of the basic parsing tools may include accuracy, efficiency, supported data formats, scalability, fault tolerance, and security.

[0085] Among them, the collaborative benefits can be calculated through dimensions such as historical collaboration success rate, similarity of analysis tool features, and collaboration overhead.

[0086] Specifically, the analytical returns of multiple basic tools can be calculated through exhaustive methods, greedy algorithms, dynamic programming, Nash equilibrium theory, etc. Among them, Nash equilibrium is used to describe a state in which the strategy selected by each participant is the best response based on the strategies selected by all other participants.

[0087] In one embodiment, assuming that the basic parsing tools include T1, T2, T3 and T4, the parsing benefit when each basic parsing tool is run alone is defined as R i (such as accuracy, coverage, and efficiency, etc.), build a 4*4 collaborative benefit matrix S, and calculate the sum of the collaborative gains S between each two basic analytical tools based on the historical collaboration success rate, the similarity of analytical tool features, and the collaboration cost. ij , S ij Indicates tool T i With T j The additional synergistic gain when used together may be positive or negative. Determine the permutations and combinations of multiple basic analytical tools and calculate the total analytical benefit of each permutation and combination. The total analytical benefit can be obtained by calculating the sum of the individual benefits of all tools plus the sum of the synergistic gains between any two tools. For example, if the total analytical benefit of T2, T3, and T4 is to be calculated, it can be obtained by (R2+R3+R4)+(S 23 +S 24 +S 34 ), and calculate the total analytical benefit. This covers all possible combinations and provides more complete benefit information for the decision-making of the combination analysis tool.

[0088] For example, the following code can be used to calculate whether the log matches the basic parsing tool:

[0089]

[0090] In one embodiment, a dynamic evaluation matrix can be constructed by obtaining multiple parameters of the basic parsing tool, such as progress, speed, resource consumption, security level, knowledge coverage, etc., and combining the weight of each parameter to calculate the parsing capability of the basic parsing tool. The weight of the parameter is related to the current system status (such as CPU load, memory remaining, security warning level, etc.).

[0091] In one embodiment, the basic parsing tool automatically reports its parsing capability before being added to the parsing tool library.

[0092] In this way, intelligent and adaptive evolution of analytical tool selection is achieved.

[0093] In some specific implementations, a target parsing tool is used to parse the log to obtain parsing results, including:

[0094] In response to the target parsing tool including a plurality of basic parsing tools, obtaining parsing capabilities of the plurality of basic parsing tools;

[0095] Based on the parsing capabilities of multiple basic parsing tools, logs are parsed separately to obtain multiple sub-parsing results;

[0096] Based on multiple sub-parse results, the knowledge graph is used to infer the associations between the multiple sub-parse results to obtain the parsing results of the log.

[0097] Here, knowledge graph is a method of organizing data in the form of a graph, where nodes represent entities (such as people, places, events, etc.) and edges represent the relationships between entities.

[0098] For example, assuming the log is a BMC log, the target parsing tools include ipmitool, mcelog, fw_analyzer, and sensor_parser. Among them, ipmitool is used to obtain basic hardware information and status, mcelog is used to collect and parse CPU exception logs, fw_analyzer is used to comprehensively analyze firmware and sensor data to diagnose problems, and sensor_parser is used to parse and provide sensor readings. Specifically, Figure 2 This is a schematic diagram of the parsing process of multiple basic parsing tools in the embodiment of this application, such as Figure 2 As shown, the parsing process of multiple basic parsing tools in this application includes: first parsing the log through ipmitool, supplementing the hardware topology, parsing through mcelog, providing the microcode version, parsing through fw_analyzer, adding sensor data, and parsing through sensor_parser.

[0099] Specifically, multiple sub-parse results are automatically connected in series through the knowledge flow engine. The specific process includes: building a temporary knowledge graph based on multiple sub-parse results, executing rule reasoning on the knowledge graph, determining the potential associations between multiple sub-parse results, dynamically expanding the graph based on potential associations, detecting whether there are new associations, and thus determining whether to continue reasoning. When there are no new associations, the reasoning is terminated and the final parsing result is generated. When there are new associations, the reasoning is continued.

[0100] In this way, multiple basic parsing tools can be used to collaboratively parse logs, thereby improving data availability and promoting collaboration and sharing among basic parsing tools.

[0101] In some specific implementations, the logs are parsed separately based on the parsing capabilities of multiple basic parsing tools to obtain multiple sub-parsing results, including:

[0102] Obtain log status and the parsing capabilities of multiple basic parsing tools, where the status includes log content and parsing status;

[0103] Based on the log status and the parsing capabilities of multiple basic parsing tools, the parsing order of multiple basic parsing tools is obtained through reinforcement learning algorithms;

[0104] Multiple basic parsing tools parse logs according to the parsing order.

[0105] Here, reinforcement learning algorithm is a machine learning paradigm that enables an intelligent agent to learn how to make a series of decisions through trial and error while interacting with the environment.

[0106] Here, the parsing capabilities of the parsing tool include functional features, applicable log types, parsing efficiency, etc.

[0107] Specifically, the reinforcement learning algorithm can be Q-learning or a deep Q network.

[0108] Specifically, a reinforcement learning model based on a reinforcement learning algorithm is trained in advance using a data set to obtain a reinforcement learning model. The reinforcement learning model includes the model's policy parameters, state value function or action value function, etc.

[0109] Specifically, a training data set is collected in advance, and the model is trained using a reinforcement learning algorithm to obtain a reinforcement learning model. The log status and parsing capability are input into the reinforcement learning model. The reinforcement learning model outputs the basic parsing tool that should be selected in the current state according to the current parsing state of the log. The selected basic parsing tool is used to parse the log to obtain the sub-parsing result. The current state of the log is updated according to the parsing result, and the current state is input into the reinforcement learning model to obtain the selection of the next basic parsing tool until the log is parsed.

[0110] For example, the code may be as follows:

[0111] class PathOptimizer:

[0112] def__init__(self):

[0113] self.q_table=defaultdict(lambda:np.zeros(n_actions))

[0114] defchoose_action(self,state):

[0115] ifnp.random.uniform() <self.epsilon:

[0116] return random_action()

[0117] else:

[0118] return np.argmax(self.q_table[state])

[0119] defupdate_q(self,state,action,reward,next_state):

[0120] reward=α*Accuracy+β*(1 / TimeCost)+γ*SecurityScore

[0121] self.q_table[state][action]+=self.lr*(reward+self.gamma*np.max(self.q_table[next_state])-self.q_table[state][action])

[0122] In this way, the parsing efficiency can be improved by determining the parsing order, the parsing instructions can be improved, and the flexibility and adaptability of the parsing can be enhanced.

[0123] In some specific implementations, rendering the parsing results and presenting the parsing results to the user includes:

[0124] Determine the page layout of the display page according to the tag configuration information of the display page;

[0125] Render the parsing results according to the page layout;

[0126] The rendered parsing results are sent to the display page to display the parsing results.

[0127] Here, rendering refers to the process of converting a three-dimensional scene or two-dimensional graphics into an image.

[0128] Here, the tag configuration information is a data structure used to describe the properties, style, and layout of page elements or components, including the position, size, style, and absolute or relative layout relationships between elements.

[0129] The analysis results can be displayed to the user through a display terminal, which can be a mobile phone, computer, tablet, etc.

[0130] Among them, it can be presented to users in the form of images, reports, etc.

[0131] Specifically, the parsing results are standardized, unified into the same format, and rendered to obtain a rendering result.

[0132] In one embodiment, interactive query and customized reporting capabilities may be provided.

[0133] In one embodiment, the rendered parsing results are normalized.

[0134] In one embodiment, the parsing results may be rendered through multi-threading.

[0135] In this way, by visually displaying the analysis results to users, it is possible to improve comprehensibility and enhance user experience.

[0136] In one embodiment, Figure 3 A flow chart of target analysis tool selection in the embodiment of the present application is shown in FIG. Figure 3 As shown, the process of selecting the target parsing tool in this application includes: judging whether there is a match of multiple tools (i.e., multiple basic parsing tools) based on the log feature vector (i.e., log type); if there is a single tool match, directly calling the corresponding tool for parsing; if there is a multiple tool match, creating a tool collaboration group, calculating the tool collaboration benefit matrix, and performing Nash equilibrium solution. The result is that if there is a single strategy (i.e., the equilibrium point corresponds to one basic parsing tool), the equilibrium point tool is selected as the target parsing tool; if there is a mixed strategy (i.e., the equilibrium point corresponds to multiple basic parsing tools), a tool pipeline is generated as the target parsing tool.

[0137] In one embodiment, Figure 4 This is a schematic diagram of the system in the embodiment of the present application, such as Figure 4 As shown, the system in this application includes: a log profiling engine, a parsing tool warehouse, an intelligent scheduling hub and a unified output interface.

[0138] Specifically, the log profiling engine includes a feature fingerprint library that stores various known log features and a machine learning classifier that uses an LSTM network to identify unknown log formats.

[0139] Specifically, the parsing tool repository includes a pre-integrated tool layer and a custom plug-in layer. The pre-integrated tool layer includes a variety of standard tool sets, and the custom plug-in layer supports dynamic loading of plug-ins.

[0140] Specifically, the intelligent scheduling hub includes a task scheduler and a priority queue. The task scheduler includes a directed acyclic graph scheduling algorithm, and the priority queue includes setting service quality levels.

[0141] Specifically, the unified output interface includes standardized output and a visual dashboard. Standardized output includes conversion to the OpenTelemetry format, and the visual dashboard includes the Grafana template library.

[0142] It should be understood that although Figure 1-4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0144] An embodiment of the present application also provides a log parsing device, which includes: a first processing module 501, used to obtain a log; a second processing module 502, used to extract a first feature of the log, and determine the log type of the log based on the first feature, wherein the first feature represents an attribute of the log; a third processing module 503, used to calculate the fitness of the log and at least one basic parsing tool in the parsing tool library based on the log type, obtain at least one basic parsing tool with a fitness greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determine a target parsing tool based on the parsing benefits of the multiple basic parsing tools; a fourth processing module 504, used to parse the log using the target parsing tool to obtain a parsing result.

[0145] As a preferred implementation, in an embodiment of the present application, the device also includes a preprocessing module, which is specifically used to: detect the log encoding and log format of the log; in response to detecting that there is garbled code in the log encoding, decode the garbled code and convert the garbled code into a recognizable code; in response to detecting that the log format is an unrecognizable format, convert the unrecognizable format into a recognizable format; in response to detecting that the log encoding and log format are all recognizable, convert the format of the log into a preset format, wherein the preset format is at least any one of the key-value pair format, JSON format, CSV format and XML format.

[0146] As a preferred implementation, in the embodiment of the present application, the second processing module 502 is specifically used to: extract at least one first feature of the log through a convolutional neural network; convert the at least one first feature into a vector, match the vector with a multivariate recognition model through a single instruction multiple data instruction, and obtain a second feature corresponding to the log type, wherein the second feature represents multiple attributes of the log; and input the second feature into the multivariate recognition model to determine the log type of the log.

[0147] As a preferred implementation mode, in the embodiment of the present application, the third processing module 503 is specifically used to: calculate the ratio of the intersection between the log features of the log and the parsing capabilities of at least one basic parsing tool in the parsing tool library to the union according to the log type; determine that the parsing tool matches the log in response to the ratio being greater than or equal to a preset threshold; count the number of basic parsing tools in the tool library that match the log; in response to the existence of a matching basic parsing tool, use the basic parsing tool as the target parsing tool; in response to the existence of multiple matching basic parsing tools, establish a parsing tool collaboration group; based on the parsing tool collaboration group, construct a collaboration benefit matrix; based on the collaboration benefit matrix, calculate the combined parsing benefits of multiple basic parsing tools, and determine the equilibrium point with the maximum parsing benefit; in response to the equilibrium point corresponding to a basic parsing tool, use the basic parsing tool corresponding to the equilibrium point as the target parsing tool; in response to the equilibrium point corresponding to multiple basic parsing tools, use the multiple basic parsing tools corresponding to the equilibrium point as the target parsing tools.

[0148] As a preferred implementation, in an embodiment of the present application, the fourth processing module 504 is specifically used to: in response to the target parsing tool including multiple basic parsing tools, obtain the parsing capabilities of multiple basic parsing tools; parse the logs separately according to the parsing capabilities of the multiple basic parsing tools to obtain multiple sub-parsing results; based on the multiple sub-parsing results, use the knowledge graph to infer the association between the multiple sub-parsing results to obtain the parsing results of the log.

[0149] As a preferred implementation method, in the embodiment of the present application, the fourth processing module 504 is specifically used to: obtain the status of the log and the parsing capabilities of multiple basic parsing tools, where the status includes the log content and the parsing status; based on the status of the log and the parsing capabilities of multiple basic parsing tools, obtain the parsing order of multiple basic parsing tools through a reinforcement learning algorithm; and multiple basic parsing tools parse the log according to the parsing order.

[0150] As a preferred implementation method, in an embodiment of the present application, the device also includes a rendering module, which is specifically used to: determine the page layout of the display page based on the tag configuration information of the display page; render the parsing results according to the page layout; and send the rendered parsing results to the terminal to display the parsing results.

[0151] For the description of the features in the embodiment corresponding to the log parsing device, please refer to the relevant description of the embodiment corresponding to the log parsing method, and will not be repeated here.

[0152] The embodiment of the present application further provides an electronic device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a log parsing method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0153] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1: obtaining a log; S2: extracting a first feature of the log, and determining a log type of the log based on the first feature, wherein the first feature represents an attribute of the log; S3: calculating, based on the log type, a degree of compatibility between the log and at least one basic parsing tool in a parsing tool library, obtaining at least one basic parsing tool having a degree of compatibility greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; S4: parsing the log using the target parsing tool to obtain a parsing result.

[0155] In one embodiment, when the processor executes the computer program, the following steps are further implemented: detecting the log encoding and log format of the log; in response to detecting that there is garbled code in the log encoding, decoding the garbled code and converting the garbled code into a recognizable code; in response to detecting that the log format is an unrecognizable format, converting the unrecognizable format into a recognizable format; in response to detecting that the log encoding and log format are all recognizable, converting the format of the log into a preset format, wherein the preset format is at least any one of a key-value pair format, a JSON format, a CSV format, and an XML format.

[0156] In one embodiment, when the processor executes the computer program, it further implements the following steps: extracting at least one first feature of the log through a convolutional neural network; converting the at least one first feature into a vector, matching the vector with a multivariate recognition model through a single instruction multiple data instruction to obtain a second feature corresponding to the log type, wherein the second feature represents multiple attributes of the log; and inputting the second feature into the multivariate recognition model to determine the log type of the log.

[0157] In one embodiment, the processor further implements the following steps when executing the computer program: according to the log type, calculate the ratio of the intersection between the log features of the log and the parsing capabilities of at least one basic parsing tool in the parsing tool library to the union; in response to the ratio being greater than or equal to a preset threshold, determine that the parsing tool matches the log; count the number of basic parsing tools in the tool library that match the log; in response to the existence of a matching basic parsing tool, use the basic parsing tool as the target parsing tool; in response to the existence of multiple matching basic parsing tools, establish a parsing tool collaboration group; based on the parsing tool collaboration group, construct a collaboration benefit matrix; based on the collaboration benefit matrix, calculate the combined parsing benefits of multiple basic parsing tools, and determine the equilibrium point with the maximum parsing benefit; in response to the equilibrium point corresponding to a basic parsing tool, use the basic parsing tool corresponding to the equilibrium point as the target parsing tool; in response to the equilibrium point corresponding to multiple basic parsing tools, use the multiple basic parsing tools corresponding to the equilibrium point as the target parsing tools.

[0158] In one embodiment, when the processor executes the computer program, it also implements the following steps: in response to the target parsing tool including multiple basic parsing tools, the parsing capabilities of the multiple basic parsing tools are obtained; according to the parsing capabilities of the multiple basic parsing tools, the log is parsed separately to obtain multiple sub-parsing results; based on the multiple sub-parsing results, the knowledge graph is used to infer the association between the multiple sub-parsing results to obtain the parsing result of the log.

[0159] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining the status of the log and the parsing capabilities of multiple basic parsing tools, where the status includes the log content and the parsing status; based on the status of the log and the parsing capabilities of the multiple basic parsing tools, obtaining the parsing order of the multiple basic parsing tools through a reinforcement learning algorithm; and the multiple basic parsing tools parse the log according to the parsing order.

[0160] In one embodiment, when the processor executes the computer program, it also implements the following steps: determining the page layout of the display page based on the tag configuration information of the display page; rendering the parsing result according to the page layout; and sending the rendered parsing result to the terminal to display the parsing result.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: S1: obtaining a log, wherein a first feature characterizes an attribute of the log; S2: extracting the first feature of the log, and determining the log type of the log based on the first feature; S3: calculating the fitness of the log and at least one basic parsing tool in the parsing tool library based on the log type, obtaining at least one basic parsing tool with a fitness greater than or equal to a preset threshold, and in response to the existence of multiple basic parsing tools, determining a target parsing tool based on the parsing benefits of the multiple basic parsing tools; S4: parsing the log using the target parsing tool to obtain a parsing result.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: detecting the log encoding and log format of the log; in response to detecting that there is garbled code in the log encoding, decoding the garbled code and converting the garbled code into a recognizable code; in response to detecting that the log format is an unrecognizable format, converting the unrecognizable format into a recognizable format; in response to detecting that the log encoding and log format are all recognizable, converting the format of the log into a preset format, wherein the preset format is at least any one of a key-value pair format, a JSON format, a CSV format, and an XML format.

[0163] In one embodiment, when the processor executes the computer program, it further implements the following steps: extracting at least one first feature of the log through a convolutional neural network; converting the at least one first feature into a vector, matching the vector with a multivariate recognition model through a single instruction multiple data instruction to obtain a second feature corresponding to the log type, wherein the second feature represents multiple attributes of the log; and inputting the second feature into the multivariate recognition model to determine the log type of the log.

[0164] In one embodiment, the computer program further implements the following steps when executed by the processor: according to the log type, calculating the ratio of the intersection between the log features of the log and the parsing capabilities of at least one basic parsing tool in the parsing tool library to the union; in response to the ratio being greater than or equal to a preset threshold, determining that the parsing tool matches the log; counting the number of basic parsing tools in the tool library that match the log; in response to the existence of a matching basic parsing tool, taking the basic parsing tool as the target parsing tool; in response to the existence of multiple matching basic parsing tools, establishing a parsing tool collaboration group; based on the parsing tool collaboration group, constructing a collaboration benefit matrix; based on the collaboration benefit matrix, calculating the combined parsing benefits of multiple basic parsing tools, and determining the equilibrium point with the maximum parsing benefit; in response to the equilibrium point corresponding to a basic parsing tool, taking the basic parsing tool corresponding to the equilibrium point as the target parsing tool; in response to the equilibrium point corresponding to multiple basic parsing tools, taking the multiple basic parsing tools corresponding to the equilibrium point as the target parsing tools.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to the target parsing tool including multiple basic parsing tools, the parsing capabilities of the multiple basic parsing tools are obtained; according to the parsing capabilities of the multiple basic parsing tools, the log is parsed separately to obtain multiple sub-parsing results; based on the multiple sub-parsing results, the knowledge graph is used to infer the association between the multiple sub-parsing results to obtain the parsing result of the log.

[0166] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the status of the log and the parsing capabilities of multiple basic parsing tools, where the status includes the log content and the parsing status; based on the status of the log and the parsing capabilities of the multiple basic parsing tools, a parsing order of the multiple basic parsing tools is obtained through a reinforcement learning algorithm; and the multiple basic parsing tools parse the log according to the parsing order.

[0167] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the page layout of the display page based on the tag configuration information of the display page; rendering the parsing results according to the page layout; and sending the rendered parsing results to the terminal to display the parsing results.

[0168] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0169] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A log parsing method, characterized in that: The method comprises: Get logs; Extracting a first feature of the log, and determining a log type of the log based on the first feature, wherein the first feature represents an attribute of the log; Calculating, based on the log type, a compatibility between the log and at least one basic parsing tool in a parsing tool library, obtaining at least one basic parsing tool whose compatibility is greater than or equal to a preset threshold, and determining a target parsing tool based on parsing benefits of the multiple basic parsing tools if there are multiple basic parsing tools; The target parsing tool is used to parse the log to obtain a parsing result.

2. The log parsing method according to claim 1, characterized in that: After obtaining the log, the method further includes: Detecting the log code and log format of the log; In response to detecting that there is a garbled code in the log code, decoding the garbled code and converting the garbled code into a recognizable code; In response to detecting that the log format is an unrecognizable format, converting the unrecognizable format into a recognizable format; In response to detecting that the log code and the log format are all recognizable, the format of the log is converted into a preset format, wherein the preset format is at least any one of a key-value pair format, a JSON format, a CSV format, and an XML format.

3. The log parsing method according to claim 1, wherein: The extracting the first feature of the log and determining the log type of the log according to the first feature includes: Extracting at least one first feature of the log by a convolutional neural network; Converting the at least one first feature into a vector, and matching the vector with a multivariate recognition model using a single instruction multiple data instruction to obtain a second feature corresponding to the log type, wherein the second feature represents multiple attributes of the log; The second feature is input into the multivariate recognition model to determine the log type of the log.

4. The log parsing method according to claim 1, wherein: The step of calculating, based on the log type, a compatibility between the log and at least one basic parsing tool in a parsing tool library, obtaining at least one basic parsing tool whose compatibility is greater than or equal to a preset threshold, and determining a target parsing tool based on parsing benefits of the multiple basic parsing tools in response to the existence of multiple basic parsing tools, includes: Calculating, according to the log type, a ratio of an intersection of log features of the log and a parsing capability of at least one basic parsing tool in the parsing tool library to a union; In response to the ratio being greater than or equal to a preset threshold, determining that the parsing tool matches the log; and counting the number of basic parsing tools in the tool library that match the log; In response to a matching basic parsing tool existing, using the basic parsing tool as a target parsing tool; In response to the presence of multiple matching basic parsing tools, establishing a parsing tool collaboration group; Constructing a synergy benefit matrix based on the analytical tool synergy group; Calculate the combined analytical benefits of multiple basic analytical tools based on the synergistic benefit matrix and determine the equilibrium point where the analytical benefits are maximized; In response to the equilibrium point corresponding to a basic parsing tool, using the basic parsing tool corresponding to the equilibrium point as a target parsing tool; In response to the equilibrium point corresponding to a plurality of basic parsing tools, the plurality of basic parsing tools corresponding to the equilibrium point are used as target parsing tools.

5. The log parsing method according to claim 1, wherein: The using the target parsing tool to parse the log to obtain a parsing result includes: In response to the target parsing tool including a plurality of basic parsing tools, obtaining parsing capabilities of the plurality of basic parsing tools; Parsing the logs separately according to the parsing capabilities of the multiple basic parsing tools to obtain multiple sub-parsing results; Based on the multiple sub-parsing results, the knowledge graph is used to infer the associations between the multiple sub-parsing results to obtain the parsing result of the log.

6. The log parsing method according to claim 5, characterized in that: The logs are parsed respectively according to the parsing capabilities of the multiple basic parsing tools to obtain multiple sub-parsing results, including: Obtaining the status of the log and the parsing capabilities of the multiple basic parsing tools, wherein the status includes log content and parsing status; Based on the state of the log and the parsing capabilities of the multiple basic parsing tools, obtaining a parsing order of the multiple basic parsing tools through a reinforcement learning algorithm; The multiple basic parsing tools parse the log according to the parsing order.

7. The log parsing method according to claim 1, characterized in that: After parsing the log using the target parsing tool to obtain a parsing result, the method further includes: Determining a page layout of the display page according to the tag configuration information of the display page; Rendering the parsing result according to the page layout; The rendered parsing result is sent to the terminal to display the parsing result.

8. A log analysis device, characterized in that: The device comprises: The first processing module is used to obtain logs; a second processing module, configured to extract a first feature of the log and determine a log type of the log based on the first feature, wherein the first feature represents an attribute of the log; a third processing module configured to calculate, based on the log type, a compatibility between the log and at least one basic parsing tool in the parsing tool library, obtain at least one basic parsing tool whose compatibility is greater than or equal to a preset threshold, and, in response to the existence of multiple basic parsing tools, determine a target parsing tool based on the parsing benefits of the multiple basic parsing tools; The fourth processing module is used to parse the log using the target parsing tool to obtain a parsing result.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the log parsing method according to any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium, characterized in that A computer program is stored in a computer-readable storage medium, wherein when the computer program is executed by a processor, the steps of the log parsing method according to any one of claims 1 to 7 are implemented.

Images