Application log analysis method and device

Abstract

The invention discloses an application log analysis method and device. The method comprises the following steps: acquiring an application log of a target application and a system resource index of the target application during operation; a log template sequence is determined according to the application logs, a system resource index sequence is determined according to the system resource indexes, and the log template sequence is used for representing a set composed of log templates corresponding to the application logs; the system resource index sequence is used for representing a system resource index set in the same time window with the log template sequence; analyzing the log template sequence and the system resource index sequence by adopting an anomaly detection model to obtain an abnormal log template; and displaying the abnormal log template to the target object. The technical problems that a log processing mechanism in related technologies highly depends on manual intervention, and the root cause positioning efficiency is low are solved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for analyzing application logs. Background Technology

[0002] With the widespread adoption of cloud computing and microservice architectures, the log data generated by modern application systems is experiencing explosive growth and increasing complexity. In distributed systems and cloud-native architectures, application logs, as a crucial data source for operations monitoring, fault diagnosis, and business analysis, are becoming increasingly valuable. In this era of surging data volumes, system throughput is extremely high. To minimize the impact of failures on users and reduce losses, it is essential to provide rapid alerts, rapid handling, and rapid recovery for abnormal scenarios. Therefore, the analysis of abnormal logs is a critical step. Through real-time log collection and analysis, enterprises can quickly identify system anomalies, diagnose the causes of failures, and achieve proactive detection, thereby ensuring high service availability and business continuity. However, the log processing mechanisms in related technologies heavily rely on manual intervention, resulting in low efficiency in root cause localization.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an application log analysis method and apparatus to at least solve the technical problem in related technologies that log processing mechanisms are highly dependent on manual intervention and have low root cause localization efficiency.

[0005] According to one aspect of the embodiments of this application, an application log analysis method is provided, comprising: acquiring application logs of a target application and system resource indicators of the target application during runtime; determining a log template sequence based on the application logs and a system resource indicator sequence based on the system resource indicators, wherein the log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence; analyzing the log template sequence and the system resource indicator sequence using an anomaly detection model to obtain an abnormal log template; and displaying the abnormal log template to a target object.

[0006] Optionally, determining the log template sequence based on application logs includes: identifying dynamic variables and static text information in the application logs; grouping the application logs based on the static text information and retaining the position information of the dynamic variables to obtain the log templates corresponding to the application logs; storing the log templates in a log template library, wherein each log template corresponds to a template ID; after all log templates are stored in the log template library, retrieving the latest updated first preset number of target log templates from the log template library, and filling or truncating the target log templates to a fixed length to obtain the log template sequence.

[0007] Optionally, before using the anomaly detection model to analyze the log template sequence and system resource indicator sequence, the method further includes: obtaining the log timestamp corresponding to the target log template in the log template sequence; obtaining the system resource indicators under the log timestamp to obtain the system resource indicator sequence; and normalizing the system resource indicator sequence to obtain the indicator matrix.

[0008] Optionally, an anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain anomaly log templates. This includes: using the anomaly detection model to determine the context vector corresponding to the log template sequence and the indicator feature vector corresponding to the system resource indicator sequence; concatenating the context vector and the indicator feature vector to obtain a fused feature vector; using the classifier of the anomaly detection model to classify the fused feature vector for anomalies to obtain the log anomaly probability; and identifying log templates with a log anomaly probability greater than the anomaly probability threshold as anomaly log templates.

[0009] Optionally, the context vector corresponding to the log template sequence is determined using an anomaly detection model, including: converting the log template sequence into a first vector representation using the embedding layer of the anomaly detection model; obtaining the hidden state of the log template sequence at each time step using the bidirectional long short-term memory layer of the anomaly detection model; determining the weight of each time step using the attention layer of the anomaly detection model; and performing a weighted summation based on the weights and hidden states to obtain the context vector.

[0010] Optionally, determining the indicator feature vector corresponding to the system resource indicator sequence includes: processing the system resource indicator sequence using the long short-term memory layer in the anomaly detection model, and determining the output of the last time step as the indicator feature vector.

[0011] Optionally, the method further includes: obtaining the update application logs of the target application; determining the similarity between the update application logs and the log templates stored in the log template library; and creating an update log template corresponding to the update application logs if the similarity is less than a threshold.

[0012] Optionally, the method further includes: obtaining an exception log template and the first timestamp of the exception log corresponding to the exception log template; determining the target system resource indicators under the first timestamp; determining auxiliary analysis tags based on the exception log template and the target system resource indicators, and displaying the auxiliary analysis tags to the target object.

[0013] Optionally, the method further includes: obtaining the target code block of the exception log corresponding to the exception log template, and obtaining the context information corresponding to the exception log; determining the functional description of the target code block through the application programming interface; and determining the root cause inference and reference solution of the exception log from the historical reference library based on the context information and functional description.

[0014] According to another aspect of the embodiments of this application, an application log analysis apparatus is also provided, comprising: an acquisition module, configured to acquire application logs of a target application and system resource indicators of the target application during runtime; a determination module, configured to determine a log template sequence based on the application logs and a system resource indicator sequence based on the system resource indicators, wherein the log template sequence represents a set composed of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence; an analysis module, configured to analyze the log template sequence and the system resource indicator sequence using an anomaly detection model to obtain an abnormal log template; and a display module, configured to display the abnormal log template to a target object.

[0015] In this embodiment, the application logs of the target application and the system resource indicators of the target application during runtime are obtained; a log template sequence is determined based on the application logs, and a system resource indicator sequence is determined based on the system resource indicators. The log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence. An anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain an abnormal log template. The abnormal log template is then displayed to the target object, achieving the goal of quickly locating and understanding abnormal logs. This reduces manual intervention and improves the efficiency of root cause localization, thereby solving the technical problem of log processing mechanisms in related technologies being highly dependent on manual intervention and having low root cause localization efficiency. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1This is a hardware structure block diagram of a computer terminal for implementing an application log analysis method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart of an application log analysis method according to an embodiment of this application;

[0019] Figure 3 This is a structural diagram of an application log analysis system according to an embodiment of this application;

[0020] Figure 4 This is a structural diagram of an application log analysis device according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0024] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0025] Application: or application program, which in this application refers to a program that implements a specific function.

[0026] Application logs: These are records output in text format during program execution. Logs can contain a variety of information, such as thread ID, log timestamp, error messages, etc.

[0027] System resource metrics: numerical values ​​of system environment resources during program execution, including but not limited to CPU and memory usage.

[0028] Code repository: A code repository built by the program development project team, which stores the code of the relevant application.

[0029] In related technologies, mainstream log processing mechanisms primarily employ rule-based matching, keyword filtering, or statistical models (such as frequency- or time-window-based anomaly detection), coupled with alarm systems for real-time monitoring. Typical scenarios include triggering alarms when keywords like ERROR or Exception frequently appear in logs, notifying operations personnel to intervene. However, with increasingly complex system architectures, these solutions often lack adaptability to diverse log structures and specific root cause analysis and reference solutions for abnormal logs. Furthermore, accumulated operational knowledge, such as historical fault cases and solutions, is often not effectively preserved and reused. Specifically, this includes the following issues:

[0030] 1. Diverse log structures and high parsing difficulty: Different applications, services, and even different versions of the same service may use completely different log formats (such as structured JSON, unstructured text, Syslog, etc.), making it difficult to generalize traditional parsing methods based on regular expressions or fixed templates. Customized rules are required for different log sources, resulting in extremely high maintenance costs.

[0031] 2. Anomaly detection lacks contextual information, resulting in a high false alarm rate: Current methods typically analyze single log entries or short sequences in isolation, ignoring the overall system environment (such as resource utilization). For example, a "Database connection failed" log entry could be due to a database failure or a momentary network fluctuation, but existing detection mechanisms cannot combine indicators such as CPU and network latency for a comprehensive judgment, leading to a large number of invalid alarms.

[0032] 3. High reliance on manual intervention, resulting in low root cause analysis efficiency: Most anomaly detection systems only provide simple error messages (such as "10 ERROR logs detected"), and cannot automatically correlate related logs, trace call chains, or recommend solutions. Operations personnel still need to manually sift through massive amounts of logs and compare them with historical failure cases, which is time-consuming and prone to missing crucial clues.

[0033] These challenges severely restrict the real-time performance and effectiveness of log analysis. Against the backdrop of accelerated digital transformation, there is an urgent need to build a real-time detection solution that can adapt to multi-source log structures, intelligently integrate system contexts, and provide interpretable analysis.

[0034] To address the aforementioned issues, related technologies have proposed a solution, but this solution focuses more on explicit anomaly feature extraction and matching analysis. It suffers from the following problems: 1. Insufficient proactive capture of implicit anomalies; 2. Lack of anomaly observation priority; 3. Insufficient anomaly contextual relevance.

[0035] To address the problems existing in related technologies, embodiments of this application provide a method for analyzing application logs, which can be run on... Figure 1 The computer terminal shown is described below.

[0036] The application log analysis method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an application log analysis method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the application log analysis method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the application log analysis method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0040] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0041] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0042] In the above operating environment, this application provides an embodiment of an application log analysis method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] Figure 2This is a flowchart of an application log analysis method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0044] Step S202: Obtain the application logs of the target application and the system resource metrics of the target application during runtime.

[0045] In step S202 above, the application (program) generates various types and formats of log data during its operation, such as program status, event records, error messages, performance metrics, etc. By deploying a log collection agent, using log management tools, or directly accessing the application's log output, the log data generated by the target application can be obtained in real time or periodically.

[0046] System resource metrics refer to data on resource usage provided by the operating system and underlying hardware during the operation of a target application. System resource metrics include, but are not limited to, CPU utilization, memory utilization, disk I / O, network latency, and throughput.

[0047] The aforementioned application logs and system resource metrics can be collected through the data transmission pipeline of the data acquisition module. This data acquisition module provides data transmission between the application side and the analysis platform side using a "producer-consumer" model, including multi-format log data and providing system context information. This data acquisition module monitors and collects system resource metrics during the target application's runtime, and the data transmission pipeline establishes corresponding main application data transmission for the aforementioned two types of data.

[0048] Combining application logs and system resource metrics for anomaly detection and analysis can provide a comprehensive view of the system's health status. System resource metrics can fill the gaps in the complete description of the system's operating status that individual log data cannot provide, thereby more accurately locating the cause of the problem.

[0049] Step S204: Determine the log template sequence based on the application logs and the system resource indicator sequence based on the system resource indicators. The log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence.

[0050] In step S204 above, the log template is a structured representation extracted from the original application logs. By removing dynamically changing parts (such as timestamps, IP addresses, variable values, etc.), the basic structure and fixed text of the logs are retained, resulting in a standard template. The construction of the log template sequence allows for the standardization of originally unstructured or semi-structured log data, reducing the complexity of log analysis. Through templated processing, the system can identify different instances of similar log events. This log template sequence is used as input for subsequent anomaly detection models, enabling the models to learn the frequency, patterns, and trends of log events, thereby determining whether abnormal logs have been generated.

[0051] The system resource indicator sequence is a collection of system resource usage data within the same time window as the log template sequence. It reflects the system-level load and performance under different application log templates. The system resource indicator sequence provides contextual information for the analysis of abnormal logs, enabling accurate determination of whether log anomalies are related to system resource bottlenecks.

[0052] Step S206: The anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain the anomaly log template.

[0053] In step S206 above, the anomaly detection model can capture the patterns of application logs and system resource metrics changing over time, as well as the potential correlations between them. Through training, the model learns the characteristics of log template sequences and system resource metric sequences under normal operating conditions, as well as deviation patterns under abnormal conditions, thereby enabling it to detect abnormal log templates.

[0054] Step S208: Display the exception log template to the target object.

[0055] In step S208 above, the detected abnormal log templates are summarized and displayed to the target audience through a visualization interface. For example, the following can be displayed: the specific content of the abnormal log template, the frequency or severity of the abnormality, the timestamp of the abnormality, the status of related system resource indicators, and the context information of the abnormality (such as application name, service module, host information, etc.). The visualization interface can be an alarm panel, an operations and maintenance console, or a dedicated abnormality analysis view. The display method will be sorted according to the importance and urgency of the abnormal log templates, highlighting keywords or abnormal information in the abnormal log templates to remind the target audience to quickly identify the problem.

[0056] Through steps S202 to S208, the goal of quickly locating and understanding abnormal logs is achieved, thereby reducing manual intervention and improving the efficiency of root cause localization. This solves the technical problem of log processing mechanisms in related technologies being highly dependent on manual intervention and having low efficiency in root cause localization. The following is an explanation.

[0057] In step S204 of the above application log analysis method, determining the log template sequence based on the application log includes: identifying dynamic variables and static text information in the application log; grouping the application log based on the static text information and retaining the position information of the dynamic variables to obtain the log template corresponding to the application log; storing the log template in the log template library, wherein each log template corresponds to a template ID; after all log templates are stored in the log template library, obtaining the IDs of the latest updated first preset number of target log templates from the log template library; and filling or truncating the IDs of the target log templates to a fixed length to obtain the log template sequence.

[0058] In some embodiments of this application, consumers consume log messages from the data transmission pipeline and store the log messages in a time-series database. For system metric data, log timestamps are also synchronously stored in the database. For the log data, a clustering algorithm such as Drain3 is used to identify dynamic variables and static text information in the logs and extract them into log templates. A unique log template sequence is assigned to each unique log template. And store it in the log template library.

[0059] Dynamic variables refer to the changing parts of the log, such as timestamps, IP addresses, usernames, and the specific values ​​of error codes. Different log records have different values ​​for dynamic variables, so they are replaced with placeholders or general identifiers in the log template. Static text information refers to the fixed and unchanging parts of the log, such as the descriptive text of error messages and the descriptive statements of system events. Static text information is retained in the log template to identify the type and context of log events.

[0060] During the creation of log templates, application logs are grouped based on static text information. If two logs have the same static text information, they are considered to belong to the same group and have the same behavior pattern. For each group of logs, a log template is constructed that retains the static text information and uses specific placeholders or tags (e.g., {}). <var>The dynamic variables were replaced with other variables (e.g., time, numerical value, string). The type information of the dynamic variables (e.g., time, numerical value, string) was also recorded. Each completed log template was assigned a unique template ID and stored in the log template library. The template ID serves as a unique identifier for the log template, facilitating subsequent retrieval and management.

[0061] In the log template library, a preset number of target log templates are selected based on the latest updated timestamp. Each application log corresponds to a timestamp when it is generated. When determining the target log templates, the log templates in the log template library can be sorted according to the timestamps when different application logs are generated. The preset number of the latest updated log templates are determined as the target log templates. The preset number can be the size of a sliding window. After obtaining the ID of the target log template, it is padded or truncated to obtain the log template sequence.

[0062] In step S206 of the above application log analysis method, before using the anomaly detection model to analyze the log template sequence and system resource indicator sequence, the method further includes: obtaining the log timestamp corresponding to the target log template in the log template sequence; obtaining the system resource indicators under the log timestamp to obtain the system resource indicator sequence; and normalizing the system resource indicator sequence to obtain the indicator matrix.

[0063] In some embodiments of this application, the log timestamp is the recording time of the original application log corresponding to the target log template. After obtaining the log timestamp of the target log template, the system resource indicators corresponding to the log timestamp are obtained, thus obtaining the aforementioned system resource indicator sequence. The construction of the system resource indicator sequence needs to ensure that the time windows of the other log template sequences are consistent, that is, each data point in the system resource indicator sequence corresponds to the actual state of the system resources under the log timestamp. Since different types of system resource indicators have different scales and units, directly inputting system resource indicators with different units into the anomaly detection model will affect the model's predictive ability. Therefore, it is necessary to normalize the system resource indicator sequence and construct an indicator matrix from the normalized system resource indicator sequence, where each row represents a set of resource indicators at a time point, and each column represents a specific resource type.

[0064] In step S206 of the above application log analysis method, an anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain anomaly log templates. This includes: using the anomaly detection model to determine the context vector corresponding to the log template sequence and the indicator feature vector corresponding to the system resource indicator sequence; concatenating the context vector and the indicator feature vector to obtain a fused feature vector; using the classifier of the anomaly detection model to classify the fused feature vector for anomalies to obtain the log anomaly probability; and identifying log templates with a log anomaly probability greater than the anomaly probability threshold as anomaly log templates.

[0065] In the above steps, the context vector corresponding to the log template sequence is determined using an anomaly detection model, including: converting the log template sequence into a first vector representation using the embedding layer of the anomaly detection model; obtaining the hidden state of the log template sequence at each time step using the bidirectional long short-term memory layer of the anomaly detection model; determining the weight of each time step using the attention layer of the anomaly detection model; and performing a weighted summation based on the weights and hidden states to obtain the context vector.

[0066] In the above steps, determining the indicator feature vector corresponding to the system resource indicator sequence includes: processing the system resource indicator sequence using the long short-term memory layer in the anomaly detection model, and determining the output of the last time step as the indicator feature vector.

[0067] In some embodiments of this application, the context vector is the vector representation of the target log template corresponding to the ID of the target log template in the log template sequence. The embedding layer of the anomaly detection model transforms each template ID in the log template sequence into a first vector representation, thereby mapping discrete text information into a continuous vector space, facilitating the model's understanding of semantics and patterns. A bidirectional Long Short-Term Memory (LSTM) layer captures the temporal dependencies and bidirectional temporal information (i.e., past and future dependencies) in the target log templates corresponding to the log template sequence, thus obtaining the hidden state of the log template sequence at each time step. The hidden state contains contextual information about the events before and after that time step, reflecting the relationship and pattern between log events and events before and after them. The attention layer allows the model to focus on certain more important time steps while ignoring less relevant information when processing sequence data. In anomaly detection, the attention layer assigns a weight to each time step, the magnitude of which reflects the relative importance of the corresponding log template in the overall sequence. Based on the weights calculated by the attention layer and the hidden state of each time step, a weighted summation operation is performed to obtain the context vector of the entire log template sequence. It not only includes the original characteristics of the logs in the log template sequence, but also incorporates the importance information of the logs' changing trends, patterns, and exception contexts over time.

[0068] The size of the log template sequence can be n, represented as:

[0069]

[0070] The number of system resource indicators is m. Indicating the log template sequence The monitoring status of j system resource metrics corresponding to the timestamp is represented by the following metric matrix:

[0071]

[0072] The feature vector of the system resource indicator sequence reflects the system's operating status within a specific time window, such as CPU utilization, memory consumption, and disk I / O. Similar to the log template sequence, the system resource indicators are processed through an embedding layer and a Long Short-Term Memory (LSTM) layer to obtain a feature vector that represents the system resource status and changing trends. The output of the LSTM at the last time step is determined as the indicator feature vector.

[0073] The extracted context vector and indicator feature vector are concatenated to obtain a fused feature vector. This fused vector contains all the information from the log template sequence and the system resource indicator sequence, providing a comprehensive data perspective for the anomaly detection model, taking into account both application behavior and system environment factors. The fused feature vector is input into the classifier of the anomaly detection model. The classifier can be a fully connected layer in the model, followed by an activation function (such as Sigmoid or Softmax) to output the anomaly detection result. The output of the classifier is a probability value, representing the likelihood that the model judges the log template sequence to be abnormal within the current time window, i.e., the log anomaly probability mentioned above. The closer this log anomaly probability is to 1, the more likely the application log corresponding to the log template sequence is to contain an abnormal event.

[0074] A pre-defined anomaly probability threshold is used to distinguish between normal logs and anomalous log templates. The anomaly probability of logs obtained within each time window is compared with the anomaly probability threshold, and log templates with an anomaly probability greater than the threshold are identified as anomalous log templates.

[0075] In another optional embodiment, if the log anomaly probability exceeds the threshold, the log template sequence within the current time window is considered to be abnormal. Based on the determination of the abnormal log template sequence, the log template with a higher weight (e.g., greater than the weight threshold) in the log template sequence is located through the weight distribution of the attention layer in the anomaly detection model, and the abnormal log template is obtained.

[0076] The above-mentioned method for analyzing application logs also includes: obtaining the update application logs of the target application; determining the similarity between the update application logs and the log templates stored in the log template library; and creating an update log template corresponding to the update application logs if the similarity is less than a threshold.

[0077] In some embodiments of this application, when the target application generates new application logs, i.e., the aforementioned updated application logs, the similarity is calculated by comparing each updated application log with existing log templates in the log template library. Algorithms specifically designed for log parsing, such as the Drain algorithm, can be used. The similarity calculation primarily focuses on the static text information in the logs, while also considering the type and position of dynamic variables. If the similarity between the updated application log and an existing log template in terms of static text and the type and position of dynamic variables is greater than or equal to a threshold, no new log template needs to be created. If the similarity is less than the threshold, the creation of a new template is triggered, i.e., an updated log template is created and assigned a unique template ID.

[0078] The above-mentioned method for analyzing application logs also includes: obtaining an exception log template and the first timestamp of the exception log corresponding to the exception log template; determining the target system resource indicators under the first timestamp; determining auxiliary analysis tags based on the exception log template and the target system resource indicators, and displaying the auxiliary analysis tags to the target object.

[0079] In some embodiments of this application, the first timestamp is the time when the original exception log corresponding to the exception log template was generated. By obtaining the target system resource indicators under the first timestamp, and based on the exception log template and the target system resource indicators, auxiliary analysis tags are generated, such as "high risk of memory leak," "peak CPU utilization," "excessive network latency," etc. These auxiliary analysis tags can help operations and maintenance personnel quickly locate potential problems at the system resource level. The obtained auxiliary analysis tags are then integrated into the exception detection report and presented to operations and maintenance personnel through graphical interfaces, console output, or automatic email notifications. Simultaneously, the obtained auxiliary analysis tags may also be used in automated decision-making processes to guide the system in taking corresponding fault recovery or resource optimization measures.

[0080] The above-mentioned method for analyzing application logs also includes: obtaining the target code block of the exception log corresponding to the exception log template, and obtaining the context information corresponding to the exception log; determining the functional description of the target code block through the application programming interface; and determining the root cause inference and reference solution of the exception log from the historical reference library based on the context information and functional description.

[0081] In some embodiments of this application, once the exception log template is identified, the application code block that generated the exception log, i.e., the aforementioned target code block, is located by tracing back the source of the corresponding exception log. The contextual information of the exception log includes the system state at the time the exception log occurred, such as CPU load, memory usage, network status, and other relevant log information. This contextual information provides the environmental background for the exception. By calling an API that integrates code repository and natural language processing capabilities, the system can parse the semantics of the target code block and extract its functional description. The functional description helps to understand the role and expected behavior of the target code block. Using the contextual information of the exception log and the functional description of the code block as query conditions, the system searches for patterns of similar exception cases in the historical reference library. Through comparison, a preliminary inference of the root cause of the exception log can be obtained, such as "memory leak" or "external service interruption." Solutions related to this root cause are then searched from the historical reference library. These solutions include the experience and technical means used by past operations and maintenance personnel and developers to solve similar problems. The reference solution can be a detailed remediation guide or a suggested code modification or configuration adjustment strategy.

[0082] The following combination Figure 3 The analysis method for the above application logs will be further explained. Figure 3 This is a structural diagram of an application log analysis system according to an embodiment of this application, such as... Figure 3 As shown, it includes: a data acquisition module, a data analysis module, and a view analysis module. The main functions of the relevant modules are as follows:

[0083] (1) Data Acquisition Module: A data transmission pipeline is established through the "producer-consumer" model to collect application logs and system resource indicators of the (target) application. The system resource indicators can be obtained through the system's built-in monitoring tools such as vmstat. To reduce the impact on application performance, the producer, i.e. the application, transmits log messages to the log message queue asynchronously to provide basic data for the data analysis module.

[0084] (2) Data Analysis Module: This module is mainly responsible for persistent storage and data analysis. It persistently stores raw data, automatically extracts log templates from log data (i.e., application logs) and stores them in a log template library (or template database), and processes system indicator data with corresponding timestamps. It obtains log template sequences from the log template library and determines the corresponding system resource indicator sequences. These two sequences are then input into the anomaly detection model to analyze the probability of abnormal logs within the relevant window. Combined with auxiliary analysis labels generated from the system indicator data, it reduces the false positives for occasional system anomalies.

[0085] The data analysis module serves as an analysis platform, assisting in parsing the log data acquired by the data acquisition module and identifying abnormal logs. It extracts diverse logs into log templates and analyzes them using anomaly detection models. Simultaneously, it performs multimodal fusion of log template sequences and system resource indicator sequences, achieving more accurate root cause localization through joint modeling.

[0086] (3) View Analysis Module: This module is mainly responsible for interacting with operations and maintenance personnel. After obtaining the abnormal logs corresponding to the abnormal log templates from the data analysis module, it draws a visual abnormal view based on the weight, so that operations and maintenance personnel can prioritize and investigate abnormal matters. At the same time, it combines the knowledge base (including business rules and code repositories) to perform intelligent analysis of abnormalities, provide abnormal root cause inference analysis and solution references, and generate interpretable analysis outputs for the target objects.

[0087] The view analysis module provides an interactive platform for operations and maintenance personnel. Its main functions include visualizing abnormal logs, providing alarm notifications, and offering explanations, analyses, and solution references for abnormal content. Specifically, upon detecting an abnormal log, it provides a visual view to show the abnormal log information to operations and maintenance personnel, while simultaneously sending alarm information via SMS and other means. For example, it displays an abnormal log template keyword graph based on attention weights; the higher the weight, the brighter the display. When anomalies are also found in the auxiliary analysis tags of system metrics, they are highlighted, facilitating quick troubleshooting of key anomalies by operations and maintenance personnel.

[0088] The method in this application embodiment not only provides rapid location of anomaly logs but also supports the interpretation and analysis of anomaly content, providing a reference for operations and maintenance personnel to understand the root cause and resolve anomalies. Therefore, it is first necessary to build a knowledge base, the content of which is not limited to code libraries, business rules, etc. This knowledge base is then connected to an artificial intelligence platform, and the platform's API is used to scan and learn from the knowledge base. When anomaly logs are detected, the location of the log within a code block is located by reverse engineering based on data from the template library and persistent database where the log is located. The API is used to perform functional analysis of the code block containing the log, and finally, a textual description of the root cause inference and reference solutions for the relevant anomalies is provided.

[0089] The application log analysis method provided in this application aims to offer intelligent anomaly log analysis and processing. It uses a message queue to collect and transmit data, supports diverse log format processing, and incorporates system resource information to assist in anomaly log analysis and detection. Finally, it provides anomaly detection result display and advanced analysis. Addressing the issue of numerous log formats and the high manual maintenance costs of traditional rule matching, this application uses the Drain3 algorithm to identify log text and extract log templates. It also supports adding new log templates through log template similarity calculation, improving log format adaptability. To address the lack of contextual information in anomaly log detection, this application uses the LSTM algorithm to process and analyze log sequences within a sliding window. Simultaneously, it collects corresponding system resource indicators to assist in analysis, enhancing the distinguishability of anomaly logs and reducing false detections caused by occasional system fluctuations. To address the insufficient utilization of historical experience, this application constructs a knowledge base containing code libraries, business rules, and other information. Based on this knowledge base, it uses artificial intelligence technology to analyze anomaly logs, enabling operations and maintenance personnel to avoid large-scale anomaly investigation and improve effective handling time. Therefore, the method provided in this application embodiment has advantages over traditional anomaly log analysis schemes, such as wide adaptability, high accuracy, strong interpretability, and experience reuse.

[0090] Figure 4 This is a structural diagram of an application log analysis device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0091] The acquisition module 40 is used to acquire the application logs of the target application and the system resource indicators of the target application during runtime.

[0092] The determination module 42 is used to determine the log template sequence based on the application logs and the system resource indicator sequence based on the system resource indicators. The log template sequence is used to represent a set of log templates corresponding to multiple application logs, and the system resource indicator sequence is used to represent a set of system resource indicators within the same time window as the log template sequence.

[0093] Analysis module 44 is used to analyze the log template sequence and system resource indicator sequence using an anomaly detection model to obtain anomaly log templates;

[0094] Display module 46 is used to display the exception log template to the target object.

[0095] Through the acquisition module, determination module, analysis module, and display module in the above-mentioned application log analysis device, the goal of quickly locating and understanding abnormal logs is achieved, thereby reducing manual intervention and improving the efficiency of root cause location. This solves the technical problem of log processing mechanisms in related technologies being highly dependent on manual intervention and having low efficiency in root cause location.

[0096] In the determination module of the aforementioned application log analysis device, the determination module is also used to identify dynamic variables and static text information in the application log; group the application logs according to the static text information and retain the position information of the dynamic variables to obtain the log templates corresponding to the application logs, and store the log templates in the log template library, wherein each log template corresponds to a template ID; after all log templates are stored in the log template library, the latest updated first preset number of target log templates are obtained from the log template library, and the target log templates are filled or truncated to a fixed length to obtain the log template sequence.

[0097] In the determination module of the aforementioned application log analysis device, the determination module is further used to obtain the log timestamp corresponding to the target log template in the log template sequence; obtain the system resource indicators under the log timestamp to obtain the system resource indicator sequence; and perform normalization processing on the system resource indicator sequence to obtain the indicator matrix.

[0098] In the analysis module of the aforementioned application log analysis device, the analysis module is further used to determine the context vector corresponding to the log template sequence and the indicator feature vector corresponding to the system resource indicator sequence using an anomaly detection model; to concatenate the context vector and the indicator feature vector to obtain a fused feature vector; to use the classifier of the anomaly detection model to classify the fused feature vector for anomalies and obtain the log anomaly probability; and to determine the log templates with a log anomaly probability greater than the anomaly probability threshold as abnormal log templates.

[0099] In the analysis module of the aforementioned application log analysis device, the analysis module is further used to convert the log template sequence into a first vector representation using the embedding layer of the anomaly detection model; to obtain the hidden state of the log template sequence at each time step using the bidirectional long short-term memory layer of the anomaly detection model; to determine the weight of each time step using the attention layer of the anomaly detection model; and to obtain the context vector by performing a weighted summation based on the weight and the hidden state.

[0100] In the analysis module of the aforementioned application log analysis device, the analysis module is also used to process the system resource indicator sequence using the long short-term memory layer in the anomaly detection model, and determine the output of the last time step as the indicator feature vector.

[0101] The application log analysis device described above also includes a processing module 48, which is used to obtain the update application log of the target application; determine the similarity between the update application log and the log template stored in the log template library; and create an update log template corresponding to the update application log if the similarity is less than a threshold.

[0102] In the display module of the aforementioned application log analysis device, the display module is also used to obtain the abnormal log template and the first timestamp of the abnormal log corresponding to the abnormal log template; determine the target system resource indicators under the first timestamp; determine auxiliary analysis tags based on the abnormal log template and the target system resource indicators, and display the auxiliary analysis tags to the target object.

[0103] In the processing module of the aforementioned application log analysis device, the processing module is also used to obtain the target code block of the exception log corresponding to the exception log template, and to obtain the context information corresponding to the exception log; to determine the functional description of the target code block through the application programming interface; and to determine the root cause inference and reference solution of the exception log from the historical reference library based on the context information and functional description.

[0104] It should be noted that, Figure 4 The application log analysis device shown is used to perform... Figure 2 The application log analysis method shown above is also applicable to the application log analysis device, and will not be repeated here.

[0105] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory to execute program instructions that perform the following functions: acquiring application logs of a target application and system resource indicators of the target application during runtime; determining a log template sequence based on the application logs and a system resource indicator sequence based on the system resource indicators, wherein the log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence; analyzing the log template sequence and the system resource indicator sequence using an anomaly detection model to obtain an abnormal log template; and displaying the abnormal log template to the target object.

[0106] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The application log analysis method shown above is also applicable to this electronic device, and will not be repeated here.

[0107] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following application log analysis method by running the computer program: acquiring the application logs of a target application and system resource indicators of the target application during runtime; determining a log template sequence based on the application logs and a system resource indicator sequence based on the system resource indicators, wherein the log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence; analyzing the log template sequence and the system resource indicator sequence using an anomaly detection model to obtain an abnormal log template; and displaying the abnormal log template to the target object.

[0108] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The analysis method for application logs shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0109] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the application log analysis method in various embodiments of this application.

[0110] This application also provides a computer program that, when executed by a processor, implements the steps of the application log analysis method in various embodiments of this application.

[0111] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0117] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / var>

Claims

1. A method for analyzing application logs, characterized in that, include: Obtain the application logs of the target application, as well as the system resource metrics of the target application during runtime; A log template sequence is determined based on the application logs, and a system resource indicator sequence is determined based on the system resource indicators, wherein the log template sequence represents a set of log templates corresponding to multiple application logs, and the system resource indicator sequence represents a set of system resource indicators within the same time window as the log template sequence; An anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain anomaly log templates; The exception log template is displayed to the target object.

2. The method according to claim 1, characterized in that, Based on the application logs, a log template sequence is determined, including: Identify dynamic variables and static text information in the application logs; The application logs are grouped according to the static text information, and the position information of the dynamic variables is retained to obtain the log templates corresponding to the application logs. The log templates are stored in the log template library, wherein each log template corresponds to a template ID. After all log templates are stored in the log template library, the IDs of the latest updated first preset number of target log templates are obtained from the log template library. The IDs of the target log templates are then padded or truncated to a fixed length to obtain the log template sequence.

3. The method according to claim 2, characterized in that, Before analyzing the log template sequence and the system resource indicator sequence using the anomaly detection model, the method further includes: Obtain the log timestamp corresponding to the target log template in the log template sequence; Obtain the system resource metrics under the log timestamp to obtain the system resource metric sequence; The system resource index sequence is normalized to obtain an index matrix.

4. The method according to claim 1, characterized in that, An anomaly detection model is used to analyze the log template sequence and the system resource indicator sequence to obtain an anomaly log template, including: The anomaly detection model is used to determine the context vector corresponding to the log template sequence and the indicator feature vector corresponding to the system resource indicator sequence. By concatenating the context vector and the indicator feature vector, a fused feature vector is obtained; The classifier of the aforementioned anomaly detection model is used to classify the fused feature vector for anomalies, thereby obtaining the log anomaly probability; Log templates with an anomaly probability greater than the anomaly probability threshold are identified as the abnormal log templates.

5. The method according to claim 4, characterized in that, The anomaly detection model is used to determine the context vector corresponding to the log template sequence, including: The log template sequence is converted into a first vector representation using the embedding layer of the anomaly detection model; The hidden state of the log template sequence at each time step is obtained using the bidirectional long short-term memory layer of the anomaly detection model. The attention layer of the anomaly detection model is used to determine the weights at each time step; The context vector is obtained by weighted summation based on the weights and the hidden states.

6. The method according to claim 4, characterized in that, Determining the indicator feature vector corresponding to the system resource indicator sequence includes: The system resource index sequence is processed using the long short-term memory layer in the anomaly detection model, and the output of the last time step is determined as the index feature vector.

7. The method according to claim 2, characterized in that, The method further includes: Obtain the update application logs of the target application; Determine the similarity between the updated application log and the log templates stored in the log template library; If the similarity is less than the threshold, an update log template corresponding to the updated application log is created.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the exception log template and the first timestamp of the exception log corresponding to the exception log template; Determine the target system resource metrics under the first timestamp; Based on the anomaly log template and the target system resource indicators, auxiliary analysis tags are determined and displayed to the target object.

9. The method according to claim 8, characterized in that, The method further includes: Obtain the target code block of the exception log corresponding to the exception log template, and obtain the context information corresponding to the exception log; The functional description of the target code block is determined through the application programming interface; Based on the context information and the function description, the root cause inference and reference solution of the anomaly log are determined from the historical reference library.

10. An application log analysis device, characterized in that, include: The acquisition module is used to acquire the application logs of the target application, as well as the system resource metrics of the target application during runtime. The determination module is used to determine a log template sequence based on the application logs and a system resource indicator sequence based on the system resource indicators, wherein the log template sequence is used to represent a set of log templates corresponding to multiple application logs, and the system resource indicator sequence is used to represent a set of system resource indicators within the same time window as the log template sequence; The analysis module is used to analyze the log template sequence and the system resource indicator sequence using an anomaly detection model to obtain an anomaly log template; The display module is used to show the exception log template to the target object.

Images