Log detection methods, log detection devices, and electronic devices

Abstract

This application provides a log detection method, a log detection device, and an electronic device. The method includes acquiring log sequence data of an application, where the log sequence data is a structured log of the application's execution process arranged chronologically within a predetermined time period; extracting multiple "golden signals" from the log sequence data, where the golden signals characterize the application's performance; counting the multiple golden signals within different time periods to obtain multiple anomaly signatures, where each anomaly signature is a unique identifier for whether the structured log contains a golden signal; and determining a persistent anomaly time period based at least on the similarity of the multiple anomaly signatures, and identifying the log sequence data corresponding to the persistent anomaly time period as persistent anomaly log data. This method solves the problem of inaccurate log anomaly detection results in the prior art.

Description

Technical Field

[0001] This application relates to the field of system log detection technology, and more specifically, to a log detection method, a log detection device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] As more and more organizations migrate to the cloud, a major challenge for operations teams is how to effectively utilize the vast amounts of information derived from multiple data sources to help maintain the robustness and availability of cloud services. Existing solutions rely on periodic log data to understand the application's state and diagnose the root causes of problems. Examples include supervised and unsupervised methods. Supervised methods preprocess and extract features from existing datasets of normal and abnormal log data, then perform data mining on the obtained normal and abnormal log features within a preset time window to build normal and abnormal behavior models. Unsupervised methods widely use principal component analysis and autoencoders, defining a scoring function to represent the degree of anomaly of data objects and ranking them according to their anomaly scores. However, these methods only focus on the data points, context, and sets that lead to false positives, resulting in low detection accuracy. Summary of the Invention

[0003] The main objective of this application is to provide a log detection method, a log detection device, a computer-readable storage medium, and an electronic device to at least solve the problem of inaccurate log anomaly detection results in the prior art.

[0004] According to one aspect of this application, a log detection method is provided, comprising: acquiring log sequence data of an application, wherein the log sequence data is a structured log of the application execution process arranged in chronological order within a predetermined time period; extracting multiple golden signals from the log sequence data, wherein the golden signals are used to characterize the performance of the application; performing counting processing on the multiple golden signals within different time periods to obtain multiple abnormal signatures, wherein the abnormal signature is a unique identifier for whether the structured log has the golden signal; determining a continuous abnormal time period based at least on the similarity of the multiple abnormal signatures, and determining the log sequence data corresponding to the continuous abnormal time period as continuous abnormal log data.

[0005] Optionally, obtaining the application's log sequence data includes: obtaining the application's raw log information, wherein the raw log information is unstructured free text information generated by the application during operation; using the Drain algorithm to transform the raw log information to obtain structured log events; and using a sliding time window to group the log events to obtain the log sequence data.

[0006] Optionally, the golden signals include error signals or unavailability signals. The error signals are used to characterize errors occurring during the execution of the application, and the unavailability signals are used to characterize that the application cannot provide normal service within a predetermined time period. Extracting multiple golden signals from the log sequence data includes: obtaining a code regular expression, wherein the code regular expression is used to characterize error codes or unavailability codes in the log sequence data; performing word segmentation on the log sequence data to obtain multiple words; comparing the words with the regular expression, and determining that the log sequence data contains the golden signals if the log sequence data satisfies the regular expression.

[0007] Optionally, the golden signal includes one of a delay signal, a saturation signal, and an unavailability signal. The delay signal indicates that the application's response time to a user request is greater than a time threshold. The saturation signal indicates that the application's load is greater than a load threshold. The unavailability signal indicates that the application cannot provide normal service within a predetermined time period. Extracting multiple golden signals from the log sequence data includes: obtaining basic seed words and synonyms of the basic seed words, wherein the seed words indicate that the application is experiencing at least one of delay, saturation, and unavailability; constructing a rule dictionary based on the basic seed words and the synonyms; performing word segmentation on the log sequence data to obtain multiple words; calculating the cosine distance between the words and the words in the rule dictionary to obtain a first cosine distance; and determining the log sequence data as the golden signal if the first cosine distance is less than a first predetermined distance.

[0008] Optionally, the golden signal includes an anomaly signal, which is used to characterize an abnormal situation during the operation of the application. Extracting multiple golden signals from the log sequence data includes: obtaining a negative word set, wherein the negative word set includes multiple negative words, which are used to characterize negative sentiment towards the application; calculating the intersection of the negative word set and the rule dictionary to obtain a rule intersection; constructing a negative sentiment dictionary based on the sentiment polarity of the rule intersection; performing word segmentation on the log sequence data to obtain multiple words; calculating the cosine distance between the words and the words in the negative sentiment dictionary to obtain a second cosine distance; and determining the log sequence data as the golden signal if the second cosine distance is less than a second predetermined distance.

[0009] Optionally, multiple gold signals within different time periods are counted to obtain multiple abnormal signatures, including: obtaining abnormal data from the log sequence data and determining the abnormal time period corresponding to the abnormal data, wherein the abnormal data is used to characterize abnormal behavior of the application; obtaining the number of log lines corresponding to the abnormal time period, wherein the number of log lines includes multiple log sequence data; and counting the gold signals with the same number of log lines to obtain the abnormal signature.

[0010] Optionally, the continuous abnormal time period is determined at least based on the similarity of the multiple abnormal signatures, including: calculating one of the Jensen-Shannon divergence, Kourbach-Leibler divergence, and cosine distance in the vector space of the multiple abnormal signatures to obtain the similarity of the multiple abnormal signatures; if the similarity is greater than a similarity threshold, the time period corresponding to the multiple abnormal signatures is determined as the continuous abnormal time period.

[0011] According to another aspect of this application, a log detection apparatus is provided, comprising: an acquisition unit for acquiring log sequence data of an application, wherein the log sequence data is a structured log of the application execution process arranged in chronological order within a predetermined time period; an extraction unit for extracting multiple golden signals from the log sequence data, wherein the golden signals are used to characterize the performance of the application; a processing unit for counting the multiple golden signals within different time periods to obtain multiple abnormal signatures, wherein the abnormal signature is a unique identifier for whether the structured log has the golden signal; and a determination unit for determining a continuous abnormal time period based at least on the similarity of the multiple abnormal signatures, and determining the log sequence data corresponding to the continuous abnormal time period as continuous abnormal log data.

[0012] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0013] According to another aspect of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute any of the methods described by the computer program.

[0014] The technical solution of this application first obtains the log sequence data of the application; then, it extracts multiple golden signals from the log sequence data, which are used to characterize the performance of the application; next, it counts the multiple golden signals in different time periods to obtain multiple anomaly signatures, which are unique identifiers for whether a structured log has a golden signal; finally, it determines the continuous anomaly time period based at least on the similarity of the multiple anomaly signatures, and determines the log sequence data corresponding to the continuous anomaly time period as continuous anomaly log data. By counting and aggregating the golden signals of all logs within a predetermined time period, the influence of false positive anomalies in log anomalies can be suppressed, thereby improving the accuracy of log anomaly detection results and solving the problem of inaccurate log anomaly detection results in the prior art. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for an execution log detection method provided in an embodiment of this application is shown;

[0017] Figure 2 A schematic flowchart of a log detection method according to an embodiment of this application is shown;

[0018] Figure 3 A detailed flowchart of a log detection method according to an embodiment of this application is shown;

[0019] Figure 4 A structural block diagram of a log detection device provided according to an embodiment of this application is shown.

[0020] The above figures include the following reference numerals:

[0021] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] As described in the background section, existing technologies only focus on abnormal data points, contexts, and sets that lead to false positives, resulting in low detection accuracy. To address the above issues, embodiments of this application provide a log detection method, a log detection device, a computer-readable storage medium, and an electronic device.

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a log detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the log detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] This embodiment provides a log detection method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Figure 2 This is a flowchart of a log detection method according to an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:

[0031] Step S201: Obtain the application's log sequence data, wherein the log sequence data is a structured log of the application's execution process arranged in chronological order within a predetermined time period.

[0032] Specifically, structured logs are a log format that organizes log information in a way that is easy to parse and analyze. This format typically includes fixed fields and values, where fields represent specific attributes of the log information, such as timestamps, log levels, event types, and user information. Structured logs enable log data to be efficiently processed and queried by automated tools.

[0033] Step S202: Extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0034] Specifically, "golden signals" are key indicators for monitoring the health and performance of an application, such as errors, latency, saturation, traffic, unavailability, and anomalies. These golden signals can be categorized based on the severity of log anomalies. Since the information captured in logs is closely related to the application and its state, and essentially contains these signals, identifying golden signals from the logs is crucial for subsequent analysis.

[0035] Step S203: Count the multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0036] Specifically, for example, if there are 6 log entries in a time window, 1 of which is detected as an anomaly, 1 as an error, and the remaining 4 as normal, then the current anomaly signature is (1,1,0,0,0,0).

[0037] Step S204: Based on the similarity of at least a number of the above-mentioned abnormal signatures, determine the continuous abnormal time period, and determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0038] Specifically, since some anomalous signatures share similarities, these similarities can be used to detect persistent anomalous regions. This method can be applied to the post-filtering system of any existing log anomaly detector.

[0039] This embodiment first acquires application log sequence data; then, it extracts multiple "golden signals" from the log sequence data, which characterize the application's performance; next, it counts these golden signals across different time periods to obtain multiple anomaly signatures, each a unique identifier for whether a structured log contains a golden signal; finally, based at least on the similarity of the multiple anomaly signatures, it determines a persistent anomaly time period and identifies the log sequence data corresponding to this period as persistently anomaly log data. By counting and aggregating the golden signals of all logs within a predetermined time period, the influence of false positives in log anomalies can be suppressed, thereby improving the accuracy of log anomaly detection results and solving the problem of inaccurate log anomaly detection results in existing technologies.

[0040] In specific implementation, step S201 can be achieved through the following steps: Step S2011, obtain the raw log information of the application, wherein the raw log information is unstructured free text information generated by the application during operation; Step S2012, use the Drain algorithm to transform the raw log information to obtain structured log events; Step S2013, use a sliding time window to group the log events to obtain the log sequence data. This method can further quickly obtain the log sequence data.

[0041] Specifically, the Drain algorithm is an online log template mining method that extracts templates from log message streams in real time. This algorithm uses a fixed-depth parse tree to guide the search process for log groups. The specific steps of the Drain algorithm are as follows: First, the raw logs are preprocessed using simple regular expressions based on professional knowledge, such as IP addresses and block IDs, to remove identifiers from the raw log messages. The purpose of this step is to simplify the log messages, making them easier to parse. Next, a search is performed based on log message length, starting from the root node of the parse tree, which distinguishes log groups based on the number of identifiers in the log messages. A path is selected based on the number of identifiers in the preprocessed log messages. This step is based on the assumption that log messages with the same log events are likely to have the same log message length. A search is then performed using identifiers, traversing from the first-level nodes determined in the previous steps to the leaf nodes. This step assumes that the identifier at the beginning of the log message is more likely to be constant. Finally, a search is performed using identifier similarity: before this step, a leaf node containing a list of log groups has been traversed. The most suitable log group is selected from the list by calculating the similarity between the log message and the log events in each log group. If the log group with the highest similarity is found and its similarity is greater than or equal to a predefined threshold, that group is returned as the best match; otherwise, a flag is returned indicating that no suitable match was found. The parse tree is updated: If a suitable log group is returned as described above, the log ID of the current log message is added to the log ID in the returned log group, and the log events in the returned log group are updated. If no suitable log group is found, a new log group is created based on the current log message, and the parse tree is updated. Traversing from the root node to the leaf node that should contain the new log group, new internal nodes and leaf nodes are added accordingly along this path. The key to the Drain algorithm is that it uses a fixed-depth parse tree to limit the number of nodes visited during the search process, thereby improving search efficiency. Furthermore, it can adapt to real-time changing log streams by continuously learning and extracting log templates from the original log entries.

[0042] To quickly extract the aforementioned error signals or unavailability signals, the aforementioned "golden signals" include error signals or unavailability signals. The error signals are used to characterize errors occurring during the execution of the aforementioned application, and the unavailability signals are used to characterize the aforementioned application's inability to provide normal service within a predetermined time period. The aforementioned step S202 of this application can be implemented through the following steps: Step S2021, obtaining a code regular expression, wherein the aforementioned code regular expression is used to characterize the error code or unavailability code of the aforementioned log sequence data; Step S2022, performing word segmentation processing on the aforementioned log sequence data to obtain multiple words; Step S2023, comparing the aforementioned words with the aforementioned regular expression, and determining that the aforementioned log sequence data contains the aforementioned "golden signals" if the aforementioned log sequence data satisfies the aforementioned regular expression.

[0043] Specifically, taking error signals as an example, the error codes mentioned above are mainly HTTP 5xx error codes and APM (Application Performance Management) error codes. By using dictionaries or regular expressions corresponding to the error codes, markers can be detected, thereby quickly extracting the aforementioned key signals. Taking unavailability signals as an example, these occur when 4xx HTTP codes occur or resources cannot be found during a service request, requiring more urgent handling. Regular expressions can be used to detect 4xx HTTP codes, thereby quickly extracting the aforementioned key signals.

[0044] The aforementioned "golden signal" includes one of a delay signal, a saturation signal, and an unavailability signal. The delay signal indicates that the application's response time to a user request exceeds a time threshold; the saturation signal indicates that the application's load exceeds a load threshold; and the unavailability signal indicates that the application cannot provide normal service within a predetermined time period. Step S202 can also be implemented in other ways, such as: Step S2024, obtaining basic seed words and their synonyms, where the seed words indicate at least one of delay, saturation, and unavailability in the application; Step S2025, constructing a rule dictionary based on the basic seed words and their synonyms; Step S2026, performing word segmentation on the log sequence data to obtain multiple words; Step S2027, calculating the cosine distance between the words and the words in the rule dictionary to obtain a first cosine distance; Step S2028, determining the log sequence data as the aforementioned "golden signal" if the first cosine distance is less than a first predetermined distance. This method can further quickly obtain one of the aforementioned delay signal, saturation signal, and unavailability signal.

[0045] Specifically, taking the unavailability signal as an example, a dictionary is constructed using seed words such as "available," "reachable," "connected," and "search." For any application, low latency is required. Taking the latency signal as an example, the constructed model includes some manually identified seed words, such as "time-consuming" and "latency," and the word in the embedding space with the closest cosine distance to the seed word is used as the dictionary entry for the latency category. The aforementioned saturation signal also refers to a measure of the application's resource load. By understanding the traffic that the service can effectively handle, potential resource failures can be predicted in advance to avoid errors for clients. The following seed words, "saturation," "threshold," "storage," and "bandwidth," can be used to construct the dictionary for saturation signal categories.

[0046] In some embodiments, the aforementioned "golden signal" includes an anomaly signal, which is used to characterize an abnormal situation occurring during the operation of the application. Step S202 can be further implemented through the following steps: Step S2051, obtaining a set of negative words, wherein the set of negative words includes multiple negative words, which characterize negative sentiment towards the application; Step S2052, calculating the intersection of the set of negative words and the rule dictionary to obtain a rule intersection; Step S2053, constructing a negative sentiment dictionary based on the sentiment polarity of the rule intersection; Step S2054, performing word segmentation on the log sequence data to obtain multiple words; Step S2055, calculating the cosine distance between the words and the words in the negative sentiment dictionary to obtain a second cosine distance; Step S2056, if the second cosine distance is less than a second predetermined distance, determining the log sequence data as the aforementioned "golden signal". This method can further quickly obtain the aforementioned anomaly signal.

[0047] Specifically, the aforementioned abnormal signals mainly include exception stack traces, error codes in middleware or the operating system, or any negative messages. Regular expressions are used to detect markers indicating exceptions, such as "File Not Found Exception." A log entry containing a negative message, such as "Unable to create an order due to an unspecified I / O error," can also be considered an exception. To quickly capture negative messages in logs, a negative sentiment lexicon for IT vocabulary can be constructed using the intersection of a domain-specific GloVe word embedding model and negatively related words in the Vader sentiment lexicon.

[0048] Step S203 above can be implemented through the following steps: Step S2031, obtain abnormal data from the log sequence data and determine the abnormal time period corresponding to the abnormal data, wherein the abnormal data is used to characterize abnormal behavior of the application; Step S2032, obtain the number of log lines corresponding to the abnormal time period, wherein the number of log lines includes multiple log sequence data; Step S2033, count the gold signals with the same number of log lines to obtain the abnormal signature. This method can further quickly achieve the aggregation processing of gold signals.

[0049] A log exception is triggered when a log window exhibits anomalous behavior. In log exception detection, the window size *w* for the anomalous time period is a configurable parameter, and the signature of the exception can be calculated using the log lines within window *w*. Specifically, for an anomalous data point *i*, assume the log window *w* starts from time *t*. s to t e Let L be the number of log lines in the time period. Obtain the golden signal of each log in L, and then aggregate the golden signals of these L logs by counting and aggregating them to obtain the classification count distribution of abnormal data point i, which is the above-mentioned abnormal signature.

[0050] In another embodiment, step S204 can be implemented through the following steps: Step S2041, calculate one of the Jensen-Shannon divergence, Kourbach-Leibler divergence, and cosine distance in the vector space for the multiple anomalous signatures to obtain the similarity of the multiple anomalous signatures; Step S2042, if the similarity is greater than a similarity threshold, determine the time period corresponding to the multiple anomalous signatures as the continuous anomalous time period. This method establishes continuous similarity by using anomalous features, which can further quickly obtain the continuous anomalous time period.

[0051] Because persistent anomalies lead to increased task cascading latency, the overall execution cost of system tasks is higher than that of transient anomalies. Existing technologies detect persistent anomalies by considering the temporal evolution of anomalies in an unsupervised manner. However, this method does not use the features of the anomalies to establish persistent similarity; instead, it captures a series of persistent anomalous regions. Furthermore, the related models have many parameters, requiring tuning for each application, and the workload increases exponentially when the model is deployed to a new environment. Specifically, given anomaly data points X in a time series with a range of x1, x2, ..., xn, first, the time series vector is standardized to obtain a basic probability distribution. Then, the persistence detection algorithm described above is used to detect persistent regions. This algorithm can provide three parameters—minimum duration, minimum threshold, and tolerance—to control the effect. The minimum duration ensures that the persistent region must be at least for a specified duration, set by the operations engineer, depending on the nature of the monitored application. The minimum threshold captures the minimum similarity that anomaly data points must satisfy as part of a persistent region, controlling to some extent the compactness of signatures appearing in the persistent region. The tolerance takes the number of data points that do not meet the similarity criteria as part of the persistent region. Finally, the detection results of log anomalies are output. Furthermore, the method effectively reduces the number of false alarms in anomaly detection, filtering out transient anomalies that do not require intervention, helping operations personnel focus on critical issues. Moreover, the entire method can be used with existing anomaly detection technologies as a post-filtering system; the system method is insensitive to module parameters and is relatively easy to deploy in new environments.

[0052] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the log detection method of this application will be described in detail below with reference to specific embodiments.

[0053] This embodiment relates to a specific log detection method, such as... Figure 3 As shown, it includes the following steps:

[0054] Step S1: Collect log information from multiple microservices to obtain log data;

[0055] Step S2: Perform gold signal detection on the log data to obtain the gold signal;

[0056] Step S3: Format the log data and perform anomaly detection to obtain an anomaly window;

[0057] Step S4: Analyze the anomaly type of the gold signal within the anomaly window to obtain the analysis results;

[0058] Step S5: Continuously monitor the analysis results to obtain persistent abnormal areas;

[0059] Step S6: Output the log anomaly detection result for the persistent abnormal region.

[0060] This application also provides a log detection device. It should be noted that the log detection device of this application can be used to execute the log detection method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0061] The log detection device provided in the embodiments of this application will be described below.

[0062] Figure 4 This is a schematic diagram of a log detection device according to an embodiment of this application. Figure 4 As shown, the device includes:

[0063] The acquisition unit 10 is used to acquire the log sequence data of the application, wherein the log sequence data is a structured log of the application execution process arranged in chronological order within a predetermined time period.

[0064] Specifically, structured logs are a log format that organizes log information in a way that is easy to parse and analyze. This format typically includes fixed fields and values, where fields represent specific attributes of the log information, such as timestamps, log levels, event types, and user information. Structured logs enable log data to be efficiently processed and queried by automated tools.

[0065] Extraction unit 20 is used to extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0066] Specifically, "golden signals" are key indicators for monitoring the health and performance of an application, such as errors, latency, saturation, traffic, unavailability, and anomalies. These golden signals can be categorized based on the severity of log anomalies. Since the information captured in logs is closely related to the application and its state, and essentially contains these signals, identifying golden signals from the logs is crucial for subsequent analysis.

[0067] Processing unit 30 is used to count multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0068] Specifically, for example, if there are 6 log entries in a time window, 1 of which is detected as an anomaly, 1 as an error, and the remaining 4 as normal, then the current anomaly signature is (1,1,0,0,0,0).

[0069] The determining unit 40 is used to determine a continuous abnormal time period based at least on the similarity of multiple of the above-mentioned abnormal signatures, and to determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0070] Specifically, since some anomalous signatures share similarities, these similarities can be used to detect persistent anomalous regions. This method can be applied to the post-filtering system of any existing log anomaly detector.

[0071] In this embodiment, the acquisition unit acquires the log sequence data of the application; the extraction unit extracts multiple golden signals from the log sequence data, which are used to characterize the application's performance; the processing unit counts the multiple golden signals within different time periods to obtain multiple anomaly signatures, which are unique identifiers for whether a structured log has a golden signal; and the determination unit determines the continuous anomaly time period based at least on the similarity of the multiple anomaly signatures, and determines the log sequence data corresponding to the continuous anomaly time period as continuous anomaly log data. By counting and aggregating the golden signals of all logs within a predetermined time period, the influence of false positive anomalies in log anomalies can be suppressed, thereby improving the accuracy of log anomaly detection results and solving the problem of inaccurate log anomaly detection results in the prior art.

[0072] In its specific implementation, the aforementioned acquisition unit includes a first acquisition module, a first processing module, and a second processing module. The first acquisition module acquires the raw log information of the application, which is unstructured free text information generated during the application's operation. The first processing module uses the Drain algorithm to transform the raw log information into structured log events. The second processing module uses a sliding time window to group the log events, obtaining the log sequence data. This device can further acquire the log sequence data quickly.

[0073] Specifically, the Drain algorithm is an online log template mining method that extracts templates from log message streams in real time. This algorithm uses a fixed-depth parse tree to guide the search process for log groups. The specific steps of the Drain algorithm are as follows: First, the raw logs are preprocessed using simple regular expressions based on professional knowledge, such as IP addresses and block IDs, to remove identifiers from the raw log messages. The purpose of this step is to simplify the log messages, making them easier to parse. Next, a search is performed based on log message length, starting from the root node of the parse tree, which distinguishes log groups based on the number of identifiers in the log messages. A path is selected based on the number of identifiers in the preprocessed log messages. This step is based on the assumption that log messages with the same log events are likely to have the same log message length. A search is then performed using identifiers, traversing from the first-level nodes determined in the previous steps to the leaf nodes. This step assumes that the identifier at the beginning of the log message is more likely to be constant. Finally, a search is performed using identifier similarity: before this step, a leaf node containing a list of log groups has been traversed. The most suitable log group is selected from the list by calculating the similarity between the log message and the log events in each log group. If the log group with the highest similarity is found and its similarity is greater than or equal to a predefined threshold, that group is returned as the best match; otherwise, a flag is returned indicating that no suitable match was found. The parse tree is updated: If a suitable log group is returned as described above, the log ID of the current log message is added to the log ID in the returned log group, and the log events in the returned log group are updated. If no suitable log group is found, a new log group is created based on the current log message, and the parse tree is updated. Traversing from the root node to the leaf node that should contain the new log group, new internal nodes and leaf nodes are added accordingly along this path. The key to the Drain algorithm is that it uses a fixed-depth parse tree to limit the number of nodes visited during the search process, thereby improving search efficiency. Furthermore, it can adapt to real-time changing log streams by continuously learning and extracting log templates from the original log entries.

[0074] To quickly extract the aforementioned error signals or unavailability signals, the aforementioned "golden signals" include error signals or unavailability signals. The error signals are used to characterize errors occurring during the execution of the aforementioned application, and the unavailability signals are used to characterize the aforementioned application's inability to provide normal service within a predetermined time period. The extraction unit of this application includes a second acquisition module, a third processing module, and a first determination module. The second acquisition module is used to acquire a code regular expression, wherein the code regular expression is used to characterize the error code or unavailability code of the aforementioned log sequence data. The third processing module is used to perform word segmentation processing on the aforementioned log sequence data to obtain multiple words. The first determination module is used to compare the aforementioned words with the aforementioned regular expression, and if the aforementioned log sequence data satisfies the aforementioned regular expression, it determines that the aforementioned log sequence data contains the aforementioned "golden signals".

[0075] Specifically, taking error signals as an example, the error codes mentioned above are mainly HTTP 5xx error codes and APM (Application Performance Management) error codes. By using dictionaries or regular expressions corresponding to the error codes, markers can be detected, thereby quickly extracting the aforementioned key signals. Taking unavailability signals as an example, these occur when 4xx HTTP codes occur or resources cannot be found during a service request, requiring more urgent handling. Regular expressions can be used to detect 4xx HTTP codes, thereby quickly extracting the aforementioned key signals.

[0076] The aforementioned "golden signal" includes one of a delay signal, a saturation signal, and an unavailability signal. The delay signal indicates that the application's response time to a user request exceeds a time threshold; the saturation signal indicates that the application's load exceeds a load threshold; and the unavailability signal indicates that the application cannot provide normal service within a predetermined time period. The extraction unit further includes a third acquisition module, a first construction module, a fourth processing module, a first calculation module, and a second determination module. The third acquisition module acquires basic seed words and their synonyms, whereby the seed words indicate at least one of delay, saturation, and unavailability in the application. The first construction module constructs a rule dictionary based on the basic seed words and their synonyms. The fourth processing module performs word segmentation on the log sequence data to obtain multiple words. The first calculation module calculates the cosine distance between the words and the words in the rule dictionary to obtain a first cosine distance. The second determination module determines the log sequence data as the "golden signal" if the first cosine distance is less than a first predetermined distance. This device can further quickly acquire one of the aforementioned delay signal, saturation signal, and unavailability signal.

[0077] Specifically, taking the unavailability signal as an example, a dictionary is constructed using seed words such as "available," "reachable," "connected," and "search." For any application, low latency is required. Taking the latency signal as an example, the constructed model includes some manually identified seed words, such as "time-consuming" and "latency," and the word in the embedding space with the closest cosine distance to the seed word is used as the dictionary entry for the latency category. The aforementioned saturation signal also refers to a measure of the application's resource load. By understanding the traffic that the service can effectively handle, potential resource failures can be predicted in advance to avoid errors for clients. The following seed words, "saturation," "threshold," "storage," and "bandwidth," can be used to construct the dictionary for saturation signal categories.

[0078] In some embodiments, the aforementioned "golden signal" includes an anomaly signal, which characterizes an abnormal situation occurring during the operation of the application. The extraction unit includes a fourth acquisition module, a second calculation module, a second construction module, a fifth processing module, a third calculation module, and a third determination module. The fourth acquisition module acquires a set of negative words, which includes multiple negative words characterizing negative sentiment towards the application. The second calculation module calculates the intersection of the negative word set and the rule dictionary to obtain a rule intersection. The second construction module constructs a negative sentiment dictionary based on the sentiment polarity of the rule intersection. The fifth processing module performs word segmentation on the log sequence data to obtain multiple words. The third calculation module calculates the cosine distance between the words and the words in the negative sentiment dictionary to obtain a second cosine distance. The third determination module determines the log sequence data as the "golden signal" if the second cosine distance is less than a second predetermined distance. This device can further quickly acquire the aforementioned anomaly signal.

[0079] Specifically, the aforementioned abnormal signals mainly include exception stack traces, error codes in middleware or the operating system, or any negative messages. Regular expressions are used to detect markers indicating exceptions, such as "File Not Found Exception." A log entry containing a negative message, such as "Unable to create an order due to an unspecified I / O error," can also be considered an exception. To quickly capture negative messages in logs, a negative sentiment lexicon for IT vocabulary can be constructed using the intersection of a domain-specific GloVe word embedding model and negatively related words in the Vader sentiment lexicon.

[0080] The aforementioned processing unit includes a fifth acquisition module, a sixth acquisition module, and a sixth processing module. The fifth acquisition module acquires abnormal data from the log sequence data and determines the abnormal time period corresponding to the abnormal data. The abnormal data characterizes abnormal behavior in the application. The sixth acquisition module acquires the number of log lines corresponding to the abnormal time period, where the number of log lines includes multiple log sequence data. The sixth processing module counts the gold signals with the same number of log lines to obtain the abnormal signature. This device can further rapidly perform gold signal aggregation processing.

[0081] A log exception is triggered when a log window exhibits anomalous behavior. In log exception detection, the window size *w* for the anomalous time period is a configurable parameter, and the signature of the exception can be calculated using the log lines within window *w*. Specifically, for an anomalous data point *i*, assume the log window *w* starts from time *t*. s to t e Let L be the number of log lines in the time period. Obtain the golden signal of each log in L, and then aggregate the golden signals of these L logs by counting and aggregating them to obtain the classification count distribution of abnormal data point i, which is the above-mentioned abnormal signature.

[0082] In another embodiment, the determining unit includes a fourth calculation module and a fourth determining module. The fourth calculation module is used to calculate one of the Jensen-Shannon divergence, Kourbach-Leibler divergence, and cosine distance in the vector space for the multiple anomalous signatures to obtain the similarity of the multiple anomalous signatures. The fourth determining module is used to determine the time period corresponding to the multiple anomalous signatures as the continuous anomalous time period when the similarity is greater than a similarity threshold. This device establishes continuous similarity by using anomalous features, which can further quickly obtain the continuous anomalous time period.

[0083] Because persistent anomalies lead to increased task cascading latency, the overall execution cost of system tasks is higher than that of transient anomalies. Existing technologies detect persistent anomalies by considering the temporal evolution of anomalies in an unsupervised manner. However, this method does not use the features of the anomalies to establish persistent similarity; instead, it captures a series of persistent anomalous regions. Furthermore, the related models have many parameters, requiring tuning for each application, and the workload increases exponentially when the model is deployed to a new environment. Specifically, given anomaly data points X in a time series with a range of x1, x2, ..., xn, first, the time series vector is standardized to obtain a basic probability distribution. Then, the persistence detection algorithm described above is used to detect persistent regions. This algorithm can provide three parameters—minimum duration, minimum threshold, and tolerance—to control the effect. The minimum duration ensures that the persistent region must be at least for a specified duration, set by the operations engineer, depending on the nature of the monitored application. The minimum threshold captures the minimum similarity that anomaly data points must satisfy as part of a persistent region, controlling to some extent the compactness of signatures appearing in the persistent region. The tolerance takes the number of data points that do not meet the similarity criteria as part of the persistent region. Finally, the detection results of log anomalies are output. Furthermore, the method effectively reduces the number of false alarms in anomaly detection, filtering out transient anomalies that do not require intervention, helping operations personnel focus on critical issues. Moreover, the entire method can be used with existing anomaly detection technologies as a post-filtering system; the system method is insensitive to module parameters and is relatively easy to deploy in new environments.

[0084] The aforementioned log detection device includes a processor and a memory. The acquisition unit, extraction unit, processing unit, and determination unit are all stored as program units in the memory, and the processor executes these program units to achieve their respective functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0085] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and logs can be monitored by adjusting kernel parameters.

[0086] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0087] This invention provides a computer-readable storage medium including a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute the log detection method.

[0088] Specifically, log detection methods include:

[0089] Step S201: Obtain the application's log sequence data, wherein the log sequence data is a structured log of the application's execution process arranged in chronological order within a predetermined time period.

[0090] Specifically, structured logs are a log format that organizes log information in a way that is easy to parse and analyze. This format typically includes fixed fields and values, where fields represent specific attributes of the log information, such as timestamps, log levels, event types, and user information. Structured logs enable log data to be efficiently processed and queried by automated tools.

[0091] Step S202: Extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0092] Specifically, "golden signals" are key indicators for monitoring the health and performance of an application, such as errors, latency, saturation, traffic, unavailability, and anomalies. These golden signals can be categorized based on the severity of log anomalies. Since the information captured in logs is closely related to the application and its state, and essentially contains these signals, identifying golden signals from the logs is crucial for subsequent analysis.

[0093] Step S203: Count the multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0094] Specifically, for example, if there are 6 log entries in a time window, 1 of which is detected as an anomaly, 1 as an error, and the remaining 4 as normal, then the current anomaly signature is (1,1,0,0,0,0).

[0095] Step S204: Based on the similarity of at least a number of the above-mentioned abnormal signatures, determine the continuous abnormal time period, and determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0096] Specifically, since some anomalous signatures share similarities, these similarities can be used to detect persistent anomalous regions. This method can be applied to the post-filtering system of any existing log anomaly detector.

[0097] This invention provides a processor for running a program, wherein the program executes the log detection method described above.

[0098] Specifically, log detection methods include:

[0099] Step S201: Obtain the application's log sequence data, wherein the log sequence data is a structured log of the application's execution process arranged in chronological order within a predetermined time period.

[0100] Specifically, structured logs are a log format that organizes log information in a way that is easy to parse and analyze. This format typically includes fixed fields and values, where fields represent specific attributes of the log information, such as timestamps, log levels, event types, and user information. Structured logs enable log data to be efficiently processed and queried by automated tools.

[0101] Step S202: Extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0102] Specifically, "golden signals" are key indicators for monitoring the health and performance of an application, such as errors, latency, saturation, traffic, unavailability, and anomalies. These golden signals can be categorized based on the severity of log anomalies. Since the information captured in logs is closely related to the application and its state, and essentially contains these signals, identifying golden signals from the logs is crucial for subsequent analysis.

[0103] Step S203: Count the multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0104] Specifically, for example, if there are 6 log entries in a time window, 1 of which is detected as an anomaly, 1 as an error, and the remaining 4 as normal, then the current anomaly signature is (1,1,0,0,0,0).

[0105] Step S204: Based on the similarity of at least a number of the above-mentioned abnormal signatures, determine the continuous abnormal time period, and determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0106] Specifically, since some anomalous signatures share similarities, these similarities can be used to detect persistent anomalous regions. This method can be applied to the post-filtering system of any existing log anomaly detector.

[0107] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0108] Step S201: Obtain the application's log sequence data, wherein the log sequence data is a structured log of the application's execution process arranged in chronological order within a predetermined time period.

[0109] Step S202: Extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0110] Step S203: Count the multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0111] Step S204: Based on the similarity of at least a number of the above-mentioned abnormal signatures, determine the continuous abnormal time period, and determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0112] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0113] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0114] Step S201: Obtain the application's log sequence data, wherein the log sequence data is a structured log of the application's execution process arranged in chronological order within a predetermined time period.

[0115] Step S202: Extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application.

[0116] Step S203: Count the multiple gold signals in different time periods to obtain multiple abnormal signatures. The abnormal signature is a unique identifier for whether the structured log has the gold signal.

[0117] Step S204: Based on the similarity of at least a number of the above-mentioned abnormal signatures, determine the continuous abnormal time period, and determine the above-mentioned log sequence data corresponding to the above-mentioned continuous abnormal time period as continuous abnormal log data.

[0118] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0127] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0128] 1) The log detection method of this application first acquires the log sequence data of the application; then, it extracts multiple golden signals from the log sequence data, which are used to characterize the performance of the application; next, it counts the multiple golden signals in different time periods to obtain multiple anomaly signatures, which are unique identifiers for whether the structured log has a golden signal; finally, it determines the continuous anomaly time period based at least on the similarity of the multiple anomaly signatures, and determines the log sequence data corresponding to the continuous anomaly time period as continuous anomaly log data. By counting and aggregating the golden signals of all logs within a predetermined time period, the influence of false positive anomalies in log anomalies can be suppressed, thereby improving the accuracy of log anomaly detection results and solving the problem of inaccurate log anomaly detection results in the prior art.

[0129] 2) The log detection apparatus of this application comprises: an acquisition unit acquiring log sequence data of an application; an extraction unit extracting multiple golden signals from the log sequence data, the golden signals being used to characterize the application's performance; a processing unit counting the multiple golden signals within different time periods to obtain multiple abnormal signatures, the abnormal signature being a unique identifier for whether a structured log has a golden signal; and a determination unit determining a continuous abnormal time period based at least on the similarity of the multiple abnormal signatures, and determining the log sequence data corresponding to the continuous abnormal time period as continuous abnormal log data. By counting and aggregating the golden signals of all logs within a predetermined time period, the influence of false positive anomalies in log anomalies can be suppressed, thereby improving the accuracy of log anomaly detection results and solving the problem of inaccurate log anomaly detection results in the prior art.

[0130] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A log detection method, characterized in that, include: Obtain application log sequence data, wherein the log sequence data is a structured log of the application execution process arranged in chronological order within a predetermined time period; Multiple gold signals are extracted from the log sequence data, wherein the gold signals are used to characterize the performance of the application; Multiple gold signals within different time periods are counted to obtain multiple abnormal signatures, and the abnormal signature is a unique identifier for whether the structured log contains the gold signal. Based at least on the similarity of multiple abnormal signatures, a continuous abnormal time period is determined, and the log sequence data corresponding to the continuous abnormal time period is determined to be continuous abnormal log data. The golden signals include one of a delay signal, a saturation signal, and an unavailability signal. The delay signal indicates that the application's response time to a user request exceeds a time threshold. The saturation signal indicates that the application's load exceeds a load threshold. The unavailability signal indicates that the application cannot provide normal service within a predetermined time period. Multiple golden signals are extracted from the log sequence data, including: Obtain basic seed words and synonyms of the basic seed words, wherein the seed words are used to characterize at least one of the following: delay, saturation, and unavailability of the application; Construct a rule dictionary based on the basic seed words and the synonyms; The log sequence data is segmented into words to obtain multiple words; Calculate the cosine distance between the word and the words in the rule dictionary to obtain the first cosine distance; If the first cosine distance is less than the first predetermined distance, the log sequence data is determined to be the golden signal; The gold signals include anomaly signals, which are used to characterize abnormal situations that occur during the operation of the application. Multiple gold signals are extracted from the log sequence data, including: Obtain a set of negative words, wherein the set of negative words includes multiple negative words, which are used to characterize negative sentiment towards the application; Calculate the intersection of the negative word set and the rule dictionary to obtain the rule intersection; A negative emotion dictionary is constructed based on the emotional polarity of the intersection of the aforementioned rules; The log sequence data is segmented into words to obtain multiple words; Calculate the cosine distance between the word and the words in the negative emotion dictionary to obtain the second cosine distance; If the second cosine distance is less than the second predetermined distance, the log sequence data is determined to be the golden signal.

2. The method according to claim 1, characterized in that, Obtain the application's log sequence data, including: Obtain the raw log information of the application, wherein the raw log information is unstructured free text information generated by the application during its operation; The Drain algorithm is used to transform the original log information to obtain structured log events; The log events are grouped using a sliding time window to obtain the log sequence data.

3. The method according to claim 1, characterized in that, The golden signals include error signals or unavailability signals. Error signals indicate that an error has occurred during application execution, and unavailability signals indicate that the application cannot provide normal service within a predetermined time period. Multiple golden signals are extracted from the log sequence data, including: Obtain a code regular expression, wherein the code regular expression is used to characterize the error code or the unusable code of the log sequence data; The log sequence data is segmented into words to obtain multiple words; By comparing the words with the regular expression, if the log sequence data satisfies the regular expression, the gold signal is determined to be in the log sequence data.

4. The method according to claim 1, characterized in that, Multiple gold signals within different time periods are counted to obtain multiple abnormal signatures, including: Obtain abnormal data from the log sequence data and determine the abnormal time period corresponding to the abnormal data, wherein the abnormal data is used to characterize abnormal behavior of the application. Obtain the number of log lines corresponding to the abnormal time period, wherein the number of log lines includes multiple log sequence data; The abnormal signature is obtained by counting the gold signals for the same number of log lines.

5. The method according to claim 1, characterized in that, The duration of the sustained anomaly is determined based on at least the similarity of multiple said anomaly signatures, including: The similarity of the multiple anomalous signatures is obtained by calculating one of the Jensen-Shannon divergence, Kourbach-Leibler divergence, and cosine distance in the vector space. If the similarity is greater than the similarity threshold, the time period corresponding to the multiple abnormal signatures is determined as the continuous abnormal time period.

6. A log detection device, characterized in that, include: The acquisition unit is used to acquire the log sequence data of the application, wherein the log sequence data is a structured log of the application execution process arranged in chronological order within a predetermined time period; An extraction unit is configured to extract multiple gold signals from the log sequence data, wherein the gold signals are used to characterize the performance of the application. The processing unit is used to count multiple gold signals within different time periods to obtain multiple abnormal signatures, wherein the abnormal signature is a unique identifier for whether the structured log contains the gold signal. The determining unit is configured to determine a continuous abnormal time period based at least on the similarity of multiple abnormal signatures, and to determine the log sequence data corresponding to the continuous abnormal time period as continuous abnormal log data. The golden signal includes one of a delay signal, a saturation signal, and an unavailable signal. The delay signal indicates that the application's response time to a user request is greater than a time threshold. The saturation signal indicates that the application's load is greater than a load threshold. The unavailable signal indicates that the application cannot provide normal service within a predetermined time period. The extraction unit further includes a third acquisition module, a first construction module, a fourth processing module, a first calculation module, and a second determination module. The third acquisition module is used to acquire basic seed words and synonyms of the basic seed words. The seed words are used to indicate that the application is experiencing at least one of delay, saturation, and unavailability. The first construction module is used to construct a rule dictionary based on the basic seed words and the synonyms. The fourth processing module is used to perform word segmentation on the log sequence data to obtain multiple words. The first calculation module is used to calculate the cosine distance between the words and the words in the rule dictionary to obtain a first cosine distance. The second determination module is used to determine that the log sequence data is the golden signal if the first cosine distance is less than a first predetermined distance. The golden signal includes an anomaly signal, which is used to characterize an abnormal situation during the operation of the application. The extraction unit includes a fourth acquisition module, a second calculation module, a second construction module, a fifth processing module, a third calculation module, and a third determination module. The fourth acquisition module is used to acquire a negative word set, wherein the negative word set includes multiple negative words, which are used to characterize negative sentiment towards the application. The second calculation module is used to calculate the intersection of the negative word set and the rule dictionary to obtain a rule intersection. The second construction module is used to construct a negative sentiment dictionary based on the sentiment polarity of the rule intersection. The fifth processing module is used to perform word segmentation on the log sequence data to obtain multiple words. The third calculation module is used to calculate the cosine distance between the words and the words in the negative sentiment dictionary to obtain a second cosine distance. The third determination module is used to determine the log sequence data as the golden signal if the second cosine distance is less than a second predetermined distance.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 5.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 5 through the computer program.

Images