Abnormal log detection method and device, storage medium and program product

Abstract

Embodiments of the invention provide an abnormal log detection method and device, a storage medium and a program product. The method comprises the steps of obtaining a first log template set of a first log set; determining potential abnormal log templates in the first log template set according to the first log template set and a second log template set, wherein the second log template set comprises a second log template constructed according to a stable second log set; determining a first log matched with the potential abnormal log template from the first log set, and screening a second log related to the semantics of the first log from the second log set; and performing semantic comparison on the first log and the second log through the first model, and determining whether the first log is an abnormal log or not according to a comparison result. By means of the method, efficient and real-time abnormal log detection can be achieved, calculation and storage expenses are remarkably reduced, the accuracy rate and recall rate of abnormal detection are improved, the method can adapt to different log formats and styles between different software systems and assemblies, and the operation and maintenance cost is greatly reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an abnormal log detection method, device, storage medium, and program product. Background Technology

[0002] Current software systems (such as large-scale distributed systems, cloud computing platforms, and microservice architectures) are increasingly large and complex. Their multi-component and multi-layered interactions make real-time monitoring and rapid understanding of their operational status difficult. In such systems, failures (such as performance bottlenecks, service interruptions, and security vulnerability exploits) are inevitable and have significant impacts. Timely detection and localization of these system anomalies are crucial for ensuring the reliability, availability, and security of services.

[0003] During operation, the system generates massive amounts of log data, which are essentially ordered, time-series records of system events, states, and operations. Log anomaly detection refers to the process of automatically analyzing the log data generated by the system to identify anomalies that significantly deviate from expected normal patterns (including event sequences, log content, frequency of occurrence, etc.) or indicate potential problems. Its goal is to discover problems before they cause significant impact.

[0004] However, existing abnormal log detection methods still face significant challenges in implementing efficient log anomaly detection in complex and ever-changing production environments, and have limitations in terms of accuracy, efficiency, and robustness. Summary of the Invention

[0005] This disclosure provides an abnormal log detection method, device, storage medium, and program product to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this disclosure provide an anomaly log detection method, including:

[0007] Obtain the first log set, and parse the first log set to obtain the first log template set;

[0008] Based on the first log template set and the second log template set, potential abnormal log templates in the first log template set are determined; wherein, the second log template set includes a second log template constructed based on a steady-state second log set;

[0009] From the first log set, determine the first log that matches the potential abnormal log template, and from the second log set, filter the second log that is semantically related to the first log;

[0010] The first model is used to perform a semantic comparison between the first log and the second log, and the first log is determined to be an abnormal log based on the comparison result.

[0011] Secondly, embodiments of this disclosure provide an anomaly log detection device, comprising:

[0012] The acquisition unit is used to acquire the first log set and parse the first log set to obtain the first log template set.

[0013] A potential abnormal log template determination unit is configured to determine potential abnormal log templates in the first log template set based on the first log template set and the second log template set; wherein, the second log template set includes a second log template constructed based on a steady-state second log set;

[0014] A semantically related log filtering unit is used to determine a first log that matches the potential abnormal log template from the first log set, and to filter a second log that is semantically related to the first log from the second log set;

[0015] An anomaly determination unit is used to perform a semantic comparison between the first log and the second log using a first model, and determine whether the first log is an anomaly log based on the comparison result. Semantic-related log filtering unit.

[0016] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the anomaly log detection method as described in the first aspect and various possible designs of the first aspect.

[0019] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the abnormal log detection method described in the first aspect and various possible designs of the first aspect.

[0020] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the anomaly log detection method as described in the first aspect and various possible designs of the first aspect.

[0021] The abnormal log detection method, device, storage medium, and program product provided in this disclosure involve: acquiring a first log set; parsing the first log set to obtain a first log template set; determining potential abnormal log templates in the first log template set based on the first log template set and a second log template set; wherein the second log template set includes a second log template constructed based on a steady-state second log set; determining a first log matching the potential abnormal log template from the first log set; filtering a second log semantically related to the first log from the second log set; performing a semantic comparison between the first log and the second log using a first model; and determining whether the first log is an abnormal log based on the comparison result. By first filtering out suspicious log templates at the log template level and then performing semantic judgment at the original log semantic level, efficient and real-time anomaly log detection is achieved, significantly reducing computational and storage overhead. Since the first model is used to semantically judge the first log matching the potential anomaly log template, the problems of information loss and false positives / false negatives caused by processing complex log semantics are overcome, improving the accuracy and recall of anomaly detection. Furthermore, since this embodiment does not limit specific log formats or have built-in rules binding specific log formats, it can adapt to the different log formats and styles between different software systems and components, avoiding model failure or rule maintenance burden caused by changes in log structure. It can effectively cope with the dynamic changes in log features (i.e., log drift) caused by the continuous deployment and evolution of software systems, ensuring the long-term effectiveness of the detection model. It eliminates the need for model retraining or cumbersome rule maintenance, achieving automated closed-loop operation and significantly reducing maintenance costs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A scenario example diagram illustrating an anomaly log detection method provided in an embodiment of this disclosure;

[0024] Figure 2 This is a schematic flowchart of an anomaly log detection method provided in an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic flowchart of an anomaly log detection method provided in another embodiment of the present disclosure;

[0026] Figure 4 This is a structural block diagram of an anomaly log detection device provided in an embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] During operation, the system generates massive amounts of log data, which are essentially ordered, time-series records of system events, states, and operations. As a direct reflection of internal system behavior, logs are characterized by rich data sources, broad coverage, and detailed recording, making them a valuable data source for monitoring and diagnosing system health. Log anomaly detection refers to the process of automatically analyzing system-generated log data to identify anomalies that significantly deviate from expected normal patterns (including event sequences, log content, and frequency of occurrence) or indicate potential problems. Its goal is to discover problems before they cause significant impact.

[0030] Existing methods for detecting log anomalies still face significant challenges in implementing efficient log anomaly detection in complex and ever-changing production environments, including:

[0031] The inconsistency and variability of log formats make it difficult for data-specific trained models to generalize and become universal.

[0032] The demands on processing performance from massive, high-speed log generation;

[0033] The evolution of system patterns leads to model aging and failure;

[0034] To address the aforementioned challenges, particularly the complexity and variability of log data, the high processing performance requirements, and the limitations of existing methods in terms of accuracy, efficiency, and robustness.

[0035] To address the aforementioned technical issues, this disclosure provides an anomaly log detection method. By first filtering out potential anomaly log templates at the log template level and then performing semantic judgment at the original log semantic level, it achieves efficient and real-time anomaly log detection, significantly reducing computational and storage overhead. It overcomes the information loss and false positive / false negative issues caused by processing complex log semantics, improving the accuracy and recall of anomaly detection. Furthermore, it does not have built-in rules binding to specific log formats, allowing it to adapt to the different log formats and styles of different software systems and components, avoiding model failure or rule maintenance burden caused by changes in log structure. It can effectively cope with the dynamic changes in log features (i.e., log drift) caused by the continuous deployment and evolution of software systems, ensuring the long-term effectiveness of the detection model. It eliminates the need for model retraining or cumbersome rule maintenance, achieving automated closed-loop operation and significantly reducing maintenance costs.

[0036] The application scenarios of the abnormal log detection method in this disclosure embodiment are as follows: Figure 1 As shown, a first log template set of a first log set can be obtained; potential abnormal log templates in the first log template set can be determined based on the first log template set and the second log template set, wherein the second log template set includes a second log template constructed based on the steady-state second log set; a first log matching the potential abnormal log template can be determined from the first log set, and a second log related to the semantics of the first log can be filtered from the second log set; the first log and the second log can be semantically compared using a first model, and whether the first log is an abnormal log can be determined based on the comparison result.

[0037] The activation of related functions, the acquisition of data, the processing and storage of data, etc., in the embodiments disclosed herein shall all require prior authorization from the user and other rights holders associated with the user, and shall comply with the relevant laws and regulations and the agreements between rights holders.

[0038] The abnormal log detection method of this disclosure will be described in detail below with reference to specific embodiments.

[0039] refer to Figure 2 , Figure 2 This is a schematic flowchart of an anomaly log detection method according to an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices or servers. The anomaly log detection method includes:

[0040] S201. Obtain the first log set and parse the first log set to obtain the first log template set.

[0041] In this embodiment, a set of logs to be detected is first obtained from the log storage system, denoted as the first log set. The first log set includes multiple logs to be detected. Considering that each log usually contains some dynamic parameters, and the dynamic parameters may differ between different logs, in order to reduce the complexity of the subsequent detection process and provide standardized input for the subsequent detection process, in this embodiment, a log template can be extracted from each log in the first log set to construct a first log template set. In this way, subsequent processing only needs to process these log templates to achieve the effect of processing the logs corresponding to these templates.

[0042] For any log extraction template, for example, the specific value of a dynamic parameter can be replaced with a wildcard (such as an asterisk " "). (etc.), such as the following example of a log:

[0043] mg_id=None legend_name = dbatman_err{"AccountId": "REDACTED" , "Region": "REDACTED",….."error":"client_error"}……timestamps=[REDACTED,REDACTED]

[0044] The extracted log template is:

[0045] mg_id=None legend_name = dbatman_err{"AccountId": " ", "Region": " ",….. "error": " "} ……timestamps=[ , ]

[0046] It should be noted that any feasible method can be used to extract log templates in this embodiment, and no limitation is imposed. For example, a log parsing tool (such as Drain log parsing tool) can be used to parse each log in the first log set into log templates and dynamic parameters according to the structured form of log templates and dynamic parameters. All the parsed log templates are then aggregated to construct the first log template set. Aggregation combines identical log templates into a single log template, achieving the effect of deduplication in the first log template set, making it a set of log templates without duplicate templates. This embodiment can efficiently and accurately extract log templates using a log parsing tool, without requiring large-scale training, and with relatively low resource consumption.

[0047] S202. Based on the first log template set and the second log template set, determine the potential abnormal log templates in the first log template set; wherein, the second log template set includes a second log template constructed based on the steady-state second log set.

[0048] In this embodiment, a steady-state second log set can be obtained. This second log set can be a set of historical logs within a selected historical baseline time window (such as the past 7 days), wherein there are no abnormal logs in the second log set (or abnormal logs have been filtered out). Log templates can be extracted from the second logs in the same manner as in S201 to construct a second log template set.

[0049] The second log template set is used as a benchmark. The first log template set is compared with the second log template set. If there are new log templates in the first log template set compared with the second log template set, the new log templates may be abnormal. These new log templates can be regarded as potential abnormal log templates, or these new log templates can be further filtered and regarded as potential abnormal log templates. Then, the subsequent detection process can only process these potential abnormal log templates, which can reduce the amount of data.

[0050] Optionally, when determining potential abnormal log templates in the first log template set based on the first and second log template sets, the difference between the first and second log template sets can be obtained. This yields the newly added log templates in the first log template set compared to the second log template set. For example, if the first log template set is S_current and the second log template set is S_historical, then the difference S_suspected = S_current - S_historical. The difference S_suspected includes the newly added log templates in the first log template set compared to the second log template set. Furthermore, potential abnormal log templates can be determined based on these newly added log templates. Of course, the method for determining newly added log templates is not limited to the above; other feasible methods can also be used.

[0051] Optionally, when determining potential abnormal log templates based on newly added log templates, these newly added log templates can be directly used as potential abnormal log templates; or, considering that some newly added log templates may be highly similar to the second log template and belong to variants of the second log template rather than entirely new log templates, this embodiment can further filter the newly added log templates based on the similarity between the newly added log templates and the second log templates.

[0052] Specifically, the similarity between any newly added log template and each of the second log templates in the second log template set can be obtained, and the maximum similarity can be determined. The maximum similarity is the similarity between the newly added log template and the most similar second log template. If the maximum similarity exceeds a preset threshold, it means that the newly added log template is highly similar to the second log template, and may be a variant of the second log template. Therefore, the newly added log template is not a potentially abnormal log template and can be removed. If the maximum similarity does not exceed the preset similarity threshold, the newly added log template is determined to be a potentially abnormal log template. Through the above filtering based on log template similarity, the remaining newly added log templates are recorded as potentially abnormal log templates.

[0053] The similarity between log templates can be determined by using general text embedding technology to convert the log templates into high-dimensional vector representations, and then calculating the cosine similarity between the high-dimensional vectors of the two log templates. Of course, other feasible methods can also be used to determine the similarity between log templates, and this embodiment does not impose any limitations.

[0054] S203. Determine the first log that matches the potential abnormal log template from the first log set, and filter the second log that is semantically related to the first log from the second log set.

[0055] In this embodiment, since the aforementioned steps only screen potential abnormal log templates at the log template level and ignore whether they are semantically the same or similar to the second log, this embodiment can make further semantic judgments after determining the potential abnormal log templates.

[0056] Specifically, for any potential abnormal log template, at least one original log corresponding to the potential abnormal log template can be selected from the first log set, that is, the first log matching the potential abnormal log template (the selection method can be random selection or other selection methods, where since all first logs matching a potential abnormal log template have the same structure and semantics, it is preferable to select only one first log). Further, it can be determined whether there is a second log with the same semantics in the second log set. However, considering that if all second logs in the second log set are semantically compared with the suspicious logs, the number of second logs will be too large, and the attention will be scattered and the effect will be poor; if the second logs used for semantic comparison are directly selected based on vector similarity, the cost of vectorizing all second logs will be too high, and the semantic loss problem from log to vector will occur, and the effect will not be ideal; if the second logs used for semantic comparison are selected based on manual rules, the robustness of automatic adaptation according to log characteristics cannot be achieved; therefore, in this embodiment, one or more keywords representing the core semantics are extracted from the first log, a query statement is constructed based on the keywords, and log retrieval is performed from the second log set, that is, the second logs including the keywords are retrieved, and the retrieved second logs are used as the second logs for semantic comparison, that is, as the second logs semantically related to the first log.

[0057] The keywords representing the core semantics are strings that can characterize the core semantic content of suspicious logs. These can be keyword fragments containing as many words as possible, including core content such as function names, error classes, protocol verbs, and component identifiers. They must be precise, continuous strings, excluding wildcards, regular expressions, quotation marks, or extra characters, and also excluding variable parameters (such as IDs, timestamps, IP addresses, numerical values, and status values). Of course, keywords representing the core semantics can also be keywords that meet other rules or conditions.

[0058] Extracting keywords representing core semantics from the first log can be done in any feasible way. In this embodiment, it can be done using a model, which can be a Large Language Model (LLM) or other deep learning models. In order to distinguish it from the first model in S204 below, the model used to extract keyword fragments representing core semantic content from suspicious logs in this embodiment is called the second model. Of course, the second model can be the same model as the first model or a different model.

[0059] More specifically, a first prompt instruction can be constructed to identify keywords representing core semantic content. The first prompt instruction guides the second model to extract keywords representing core semantics from the first log. This may include rules for the aforementioned keywords, which can guide the second model to focus on keyword fragments with fixed structures and representing core semantic content in the first log (including core content such as function names, error classes, protocol verbs, and component identifiers), while excluding variable parameters (such as ID, timestamp, IP address, numerical value, and status value).

[0060] After obtaining the keywords representing the core semantic content in the logs, a second log set can be queried based on these keywords. Specifically, a query statement can be constructed based on the keywords, conforming to the query specifications of the log storage system (such as AND logical connections). For example, the query statement can be constructed by connecting the keywords using AND logic: 'Keyword fragment 1' and 'Keyword fragment 2' and 'Keyword fragment 3'. Querying the second log set based on keywords representing the core semantics in suspicious logs avoids double semantic loss (original log → template → vector). The core semantics of the retrieved second logs are highly matched with those of the first logs, greatly improving the accuracy and efficiency of subsequent semantic comparisons. Furthermore, the query process is precise and focused, yielding fewer search results, and is fast, directly based on the log storage system without the need to build additional vector storage and retrieval systems.

[0061] Optionally, the second model can directly combine keywords into query statements according to the query specifications of the log storage system. This can generate query statements dynamically in real time without the need for manual rule setting, making the query statements naturally adaptable to different log formats. Of course, other means such as scripts can also be used to combine keyword fragments into query statements.

[0062] Furthermore, based on the query statement, second logs containing keywords can be queried from the second log set as candidate logs for the first log. These candidate logs can be directly used as second logs for semantic comparison with the first log. Optionally, in this embodiment, to further optimize the semantic comparison process, improve representativeness and comparability, and provide the most favorable comparison basis, the candidate logs of the first log can be further filtered. The similarity between the first log and each candidate log can be obtained, such as Jaccard similarity (similarity based on common tokens). The candidate logs with the highest similarity ranking (the top N candidate logs, N=3) are selected as second logs semantically related to the first log. Through this further filtering of candidate logs, the focus can be on second logs semantically related to the first log, improving representativeness and comparability. Furthermore, Jaccard similarity calculation is efficient and suitable for similarity comparisons between a small number of logs.

[0063] S204. Perform a semantic comparison between the first log and the second log using the first model, and determine whether the first log is an abnormal log based on the comparison result.

[0064] In this embodiment, after obtaining the second log which is semantically related to the first log, a first model can be used to perform a semantic comparison between the first and second logs to determine whether they have the same semantics. The first model can be a large language model or other deep learning models. The semantic comparison mainly relies on the semantic understanding capability of the first model, but it also needs to be combined with external prior knowledge, including but not limited to system knowledge (such as component functions, common error patterns, etc.) and expert second experience, to enhance the contextual understanding capability of the first model and reduce illusions. Optionally, prior knowledge can be accessed through an external knowledge base. The first log, the second log, and prior knowledge can be input into the first model, and the first model outputs a comparison result indicating whether the first and second logs have the same semantics.

[0065] Optionally, a second prompt can be constructed based on the first log, the second log, and prior knowledge for semantic comparison of the logs. This second prompt includes criteria for determining semantic similarity. For example, if the first log and any second log are highly similar in core events, context, and structure, differing only in variable parameters such as timestamps, IP addresses, usernames, and region names, or with minor differences in descriptive terminology, then the first log and the second log are semantically identical. Of course, the criteria are not limited to those listed above; other criteria capable of determining semantic similarity are also acceptable. Furthermore, the second prompt can be input into the first model to guide it in performing a semantic comparison between the first and second logs, outputting a comparison result indicating whether the first and second logs are semantically identical. Optionally, the first model can also output the reasoning behind the judgment for the semantic similarity comparison result.

[0066] If it is determined that the first log has the same semantics as any second log, it means that the first log is a variant of the second log. Therefore, it can be determined that the first log, as well as the corresponding potential exception log template and all the first logs corresponding to the potential exception log template, do not have any exceptions. If it is determined that the first log has different semantics from all the second logs, it is determined that the first log, as well as the corresponding potential exception log template and all the first logs corresponding to the potential exception log template, have exceptions.

[0067] The abnormal log detection method provided in this embodiment obtains a first log set and parses the first log set to obtain a first log template set; based on the first log template set and a second log template set, it determines potential abnormal log templates in the first log template set; wherein, the second log template set includes a second log template constructed based on a steady-state second log set; it determines a first log matching the potential abnormal log template from the first log set, and filters a second log semantically related to the first log from the second log set; it performs a semantic comparison between the first log and the second log using a first model, and determines whether the first log is an abnormal log based on the comparison result. By first filtering out potential abnormal log templates at the log template level and then performing semantic judgment at the original log semantic level, it achieves efficient and real-time abnormal log detection, significantly reducing computational and storage overhead. It overcomes the information loss and false positive / false negative issues caused by processing complex log semantics, improving the accuracy and recall of abnormal detection. Furthermore, it does not have built-in rules binding to specific log formats, allowing it to adapt to the different log formats and styles between different software systems and components, avoiding model failure or rule maintenance burden caused by changes in log structure. It can effectively cope with the dynamic changes in log features (i.e., log drift) caused by the continuous deployment and evolution of software systems, ensuring the long-term effectiveness of the detection model. It eliminates the need for model retraining or cumbersome rule maintenance, achieving automated closed-loop and significantly reducing operation and maintenance costs.

[0068] Based on any of the above embodiments, considering the inherent limitations (not perfect) of log parsing algorithms when extracting log templates, which may lead to semantically equivalent but template heterogeneous problems—that is, logs describing the same abnormal event may be parsed into multiple different log templates—the above-mentioned abnormal log detection methods may all identify them as abnormal, thus generating redundant alarms. Therefore, after determining that potential abnormal log templates and their corresponding first logs are abnormal, based on semantic equivalence, these log templates determined to be abnormal and their corresponding abnormal logs can be automatically merged to reduce the final number of alarm entries (alarm count) and improve user experience. Specifically, this can be achieved as follows: Figure 3 The above includes:

[0069] S301. The log templates and corresponding abnormal logs that are determined to be abnormal are taken as abnormal information. The fourth model is called to execute the fourth prompt instruction on the abnormal information to identify semantically similar abnormal information and perform merging and clustering. The fourth prompt instruction is used to instruct the fourth model to perform abnormal clustering based on the semantic content of the abnormal information.

[0070] S302. Log anomaly alerts are generated based on the anomaly information after merging and clustering.

[0071] In this embodiment, for any log template determined to be abnormal, at least one corresponding abnormal log can be selected. The potential abnormal log template and the at least one abnormal log are used to construct an abnormal message. Based on these abnormal messages, a fourth model can be used to identify whether the semantics are the same. Specifically, it can identify whether there are abnormal messages describing the same or similar events or root causes of problems. If so, they are grouped into a merging group, that is, these abnormal messages are identified as semantically identical abnormal messages. Then, the abnormal log templates and corresponding abnormal logs corresponding to the semantically identical abnormal messages are merged to reduce redundancy. Optionally, a representative log template can be selected for each merging group. The abnormal log template with the most corresponding first log entries can be selected, or other selection methods can be used. This embodiment does not limit this selection.

[0072] Optionally, in addition to outputting merged groups containing semantically identical anomaly information, the fourth model can also output the reasons for determining semantic similarity, i.e., the merging reasons. Furthermore, it can output a concise and semantically clear alarm view, avoiding alarm storms and greatly improving the diagnostic efficiency of operations and maintenance personnel.

[0073] Based on any of the above embodiments, when obtaining the first log set in S201, considering that some traditional anomaly log detection methods typically pull the full amount of logs from the log storage system, most of which are repetitive, high-frequency, and have no detection value, leading to exponential data expansion in subsequent parsing, filtering, and comparison processes; or that some traditional anomaly log detection methods introduce vectors for retrieval, but still require vectorizing massive amounts of logs at once, resulting in high resource consumption and noisy recall results, this embodiment performs a pre-emptive automatic retrieval process when obtaining the first log set, compressing the size of the logs to be detected and retaining only logs with detection value, thereby significantly reducing the computation and storage overhead of subsequent steps. The specific process may include:

[0074] The log retrieval statement is automatically constructed and iteratively optimized by the third model, triggering semantic noise reduction and compression of the original logs of the current window in the log storage system to obtain the first log set; wherein, at least one of the log retrieval statements is used to filter out log templates that appear repeatedly but have no practical meaning.

[0075] In this embodiment, a first set of raw logs can be randomly collected from the log storage system (the query statement used can be an asterisk " "). The first original log set includes a certain number of logs. The first original log set is input into the third model. Based on the third model, repetitive but meaningless log templates are extracted from the first original log set. The third model can be a large language model or other deep learning model. Relying on its context understanding ability, it can distinguish between templated noise and potential anomalies, and achieve semantic-level noise reduction. Templated noise refers to repetitive but meaningless log templates, such as a large number of logs like "INFO GC took 50 ms" and "DEBUG heartbeat ok", which are useless for troubleshooting. Potential anomalies are logs that appear very infrequently but are semantically obviously abnormal (such as "NullPointerException inPaymentService").

[0076] After extracting recurring but meaningless log templates, you can construct a query statement to filter out these recurring but meaningless log templates. For example, you can use NOT logic to filter out these recurring but meaningless log templates. As an example, a query statement could be: NOT "GCtook" AND NOT "heartbeat ok" AND NOT "mirror recv error".

[0077] Furthermore, based on the query statement, a second original log set is randomly collected from the log storage system, and then the second original log set is input into the third model. Based on the third model, recurring but meaningless log templates are extracted from the second original log set and updated into the query statement, so that the query statement becomes a query statement that filters out more recurring but meaningless log templates. For example, based on the above example, adding newly extracted recurring but meaningless log templates results in a new query statement: NOT "GC took" AND NOT "heartbeat ok" AND NOT "mirror recv error" AND NOT "EndpointIsInvalid".

[0078] Furthermore, a third set of original logs is randomly collected from the log storage system based on a new query statement. This process is iterated until a termination condition is met. The termination condition includes the number of iterations reaching a preset threshold or the number of logs in the original log set being less than a preset threshold. Finally, the original log set retrieved when the termination condition is met is determined as the first log set. This embodiment adopts an iterative, agentic paradigm, with a closed-loop process led by a third model: log sampling → feature analysis (repeated but meaningless log templates) → query refinement (constructing query statements) → resampling. This achieves semantic-level noise reduction and high compression, significantly reducing the data scale of subsequent processing, reducing computational and storage costs, and reducing load. Moreover, there are no manual rules throughout the process; the repeated but meaningless log templates and query statements are all generated by the third model, naturally adapting to log format evolution and exhibiting high robustness. Furthermore, the repeated but meaningless log templates and query statements in each iteration can be recorded, facilitating auditing and debugging and improving interpretability.

[0079] It should be noted that the first model, second model, third model, and fourth model used in the above embodiments can be the same model, different models, or some of them can be the same model. This embodiment does not impose any restrictions.

[0080] Corresponding to the anomaly log detection method in the above embodiment, Figure 4 This is a structural block diagram of an anomaly log detection device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The abnormal log detection device 400 includes:

[0081] Acquisition unit 401 is used to acquire the first log set and parse the first log set to obtain the first log template set;

[0082] The potential abnormal log template determination unit 402 is used to determine potential abnormal log templates in the first log template set based on the first log template set and the second log template set; wherein, the second log template set includes a second log template constructed based on the steady-state second log set;

[0083] The semantically related log filtering unit 403 is used to determine the first log that matches the potential abnormal log template from the first log set, and to filter the second log that is semantically related to the first log from the second log set;

[0084] Anomaly determination unit 404 is used to perform a semantic comparison between the first log and the second log using a first model, and determine whether the first log is an anomalous log based on the comparison result. Semantic-related log filtering unit.

[0085] In one or more embodiments of this disclosure, when the potential abnormal log template determining unit 402 determines the potential abnormal log template in the first log template set based on the first log template set and the second log template set, it is configured to:

[0086] Determine the difference between the first log template set and the second log template set. The difference includes newly added log templates in the first log template set relative to the second log template set, which serve as potential exception log templates.

[0087] In one or more embodiments of this disclosure, the potential anomaly log template determination unit 402 is further configured to:

[0088] The similarity between any newly added log template and each of the second log templates in the second log template set is determined. If the maximum similarity does not exceed a preset threshold, the newly added log template is then identified as the potential abnormal log template.

[0089] In one or more embodiments of this disclosure, when the semantically related log filtering unit 403 determines a first log matching the potential abnormal log template from the first log set and filters a second log semantically related to the first log from the second log set, it is configured to:

[0090] A first prompt instruction is constructed to identify keyword fragments for core semantic content, and a second model extracts keywords representing core semantics from the first log based on the first prompt instruction;

[0091] Query candidate logs containing the keywords from the second log set;

[0092] Determine the similarity between the first log and each of the candidate logs, and select the candidate log with the higher similarity ranking as the second log that is semantically related to the first log.

[0093] In one or more embodiments of this disclosure, the anomaly determination unit 404, when performing a semantic comparison between the first log and the second log using a first model, is configured to:

[0094] Construct a second prompt instruction for log semantic comparison, wherein the second prompt instruction includes a metric for log semantic similarity;

[0095] The first model performs a semantic comparison between the first log and the second log according to the second prompt instruction and outputs the comparison result, which indicates whether the first log and the second log have the same semantics.

[0096] In one or more embodiments of this disclosure, when the acquisition unit 401 acquires the first log set, it is used to:

[0097] The log retrieval statement is automatically constructed and iteratively optimized by the third model, triggering semantic noise reduction and compression of the original logs of the current window in the log storage system to obtain the first log set; wherein, at least one of the log retrieval statements is used to filter out log templates that appear repeatedly but have no practical meaning.

[0098] In one or more embodiments of this disclosure, the anomaly determination unit 404 is further configured to:

[0099] The log templates and corresponding abnormal logs that are determined to be abnormal are taken as abnormal information. The fourth model is invoked to execute the fourth prompt instruction on the abnormal information, identify abnormal information with similar semantics, merge and cluster them, and issue log abnormal alarms based on the merged and clustered abnormal information. The fourth prompt instruction is used to instruct the fourth model to perform abnormal clustering based on the semantic content of the abnormal information.

[0100] In one or more embodiments of this disclosure, the acquisition unit 401, when parsing the first log set to obtain the first log template set, is used to:

[0101] Based on the structured form of log templates and dynamic parameters, each log in the first log set is parsed, and all the parsed log templates are aggregated to generate the first log template set.

[0102] Semantic-related log filtering unit

[0103] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0104] To implement the above embodiments, this disclosure also provides an electronic device.

[0105] refer to Figure 5The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0106] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0107] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0108] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0109] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0110] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0111] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0112] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0115] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0116] The electronic device, computer-readable storage medium, and computer program product provided in this disclosure embodiment obtain a first log template set of a first log set; determine potential abnormal log templates in the first log template set according to the first log template set and a second log template set, wherein the second log template set includes a second log template constructed based on a steady-state second log set; determine a first log matching the potential abnormal log template from the first log set; filter second logs semantically related to the first log from the second log set; perform a semantic comparison between the first log and the second log using a first model; and determine whether the first log is an abnormal log based on the comparison result. By first filtering out suspicious log templates at the log template level and then performing semantic judgment at the original log semantic level, it achieves efficient and real-time anomaly log detection, significantly reducing computational and storage overhead. It overcomes the information loss and false positive / false negative issues caused by processing complex log semantics, improving the accuracy and recall of anomaly detection. Furthermore, it does not have built-in rules binding to specific log formats, allowing it to adapt to the different log formats and styles between different software systems and components, avoiding model failure or rule maintenance burden caused by changes in log structure. It can effectively cope with the dynamic changes in log features (i.e., log drift) caused by the continuous deployment and evolution of software systems, ensuring the long-term effectiveness of the detection model. It eliminates the need for model retraining or cumbersome rule maintenance, achieving automated closed-loop and significantly reducing operation and maintenance costs.

[0117] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0118] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0119] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An anomaly log detection method, characterized in that, include: Obtain the first log set, and parse the first log set to obtain the first log template set; Based on the first log template set and the second log template set, potential abnormal log templates in the first log template set are determined; wherein, the second log template set includes a second log template constructed based on a steady-state second log set; From the first log set, determine the first log that matches the potential abnormal log template, and from the second log set, filter the second log that is semantically related to the first log; The first model is used to perform a semantic comparison between the first log and the second log, and the first log is determined to be an abnormal log based on the comparison result.

2. The method according to claim 1, characterized in that, The step of determining potential abnormal log templates in the first log template set based on the first log template set and the second log template set includes: Determine the difference between the first log template set and the second log template set. The difference includes newly added log templates in the first log template set relative to the second log template set, which serve as potential exception log templates.

3. The method according to claim 2, characterized in that, The method further includes: Determine the similarity between any newly added log template and each of the second log templates in the second log template set. If the maximum similarity does not exceed a preset threshold, the newly added log template is identified as the potential abnormal log template.

4. The method according to claim 1, characterized in that, The step of determining the first log that matches the potential abnormal log template from the first log set, and filtering the second logs that are semantically related to the first log from the second log set, includes: A first prompt instruction is constructed to identify keywords for core semantic content, and a second model is used to extract keywords representing core semantics from the first log based on the first prompt instruction; Query candidate logs containing the keywords from the second log set; Determine the similarity between the first log and each of the candidate logs, and select the candidate log with the higher similarity ranking as the second log that is semantically related to the first log.

5. The method according to claim 1, characterized in that, The semantic comparison of the first log and the second log using the first model includes: Construct a second prompt instruction for log semantic comparison, wherein the second prompt instruction includes a metric for log semantic similarity; The first model performs a semantic comparison between the first log and the second log according to the second prompt instruction and outputs the comparison result, which indicates whether the first log and the second log have the same semantics.

6. The method according to any one of claims 1-5, characterized in that, The process of obtaining the first log set includes: The log retrieval statement is automatically constructed and iteratively optimized by the third model, triggering semantic noise reduction and compression of the original logs of the current window in the log storage system to obtain the first log set; wherein, at least one of the log retrieval statements is used to filter out log templates that appear repeatedly but have no practical meaning.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: The log templates and corresponding abnormal logs that are determined to be abnormal are taken as abnormal information. The fourth model is called to execute the fourth prompt instruction for the abnormal information, identify abnormal information with similar semantics, merge and cluster them, and issue log abnormal alarms based on the merged and clustered abnormal information. The fourth prompt instruction is used to instruct the fourth model to perform abnormal clustering based on the semantic content of the abnormal information.

8. The method according to any one of claims 1-5, characterized in that, The parsing of the first log set yields a first log template set, including: Based on the structured form of log templates and dynamic parameters, each log in the first log set is parsed, and all the parsed log templates are aggregated to generate the first log template set.

9. An anomaly log detection device, characterized in that, include: The acquisition unit is used to acquire the first log set and parse the first log set to obtain the first log template set. A potential abnormal log template determination unit is configured to determine potential abnormal log templates in the first log template set based on the first log template set and the second log template set; wherein, the second log template set includes a second log template constructed based on a steady-state second log set; A semantically related log filtering unit is used to determine a first log that matches the potential abnormal log template from the first log set, and to filter a second log that is semantically related to the first log from the second log set; An anomaly determination unit is used to perform a semantic comparison between the first log and the second log using a first model, and determine whether the first log is an anomaly log based on the comparison result.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images