A log failure detection method and related apparatus

Abstract

The embodiment of the application provides a log fault detection method and related device, through initial detection of log data based on a lightweight binary classification model, a first detection result is obtained; when the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain keywords of the log data; the keywords are matched with fault classification nodes in a knowledge base to determine fault classification nodes corresponding to the keywords; the keywords are matched with fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keywords; and a second detection result is determined based on a matching degree of the at least one fault semantic intent node and the keywords. In the embodiment of the application, the log data is initially detected based on the lightweight binary classification model, the depth explainability of the detection result is improved in combination with the knowledge base, and the accuracy of log fault detection is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log fault detection method and related apparatus. Background Technology

[0002] A log is a collection of event information automatically recorded by a system, application, or device during operation. It records critical information such as operational status, errors, warnings, and performance data. It serves as a data source for operations, development, and security teams to analyze system behavior, troubleshoot problems, test performance, and detect faults.

[0003] Currently, log fault detection relies on rule-based matching mechanisms (such as regular expression matching and keyword alerting rules) and statistical analysis. However, given the complexity and diversity of log formats and the ambiguity of fault information semantics, current detection methods suffer from low accuracy.

[0004] Therefore, there is an urgent need for a solution to address the aforementioned technical problems. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a log fault detection method and related apparatus.

[0006] The embodiments of this application disclose the following technical solutions: Firstly, embodiments of this application provide a log fault detection method, including: A lightweight binary classification model is used to perform initial detection on log data and obtain the first detection result. The lightweight binary classification model is trained with historical log data as input and log level as the output target. When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain the keywords of the log data; The keywords are matched with fault classification nodes in the knowledge base to determine the fault classification nodes corresponding to the keywords; Match the keywords with the fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keywords; Based on the matching degree between the at least one fault semantic intent node and the keyword, a second detection result of the log data is determined.

[0007] In one possible implementation, when the first detection result indicates a fault in the log data, keyword extraction is performed on the log data to obtain the keywords of the log data, including: When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain multiple initial keywords; Based on the term frequency and inverse document frequency of the plurality of initial keywords in the log data, a first number of initial keywords are determined from the plurality of initial keywords, and the first number of initial keywords are determined as the keywords of the log data.

[0008] In one possible implementation, the first detection result includes the service name corresponding to the log data, and the step of matching the keyword with fault classification nodes in the knowledge base to determine the fault classification node corresponding to the keyword includes: Based on the service name corresponding to the log data, the fault classification nodes in the knowledge base are filtered to obtain fault classification nodes that match the service name corresponding to the log data. The keyword is matched with the fault classification node that matches the service name corresponding to the log data to determine the fault classification node corresponding to the keyword.

[0009] In one possible implementation, determining the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword includes: Based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, the matching degree between the at least one fault semantic intent node and the keyword is determined; When the matching degree is greater than a preset threshold, a second detection result of the log data is generated based on the at least one fault semantic intent node.

[0010] In one possible implementation, after determining the matching degree between the at least one fault semantic intent node and the keyword based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, the method further includes: When the matching degree is less than or equal to a preset threshold, fault detection is performed on the log data based on the keyword, a second detection result of the log data is generated, and the knowledge base is updated based on the second detection result.

[0011] In one possible implementation, determining the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword includes: Based on the semantic information in the at least one fault semantic intent node and the semantic information in the log data, determine the vector similarity between the log data and the at least one fault semantic intent node; Based on the matching degree and the vector similarity, a second detection result for the log data is determined.

[0012] In one possible implementation, the steps for constructing the knowledge base include: Based on key fields extracted from historical log data, the historical log data is annotated to obtain annotated historical log data. Based on the timestamp and format characteristics of the annotated historical log data, the annotated historical log data is merged to obtain merged historical log data. Fault detection is performed on the merged historical log data to generate log fault information; The log fault information is transformed using a graph database engine to generate a graph-structured knowledge base.

[0013] Second aspect: This application provides a log fault detection device, including: The system comprises a detection unit, an extraction unit, a first matching unit, a second matching unit, and a determination unit. The detection unit is used to perform initial detection on log data based on a lightweight binary classification model to obtain a first detection result; the lightweight binary classification model is trained with historical log data as input and log level as the output target. The extraction unit is used to extract keywords from the log data when the first detection result indicates that the log data is faulty, thereby obtaining the keywords of the log data. The first matching unit is used to match the keyword with the fault classification node in the knowledge base to determine the fault classification node corresponding to the keyword; The second matching unit is used to match the keyword with the fault semantic intent node under the fault classification node in the knowledge base, and determine at least one fault semantic intent node corresponding to the keyword. The determining unit is used to determine a second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword.

[0014] Third aspect: This application provides a computer device, which includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of a log fault detection method as described above, according to the instructions in the program code.

[0015] Fourth aspect: This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a log fault detection method as described above.

[0016] Fifth aspect: This application provides a computer program product, which, when run on a computer, executes the steps of a log fault detection method as described above.

[0017] Sixth aspect: This application provides a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, so that the chip implements a log fault detection method as described above.

[0018] Compared with the prior art, the embodiments of this application have the following beneficial effects: This application provides a log fault detection method and related apparatus. It performs initial detection on log data based on a lightweight binary classification model to obtain a first detection result. When the first detection result indicates a fault in the log data, keywords are extracted from the log data to obtain keywords. The keywords are matched with fault classification nodes in a knowledge base to determine the fault classification node corresponding to the keyword. The keywords are then matched with fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keyword. Based on the matching degree between the at least one fault semantic intent node and the keyword, a second detection result for the log data is determined. This application employs a hierarchical detection method, performing initial detection on log data based on a lightweight binary classification model, and combining a knowledge base including fault classification nodes and fault semantic intent nodes to determine the deep interpretability of the detection results, effectively improving the accuracy of log fault detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of a log fault detection method provided in an embodiment of this application; Figure 2 A flowchart of a log fault detection method provided in an embodiment of this application; Figure 3A schematic diagram of a knowledge base structure provided in an embodiment of this application; Figure 4 A schematic diagram of sample knowledge base data provided in an embodiment of this application; Figure 5 A schematic diagram of a log fault detection method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a log fault detection device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0024] Currently, log fault detection relies on rule-based matching mechanisms (such as regular expression matching and keyword alerting rules) and statistical analysis. However, given the complexity and diversity of log formats and the ambiguity of fault information semantics, current detection methods suffer from low accuracy.

[0025] Based on this, embodiments of this application provide a log fault detection method and related apparatus. The method involves initial detection of log data using a lightweight binary classification model to obtain a first detection result. When the first detection result indicates a fault in the log data, keyword extraction is performed on the log data to obtain keywords. These keywords are then matched with fault classification nodes in a knowledge base to determine the fault classification node corresponding to the keyword. The keywords are further matched with fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keyword. Finally, a second detection result for the log data is determined based on the matching degree between the at least one fault semantic intent node and the keyword.

[0026] This application employs a hierarchical detection method, which performs initial detection on log data based on a lightweight binary classification model. It combines a knowledge base including fault classification nodes and fault semantic intent nodes to determine the deep interpretability of the detection results, effectively improving the accuracy of log fault detection.

[0027] like Figure 1 As shown in the figure, this figure is a schematic diagram of an application scenario of a log fault detection method provided in an embodiment of this application. The log fault detection method can be implemented based on the interaction between the client 101 and the server 102.

[0028] It is understood that the deployment method of client 101 and server 102 is not specifically limited in this embodiment. For example, client 101 and server 102 can be deployed independently or jointly.

[0029] In this embodiment, client 101 sends log data to server 102. After receiving the log data, server 102 can perform initial detection on the log data based on a lightweight binary classification model to obtain a first detection result. The lightweight binary classification model is trained using historical log data as input and log level as the output target. When the first detection result indicates a fault in the log data, keywords are extracted from the log data to obtain keywords. The keywords are matched with fault classification nodes in the knowledge base to determine the fault classification node corresponding to the keywords. The keywords are matched with fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keywords. Based on the matching degree between the at least one fault semantic intent node and the keywords, a second detection result for the log data is determined.

[0030] The following is combined with Figure 2 This application introduces a log fault detection method provided by an embodiment, such as... Figure 2 As shown, this figure is a flowchart of a log fault detection method provided in an embodiment of this application, including S201-S205.

[0031] S201. Based on a lightweight binary classification model, perform initial detection on the log data to obtain the first detection result.

[0032] The lightweight binary classification model is trained using historical log data as input and log level as the output target.

[0033] For example, this lightweight binary classification model can be obtained by fine-tuning a pre-trained language model based on the DistilBERT architecture, and used to perform initial detection on log data to determine whether the log data is faulty.

[0034] DistilBERT is a lightweight variant of Bidirectional Encoder Representations from Transformers (BERT). It compresses model size through knowledge distillation, aiming to significantly reduce computational resource consumption while preserving the core performance of BERT.

[0035] This lightweight binary classification model, obtained by fine-tuning a pre-trained language model based on the DistilBERT architecture, can significantly reduce inference latency and resource consumption while maintaining high semantic understanding capabilities, making it suitable for real-time detection scenarios with high throughput and low latency.

[0036] In this embodiment, the lightweight binary classification model retains the multi-head self-attention mechanism and feedforward neural network structure of the standard Transformer, supports deep semantic modeling of input text, and is suitable for semantic classification tasks of log data content.

[0037] The training data for this lightweight binary classification model is historical log data. Based on equation (1), the historical log data can be processed as shown in equation (1) below: (1) in, Indicates the service name entity; Enumerated values ​​representing log levels, such as errors (ERROR), warnings (WARN), etc. [SERVICE], [LEVEL], and [MSG] represent learnable special tags used to guide the model to focus on different semantic domains.

[0038] In the process of training the lightweight binary classification model, the method provided in this application embodiment truncates the input historical log data based on the DistilBERT tokenizer to ensure that its length does not exceed 512 tokens, and simultaneously generates the corresponding numeric encoding sequence (input_ids) and attention mask (attention_mask).

[0039] Based on this, the processed historical log data is input into the pre-trained model, and deep semantic features of the historical log data are extracted through a multi-layer Transformer structure. The output feature information includes the hidden state of context information or the global representation after pooling. Then, through a fully connected classification layer, the extracted feature information is mapped to the category space corresponding to the target task.

[0040] During the training phase, the cross-entropy loss function is used to calculate the deviation between the predicted results and the true labels, and the AdamW optimizer is used to update the parameters. The initial learning rate can be set to 2. -5 Set the eps parameter to 1 -8 To prevent overfitting, an early stopping mechanism can be introduced during training. Training is terminated when the F1 score on the validation set fails to improve for three consecutive rounds, resulting in a lightweight binary classification model with optimized performance.

[0041] After obtaining the lightweight binary classification model, the log data can be initially detected based on the lightweight binary classification model in this embodiment of the application to obtain a first detection result, which can indicate whether the log data is normal or the log data is faulty.

[0042] In one possible implementation, to balance the real-time performance of detection with the system throughput, this embodiment of the application may employ a small-batch processing strategy based on a time sliding window to group and aggregate multiple log data before performing initial detection on the log data based on the lightweight binary classification model, thereby obtaining the grouped and aggregated log data.

[0043] For example, with the window length set to 10 seconds, a continuous data stream can be divided into discrete "time blocks". For instance, log data in the range [T, T+10] can be used as the first group of log data, log data in the range [T+11, T+20] can be used as the second group of log data, and so on, to obtain multiple groups of log data.

[0044] For each group of log data, the log data is aggregated based on resource identifier fields of service instances, such as service name (service_name) and Pod IP address (pod_ip), to obtain grouped and aggregated log data.

[0045] After the grouped and aggregated log data is processed into structured log data, the lightweight binary classification model is used to perform initial detection on the structured log data, and the first detection result can be obtained.

[0046] When the first detection result indicates that the log data is normal, a "normal" flag can be returned and the detection process can be terminated; when the first detection result indicates that the log data is faulty, S202 can be executed.

[0047] S202. When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain the keywords of the log data.

[0048] In one possible implementation, when the first detection result indicates a fault in the log data, keyword extraction is performed on the log data to obtain multiple initial keywords; based on the term frequency (TF) and inverse document frequency (IDF) of the multiple initial keywords in the log data, a first number of initial keywords are determined from the multiple initial keywords, and the first number of initial keywords are determined as the keywords of the log data.

[0049] For example, in the embodiments of this application, an improved TextRank-TF-IDF fusion algorithm can be used as a lightweight keyword extraction model. This paper constructs a word co-occurrence graph by statistically analyzing the co-occurrence relationships of words in log data. Based on this, the TextRank algorithm is used to iteratively calculate the word co-occurrence graph, generating word weights based on the graph structure. The TF-IDF algorithm is then used to calculate the statistical importance of multiple initial keywords in the log data, obtaining the first weight corresponding to word frequency and the second weight corresponding to inverse document frequency. The word frequency and inverse document frequency are then weighted and fused to obtain a comprehensive score for the multiple initial keywords. These initial keywords are then sorted in descending order of their comprehensive scores, and the top few initial keywords are extracted, achieving high-precision log keyword extraction with low resource consumption.

[0050] It is understood that the first quantity is not specifically limited in the embodiments of this application, and can be set according to actual needs. For example, the first quantity can be 3, 4, 5, etc.

[0051] S203. Match the keywords with the fault classification nodes in the knowledge base to determine the fault classification node corresponding to the keywords.

[0052] Among them, the fault classification nodes in the knowledge base are used to represent the hierarchy of each type in the classification tree.

[0053] In one possible implementation, the knowledge base construction steps include: annotating the historical log data based on key fields extracted from historical log data to obtain annotated historical log data.

[0054] For example, after loading historical log data, key fields in the historical log data can be identified and extracted based on regular expressions, and the plain text log data can be parsed into structured log data.

[0055] This key field may include, but is not limited to, timestamp, service name, instance IP, thread name, log level, class name, and log text.

[0056] By using key fields extracted from historical log data, the type of log text within the log data can be determined during the annotation process. The log text type can include static text and dynamic variables.

[0057] Based on the type of log text in the log data, the historical log data can be annotated to obtain annotated historical log data.

[0058] For example, based on the fine-tuned DistilBERT semantic labeling model, dynamic variables (such as time, service name, etc.) in log data can be automatically identified and labeled with predefined semantic tags (such as...). <duration>,<SERVICE_NAME> The annotation of historical log data is generated by taking the values ​​of the log data and using them as parameters. This annotated historical log data can be a structured parameterized log intent representation, thus preserving the semantic integrity of the log data while maintaining good machine readability, providing a structured foundation for subsequent processing.

[0059] After obtaining the labeled historical log data, the labeled historical log data can be merged based on the timestamp and format characteristics of the labeled historical log data to obtain the merged historical log data.

[0060] For example, the format features include stack features, and the merged historical log data corresponding to the stack features includes stack trace events. This application embodiment employs a multi-line log merging strategy based on timestamps and format features. The line in the annotated historical log data that begins with a timestamp is identified as an event header, serving as the starting marker for the stack trace event. Starting from the line containing the event header, the format features of subsequent lines are continuously detected. If a line begins with a stack feature (such as "at", "Causedby", or a specific indentation), it is determined to be a stack line, and it is merged with the event header to generate a complete stack trace event, supporting subsequent efficient fault detection and analysis processes.

[0061] Specific indentation may include, but is not limited to, a leading space.

[0062] In one possible implementation, before merging the labeled historical log data to obtain the merged historical log data, the labeled historical log data can be deduplicated to obtain the deduplicated historical log data.

[0063] For example, in order to eliminate duplicate entries in the labeled historical log data, reduce redundant calculations and memory overhead, and improve data processing efficiency, a time window-based deduplication mechanism is introduced in this application embodiment.

[0064] In this embodiment, the service_name, pod_ip, and log text are combined to generate an MD5 hash value, which is then used as the composite key. By retaining only the first log entry for the same composite key within a sliding window of a preset time (e.g., 5 minutes) and recording its cumulative occurrence count, the number of log entries can be significantly reduced, log storm interference can be suppressed, and the performance of computing resource allocation and processing links can be optimized.

[0065] After obtaining the deduplicated historical log data, the deduplicated historical log data can be merged to obtain the merged historical log data.

[0066] It is understandable that the process of merging the deduplicated historical log data is the same as the process of merging the labeled historical log data described above, and will not be repeated here.

[0067] After obtaining the merged historical log data, fault detection is performed on the merged historical log data to generate log fault information.

[0068] For example, in this embodiment of the application, the merged historical log data can be batch input into the Chain-of-Thought (CoT) large model. Through preset prompts, the CoT large model is guided to perform chain-like reasoning and output log fault information.

[0069] The batch log size for inputting a large CoT model can be set based on the context length of the large language model. For example, with a context window length of 128K tokens, the batch log size can be 64K.

[0070] After obtaining the log fault information, the log fault information can be transformed based on the graph database engine to generate a graph-structured knowledge base.

[0071] The graph database engine can include, but is not limited to, the Neo4j graph database engine, used to transform unstructured log fault information into a graph-structured knowledge base with explicit semantic relationships and vectorized representation. This generated knowledge base, by defining node types, attributes, and relationships between nodes, enables hierarchical organization and storage of fault knowledge.

[0072] For example, the knowledge base contains three node types: Fault Type, Semantic Intent, and Log Example. The attribute definitions for each node are shown in Table 1. Table 1

[0073] In this application's embodiments, the relationships between nodes are connected through directed relationships to express specific semantics. These relationships include, but are not limited to, subclass relationships (SUBCLASS_OF), instance relationships (INSTANCE_OF), and sample relationships (EXEMPLIFIED_BY).

[0074] For example, subclass relationships are used to construct a multi-level taxonomy of fault types. The start and end points of a subclass relationship are pointed to by the fault category node of the child level and the fault category node of the parent level. For example, the first-level category "Hystrix Fault" can be the parent class of the second-level category "Timeout Config Conflict".

[0075] Instance relationships are used to indicate that a specific fault semantic instance belongs to a particular fault category. The start and end points of an instance relationship are the fault semantic intent node pointing to the most specific (usually a second-level) fault category node.

[0076] An instance relation is used to represent one or more specific log entries as empirical examples of a certain fault semantic intent. The start and end points of the instance relation are pointed to by the fault semantic intent node and one or more fault log instance nodes.

[0077] like Figures 3-4 As shown, these are schematic diagrams of a knowledge base structure and sample data of a knowledge base provided in an embodiment of this application, respectively.

[0078] In this embodiment, fault classification nodes at all levels can be created based on the fault classification system, and these fault classification nodes can be interconnected based on the subclass relationship to construct a complete fault classification tree.

[0079] Based on this, for each specific fault case, a corresponding fault semantic intent node is created, and it is associated with the lowest-level fault classification node in the fault classification tree through instance relationships. Then, each relevant log in the case is created as a fault log example node, and the fault log example node is connected to its corresponding fault semantic intent node using example relationships.

[0080] In terms of semantic processing, the pre-trained lightweight semantic embedding model bge-m3 was selected to perform vectorization calculations on the fault_intent field of each fault semantic intent node and the relevant_log field of each fault log sample node, generating 1024-dimensional floating-point vectors, and storing these floating-point vectors in the semantic_vector attribute of the corresponding node.

[0081] By persistently storing the nodes created above, the attributes contained in the nodes, and the relationships between the nodes into the Neo4j graph database, a knowledge base with a graph structure can be generated.

[0082] Therefore, by matching the keywords with fault classification nodes in the knowledge base, the fault classification node corresponding to the keywords can be determined.

[0083] S204. Match the keyword with the fault semantic intent node under the fault classification node in the knowledge base to determine at least one fault semantic intent node corresponding to the keyword.

[0084] In this embodiment of the application, based on the service name corresponding to the log data, the fault classification nodes in the knowledge base are filtered to obtain the fault classification nodes that match the service name corresponding to the log data; by matching the keyword with the fault classification node that matches the service name corresponding to the log data, the fault classification node corresponding to the keyword can be determined.

[0085] For example, in combination Figure 3 The knowledge base structure diagram shown is introduced below. In this embodiment, the fault classification nodes in the knowledge base can be initially screened based on service_name. The extracted keywords are fuzzily matched with the fault_type field of the screened fault classification nodes to determine the fault classification node corresponding to the keywords.

[0086] S205. Based on the matching degree between the at least one fault semantic intent node and the keyword, determine the second detection result of the log data.

[0087] In one possible implementation, the matching degree between the at least one fault semantic intent node and the keyword can be determined based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data.

[0088] For example, the process of determining the matching degree between the at least one fault semantic intent node and the keyword based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, is as shown in Equation (2): (2) in, Indicates the keyword matching degree; T represents log data; This indicates at least one fault semantic intent node corresponding to the keyword; This indicates the number of keywords in the log data; Represents edit distance similarity; f takes values ​​from K is the log intent; K is a set representing the set of all keywords extracted from log data T; k represents each keyword in log data T.

[0089] When the matching degree is greater than a preset threshold, a second detection result of the log data is generated based on the at least one fault semantic intent node.

[0090] In one possible implementation, during the process of determining the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword, the vector similarity between the log data and the at least one fault semantic intent node can be determined based on the semantic information in the at least one fault semantic intent node and the semantic information of the log data; the second detection result of the log data is determined based on the matching degree and the vector similarity.

[0091] For example, in this embodiment of the application, to improve the retrieval efficiency and real-time performance of the knowledge base, an Approximate Nearest Neighbor (ANN) technique is introduced to optimize the similarity matching process of semantic vectors. For each fault semantic intent node corresponding to the keyword, the cosine similarity between the semantic information of its real-time log data and the semantic information in the fault semantic intent nodes stored in the knowledge base is calculated using a lightweight embedding model.

[0092] The semantic information in the fault semantic intent node can be determined based on the semantic_vector in the fault log sample node under that fault semantic intent node. The semantic_vector in the fault log sample node is the semantic vector of the original log text content.

[0093] Lightweight embedding model used (BGE-M3) generates a vector representation of the semantic information of real-time log data (T), i.e., a semantic vector. As shown in equation (3): (3) Semantic information in fault semantic intent nodes ( The cosine similarity between the semantic information of the log data and the log data is shown in Equation (4) as follows: (4) After determining the vector similarity between the log data and the at least one fault semantic intent node, a second detection result of the log data can be determined based on the matching degree and the vector similarity.

[0094] For example, based on the matching degree and the vector similarity, a weighted score can be determined as shown in equation (5): (5) in, This represents the weighted score. The keyword matching degree can be calculated according to formula (2); It is the semantic vector similarity, which can be calculated according to formula (4); α, , These are the first weight, second weight, and third weight, respectively, α+ =1.

[0095] It is understood that, in the embodiments of this application, α, , Furthermore, the value of the preset threshold is not specifically limited and can be set according to actual needs. For example: α=0.3, β=0.4, γ=0.3, and the preset threshold is 0.8.

[0096] When the weighted score is greater than a preset threshold, a second detection result of the log data is generated. This second detection result may include information such as fault_type, fault_title, fault_intent, solutions, and impact_scope.

[0097] When the matching degree is less than or equal to a preset threshold, in this embodiment of the application, fault detection can be performed on the log data based on the keywords, a second detection result of the log data can be generated, and the knowledge base can be updated based on the second detection result.

[0098] For example, in this embodiment of the application, if no matching abnormal log data is found in the knowledge base, an "unknown fault" detection process can be initiated. By marking the log data as a "new anomaly," and based on the processing method of historical log data during the knowledge base construction process, the log data marked as "new anomaly" is processed, and the processed log data is updated in the knowledge base, thereby realizing the continuous accumulation and updating of knowledge.

[0099] In summary, the embodiments of this application adopt a hierarchical detection method, which performs initial detection on log data based on a lightweight binary classification model, and combines a knowledge base including fault classification nodes and fault semantic intent nodes to determine the deep interpretability of the detection results, thereby effectively improving the accuracy of log fault detection.

[0100] To facilitate understanding, the following will be combined with... Figure 5 This application provides an overall overview of a log fault detection method provided in its embodiments.

[0101] In this embodiment, the offline training module for historical log data and the fault detection module for real-time log data are connected through a knowledge base that enhances semantics using a multi-level fault graph.

[0102] The offline training module for historical log data is used to mine faults from historical log data, build and continuously update a knowledge base with enhanced semantics of multi-level fault graphs. The construction steps of this knowledge base are as described in S203 above, and will not be repeated here.

[0103] The real-time log data fault detection module adopts a four-layer architecture, including lightweight model initial screening, multi-level knowledge base retrieval, automatic analysis of unknown faults, and intelligent caching acceleration. Combined with a knowledge base that enhances semantics through multi-level fault graphs and large model inference capabilities, it can process real-time incoming log data, achieving efficient and continuously evolving fault detection. Figure 5 As shown in the figure, this is a schematic diagram of a log fault detection method provided in an embodiment of this application, including: S501. Perform small-batch aggregation and structured parsing on real-time log data to obtain structured log data.

[0104] In this embodiment of the application, a small-batch processing strategy based on a time sliding window can be used to group and aggregate multiple log data to obtain grouped and aggregated log data.

[0105] For each group of log data, the log data is aggregated based on resource identifier fields of service instances, such as service name (service_name) and Pod IP address (pod_ip), to obtain grouped and aggregated log data.

[0106] The grouped and aggregated log data is processed into structured log data.

[0107] S502, intelligent log data deduplication and caching acceleration.

[0108] In this embodiment, efficient log data detection can be achieved through content fingerprinting and cache optimization mechanisms based on the intelligent log data deduplication and caching acceleration engine (LogDDR).

[0109] For example, a unique MD5 content fingerprint can be generated based on the service_name, pod_ip, and message in the logs, enabling accurate identification of duplicate logs.

[0110] In this embodiment, a multi-dimensional cache management strategy is adopted. For each cached item, the creation time, last access time, and number of hits are recorded. Low-frequency items with a creation time of more than 24 hours or a number of hits of less than 3 are prioritized for elimination to ensure the effectiveness of the cache.

[0111] Meanwhile, the intelligent log data deduplication and caching acceleration engine provided in this application embodiment can support a dynamic cache update mechanism. When the knowledge base changes due to an "unknown fault" closed-loop process, it can automatically trigger the relevant cache invalidation and refresh, and release cache space through periodic self-cleaning tasks. Based on this mechanism, the overhead caused by duplicate detection calculations can be effectively reduced, and the system throughput and response efficiency can be improved.

[0112] S503, Determine if the cache was hit.

[0113] If the fingerprint hits a valid cache, execute S509, return or cache the detection result to avoid repetitive small model reasoning and knowledge base recall processes.

[0114] If the fingerprint does not hit a valid cache, then execute S504.

[0115] S504. Initial detection of structured log data based on a lightweight binary classification model.

[0116] The process of initial detection of structured log data based on a lightweight binary classification model is described in section S201 above and will not be repeated here.

[0117] S505. Determine whether the structured log data is abnormal log data.

[0118] When the structured log data is determined to be abnormal, the deep fault detection process can be initiated and S506 can be executed; when the structured log data is determined to be normal, the "normal" flag can be returned and S509 can be executed to end the detection process.

[0119] S506, Recall multi-level fault diagrams to enhance semantic knowledge base.

[0120] In this embodiment of the application, fault detection can be performed on log data based on a pre-built knowledge base. The process is the same as S203-S205 above, and will not be repeated here.

[0121] S507. Determine whether the recall was successful.

[0122] In this embodiment, a weighted score can be obtained based on the matching degree and vector similarity. If the weighted score is greater than a preset threshold, the recall is determined to be successful, and step S509 is executed; if the weighted score is less than or equal to the preset threshold, step S508 is executed.

[0123] S508, Handle unknown faults and update the knowledge base.

[0124] In this embodiment of the application, if no matching abnormal log data can be found in the knowledge base, the "unknown fault" detection process can be initiated.

[0125] For example, the log data is marked as "new anomaly". Based on the way historical log data is processed during the construction of the knowledge base, the log data marked as "new anomaly" is processed and the processed log data is updated in the knowledge base to achieve continuous accumulation and updating of knowledge.

[0126] S509, Return or cache the detection result.

[0127] The detection results may include, but are not limited to, information such as fault_type, fault_title, fault_intent, solutions, and impact_scope.

[0128] In summary, the embodiments of this application construct a knowledge base that includes semantics, classification, and dependency relationships, and introduce semantic similarity recall and CoT large model reasoning, which improves the depth of fault detection and semantic understanding capabilities, thereby effectively improving the accuracy of log fault detection.

[0129] Meanwhile, this embodiment, based on a knowledge base, enables the accumulation and reuse of knowledge, rather than simply fine-tuning or transferring models, or merely fusing feature vectors and calculating anomaly scores. This embodiment combines large and small models and introduces a knowledge base to detect log data, calling the CoT large model when necessary, significantly reducing computational costs and latency.

[0130] This application provides a log fault detection device, see [link to relevant documentation]. Figure 6 The figure is a schematic diagram of the structure of a log fault detection device provided in an embodiment of this application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiments of the above method. Some contents will not be repeated.

[0131] A log fault detection device 6100 includes: The system includes a detection unit 6101, an extraction unit 6102, a first matching unit 6103, a second matching unit 6104, and a determination unit 6105. The detection unit 6101 is used to perform initial detection on log data based on a lightweight binary classification model to obtain a first detection result; the lightweight binary classification model is trained with historical log data as input and log level as the output target. The extraction unit 6102 is used to extract keywords from the log data when the first detection result indicates that the log data is faulty, thereby obtaining the keywords of the log data. The first matching unit 6103 is used to match the keyword with the fault classification node in the knowledge base to determine the fault classification node corresponding to the keyword; The second matching unit 6104 is used to match the keyword with the fault semantic intent node under the fault classification node in the knowledge base to determine at least one fault semantic intent node corresponding to the keyword. The determining unit 6105 is used to determine the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword.

[0132] In one possible implementation, the extraction unit is specifically used for: When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain multiple initial keywords; Based on the term frequency and inverse document frequency of the plurality of initial keywords in the log data, a first number of initial keywords are determined from the plurality of initial keywords, and the first number of initial keywords are determined as the keywords of the log data.

[0133] In one possible implementation, the first matching unit is specifically used for: Based on the service name corresponding to the log data, the fault classification nodes in the knowledge base are filtered to obtain fault classification nodes that match the service name corresponding to the log data. The keyword is matched with the fault classification node that matches the service name corresponding to the log data to determine the fault classification node corresponding to the keyword.

[0134] In one possible implementation, the determining unit is specifically used for: Based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, the matching degree between the at least one fault semantic intent node and the keyword is determined; When the matching degree is greater than a preset threshold, a second detection result of the log data is generated based on the at least one fault semantic intent node.

[0135] In one possible implementation, the apparatus further includes: an update unit; The update unit is used to perform fault detection on the log data based on the keyword when the matching degree is less than or equal to a preset threshold, generate a second detection result of the log data, and update the knowledge base based on the second detection result.

[0136] In one possible implementation, the determining unit is specifically used for: Based on the semantic information in the at least one fault semantic intent node and the semantic information in the log data, determine the vector similarity between the log data and the at least one fault semantic intent node; Based on the matching degree and the vector similarity, a second detection result for the log data is determined.

[0137] In one possible implementation, the apparatus further includes a knowledge base construction unit; The knowledge base construction unit is used to annotate the historical log data based on key fields extracted from the historical log data to obtain annotated historical log data; merge the annotated historical log data based on the timestamp and format characteristics of the annotated historical log data to obtain merged historical log data; perform fault detection on the merged historical log data to generate log fault information; and transform the log fault information based on a graph database engine to generate a graph-structured knowledge base.

[0138] In summary, the embodiments of this application adopt a hierarchical detection mechanism, which performs initial detection on log data based on a lightweight binary classification model, and combines a knowledge base including fault classification nodes and fault semantic intent nodes to determine the deep interpretability of the detection results, thereby effectively improving the accuracy of log fault detection.

[0139] This application provides a computer device, which includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of a log fault detection method as described above, according to the instructions in the program code.

[0140] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a log fault detection method as described above.

[0141] This application provides a computer program product that, when run on a computer, executes the steps of a log fault detection method as described above.

[0142] This application provides a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, thereby enabling the chip to implement a log fault detection method as described above.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / duration>

Claims

1. A log fault detection method, characterized in that, include: Based on a lightweight binary classification model, initial detection is performed on the log data to obtain the first detection result; The lightweight binary classification model is trained using historical log data as input and log level as the output target. When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain the keywords of the log data; The keywords are matched with fault classification nodes in the knowledge base to determine the fault classification nodes corresponding to the keywords; Match the keywords with the fault semantic intent nodes under the fault classification nodes in the knowledge base to determine at least one fault semantic intent node corresponding to the keywords; Based on the matching degree between the at least one fault semantic intent node and the keyword, a second detection result of the log data is determined.

2. The method according to claim 1, characterized in that, When the first detection result indicates a fault in the log data, keyword extraction is performed on the log data to obtain the keywords of the log data, including: When the first detection result indicates that the log data is faulty, keyword extraction is performed on the log data to obtain multiple initial keywords; Based on the term frequency and inverse document frequency of the plurality of initial keywords in the log data, a first number of initial keywords are determined from the plurality of initial keywords, and the first number of initial keywords are determined as the keywords of the log data.

3. The method according to claim 1, characterized in that, The first detection result includes the service name corresponding to the log data. The step of matching the keyword with fault classification nodes in the knowledge base to determine the fault classification node corresponding to the keyword includes: Based on the service name corresponding to the log data, the fault classification nodes in the knowledge base are filtered to obtain fault classification nodes that match the service name corresponding to the log data. The keyword is matched with the fault classification node that matches the service name corresponding to the log data to determine the fault classification node corresponding to the keyword.

4. The method according to claim 1, characterized in that, The determination of the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword includes: Based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, the matching degree between the at least one fault semantic intent node and the keyword is determined; When the matching degree is greater than a preset threshold, a second detection result of the log data is generated based on the at least one fault semantic intent node.

5. The method according to claim 4, characterized in that, After determining the matching degree between the at least one fault semantic intent node and the keyword based on the similarity value between the at least one fault semantic intent node and the keyword, and the number of keywords in the log data, the method further includes: When the matching degree is less than or equal to a preset threshold, fault detection is performed on the log data based on the keyword, a second detection result of the log data is generated, and the knowledge base is updated based on the second detection result.

6. The method according to claim 1, characterized in that, The determination of the second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword includes: Based on the semantic information in the at least one fault semantic intent node and the semantic information in the log data, determine the vector similarity between the log data and the at least one fault semantic intent node; Based on the matching degree and the vector similarity, a second detection result for the log data is determined.

7. The method according to any one of claims 1-6, characterized in that, The steps for constructing the knowledge base include: Based on key fields extracted from historical log data, the historical log data is annotated to obtain annotated historical log data. Based on the timestamp and format characteristics of the annotated historical log data, the annotated historical log data is merged to obtain merged historical log data. Fault detection is performed on the merged historical log data to generate log fault information; The log fault information is transformed using a graph database engine to generate a graph-structured knowledge base.

8. A log fault detection device, characterized in that, include: The system comprises a detection unit, an extraction unit, a first matching unit, a second matching unit, and a determination unit. The detection unit is used to perform initial detection on log data based on a lightweight binary classification model to obtain a first detection result; the lightweight binary classification model is trained with historical log data as input and log level as the output target. The extraction unit is used to extract keywords from the log data when the first detection result indicates that the log data is faulty, thereby obtaining the keywords of the log data. The first matching unit is used to match the keyword with the fault classification node in the knowledge base to determine the fault classification node corresponding to the keyword; The second matching unit is used to match the keyword with the fault semantic intent node under the fault classification node in the knowledge base, and determine at least one fault semantic intent node corresponding to the keyword. The determining unit is used to determine a second detection result of the log data based on the matching degree between the at least one fault semantic intent node and the keyword.

9. A computer device, characterized in that, The computer device includes: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of a log fault detection method as described in any one of claims 1-7 according to instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a log fault detection method as described in any one of claims 1-7.

Images