Log detection method and device, storage medium, and computer program product

Abstract

The application provides a log detection method and device, a storage medium and a computer program product. The method comprises the following steps: performing word segmentation on a log to be detected by using a statistical-based sub-word segmentation algorithm to obtain a token sequence corresponding to the log to be detected; generating a semantic vector of each token in the token sequence by using a semantic embedding model based on a character fragment combination mechanism; constructing an anomaly detection feature of the log to be detected based on the semantic vector of each token in the token sequence; and inputting the anomaly detection feature of the log to be detected into an anomaly detection model for anomaly detection to obtain an anomaly detection score of the log to be detected.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a log detection method and apparatus, storage medium, and computer program product. Background Technology

[0002] In existing technologies, the main methods for anomaly detection in web logs fall into the following categories: Based on rule / statistical or lightweight machine learning hybrid solutions, these solutions match known attack signatures using a pre-defined regular expression library or keyword library, or detect anomalies by statistically analyzing thresholds for specific events, and then overlay a simple machine learning model for secondary confirmation.

[0003] Schemes based on a single deep learning model typically employ a mainstream model for analysis. For example, they may use Long Short-Term Memory (LSTM) networks or variants of Recurrent Neural Networks (RNNs) to capture the temporal dependencies of log sequences, or use Graph Neural Networks (GNNs) to analyze the relationships between entities such as Internet Protocol (IP) and Uniform Resource Locators (URLs). The implementation involves preprocessing the logs into sequences or graph structures before inputting them into a single model for training and inference.

[0004] Feature engineering solutions that rely on explicit parsing first extract fields (such as IP, time, URL, status code, etc.) from semi-structured log text using regular expressions.

[0005] All the above solutions involve log text preprocessing, which heavily relies on fragile regular expressions for log parsing. This makes it unsuitable for adapting to the format drift caused by rapid iterations in online business. If upstream business changes alter the log format, field order, or delimiters, the parsing logic fails. Furthermore, word embedding methods cannot effectively address out-of-vocabulary words, and their effectiveness is highly dependent on the accuracy of the word segmentation tool. They cannot handle minor changes in log format, lack detection capabilities for new and unknown attacks, and require manual updates to rule bases and thresholds, resulting in insufficient adaptability and high maintenance costs. Therefore, current log preprocessing capabilities for anomaly detection are insufficient and cannot meet the efficient detection needs of complex business scenarios. Summary of the Invention

[0006] This application provides a log detection method and apparatus, a storage medium, and a computer program product.

[0007] This application provides a log detection method, including: The log to be detected is segmented using a statistical sub-word segmentation algorithm to obtain the token sequence corresponding to the log to be detected; A semantic embedding model based on character fragment combination mechanism is used to generate a semantic vector for each token in the token sequence; Based on the semantic vector of each token in the token sequence, anomaly detection features of the log to be detected are constructed; The anomaly detection features of the log to be detected are input into the anomaly detection model for anomaly detection, and an anomaly detection score of the log to be detected is obtained.

[0008] This application provides a log detection device, including: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the above-described log detection method.

[0009] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described log detection method.

[0010] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described log detection method.

[0011] This application provides a log detection method, apparatus, storage medium, and computer program product. The method includes: segmenting the log to be detected using a statistical sub-word segmentation algorithm to obtain a token sequence corresponding to the log; generating a semantic vector for each token in the token sequence using a semantic embedding model based on a character fragment combination mechanism; constructing anomaly detection features of the log to be detected based on the semantic vector of each token in the token sequence; and inputting the anomaly detection features of the log to be detected into an anomaly detection model for anomaly detection to obtain an anomaly detection score for the log. The technical solution provided in this application, through sub-word segmentation and character-level semantic embedding, eliminates the need for any manually configured parsing rules and can automatically adapt to arbitrary shifts in log format, field order, and delimiters, improving log preprocessing capabilities. It fundamentally solves the problem of traditional solutions causing system paralysis due to upstream business changes, thereby improving log detection performance and enhancing the detection capability and generalization performance against unknown attacks. Attached Figure Description

[0012] Figure 1This is a general architecture diagram of a log detection method provided in an embodiment of this application; Figure 2 A flowchart illustrating a log detection method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating a process for constructing anomaly detection features, provided as an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an anomaly detection model provided in an embodiment of this application; Figure 5 This application provides a schematic diagram of a process for generating exception explanation information. Figure 6 A flowchart illustrating the training process of an anomaly detection model provided in an embodiment of this application; Figure 7 A schematic diagram of the structure of a log detection device provided in this application embodiment. Figure 1 ; Figure 8 A schematic diagram of the structure of a log detection device provided in this application embodiment. Figure 2 . Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The embodiments below can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0015] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0016] Existing log anomaly detection technologies have the following problems: Parsing vulnerabilities and high maintenance costs: It relies heavily on fragile regular expressions for log parsing and cannot adapt to the log format drift caused by rapid iteration of online business. Limitations of detection capabilities: Analysis typically employs a single mainstream model, such as using Long Short-Term Memory (LSTM) networks or variants of Recurrent Neural Networks (RNNs) to capture temporal dependencies in log sequences, or using Graph Neural Networks (GNNs) to analyze relationships between entities like Internet Protocol (IP) and Uniform Resource Locators (URLs). This approach involves preprocessing logs into sequences or graph structures before inputting them into a single model for training and inference. However, this model approach suffers from a limited perspective. Temporal models struggle to capture the lateral correlations of coordinated attacks, while graph models are insensitive to local mutations and subtle temporal patterns, making it difficult to balance global contextual information (such as a complete attack chain) with local details (such as an anomalous parameter). Furthermore, complex graph models often fail to meet low-latency real-time requirements, resulting in insufficient detection capabilities for complex and unknown attacks.

[0017] The contradiction between real-time performance and resource consumption: Although complex models have good performance, they have high inference latency and large resource consumption, making them difficult to deploy on resource-constrained nodes and unable to meet the real-time detection requirements of high-concurrency businesses.

[0018] This application aims to solve the aforementioned technical problems by providing a log detection method that requires no regular expression parsing, is robust to log format drift, can balance global context and local mutation detection, supports unsupervised learning and low-latency deployment, and has good interpretability. The log detection method is implemented using a log detection device. The log detection device can be any electronic device, and this application does not limit its implementation.

[0019] The log detection method provided in this application has the following overall architecture: Figure 1 As shown. This application aims to address the problems of fragile log parsing, limited model perspective, poor real-time performance, and poor interpretability in existing technologies. The core technical solution includes: Character-level preprocessing utilizes a statistical sub-word segmentation algorithm combined with a semantic embedding model that combines character fragment combination (character n-gram) mechanism. Each n-gram learns an embedding vector, and the final text representation is the aggregation of these n-gram vectors. This directly processes the raw log byte stream, avoiding manual regular expression parsing. Construct a parallel dual-channel heterogeneous neural network, in which one neural network channel is used to capture global contextual dependencies and / or long-range dependencies in logs, and the other neural network channel is used to capture local patterns, short-term mutations and rhythmic features in logs; Deeply fuse dual-channel features through a cross-attention mechanism; An unsupervised paradigm combining mask reconstruction and contrastive learning is used for model training; Anomaly scores are calculated based on multi-dimensional information such as reconstruction error and comparison similarity, and interpretable attribution analysis is provided. This invention has significant advantages such as robustness to log format changes, comprehensive detection dimensions, support for low-latency deployment, and interpretable results.

[0020] Based on the above, the technical solution provided in this application is described in detail below.

[0021] Figure 2 This is a flowchart illustrating a log detection method provided in an embodiment of this application. As shown in the figure, the log detection method in this embodiment mainly includes the following steps: S101. Use a statistical sub-word segmentation algorithm to segment the log to be detected, and obtain the token sequence corresponding to the log to be detected.

[0022] In the embodiments of this application, the log detection device can use a statistical sub-word segmentation algorithm to segment the log to be detected and obtain the token sequence corresponding to the log to be detected.

[0023] It should be noted that, in the embodiments of this application, the statistical sub-word segmentation algorithm can be a Byte-Pair Encoding (BPE) algorithm, which uses a data-driven approach to gradually merge frequently co-occurring character pairs into new symbols, thereby finding the optimal "sub-word" unit between characters and complete words. It can also be a WordPiece or SentencePiece algorithm, etc., which are not limited in the embodiments of this application.

[0024] It should be noted that, in the embodiments of this application, taking the BPE algorithm as an example, a BPE word segmenter / algorithm with a vocabulary size of 8k to 16k can be trained directly on the byte stream of massive raw logs without relying on any preset rules (experiments have verified that this range has the best vocabulary coverage and computational efficiency for log data). During online inference, the BPE algorithm performs streaming word segmentation on the input log to be detected, converting it into a token sequence. The complexity of this process is linearly related to the log length L, O(L), which is highly efficient and can adapt to any format.

[0025] S102. Using a semantic embedding model based on character fragment combination mechanism, generate the semantic vector of each token in the token sequence.

[0026] In the embodiments of this application, after obtaining the token sequence corresponding to the log to be detected, the log detection device can use a semantic embedding model based on the character fragment combination mechanism to generate a semantic vector for each token in the token sequence.

[0027] It should be noted that, in the embodiments of this application, a semantic embedding model, such as the FastText model, can be used, utilizing a character fragment combination mechanism, such as character-level n-grams, where n is typically set to 3 to 5, to balance local semantic capture with computational overhead information, to generate a semantic vector d for each token. s =128. Its advantage lies in the fact that even when encountering rare tokens not found in the vocabulary (Out-of-Vocabulary (OOV) problem), it can still construct a meaningful approximate semantic vector by combining the internal character n-grams, thus enhancing the model's generalization ability.

[0028] S103. Based on the semantic vector of each token in the token sequence, construct the anomaly detection features of the log to be detected.

[0029] In the embodiments of this application, the log detection device can construct anomaly detection features of the log to be detected based on the semantic vector of each token in the token sequence.

[0030] In embodiments of this application, the log detection device constructs anomaly detection features of the log to be detected based on the semantic vector of each token in the token sequence, including: generating a first vector based on the log to be detected; concatenating the semantic vector of each token in the token sequence with the first vector and performing layer normalization processing to obtain the anomaly detection features of the log to be detected; wherein, the first vector includes a time vector, or a time vector and a graph statistical vector; the time vector is obtained by decomposing the absolute timestamp in the log to be detected into multiple periodic features and fusing them with the relative position encoding of the log to be detected within its session; the graph statistical vector is obtained by constructing a bipartite graph based on the IP address and URL template of the log to be detected within a sliding time window and calculating statistical indicators.

[0031] It should be noted that, in the embodiments of this application, the absolute timestamp in the log to be detected can be decomposed into multiple periodic features (such as: hours of a day, days of a week, whether it is a holiday, whether it is a business peak, etc.). Each periodic feature is mapped to a low-dimensional vector through a learnable embedding layer, and then all vectors are concatenated and fused and dimensionality reduced through a linear layer, and the relative position encoding of the log within its session is superimposed to generate a time vector d. t =32.

[0032] It should be noted that, in the embodiments of this application, to enhance the detection of collaborative anomalies, a bipartite graph between IP addresses and URL templates can be maintained within a sliding time window (the window size is set to 5-15 minutes, adaptively adjusted according to peak and off-peak traffic). Statistical indicators such as the in-degree increment Δdeg of nodes, the unpopularity degree based on global frequency, and Jaccard similarity are calculated in real time to form a low-dimensional graph statistical vector dg=16.

[0033] It should be noted that in the embodiments of this application, lightweight graph statistics are performed to obtain graph statistical feature vectors, which are mainly applicable to scenarios of detecting cooperative attacks. This method can be selectively implemented according to actual needs and application scenarios, and is not limited in the embodiments of this application.

[0034] It should be noted that, in the embodiments of this application, see... Figure 3 It can concatenate semantic vectors, temporal vectors, and optional graph statistical vectors, and then process them through layer normalization to form the final anomaly detection features. Its expression is shown in the following formula (1): (1) in, Representation layer normalization, A semantic vector representing the token. Represents the time vector. Represents a graph statistical vector. Indicates splicing.

[0035] S104. Input the anomaly detection features of the log to be detected into the anomaly detection model to perform anomaly detection and obtain the anomaly detection score of the log to be detected.

[0036] In the embodiments of this application, after obtaining the abnormal detection features of the log to be detected, the log detection device can input the abnormal detection features into the abnormal detection model to perform abnormal detection and obtain the abnormal detection score of the log to be detected.

[0037] In the embodiments of this application, see Figure 4 The anomaly detection model includes a first neural network channel and a second neural network channel that complement each other. The anomaly detection features of the log to be detected are fed into the first neural network channel and the second neural network channel in parallel as input; The first neural network channel is used to capture global contextual dependencies and / or long-distance dependencies in the logs; The second neural network channel is used to capture local patterns, short-term mutations, and rhythmic features in the logs.

[0038] It should be noted that, in the embodiments of this application, the first neural network channel can be an improved Swin-Transformer based on a shifted window mechanism, used to capture the global context and long-range dependencies of the logs. This application improves the Swin-Transformer for one-dimensional sequence data by dividing the sequence into non-overlapping local windows (window size can be set to 32 or 64), and performing standard self-attention within the window. To achieve cross-window information interaction, a one-dimensional shifted window mechanism (SWM) is adopted, alternating between regular window partitioning and offset window partitioning that slides half the window size in consecutive Transformer layers to maintain cross-window information flow. In a preferred embodiment, the number of layers is set to L. A =3, number of channels h=256. This makes the computational complexity of the model approximately linear with the sequence length n, O(n), instead of the squared relationship O(n²) of the standard Transformer. While maintaining the ability to capture global dependencies, it greatly improves the efficiency of processing long sequences and meets the low latency requirements.

[0039] It should be noted that, in the embodiments of this application, the second neural network channel can be a lightweight temporal convolutional network (TCN) used to capture local patterns, short-term mutations, and rhythmic features in the logs. It consists of multiple layers of causal convolution, ensuring that the model does not see future information when processing the current time step. In a preferred embodiment, the number of layers is set to L. B =6, kernel size k=3, and dilation rate r increases exponentially by {1,2,4,8,...}, allowing its receptive field to expand exponentially with the number of layers. Combined with residual connections and group normalization, TCN can efficiently learn temporal features with a small number of parameters.

[0040] In the embodiments of this application, see Figure 4 The anomaly detection model also includes a fusion network layer; the fusion network layer includes: The first cross-attention layer is used to calculate the first attention-weighted representation by using the output of the first neural network channel as a query and the output of the second neural network channel as the key and value. The second cross-attention layer is used to calculate the second attention weighted representation by using the output of the second neural network channel as the query and the output of the first neural network channel as the key and value. A bidirectional representation fusion layer is used to fuse the first attention-weighted representation and the second attention-weighted representation to obtain fused features; The gated fusion unit is used to fuse the outputs of the first neural network channel, the outputs of the second neural network channel, and the fusion features to obtain the cross-attention output of the log to be detected.

[0041] It should be noted that, in the embodiments of this application, see... Figure 4 To achieve dual-channel information complementarity, a cross-attention (CA) mechanism is used for deep interaction. Specifically, the output of one neural network channel is used as a query, and the output of the other neural network channel is used as a key and value to calculate the attention-weighted representation, and vice versa, thereby generating bidirectional fusion features. Subsequently, a gated fusion unit (GFU) is used to dynamically combine the output A of the first neural network channel, the output B of the second neural network channel, and the fusion feature Z. Optional graph statistics vectors can be used as bias terms to input the gated fusion unit, affecting the fusion weights. Its fusion logic can be expressed as the following equation (2): (2) in, It is a cross-attention output. It is the Sigmoid activation function. These are learnable gating weights. For gating parameters, For linear transformation, Indicates splicing, This is element-wise multiplication.

[0042] In the embodiments of this application, the anomaly detection score is obtained by standardizing and weighting the anomaly indicators of the cross-attention output of the log to be detected on multiple detection dimensions.

[0043] It should be noted that, in the embodiments of this application, the anomaly detection model may also include other network layers or units, which can extract anomaly indicators from the cross-attention output of the log to be detected on multiple detection dimensions, and then standardize and weightedly fuse them to obtain an anomaly detection score.

[0044] It should be noted that, in the embodiments of this application, the anomaly detection score of a log entry can be composed of three weighted parts: its reconstruction error, its similarity to the normal pattern center, and its anomaly score from graph statistics. Its expression is as follows (3): (3) In actual calculations, the three outlier indicators are first subjected to quantile transformation or Z-score standardization (represented by the Norm() function) to unify the dimensions and distribution, thereby enhancing the stability of the scores. The weight parameters α, β, and γ can be optimized based on the performance on the validation set, and are not limited in this embodiment.

[0045] In the embodiments of this application, the log detection device may further perform the following steps after obtaining the anomaly detection score of the log to be detected: if the anomaly detection score is higher than the anomaly threshold, obtain the cross-attention matrix and the reconstruction error of each token in the anomaly detection model; based on the cross-attention matrix and the reconstruction error of each token, identify abnormal log fragments from the log to be detected and generate anomaly explanation information.

[0046] It should be noted that, in the embodiments of this application, as... Figure 5 As shown, when an anomaly is detected (i.e., the anomaly detection score exceeds the anomaly threshold), interpretability is also provided. By analyzing the cross-attention matrix within the model and the reconstruction error of each token, the anomaly log segment (key substring or time point) that contributes most to the anomaly determination can be located, generating anomaly explanation information. This anomaly explanation information can be presented as a heatmap highlighting the anomaly log segment, generating a traceable anomaly attribution path. Furthermore, it can also include a snapshot of the bipartite graph hotspot subgraph combined with the time of the anomaly occurrence, providing security analysts with rich, multi-dimensional tracing evidence.

[0047] It should be noted that, in the embodiments of this application, bucketing can be performed according to different business lines or URL templates, that is, continuous or discrete score values ​​are divided into several intervals (buckets) according to a certain rule, each bucket is processed or compared independently, and the quantile method is used to dynamically determine the abnormal threshold to adapt to the baseline fluctuations in different scenarios. The bucket quantile threshold calculation method can be expressed as the following formula (4): (4) in, denoted as p (p∈[0.95,0.99]), and b is the bucket identifier.

[0048] In the embodiments of this application, the log detection device can also perform unsupervised training on the anomaly detection model, which will be described in detail below.

[0049] In embodiments of this application, the log detection device may further perform the following steps: performing a first training task and / or a second training task, and training an anomaly detection model based on the loss obtained from the first training task and / or the second training task; the first training task includes: masking the anomaly detection feature portion of the sample log, reconstructing it using the anomaly detection model, and calculating the reconstruction loss based on the reconstruction bias; the second training task includes: taking the sample log and the first positive sample as a positive sample pair, and the sample log and the first negative sample as a negative sample pair; generating cross-attention outputs for each sample using the anomaly detection model, performing comparative learning between the positive sample pairs and the negative sample pairs, and calculating the comparative learning loss.

[0050] It should be noted that, in the embodiments of this application, see... Figure 6 A masked language model approach, similar to Bidirectional Encoder Representations from Transformers (BERT), can be adopted. During training, anomaly detection features of the input sample logs are randomly masked at a certain proportion (e.g., 15%-30%), and then the model is trained to predict the masked content. The masking strategy can be further optimized, for example, by dynamically increasing the masking rate of high-information-entropy substrings (e.g., random parameters), parameter key-value pairs, or historically frequent anomaly time slices, to enhance the model's ability to learn key information. Reconstruction Loss It can be expressed as the following formula (5): (5) Where N is the total number of sample logs, This represents the anomaly detection feature of the i-th sample log. This represents the anomaly detection feature of the reconstructed log of the i-th sample. It is the square of the L2 norm.

[0051] It should be noted that, in the embodiments of this application, see... Figure 6 Contrastive learning can be used for model training. To enable the model to distinguish patterns from different sessions, a contrastive learning loss (InfoNCE) is introduced. Different log fragments from the same session are treated as positive sample pairs, while log fragments from different sessions or time periods separated by a large gap are treated as negative sample pairs. The loss function forces the model to learn representations that bring positive samples closer together and push negative samples further away, thereby improving the model's ability to understand session-level semantics. Contrastive learning loss. It can be expressed as the following formula (6): (6) in, Indicates the number of positive sample pairs. This represents the cross-attention output of the i-th sample log. This represents the cross-attention output of the logs that form a positive sample pair with the i-th sample log. This represents the cross-attention output of the logs that form a negative sample pair with the i-th sample log. τ is the temperature coefficient (τ>0).

[0052] It should be noted that, in the embodiments of this application, see... Figure 6 The total training loss L can be a weighted sum of the reconstruction loss and the contrastive learning loss, expressed as follows (7): (7) in, and It is a hyperparameter used to balance the loss of the two training tasks.

[0053] Based on the above, the technical solution provided in this application improves upon the parsing capability, detection capability, real-time performance, and interpretability of log anomaly detection. Specific technical advantages are as follows: Robustness and Maintenance-Free Operation: Addressing the "fragility" of existing technologies, this solution employs statistical sub-segmentation algorithms, such as the BPE algorithm, to directly process log bytes. It requires no manually configured parsing rules and automatically adapts to changes in log format, field order, and delimiters. This fundamentally solves the problem of traditional solutions causing system paralysis due to upstream business changes, significantly improving system robustness and reducing maintenance costs.

[0054] Comprehensive and high-precision detection: Addressing the limitations of existing technologies in detection capabilities, this solution utilizes a dual-channel heterogeneous architecture of Swin-Transformer and TCN, along with cross-attention fusion, to simultaneously perceive the global context of logs (such as multi-step attack chains) and local mutations (such as single anomalous parameter values). Combined with the inherent sensitivity of mask reconstruction to unknown samples, it significantly improves the recall and accuracy for complex, unknown, and variant attacks.

[0055] Balancing high performance and low latency: Addressing the shortcomings of existing technologies in the "contradiction between real-time performance and resource consumption", the model adopts the linear complexity of the Swin-Transformer and the efficient convolutional network TCN, enabling the model to maintain strong representation capabilities while having low inference latency and low resource consumption. It is capable of being deployed on resource-constrained edge nodes or meeting the real-time detection requirements of high-concurrency business.

[0056] Enhanced collaborative anomaly detection capability: By introducing optional lightweight bipartite graph statistical features and incorporating them as gating biases into the main model, the ability to identify lateral collaborative attacks such as Distributed Denial of Service (DDoS) and distributed scanning is effectively enhanced, and the computational cost is far lower than deploying a complete, heavyweight GNN model.

[0057] Excellent interpretability and traceability: Addressing the shortcomings of traditional deep learning models as "black boxes," it can generate anomaly log fragments based on attention weights and reconstruction errors, and generate anomaly explanation information, including but not limited to generating attribution heatmaps and automatically highlighting the root causes of anomalies (key characters, parameters, or time points). This provides security operations personnel with intuitive and reliable decision-making basis, greatly improving the operability of alarms and the efficiency of investigation.

[0058] This application also provides a log detection device. Figure 7 A schematic diagram of the structure of a log detection device provided in this application embodiment. Figure 1 .like Figure 7 As shown, the log detection device 1 includes: The log processing module 11 is used to segment the log to be detected using a statistical sub-word segmentation algorithm to obtain the token sequence corresponding to the log to be detected; to generate the semantic vector of each token in the token sequence using a semantic embedding model based on character fragment combination mechanism; and to construct the anomaly detection features of the log to be detected based on the semantic vector of each token in the token sequence. The anomaly detection module 12 is used to input the anomaly detection features of the log to be detected into the anomaly detection model for anomaly detection, and obtain the anomaly detection score of the log to be detected.

[0059] In one embodiment of this application, the log processing module 11 is used to generate a first vector based on the log to be detected; and to perform layer normalization processing after concatenating the semantic vector of each token in the token sequence with the first vector to obtain the anomaly detection features of the log to be detected. Wherein, the first vector includes a time vector, or the time vector and a graph statistics vector; The time vector is obtained by decomposing the absolute timestamp in the log to be detected into multiple periodic features and fusing them with the relative position encoding of the log to be detected within its session. The graph statistical vector is obtained by constructing a bipartite graph based on the IP address and URL template of the log to be detected within a sliding time window, and then calculating statistical indicators.

[0060] In one embodiment of this application, the anomaly detection model includes a first neural network channel and a second neural network channel that complement each other. The anomaly detection features of the log to be detected are fed into the first neural network channel and the second neural network channel in parallel as inputs. The first neural network channel is used to capture global contextual dependencies and / or long-distance dependencies in the logs; The second neural network channel is used to capture local patterns, short-term mutations, and rhythmic features in the logs.

[0061] In one embodiment of this application, the anomaly detection model further includes a fusion network layer; The fusion network layer includes: The first cross-attention layer is used to calculate the first attention-weighted representation by using the output of the first neural network channel as a query and the output of the second neural network channel as a key and value. The second cross-attention layer is used to calculate the second attention-weighted representation using the output of the second neural network channel as a query and the output of the first neural network channel as a key and value. A bidirectional representation fusion layer is used to fuse the first attention-weighted representation and the second attention-weighted representation to obtain fused features; The gated fusion unit is used to fuse the output of the first neural network channel, the output of the second neural network channel, and the fusion feature to obtain the cross-attention output of the log to be detected.

[0062] In one embodiment of this application, the anomaly detection score is obtained by standardizing and weighting the anomaly indicators of the cross-attention output of the log to be detected across multiple detection dimensions.

[0063] In one embodiment of this application, the log detection device 1 further includes an anomaly alerting module (not shown in the figure), which is used to obtain the cross-attention matrix and the reconstruction error of each token in the anomaly detection model if the anomaly detection score is higher than the anomaly threshold; and to identify abnormal log fragments from the log to be detected based on the cross-attention matrix and the reconstruction error of each token, and generate anomaly explanation information.

[0064] In one embodiment of this application, the log detection device 1 further includes a model training module (not shown in the figure), which is used to perform a first training task and / or a second training task, and train the anomaly detection model based on the loss obtained from the first training task and / or the second training task; The first training task includes: masking the anomaly detection feature portion of the sample logs, reconstructing it using the anomaly detection model, and calculating the reconstruction loss based on the reconstruction bias; The second training task includes: taking the sample log and the first positive sample as a positive sample pair, and the sample log and the first negative sample as a negative sample pair; using the anomaly detection model to generate cross-attention outputs for each sample, performing comparative learning between the positive sample pair and the negative sample pair, and calculating the comparative learning loss.

[0065] Based on the same inventive concept Figure 8 A schematic diagram of the structure of a log detection device provided in this application embodiment. Figure 2 .like Figure 8 As shown in the embodiments of this application, the log detection device 1 includes: a processor 13, a memory 14, and a communication bus 15; The communication bus 15 is used to realize the communication connection between the processor 13 and the memory 14; The processor 13 is used to execute one or more computer programs stored in the memory 14 to implement the above-described log detection method.

[0066] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described log detection method.

[0067] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described log detection method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-described memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0069] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A log detection method, characterized in that, include: The log to be detected is segmented using a statistical sub-word segmentation algorithm to obtain the token sequence corresponding to the log to be detected; A semantic embedding model based on character fragment combination mechanism is used to generate a semantic vector for each token in the token sequence; Based on the semantic vector of each token in the token sequence, anomaly detection features of the log to be detected are constructed; The anomaly detection features of the log to be detected are input into the anomaly detection model for anomaly detection, and an anomaly detection score of the log to be detected is obtained.

2. The method according to claim 1, characterized in that, The step of constructing anomaly detection features corresponding to the log to be detected based on the semantic vector of each token in the token sequence includes: A first vector is generated based on the log to be detected; The semantic vector of each token in the token sequence is concatenated with the first vector and then subjected to layer normalization to obtain the anomaly detection features of the log to be detected. Wherein, the first vector includes a time vector, or the time vector and a graph statistics vector; The time vector is obtained by decomposing the absolute timestamp in the log to be detected into multiple periodic features and fusing them with the relative position encoding of the log to be detected within its session. The graph statistical vector is obtained by constructing a bipartite graph based on the Internet Protocol IP address and Uniform Resource Locator (URL) template of the log to be detected within a sliding time window, and then calculating statistical indicators.

3. The method according to claim 1, characterized in that, The anomaly detection model includes a first neural network channel and a second neural network channel that complement each other. The anomaly detection features of the log to be detected are fed into the first neural network channel and the second neural network channel in parallel as inputs. The first neural network channel is used to capture global contextual dependencies and / or long-distance dependencies in the logs; The second neural network channel is used to capture local patterns, short-term mutations, and rhythmic features in the logs.

4. The method according to claim 3, characterized in that, The anomaly detection model also includes a fusion network layer; The fusion network layer includes: The first cross-attention layer is used to calculate the first attention-weighted representation by using the output of the first neural network channel as a query and the output of the second neural network channel as a key and value. The second cross-attention layer is used to calculate the second attention-weighted representation using the output of the second neural network channel as a query and the output of the first neural network channel as a key and value. A bidirectional representation fusion layer is used to fuse the first attention-weighted representation and the second attention-weighted representation to obtain fused features; The gated fusion unit is used to fuse the output of the first neural network channel, the output of the second neural network channel, and the fusion feature to obtain the cross-attention output of the log to be detected.

5. The method according to any one of claims 1-4, characterized in that, The anomaly detection score is obtained by standardizing and weighting the anomaly indicators of the cross-attention output of the log to be detected across multiple detection dimensions.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: If the anomaly detection score is higher than the anomaly threshold, obtain the cross-attention matrix and the reconstruction error of each token within the anomaly detection model; Based on the cross-attention matrix and the reconstruction error of each token, abnormal log fragments are identified from the logs to be detected, and abnormal explanation information is generated.

7. The method according to any one of claims 1-4, characterized in that, The method further includes: Perform a first training task and / or a second training task, and train the anomaly detection model based on the loss obtained from the first training task and / or the second training task; The first training task includes: masking the anomaly detection feature portion of the sample logs, reconstructing it using the anomaly detection model, and calculating the reconstruction loss based on the reconstruction bias; The second training task includes: taking the sample log and the first positive sample as a positive sample pair, and the sample log and the first negative sample as a negative sample pair; using the anomaly detection model to generate cross-attention outputs for each sample, performing comparative learning between the positive sample pair and the negative sample pair, and calculating the comparative learning loss.

8. A log detection device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images