Cross-system log anomaly detection method based on multi-layer transformer semantic alignment and transfer learning

Abstract

The application is a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning, including the following contents: uniformly preprocessing original log data from different systems; constructing a multi-layer Transformer representation learning module, including a word-level Transformer layer, a log event-level Transformer layer and cross-system semantic interaction and consistency modeling; under the transfer learning framework, optimizing through a contrastive learning loss function to make the semantic representations of the source system and the target system in the embedding space semantically aligned, obtaining a cross-system consistent log semantic matrix; and using an anomaly detection model based on the cross-system consistent log semantic matrix to complete anomaly determination. Not only can the deep modeling capability of log semantic structure and context dependency be improved, but also the alignment and transfer of the unified semantic space can be realized, and the detection accuracy and system universality can be improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer system operation and maintenance technology, specifically a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning. Background Technology

[0002] With the widespread application of cloud computing, microservices, and distributed systems, the scale of log data generated during system operation is growing exponentially, and the complexity of logs is also increasing dramatically. Logs, as a crucial record of system status and faults, comprehensively document the system's behavior, events, and state changes during operation. In complex systems, logs serve as the unique, unified "language" among modules, and their intelligent analysis and anomaly detection have become key means to ensure stable system operation. Traditional log anomaly detection methods mainly rely on manual rules, statistical thresholds, or template-based clustering. These methods suffer from drawbacks such as a lack of semantic understanding and poor cross-system transfer performance when dealing with logs from multi-source heterogeneous systems. Therefore, a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning is needed to overcome these shortcomings.

[0003] Existing log anomaly detection methods generally rely on template matching strategies or shallow neural networks to model logs, exhibiting low sensitivity to semantic changes and contextual dependencies in system logs, making them ill-suited for complex and diverse log formats. Secondly, traditional log detection models lack semantic transfer mechanisms, leading to significant performance degradation on new systems. This necessitates re-labeling and retraining, resulting in high costs and low efficiency. Furthermore, existing methods often rely on large amounts of labeled data or require model retraining when new systems emerge, making it difficult to quickly adapt to new systems with limited or unlabeled samples, demonstrating insufficient generalization ability.

[0004] In recent years, some papers on log anomaly detection methods have had shortcomings: CLSTMlog (Li J, Zhang H, Wang P, et al. Log anomaly detection method based on CNN and LSTM fusion [C] / / Second International Conference on Informatics, Networking, and Computing. 2024: 1307818.) This method integrates local pattern recognition (CNN) and temporal feature recognition (LSTM), has a relatively simple structure, and is easy to train, but lacks semantic depth, has weak cross-system generalization, and lacks a few-shot adaptation mechanism, resulting in limited transfer performance. LogMS (Yu Z, Yang S, Zhang X, et al. LogMS: a multi-stage log anomaly detection method based on multi-source information fusion and probability label estimation [J]. Frontiers in Physics, 2024, 12: 1401857.) This method introduces multi-source information fusion, relies on templated parsing and manually designed features, but its temporal feature representation is unreasonable, and its cross-system transfer and adaptation capabilities are weak. BERT-Log (Chen S, Wang Q, Li F, et al. BERT-Log: Anomalydetection for system logs based on pre-trained language model [J]. IAppliedArtificial Intelligence, 2022, 36(1).) introduces a predictive training language model, which improves the ability to extract semantic features, but still requires a large number of labeled samples; the semantic space is sensitive to cross-system changes and lacks a dedicated strategy for fast adaptation with few samples.

[0005] In summary, existing log anomaly detection methods still have significant shortcomings in terms of insufficient semantic understanding depth, limited cross-system transfer performance, and weak model adaptability. There is an urgent need for a log anomaly detection method that can deeply model log semantic features, achieve cross-system semantic alignment, and has rapid adaptation capabilities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention aims to solve the technical problem of proposing a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning. This method not only improves the deep modeling capability of log semantic structure and contextual dependencies but also achieves alignment and transfer within a unified semantic space. Furthermore, it enables a rapid adaptation mechanism under conditions of few samples. This invention not only ensures the model's detection performance in new systems but also improves detection accuracy and system versatility.

[0007] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows: In a first aspect, the present invention provides a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning, the method comprising the following: The raw log data from different systems is preprocessed in a unified manner to extract key fields including time, log level and text content, and standardized log input is obtained through template and word segmentation to obtain log word vector sequences. Construct a multi-layer Transformer representation learning module, including a word-level Transformer layer, a log event-level Transformer layer, and cross-system semantic interaction and consistency modeling; Introduce log-level weights in the word-level Transformer layer. and log-level embedding vector The word vectors of the logs are fused according to the following formula, and the fused word vectors are used to construct an enhanced log word vector sequence. ; , ,

[0008] in, It is the first The original word vector representation of each word. It is the first The word vector representation after word enhancement; level represents the log level, ERROR represents an error, WARN represents a warning, and INFO represents information; It is the first The embedding vector of each word at the log level; It is the number of words contained in the log; It is the first The weight of each word at the log level; The enhanced log word vector sequence After processing by a multi-head self-attention mechanism, the vectors are then normalized by an additive layer with the word vector sequence of the current log, resulting in the output representation matrix of the log word-level Transformer. ; Through aggregation functions Compression is performed to obtain a log-level semantic representation vector. Positional encoding is added to the log-level semantic representation vector to obtain an enhanced log event representation. This enhanced log event representation is then input into a log event-level Transformer layer, where a self-attention mechanism captures global dependencies and temporal relationships between log events, yielding a log event-level semantic aggregation result. ; In cross-system semantic interaction and consistency modeling, the event-level semantic aggregation results of the source system logs are used as query vectors, and the event-level semantic aggregation results of the target system logs are used as key-value vectors. Attention weights are used to calculate the semantic association strength between the source and target system logs. Based on this association strength, the event-level semantic aggregation results of the source system logs are weighted and fused, thereby achieving cross-system semantic interaction and alignment, and obtaining a system-level semantic representation. ; Within the transfer learning framework, the contrastive learning loss function is optimized to semantically align the log semantic representations of the source and target systems in the embedding space, thereby obtaining a consistent log semantic matrix across systems. Anomaly detection is performed using an anomaly detection model based on a consistent log semantic matrix across systems.

[0009] Furthermore, the semantic alignment process is as follows: The log event-level semantic aggregation results of the multi-layer Transformer representation learning module of the source system and the target system. and system-level semantic representation Weighted fusion is performed to obtain the fused semantic vector representations of the source system and the target system, and feature standardization is performed to obtain the feature-standardized semantic vector representation. Based on the feature-standardized semantic vector representation, the similarity between positive and negative samples of the source system and the target system is calculated. The InfoNCE loss is used for alignment training, so that the semantic features of the source domain and the target domain tend to be close in the embedding space, thus completing the semantic distribution alignment. After semantic distribution alignment, a semantic preservation adjustment term is introduced, and the formula for calculating the semantic preservation loss is as follows:

[0010] in, and These are the source system and target system log semantic matrices after normalization and linear mapping operations, respectively. and These are the semantic matrices of the source and target system logs after semantic alignment. Optimization training is performed under the joint constraints of semantic alignment loss and semantic preservation loss to obtain a consistent log semantic matrix across systems.

[0011] Furthermore, the anomaly detection model adopts a prototype network-based anomaly detection model, using the cross-system consistent log semantic matrix as input samples, obtaining anomaly scores by calculating the distance between the input samples and normal category prototypes, and completing anomaly determination based on preset thresholds.

[0012] Furthermore, a prototype feedback adjustment unit is introduced into the semantic alignment process to constrain and optimize the alignment results using normal category prototypes, so that the semantic alignment results not only satisfy cross-system semantic consistency, but also conform to the normal behavioral pattern structure.

[0013] Furthermore, in the prototype feedback adjustment unit, the prototype feedback constraint term is:

[0014] in: This represents the number of log samples. This represents the Euclidean distance, a constraint term used to measure the degree of deviation between the log semantic representation and the center of normal behavior; This is the semantically aligned log semantic vector representation. The semantic center vector of the normal category prototype.

[0015] Secondly, this invention provides a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning, the method comprising the following: The raw log data from different systems is preprocessed in a unified manner to extract key fields including time, log level and text content, and standardized log input is obtained through template and word segmentation to obtain log word vector sequences. An anomaly detection network is constructed, comprising a multi-layer Transformer representation learning module, a semantic alignment module, and an anomaly detection module based on a prototype network. The multi-layer Transformer representation learning module is implemented using a three-layer Transformer, including a word-level Transformer layer, a log event-level Transformer layer, and cross-system semantic interaction and consistency modeling. The multi-layer Transformer represents the output of the log event-level Transformer layer of the learning module, which integrates cross-system semantic interaction and consistency modeling, and serves as the input to the semantic alignment module. The semantic alignment module outputs a cross-system consistent log semantic matrix, which is used as input to the prototype network-based anomaly detection module; the prototype network-based anomaly detection module outputs anomaly scores, and the results are used to determine the anomaly score. The total loss function of the anomaly detection network for:

[0016] in, To preserve the weight parameters for semantics, For prototype feedback weight parameters; For prototype feedback constraints; Loss due to semantic preservation; For semantic alignment loss; The anomaly detection network is trained using the total loss function, and the trained anomaly detection network is used for cross-system log anomaly detection.

[0017] Thirdly, the present invention provides a cross-system log anomaly detection system based on multi-layer Transformer semantic alignment and transfer learning. The system executes the method described above, including: a log preprocessing module, a multi-layer Transformer representation learning module, and a cross-system semantic alignment and transfer learning module.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention significantly enhances the expressive power of log semantic features by deeply modeling the contextual dependencies of logs using multi-layer Transformers. By introducing semantic alignment, a unified semantic space mapping is established across different systems, effectively solving the problem of inconsistent feature distribution across systems. Combined with transfer learning, the model can quickly transfer to new systems with limited samples, achieving efficient adaptation and detection. This invention constructs a complete technical chain from semantic understanding and cross-system alignment to rapid learning, improving the accuracy, robustness, and versatility of cross-system log anomaly detection, and has high engineering practical value and promotional significance.

[0019] This invention combines multi-layer Transformer, semantic alignment, and transfer learning to extract the complex semantics required for log anomaly detection, achieving a unified expression of log semantics and efficient model transfer across heterogeneous systems, thereby improving the accuracy and robustness of cross-system log anomaly detection. This invention not only ensures consistency in the cross-system semantic space but also performs structural optimizations on the feature extraction and rapid adaptation processes to ensure the model better serves the log anomaly detection task, rather than merely performing shallow feature transfer between the source and target systems. Through this optimized design, semantic alignment consistency is maintained across cross-system tasks while improving detection accuracy and sensitivity, significantly enhancing the system's ability to identify fine-grained anomaly features. Furthermore, this invention employs an anomaly detection module based on a prototype network in the transfer learning process. This lightweight structure only requires calculating the feature distance between the sample and the class prototype to complete the classification judgment, avoiding repeated updates of complex gradients and significantly reducing computational overhead. Rapid adaptation can be completed without additional parameter training during the inference phase, thus exhibiting higher efficiency and stability when processing large-scale, multi-source heterogeneous log data. This not only improves cross-system migration performance, but also enables the main computing resources to be concentrated on deep semantic extraction and anomaly feature recognition tasks of multi-layer Transformers, further improving detection accuracy and real-time response capabilities.

[0020] This invention employs a multi-layered Transformer architecture to enhance the multi-level modeling capabilities of log semantics. It primarily comprises three Transformer representation layers. The first layer encodes log word-level sequences, capturing key trigger words and their contextual relationships to generate a unified underlying semantic representation. The second layer performs deep semantic modeling for structured log events, further enhancing the expressive power of log templates and field semantics. The third layer integrates broader contextual information based on event semantics to generate system-level semantics for logs, improving the global consistency and discriminability of log representation. This three-layer Transformer architecture, through inter-layer residual connections and layer normalization mechanisms, maintains high stability and robustness in large-scale log streams, effectively addressing complex issues such as system heterogeneity, data sparsity, and log pattern drift.

[0021] Furthermore, in the log semantic representation stage, this invention introduces log level weights and log level embedding vectors in the word-level Transformer layer. By performing differentiated modeling for different log levels, the model can simultaneously perceive the semantics of log content and the severity information of logs during the semantic representation process, thereby improving the sensitivity and discrimination ability of abnormal log identification.

[0022] Furthermore, in the semantic alignment process, this invention introduces a prototype feedback adjustment unit, which uses the normal category prototype output by the anomaly detection model based on the prototype network to provide feedback constraints on the semantic alignment results. This ensures that the unified semantic representation not only meets the cross-system consistency requirements but also conforms to the normal behavior pattern structure, thereby improving the consistency and stability between the semantic alignment results and the anomaly detection task.

[0023] This invention fully considers the differences in semantic expression, format structure, and vocabulary distribution among different system logs. Based on the semantic features extracted by a multi-layer Transformer, this invention performs semantic alignment to achieve unified mapping and alignment of log features from different systems. Simultaneously, to prevent semantic confusion caused by over-alignment, this invention sets a semantic preservation adjustment term during the training phase, thereby balancing the needs of "feature alignment" and "system difference preservation." This invention effectively solves the problems of inconsistent feature distribution and difficulty in unifying the semantic space among different systems in traditional log anomaly detection methods, enabling dynamic modeling of semantic relationships between systems and improving the universality and transferability of log features.

[0024] This invention introduces a rapid adaptation mechanism based on Prototypical Network (ProtoNet) into a unified feature space after semantic alignment. This mechanism enables different categories of ProtoNet to naturally cluster in the embedding space through semantic alignment and unified feature space mapping, avoiding domain shift. Simultaneously, prototypes are constructed on the system-level semantic representation after semantic alignment, using normal class prototype modeling and distance thresholding to identify anomalies, forming a stable embedding space suitable for situations where log anomaly types are not exhaustive. This addresses the problems of scarce labeled samples and low model transfer efficiency between different systems. By training the model on both the source and target systems using prototype metric learning, the model learns transferable feature metrics and category prototype representations, enabling rapid adaptation to new systems and anomaly identification under limited sample conditions. Furthermore, the ProtoNet structure exhibits good interpretability and stability. Different category prototypes can be considered as shared semantic centers across systems, and the distance between a sample and a prototype reflects the degree of log anomaly, thus achieving interpretable anomaly detection based on semantic similarity. This invention realizes the transformation of the model from "feature transfer" to "knowledge transfer", which enhances the model's rapid adaptability and generalization performance in different log systems. The module structure is lightweight and computationally efficient, which can significantly improve detection robustness and response speed in cross-system and low-sample scenarios, ensuring that log anomaly detection maintains stable and reliable performance in dynamic system environments.

[0025] Meanwhile, by introducing log-level weights and log-level embedding vectors in the word-level semantic representation stage, this invention can effectively enhance the model's sensitivity to high-risk logs (such as ERROR, WARN, etc.) and improve the accuracy of anomaly identification.

[0026] By introducing prototype feedback regulation during the semantic alignment process, the unified semantic representation can be guided by the normal behavioral pattern structure, thereby reducing invalid alignment and further improving the stability and generalization ability of the cross-system anomaly detection model. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 This is a schematic diagram of a word-level Transformer layer according to an embodiment of the present invention; wherein, It is the first The original word vector representation of each word. It is the first The word vector representation after word enhancement. Trm is an abbreviation for Transformer. It is the first The contextual semantic vector of each log term, i.e., the output representation of the log term-level Transformer; This represents a log-level semantic representation vector, used to characterize the global semantic features of the entire log entry.

[0028] Figure 3 The diagrams for the log event-level Transformer layer and cross-system semantic interaction and consistency modeling are identical, showing that they share the same network structure.

[0029] Figure 4 This is a schematic diagram of the overall structure of an anomaly detection network according to an embodiment of the present invention. Detailed Implementation

[0030] The present invention will be further explained below with reference to the embodiments and accompanying drawings, but this is not intended to limit the scope of protection of this application.

[0031] Example 1 This embodiment presents a cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning. The method includes the following steps: Step 1: Log Preprocessing Module Before formally performing log semantic modeling, the raw log data from different systems is first standardized through a log preprocessing module. The main purpose is to eliminate differences in log format, field structure, and encoding methods, providing standardized input for the subsequent Transformer model. Specifically, firstly, the raw log text is timestamped and parsed, removing redundant information and special symbols, and extracting key fields such as time, module, log level, and content. Secondly, the log content is templated and segmented, abstracting dynamic variables in the log and reducing the impact of semantic noise. This invention employs a BERT-based dynamic word vector embedding model to map the segmented log text into context-dependent semantic vector representations. This method can dynamically adjust word meanings according to different contexts in the log, thus maintaining semantic consistency across log corpora from different systems. After processing by this module, the semantic structure of log data from different systems is initially unified, providing directly input feature representations for the multi-layer Transformer representation learning module.

[0032] Step 2: Multi-layer Transformer Representation Learning Module This module consists of a three-layer Transformer structure: a word-level Transformer layer (word-level semantic encoding), a log event-level Transformer layer (log event-level semantic aggregation), and a cross-system semantic interaction and consistency modeling layer (log system-level Transformer layer).

[0033] 2.1 Word-level Transformer Layer The word-level Transformer layer is used to model the semantic relationships within a single log entry and capture the contextual dependencies between words.

[0034] First, suppose a log entry contains, after word segmentation... The word vector sequence of words is represented as follows:

[0035] in, It is the first The original word vector representation of each word. It is the number of words contained in the log. It is the dimension of the word vector.

[0036] To enhance the model's ability to perceive the semantic differences and importance of different log levels, this invention assigns corresponding weights to different log levels based on log level information. This is used to adjust the influence of different log levels on semantic representation. Log level embedding vectors are added. By combining weight adjustment, a joint modeling of log semantics and importance is achieved, and an enhanced log word vector sequence is constructed. :

[0037]

[0038]

[0039] in, It is a log-level weight. It is the first The weight of each word at the log level. The log levels are: ERROR for errors, WARN for warnings, and INFO for information. These are log-level embedding vectors, initialized using a preset initialization strategy. This strategy includes assigning values ​​to the embedding vectors using a random distribution, ensuring a differentiated distribution of embedding vectors for each log level in the initial stage. During model training, these embedding vectors are continuously updated as learnable parameters through backpropagation, thereby forming stable and distinguishable representations in the shared semantic space. It is the first The embedding vector of each word at the log level; It is the first The original word vector representation of each word. It is the first The word vector representation after word enhancement.

[0040] The log level weights are used to weight and modulate log word vectors according to the importance of the log level, and the log level embedding vectors are used to represent the semantic differences between different log levels. By introducing the log level weights and log level embedding vectors into the log semantic representation process, joint modeling of log level importance information and semantic information is achieved, thereby improving the model's ability to identify abnormal logs.

[0041] The semantic representation of the input sequence is calculated using a multi-head self-attention mechanism, and the query matrix is ​​first obtained through a linear transformation. Key matrix Sum matrix And calculate the output of a single attention head:

[0042] in, This is the learnable parameter matrix. It is a scaling factor. It is a word relevance matrix; This represents the output of a single attention head in a specific semantic subspace.

[0043] Furthermore, multiple attention heads are computed in parallel using a multi-head mechanism, and their outputs are concatenated and linearly transformed.

[0044] Each of them Indicates the first The output of each attention head. It is a query matrix. It is a key matrix. It is a value matrix. It is the first The query projection matrix of each attention head. It is the first The key projection matrix of each attention head It is the first The projection matrix of the values ​​of each attention head, This indicates that the data is assembled in columns. It outputs a linear transformation matrix.

[0045] Finally, the output representation matrix of the log word-level Transformer is obtained. :

[0046] in, It is a sequence of log word vectors. It is a layer normalization function. It is the first The contextual semantic vector of each log term.

[0047] Based on this, aggregate functions are used to... After compression, the log-level semantic representation vector is obtained:

[0048] in, This represents an aggregation operation on a sequence representation. This represents a log-level semantic representation vector, used to characterize the global semantic features of the entire log entry.

[0049] 2.2 Log Event-Level Transformer Layer The log event-level Transformer layer models the sequence of log-level semantic representation vectors obtained from the word-level Transformer layer. It aims to perform deep abstraction of log templates and their field semantics, thereby enhancing the log event-level representation capabilities.

[0050] Let the log-level semantic representation vector sequence be... ,in Indicates the first The log-level semantic representation vector is obtained by encoding each log event using a word-level Transformer and then performing an aggregation operation. This represents the number of log events contained in a single sample instance.

[0051] To incorporate the temporal sequence information of log events, positional encoding is added to the log-level semantic representation vector:

[0052] in, It is the first An enhanced log event indicates that... This refers to the position code in the 1st position. The encoded vector at each position.

[0053] Subsequently, the enhanced log event representation is input into the log event-level Transformer layer. Through a self-attention mechanism, the global dependencies and temporal correlations between log events are captured to obtain the log event-level semantic aggregation result. :

[0054]

[0055] in, , It is a query matrix. It is a key matrix. It is a value matrix. This is the learnable parameter matrix. It is a scaling factor. It is a word correlation matrix.

[0056] The obtained log event-level semantic aggregation results The semantic dependencies, contextual logical relationships, and temporal evolution characteristics among log templates are comprehensively modeled. This layer can effectively characterize the propagation path and contextual association patterns of abnormal events in the log sequence, providing a stable and more discriminative semantic representation of events for subsequent higher-level semantic fusion.

[0057] 2.3 Cross-system semantic interaction and consistency modeling Building upon the log event-level semantic aggregation results, the third-layer Transformer further integrates broader contextual information to improve the global consistency and discriminative ability of log representation across different system environments. This layer primarily addresses cross-system log semantic modeling needs, achieving the construction of a unified representation space through system-level semantic interaction.

[0058] Let the log semantic matrices of the source system and the target system be respectively:

[0059] in, It is the first in the source system Log event-level semantic aggregation results of log events. It is the first in the target system The log event-level semantic aggregation result of each log event; that is, at this time, all log event-level semantic aggregation results constitute the log semantic matrix. , These represent the number of log event-level semantic aggregation results from the source system and the target system, respectively. Indicates the feature dimension.

[0060] In cross-system semantic interaction and consistency modeling, a cross-system attention mechanism (Cross-System Attention) is used. This mechanism employs the source system log event-level semantic aggregation result as the query vector and the target system log event-level semantic aggregation result as the key-value vector. Attention weights are used to calculate the semantic association strength between the source and target system logs. Based on this association strength, the event-level feature representations of the source system logs are weighted and fused, thereby achieving cross-system semantic interaction and alignment. The formula for modeling the semantic interaction between the source and target domains is as follows:

[0061] in, For learnable parameter matrix, It is the scaling factor.

[0062] The semantic association strength of log event-level semantic aggregation results is enhanced by leveraging the context of cross-system attention output. Residual fusion and normalization are then performed with the source system's log event-level semantic aggregation results to obtain a system-level semantic representation. :

[0063] in, It is a layer normalization function.

[0064] Cross-system semantic interaction and consistency modeling can establish deep semantic mapping relationships between logs from different systems, effectively alleviate the semantic offset problem caused by system differences, realize the sharing, alignment and compensation of log event semantics, thereby improving the generalization ability of the model in cross-system scenarios.

[0065] Step 3: Cross-system semantic alignment and transfer learning module The cross-system semantic alignment and transfer learning module of this invention is used to understand and solve the problem of inconsistent semantic space distribution of logs from different systems, and to achieve the unification and mapping of semantic features between the source system and the target system.

[0066] The cross-system semantic alignment and transfer learning module significantly improves the model's cross-domain adaptability through semantic distribution alignment and mechanism difference compensation mechanisms.

[0067] Semantic distribution alignment mechanism refers to optimizing the contrastive learning loss function within the transfer learning framework to make the overall distribution of the log semantic representations of the source system and the target system in the embedding space more consistent, thereby reducing the semantic offset between different systems.

[0068] The difference compensation mechanism refers to the introduction of a semantic preservation adjustment term during the semantic distribution alignment process to adjust the feature differences before and after alignment, preserve the system-specific semantic information, prevent feature expression degradation caused by over-alignment, and thus establish a balance between cross-domain consistency and system specificity.

[0069] 3.1 Feature Input Unit The feature input unit receives log event-level semantic aggregation results from the multi-layer Transformer representation learning module. and system-level semantic representation System-level semantic representation The results are integrated into the log event-level semantic aggregation results, and the two are jointly modeled.

[0070] Let the log semantic matrices of the source system and the target system be represented as follows:

[0071]

[0072]

[0073]

[0074] in, It is the first in the source system Log event-level semantic aggregation results of log events. It is the first in the target system Log event-level semantic aggregation results for each log event. It is the source system after fusing system-level semantic representation. Semantic vector representation of a log event, It is the target system after fusing system-level semantic representation. The semantic vector representation of each log event; here, the log event-level semantic aggregation result after fusing the system-level semantic representation constitutes the log semantic matrix. It is the system-level semantic modulation coefficient, used to control the degree of influence of system-level semantic representation on the event-level semantic aggregation result, and is set to 0.2 here. , These represent the system-level semantic representations of the source system and the target system, respectively. , These represent the number of log event-level semantic aggregation results from the source system and the target system, respectively. Indicates the feature dimension.

[0075] In the above manner, log event-level semantic vector representation ( , While maintaining fine-grained semantic information, it integrates global semantic context information, thereby providing more stable and globally consistent feature inputs for subsequent semantic alignment.

[0076] 3.2 Feature Normalization Layer To eliminate differences in characteristic scale and distribution between different systems, this invention... , Normalization and linear mapping operations are performed on the above to obtain the feature-standardized semantic vector representations of the source system and the target system, respectively. , :

[0077] in, It is the source system after fusing system-level semantic representation. Semantic vector representation of a log event, It is the target system after fusing system-level semantic representation. Semantic vector representation of a log event. Let be the linear transformation matrix between the source and target domains. This is a bias term.

[0078] Normalization ensures that the feature distributions of different systems are comparable at the same scale, providing a consistent spatial benchmark for subsequent semantic distance measurements.

[0079] 3.3 Semantic Alignment Training Unit Based on Transfer Learning Within the transfer learning framework, this unit performs alignment training on the log semantic representations of the source and target systems to reduce the semantic distribution differences between different systems and achieve cross-system semantic alignment.

[0080] Feature-normalized semantic vector representation of the source system Let be the log sample of the source system, and be the feature-normalized semantic vector representation of the target system. Let be the log sample of the target system. Cosine similarity is used to calculate the similarity between the log samples of the source system and the target system: in, Represents the similarity function; Adopting based on semantic alignment loss The definition is as follows:

[0081] in, Indicates the relationship between the target system and the source sample Semantically similar positive samples; This is a temperature coefficient used to adjust the contrast intensity; in the denominator... The total number of negative samples; This is the number of positive sample pairs used to calculate the contrastive loss, where N represents the number of samples in the source system. Indicates the relationship between the target system and the source sample Semantically dissimilar negative samples.

[0082] By optimizing semantic alignment loss This makes the semantic features of the source domain and the target domain more similar in the embedding space, thus completing the semantic distribution alignment.

[0083] 3.4 Semantic Preservation Adjustment Unit This invention introduces a semantic preservation adjustment term after semantic distribution alignment to prevent semantic information loss caused by over-alignment.

[0084] Let the difference between the log semantic matrix composed of the feature-normalized semantic vector representations and the semantically aligned feature log semantic matrix be used as the semantic preservation adjustment term:

[0085]

[0086] in, and These are the source and target system log semantic matrices after normalization and linear mapping operations as described in section 3.2. and These are the source system and target system log semantic matrices after semantic alignment as described in section 3.3. It is the first normalized and linearly mapped part of the source system. The feature-normalized semantic vector representation of each log event. It is the first normalized and linearly mapped operation in the target system. The feature-standardized semantic vector representation of each log event; at this point, , These represent the source system and the target system after the normalization and linear mapping operations in section 3.2, respectively.

[0087] By introducing balance parameters Combining semantic alignment loss and semantic preservation loss:

[0088] in, This is a balance parameter between semantic alignment loss and semantic preservation loss, used to adjust the weight relationship between cross-system semantic alignment and original semantic preservation. The value range is 0.01 to 1, with a preferred value of 0.1. When When a larger value is taken, the model tends to preserve the original semantic structure of the system; when When smaller values ​​are taken, the model focuses more on cross-system semantic alignment.

[0089] 3.5 Output and Effects During the training phase, the total loss function is used. As a constraint, the model parameters are optimized under the joint constraints of semantic alignment loss and semantic preservation loss, so that the log semantic representations of the source system and the target system are gradually mapped to a unified semantic space. After optimization convergence, the optimized semantic alignment module and the multi-layer Transformer representation learning module are obtained. The optimized semantic alignment module and the multi-layer Transformer representation learning module output a cross-system consistent log semantic matrix, denoted as . .

[0090] The cross-system consistent log semantic matrix It serves as input to an anomaly detection module based on a prototype network, supporting anomaly identification of different system log data in a unified semantic space.

[0091] This invention effectively overcomes the feature space shift problem caused by corpus differences in traditional methods, thereby improving the robustness and adaptability of cross-system log anomaly detection.

[0092] Step 4: Anomaly Detection Module Based on Prototype Network 4.1 Feature Input Unit This unit receives the cross-system consistent log semantic matrix output from the previous module. It is divided into a support set S and a query set Q:

[0093] in, It is the first The semantic vector representation of a log entry in a unified semantic space, the It has integrated log event-level semantic information with system-level semantic information and undergone cross-system semantic alignment processing; Indicates the first The semantic vector representation of each query log, its construction method is similar to Consistent; For its corresponding category label, It is the number of samples in the support set. This is the number of samples in the query set.

[0094] The support set is used for prototype calculation, and the query set is used for category determination.

[0095] 4.2 Prototype Building Block By calculating the mean vector of similar samples in the support set, the centroid of each category (i.e., the category prototype) is constructed:

[0096] in, Indicates belonging to the first The support set of the class is a sample set. The number of samples in this category. It is the first The semantic vector representation of a log entry in a unified semantic space, the It has integrated log event-level semantic information with system-level semantic information and undergone cross-system semantic alignment processing. Indicates category The semantic center vector.

[0097] The basic idea of ​​prototype networks is to create a prototype representation for each category. For a query requiring classification, the distance between the prototype vector of the category and the query point is calculated. This completes the construction of the prototype, allowing the sample distribution characteristics to be represented based on the category centers.

[0098] 4.3 Measurement Calculation Unit This unit is used to calculate the distance between the query sample and the prototype of each category, which is used to represent the similarity between the sample and the category center.

[0099] The distance metric function uses Euclidean distance, defined as follows:

[0100] in, Indicates the first The semantic vector representation of the query log, the It has integrated log event-level semantic information with system-level semantic information and undergone cross-system semantic alignment processing. Indicates category The semantic center vector.

[0101] The Euclidean distance reflects the degree of difference between a sample and the class center; the smaller the distance, the closer the sample is to the class.

[0102] 4.4 Anomaly Detection Unit In log anomaly detection tasks, the system typically only includes two categories: "normal" and "abnormal".

[0103] For the log sample to be detected, the anomaly score is defined by its distance from the normal category prototype: in, The anomaly score, calculated using the prototype building unit formula, is the semantic center vector of the normal category prototype. The higher the anomaly score, the more significantly the log deviates from the normal behavior pattern. Indicates the first The abnormal score of each log entry.

[0104] 4.5 Anomaly Detection Unit To achieve anomaly detection, this invention constructs a detection threshold based on the anomaly score distribution of normal samples:

[0105]

[0106]

[0107] in, This is the normal sample size. Indicates the first The semantic vector representation of a normal log entry in a unified semantic space, the It has integrated log event-level semantic information with system-level semantic information and undergone cross-system semantic alignment processing. It is the first Abnormal scores of normal samples. It is the mean of the abnormal scores of normal samples. It is the standard deviation of outlier scores. This is the adjustment coefficient, which controls the strictness of the threshold; it is set here. It is 3. It is the final anomaly detection threshold.

[0108] When abnormal scores Exceeding the threshold If the log sequence is then deemed abnormal, it is considered to be abnormal.

[0109] Example 2 In this embodiment of the invention, a prototype feedback adjustment unit is introduced into the cross-system semantic alignment and transfer learning module to further improve the consistency between cross-system semantic representation and anomaly detection tasks on the basis of completing semantic alignment training.

[0110] The prototype feedback unit uses the normal category prototypes built in the anomaly detection module based on the prototype network to constrain and optimize the semantic alignment process, so that the semantic alignment results not only satisfy cross-system semantic consistency, but also conform to the normal behavior pattern structure.

[0111] Specifically, let the log semantic vector output by the cross-system semantic alignment and transfer learning module be represented as follows: Let the normal category prototype be: It can be calculated using the prototype building unit formula and is the semantic center vector of the normal category prototype.

[0112] The prototype feedback constraint term is then defined as:

[0113] in: This represents the number of log samples. The Euclidean distance is used to measure the degree of deviation between the log semantic representation and the center of normal behavior.

[0114] Combining semantic alignment loss, semantic preservation loss, and prototype feedback constraint, the overall optimization objective function of this invention is defined as:

[0115] in, To preserve the weight parameters for semantics, These are the prototype feedback weight parameters.

[0116] Under these conditions, during the training phase, the total loss function described above is minimized. This allows the semantic alignment process to be guided by the normal behavioral pattern structure, thereby improving the stability and generalization ability of the cross-system log anomaly detection model.

[0117] It should be noted that the normal category prototype is not a fixed preset parameter, but is dynamically calculated from the normal samples in the support set in the current training round, and is used as a feedback constraint signal input to the semantic alignment module to participate in joint optimization, thereby forming a collaborative training mechanism between semantic alignment and anomaly detection.

[0118] Example 3 In this embodiment, the source system log set and the target system log set are respectively: Follow these steps to perform anomaly detection.

[0119] 1. Log preprocessing First, the date and time fields of each log entry are retained and converted to a standard time format for use as time feature input. Log levels (such as INFO, WARN, ERROR) are also preserved to enhance the ability to express anomalies. Second, the log content text is standardized: punctuation and special characters are removed, redundant information such as paths and random IDs is filtered out, and numerical entities with system differences, such as IP addresses and port numbers, are uniformly replaced with common identifiers. For example, "192.168.0.1" is replaced with "ip", and port numbers such as "3306" are replaced with "port". Furthermore, all letters are converted to lowercase, and spaces are used to separate words for easier subsequent word segmentation and modeling.

[0120] After the above preprocessing, the first A log entry can be represented as a sequence of terms:

[0121] in, For the preprocessed first 1 log entry; For the first in this log One standardized term; This represents the length of the terms in the log.

[0122] The preprocessed logs still retain the time, log level, and simplified semantic content.

[0123] Figure 4 In the example log templates, msb getService:<*>, <*> pod <*> in minion, and watch of <*> is Opened, <*> represents a wildcard for variable arguments.

[0124] 2. Log Semantic Embedding Based on Multi-Layer Transformer Representation Learning Module Based on the above, this invention employs a BERT-based dynamic word vector embedding method to embed each log entry from the support set and query set. and The mapping is transformed into a context-dependent sequence of word vectors, which can automatically adjust the semantic expression according to the context, ensuring consistency and transferability of semantic representations obtained in different system semantic environments. After processing, standardized semantic inputs for the source and target systems can be obtained respectively, providing a unified feature foundation for subsequent multi-layer Transformer representation learning, thereby achieving preliminary alignment of cross-system log semantics.

[0125] After completing the preprocessing stage described above, the log semantic representation extraction stage of the multi-layer Transformer representation learning module officially begins. During word-level semantic encoding, log-level weights and embedding vectors are fused with word vectors to enhance the model's ability to perceive information about different log severity levels. At this point, the source system logs and target system logs are input into the three-layer Transformer representation layer, yielding deep semantic vector features.

[0126] in, This represents a three-layer Transformer semantic extraction structure.

[0127] 3. Feature standardization processing To further eliminate the differences in feature distribution between the source system and the target system logs, the output of the log event-level Transformer layer of the multi-layer Transformer representation learning module, after incorporating cross-system semantic interaction and consistency modeling, is used as the input of the semantic alignment module, and this input is standardized and aligned.

[0128] Let the log characteristics of the source system and the target system be represented as follows:

[0129] in, It is the first in the source system Semantic vector representation of a log event, It is the first in the target system Semantic vector representation of a log event.

[0130] right and Apply linear mapping and normalization operations respectively:

[0131] in, Let be a trainable linear mapping matrix. This is a bias term.

[0132] 4. Transfer learning semantic alignment After standardization, cross-system log features are aligned using transfer learning.

[0133] First, a distribution alignment strategy is employed to ensure that the log characteristics of the source and target systems maintain consistency at the statistical distribution level. Taking the maximum mean difference as an example, the distribution alignment loss is:

[0134] Secondly, in the process of transfer learning alignment, based on The alignment loss function is:

[0135] in, Indicates the source sample Semantically similar positive samples; This is a temperature coefficient used to adjust the contrast intensity; in the denominator... The total number of negative samples. During semantic alignment optimization, a feedback constraint term formed by normal category prototypes is introduced to structurally guide the unified semantic representation, thereby improving the consistency between the alignment results and the anomaly detection task.

[0136] 5. Anomaly Detection Model Based on Prototype Networks After semantic alignment, log features in the unified semantic space are divided into support set and query set for building an anomaly detection model based on the prototype network.

[0137] For samples of the same class in the support set, calculate their feature mean as the prototype vector for that class:

[0138] in, Indicates belonging to the first The support set of the class is a sample set. For feature extraction function, Indicates category The semantic center vector.

[0139] In practical applications, when the model is migrated to a new target system At this time, only a small number of labeled log samples are needed to build new category prototypes without retraining the overall model.

[0140] The system uses query log samples Compared with normal category prototypes Calculate the anomaly score based on the distance between them: , and with preset threshold :

[0141]

[0142]

[0143] The system compares the scores, and when the anomaly score exceeds the threshold, the log or log sequence is determined to be abnormal.

[0144] in, For adjustment coefficients, This represents the mean score of abnormal samples from normal samples. The standard deviation is denoted as .

[0145] The features of the above embodiments can be combined with each other without contradiction.

[0146] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning, characterized in that, The method includes the following: The raw log data from different systems is preprocessed in a unified manner to extract key fields including time, log level and text content, and standardized log input is obtained through template and word segmentation to obtain log word vector sequences. Construct a multi-layer Transformer representation learning module, including a word-level Transformer layer, a log event-level Transformer layer, and cross-system semantic interaction and consistency modeling; Introduce log-level weights in the word-level Transformer layer. and log-level embedding vector The word vectors of the logs are fused according to the following formula, and the fused word vectors are used to construct an enhanced log word vector sequence. ; ， ， ， in, It is the first The original word vector representation of each word. It is the first The word vector representation after word enhancement; level represents the log level, ERROR represents an error, WARN represents a warning, and INFO represents information; It is the first The embedding vector of each word at the log level; It is the number of words contained in the log; It is the first The weight of each word at the log level; The enhanced log word vector sequence After processing by a multi-head self-attention mechanism, the vectors are then normalized by an additive layer with the word vector sequence of the current log, resulting in the output representation matrix of the log word-level Transformer. ; Through aggregation functions Compression is performed to obtain a log-level semantic representation vector. Positional encoding is added to the log-level semantic representation vector to obtain an enhanced log event representation. This enhanced log event representation is then input into a log event-level Transformer layer, where a self-attention mechanism captures global dependencies and temporal relationships between log events, yielding a log event-level semantic aggregation result. ; In cross-system semantic interaction and consistency modeling, the event-level semantic aggregation results of the source system logs are used as query vectors, and the event-level semantic aggregation results of the target system logs are used as key-value vectors. Attention weights are used to calculate the semantic association strength between the source and target system logs. Based on this association strength, the event-level semantic aggregation results of the source system logs are weighted and fused, thereby achieving cross-system semantic interaction and alignment, and obtaining a system-level semantic representation. ; Within the transfer learning framework, the contrastive learning loss function is optimized to semantically align the log semantic representations of the source and target systems in the embedding space, thereby obtaining a consistent log semantic matrix across systems. Anomaly detection is performed using an anomaly detection model based on a consistent log semantic matrix across systems.

2. The method according to claim 1, characterized in that, The semantic alignment process is as follows: The log event-level semantic aggregation results of the multi-layer Transformer representation learning module of the source system and the target system. and system-level semantic representation Weighted fusion is performed to obtain the fused semantic vector representations of the source system and the target system, and feature standardization is performed to obtain the feature-standardized semantic vector representation. Based on the feature-standardized semantic vector representation, the similarity between positive and negative samples of the source system and the target system is calculated. The InfoNCE loss is used for alignment training, so that the semantic features of the source domain and the target domain tend to be close in the embedding space, thus completing the semantic distribution alignment. After semantic distribution alignment, a semantic preservation adjustment term is introduced, and the formula for calculating the semantic preservation loss is as follows: ， in, and These are the source system and target system log semantic matrices after normalization and linear mapping operations, respectively. and These are the semantic matrices of the source and target system logs after semantic alignment. Optimization training is performed under the joint constraints of semantic alignment loss and semantic preservation loss to obtain a consistent log semantic matrix across systems.

3. The method according to claim 1, characterized in that, The anomaly detection model adopts a prototype network-based anomaly detection model. It uses the cross-system consistent log semantic matrix as input sample, calculates the distance between the input sample and the normal category prototype to obtain an anomaly score, and completes the anomaly judgment based on a preset threshold.

4. The method according to claim 3, characterized in that, In the semantic alignment process, a prototype feedback adjustment unit is introduced to constrain and optimize the alignment results using normal category prototypes, so that the semantic alignment results not only meet cross-system semantic consistency, but also conform to the normal behavior pattern structure.

5. The method according to claim 4, characterized in that, In the prototype feedback adjustment unit, the prototype feedback constraint term is: ， in: This represents the number of log samples. This represents the Euclidean distance, a constraint term used to measure the degree of deviation between the log semantic representation and the center of normal behavior; This is the semantically aligned log semantic vector representation. The semantic center vector of the normal category prototype.

6. A cross-system log anomaly detection method based on multi-layer Transformer semantic alignment and transfer learning, characterized in that, The method includes the following: The raw log data from different systems is preprocessed in a unified manner to extract key fields including time, log level and text content, and standardized log input is obtained through template and word segmentation to obtain log word vector sequences. An anomaly detection network is constructed, comprising a multi-layer Transformer representation learning module, a semantic alignment module, and an anomaly detection module based on a prototype network. The multi-layer Transformer representation learning module is implemented using a three-layer Transformer, including a word-level Transformer layer, a log event-level Transformer layer, and cross-system semantic interaction and consistency modeling. The multi-layer Transformer represents the output of the log event-level Transformer layer of the learning module, which integrates cross-system semantic interaction and consistency modeling, and serves as the input to the semantic alignment module. The semantic alignment module outputs a cross-system consistent log semantic matrix, which is used as input to the prototype network-based anomaly detection module; the prototype network-based anomaly detection module outputs anomaly scores, and the results are used to determine the anomaly scores. The total loss function of the anomaly detection network for: ， in, To preserve weight parameters for semantics, For prototype feedback weight parameters; For prototype feedback constraints; Loss due to semantic preservation; For semantic alignment loss; The anomaly detection network is trained using the total loss function, and the trained anomaly detection network is used for cross-system log anomaly detection.

7. A cross-system log anomaly detection system based on multi-layer Transformer semantic alignment and transfer learning, characterized in that, The system performs the method described in any one of claims 1-6.

Images