Small base station log anomaly detection method and system fusing LSTM and Transform

Abstract

The invention relates to a small base station log anomaly detection method and a small base station log anomaly detection system fusing LSTM (Long Short Term Memory) and Transform. The method comprises the following steps: collecting original logs of each specialty from base station equipment; splitting and analyzing each original log to obtain a log template corresponding to the original log; constructing a log vector dictionary; dividing a log template corresponding to the original log into a log sequence through a sliding window mechanism, and obtaining a log sequence semantic vector in combination with the log vector dictionary; the method comprises the steps that a log sequence semantic vector is input into a BiLSTM-Transform model, a BiLSTM-Transform model is constructed, the log sequence semantic vector is input into the BiLSTM-Transform model to be trained, the trained BiLSTM-Transform model is obtained, and log anomaly detection is carried out through the trained BiLSTM-Transform model. According to the method, short-time dependence is captured through LSTM, a global sequence relation is captured through Transform, and log features are fully captured, so that the model has anti-noise capability, and meanwhile, multi-modal features can be fully fused to complete log anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and system for detecting anomalies in small base station logs that integrates LSTM and Transformer. Background Technology

[0002] With the development of 5G and the Internet of Things (IoT), the number of small cells has surged, forming ultra-dense networks. Each base station can generate approximately 50GB of logs per hour, covering dynamic information such as device status, user connections, and traffic load. Traditional manual inspections cannot handle this volume of data, necessitating automated anomaly detection technology to improve operational efficiency. In current communication system operations and maintenance, fault location is extremely challenging for staff. Maintenance personnel spend 90% of their time on fault location, resulting in low efficiency. Fault location is highly dependent on experts; 90% of problems cannot be solved by non-experts, placing immense pressure on on-site maintenance. Frontline staff generally prefer to directly seek help from experts, hindering their own knowledge accumulation and professional development. Therefore, a complete and automated fault location method is urgently needed, leveraging artificial intelligence for automated log analysis to replace simple rule-based fault location. This is a pressing process solution in the current communication system operations and maintenance field.

[0003] Existing log anomaly detection methods typically involve four core steps: log collection, log parsing, feature extraction, and anomaly detection. Log collection involves acquiring relevant logs generated from network devices, operating systems, and service programs. Log parsing transforms the collected unstructured log data into structured data. This process involves identifying known log events, extracting a unified template, and extracting parameters to construct a log sequence for the detection model. Feature extraction uses machine learning or statistical analysis techniques to analyze and mine the inherent feature information of the log data. Anomaly detection inputs the extracted log features (vector representation) into a machine learning or deep learning model for training, generating an anomaly detection model. This model can be used for anomaly identification in subsequent logs. Supervised learning requires a large amount of manually labeled data, but base station logs are often numerous and diverse, requiring significant manpower for labeling, making it impractical for anomaly detection in actual base station logs. Unsupervised learning, on the other hand, only requires training on normal logs and uses a regression task to identify log anomalies. This method is feasible for anomaly detection in actual base station logs and therefore warrants further investigation. Log data is characterized by its unstructured, diverse, and unstable nature, along with strong contextual dependencies, posing a significant challenge to log sequence anomaly detection. Previous studies, such as Support Vector Machines (SVM), Principal Component Analysis (PCA), and Invariant Mining (IM), represent log sequences as log counting vectors, ignoring the semantic information within log events and failing to address the ever-evolving log statements. Currently, earlier rule-based and statistical log anomaly detection methods are no longer sufficient to meet the rapid development and iteration of software systems. Machine learning-based log anomaly detection suffers from weaker feature extraction capabilities, poor adaptability, higher manual costs, and lower accuracy compared to deep learning. Therefore, current research on log anomaly detection focuses on deep learning methods. However, real-world logs often contain numerous irrelevant parameters that interfere with key semantic extraction, and existing methods often neglect denoising the log sequence, resulting in accuracy levels that fall short of ideal standards.

[0004] Log parsing is often a crucial step in log anomaly detection, and its accuracy directly impacts the accuracy of downstream tasks. Base station logs, in particular, are highly interdisciplinary. Existing log parsers often fail to identify parameters within the logs, potentially affecting downstream analysis. Large Language Models (LLMs) perform well in text and code-related tasks, and logs, which can be intuitively seen as a combination of text and code-like variables, can be leveraged to treat log parsing as a translation task, transforming log content into log templates.

[0005] Meanwhile, most existing anomaly detection models are RNN models. RNN models may suffer from gradient vanishing or exploding problems when processing long sequences, making it difficult to learn the dependencies in long sequences and thus affecting the final accuracy. Summary of the Invention

[0006] Therefore, it is necessary to provide a more accurate method and system for detecting anomalies in small cell logs that integrates LSTM and Transformer to address the aforementioned technical problems.

[0007] One method for detecting anomalies in small cell logs that integrates LSTM and Transformer includes: S1: Collect raw logs from various specialties from base station equipment; S2: Split and parse each original log entry to obtain the log template corresponding to the original log; S3: Construct a log vector dictionary; S4: The log templates corresponding to the original logs are divided into log sequences using a sliding window mechanism, and the log sequence semantic vectors are obtained by combining them with the log vector dictionary; S5: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model for log anomaly detection.

[0008] Further, in step S2, the Flan-T5 model is constructed. The Flan-T5 model is used to split and parse each original log to obtain the log template corresponding to the original log. The Flan-T5 model is a pre-trained language model based on the T5 model and has been fine-tuned for the task to improve its performance in various natural language processing tasks. The T5 model is a text generation framework designed to transform all NLP tasks into text-to-text tasks, simplifying the application of the model in multi-task learning.

[0009] Furthermore, constructing the Flan-T5 model also includes: S2.1: Fifty training samples are formed by sampling a small number of logs and their related log templates through clustering and log sampling techniques; S2.2: Design a prompt template. Prompts are user-provided inputs, including: queries, instructions, or questions; S2.3: Train the Flan-T5 model using training samples and prompt templates. During training, provide the task description, instructions, inputs, and responses to the Flan-T5 model (LLM). During parsing, only provide the description, instructions, and inputs, and require the Flan-T5 model to generate responses to obtain the log template.

[0010] Furthermore, step S3 specifically includes: A sentence-BERT model is constructed and pre-trained to obtain a fine-tuned sentence-BERT model. This fine-tuned model is then used to vectorize log templates, constructing a log vector dictionary. Furthermore, a sentence-BERT model is constructed and pre-trained, resulting in a fine-tuned sentence-BERT model, including: S3.1 Training Corpus Preparation: The log template generated by flan-t5-small is used as the basic corpus; S3.2 Data Augmentation Strategy: Apply an improved random deletion scheme to randomly delete 15%-25% of non-key tokens in each training sample to ensure that the template retains complete semantics after deletion; add synonym replacement enhancement, build a log-specific thesaurus, and apply random replacement in 20% of the training samples; S3.3 Constructing Training Samples: Define a function to generate self-supervised training samples, generate positive sample sentence pairs, generate negative sample sentence pairs, and all samples contain similarity labels, with positive examples being 1 and negative examples being 0. S3.4 Model Configuration and Training: Load the pre-trained MiniLM model, which is built on the Sentence-BERT framework and adopts a dual-tower (Siamese Network) structure: the input text is mapped to a fixed-dimensional vector (such as 384-dimensional) through a Transformer encoder with shared weights, and then the sentence vector is output through a pooling layer; The learning rate is preset to 2e-5 and can be dynamically adjusted; the batch size is set to 8 to 32, and the training epochs are 100; the training stops in real time based on the change of the loss value; the contrast loss function is applied to optimize the similarity score of sentence pairs, and the pooling layer is modified to the CLS strategy. Finally, the fine-tuned sentence-bert model is trained and output.

[0011] Furthermore, in step S4, dividing the log template corresponding to the original log into log sequences using the sliding window mechanism also includes: Wavelet denoising is performed using a sliding window mechanism to remove noise from the data and retain useful signal components. The denoised data is then normalized to obtain the log sequence.

[0012] Further, in step S5, the BiLSTM-Transformer model includes: Model Denoising Layer: A three-layer convolutional denoising layer is constructed. The preprocessed data is then subjected to convolutional denoising to enhance local features. Residual learning is used to construct the feature input data, which is composed of the denoised data and the undenoised data in a 7:3 ratio. The residual connection is used to balance denoising and information preservation, thus providing higher quality feature representations for subsequent model learning. Feature Transformation Layer: The feature transformation layer is the first learnable transformation layer of the model, which maps the original embedding space to the model optimization space, creating feature representations that are more suitable for temporal modeling; Bidirectional LSTM layer: Combining past and future information, it provides a complete context for each time step, enhances the temporal awareness of features, performs fine-grained local temporal modeling, outputs features rich in temporal information, and provides optimized input for Transformer; Transformer layer: The output of LSTM is encoded with positional codes. The positionally encoded sequence is then input into the Transformer encoder, which includes a multi-head self-attention network and a feedforward network. A triangular mask is generated to prevent future information leakage. Based on the temporal features extracted by LSTM, global dependencies in the sequence are further captured. Attention mechanism layer: The representation of each time step of the Transformer output is weighted, and the time step most important to the current prediction task is selected. A fixed-size context vector is obtained by weighted summation. This vector integrates important information from the entire sequence and is used for the final prediction. Output layer: The context vector obtained through the attention mechanism is passed through two fully connected layers to generate the final output; Loss function and optimization strategy: The total loss consists of two parts: prediction loss and denoising loss. Prediction loss is the mean square error between the predicted output and the true label, and denoising loss is the mean square error between the output of the denoising layer and the original input.

[0013] Further, in step S5, the log sequence semantic vector is input into the BiLSTM-Transformer model for training, resulting in the trained BiLSTM-Transformer model as follows: Data preprocessing: Wavelet denoising: For each feature dimension Handle independently:

[0014] in It is wavelet transform. It is a soft thresholding function, where d is the feature dimension; Normalization process:

[0015] Where x is the data after wavelet denoising, i.e., the above.

[0016] enter B is the batch size, and W is the window size; Perform a convolution operation on the input X, specifically by transposing X from [B,W,384] to [B,384,W]. Through convolutional blocks: Convolution 1: C1 = Conv1d (in_channels = 384, conv_channels = 96, kernel_size = 5, padding = 2) ReLU activation: A1 = ReLU(C1) Batch normalization: B1 = BatchNorm1d(A1) Dropout: D1 = Dropout (B1, p = 0.2) Convolution 2: C2 = Conv1d (in_channels =96, conv_channels = 96, kernel_size = 3, padding = 1) ReLU activation: A2 = ReLU ( C2 ) Convolution 3: C3 = Conv1d (in_channels =96, conv_channels = 384, kernel_size = 3, padding = 1) Batch normalization: B2 = BatchNorm1d(A3) Transpose the data back to [B,W,384] and use residual connections to construct the final denoised data: denoised_X = 0.7 × B² + 0.3 × X; The output is as follows:

[0017]

[0018] in It is the ReLU activation function. It is the output obtained after passing through the convolution block. This is the final denoised data; Feature transformation layer:

[0019] in It is the input projection layer weight matrix; It is the bias vector of the input projection layer; It is the feature sequence after projection and activation; Where h is the sequence length and h is the hidden layer size; Bidirectional LSTM layer:

[0020]

[0021] in This refers to the hidden state of the LSTM at time step t. This refers to the cell state of LSTM at time step t; It is the input of the previous time step, which is the input data received by the LSTM unit at the current time t; it comes from the output of the previous module at time t-1. It is the hidden state of the previous time step. It represents the cell state at the previous time step; It is a forward LSTM that processes the entire sequence from left to right. Then, the set of hidden states output at each time step; Backward LSTM processes the entire sequence from right to left. Then, the set of hidden states output at each time step; The bidirectional outputs are concatenated; because the forward and backward outputs are concatenated, the feature dimension becomes 2h; this contains the past and future contextual information for each time step;

[0022]

[0023]

[0024] PE: Position Encoding Matrix; Since the Transformer itself does not have a loop or convolution structure, it cannot perceive the order of the sequence, so position information needs to be manually injected. pos: The position index in the sequence (0, 1, 2, ..., τ-1); i: Dimension index (0, 1, 2, ..., d-1), used to alternate between sine and cosine functions; d: The dimension of the positional encoding, which must be added to the matrix. The feature dimensions are consistent, i.e., d = 2h; Using Dropout, we can randomly mask out a portion of the location information here to prevent overfitting; The sequence representation after adding positional information; this is the input to the Transformer encoder. Transformer layer:

[0025] TransformerEncoder: A module consisting of multiple stacked TransformerEncoder Layers; each layer contains: Multi-head self-attention: This mechanism allows each element in a sequence to pay attention to all other elements in the sequence, efficiently capturing global dependencies. Feedforward neural networks: perform nonlinear transformations on the output of self-attention; Dimensions and Inputs Consistent; Attention mechanism layer:

[0026] The weight matrix for the attention mechanism; used to project the Transformer's output H_trans onto a new space. Bias vector of attention mechanism v: Weight vector of the attention scoring function Attention weights: This is a vector calculated using the softmax function, representing the importance of each time step to the final task. The sum of all weights is 1.

[0027] c: Context vector; it is the weighted sum of the entire sequence, a condensation and summary of the information that the model considers most helpful for making the final prediction; Output layer:

[0028]

[0029] A 384-dimensional vector is output through two fully connected layers; It is the output of the first fully connected layer. This is the weight matrix of the first fully connected layer. It is the bias vector of the first fully connected layer; This is the final predicted output. This is the weight matrix of the second fully connected layer. It is the bias vector of the second fully connected layer; Loss function:

[0030]

[0031]

[0032] It is the main prediction loss, using mean squared error, used to measure the difference between the model's final output and the true value; It is the actual value. It is a predicted value; It is the denoising loss; it measures the ability of the denoising layer to reconstruct the input sequence. It is the original input sequence; It is the denoised sequence; It is an auxiliary loss function, its purpose being to guide the denoising layer to learn how to effectively denoise and extract features from the input data, thereby providing input information for the model's main task; (The formula is incomplete and requires further context.) This indicates that the mean value (MSE) is calculated over the entire input sequence. This is the total loss, where 0.1 is the weighting coefficient; this is a hyperparameter used to balance the importance of the main loss and auxiliary losses. Loss function during testing phase: .

[0033] Furthermore, in step S5, log anomaly detection is performed using the trained BiLSTM-Transformer model, specifically as follows: The loss function uses a single MSE function; Set a threshold; if the loss function MSE is greater than the threshold, it is considered abnormal; otherwise, it is considered normal.

[0034] Furthermore, this invention provides a small cell log anomaly detection system that integrates LSTM and Transformer, comprising: Data Acquisition Module: Collects raw logs from various disciplines from base station equipment; Splitting and parsing module: Splits and parses each original log entry to obtain the log template corresponding to the original log; Vector dictionary module: Constructs a log vector dictionary; Semantic Vector Module: The log template corresponding to the original log is divided into log sequences through a sliding window mechanism, and the log sequence semantic vector is obtained by combining the log vector dictionary; Model detection module: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model to detect log anomalies.

[0035] Therefore, this invention proposes a method that captures short-term dependencies using LSTM and global sequence relationships using Transformer, fully capturing log features to make the model robust to noise, while also fully integrating multimodal features to complete log anomaly detection. Attached Figure Description

[0036] Figure 1 This is a flowchart of a small cell log anomaly detection method that integrates LSTM and Transformer in one embodiment. Figure 2 Here is a block diagram of a small cell log anomaly detection system that integrates LSTM and Transformer in one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] Example 1: This embodiment provides, as follows: Figure 1 The method for detecting anomalies in small cell logs, which integrates LSTM and Transformer, includes: S1: Collect raw logs from various specialties from base station equipment; S2: Split and parse each original log entry to obtain the log template corresponding to the original log; S3: Construct a log vector dictionary; S4: The log templates corresponding to the original logs are divided into log sequences using a sliding window mechanism, and the log sequence semantic vectors are obtained by combining them with the log vector dictionary; S5: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model for log anomaly detection.

[0038] This embodiment proposes a method that uses LSTM to capture short-term dependencies and Transformer to capture global sequence relationships, fully capturing log features to make the model robust to noise and fully integrate multimodal features to complete log anomaly detection.

[0039] Example 2: This embodiment further discloses information based on Embodiment 1: Further, in step S2, the Flan-T5 model is constructed. The Flan-T5 model is used to split and parse each original log to obtain the log template corresponding to the original log. The Flan-T5 model is a pre-trained language model based on the T5 model and has been fine-tuned for the task to improve its performance in various natural language processing tasks. The T5 model is a text generation framework designed to transform all NLP tasks into text-to-text tasks, simplifying the application of the model in multi-task learning.

[0040] Furthermore, constructing the Flan-T5 model also includes: S2.1: Fifty training samples are formed by sampling a small number of logs and their related log templates through clustering and log sampling techniques; S2.2: Design a prompt template. Prompts are user-provided inputs, including: queries, instructions, or questions; S2.3: Train the Flan-T5 model using training samples and prompt templates. During training, provide the task description, instructions, inputs, and responses to the Flan-T5 model (LLM). During parsing, only provide the description, instructions, and inputs, and require the Flan-T5 model to generate responses to obtain the log template.

[0041] Furthermore, step S3 specifically includes: A sentence-BERT model is constructed and pre-trained to obtain a fine-tuned sentence-BERT model. This fine-tuned model is then used to vectorize log templates, constructing a log vector dictionary. Furthermore, a sentence-BERT model is constructed and pre-trained, resulting in a fine-tuned sentence-BERT model, including: S3.1 Training Corpus Preparation: The log template generated by flan-t5-small is used as the basic corpus; S3.2 Data Augmentation Strategy: Apply an improved random deletion scheme to randomly delete 15%-25% of non-key tokens in each training sample to ensure that the template retains complete semantics after deletion; add synonym replacement enhancement, build a log-specific thesaurus, and apply random replacement in 20% of the training samples; S3.3 Constructing Training Samples: Define a function to generate self-supervised training samples, generate positive sample sentence pairs, generate negative sample sentence pairs, and all samples contain similarity labels, with positive examples being 1 and negative examples being 0. S3.4 Model Configuration and Training: Load the pre-trained MiniLM model, which is built on the Sentence-BERT framework and adopts a dual-tower (Siamese Network) structure: the input text is mapped to a fixed-dimensional vector (such as 384-dimensional) through a Transformer encoder with shared weights, and then the sentence vector is output through a pooling layer; The learning rate is preset to 2e-5 and can be dynamically adjusted; the batch size is set to 8 to 32, and the training epochs are 100; the training stops in real time based on the change of the loss value; the contrast loss function is applied to optimize the similarity score of sentence pairs, and the pooling layer is modified to the CLS strategy. Finally, the fine-tuned sentence-bert model is trained and output.

[0042] Furthermore, in step S4, dividing the log template corresponding to the original log into log sequences using the sliding window mechanism also includes: Wavelet denoising is performed using a sliding window mechanism to remove noise from the data and retain useful signal components. The denoised data is then normalized to obtain the log sequence.

[0043] Further, in step S5, the BiLSTM-Transformer model includes: Model Denoising Layer: A three-layer convolutional denoising layer is constructed. The preprocessed data is then subjected to convolutional denoising to enhance local features. Residual learning is used to construct the feature input data, which is composed of the denoised data and the undenoised data in a 7:3 ratio. The residual connection is used to balance denoising and information preservation, thus providing higher quality feature representations for subsequent model learning. Feature Transformation Layer: The feature transformation layer is the first learnable transformation layer of the model, which maps the original embedding space to the model optimization space, creating feature representations that are more suitable for temporal modeling; Bidirectional LSTM layer: Combining past and future information, it provides a complete context for each time step, enhances the temporal awareness of features, performs fine-grained local temporal modeling, outputs features rich in temporal information, and provides optimized input for Transformer; Transformer layer: The output of LSTM is encoded with positional codes. The positionally encoded sequence is then input into the Transformer encoder, which includes a multi-head self-attention network and a feedforward network. A triangular mask is generated to prevent future information leakage. Based on the temporal features extracted by LSTM, global dependencies in the sequence are further captured. Attention mechanism layer: The representation of each time step of the Transformer output is weighted, and the time step most important to the current prediction task is selected. A fixed-size context vector is obtained by weighted summation. This vector integrates important information from the entire sequence and is used for the final prediction. Output layer: The context vector obtained through the attention mechanism is passed through two fully connected layers to generate the final output; Loss function and optimization strategy: The total loss consists of two parts: prediction loss and denoising loss. Prediction loss is the mean square error between the predicted output and the true label, and denoising loss is the mean square error between the output of the denoising layer and the original input.

[0044] Further, in step S5, the log sequence semantic vector is input into the BiLSTM-Transformer model for training, resulting in the trained BiLSTM-Transformer model as follows: Data preprocessing: Wavelet denoising: For each feature dimension Handle independently:

[0045] in It is wavelet transform. It is a soft thresholding function, where d is the feature dimension; Normalization process:

[0046] Where x is the data after wavelet denoising, i.e., the above.

[0047] enter B is the batch size, and W is the window size; Perform a convolution operation on the input X, specifically by transposing X from [B,W,384] to [B,384,W]. Through convolutional blocks: Convolution 1: C1 = Conv1d (in_channels = 384, conv_channels = 96, kernel_size = 5, padding = 2) ReLU activation: A1 = ReLU(C1) Batch normalization: B1 = BatchNorm1d(A1) Dropout: D1 = Dropout (B1, p = 0.2) Convolution 2: C2 = Conv1d (in_channels =96, conv_channels = 96, kernel_size = 3, padding = 1) ReLU activation: A2 = ReLU ( C2 ) Convolution 3: C3 = Conv1d (in_channels =96, conv_channels = 384, kernel_size = 3, padding = 1) Batch normalization: B2 = BatchNorm1d(A3) Transpose the data back to [B,W,384] and use residual connections to construct the final denoised data: denoised_X = 0.7 × B² + 0.3 × X; The output is as follows:

[0048]

[0049] in It is the ReLU activation function. It is the output obtained after passing through the convolution block. This is the final denoised data; Feature transformation layer:

[0050] in It is the input projection layer weight matrix; It is the bias vector of the input projection layer; It is the feature sequence after projection and activation; Where h is the sequence length and h is the hidden layer size; Bidirectional LSTM layer:

[0051]

[0052] in This refers to the hidden state of the LSTM at time step t. This refers to the cell state of LSTM at time step t; It is the input of the previous time step, which is the input data received by the LSTM unit at the current time t; it comes from the output of the previous module at time t-1. It is the hidden state of the previous time step. It represents the cell state at the previous time step; It is a forward LSTM that processes the entire sequence from left to right. Then, the set of hidden states output at each time step; Backward LSTM processes the entire sequence from right to left. Then, the set of hidden states output at each time step; The bidirectional outputs are concatenated; because the forward and backward outputs are concatenated, the feature dimension becomes 2h; this contains the past and future contextual information for each time step;

[0053]

[0054]

[0055] PE: Position Encoding Matrix; Since the Transformer itself does not have a loop or convolution structure, it cannot perceive the order of the sequence, so position information needs to be manually injected. pos: The position index in the sequence (0, 1, 2, ..., τ-1); i: Dimension index (0, 1, 2, ..., d-1), used to alternate between sine and cosine functions; d: The dimension of the positional encoding, which must be added to the matrix. The feature dimensions are consistent, i.e., d = 2h; Using Dropout, we can randomly mask out a portion of the location information here to prevent overfitting; The sequence representation after adding positional information; this is the input to the Transformer encoder. Transformer layer:

[0056] TransformerEncoder: A module consisting of multiple stacked TransformerEncoder Layers; each layer contains: Multi-head self-attention: This mechanism allows each element in a sequence to pay attention to all other elements in the sequence, efficiently capturing global dependencies. Feedforward neural networks: perform nonlinear transformations on the output of self-attention; Dimensions and Inputs Consistent; Attention mechanism layer:

[0057] The weight matrix for the attention mechanism; used to project the Transformer's output H_trans onto a new space. Bias vector of attention mechanism v: Weight vector of the attention scoring function Attention weights: This is a vector calculated using the softmax function, representing the importance of each time step to the final task. The sum of all weights is 1.

[0058] c: Context vector; it is the weighted sum of the entire sequence, a condensation and summary of the information that the model considers most helpful for making the final prediction; Output layer:

[0059]

[0060] A 384-dimensional vector is output through two fully connected layers; It is the output of the first fully connected layer. This is the weight matrix of the first fully connected layer. It is the bias vector of the first fully connected layer; This is the final predicted output. This is the weight matrix of the second fully connected layer. It is the bias vector of the second fully connected layer; Loss function:

[0061]

[0062]

[0063] It is the main prediction loss, using mean squared error, used to measure the difference between the model's final output and the true value; It is the actual value. It is a predicted value; It is the denoising loss; it measures the ability of the denoising layer to reconstruct the input sequence. It is the original input sequence; It is the denoised sequence; It is an auxiliary loss function, its purpose being to guide the denoising layer to learn how to effectively denoise and extract features from the input data, thereby providing input information for the model's main task; (The formula is incomplete and requires further context.) This indicates that the mean value (MSE) is calculated over the entire input sequence. This is the total loss, where 0.1 is the weighting coefficient; this is a hyperparameter used to balance the importance of the main loss and auxiliary losses. Loss function during testing phase: .

[0064] Furthermore, in step S5, log anomaly detection is performed using the trained BiLSTM-Transformer model, specifically as follows: The loss function uses a single MSE function; Set a threshold; if the loss function MSE is greater than the threshold, it is considered abnormal; otherwise, it is considered normal.

[0065] This embodiment proposes a method that uses LSTM to capture short-term dependencies and Transformer to capture global sequence relationships, fully capturing log features to make the model robust to noise and fully integrate multimodal features to complete log anomaly detection.

[0066] Example 3: This embodiment provides, as follows: Figure 2 The small cell log anomaly detection system shown includes: Data Acquisition Module: Collects raw logs from various disciplines from base station equipment; Splitting and parsing module: Splits and parses each original log entry to obtain the log template corresponding to the original log; Vector dictionary module: Constructs a log vector dictionary; Semantic Vector Module: The log template corresponding to the original log is divided into log sequences through a sliding window mechanism, and the log sequence semantic vector is obtained by combining the log vector dictionary; Model detection module: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model to detect log anomalies.

[0067] This embodiment proposes a method that uses LSTM to capture short-term dependencies and Transformer to capture global sequence relationships, fully capturing log features to make the model robust to noise and fully integrate multimodal features to complete log anomaly detection.

[0068] Example 4: A method for detecting log anomalies, comprising the following steps: Step 1: Collect logs: Collect raw logs from various specialties from the base station equipment.

[0069] Step 2, Data Preprocessing and Log Parsing: Each original log entry is split and LLM is used to parse the log events to obtain the log template corresponding to the log entry.

[0070] Step 3, Semantic Vectorization: Pre-train the Sentence-BERT model and use the fine-tuned Sentence-BERT to vectorize the log model, constructing a log vector dictionary.

[0071] Step 4: Reorder the log sequence based on log timestamps and specific data items of log entries. Use a sliding window mechanism to divide all log entries into log sequences, and combine this with the semantic vectorization method in Step 3 to obtain the log sequence semantic vector. Step 5, Anomaly Detection: Input the log sequence semantic vector into the Denoising-BiLSTM-Trans model for training to obtain the trained DBLTlog model. During the testing phase, the trained DBLTlog model is used for log anomaly detection.

[0072] Specifically, the log parsing in step 2 uses the T5logs method, which is a text-to-text log parsing method based on Flan-T5 (Fine-tuned Language Net T5). The Flan-T5 model is essentially a pre-trained language model based on the T5 (Text-to-Text Transfer Transformer) model, fine-tuned for specific tasks to improve its performance across various natural language processing tasks. T5 is a unified text generation framework designed to transform all NLP tasks into text-to-text tasks, simplifying the application of models in multi-task learning.

[0073] Includes the following steps: S1. First, use clustering and log sampling techniques to sample a small number of logs and their related log templates to form fifty training samples for training the LLM.

[0074] S2. After generating training samples, design a prompt template. The prompt is user-provided input, such as a query, instruction, or question, used to guide large language models (LLMs) and indicate their behavior in a specific task. The prompt template is shown below: "Parse the raw log to log template: '{Raw log}'." "{Log template}" S3. During training, provide the task description, instructions, inputs, and responses to the LLM. During parsing, only provide the description, instructions, and inputs, and request the LLM to generate a response, resulting in a log template statement.

[0075] Further, in step 3, the Sentence-BERT model is fine-tuned using the log template statements obtained in step 2 to generate the V-SBERT (Vectors Sentence BERT) model, and a semantic dictionary is constructed. The V-SBERT model is then used for template semantic matching, sorting the log templates by timestamp and converting them into log template semantic vectors. The specific steps are as follows: S1. Training Corpus Preparation: The log templates generated by flan-t5-small are used as the basic corpus. Strict quality control is implemented to filter out low-quality templates (such as those with a length of <5 tokens or >256 tokens). The semantic accuracy of high-frequency templates (Top 1000 templates) is appropriately verified manually.

[0076] S2. Data Augmentation Strategy: Apply an improved random deletion scheme to randomly delete 15%-25% of non-critical tokens from each training sample (retaining core placeholders such as [IP] / [ERROR]), ensuring that the template retains complete semantics after deletion (verified through a pre-trained language model). Add synonym replacement enhancement by building a log-specific synonym library (e.g., error / fault, failed / unsuccessful) and applying random replacements to 20% of the training samples.

[0077] S3. Construct training samples: Define the generate_train_examples function to generate self-supervised training samples (positive examples + negative examples), generate positive sample sentence pairs (initial sentence + enhanced sentence), generate negative sample sentence pairs (initial sentence + random different sentences), and all samples contain similarity labels (1 / 0), with 1 for positive examples and 0 for negative examples. S4. Model Configuration and Training: Load the pre-trained MiniLM model, which is built on the Sentence-BERT framework and adopts a dual-tower (Siamese Network) structure: the input text is mapped to a fixed-dimensional vector (such as 384-dimensional) through a Transformer encoder with shared weights, and then the sentence vector is output through a pooling layer; The preset learning rate is 2e-5, which can be dynamically adjusted. The batch size is set to 8 to 32, and the training epochs are 100. The training stops in real time based on changes in the loss value. A contrastive loss function is applied to optimize the similarity scores of sentence pairs, and the pooling layer is modified to use the CLS strategy. Finally, the trained output is used to fine-tune the model.

[0078] S5. Construct a semantic vector dictionary and use the trained V-SBERT model to perform template semantic matching, converting the log template into a log semantic sequence vector arranged by timestamps.

[0079] Finally, the anomaly detection part of step 5 specifically includes: Anomaly detection model training section: S1. Data preprocessing: Training data is constructed using a sliding window, and wavelet denoising is performed on the window sequence to remove noise from the data and retain useful signal components. Then, the denoised data is normalized.

[0080] S2, Model Architecture: S21. Model Denoising Layer: A three-layer convolutional denoising layer is constructed. The preprocessed data is then subjected to convolutional denoising to enhance local features. Residual learning is used to construct the feature input data, which is composed of the denoised data and the undenoised data in a 7:3 ratio. Residual connections are used to balance denoising and information preservation, providing higher-quality feature representations for subsequent model learning.

[0081] S22, Feature Transformation Layer: The feature transformation layer is the first learnable transformation layer of the model, which maps the original embedding space to the model optimization space, creating a feature representation that is more suitable for time series modeling.

[0082] S23, a bidirectional LSTM layer, combines past and future information to provide a complete context for each time step, enhancing the temporal awareness of features, performing fine-grained local temporal modeling, and outputting features rich in temporal information to provide optimized input for the Transformer.

[0083] S24, Transformer layer, adds positional encoding to the output of LSTM, and inputs the positionally encoded sequence into the Transformer encoder. The encoder includes a multi-head self-attention and feedforward network, and generates a triangular mask to prevent future information leakage. Based on the temporal features extracted by LSTM, it further captures the global dependencies in the sequence.

[0084] S25. Attention mechanism: The representation of each time step in the Transformer output is weighted, and the time step most important to the current prediction task is selected. A fixed-size context vector is obtained by weighted summation. This vector integrates important information from the entire sequence and is used for the final prediction.

[0085] S26, the output layer, generates the final output by passing the context vector obtained through the attention mechanism through two fully connected layers.

[0086] S3. Loss Function and Optimization Strategy: The total loss consists of two parts: prediction loss and denoising loss. Prediction loss is the mean square error between the predicted output and the true label, and denoising loss is the mean square error between the output of the denoising layer and the original input. Anomaly detection model detection section: S1, Data Processing Section: Same as above S2, model architecture is the same as above. S3. The loss function adopts a single MSE function. Since the validation process only focuses on the model's prediction ability rather than the feature learning ability, and the denoising loss is not directly related to the final prediction task, the noise loss during the training process is removed in order to reduce the amount of computation.

[0087] S4. Set a threshold. If the loss function MSE is greater than the threshold, it is considered abnormal; otherwise, it is considered normal.

[0088] Mathematical model expression of the anomaly detection process: Step 1: Collect logs. Let the system log set L = { }; Step 2: Log parsing. Use T5logs to parse the log collection L into a log template sequence. The structured text information remaining after removing the variable parameter part of the system log through the log parser T5logs is the log template.

[0089] Step 3: Feature extraction. Input the log template sequence in T into the trained V-SBERT model to obtain the corresponding log template semantic vector sequence. ; Step 4: Construct a semantic vector sequence sliding window: Let the sliding window size be τ, and let... , The corresponding sliding window is denoted as (V, The generation method is as follows: like Then (V, )=

[0090] Otherwise, (V, )= ; Step 5, anomaly detection: Assume the above output sliding window (V, )for , is the input to the anomaly detection module, where N is the number of sequences, T is the sequence length, and 384 is the feature dimension (Sentence-bert output sequence is 384-dimensional). Training phase: S1. Data Preprocessing: Wavelet denoising: For each feature dimension Handle independently:

[0091] in It is wavelet transform. It is a soft thresholding function, and d is the feature dimension.

[0092] Normalization process:

[0093] Where x is the data after wavelet denoising, i.e., the above.

[0094] S2, Model Hidden Layer: Model Input B is the batch size, and W is the window size; S21, Denoising Layer: The denoising layer performs a convolution operation on the input X. Specifically, it transposes X from [B,W,384] to [B,384,W]. Through convolutional blocks: Convolution 1: C1 = Conv1d (in_channels = 384, conv_channels = 96, kernel_size = 5, padding = 2) ReLU activation: A1 = ReLU(C1) Batch normalization: B1 = BatchNorm1d(A1) Dropout: D1 = Dropout (B1, p = 0.2) Convolution 2: C2 = Conv1d (in_channels =96, conv_channels = 96, kernel_size = 3, padding = 1) ReLU activation: A2 = ReLU ( C2 ) Convolution 3: C3 = Conv1d (in_channels =96, conv_channels = 384, kernel_size = 3, padding = 1) Batch normalization: B2 = BatchNorm1d(A3) Transpose the data back to [B,W,384] and use residual connections to construct the final denoised data: denoised_X = 0.7 × B² + 0.3 × X; The output is as follows:

[0095]

[0096] in It is the ReLU activation function. It is the output obtained after passing through the convolution block. This is the final denoised data.

[0097] S22, Feature Projection Layer:

[0098] in It is the input projection layer weight matrix; It is the bias vector of the input projection layer; It is the feature sequence after projection and activation; Where h is the sequence length and h is the hidden layer size; S23, Bidirectional LSTM layer:

[0099]

[0100] in This refers to the hidden state of the LSTM at time step t. This refers to the cell state of LSTM at time step t; This is the input from the previous time step, the input data received by the LSTM unit at the current time t. It comes from the previous module (i.e., the feature projection layer). The output at time t-1.

[0101] It is the hidden state of the previous time step. It represents the cell state at the previous time step; It is a forward LSTM that processes the entire sequence from left to right. Then, the set of hidden states output at each time step.

[0102] Backward LSTM processes the entire sequence from right to left. Then, the set of hidden states output at each time step.

[0103] The bidirectional outputs are concatenated; because the forward and backward outputs are concatenated, the feature dimension becomes 2h. This includes past and future contextual information for each time step; S24, Location Coding Layer:

[0104]

[0105]

[0106] PE: Position Encoding Matrix. Since the Transformer itself lacks recurrent or convolutional structures and cannot perceive the sequence order, positional information needs to be manually injected. pos: The position index in the sequence (0, 1, 2, ..., τ-1); i: Dimension index (0, 1, 2, ..., d-1). Used to alternate between sine and cosine functions; d: The dimension of the positional encoding. It must be added to the matrix to be added. The feature dimensions are consistent, i.e., d = 2h; Using Dropout, we can randomly mask out a portion of the location information here to prevent overfitting; The sequence representation after adding positional information. This is the input to the Transformer encoder; S25, Transformer encoder layer:

[0107] TransformerEncoder: A module consisting of multiple stacked TransformerEncoder Layers. Each layer contains: Multi-head self-attention: This mechanism allows each element in a sequence to pay attention to all other elements in the sequence, efficiently capturing global dependencies. Feed-forward network: This mechanism performs a non-linear transformation on the output of the self-attention mechanism.

[0108] Dimensions and Inputs Consistent; S26. Attention Mechanism:

[0109]

[0110] The weight matrix for the attention mechanism. Used to project the Transformer's output H_trans onto a new space. Bias vector of the attention mechanism v: Weight vector of the attention scoring function : Attention weights. This is a vector calculated using the softmax function, representing the importance of each time step to the final task. The sum of all weights is 1. c: Context vector. This is a weighted sum of the entire sequence, a condensed and summarized representation of the information the model deems most helpful for the final prediction.

[0111] S27, Output Layer:

[0112]

[0113] A 384-dimensional vector is output through two fully connected layers; It is the output of the first fully connected layer. This is the weight matrix of the first fully connected layer. It is the bias vector of the first fully connected layer; This is the final predicted output. This is the weight matrix of the second fully connected layer. It is the bias vector of the second fully connected layer; S3, Loss Function:

[0114]

[0115]

[0116] It is the main prediction loss, which uses mean squared error (MSE) to measure the difference between the model's final output and the true value; It is the actual value. It is a predicted value; It is the denoising loss; it measures the ability of the denoising layer to reconstruct the input sequence. It is the original input sequence; It is the denoised sequence; This is an auxiliary loss function, designed to guide the denoising layer in learning how to effectively denoise and extract features from the input data, thereby providing cleaner and more useful input information for the model's main task (prediction). (The formula is incomplete.) This indicates that the average MSE is calculated over the entire input sequence (all dimensions across all time steps). This is the total loss, where 0.1 is the weighting coefficient; this is a hyperparameter used to balance the importance of the main loss and auxiliary losses. Testing phase: The model structure and data preprocessing are consistent with the training process described above; Loss function:

[0117] Performance metrics: Precision Calculation formula: TP / (TP + FP) This reflects the proportion of logs that the model identifies as abnormal, but are actually abnormal. Measuring the accuracy of model judgments Recall Calculation formula: TP / (TP + FN) This reflects the proportion of actual anomaly logs that were correctly identified by the model. Measuring the model's ability to detect anomalies F1 Score Calculation formula: 2(Precision × Recall) / (Precision + Recall) Harmonic mean of precision and recall Comprehensive evaluation of the overall performance of the model The meanings of each parameter are as follows:

[0118] Anomaly detection formula:

[0119] in This is the abnormal threshold; In this embodiment, the loss curve of the remodeling model tends to be the lowest after 30 rounds; using base station log L1, the values ​​of various indicators are all well achieved.

[0120] This embodiment uses the T5logs log parsing method to achieve efficient log parsing and provide accurate log template sequences for downstream tasks. It uses Sentence-BERT to extract semantic behavioral features from the log templates, efficiently solving the problems of sample duplication and information extraction, and providing accurate log feature vector sequences for subsequent detection tasks. The detection part uses an LSTM and Transformer model architecture with a denoising sub-network. The denoising module suppresses irrelevant noise, allowing the model to focus more on learning and predicting relevant patterns, improving prediction accuracy and generalization ability. Through a hybrid architecture of BI-LSTM and Transformer encoder layers, it captures features from both local and global sequences, fully integrating multimodal features to achieve efficient and accurate anomaly detection.

Claims

1. A small base station log anomaly detection method fusing LSTM and Transformer, characterized in that, include: S1: Collect raw logs from various specialties from base station equipment; S2: Split and parse each original log entry to obtain the log template corresponding to the original log; S3: Construct a log vector dictionary; S4: The log templates corresponding to the original logs are divided into log sequences using a sliding window mechanism, and the log sequence semantic vectors are obtained by combining them with the log vector dictionary; S5: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model for log anomaly detection.

2. The method for detecting small cell log anomalies by fusing LSTM and Transformer as described in claim 1, characterized in that, In step S2, the Flan-T5 model is constructed. The Flan-T5 model is used to split and parse each original log to obtain the log template corresponding to the original log. The Flan-T5 model is a pre-trained language model based on the T5 model and has been fine-tuned for tasks to improve its performance in various natural language processing tasks. The T5 model is a text generation framework designed to transform all NLP tasks into text-to-text tasks, simplifying the application of the model in multi-task learning.

3. The method for detecting small cell log anomalies by fusing LSTM and Transformer as described in claim 1, characterized in that, Building the Flan-T5 model also includes: S2.1: Fifty training samples are formed by sampling a small number of logs and their related log templates through clustering and log sampling techniques; S2.2: Design a prompt template. Prompts are user-provided inputs, including: queries, instructions, or questions; S2.3: Train the Flan-T5 model using training samples and prompt templates. During training, provide the task description, instructions, inputs, and responses to the Flan-T5 model (LLM). During parsing, only provide the description, instructions, and inputs, and require the Flan-T5 model to generate responses to obtain the log template.

4. The method for detecting small cell log anomalies by fusing LSTM and Transformer as described in claim 3, characterized in that, Step S3 specifically includes: Build and pre-train a sentence-bert model to obtain a fine-tuned sentence-bert model, and use the fine-tuned sentence-bert model to vectorize log templates and construct a log vector dictionary.

5. The method for detecting log anomalies in small base stations by fusing LSTM and Transformer as described in claim 4, characterized in that, The sentence-BERT model was constructed and pre-trained, and the fine-tuned sentence-BERT model was obtained, including: S3.1 Training Corpus Preparation: The log template generated by flan-t5-small is used as the basic corpus; S3.2 Data Augmentation Strategy: Apply an improved random deletion scheme to randomly delete 15%-25% of non-key tokens in each training sample to ensure that the template still maintains complete semantics after deletion; add synonym replacement enhancement, build a log-specific thesaurus, and apply random replacement in 20% of the training samples; S3.3 Constructing Training Samples: Define a function to generate self-supervised training samples, generate positive sample sentence pairs, generate negative sample sentence pairs, and all samples contain similarity labels, with positive examples being 1 and negative examples being 0. S3.4 Model Configuration and Training: Load the pre-trained MiniLM model, which is built on the Sentence-BERT framework and adopts a dual-tower (Siamese Network) structure: the input text is mapped to a fixed-dimensional vector (such as 384-dimensional) through a Transformer encoder with shared weights, and then the sentence vector is output through a pooling layer; The learning rate is set to 2e-5 and can be dynamically adjusted; the batch size is set to 8 to 32, and the training epochs are 100; the training stops in real time based on the change of the loss value; the contrast loss function is applied to optimize the similarity score of sentence pairs, and the pooling layer is modified to the CLS strategy. Finally, the fine-tuned sentence-bert model is trained and output.

6. The method for detecting small cell log anomalies by fusing LSTM and Transformer as described in claim 1, characterized in that, In step S4, dividing the log template corresponding to the original log into log sequences using the sliding window mechanism also includes: Wavelet denoising is performed using a sliding window mechanism to remove noise from the data and retain useful signal components. The denoised data is then normalized to obtain the log sequence.

7. The method for detecting small cell log anomalies by fusing LSTM and Transformer as described in claim 1, characterized in that, In step S5, the BiLSTM-Transformer model includes: Model Denoising Layer: A three-layer convolutional denoising layer is constructed. The preprocessed data is then subjected to convolutional denoising to enhance local features. Residual learning is used to construct the feature input data, which is composed of the denoised data and the undenoised data in a 7:3 ratio. The residual connection is used to balance denoising and information preservation, thus providing higher quality feature representations for subsequent model learning. Feature Transformation Layer: The feature transformation layer is the first learnable transformation layer of the model, which maps the original embedding space to the model optimization space, creating feature representations that are more suitable for temporal modeling; Bidirectional LSTM layer: Combining past and future information, it provides a complete context for each time step, enhances the temporal awareness of features, performs fine-grained local temporal modeling, outputs features rich in temporal information, and provides optimized input for Transformer; Transformer layer: The output of LSTM is added with position encoding. The position-encoded sequence is then input into the Transformer encoder, which includes a multi-head self-attention network and a feedforward network. A triangular mask is generated to prevent future information leakage. Based on the temporal features extracted by LSTM, the global dependencies in the sequence are further captured. Attention mechanism layer: The representation of each time step of the Transformer output is weighted, and the time step most important to the current prediction task is selected. A fixed-size context vector is obtained by weighted summation. This vector integrates important information from the entire sequence and is used for the final prediction. Output layer: The context vector obtained through the attention mechanism is passed through two fully connected layers to generate the final output; Loss function and optimization strategy: The total loss consists of two parts: prediction loss and denoising loss. Prediction loss is the mean square error between the predicted output and the true label, and denoising loss is the mean square error between the output of the denoising layer and the original input.

8. The method for detecting log anomalies in small base stations by fusing LSTM and Transformer as described in claim 1, characterized in that, In step S5, the log sequence semantic vector is input into the BiLSTM-Transformer model for training, resulting in the trained BiLSTM-Transformer model as follows: Data preprocessing: Wavelet denoising: For each feature dimension Handle independently: in It is wavelet transform. It is a soft thresholding function, where d is the feature dimension; Normalization process: Where x is the data after wavelet denoising, i.e., the above. enter B is the batch size, and W is the window size; Perform a convolution operation on the input X, specifically by transposing X from [B,W,384] to [B,384,W]. Through convolutional blocks: Convolution 1: C1 = Conv1d (in_channels = 384, conv_channels = 96, kernel_size= 5, padding = 2) ReLU activation: A1 = ReLU(C1) Batch normalization: B1 = BatchNorm1d(A1) Dropout: D1 = Dropout (B1, p = 0.2) Convolution 2: C2 = Conv1d (in_channels =96, conv_channels = 96, kernel_size =3, padding = 1) ReLU activation: A2 = ReLU ( C2 ) Convolution 3: C3 = Conv1d (in_channels =96, conv_channels = 384, kernel_size =3, padding = 1) Batch normalization: B2 = BatchNorm1d(A3) Transpose the data back to [B,W,384] and use residual connections to construct the final denoised data: denoised_X = 0.7 × B² + 0.3 × X; The output is as follows: in It is the ReLU activation function. It is the output obtained after passing through the convolution block. This is the final denoised data; Feature transformation layer: in It is the input projection layer weight matrix; It is the bias vector of the input projection layer; It is the feature sequence after projection and activation; Where h is the sequence length and h is the hidden layer size; Bidirectional LSTM layer: in This refers to the hidden state of the LSTM at time step t. This refers to the cell state of the LSTM at time step t; It is the input of the previous time step, which is the input data received by the LSTM unit at the current time t; it comes from the output of the previous module at time t-1. It is the hidden state of the previous time step. It represents the cell state at the previous time step; It is a forward LSTM that processes the entire sequence from left to right. Then, the set of hidden states output at each time step; Backward LSTM processes the entire sequence from right to left. Then, the set of hidden states output at each time step; The bidirectional outputs are concatenated; because the forward and backward outputs are concatenated, the feature dimension becomes 2h; this contains the past and future contextual information for each time step; PE: Position Encoding Matrix; Since the Transformer itself does not have a loop or convolution structure, it cannot perceive the order of the sequence, so position information needs to be manually injected. pos: The position index in the sequence (0, 1, 2, ..., τ-1); i: Dimension index (0, 1, 2, ..., d-1), used to alternate between sine and cosine functions; d: The dimension of the positional encoding, which must be added to the matrix. The feature dimensions are consistent, i.e., d = 2h; Using Dropout, we can randomly mask out a portion of the location information here to prevent overfitting; The sequence representation after adding positional information; this is the input to the Transformer encoder. Transformer layer: TransformerEncoder: A module consisting of multiple stacked TransformerEncoder Layers; each layer contains: Multi-head self-attention: This mechanism allows each element in a sequence to pay attention to all other elements in the sequence, efficiently capturing global dependencies. Feedforward neural networks: perform nonlinear transformations on the output of self-attention; Dimensions and Inputs Consistent; Attention mechanism layer: The weight matrix for the attention mechanism; used to project the Transformer's output H_trans onto a new space. Bias vector of attention mechanism v: Weight vector of the attention scoring function Attention weights; It is a vector, calculated using the softmax function, representing the importance of each time step to the final task; the sum of all weights is 1. c: Context vector; it is the weighted sum of the entire sequence, a condensation and summary of the information that the model considers most helpful for making the final prediction; Output layer: A 384-dimensional vector is output through two fully connected layers; It is the output of the first fully connected layer. This is the weight matrix of the first fully connected layer. It is the bias vector of the first fully connected layer; This is the final predicted output. This is the weight matrix of the second fully connected layer. It is the bias vector of the second fully connected layer; Loss function: It is the main prediction loss, using mean squared error, used to measure the difference between the model's final output and the true value; It is the actual value. It is a predicted value; It is the denoising loss; it measures the ability of the denoising layer to reconstruct the input sequence. It is the original input sequence; It is the denoised sequence; It is an auxiliary loss function, its purpose being to guide the denoising layer to learn how to effectively denoise and extract features from the input data, thereby providing input information for the model's main task; (The formula is incomplete and requires further context.) This indicates that the mean value (MSE) is calculated over the entire input sequence. This is the total loss, where 0.1 is the weighting coefficient; This is a hyperparameter used to balance the importance of the main loss and the auxiliary loss; Loss function during testing phase: 。 9. The method for detecting log anomalies in small base stations by fusing LSTM and Transformer as described in claim 1, characterized in that, In step S5, log anomaly detection is performed using the trained BiLSTM-Transformer model, specifically as follows: The loss function uses a single MSE function; Set a threshold; if the loss function MSE is greater than the threshold, it is considered abnormal; otherwise, it is considered normal.

10. A small cell log anomaly detection system integrating LSTM and Transformer, characterized in that, include: Data Acquisition Module: Collects raw logs from various disciplines from base station equipment; Splitting and parsing module: Splits and parses each original log entry to obtain the log template corresponding to the original log; Vector dictionary module: Constructs log vector dictionaries; Semantic Vector Module: The log template corresponding to the original log is divided into log sequences through a sliding window mechanism, and the log sequence semantic vector is obtained by combining the log vector dictionary; Model detection module: Construct a BiLSTM-Transformer model, input the log sequence semantic vector into the BiLSTM-Transformer model for training, obtain the trained BiLSTM-Transformer model, and use the trained BiLSTM-Transformer model to detect log anomalies.

Images