A security alarm false alarm identification method based on deep learning and text analysis

Abstract

The application discloses a kind of based on deep learning and security alarm false alarm identification method of text analysis, 1) data acquisition and processing stage;2) through the multi-dimensional feature of extraction log, network attack event is detected using recurrent neural network RNN;3) text analysis based on Word2Vec and the false alarm of network security event of Transformer detection;4) abnormal result feedback, the application is extracted by feature and RNN model design, especially using LSTM unit handles time dependence, improve the identification ability of model to security event, this method does not depend on traditional signature-based detection, reduces the dependence on false alarm;Combining the text analysis technology based on Word2Vec and Transformer, semantic information in alarm content is deeply mined, the accuracy of false alarm identification is significantly improved, and through abnormal result feedback to operation and maintenance personnel, more efficient and accurate network security protection solution is provided.

Description

Technical Field

[0001] The present invention relates to the field of network security and machine learning technology, and in particular to a method for identifying false alarms of security alerts based on deep learning and text analysis. Background Art

[0002] In modern enterprise network environments, false positive security alerts are an increasingly serious problem. As network attacks continue to grow in complexity and diversity, traditional signature-based intrusion detection systems (IDS) and anomaly-based detection methods often generate a large number of false positives when identifying real threats. These false positives not only consume a significant amount of time and effort from operations and maintenance personnel but can also disrupt normal enterprise operations.

[0003] Existing security alert processing methods primarily rely on analyzing threat indicators within alert events or extracting contextual information by analyzing the sequence of alert events. However, these methods have significant shortcomings in integrating threat indicators and contextual information. For example, they may overlook key information within alert attributes, such as whether the source IP address is from a foreign country or whether the alert belongs to a category with a well-defined payload (e.g., ransomware and botnets). Furthermore, existing methods often fail to fully utilize information provided by cross-organizational intrusion detection system (IPS) monitors when analyzing alert context, such as the correlation between source and destination IP addresses.

[0004] To improve the efficiency of network attack detection and the accuracy of alert processing, a new approach is needed to identify attacks and filter false positives. Technologies based on deep learning and text analytics offer new possibilities for addressing this issue. Deep learning models can automatically learn complex patterns from large amounts of data, while text analytics can deeply mine the semantic information within alert content to more accurately distinguish between normal and malicious activity. The combination of these two technologies can more accurately distinguish between normal and malicious activity, significantly improving the efficiency and accuracy of alert processing.

[0005] Therefore, this paper proposes a method for identifying false alarms in security alerts based on deep learning and text analysis. This method aims to improve the accuracy of false alarm identification and reduce disruption to normal enterprise operations by comprehensively utilizing information such as individual alert characteristics and contextual relationships. This method will help operations personnel handle alerts more effectively, thereby improving the performance of the entire network security system. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a security alarm false alarm identification method based on deep learning and text analysis.

[0007] A security alert false alarm identification method based on deep learning and text analysis includes the following steps:

[0008] Step 1) Data collection and processing: Collect network traffic, security events, and system call logs from key data sources in vulnerable industries such as enterprises, schools, and factories. Use Logstash to integrate, clean, and standardize the data, remove invalid data, and convert it into a unified JSON format to provide a structured data set for subsequent analysis.

[0009] Step 2) The system extracts multidimensional features from logs and uses a recurrent neural network (RNN) to detect cyberattacks. Key fields are extracted from the data set through a thorough review, feature vectors are constructed, and an RNN model is designed to learn temporal dependencies using LSTM units. Feature encoding and normalization are used to optimize model training, enabling the model to identify abnormal behaviors from a large number of normal behaviors.

[0010] Normal behavior refers to expected, regular activities in a network system that do not pose security risks or threats. For example, a user accessing an internal company website, using authorized applications, or performing daily work tasks during working hours are all examples of normal behavior.

[0011] Abnormal behavior refers to activities that deviate from normal patterns and may indicate a security threat or issue. Abnormal behavior may include unauthorized access attempts, unusual network traffic patterns, suspicious login activity, malware execution, data leakage, etc. These behaviors may indicate that a malicious user is attempting to compromise system security or that the system may have been compromised.

[0012] Identifying abnormal behavior from a large number of normal behaviors means: using machine learning models to learn and understand the patterns of normal behavior and to identify behaviors that are significantly different from these patterns.

[0013] Step 3) Detect false positives of network security incidents using text analysis based on Word2Vec and Transformer: Combine the Word2Vec and Transformer models to perform text analysis. By segmenting and encoding the RNN model detection results and using the self-attention mechanism to deeply understand (analyze) the content of network traffic packets, a Transformer-based classifier is trained to identify and reduce false positives in security alerts.

[0014] Step 4) Abnormal result feedback: The abnormal detection results after deep learning and text analysis are fed back to the operation and maintenance personnel to distinguish between benign and abnormal alarms, guide further evaluation and processing, and thus optimize the performance of the network security protection system.

[0015] In step 1), the detailed steps of the data collection and processing stage are as follows:

[0016] (1-1) Data Collection Phase: Through close collaboration with the cybersecurity team, we conducted a comprehensive review of the network architecture of enterprises, schools, factories, and other industries to identify key data sources and categorize these data sources, laying the foundation for configuring Logstash input plugins. As a data processing engine, Logstash is responsible for unifying and standardizing data collection, processing, and forwarding, ensuring the efficiency, stability, and security of the process. Data sources are configured on Logstash and filtered as necessary. They are then forwarded to Kafka topics via the Kafka output plugin. Each topic is dedicated to a specific data type, improving collection efficiency. At the same time, the stability and security of the data collection system are guaranteed through the Prometheus monitoring tool and regular system audits.

[0017] (1-2) Data Preprocessing: Real-time processing of Kafka data is achieved through applications based on the Kafka consumer API. Consumer groups are leveraged to improve data processing efficiency and system fault tolerance. Data cleaning involves data validation, regular expression filtering, and deduplication to ensure data accuracy and uniqueness. Data conversion logic includes timestamp standardization, data type conversion, and data windowing to accommodate diverse analysis needs. Data formatting converts the converted data into JSON format, using serialization and deserialization techniques to maintain standardized data structures while preserving semantic information.

[0018] In step 2), the system extracts multidimensional features of the log and uses a recurrent neural network (RNN) to detect network attack events. The detailed steps are as follows:

[0019] (2-1) Feature Extraction: By reviewing and cleaning the raw log data, key fields such as IP addresses, port numbers, and protocol types are identified and converted into a structured, standardized format that is easily understood by the model. Timestamps are standardized and numerical features are normalized to ensure data consistency and accuracy. These processing steps transform the log data into standardized feature vectors suitable for model learning, laying the foundation for subsequent RNN model training.

[0020] (2-2) RNN model design: In the model design phase, LSTM units are used to build the hidden layer to process sequence data and capture time-dependent features in the data (time-dependent features refer to features in the data that change over time, which are crucial to understanding the development of events and predicting future behavior, such as frequent data access requests in a short period of time, or specific types of network traffic patterns, which change over time and may indicate security threats), while introducing a dropout layer to prevent overfitting. The input layer of the model receives the processed feature vector, while the output layer outputs the binary classification prediction probability of security events through the sigmoid activation function. This design allows the RNN model to efficiently detect existing security threats.

[0021] (2-3) Model training: First, a binary cross-entropy loss function is set to optimize the binary classification problem, and data with a batch size of 256 is used for training, and 50 epochs are set to ensure that the model is fully learned, with a learning rate of 0.01, combined with the Adam optimizer, which can adaptively adjust the learning rate, speed up convergence and improve performance. During training, the model calculates the output through forward propagation, updates the weights through backpropagation, and iterates to minimize the loss, ultimately training an RNN model that can accurately identify attack events.

[0022] In step 3), the detailed steps of the text analysis network security event false alarm detection based on Word2Vec and Transformer are as follows:

[0023] (3-1) Use the tokenizer for tokenization: The tokenizer is responsible for extracting key information from security alert-related network traffic data packets, including network communication quintuples (source / destination IP, source / destination port and protocol number) and request and response headers and bodies (request and response headers and request and response bodies). It performs decoding, standardizes special characters, splits the decoded string into words, and filters out irrelevant invalid strings such as numbers, HTML tags, etc., and finally generates a word sequence useful for text analysis, providing preparation for subsequent Word2Vec encoding and Transformer-based model processing.

[0024] (3-2) Word2Vec encoding: Use the pre-trained Word2Vec model to convert the word sequence output by the tokenizer into dense vector form, which can capture the semantic relationship and context information between words, providing rich feature representation for the Transformer-based text analysis model.

[0025] (3-3) Transformer-based classifier: An advanced neural network model is constructed, which integrates the vectors generated by Word2Vec using embedding layers and position encoding, and preserves the word order information. Through multiple encoder layers and the application of self-attention mechanism, the model can deeply understand the context and long-distance dependence in the text. Finally, through a fully connected output layer using sigmoid activation function for binary classification prediction, the probability score representing the possibility of false alarm of the activity is output, realizing efficient identification and classification of normal activities and malicious behaviors in security alerts.

[0026] (3-4) Anomaly detection: After training and verification, the Transformer classifier can be deployed in real-time network monitoring systems to analyze traffic data packets and identify and filter false positives in security alerts.

[0027] In step 4), the detailed steps of abnormal result feedback are as follows:

[0028] (4-1) Abnormal result feedback: The analyzed security alerts are classified as false alarms or real security threats. The operation and maintenance personnel ignore or remove false alarms according to the classification results to reduce interference; and respond and handle the confirmed security threats in an emergency to ensure rapid and effective response to real attacks and improve the accuracy and efficiency of network security protection.

[0029] The beneficial effects of the present application mainly include:

[0030] 1) In the data collection and preprocessing stage, through cooperation with the network security team, the network architecture of enterprises, schools and factories in industries vulnerable to attacks is comprehensively reviewed, the key data sources are determined, and Logstash is used for unified processing and standardization of data, ensuring the efficiency and stability of data collection, processing and forwarding.

[0031] 2) In the data preprocessing stage, by using an application based on Kafka consumer API, real-time reception, cleaning, formatting and conversion of data are realized, improving the efficiency of data processing and data quality.

[0032] 3) Through feature extraction and RNN model design, especially using LSTM units to handle time dependence, the model's ability to identify security events is improved, and this method does not rely on traditional signature-based detection, reducing the dependence on false alarms.

[0033] 4) Combined with Word2Vec and Transformer-based text analysis technology, the semantic information in the alert content is deeply mined, significantly improving the accuracy of false alarm identification, and the abnormal result feedback to the operation and maintenance personnel provides a more efficient and accurate network security protection solution. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings.

[0036] Reference Figure 1 A security alarm false alarm identification method based on deep learning and text analysis includes the following steps:

[0037] Step (1) Data collection and processing phase: Collect network traffic, security events, and system call logs from key data sources in vulnerable industries such as enterprises, schools, and factories. Use Logstash to integrate, clean, and standardize the data, remove invalid data, and convert them into a unified JSON format to provide a structured data set for subsequent analysis.

[0038] In step (1), the detailed steps of data collection and preprocessing are as follows:

[0039] (1-1) Data Collection Phase: During the data collection process, through long-term close collaboration with the network security team, we conducted a comprehensive review of the network architecture of enterprises, schools, factories, and other industries, and identified key data sources. These data sources include but are not limited to network edge devices, servers, applications, and endpoints. Subsequently, the data was carefully classified into network traffic logs, security event logs, and system call logs, providing a foundation for configuring appropriate input plugins for Logstash.

[0040] Logstash is an open-source, server-side data processing engine that dynamically unifies data from various sources and standardizes it to a selected destination. This method uses Logstash to ensure efficient and stable data collection, processing, and forwarding, while also ensuring data security. After identifying and categorizing the data sources, install and configure Logstash.

[0041] Logstash was installed on an appropriate server or virtual machine, ensuring access to all key data sources. Subsequently, Logstash input plugins were configured for different types of log data. For example, the file input plugin was used to monitor and read file logs, and the syslog input plugin was used to receive logs sent over the network. After configuring the input plugins, Logstash filters were further configured to apply necessary filtering rules, such as filtering based on log level, error code, or other key fields. The filtered data was forwarded to Kafka via the Kafka output plugin.

[0042] In Kafka, specialized Kafka topics are designed based on the log type, ensuring that each topic is dedicated to a specific type of data for easier management and processing. At the same time, an appropriate number of partitions is configured for each topic to support parallel processing and improve throughput.

[0043] To ensure the stability and efficiency of the data collection system, we also monitor and maintain the data collection process. We use the Prometheus monitoring tool to monitor Kafka performance metrics and Logstash log processing status. We also conduct regular system audits to check the security and efficiency of data collection and transmission, ensuring a smooth and secure data collection process.

[0044] (1-2) Data Preprocessing: During this stage, the data received from Kafka needs to be converted into a clean, structured form suitable for further analysis. To this end, we first used consumer applications based on the Kafka Consumer API. These applications subscribed to specific Kafka topics to receive real-time data streams. By configuring consumer groups, we enabled the collaboration of multiple consumer instances, which not only improved data processing efficiency but also enhanced the system's fault tolerance.

[0045] During the data cleansing phase, a data validation mechanism was first implemented to identify and exclude malformed or incomplete log entries. This included checking the integrity of data fields and validating data types. For example, if a log entry is expected to contain the "id," "timestamp," and "action_type" fields, the validation mechanism would check their presence and verify that the "timestamp" field is in a valid time format. Secondly, regular expressions were used to filter out irrelevant or low-value data, for example, removing meaningless log padding or ad traffic. Furthermore, data deduplication was performed to identify and exclude duplicate log entries, ensuring data uniqueness.

[0046] The data conversion logic is implemented through the following three data conversion logics:

[0047] 1) Timestamp standardization: In the processed data, timestamps may exist in various formats, such as "2024-07-24 14:30:00" or "20240724143000". To unify the time format, we convert all timestamps to the ISO 8601 standard format (i.e., "YYYY-MM-DDTHH:MM:SS"); for example, "20240724143000" is converted to "2024-07-24T14:30:00".

[0048] 2) Data type conversion: Data type conversion converts data from one format to another to meet analysis requirements. For example, some fields may be string types, but mathematical operations may be performed on them, such as converting the string "123.45" to the floating-point number 123.45. In addition, some date fields may need to be converted from string to date type to perform date-related calculations and comparisons.

[0049] 3) Data Windowing: For time series data, use windowing techniques to split the data into fixed-size blocks. For example, if you need to analyze hourly traffic patterns, split the continuous timestamp data into hourly windows. For example, if there is data from "2024-07-24T14:00:00" to "2024-07-24T15:00:00", split it into a window from 14:00 to 15:00, and analyze all the data within this window as a group.

[0050] Data formatting is another key step in data preprocessing. The processed data is uniformly converted to JSON format, as JSON is highly flexible and easy to read, making it ideal for data exchange and storage. During the conversion process, we employ serialization and deserialization techniques to ensure standardized data structures while preserving the data's semantics and context.

[0051] Step (2) The system extracts multidimensional features of the logs and uses a recurrent neural network (RNN) to detect network attack events: by exhaustively reviewing the data set, key fields are extracted, feature vectors are constructed, and an RNN model is designed to learn time dependencies using LSTM units. Model training is optimized through feature encoding and normalization processing, so that the model can identify abnormal behaviors from a large number of normal behaviors.

[0052] In step (2), the system extracts multidimensional features from the logs and uses a recurrent neural network (RNN) to detect network attack events. The detailed steps are as follows:

[0053] (2-1) Feature extraction: This is a key step in converting raw log data into usable information for the model. First, data review is performed. This process involves a detailed analysis of each event in the dataset to identify key fields such as the source IP address, destination IP address, port number, protocol type, and login attempt timestamp. These fields are fundamental to understanding security events, so it is crucial to ensure they are accurately recorded in the dataset.

[0054] Feature construction is then performed, which is the process of converting the extracted data into useful features. The source IP address and destination IP address need to be converted into country or region information through geolocation coding, which helps to identify abnormal behavior patterns that may be related to geographic location. For example, the source IP address in the original data is "204.20.25.36". Through the geolocation coding service, it can be determined that this IP address actually corresponds to a data center in Ohio, USA. In feature construction, this IP address can be converted into a country code (such as "US") and a region label (such as "Ohio"). If a corporate network does not usually communicate with the region, but the system detects a large number of requests from the region, this may indicate abnormal behavior.

[0055] The port number and protocol type will be converted into one-hot encoding for processing based on the needs of the RNN model, allowing the model to better understand this categorical data. For example, if the port number is 80, which is typically used for HTTP communication, and the protocol type is "TCP," in one-hot encoding, the port number is represented as a binary vector in which the 80th bit is 1 and the rest are 0s; the protocol type "TCP" is also converted into a binary vector in which the bit corresponding to the "TCP" position is 1, and the positions of other protocols such as "UDP" are 0. In this way, each feature is converted into a two-dimensional vector form, which can be more easily processed by the model.

[0056] The numerical features are then normalized. Normalization helps the model process data of different scales more effectively by scaling the feature values ​​to the range of 0 to 1. This is done using formula (1), which subtracts the minimum value of each feature from the feature value and then divides it by the difference between the maximum and minimum values ​​of the feature.

[0057]

[0058] Finally, feature encoding is the process of converting categorical features into a numerical format that the model can process. For example, a categorical feature such as protocol type would be encoded using e -hot encoding is converted to a binary vector so that each protocol type is represented by a unique binary pattern. This process not only improves the model's ability to handle categorical data, but also prevents the model from being biased towards certain categories during training.

[0059] After the above steps, the original log data is converted into a set of rich, accurate feature vectors suitable for model learning, laying the foundation for subsequent RNN model design and training.

[0060] (2-2) RNN model design: Recurrent neural networks (RNNs) can capture temporal dependencies in data, which is crucial in security incident analysis. Model design is divided into the following four steps:

[0061] 2-2-1) Define the input layer of the model. The input data is the feature vector sequence obtained in the feature extraction stage.

[0062] 2-2-2) Design the hidden layer, choosing LSTM as the core unit type because LSTMs excel at processing sequential data, particularly their ability to learn and memorize long-term dependencies. The LSTM unit structure consists of three gates: an input gate, a forget gate, and an output gate. These gates work together to allow the network to retain or forget information as necessary. In this way, LSTM networks avoid the vanishing or exploding gradient problems of traditional RNNs, resulting in superior performance in sequence prediction tasks.

[0063] When configuring the LSTM layers, set the number of units to 128 for each LSTM layer and the appropriate number of neurons for each LSTM unit. The number of neurons depends on the complexity of the data and the task requirements. Typically, the initial hidden layer may have more neurons to capture a wider range of features, while subsequent layers may have fewer neurons to achieve higher levels of abstraction and pattern recognition.

[0064] 2-2-3) Add a dropout layer after the LSTM layer to randomly drop some neurons to prevent overfitting of the model. For example, setting the dropout rate to 0.2 means that 20% of the neurons in each layer will be randomly dropped during training.

[0065] 2-2-4) Define the output layer. For binary classification problems (such as false positives and non-false positives), the output layer usually has only one neuron and uses the sigmoid activation function. The activation function formula is as follows:

[0066]

[0067] in, σ is the sigmoid function, and z is the value input to the sigmoid function.

[0068] (2-3) Model training: Train the RNN model to enable it to distinguish false positives from real security incidents.

[0069] First, define the loss function. For the binary classification problem, we use the binary cross entropy loss function, the formula is as follows:

[0070]

[0071] Where N is the number of samples, y i is the true label of the i-th sample (0 or 1), p i is the probability that the model predicts that the i-th sample is 1.

[0072] Next, we set the model parameters. In this study, the batch size was set to 256 to efficiently process data during training. The number of training epochs was set to 50, ensuring the model could learn the data characteristics over a sufficient period of time while avoiding overtraining. The learning rate was set to 0.01 to ensure an appropriate step size during the weight update process, ensuring rapid convergence while avoiding skipping the optimal solution.

[0073] Finally, select the optimizer. This method uses the Adam optimizer to update the network weights. The Adam optimizer is an adaptive learning rate optimization algorithm that combines the advantages of RMSprop and Momentum. It can provide different learning rates for different parameters, thereby accelerating convergence and improving model performance. During training, the Adam optimizer dynamically adjusts the learning rate of each parameter, allowing the network to learn more efficiently.

[0074] During the model training phase, the model is trained multiple times using the training dataset. Each iteration calculates the output through forward propagation, and then updates the weights through backpropagation. During this process, a loss function serves as a metric for evaluating model performance and guiding weight updates. The binary cross-entropy loss function is particularly well-suited for binary classification problems, as it measures the difference between the model's predicted probability distribution and the true label.

[0075] (2-4) Anomaly Detection: Logistic regression is used as a classifier for anomaly detection. First, features are extracted from the security event sequence. These features are obtained through a previously trained RNN model and can represent the key attributes of the event. Next, these feature vectors are used to train a logistic regression model, and the model parameters are adjusted using formula (3).

[0076] After the logistic regression model is trained, a probability threshold Y is set based on the training results. During the actual detection process, events exceeding the threshold Y are classified as anomalies, although there will naturally be a certain amount of false positives. The threshold is selected based on performance evaluation on the validation set to balance detection sensitivity and specificity. When the logistic regression model receives a new feature vector, it outputs the probability that the event belongs to the anomaly category. If this probability value exceeds the preset threshold, the event triggers an anomaly alert and is classified as an anomaly, thus completing the anomaly detection process.

[0077] Step (3) Detect false positives of network security incidents based on text analysis of Word2Vec and Transformer: Combine Word2Vec and Transformer technologies to perform text analysis, segment and encode the detection results of the RNN model, and use the self-attention mechanism to deeply understand the content of network traffic data packets, and train a Transformer-based classifier to identify and reduce false positives in security alerts.

[0078] In step (3), the detailed steps for detecting false positives of cybersecurity incidents based on text analysis using Word2Vec and Transformer are as follows:

[0079] (3-1) Use word segmentation:

[0080] 3-1-1) The tokenizer is responsible for extracting relevant information from network traffic packets that trigger security alerts. It captures the network communication five-tuple (source / destination IP, source / destination port, and protocol number), as well as the request and response headers and bodies. For example, suppose a packet has a source IP of 208.65.25.12, a destination IP of 208.80.152.201, a source port of 4444, a destination port of 80, and a protocol of TCP. The tokenizer extracts this information from the packet as part of the network communication five-tuple.

[0081] 3-1-2) Decode the request / response header and body and normalize special strings. For example, an HTTP request may contain the following header information:

[0082] GET / index.html HTTP / 1.1

[0083] Host:www.example.com

[0084] User-Agent:Mozilla / 5.0

[0085] The response header may contain the following information:

[0086] HTTP / 1.1 200 OK

[0087] Content-Type: text / html; charset=UTF-8

[0088] The tokenizer recognizes and extracts these headers, followed by the body content.

[0089] 3-1-3) Split the decoded string into words, using newlines and spaces as delimiters. For example, if the request contains URL encoding, such as GET / search?q=example%20test HTTP / 1.1, the tokenizer needs to decode it into GET / search?q=example test HTTP / 1.1. Similarly, if the body contains escape characters or HTML entities such as & or <, the tokenizer will also decode them accordingly.

[0090] 3-1-4) Split the decoded string into words, using newlines and spaces as delimiters. For example, for the query parameter q=example test in the request body, after decoding, the word segmenter will split it into words ["example", "test"] based on spaces.

[0091] 3-1-5) Finally, the tokenizer will filter out irrelevant invalid strings, such as numbers, web page elements, or formatting symbols. For example, if the data packet contains the JavaScript code var num = 123, the tokenizer will recognize 123 as a number and may filter it out based on the context as an invalid string. At the same time, if there is or <script>的HTML标签，也会被识别并过滤。

[0092] 经过上述步骤处理后，假设原始数据包含以下文本User-Agent:Mozilla / 5.0，分词器最终会生成一个单词序列，如["User","Agent","Mozilla","5.0"]。分词器的这些步骤确保了只有对文本分析有用的文本内容被保留并用于后续的Word2Vec编码和Transformer模型处理。

[0093] (3-2)Word2Vec编码：Word2Vec能够捕捉单词之间的语义含义和上下文关系，本方法使用预训练的Word2Vec模型将单词转换为密集的向量表示。

[0094] 分词器的输出被送入Word2Vec模型，为每个单词生成向量，提供丰富的特征集以捕捉流量数据包内容的语义信息。

[0095] (3-3)基于Transformer的分类器：文本分析检测的核心是基于Transformer的神经网络模型，该模型擅长处理序列数据并捕捉文本中的长距离依赖关系。Transforme模型包括以下四部分：

[0096] 1、嵌入层：将Word2Vec向量转换为适合Transformer模型的格式。

[0097] 2、位置编码：向嵌入添加位置信息，以保留单词在序列中的顺序。

[0098] 3、编码器层：由多个编码器层组成，每层都应用自注意力机制来理解文本中的上下文。每层由多头注意力机制和前馈网络组成。

[0099] 4、输出层：一个全连接层，从Transformer编码器的输出中接收信息，并执行二分类，以确定活动是良性的还是恶意的。该层使用sigmoid激活函数输出概率分数，指示活动为误报的可能性。

[0100] 对模型进行训练时，采用二元交叉熵损失函数训练模型(公式2)，适合区分良性和恶意警报的二分类任务，并使用Adam优化器进行高效训练，在训练过程中不断调整学习率和其他超参数以实现最佳性能。除此之外，根据实际情况，还可以应用正则化技术，如dropout或L2正则化，以防止过拟合。

[0101] 经过训练，基于Word2Vec和Transformer的文本分析检测器可以从异常警报中检测出来正常的活动，以消除误报。

[0102] (3-4)异常检测：将经过训练和验证的Transformer分类器被部署在实时网络监控系统中，对流量数据包进行分析，以识别并过滤安全警报中的误报。此过程涉及将实时捕获的数据包通过分词、Word2Vec编码和Transformer模型处理，从而为每个事件生成一个表示其为误报的概率分数。根据设定的阈值，系统自动将那些概率分数超过阈值的事件分类为异常，而低于阈值的事件则被视为正常活动。

[0103] 步骤(4)异常结果反馈：将经过深度学习和文本分析的异常检测结果反馈给运维人员，区分良性和异常警报，指导进一步的评估和处理，从而优化网络安全防护体系的性能。

[0104] 步骤(4)中，异常结果反馈的详细步骤如下：

[0105] (4-1)异常结果反馈：经过上述两种异常检测方式，警报信息会被分为以下两类：

[0106] 1、良性警报：这些警报被识别为误报，即它们实际上并不代表恶意活动，而是正常业务操作或其他无害行为。

[0107] 2、异常警报：这些警报被确认为真正的安全威胁，需要运维人员进一步调查和采取相应的防御措施。

[0108] 将分析结果呈现给运维人员，运维人员可以根据分析结果对警报进行进一步的评估和处理，对于被标记为良性的警报，运维人员可能会选择忽略或将它们从警报队列中移除，以减少误报对企业正常运营的影响；对于被标记为异常的警报，运维人员可以进行深入的调查，采取诸如隔离受感染的系统、更新防火墙规则或进一步分析攻击来源等措施。

[0109] 以上操作有助于运维人员更有效地识别真正的攻击，减少正常网络活动对正常业务的干扰，同时确保对真实攻击的有效响应。本方法的实施将为网络安全防护提供一种更为高效和准确的解决方案。< / script>

Claims

1. A security alarm false alarm identification method based on deep learning and text analysis, characterized in that: The steps include: 1) Data collection and processing: Network traffic, security events, and system call logs are collected from relevant data sources. Logstash is used to integrate, clean, and standardize the data, remove invalid data, and convert it into a unified JSON format to obtain a structured data set. 2) Detect network attack events using recurrent neural networks (RNNs) by extracting multidimensional features from logs: Key fields are extracted from a thorough review of the dataset, feature vectors are constructed, and an RNN model is designed to learn temporal dependencies using LSTM units. Feature encoding and normalization are used to optimize model training, allowing the model to leverage a large amount of normal behavior to identify abnormal behavior. Normal behavior refers to expected, routine activities in a network system that do not pose security risks or threats. Abnormal behavior refers to activities that deviate from normal behavior and can pose security risks or threats. 3) Detecting false positives in cybersecurity incidents using Word2Vec and Transformer-based text analysis: Combining Word2Vec and Transformer models for text analysis, the RNN model performs word segmentation and encoding on detection results, and uses a self-attention mechanism to analyze the content of network traffic packets to identify and extract features. This allows for training a Transformer-based classifier to identify and reduce false positives in security alerts. 4) Abnormal result feedback: The anomaly detection results from step 2) deep learning and step 3) text analysis are fed back to operations and maintenance personnel to manually distinguish between benign and abnormal alerts, guide further evaluation and processing, and thus optimize the performance of the network security protection system.

2. The method for identifying false alarms based on deep learning and text analysis according to claim 1, characterized in that: The specific steps of step 1) are as follows: 1-1) Data Collection Phase: The network architecture is reviewed to identify and categorize data sources. Data sources are configured on Logstash, filtered through filters, and then forwarded to Kafka topics using a Kafka output plugin. Each topic is dedicated to a specific data type. Furthermore, the Prometheus monitoring tool and regular system audits ensure the stability and security of the data collection system. 1-2) Data preprocessing phase: By using applications based on the Kafka consumer API, real-time processing of Kafka received data is achieved; The data cleaning process includes data verification, regular expression filtering, and deduplication to ensure the accuracy and uniqueness of the data; Data conversion logic includes timestamp standardization, data type conversion, and data windowing to accommodate different analysis needs; The data formatting phase converts the converted data into JSON format, using serialization and deserialization techniques to keep the data structure standardized while retaining semantic information.

3. The method for identifying false alarms based on deep learning and text analysis according to claim 1, characterized in that: The specific steps of step 2) are as follows: 2-1) Feature Extraction: By reviewing and cleaning the raw log data, key fields such as IP addresses, port numbers, and protocol types are identified and converted into a structured and standardized form. Timestamps are standardized and numerical features are normalized, converting the log data into standardized feature vectors suitable for model learning. 2-2) RNN Model Design: During the model design phase, LSTM units were used to construct hidden layers to process sequence data and capture the time-dependent features in the data. A dropout layer was also introduced to prevent overfitting. The model's input layer receives the processed feature vector, while the output layer outputs the binary prediction probability of the security event using a sigmoid activation function. 2-3) Model Training: We used a binary cross-entropy loss function to optimize the binary classification problem. We used a batch size of 256 for training and set 50 epochs to ensure sufficient model learning. We set the learning rate to 0.01 and used the Adam optimizer to adaptively adjust the learning rate to accelerate convergence and improve performance. During the training process, the model calculates outputs through forward propagation and updates weights through backpropagation, iterating continuously to minimize losses. Ultimately, an RNN model is trained that can accurately identify attack events. 2-4) Anomaly Detection: Use logistic regression as a classifier for anomaly detection. Extract features from the sequence of security events to train the logistic regression model. Set a probability threshold Y based on the training results. During the detection process, events exceeding the threshold Y are classified as anomalies, triggering an anomaly alarm, and are therefore judged as anomalies, thus completing the anomaly detection process.

4. The method for identifying false alarms based on deep learning and text analysis according to claim 1, characterized in that: The specific steps of step 3) are as follows: (3-1) Use a word segmenter to perform word segmentation: Use a word segmenter to extract key information from network traffic packets related to security alerts, including the network communication five-tuple {source / destination IP, source / destination port and protocol number} and the header and body of the request and response; perform decoding, standardize special strings, split the decoded strings into words, and filter out irrelevant invalid strings including numbers and HTML tags, and finally generate a word sequence; (3-2) Word2Vec encoding: The pre-trained Word2Vec model is used to convert the word sequence output by the word segmenter into a dense vector form, which can capture the semantic relationship and contextual information between words and provide feature representation for the Transformer-based text analysis model; (3-3) Transformer-based classifier: A neural network model is constructed, using embedding layers and positional encoding to integrate the vectors generated by Word2Vec and preserve word order information. Multiple encoder layers and a self-attention mechanism are used to understand the context and long-range dependencies in the text. Finally, a fully connected output layer uses a sigmoid activation function to perform binary classification predictions and output a probability score indicating the likelihood that the activity is a false alarm. This achieves efficient recognition and classification of normal and abnormal behaviors in security alerts. 3-4) Anomaly Detection: After training and validation, the Transformer classifier is deployed in a real-time network monitoring system to analyze traffic packets and identify and filter false positives in security alerts.

Images