Safety alarm false alarm identification method based on deep learning and text analysis

Abstract

The invention discloses a safety alarm false alarm identification method based on deep learning and text analysis. The method comprises the following steps: 1) a data acquisition and processing stage; 2) detecting a network attack event by using a recurrent neural network (RNN) through extracting multi-dimensional features of a log; 3) detecting the false alarm of the network security event based on the text analysis of Word2Vec and Transformer; according to the method, through feature extraction and RNN model design, especially by adopting an LSTM unit to process time dependence, the identification capability of the model on the security event is improved, the method does not depend on traditional signature-based detection, and the dependence on misinformation is reduced; according to the method, the semantic information in the alarm content is deeply mined in combination with a text analysis technology based on Word2Vec and Transform, the accuracy of false alarm recognition is remarkably improved, an abnormal result is fed back to operation and maintenance personnel, and a more efficient and accurate network security protection solution is provided.

Description

Technical Field

[0001] The present invention relates to the field of network security and machine learning technology, and in particular to a security alarm false alarm recognition method based on deep learning and text analysis. Background Art

[0002] In modern enterprise network environments, false positive identification of security alerts is an increasingly serious problem. As the complexity and diversity of network attacks continue to increase, traditional signature-based intrusion detection systems (IDS) and anomaly-based detection methods often generate a large number of false positives when identifying real threats. These false positives not only consume a lot of time and energy of operation and maintenance personnel, but may also interfere with the normal operation of the enterprise.

[0003] Existing security alert processing methods mainly rely on analyzing threat indicators of alert events, or obtaining contextual information by analyzing the sequence relationship of alert events. However, these methods have obvious deficiencies in integrating threat indicators and contextual information. For example, they may ignore certain key information in alert attributes, such as whether the source IP address is from a foreign country, or whether the alert belongs to a category containing clear payloads (such as ransomware and botnets). In addition, existing methods often fail to fully utilize the information provided by cross-organizational intrusion detection system (IPS) monitors when analyzing alert context, such as the correlation of source and destination IP addresses.

[0004] In order to improve the detection efficiency of network attacks and the accuracy of alarm processing, a new method is needed to identify attacks and filter false alarms. Technologies based on deep learning and text analysis provide new possibilities for solving this problem. Deep learning models can automatically learn complex patterns from large amounts of data, while text analysis can deeply mine the semantic information in the alarm content to more accurately distinguish normal activities from malicious behaviors. By combining these two technologies, normal activities and malicious behaviors can be distinguished more accurately, significantly improving the efficiency and accuracy of alarm processing.

[0005] Therefore, the present invention proposes a security alarm false alarm identification method based on deep learning and text analysis. The method aims to improve the accuracy of false alarm identification and reduce interference with the normal operation of the enterprise by comprehensively utilizing the individual characteristics of the alarm and the contextual relationship information. The implementation of this method will help operation and maintenance personnel to handle alarms more effectively, thereby improving the performance of the entire network security protection system. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a security alarm false alarm identification method based on deep learning and text analysis.

[0007] A security alarm false alarm identification method based on deep learning and text analysis includes the following steps:

[0008] Step 1) Data collection and processing stage: Collect network traffic, security events, and system call logs from key data sources in vulnerable industries such as enterprises, schools, and factories, use Logstash to integrate, clean, and standardize the data, remove invalid data, and convert it into a unified JSON format to provide a structured data set for subsequent analysis.

[0009] Step 2) The system extracts multi-dimensional features of the logs and uses a recurrent neural network (RNN) to detect cyber attack events: extract key fields by exhaustively reviewing the data set, construct feature vectors, and design an RNN model to learn time dependencies using LSTM units. Model training is optimized through feature encoding and normalization processing, so that the model can identify abnormal behaviors from a large number of normal behaviors.

[0010] Normal behavior refers to expected, regular pattern activities in a network system that do not cause security issues or threats. For example, a user accessing an internal company website, using an authorized application, or performing daily work tasks during working hours are all examples of normal behavior.

[0011] Abnormal behavior refers to activities that deviate from normal patterns and may indicate security threats or issues. Abnormal behavior may include unauthorized access attempts, unusual network traffic patterns, suspicious login activities, malware execution, data leakage, etc. These behaviors may indicate that a malicious user is trying to breach system security or that the system may have been compromised.

[0012] Identifying abnormal behaviors from a large number of normal behaviors means that through machine learning models, the patterns of normal behaviors can be learned and understood, and behaviors that are significantly different from these patterns can be identified.

[0013] Step 3) Detect false positives of network security incidents based on text analysis of Word2Vec and Transformer: Combine the Word2Vec and Transformer models to perform text analysis, segment and encode the detection results of the RNN model, and use the self-attention mechanism to deeply understand (analyze) the content of network traffic data packets, and train a Transformer-based classifier to identify and reduce false positives in security alerts.

[0014] Step 4) Abnormal result feedback: The abnormal detection results after deep learning and text analysis are fed back to the operation and maintenance personnel to distinguish between benign and abnormal alarms, guide further evaluation and processing, and thus optimize the performance of the network security protection system.

[0015] In step 1), the detailed steps of the data collection and processing stage are as follows:

[0016] (1-1) Data collection phase: Through close collaboration with the network security team, we conducted a comprehensive review of the network architecture of enterprises, schools, factories and other industries to identify key data sources, and classified these data sources to lay the foundation for configuring the input plug-in for Logstash. As a data processing engine, Logstash is responsible for unifying and standardizing data collection, processing and forwarding, ensuring the efficiency, stability and security of the process. The data source is configured on Logstash and filtered as necessary by the filter. It is then forwarded to the Kafka topic through the Kafka output plug-in. Each topic is dedicated to a data type, which improves the collection efficiency. At the same time, the stability and security of the data collection system are guaranteed through the Prometheus monitoring tool and regular system audits.

[0017] (1-2) Data preprocessing phase: By using applications based on the Kafka consumer API, real-time processing of Kafka received data is achieved, and consumer groups are used to improve data processing efficiency and system fault tolerance. The data cleaning phase includes data validation, regular expression filtering, and deduplication to ensure data accuracy and uniqueness. The data conversion logic includes timestamp standardization, data type conversion, and data windowing to meet different analysis requirements. The data formatting phase converts the converted data into JSON format, using serialization and deserialization technology to keep the data structure standardized while retaining semantic information.

[0018] In step 2), the system extracts multi-dimensional features of the log and uses a recurrent neural network (RNN) to detect network attack events. The detailed steps are as follows:

[0019] (2-1) Feature extraction: By reviewing and cleaning the original log data, key fields such as IP address, port number and protocol type are identified, and these fields are converted into a format that is easy for the model to understand (i.e. structured and standardized form). At the same time, timestamps are standardized and numerical features are normalized to ensure data consistency and accuracy. These processing steps convert log data into standardized feature vectors suitable for model learning, laying the foundation for subsequent RNN model training.

[0020] (2-2) RNN model design: In the model design phase, LSTM units are used to construct hidden layers to process sequence data and capture time-dependent features in the data (time-dependent features refer to features in the data that change over time. These features are crucial for understanding the development of events and predicting future behaviors, such as frequent data access requests in a short period of time, or specific types of network traffic patterns, which are features that change over time and may indicate security threats). At the same time, a dropout layer is introduced to prevent overfitting. The input layer of the model receives the processed feature vector, while the output layer outputs the binary prediction probability of the security event through the sigmoid activation function. This design enables the RNN model to efficiently detect existing security threats.

[0021] (2-3) Model training: First, a binary cross entropy loss function is set to optimize the binary classification problem. A batch size of 256 data is used for training, and 50 epochs are set to ensure that the model is fully learned. The learning rate is set to 0.01. With the Adam optimizer, it can adaptively adjust the learning rate, accelerate convergence and improve performance. During the training process, the model calculates the output through forward propagation, updates the weights through backpropagation, and continuously iterates to minimize the loss. Finally, an RNN model that can accurately identify attack events is trained.

[0022] In step 3), the detailed steps of detecting false positives of network security incidents based on text analysis of Word2Vec and Transformer are as follows:

[0023] (3-1) Use a word segmenter for word segmentation: The word segmenter is responsible for extracting key information from network traffic packets related to security alerts, including the network communication quintuple (source / destination IP, source / destination port, and protocol number) and the header and body of the request and response (referring to the header of the request and response and the body of the request and response). It performs decoding, standardizes special strings, splits the decoded strings into words, and filters out irrelevant invalid strings such as numbers, HTML tags, etc., and finally generates a word sequence that is useful for text analysis, preparing for subsequent Word2Vec encoding and Transformer-based model processing.

[0024] (3-2) Word2Vec encoding: The pre-trained Word2Vec model is used to convert the word sequence output by the tokenizer into dense vector form. These vectors can capture the semantic relationship and contextual information between words, providing rich feature representation for the Transformer-based text analysis model.

[0025] (3-3) Transformer-based classifier: An advanced neural network model is constructed, which uses embedding layers and positional encoding to integrate the vectors generated by Word2Vec and retain the word order information. Through multiple encoder layers and the application of self-attention mechanism, the model can deeply understand the context and long-distance dependencies in the text. Finally, a fully connected output layer uses a sigmoid activation function to perform binary classification prediction and output a probability score representing the possibility of the activity being a false alarm, achieving efficient recognition and classification of normal activities and malicious behaviors in security alerts.

[0026] (3-4) Anomaly detection: After training and verification, the Transformer classifier can be deployed in a real-time network monitoring system to analyze traffic packets and identify and filter false positives in security alerts.

[0027] In step 4), the detailed steps of abnormal result feedback are as follows:

[0028] (4-1) Abnormal result feedback: The security alerts analyzed are classified as false alarms or real security threats. Based on the classification results, the operation and maintenance personnel ignore or remove the false alarms to reduce interference; and respond to and handle the confirmed security threats urgently to ensure a quick and effective response to real attacks and improve the accuracy and efficiency of network security protection.

[0029] The beneficial effects of the present invention are mainly manifested in:

[0030] 1) During the data collection and preprocessing stage, we worked with the network security team to comprehensively review the network architecture of vulnerable industries such as enterprises, schools, and factories, identify key data sources, and use Logstash to uniformly process and standardize data, ensuring the efficiency and stability of data collection, processing, and forwarding.

[0031] 2) In the data preprocessing stage, by using applications based on the Kafka consumer API, real-time data reception, cleaning, formatting and conversion are achieved, improving data processing efficiency and data quality.

[0032] 3) Through feature extraction and RNN model design, especially the use of LSTM units to handle time dependencies, the model's ability to identify security events is improved. This method does not rely on traditional signature-based detection and reduces reliance on false positives.

[0033] 4) Combining text analysis technology based on Word2Vec and Transformer, it deeply mines the semantic information in the alarm content, significantly improves the accuracy of false alarm identification, and feeds back abnormal results to operation and maintenance personnel, providing a more efficient and accurate network security protection solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings.

[0036] Reference Figure 1 , a security alarm false alarm identification method based on deep learning and text analysis, comprising the following steps:

[0037] Step (1) Data collection and processing stage: Collect network traffic, security events, and system call logs from key data sources in vulnerable industries such as enterprises, schools, and factories, use Logstash to integrate, clean, and standardize the data, remove invalid data, and convert them into a unified JSON format to provide a structured data set for subsequent analysis.

[0038] In step (1), the detailed steps of data collection and preprocessing are as follows:

[0039] (1-1) Data collection phase: During the data collection process, through long-term close cooperation with the network security team, a comprehensive review of the network architecture of enterprises, schools, factories and other industries was conducted to identify key data sources. These data sources include but are not limited to network edge devices, servers, applications and endpoints; subsequently, the data was carefully classified into network traffic logs, security event logs and system call logs, etc., which provided the basis for configuring appropriate input plug-ins for Logstash.

[0040] Logstash is an open source server-side data processing engine that can dynamically unify data from different data sources and standardize the data to the selected destination. This method uses Logstash to ensure the efficiency and stability of data collection, processing and forwarding, as well as the security of data. After the data source is determined and classified, Logstash is installed and configured.

[0041] Logstash is installed on an appropriate server or virtual machine to ensure that it has access to all key data sources. Subsequently, Logstash input plugins are configured for different types of log data, for example, the file input plugin is used to monitor and read file logs, and the syslog input plugin is used to receive logs sent over the network. After configuring the input plugin, Logstash filters are further configured to apply necessary filtering rules, such as filtering based on log level, error code, or other key fields. The filtered data is forwarded to Kafka through the Kafka output plugin.

[0042] In Kafka, specialized Kafka topics are designed according to the log type, ensuring that each topic is dedicated to one type of data for easy management and processing. At the same time, an appropriate number of partitions is configured for each topic to support parallel processing and improve throughput.

[0043] In order to ensure the stability and efficiency of the data collection system, the data collection process is also monitored and maintained. The Prometheus monitoring tool is used to monitor the performance indicators of Kafka and the log processing status of Logstash. At the same time, regular system audits are performed to check the security and efficiency of data collection and transmission, ensuring the smoothness and security of the entire data collection process.

[0044] (1-2) Data preprocessing stage: In the data preprocessing stage, the data received from Kafka needs to be converted into a clean, structured form suitable for further analysis. To this end, we first used consumer applications based on the Kafka consumer API. These applications subscribed to specific Kafka topics to receive data streams in real time. By configuring consumer groups, we achieved the collaborative work of multiple consumer instances, which not only improved the efficiency of data processing, but also enhanced the fault tolerance of the system.

[0045] In the data cleaning phase, a data validation mechanism was first used to identify and exclude malformed or incomplete log entries. This includes integrity checks on data fields and validation of data types. For example, if a log entry is expected to contain "id", "timestamp", and "action_type" fields, the data validation mechanism will check whether these fields exist and whether "timestamp" is in a valid time format. Secondly, regular expressions are used to filter out irrelevant or low-value data, for example, to remove meaningless log padding or advertising traffic. In addition, the data is deduplicated to identify and exclude duplicate log entries to ensure data uniqueness.

[0046] The data conversion logic is implemented through the following three data conversion logics:

[0047] 1) Timestamp standardization: In the processed data, timestamps may exist in multiple formats, such as "2024-07-2414:30:00" or "20240724143000"; in order to unify the time format, we convert all timestamps to the ISO 8601 standard format (i.e. "YYYY-MM-DDTHH:MM:SS"); for example, convert "20240724143000" to "2024-07-24T14:30:00".

[0048] 2) Data type conversion: Data type conversion is the process of converting data from one format to another to meet analysis requirements. For example, some fields may be string types, but mathematical operations are actually required, such as converting the string "123.45" to the floating point number 123.45. In addition, some date fields may need to be converted from string to date type for date-related calculations and comparisons.

[0049] 3) Data windowing: For time series data, use windowing technology to split the data into fixed-size blocks; for example, if you need to analyze hourly traffic patterns, split the continuous timestamp data into one window per hour; assuming there is data from "2024-07-24T14:00:00" to "2024-07-24T15:00:00", split it into a window from 14:00 to 15:00, and analyze all the data in this window as a group.

[0050] Data formatting is another key step in data preprocessing; the processed data is uniformly converted into JSON format, because JSON format is highly flexible and easy to read, which is very suitable for data exchange and storage. During the conversion process, we used serialization and deserialization technology to ensure the standardization of data structure while retaining the semantics and contextual information of the data.

[0051] Step (2) The system extracts multi-dimensional features of the log and uses a recurrent neural network (RNN) to detect network attack events: extract key fields by exhaustively reviewing the data set, construct feature vectors, and design an RNN model to learn time dependencies using LSTM units. Model training is optimized through feature encoding and normalization processing, so that the model can identify abnormal behaviors from a large number of normal behaviors.

[0052] In step (2), the system extracts multi-dimensional features of the log and uses a recurrent neural network (RNN) to detect network attack events. The detailed steps are as follows:

[0053] (2-1) Feature extraction: It is a key step to convert raw log data into information that can be used by the model. First, data review is performed. This process involves a detailed analysis of each event in the data set to identify key fields such as the source IP address, destination IP address, port number, protocol type, and timestamp of the login attempt; these fields are the basis for understanding security events, so it is necessary to ensure that they are accurately recorded in the data set.

[0054] Feature construction is then performed, which is the process of converting the extracted data into useful features; the source IP address and the destination IP address need to be converted into country or region information through geolocation coding, which helps to identify abnormal behavior patterns that may be related to geographic location; for example, the source IP address in the original data is "204.20.25.36". Through the geolocation coding service, it can be determined that this IP address actually corresponds to a data center in Ohio, USA. In feature construction, this IP address can be converted into a country code (such as "US") and a regional label (such as "Ohio"). If a corporate network does not usually communicate with the region, but the system detects a large number of requests from the region, this may indicate abnormal behavior.

[0055] The port number and protocol type will be converted into one-hot encoding for processing according to the needs of the RNN model so that the model can better understand these categorical data. For example, if the port number is 80, it is usually used for HTTP communication, and the protocol type is "TCP". In one-hot encoding, the port number is represented as a binary vector in which the 80th bit is 1 and the rest are 0; the protocol type "TCP" will also be converted into a binary vector in which the bit corresponding to the "TCP" position is 1, and the position of other protocols such as "UDP" is 0. In this way, each feature is converted into a two-dimensional vector form, and the model can process them more easily.

[0056] The numerical features are then normalized. Normalization helps the model process data of different scales more effectively by scaling the feature values ​​to the range of 0 to 1. This is done using formula (1), which is to subtract the minimum value of each feature from the feature value and then divide it by the difference between the maximum and minimum values ​​of the feature.

[0057]

[0058] Finally, feature encoding is the process of converting categorical features into a numeric format that the model can process. For example, a categorical feature such as protocol type would be encoded using on e -hot encoding is converted to a binary vector so that each protocol type is represented by a unique binary pattern. This process not only improves the model's ability to handle categorical data, but also avoids the model from being biased towards certain categories during training.

[0059] After the above steps, the original log data is converted into a set of feature vectors that are rich, accurate and suitable for model learning, laying the foundation for subsequent RNN model design and training.

[0060] (2-2) RNN model design: Recurrent neural network (RNN) can capture the temporal dependency in data, which is crucial in security incident analysis. The model design is divided into the following four steps:

[0061] 2-2-1) Define the input layer of the model, and the input data is the feature vector sequence obtained in the feature extraction stage.

[0062] 2-2-2) Design the hidden layer and choose LSTM as the core unit type because LSTM has excellent ability to process sequence data, especially they can learn and remember long-term dependencies. The structure of the LSTM unit includes three gates: input gate, forget gate and output gate, which work together to allow the network to retain or forget information when necessary. In this way, the LSTM network can avoid the gradient vanishing or exploding problem of traditional RNN, and thus perform better in sequence prediction tasks.

[0063] When configuring the LSTM layer, set 128 units for each LSTM layer and set an appropriate number of neurons for each LSTM unit. The number of neurons depends on the complexity of the data and the requirements of the task. Typically, the initial hidden layer may have more neurons to capture a wider range of features, while subsequent layers may have fewer neurons for higher levels of abstraction and pattern recognition.

[0064] 2-2-3) Add a dropout layer. Add a dropout layer after the LSTM layer to randomly drop some neurons to prevent the model from overfitting. For example, setting the dropout rate to 0.2 means that 20% of the neurons in each layer will be randomly dropped during training.

[0065] 2-2-4) Define the output layer. For binary classification problems (such as false positives and non-false positives), the output layer usually has only one neuron and uses the sigmoid activation function. The activation function formula is as follows:

[0066]

[0067] in, σ is the sigmoid function, and z is the value input to the sigmoid function.

[0068] (2-3) Model training: Train the RNN model so that it can distinguish between false alarms and real security incidents.

[0069] First, define the loss function. For the binary classification problem, we use the binary cross entropy loss function, the formula is as follows:

[0070]

[0071] Where N is the number of samples, y i is the true label (0 or 1) of the i-th sample, p i is the probability that the model predicts that the i-th sample is 1.

[0072] Then the model parameters are set. In this study, the batch size is set to 256 to efficiently process data during training. The number of training rounds (epoch number) is set to 50 to ensure that the model can learn data features in a long enough time while avoiding overtraining. The learning rate is set to 0.01 to ensure that the step size is appropriate during the weight update process, which not only converges quickly but also avoids skipping the optimal solution.

[0073] Finally, select the optimizer. This method selects the Adam optimizer to update the network weights. The Adam optimizer is an adaptive learning rate optimization algorithm that combines the advantages of the RMSprop and Momentum methods and can provide different learning rates for different parameters, thereby accelerating convergence and improving model performance. During the training process, the Adam optimizer dynamically adjusts the learning rate of each parameter, which enables the network to learn more efficiently.

[0074] During the model training phase, the model will be trained multiple times using the training data set. Each iteration will calculate the output through forward propagation, and then update the weights through back propagation. In this process, the loss function is used as an indicator to evaluate the performance of the model and guide the update of the weights. The binary cross entropy loss function is particularly suitable for binary classification problems. It measures the difference between the probability distribution predicted by the model and the true label.

[0075] (2-4) Anomaly detection: Logistic regression is used as a classifier for anomaly detection. First, features are extracted from the security event sequence. These features are obtained through the previously trained RNN model and can represent the key attributes of the event. Then, these feature vectors are used to train the logistic regression model, and the model will adjust its parameters through formula (3).

[0076] After the logistic regression model is trained, a probability threshold Y is set based on the training results. In the actual detection process, events that exceed the threshold Y are classified as anomalies, of course there are some false positives in the middle. The selection of the threshold is based on the performance evaluation on the validation set to balance the sensitivity and specificity of the detection. When the logistic regression model receives a new feature vector, it outputs the probability that the event belongs to the anomaly category. If this probability value exceeds the preset threshold, the event will trigger an anomaly alarm, that is, it will be judged as an anomaly, thus completing the anomaly detection process.

[0077] Step (3) Detect false alarms of network security incidents based on text analysis of Word2Vec and Transformer: Combine Word2Vec and Transformer technology to perform text analysis, segment and encode the detection results of the RNN model, and use the self-attention mechanism to deeply understand the content of network traffic data packets, and train a Transformer-based classifier to identify and reduce false alarms in security alerts.

[0078] In step (3), the detailed steps of detecting false positives of network security incidents based on text analysis of Word2Vec and Transformer are as follows:

[0079] (3-1) Use the word segmenter to perform word segmentation:

[0080] 3-1-1) The tokenizer is responsible for extracting relevant information from network traffic packets that trigger security alerts. It captures the network communication quintuple (source / destination IP, source / destination port, and protocol number) as well as the header and body of the request and response. For example, suppose a packet has a source IP of 208.65.25.12, a destination IP of 208.80.152.201, a source port of 4444, a destination port of 80, and a protocol of TCP. The tokenizer extracts this information from the packet as part of the network communication quintuple.

[0081] 3-1-2) Decode the request / response header and body and standardize special strings. For example, an HTTP request may contain the following header information:

[0082] GET / index.html HTTP / 1.1

[0083] Host:www.example.com

[0084] User-Agent:Mozilla / 5.0

[0085] The response header may contain the following information:

[0086] HTTP / 1.1 200OK

[0087] Content-Type: text / html; charset=UTF-8

[0088] The tokenizer recognizes and extracts these headers, and the subsequent body content.

[0089] 3-1-3) Split the decoded string into words, using newline characters and spaces as delimiters. For example, if the request contains URL encoding, such as GET / search? q=example%20test HTTP / 1.1, the word separator needs to decode it into GET / search? q=example test HTTP / 1.1. Similarly, if the body contains escape characters or HTML entities, such as &; or <;, the word separator will also decode them accordingly.

[0090] 3-1-4) Split the decoded string into words, using line breaks and spaces as delimiters. For example, for the query parameter q=example test in the request body, after decoding, the tokenizer will split it into words ["example", "test"] based on spaces.

[0091] 3-1-5) Finally, the tokenizer will filter out irrelevant invalid strings, such as numbers, web page elements, or formatting symbols. For example, the data packet contains the JavaScript code var num = 123; the tokenizer will recognize 123 as a number and may filter it out based on the context as an invalid string. At the same time, if there is such or <script>的HTML标签，也会被识别并过滤。

[0092] 经过上述步骤处理后，假设原始数据包含以下文本User-Agent:Mozilla / 5.0，分词器最终会生成一个单词序列，如["User","Agent","Mozilla","5.0"]。分词器的这些步骤确保了只有对文本分析有用的文本内容被保留并用于后续的Word2Vec编码和Transformer模型处理。

[0093] (3-2)Word2Vec编码：Word2Vec能够捕捉单词之间的语义含义和上下文关系，本方法使用预训练的Word2Vec模型将单词转换为密集的向量表示。

[0094] 分词器的输出被送入Word2Vec模型，为每个单词生成向量，提供丰富的特征集以捕捉流量数据包内容的语义信息。

[0095] (3-3)基于Transformer的分类器：文本分析检测的核心是基于Transformer的神经网络模型，该模型擅长处理序列数据并捕捉文本中的长距离依赖关系。Transforme模型包括以下四部分：

[0096] 1、嵌入层：将Word2Vec向量转换为适合Transformer模型的格式。

[0097] 2、位置编码：向嵌入添加位置信息，以保留单词在序列中的顺序。

[0098] 3、编码器层：由多个编码器层组成，每层都应用自注意力机制来理解文本中的上下文。每层由多头注意力机制和前馈网络组成。

[0099] 4、输出层：一个全连接层，从Transformer编码器的输出中接收信息，并执行二分类，以确定活动是良性的还是恶意的。该层使用sigmoid激活函数输出概率分数，指示活动为误报的可能性。

[0100] 对模型进行训练时，采用二元交叉熵损失函数训练模型(公式2)，适合区分良性和恶意警报的二分类任务，并使用Adam优化器进行高效训练，在训练过程中不断调整学习率和其他超参数以实现最佳性能。除此之外，根据实际情况，还可以应用正则化技术，如dropout或L2正则化，以防止过拟合。

[0101] 经过训练，基于Word2Vec和Transformer的文本分析检测器可以从异常警报中检测出来正常的活动，以消除误报。

[0102] (3-4)异常检测：将经过训练和验证的Transformer分类器被部署在实时网络监控系统中，对流量数据包进行分析，以识别并过滤安全警报中的误报。此过程涉及将实时捕获的数据包通过分词、Word2Vec编码和Transformer模型处理，从而为每个事件生成一个表示其为误报的概率分数。根据设定的阈值，系统自动将那些概率分数超过阈值的事件分类为异常，而低于阈值的事件则被视为正常活动。

[0103] 步骤(4)异常结果反馈：将经过深度学习和文本分析的异常检测结果反馈给运维人员，区分良性和异常警报，指导进一步的评估和处理，从而优化网络安全防护体系的性能。

[0104] 步骤(4)中，异常结果反馈的详细步骤如下：

[0105] (4-1)异常结果反馈：经过上述两种异常检测方式，警报信息会被分为以下两类：

[0106] 1、良性警报：这些警报被识别为误报，即它们实际上并不代表恶意活动，而是正常业务操作或其他无害行为。

[0107] 2、异常警报：这些警报被确认为真正的安全威胁，需要运维人员进一步调查和采取相应的防御措施。

[0108] 将分析结果呈现给运维人员，运维人员可以根据分析结果对警报进行进一步的评估和处理，对于被标记为良性的警报，运维人员可能会选择忽略或将它们从警报队列中移除，以减少误报对企业正常运营的影响；对于被标记为异常的警报，运维人员可以进行深入的调查，采取诸如隔离受感染的系统、更新防火墙规则或进一步分析攻击来源等措施。

[0109] 以上操作有助于运维人员更有效地识别真正的攻击，减少正常网络活动对正常业务的干扰，同时确保对真实攻击的有效响应。本方法的实施将为网络安全防护提供一种更为高效和准确的解决方案。< / script>

Claims

1. A security alarm false alarm identification method based on deep learning and text analysis, characterized in that: The steps include: 1) Data collection and processing stage: Collect network traffic, security events and system call logs from relevant data sources, use Logstash to integrate, clean and standardize the data, remove invalid data, convert them into a unified JSON format, and obtain a structured data set; 2) By extracting multi-dimensional features of logs, we use recurrent neural networks (RNNs) to detect network attack events: By thoroughly reviewing the data set, we extract key fields, construct feature vectors, and design an RNN model to learn time dependencies using LSTM units. We optimize model training through feature encoding and normalization, so that the model can use a large number of normal behaviors to identify abnormal behaviors. Normal behaviors refer to expected activities in a network system that conform to regular patterns and do not cause security issues or threats. Abnormal behaviors refer to activities that deviate from normal behaviors and cause security issues or threats. 3) Detecting false positives of network security incidents based on text analysis of Word2Vec and Transformer: Combining Word2Vec and Transformer models, text analysis is performed to identify and extract features by segmenting and encoding the detection results of the RNN model and using the self-attention mechanism to analyze the content of network traffic packets, and train a Transformer-based classifier to identify and reduce false positives in security alerts; 4) Abnormal result feedback: The abnormal detection results after step 2) deep learning and step 3) text analysis are fed back to the operation and maintenance personnel to manually distinguish between benign and abnormal alarms, guide further evaluation and processing, and thus optimize the performance of the network security protection system.

2. According to claim 1, a security alarm false alarm identification method based on deep learning and text analysis is characterized in that: The specific steps of step 1) are as follows: 1-1) Data collection phase: Review the network architecture to identify the corresponding data sources and classify them; the data sources are configured on Logstash, filtered by filters first, and then forwarded to Kafka topics through the Kafka output plug-in, and each topic is dedicated to a data type; at the same time, the stability and security of the data collection system are guaranteed through the Prometheus monitoring tool and regular system audits; 1-2) Data preprocessing stage: By using applications based on the Kafka consumer API, real-time processing of Kafka received data is achieved; The data cleaning process includes data verification, regular expression filtering, and deduplication to ensure the accuracy and uniqueness of the data; Data transformation logic includes timestamp standardization, data type conversion, and data windowing to accommodate different analysis requirements; The data formatting phase converts the converted data into JSON format, using serialization and deserialization techniques to keep the data structure standardized while retaining semantic information.

3. The method for identifying false alarms of security alarms based on deep learning and text analysis according to claim 1, characterized in that: The specific steps of step 2) are as follows: 2-1) Feature extraction: By reviewing and cleaning the original log data, identifying key fields including IP address, port number and protocol type, and converting them into a structured and standardized form, at the same time, standardizing the timestamp and normalizing the numerical features, the log data is converted into a standardized feature vector suitable for model learning; 2-2) RNN model design: In the model design stage, LSTM units are used to construct hidden layers to process sequence data and capture the time-dependent features in the data, and dropout layers are introduced to prevent overfitting. The input layer of the model receives the processed feature vector, while the output layer outputs the binary prediction probability of the security event through the sigmoid activation function. 2-3) Model training: Set the binary cross entropy loss function to optimize the binary classification problem, use a batch size of 256 data for training, and set 50 epochs to ensure that the model is fully learned. The learning rate is set to 0.01, and the Adam optimizer is used to adaptively adjust the learning rate to speed up convergence and improve performance; During the training process, the model calculates the output through forward propagation, updates the weights through back propagation, and continuously iterates to minimize the loss, ultimately training an RNN model that can accurately identify attack events; 2-4) Anomaly detection: Use logistic regression as a classifier to perform anomaly detection; extract features from the sequence of security events to train the logistic regression model; set a probability threshold Y based on the training results. During the detection process, events that exceed the threshold Y are classified as anomalies, triggering an anomaly alarm, that is, they are judged as anomalies, thus completing the anomaly detection process.

4. The method for identifying false alarms of security alarms based on deep learning and text analysis according to claim 1, characterized in that: The specific steps of step 3) are as follows: (3-1) Use a word segmenter to perform word segmentation: Use a word segmenter to extract key information from network traffic packets related to security alerts, including the network communication five-tuple {source / destination IP, source / destination port and protocol number} and the header and body of the request and response; perform decoding, standardize special strings, split the decoded strings into words, and filter out irrelevant invalid strings including numbers and HTML tags, and finally generate a word sequence; (3-2) Word2Vec encoding: The pre-trained Word2Vec model is used to convert the word sequence output by the word segmenter into a dense vector form, so as to capture the semantic relationship and contextual information between words and provide feature representation for the Transformer-based text analysis model; (3-3) Transformer-based classifier: A neural network model is constructed, which uses embedding layers and positional encoding to integrate the vectors generated by Word2Vec and retain the word order information; through multiple encoder layers and the application of self-attention mechanism, the context and long-distance dependencies in the text are understood; finally, a fully connected output layer uses a sigmoid activation function to perform binary classification prediction and output a probability score indicating the possibility that the activity is a false alarm, thus achieving efficient recognition and classification of normal and abnormal behaviors in security alerts; 3-4) Anomaly Detection: After training and verification, the Transformer classifier is deployed in a real-time network monitoring system to analyze traffic packets and identify and filter false positives in security alerts.

Images