A network data security alarm intelligent collection management and alarm noise reduction processing method

Abstract

This invention discloses a method for intelligent collection, management, and noise reduction of network data security alarms. In the field of network data security technology, this invention solves the problem of heterogeneous alarm data aggregation. Through standardized management, it achieves unified collection of alarm data from security devices of different manufacturers and types, eliminating data format and semantic differences and laying a data foundation for subsequent analysis. This invention collects raw security alarm data from various heterogeneous network data security devices in real-time or near real-time through a preset adapter or API interface; it cleans and standardizes the collected raw alarm data, specifically including field mapping and normalization, format conversion, invalid data filtering, and basic field filling; and it extracts feature vectors from the standardized alarm events, including event content features, context features, correlation features, and device / rule features.

Description

Technical Field

[0001] This invention relates to the field of network data security technology, specifically to a method for intelligent collection, management, and noise reduction of network data security alarms. Background Technology

[0002] Driven by the digital wave, the situation regarding network data security is becoming increasingly severe.

[0003] However, the existing security protection system has many drawbacks: First, various security devices lack a unified alarm data standard, and the data formats and semantics are inconsistent, making it difficult to collect and analyze alarm data centrally and making aggregation difficult; Second, device defense strategies overlap, and a single security event can easily trigger multiple device alerts, generating a large number of duplicate or related alarms, forming redundant information and keeping the number of alarms high; Third, the devices themselves have false alarms, and normal business traffic can easily trigger low-risk alarms, with such invalid alarms accounting for the vast majority of the total, forming "alarm noise"; Fourth, a large number of low-value alarms can easily overwhelm critical security events, and there is a lack of intelligent sorting and filtering mechanisms, resulting in low efficiency for security operations and maintenance personnel in judgment and analysis, and difficulty in quickly identifying high-risk threats.

[0004] While some data quality assessment products can perform data integrity checks across specific dimensions, and some technologies can filter multi-format data through the configuration of Flume's collection and aggregation layers, none of them offer an integrated collection, governance, and intelligent noise reduction solution specifically for the heterogeneity and high noise characteristics of network security alarms. Therefore, there is an urgent need to develop an intelligent method for collecting, governing, and reducing noise in network data security alarms to improve alarm quality and security operation efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for intelligent collection, management, and noise reduction of network data security alarms, solving the problems of difficulty in aggregating network data security alarm data from heterogeneous products, numerous alarms, many false alarms, and difficulty in analysis and judgment.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligent collection, management, and noise reduction of network data security alarms, which specifically includes the following steps:

[0007] S1: Intelligent collection and standardized management of heterogeneous alarms;

[0008] S11: Collect raw security alarm data from various heterogeneous network data security devices in real time or near real time through preset adapters or API interfaces;

[0009] S12: Clean and standardize the collected raw alarm data, including field mapping and normalization, format conversion, invalid data filtering, and basic field filling.

[0010] S2: Feature engineering and information entropy calculation;

[0011] S21: Extract feature vectors from the standardized alarm events, wherein the feature vectors include event content features, context features, correlation features, and device / rule features;

[0012] S22: Calculate the information entropy values ​​of key feature dimensions, including event type entropy H_type, source IP entropy H_srcIP, destination IP entropy H_dstIP, destination port entropy H_dstPort, time burst entropy H_burst, and text description entropy H_text;

[0013] S23: Combine the entropy values ​​of each dimension into a multi-dimensional feature vector to represent the comprehensive information content of the alarm event;

[0014] S3: Event Importance Modeling and Real-time Scoring;

[0015] S31: Prepare labeled data, label historical alarm data and manual handling records as important alarms (label 1) and noisy alarms (label 0), select at least one supervised learning algorithm among logistic regression, gradient boosting tree, and support vector machine, use the multidimensional feature vector obtained in S2 as input and the labeled labels as output, and train the event importance classification model.

[0016] S32: For newly arrived standardized alarm events, perform feature extraction and information entropy calculation in S2 to obtain feature vectors, input them into the trained model, and output an importance score in the range of 0.0-1.0 or a high, medium, low, or noisy importance level.

[0017] S4: Dynamic Denoising and Model Self-Update

[0018] S41: Set importance score threshold and information entropy threshold. Alarms with scores below the threshold or key entropy values ​​below the threshold are judged as noise events and automatically masked or archived. Non-noise events are presented in a graded manner according to importance score. High-importance events are pushed first and trigger automatic response. Medium-importance events are merged and aggregated for display. Low-importance events are displayed in the underlying list.

[0019] S42: Continuously collect standardized data, feature entropy values, and manual handling labels of processed alarms, and retrain the model according to a preset period combined with time decay weights. After the new model is verified, it automatically replaces the old model, realizing adaptive iterative updates of the model.

[0020] Preferably, the heterogeneous network data security device includes a firewall, IDS / IPS, WAF, EDR, SIEM, and vulnerability scanner.

[0021] Preferably, the field mapping and normalization maps semantically identical fields from different sources to fields in a unified data model; the format conversion converts data in Syslog, JSON, XML, CEF, and LEEF formats into a unified JSON internal representation format; the invalid data filtering removes alarms from sources / destination IPs that are reserved addresses, test addresses, or contain specific test identifiers; and the basic field filling fills or corrects missing timestamp fields to the ISO8601 standard format.

[0022] Preferably, the event content features in S21 include event type / ID, alarm level, and keywords in the descriptive text after NLP word segmentation and stop word removal; the context features include the frequency of occurrence of the same / similar events within a specific time window and the activity and uniqueness of the source IP / destination IP / destination port; the association features include the number and type of alarms associated with the same source IP, destination IP, user, or asset; and the device / rule features include the alarm device type, manufacturer, and detection rule ID.

[0023] Preferably, the entropy values ​​of the event type entropy H_type, source IP entropy H_srcIP, destination IP entropy H_dstIP, destination port entropy H_dstPort, time burst entropy H_burst, and text description entropy H_text in S22 are calculated using the following formulas: H_type = -log2(P(type)), H_srcIP = -log2(P(srcIP)), H_dstIP = -log2(P(dstIP)), H_dstPort = -log2(P(dstPort)), and H_burst = -log2(P(burst)). Wherein, P(type) is the probability of the event type occurring in historical data, P(srcIP) is the probability distribution of the source IP triggering an alarm within a specific time window, P(dstIP) is the probability distribution of the destination IP, P(dstPort) is the probability distribution of the destination port, and P(burst) is the probability of the event burst frequency relative to the baseline frequency.

[0024] Preferably, in step S11, the adapter supports Syslog, SNMP, and HTTP / HTTPS protocols, and the API interface supports RESTful and SOAP standard interfaces.

[0025] Preferably, in step S31, the model training process uses cross-validation to optimize parameters in order to improve the model's recognition accuracy.

[0026] Preferably, in step S42, the preset model retraining cycle is once a week, and the time decay weight is set to have a higher weight for recent samples than for older samples.

[0027] Beneficial effects

[0028] This invention provides a method for intelligent collection, management, and noise reduction of network data security alarms. Compared with existing technologies, it has the following advantages:

[0029] This invention solves the problem of heterogeneous alarm data aggregation. Through standardized governance, it achieves unified collection of alarm data from security devices of different manufacturers and types, eliminating data format and semantic differences and laying a data foundation for subsequent analysis. It also achieves intelligent alarm noise reduction by quantifying the importance of alarm events based on information entropy and combining it with a supervised learning model to complete alarm importance scoring. This can accurately filter out invalid noise alarms and significantly reduce the monitoring burden on security operations personnel. Furthermore, it improves the efficiency of security incident analysis by prioritizing high-risk security incidents through alarm classification and automated response, preventing critical threats from being overwhelmed by a massive number of low-value alarms. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] A method for intelligent collection and management of network data security alarms and for handling alarm noise reduction using artificial intelligence, specifically including the following steps:

[0033] Heterogeneous Alarm Intelligent Collection and Standardized Governance S11: Raw Alarm Data Collection Raw security alarm data is collected in real-time or near real-time from various heterogeneous network data security devices such as firewalls, IDS / IPS, WAF, EDR, SIEM, and vulnerability scanners via pre-configured adapters or API interfaces. The adapters support multiple protocols such as Syslog, SNMP, and HTTP / HTTPS, while the API interfaces support standard interfaces such as RESTful and SOAP, ensuring the real-time performance and integrity of the collected data. S12: Standardized Data Processing The collected raw alarm data is cleaned and standardized to eliminate data heterogeneity.

[0034] Field Mapping and Normalization: Identify and extract key information such as timestamp, source IP, destination IP, source port, destination port, protocol, event type, alarm level, device manufacturer / model, description information, and original message digest. Map semantically similar fields from different devices to a unified data model field. For example, map “event_type” of device A and “alert_type” of device B to “event_type”.

[0035] Format conversion: Convert raw data in different formats such as Syslog, JSON, XML, CEF, and LEEF into JSON format for internal representation to ensure data structure consistency.

[0036] Invalid data filtering: Remove invalid or test-related alarms with source / destination IPs such as 10.0.0.0 / 8, 172.16.0.0 / 12, 192.168.0.0 / 16, test addresses such as 127.0.0.1, and test identifiers such as "test" or "demo".

[0037] Basic field filling: Fill or correct missing timestamps and other necessary fields. For fields without timestamps, supplement them according to the collection time or device system time, and uniformly convert them to the ISO8601 standard format.

[0038] Feature Engineering and Information Entropy Calculation S21: Feature Vector Extraction. From the standardized alarm events, feature vectors are extracted for information entropy calculation and model training, specifically divided into four categories:

[0039] Event content characteristics: Includes event type / ID, alarm level, and descriptive text keywords extracted after jieba word segmentation and stop word removal, such as "SQL injection" and "port scan".

[0040] Contextual features: include the frequency of occurrence of the same / similar events within the most recent 1 minute, 5 minutes, 1 hour, etc., as well as the activity and uniqueness of the source IP / destination IP / destination port within the corresponding time window.

[0041] Association characteristics: Includes the number and type of other alarm events associated with the same source IP, destination IP, user, or asset.

[0042] Device / Rule Characteristics: Includes the device type, manufacturer, and specific detection rule ID that generated the alarm. S22: Calculate the information entropy value. Based on information theory, calculate the information entropy of key feature dimensions to quantify the "unexpectedness" and information content of the alarm event. The higher the entropy value, the rarer and more important the event.

[0043] Event type entropy H_type: The formula is H_type = -log2(P(type)), where P(type) is the probability of this event type occurring in historical data such as the past 24 hours. For example, if the probability of an "SQL injection" event is 0.001, then H_type ≈ 9.97, indicating that this event type is rare and has a high entropy value.

[0044] Source IP entropy H_srcIP: The calculation formula is H_srcIP=-log2(P(srcIP)), where P(srcIP) is the probability distribution of the frequency of alarms triggered by a source IP within a specific time window relative to all source IPs. For example, if a source IP triggers only 1 alarm in 1 hour, while the total number of alarms during the same period is 100, then P(srcIP)=0.01, H_srcIP=6.64, indicating that the source IP triggers alarms at an extremely low frequency.

[0045] Destination IP entropy H_dstIP and destination port entropy H_dstPort: calculated as H_dstIP=-log2(P(dstIP)) and H_dstPort=-log2(P(dstPort)) respectively, where P(dstIP) is the probability distribution of the destination IP and P(dstPort) is the probability distribution of the destination port. The entropy value is higher for non-standard ports or critical asset ports that are accessed infrequently.

[0046] Temporal burst entropy H_burst: The calculation formula is H_burst=-log2(P(burst)), where P(burst) is the probability of the frequency of the same type or same source IP events relative to the baseline frequency within a short window of 1 minute before and after the event occurs. Events with a burst frequency much higher than the baseline have higher entropy values.

[0047] Text description entropy H_text (optional): Perform TF-IDF processing on the alarm description text to calculate keyword rarity entropy or text perplexity. Anomaly description text has a higher entropy value. S23: Construct feature vectors. Combine the entropy values ​​of the above dimensions into multi-dimensional feature vectors, which serve as the core basis for subsequent model training and alarm scoring.

[0048] Event Importance Modeling and Real-Time Scoring S31: Training the Importance Model

[0049] Data preparation for labeling: Collect historical alarm data and manual handling records, label valid alarms confirmed by manual handling as 1, and false alarms or ignored alarms as 0. At the same time, integrate sample data such as historical risk source IPs to form a training dataset.

[0050] Model Selection and Training: At least one supervised learning algorithm, such as Logistic Regression, GBDT, XGBoost, LightGBM, or SVM, is selected. Using the multidimensional feature vectors of S2 as input and labeled data as output, an event importance classification model is trained. During training, methods such as 5-fold cross-validation are used to optimize model parameters such as learning rate and tree depth to improve model recognition accuracy. S32: Real-time Scoring. For newly added standardized alarm events, feature extraction and information entropy calculation are first performed using S2 to obtain the corresponding feature vectors. These vectors are then input into the trained model. The model outputs an importance probability score of 0.0-1.0 or an importance level of high, medium, low, or noisy. A higher score indicates a more critical alarm event.

[0051] Dynamic Denoising and Model Self-Update S41: Dynamic Denoising

[0052] Noise filtering: Set importance score thresholds (e.g., 0.2) and information entropy thresholds (e.g., 3). High-frequency common events with scores below the thresholds or key entropy values ​​such as H_type below the thresholds are identified as noise, automatically blocked or archived, and not pushed to security operations personnel.

[0053] Tiered Presentation: Non-noise events are displayed in a tiered manner based on their scores. High-importance events with scores ≥ 0.8 are prioritized and highlighted on the dashboard, triggering automated responses such as blocking the source IP. Medium-importance events with scores ≤ 0.5 and < 0.8 are aggregated and displayed, such as merging multiple port scan alarms from the same source IP into a single alarm. Low-importance events with scores ≤ 0.2 and < 0.5 are displayed in a lower-level list for operations personnel to view as needed. S42: Model Self-Update

[0054] Data collection: Continuously collect newly generated processed alarm data, including standardized data, feature entropy values, and manual handling results, as new labeled samples.

[0055] Regular retraining: The model is retrained weekly using newly labeled data accumulated over the past four weeks, combined with time decay weights (1.0 for recent samples and 0.5 for older samples) to adapt the model to new attack patterns and network environments.

[0056] Seamless switching: Once the new model has been verified to have an accuracy of ≥95%, it will automatically replace the old online model, enabling iterative updates without human intervention and continuously ensuring the accuracy of alarm importance judgment.

[0057] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0058] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for intelligent collection, management, and noise reduction of network data security alarms, characterized in that, Includes the following steps: S1: Intelligent collection and standardized management of heterogeneous alarms; S11: Collect raw security alarm data from various heterogeneous network data security devices in real time or near real time through preset adapters or API interfaces; S12: Clean and standardize the collected raw alarm data, including field mapping and normalization, format conversion, invalid data filtering, and basic field filling. S2: Feature engineering and information entropy calculation; S21: Extract feature vectors from the standardized alarm events, wherein the feature vectors include event content features, context features, correlation features, and device / rule features; S22: Calculate the information entropy values ​​of key feature dimensions, including event type entropy H_type, source IP entropy H_srcIP, destination IP entropy H_dstIP, destination port entropy H_dstPort, time burst entropy H_burst, and text description entropy H_text; S23: Combine the entropy values ​​of each dimension into a multi-dimensional feature vector to represent the comprehensive information content of the alarm event; S3: Event Importance Modeling and Real-time Scoring; S31: Prepare labeled data, label historical alarm data and manual handling records as important alarms and noisy alarms, select at least one supervised learning algorithm from logistic regression, gradient boosting tree, and support vector machine, use the multidimensional feature vector obtained in S2 as input and the labeled labels as output, and train the event importance classification model. S32: For newly arrived standardized alarm events, perform feature extraction and information entropy calculation in S2 to obtain feature vectors, input them into the trained model, and output an importance score in the range of 0.0-1.0 or a high, medium, low, or noisy importance level. S4: Dynamic Denoising and Model Self-Update S41: Set importance score threshold and information entropy threshold. Alarms with scores below the threshold or key entropy values ​​below the threshold are judged as noise events and automatically masked or archived. Non-noise events are presented in a graded manner according to importance score. High-importance events are pushed first and trigger automatic response. Medium-importance events are merged and aggregated for display. Low-importance events are displayed in the underlying list. S42: Continuously collect standardized data, feature entropy values ​​and manual handling labels of processed alarms, retrain the model according to a preset period and time decay weight, and automatically replace the old model after the new model is verified to achieve adaptive iterative update of the model. The heterogeneous network data security devices include firewalls, IDS / IPS, WAF, EDR, SIEM, and vulnerability scanners.

2. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, The field mapping and normalization maps semantically identical fields from different sources to fields in a unified data model. The format conversion converts data in Syslog, JSON, XML, CEF, and LEEF formats into a unified JSON internal representation format. The invalid data filtering removes alarms from sources / destination IPs that are reserved addresses, test addresses, or contain specific test identifiers. The basic field filling fills in or corrects missing timestamp fields to the ISO8601 standard format.

3. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, The event content features in S21 include event type / ID, alarm level, and keywords in the descriptive text after NLP word segmentation and stop word removal. The context features include the frequency of occurrence of the same / similar events within a specific time window and the activity and uniqueness of the source IP / destination IP / destination port. The association features include the number and type of alarms associated with the same source IP, destination IP, user, or asset. The device / rule features include the alarm device type, manufacturer, and detection rule ID.

4. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, The entropy values ​​of the event type entropy H_type, source IP entropy H_srcIP, destination IP entropy H_dstIP, destination port entropy H_dstPort, time burst entropy H_burst, and text description entropy H_text in S22 are calculated using the following formulas: H_type = -log2(P(type)), H_srcIP = -log2(P(srcIP)), H_dstIP = -log2(P(dstIP)), H_dstPort = -log2(P(dstPort)), and H_burst = -log2(P(burst)). Here, P(type) is the probability of the event type occurring in historical data, P(srcIP) is the probability distribution of the source IP triggering an alarm within a specific time window, P(dstIP) is the probability distribution of the destination IP, P(dstPort) is the probability distribution of the destination port, and P(burst) is the probability of the event burst frequency relative to the baseline frequency.

5. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, In step S11, the adapter supports Syslog, SNMP, and HTTP / HTTPS protocols, and the API interface supports RESTful and SOAP standard interfaces.

6. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, In step S31, the model training process uses cross-validation to optimize parameters in order to improve the model's recognition accuracy.

7. The method for intelligent collection, management, and noise reduction of network data security alarms according to claim 1, characterized in that, In step S42, the preset model retraining cycle is once a week, and the time decay weight is set to give more weight to recent samples than to older samples.

Images