Multi-source heterogeneous network security alarm aggregation noise reduction method, system and device

Abstract

The invention discloses a multi-source heterogeneous network security alarm aggregation noise reduction method, system and device. The method comprises the following steps: step 1, constructing a network security alarm ontology model; step 2, standardization processing of multi-source alarm; 3, performing dynamic alarm aggregation based on an entity relationship; 4, multi-level alarm noise reduction and intelligent study and judgment are carried out; according to the method, deep standardization is carried out on multi-source heterogeneous alarms through the network security alarm ontology model, and a data semantic gap is eliminated; a dynamic aggregation strategy based on a core entity and a relationship is adopted, related alarms surrounding the same core entity in a specific time window are aggregated into aggregated alarms, the order of magnitude of the number of the alarms is reduced, and a complete attack scene description is formed; a three-layer progressive noise reduction mechanism is adopted, false alarms and low-value alarms are eliminated, and high credibility of output safety events is guaranteed; automatic conversion from massive original alarms to a small number of high-value security events is realized, and the security operation response speed and decision quality are improved.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a method, system, and device for aggregation and noise reduction of multi-source heterogeneous network security alarms. Background Technology

[0002] With the increasingly severe cybersecurity landscape, modern enterprises have widely deployed diverse cybersecurity devices, such as firewalls, intrusion detection systems, web application firewalls, and endpoint detection and response systems, to defend against various cyber threats. However, these security devices from different vendors and of different types exhibit significant differences in their descriptions, field definitions, and classification systems for the same security events. This results in cybersecurity alerts being multi-sourced and heterogeneous, specifically manifested in dispersed alert sources, inconsistent data formats, and inconsistent descriptions of core information.

[0003] This multi-source heterogeneous nature directly leads to the following problems: First, the number of alerts explodes, creating an alert storm, with a large amount of repetitive and redundant alert data flooding security operations scenarios; second, effective threat information is submerged in massive amounts of low-quality alerts, requiring security operations personnel to spend a lot of time filtering and analyzing alerts, resulting in low work efficiency; third, alerts across devices and platforms cannot be effectively correlated, making it difficult to reconstruct complete attack scenarios, which may ultimately lead to the overlooking of real high-risk threats, posing potential risks to enterprise network security. Existing technologies lack solutions that can systematically address the semantic gap in multi-source heterogeneous alerts and achieve accurate aggregation and efficient noise reduction, becoming a core bottleneck restricting security operation efficiency and threat identification accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and device for aggregation and noise reduction of multi-source heterogeneous network security alarms, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, this invention provides a method for aggregation and noise reduction of multi-source heterogeneous network security alarms, comprising the following steps: Step 1, constructing a network security alarm ontology model; Step 2, standardizing the processing of multi-source alarms; Step 3, dynamic alarm aggregation based on entity relationships; Step 4, multi-level alarm noise reduction and intelligent analysis.

[0006] In step one above, best practices from national standards, international standards, and mainstream industry frameworks are integrated to establish a network security alarm ontology model that includes standardization of alarm fields, alarm types, and entities and relationships in alarms.

[0007] In step two above, a streaming processing framework is used as the data processing engine to receive security alarms from multiple heterogeneous sources. After source identification and adaptation parsing, the alarms are converted into a standardized alarm data stream with a unified structure based on the network security alarm ontology model.

[0008] In step three above, the core entities and relationships in the standardized alarms are used as the key aggregation points. Combined with the time window that is dynamically adjusted according to the alarm type and behavioral characteristics, related alarms around the same core entity within a specific time window are aggregated into aggregated alarms.

[0009] In step four above, the aggregated alarms are denoised and intelligently analyzed through a three-layer progressive processing: automatic analysis and denoising based on built-in rules, adaptive denoising based on historical learning, and deep analysis and denoising based on artificial intelligence, which are executed in sequence to finally output a high-credibility security event.

[0010] Preferably, in step one, the standardization of alarm fields specifically involves: establishing a dynamically adjustable mapping table that covers the alarm formats of network security devices, mapping heterogeneous fields describing the same meaning in alarms from different data sources to unified semantic fields; the standardization of alarm types specifically involves: constructing a multi-dimensional alarm classification system based on attack targets, attack methods, vulnerability exploitation, attack consequences, and severity, and having functions for adding, deleting, and modifying alarms; the standardization of entities and relationships in alarms specifically involves: based on the standardization of alarm fields, extracting key entities and relationships between entities in alarms to form a structured knowledge graph of nodes and edges. Key entities include hosts, domain names, URLs, files, and processes, and relationships between entities include entity interaction behaviors such as connection creation, execution, and reading.

[0011] Preferably, step two specifically includes the following steps:

[0012] Step 2.1, Data Access and Processing: Real-time access to alarm data streams from network security devices via Kafka, Syslog, API, and file listening interfaces;

[0013] Step 2.2, Source Identification and Distribution Processing: Based on the device IP and alarm identifier of the alarm, automatically identify the device type and alarm format to which the alarm belongs;

[0014] Step 2.3, Adaptation and Standardization Processing: Based on the identification results, the alarm stream is distributed to the corresponding standardization processing program; the standardization processing program has built-in parsing logic for alarm formats of specific devices and mapping rules to the network security alarm ontology model, extracts key fields from the original alarm, converts data types, identifies key entities, and builds entity relationships to form a standardized alarm object;

[0015] Step 2.4, Output standardized data stream processing: Push the standardized alarm objects to the next stage message queue to form a unified structured alarm data stream.

[0016] Preferably, in step three, using the core entities and relationships in the standardized alarms as the key aggregation points specifically means: determining the core entities and core relationships from the standardized alarm data stream based on the type of network security device accessed and the alarm type defined by the network security device, and using the common core entities and core relationships as the aggregation basis.

[0017] Preferably, in step three, the time window W, which is dynamically adjusted based on the alarm type and behavioral characteristics, is calculated using the following formula:

[0018]

[0019] Among them, W base The base window is used for severity, which is the normalized alarm severity, ranging from 0 to 1. Persistence is the historical persistence estimate of this type of alarm behavior. α and γ are empirical coefficients that can be adjusted online.

[0020] Preferably, in step four, the automatic judgment and noise reduction based on built-in rules specifically involves: using an algorithm combining Bloom filters and hash tables to match standardized alarm data with rule items in a static rule base; marking the standardized alarm data that matches successfully; and entering adaptive noise reduction based on historical learning for unmarked standardized alarm data; wherein, the rule combination conditions in the static rule base include intrusion indicators, alarm types, attack tactics, attack techniques, and attack stages.

[0021] Preferably, in step four, the adaptive noise reduction based on historical learning employs methods including entity filtering, abstract relationship filtering, and contextual key feature filtering. All three filtering methods use an algorithm combining Bloom filters and hash tables. Marked alarms can have their priorities automatically adjusted by an automated orchestration process, or a response action can be executed. Unmarked alarms enter into deep analysis and noise reduction based on artificial intelligence. Specifically, entity filtering involves precisely comparing entities in standardized alarms with entities in the historical analysis entity database; if a match is found, the alarm is marked. Abstract relationship filtering involves extracting entities and relationships from standardized alarms and comparing them with abstract relationships in the historical abstract relationship analysis database; if a match is found, the alarm is marked. Contextual key feature filtering involves extracting contextual key features of entities and relationships from standardized alarms and comparing them with the historical contextual key feature analysis database; if a match is found, the alarm is marked. Contextual key features include network request data, network response data, and command execution data.

[0022] Preferably, in step four, the method for deep analysis and noise reduction based on artificial intelligence includes semantic understanding and intent analysis, automatic threat intelligence association, and contextual scenario construction. Specifically, semantic understanding and intent analysis involves using a large language model to perform deep semantic analysis on unstructured data and attack payload data in alarms to determine whether they are real attacks, thereby determining network behavior intent and threat level. Automatic threat intelligence association involves extracting key entities from aggregated alarms through an AI Agent, calling interfaces to query internal and external threat intelligence in real time, and marking the alarm if the key entity matches or is associated with malicious infrastructure or malicious samples. Contextual scenario construction involves using a large artificial intelligence model to extract standardized alarm data, comprehensively analyzing the entity relationships and behavioral logic within and across multiple aggregated alarms, constructing a complete attack story chain, and identifying abnormal behavior by combining historical alarm behavior and similar common behaviors.

[0023] This invention provides a multi-source heterogeneous network security alarm aggregation and noise reduction system, including a data access module, a parsing and standardization module, a data queue module, a dynamic alarm aggregation module, and a multi-level alarm noise reduction and intelligent judgment module. The parsing and standardization module establishes data connections with the data access module and the data queue module, respectively. The dynamic alarm aggregation module establishes data connections with the data queue module and the multi-level alarm noise reduction and intelligent judgment module, respectively. The data access module is used to access raw alarm data from various network security devices. It supports real-time aggregation of heterogeneous raw alarm data from different network security devices through Kafka, Syslog, API, and file listening interfaces, and can perform unified management operations such as adding, modifying, deleting, and searching on the data source, serving as the system's unified data entry point. The parsing and standardization module receives the raw alarm data output by the data access module and, according to the preset device alarm format parsing logic and a pre-built network security alarm ontology model, converts the fields, types, entities, and relationship mappings of heterogeneous alarms into a unified structured format. Standardized alarm data; the data queue module is used to cache and asynchronously distribute standardized alarm data output by the parsing and standardization module, decoupling upstream and downstream modules, smoothly handling data surges, and ensuring data processing stability in high-concurrency scenarios; the dynamic alarm aggregation module is used to obtain standardized alarm data from the data queue module, determine the core entities and core relationships in the standardized alarm data according to the type of network security device accessed and the alarm type defined by the device, and aggregate related standardized alarms around the same core entity within a specific time window into aggregated alarms based on common core entities and core relationships, combined with a time window dynamically adjusted according to alarm type and behavioral characteristics; the multi-layered alarm denoising and intelligent judgment module is used to receive aggregated alarms output by the dynamic alarm aggregation module, and achieves denoising and intelligent judgment through three-layer progressive processing: first, automatic judgment and denoising based on the built-in static rule base is performed, then adaptive denoising based on historical learning is performed, and finally, deep judgment and denoising based on artificial intelligence is performed, ultimately outputting highly reliable security events.

[0024] This invention provides a multi-source heterogeneous network security alarm aggregation and noise reduction device, including a memory body, a computer program and a processor body. The memory body stores a computer program that can run on the processor body, and the processor body implements the above-mentioned noise reduction method when executing the computer program.

[0025] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention deeply standardizes multi-source heterogeneous alarms through a network security alarm ontology model, eliminating the data semantic gap; it adopts a dynamic aggregation strategy based on core entities and relationships to aggregate related alarms around the same core entity within a specific time window into aggregated alarms, reducing the number of alarms by orders of magnitude and forming a complete attack scenario description; it employs a three-layer progressive noise reduction mechanism to eliminate false alarms and low-value alarms, ensuring the high credibility of output security events; and it achieves automated transformation from massive amounts of raw alarms to a small number of high-value security events, improving the speed of security operation response and the quality of decision-making. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention;

[0027] Figure 2 This is a standardized processing flowchart for multi-source alarms according to the present invention;

[0028] Figure 3 This is a system structure block diagram of the present invention;

[0029] Figure 4 This is a block diagram of the device structure of the present invention.

[0030] In the diagram: 1. Data access module; 2. Parsing and standardization module; 3. Data queue module; 4. Dynamic alarm aggregation module; 5. Multi-level alarm noise reduction and intelligent judgment module; 6. Memory unit; 7. Computer program; 8. Processor unit. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see the appendix Figure 1 - Appendix Figure 2 The present invention provides an embodiment of a method for aggregation and noise reduction of multi-source heterogeneous network security alarms, comprising the following steps: Step 1, constructing a network security alarm ontology model; Step 2, standardizing the processing of multi-source alarms; Step 3, dynamic alarm aggregation based on entity relationships; Step 4, multi-level alarm noise reduction and intelligent analysis.

[0033] In step one above, best practices from national standards, international standards, and mainstream industry frameworks are integrated to establish a network security alarm ontology model that includes standardization of alarm fields, alarm types, and entities and relationships within alarms. Specifically, alarm field standardization involves establishing a dynamically adjustable mapping table that covers the alarm formats of connected network security devices, mapping heterogeneous fields describing the same meaning from alarms from different data sources to unified semantic fields. Alarm type standardization involves constructing a multi-dimensional alarm classification system based on attack targets, attack methods, vulnerability exploitation, attack consequences, and severity, with functions for adding, deleting, and modifying alarms. Entity and relationship standardization within alarms involves extracting key entities and their relationships from alarms, forming a structured knowledge graph of nodes and edges. Key entities include hosts, domain names, URLs, files, and processes, while relationships between entities include entity interactions such as connection creation, execution, and reading.

[0034] In step two above, a streaming processing framework is used as the data processing engine to receive security alerts from multiple heterogeneous sources. After source identification and adaptation parsing, the alerts are converted into a standardized alert data stream with a unified structure based on the network security alert ontology model. This specifically includes the following steps:

[0035] 2.1 Data Access and Processing: Real-time access to alarm data streams from network security devices via Kafka, Syslog, API, and file listening interfaces;

[0036] 2.2 Source Identification and Distribution Processing: Based on the device IP and alarm identifier of the alarm, automatically identify the device type and alarm format to which the alarm belongs;

[0037] 2.3 Adaptation, parsing, and standardization: Based on the identification results, the alarm stream is distributed to the corresponding standardization processing program. The standardization processing program has built-in parsing logic for alarm formats of specific devices and mapping rules to the network security alarm ontology model. It extracts key fields from the original alarm, converts data types, identifies key entities, and builds entity relationships to form a standardized alarm object.

[0038] 2.4 Output Standardized Data Stream Processing: Push standardized alarm objects to the next stage message queue to form a unified structured alarm data stream;

[0039] In step three above, the core entities and relationships in standardized alarms are used as key aggregation points. Combined with a dynamically adjusted time window based on alarm type and behavioral characteristics, related alarms revolving around the same core entity within a specific time window are aggregated into aggregated alarms. Specifically, using the core entities and relationships in standardized alarms as key aggregation points involves: determining the core entities and core relationships from the standardized alarm data stream based on the type of network security device accessed and the alarm types defined by that device, using common core entities and core relationships as the aggregation basis; the dynamically adjusted time window W based on alarm type and behavioral characteristics uses the following formula:

[0040]

[0041] Among them, W base The base window is used for the alarm severity, which is normalized and ranges from 0 to 1. Persistence is the historical persistence estimate of this type of alarm behavior. α and γ are empirical coefficients that can be adjusted online.

[0042] In step four above, aggregated alarms undergo noise reduction and intelligent analysis through a three-layer progressive processing approach. This involves sequentially executing automatic noise reduction based on built-in rules, adaptive noise reduction based on historical learning, and deep noise reduction based on artificial intelligence, ultimately outputting high-confidence security events. Specifically, the automatic noise reduction based on built-in rules employs an algorithm combining Bloom filters and hash tables to match standardized alarm data with rule items in a static rule base. Successfully matched standardized alarm data is marked, while unmarked standardized alarm data enters the adaptive noise reduction based on historical learning. The rule combination conditions in the static rule base include intrusion indicators, alarm types, attack tactics, attack techniques, and attack stages. The adaptive noise reduction based on historical learning uses methods including entity filtering, abstract relationship filtering, and contextual key feature filtering. All three filtering methods utilize an algorithm combining Bloom filters and hash tables. Marked alarms can have their priorities automatically adjusted by an automated orchestration process or trigger response actions. Unmarked alarms enter the process based on manual... Intelligent deep analysis and noise reduction includes: entity filtering (precisely comparing entities in standardized alerts with entities in a historical entity database, marking alerts upon successful matching); abstract relationship filtering (extracting entities and relationships from standardized alerts, comparing them with abstract relationships in a historical abstract relationship database, marking alerts upon successful matching); and contextual key feature filtering (extracting contextual key features of entities and relationships from standardized alerts, comparing them with a historical contextual key feature database, marking alerts upon successful matching). Contextual key features include network request data, network response data, and command execution data. The AI-based deep analysis and noise reduction employs methods including semantic understanding and intent analysis, automatic threat intelligence association, and contextual scenario construction. Semantic understanding and intent analysis involves using a large language model to perform deep semantic analysis on unstructured data and attack payload data in alerts to determine if they are genuine attacks, thereby identifying network behavior intent and threat level. Automatic threat intelligence association utilizes AI... The agent extracts key entities from aggregated alerts and calls interfaces to query internal and external threat intelligence in real time. If a key entity matches or is associated with malicious infrastructure or malicious samples, the alert is marked. The context scenario construction is as follows: standardized alert data is extracted using a large artificial intelligence model, and the entity relationships and behavioral logic within and across multiple aggregated alerts are comprehensively analyzed to construct a complete attack story chain. Abnormal behavior is identified by combining historical alert behavior and similar common behaviors.

[0043] Please see the appendix Figure 3This invention provides an embodiment of a multi-source heterogeneous network security alarm aggregation and noise reduction system, comprising a data access module 1, a parsing and standardization module 2, a data queue module 3, a dynamic alarm aggregation module 4, and a multi-level alarm noise reduction and intelligent judgment module 5. The parsing and standardization module 2 establishes data connections with the data access module 1 and the data queue module 3, respectively. The dynamic alarm aggregation module 4 establishes data connections with the data queue module 3 and the multi-level alarm noise reduction and intelligent judgment module 5, respectively. The data access module 1 is used to access raw alarm data from various network security devices. It supports real-time aggregation of heterogeneous raw alarm data from different network security devices through Kafka, Syslog, API, and file listening interfaces, and can perform unified management operations such as adding, modifying, deleting, and searching on the data source, serving as the system's unified data entry point. The parsing and standardization module 2 receives the raw alarm data output by the data access module 1 and, according to the preset device alarm format parsing logic and the pre-built network security alarm ontology model, converts the fields, types, entities, and relationships of heterogeneous alarms into... The system provides unified, structured, and standardized alarm data. The data queue module 3 caches and asynchronously distributes the standardized alarm data output by the parsing and standardization module 2, decoupling upstream and downstream modules, smoothly handling data surges, and ensuring data processing stability in high-concurrency scenarios. The dynamic alarm aggregation module 4 retrieves standardized alarm data from the data queue module 3, determines the core entities and relationships within the standardized alarm data based on the type of network security device accessed and the alarm type defined by that device, and aggregates related standardized alarms around the same core entity within a specific time window into an aggregated alarm, using common core entities and relationships as the aggregation basis and combining a time window dynamically adjusted according to alarm type and behavioral characteristics. The multi-layered alarm noise reduction and intelligent judgment module 5 receives the aggregated alarms output by the dynamic alarm aggregation module 4, and achieves noise reduction and intelligent judgment through a three-layer progressive processing: first, automatic judgment and noise reduction based on a built-in static rule base; second, adaptive noise reduction based on historical learning; and finally, deep judgment and noise reduction based on artificial intelligence, ultimately outputting highly reliable security events.

[0044] Please see the appendix Figure 4The present invention provides an embodiment of a multi-source heterogeneous network security alarm aggregation and noise reduction device, comprising a memory body 6, a computer program 7, and a processor body 8. The memory body 6 stores the computer program 7, which can run on the processor body 8. When the processor body 8 executes the computer program 7, it implements the above-mentioned noise reduction method. The memory body 6 may be a non-transitory computer-readable storage medium, such as a solid-state drive, a mechanical hard drive, a read-only memory, a random access memory, a smart memory card, a secure digital card, a flash memory card, etc. The computer program 7 is a set of codes that implement the above-mentioned noise reduction method. The processor body 8 may be a hardware unit with data processing and program execution capabilities, such as a central processing unit, a microprocessor, or a digital signal processor.

[0045] Based on the above, the advantages of this invention are as follows: When used, it constructs a unified network security alarm ontology model, deeply standardizing heterogeneous alarms from different vendors and of different types. This transforms multi-source heterogeneous original alarms into structured entities, relationships, and object attributes, fundamentally solving the semantic gap problem between multi-source data. This makes it possible to effectively correlate and deeply analyze alarms across devices and platforms, providing a solid data foundation for subsequent accurate aggregation and threat mining. Employing a dynamic aggregation strategy based on core entities and relationships, it can construct key entities around alarm objects, aggregating and unifying a series of similar alarms. This method not only reduces the number of alarms by one to two orders of magnitude but also forms security events with complete attack contexts, enabling security analysts to understand the entire attack process more intuitively and comprehensively. A three-layer progressive noise reduction and judgment mechanism is designed: first, it quickly filters out explicit false alarms and background noise through a static rule base; then, it uses a base... The adaptive noise reduction based on historical learning adapts to the personalized scenario needs of different users, and the noise reduction range continues to expand as the historical analysis database becomes richer. Secondly, it utilizes AI large-scale models for deep semantic analysis, automatic threat intelligence association, and attack intent assessment, which can accurately identify complex and covert attacks that are difficult to detect by traditional methods. The three-layer mechanism works together to effectively eliminate low-value alerts and false alarms, accurately identify covert threats, and ensure the high credibility of the final output security events, significantly reducing the false alarm rate and false negative rate of traditional methods. Through a systematic process of standardization, aggregation, and noise reduction, it achieves automated transformation from massive amounts of raw alerts to a small number of high-value security events, eliminating the need for security operations personnel to manually screen and compare heterogeneous alerts, greatly reducing their primary analysis burden, freeing them from alert fatigue, and allowing them to focus on response and handling decisions for real threats. At the same time, the automated alert processing and analysis process improves the threat response speed of the security operations center, optimizes decision quality, and comprehensively enhances the enterprise's ability to deal with cyber threats.

[0046] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for aggregation and noise reduction of multi-source heterogeneous network security alarms, comprising the following steps: Step 1: Constructing a network security alarm ontology model; Step 2: Standardized processing of multi-source alarms; Step 3: Dynamic alarm aggregation based on entity relationships; Step 4: Multi-level alarm noise reduction and intelligent analysis; Its features include: In step one above, best practices from national standards, international standards, and mainstream industry frameworks are integrated to establish a network security alarm ontology model that includes standardization of alarm fields, alarm types, and entities and relationships in alarms. In step two above, a streaming processing framework is used as the data processing engine to receive security alarms from multiple heterogeneous sources. After source identification and adaptation parsing, the alarms are converted into a standardized alarm data stream with a unified structure based on the network security alarm ontology model. In step three above, the core entities and relationships in the standardized alarms are used as the key aggregation points. Combined with the time window that is dynamically adjusted according to the alarm type and behavioral characteristics, related alarms around the same core entity within a specific time window are aggregated into aggregated alarms. In step four above, the aggregated alarms are denoised and intelligently analyzed through a three-layer progressive processing: automatic analysis and denoising based on built-in rules, adaptive denoising based on historical learning, and deep analysis and denoising based on artificial intelligence, which are executed in sequence to finally output a high-credibility security event.

2. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step one, the standardization of alarm fields specifically involves: establishing a dynamically adjustable mapping table that covers the alarm formats of connected network security devices, mapping heterogeneous fields describing the same meaning in alarms from different data sources to unified semantic fields; the standardization of alarm types specifically involves: constructing a multi-dimensional alarm classification system based on attack targets, attack methods, vulnerability exploitation, attack consequences, and severity, and having functions for adding, deleting, and modifying alarms; the standardization of entities and relationships in alarms specifically involves: based on the standardization of alarm fields, extracting key entities and relationships between entities in alarms to form a structured knowledge graph of nodes and edges. Key entities include hosts, domain names, URLs, files, and processes, and relationships between entities include entity interaction behaviors such as connection creation, execution, and reading.

3. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: Step two specifically includes the following steps: Step 2.1, Data Access and Processing: Real-time access to alarm data streams from network security devices via Kafka, Syslog, API, and file listening interfaces; Step 2.2, Source Identification and Distribution Processing: Based on the device IP and alarm identifier of the alarm, automatically identify the device type and alarm format to which the alarm belongs; Step 2.3, Adaptation and Standardization Processing: Based on the identification results, the alarm stream is distributed to the corresponding standardization processing program; the standardization processing program has built-in parsing logic for alarm formats of specific devices and mapping rules to the network security alarm ontology model, extracts key fields from the original alarm, converts data types, identifies key entities, and builds entity relationships to form a standardized alarm object; Step 2.4, Output standardized data stream processing: Push the standardized alarm objects to the next stage message queue to form a unified structured alarm data stream.

4. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step three, taking the core entities and relationships in the standardized alarms as the key aggregation points specifically means: determining the core entities and core relationships from the standardized alarm data stream based on the type of network security device accessed and the alarm type defined by that network security device, and using the common core entities and core relationships as the aggregation basis.

5. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step three, the time window W, which is dynamically adjusted based on the alarm type and behavioral characteristics, is calculated using the following formula: Among them, W base The base window is used for severity, which is the normalized alarm severity, ranging from 0 to 1. Persistence is the historical persistence estimate of this type of alarm behavior. α and γ are empirical coefficients that can be adjusted online.

6. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step four, the automatic judgment and noise reduction based on built-in rules specifically involves: using an algorithm combining Bloom filters and hash tables to match standardized alarm data with rule items in a static rule base; marking successfully matched standardized alarm data; and entering adaptive noise reduction based on historical learning for unmarked standardized alarm data; wherein, the rule combination conditions in the static rule base include intrusion indicators, alarm types, attack tactics, attack techniques, and attack stages.

7. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step four, the adaptive noise reduction based on historical learning employs methods including entity filtering, abstract relationship filtering, and contextual key feature filtering. All three filtering methods use an algorithm combining Bloom filters and hash tables. Marked alarms can have their priorities automatically adjusted by an automated orchestration process, or a response action can be executed. Unmarked alarms enter into a deep analysis and noise reduction process based on artificial intelligence. Specifically, entity filtering involves precisely comparing entities in standardized alarms with entities in a historical analysis entity database; alarms are marked upon successful matching. Abstract relationship filtering involves extracting entities and relationships from standardized alarms and comparing them with abstract relationships in a historical abstract relationship database; alarms are marked upon successful matching. Contextual key feature filtering involves extracting contextual key features of entities and relationships from standardized alarms and comparing them with a historical contextual key feature database; alarms are marked upon successful matching. Contextual key features include network request data, network response data, and command execution data.

8. The method for aggregation and noise reduction of multi-source heterogeneous network security alarms according to claim 1, characterized in that: In step four, the AI-based deep analysis and noise reduction methods include semantic understanding and intent analysis, automatic threat intelligence association, and contextual scenario construction. Semantic understanding and intent analysis specifically involves using a large language model to perform deep semantic analysis on unstructured data and attack payload data in alerts to determine whether they represent a real attack, thereby identifying network behavior intent and threat level. Automatic threat intelligence association involves extracting key entities from aggregated alerts using an AI agent, calling interfaces to query internal and external threat intelligence in real time, and marking the alert if the key entity matches or is associated with malicious infrastructure or malicious samples. Contextual scenario construction involves using a large AI model to extract standardized alert data, comprehensively analyzing entity relationships and behavioral logic within and across multiple aggregated alerts, constructing a complete attack story chain, and identifying abnormal behavior by combining historical alert behavior and similar common behaviors.

9. A multi-source heterogeneous network security alarm aggregation and noise reduction system, comprising a data access module (1), a parsing and standardization module (2), a data queue module (3), a dynamic alarm aggregation module (4), and a multi-level alarm noise reduction and intelligent judgment module (5), characterized in that: The parsing and standardization module (2) establishes data connections with the data access module (1) and the data queue module (3) respectively, and the dynamic alarm aggregation module (4) establishes data connections with the data queue module (3) and the multi-level alarm noise reduction and intelligent judgment module (5) respectively; wherein, the data access module (1) is used to access the original alarm data of various network security devices, supports the real-time collection of heterogeneous original alarm data from different network security devices through Kafka, Syslog, API and file listening interfaces, and can perform unified management operations such as adding, modifying, deleting and searching on the data source, as the unified data entry point of the system; the parsing and standardization module (2) is used to receive the original alarm data output by the data access module (1), and convert the fields, types, entities and relation mappings of heterogeneous alarms into unified structured standardized alarm data according to the preset device alarm format parsing logic and the pre-built network security alarm ontology model; the data queue module (3) is used to process the output of the parsing and standardization module (2) The standardized alarm data is cached and asynchronously distributed to achieve decoupling of upstream and downstream modules, which can smoothly handle data floods and ensure the stability of data processing in high-concurrency scenarios. The dynamic alarm aggregation module (4) is used to obtain standardized alarm data from the data queue module (3). Based on the network security device type and the alarm type defined by the device, the core entity and core relationship in the standardized alarm data are determined. The common core entity and core relationship are used as the aggregation basis. Combined with the time window dynamically adjusted according to the alarm type and behavioral characteristics, the relevant standardized alarms around the same core entity in a specific time window are aggregated into an aggregated alarm. The multi-level alarm noise reduction and intelligent judgment module (5) is used to receive the aggregated alarm output by the dynamic alarm aggregation module (4). Noise reduction and intelligent judgment are achieved through three-layer progressive processing: first, automatic judgment and noise reduction based on the built-in static rule base are executed; then, adaptive noise reduction based on historical learning is executed; finally, deep judgment and noise reduction based on artificial intelligence is executed, and finally, a highly reliable security event is output.

10. A multi-source heterogeneous network security alarm aggregation and noise reduction device, comprising a memory body (6), a computer program (7), and a processor body (8), characterized in that: The memory body (6) stores a computer program (7) that can run on the processor body (8), and the processor body (8) implements the noise reduction method according to any one of claims 1 to 8 when executing the computer program (7).

Images