Data fusion correlation analysis method, electronic device, readable medium and program product

Through the collection and governance of multi-source heterogeneous data, combined with machine learning and timing correlation algorithms, the problem of data aggregation and identification of complex attacks in traditional network security analysis is solved, and in-depth analysis and real-time early warning of complex network attacks are achieved.

CN119066615APending Publication Date: 2024-12-03BEIJING TOPSEC NETWORK SECURITY TECH +2

Patent Information

Application Number
CN202411196633.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The aggregation and fusion problem of multi-source heterogeneous security data. Traditional network security analysis relies on single log data and is difficult to identify complex combination network attacks.

Method used

Heterogeneous data sources are obtained through multiple acquisition interfaces, data governance and standardization are carried out, multi-dimensional fusion correlation analysis model is built, and comprehensive alarm events are generated using machine learning and timing association algorithms.

Benefits of technology

It realizes unified standardization and consistency of data, improves the ability to identify complex network attacks and real-time early warning capabilities of network security threats, and improves the overall security protection level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066615A_ABST
    Figure CN119066615A_ABST
Patent Text Reader

Abstract

The invention provides a data fusion correlation analysis method. The method comprises the following steps: acquiring data from a plurality of heterogeneous data sources through a plurality of acquisition interfaces; performing data governance on the acquired data to generate a governed data set; and carrying out multi-dimensional fusion correlation analysis on the governed data set to generate a corresponding comprehensive alarm event. The disclosure also provides an electronic device, a computer readable medium, and a computer program product. Through fusion and comprehensive modeling of multi-source data, the problems that a complex attack mode is difficult to recognize, data analysis is limited by a single data source and the like are effectively solved, the detection capability of network security threats is remarkably improved, and particularly, the method is more efficient and accurate when coping with complex network combined attacks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of data fusion association analysis, and in particular to a data fusion association analysis method based on a multi-source heterogeneous scenario, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] Traditional network security analysis methods usually rely on rule matching of independent security events. However, complex network attacks often generate multiple types of log data in the network. It is difficult to identify complex and combined network attacks by analyzing these single logs alone. Therefore, it is necessary to conduct comprehensive modeling by integrating and correlating multiple data sources in order to mine potential security threats from massive data.

[0003] The multi-source heterogeneous data environment has brought about a bottleneck in data aggregation. Different security devices and manufacturers generate data in different formats and contents, which are difficult to unify and standardize in the network. The diversity and dispersion of these data have led to an urgent need for data aggregation and standardization in the network security analysis process. Summary of the invention

[0004] The embodiments of the present disclosure provide a data fusion association analysis method, an electronic device, a computer-readable medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a data fusion association analysis method, including:

[0006] Acquire data from multiple heterogeneous data sources through a variety of collection interfaces;

[0007] Performing data governance on the acquired data to generate a governed data set;

[0008] A multi-dimensional fusion correlation analysis is performed on the governed data set to generate corresponding comprehensive alarm events.

[0009] In some embodiments, data is acquired from multiple heterogeneous data sources through multiple acquisition interfaces, including:

[0010] receiving the data according to a communication protocol of the heterogeneous data source;

[0011] Parsing the data using a predefined parsing template to generate standardized data;

[0012] The standardized data is classified and stored.

[0013] In some embodiments, receiving the data according to the communication protocol of the heterogeneous data source includes:

[0014] Configure offline and real-time data receiving models;

[0015] The data is received in real time or offline according to the communication protocol of the heterogeneous data source.

[0016] In some embodiments, classifying and storing the standardized data includes:

[0017] Classifying the standardized data according to a multi-level dimension including a security event type and a security event level;

[0018] The classified data is stored in the data center.

[0019] In some embodiments, performing data governance on the acquired data to generate a governed data set includes:

[0020] The acquired data is processed in sequence according to cleaning and filtering, data conversion, data extraction, and data enrichment specifications to generate cleaned and enriched data, wherein when data that does not conform to the specifications is identified during the cleaning and filtering process, data quality warning data is generated;

[0021] The cleaned and enriched data are processed in parallel according to data summarization, data aggregation, data merging, and data deduplication specifications to generate summary and aggregated data;

[0022] The cleaned and enriched data is processed in accordance with data black identification, data white identification, and data integration specifications to generate integrated service data;

[0023] Processing the original dictionary data according to the audit verification specification and integrating it with the summary and aggregated data to generate integrated dictionary data;

[0024] The governed data set consisting of the integrated service data, the integrated dictionary data and the data quality warning data is generated.

[0025] In some embodiments, performing a multi-dimensional fusion correlation analysis on the managed data set to generate a corresponding comprehensive alarm event includes:

[0026] Build analysis module library;

[0027] Building an executable model instance in combination with the analysis modules in the analysis module library;

[0028] The executable model instance is executed to process the governed data set.

[0029] In some embodiments, executing the executable model instance to process the governed data set includes:

[0030] Using the governed data set to train a threshold deviation behavior detection algorithm model;

[0031] Based on the trained threshold deviation behavior detection algorithm model, one or more threshold reference values ​​for judging abnormal behavior are set.

[0032] In some embodiments, executing the executable model instance to process the governed data set further comprises:

[0033] Performing time series correlation analysis on the threshold reference value and the data in the treated data set by using a time series correlation algorithm;

[0034] When a network security threat event is identified from the data in the governed data set based on the time series correlation analysis, the corresponding comprehensive alarm event is generated.

[0035] In a second aspect, an embodiment of the present disclosure provides an electronic device, including:

[0036] one or more processors;

[0037] A memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion association analysis method described in the first aspect of the embodiment of the present disclosure.

[0038] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the data fusion association analysis method described in the first aspect of the embodiment of the present disclosure is implemented.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the data fusion association analysis method described in the first aspect of the embodiment of the present disclosure.

[0040] The disclosed embodiment overcomes the data aggregation bottleneck caused by diverse, dispersed, and heterogeneous data sources by establishing a data collection governance mechanism, and achieves standardization, consistency, and ordering of data sets; and constructs a comprehensive modeling and analysis method that integrates multiple data for deep correlation analysis, no longer relying solely on a single log data set. This method can deeply explore the concealment and complex combination of network attacks between network security events, thereby solving the pain points in traditional network security analysis and improving the accuracy and efficiency of network security detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a data fusion association analysis method according to an embodiment of the present disclosure.

[0042] Figure 2It is an exemplary block diagram of a multi-source heterogeneous data aggregation device according to an embodiment of the present disclosure.

[0043] Figure 3 4 is a flow chart of data import according to an embodiment of the present disclosure.

[0044] Figure 4 It is a flowchart of data governance according to an embodiment of the present disclosure.

[0045] Figure 5 It is a multi-dimensional modeling principle diagram based on massive data analysis of an embodiment of the present disclosure.

[0046] Figure 6 It is a schematic diagram of the timing correlation algorithm of the embodiment of the present disclosure.

[0047] Figure 7 It is a schematic diagram of the composition of an electronic device in an embodiment of the present disclosure.

[0048] Figure 8 It is a block diagram of a data fusion association analysis device according to an embodiment of the present disclosure.

[0049] Fig. 9 It is a schematic diagram of data fusion association analysis according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the technical solution of the present disclosure is described in detail below with reference to the accompanying drawings.

[0051] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0052] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0053] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0054] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0055] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0056] In a multi-source heterogeneous scenario, a method, device, equipment and medium for data fusion and correlation analysis are proposed. This method uses advanced algorithms, including machine learning, threshold deviation behavior detection and time series correlation analysis, and can deeply fuse and analyze various data from the network to explore the potential value of the data, thereby realizing real-time discovery and warning of hidden network security threats.

[0057] The present disclosure aims to solve the following technical problems:

[0058] 1. Aggregation and connection of multi-source heterogeneous security data

[0059] In the current network environment, the diversity and heterogeneity of data sources pose significant challenges. Data generated by different security devices and manufacturers are in different formats and are scattered in different systems, making it difficult to achieve unified data collection. This diverse and scattered data environment has become a bottleneck for data aggregation. In addition, the massive amount of data collected also has differences in fields and lacks standardization, which makes subsequent data processing and analysis difficult.

[0060] 2. Data fusion and correlation analysis

[0061] Traditional network security analysis usually relies on rule matching of a single security event. However, network attacks often generate multiple types of log data, and a single log alone cannot effectively identify complex combined attacks. Therefore, it is necessary to fuse and correlate multiple data, and use comprehensive modeling to mine the potential value of massive data to identify hidden lateral threat attacks in the network. This process is crucial to improving network security protection capabilities.

[0062] A data fusion correlation analysis method, device, equipment and medium proposed in a multi-source heterogeneous scenario have the following advantages over traditional network security analysis methods:

[0063] First, this method can effectively manage diverse and scattered heterogeneous data sources to form a standardized, consistent and ordered correlation analysis data set. This process overcomes the difficulties of traditional methods in processing heterogeneous data, ensures the uniformity and standardization of data, and provides a reliable data foundation for subsequent analysis.

[0064] Secondly, unlike traditional network security analysis methods that usually rely on a single log data set, this method integrates and correlates multiple data types to perform comprehensive modeling analysis. This fusion of multi-source data can tap the potential value of massive data, allowing complex network combination attacks to be identified earlier and more accurately.

[0065] By integrating and comprehensively modeling multi-source data, this method can effectively solve the pain points of traditional network security analysis, such as the difficulty in identifying complex attack patterns and the limitation of data analysis to a single data source. It can thus significantly improve the ability to detect network security threats, especially when dealing with complex network combination attacks. It is more efficient and accurate.

[0066] This method can break through the limitations of traditional network security analysis and achieve effective integration and correlation analysis of multiple security data. This not only improves the ability to identify complex network attacks, but also provides real-time warnings for potential lateral threats, effectively improving the overall security protection level of the network.

[0067] This method effectively solves the problem of aggregating multi-source heterogeneous data, and through in-depth analysis, improves the ability to identify and respond to complex network security threats.

[0068] A data fusion association analysis method, device, equipment and medium based on a multi-source heterogeneous scenario includes the following steps: converge multi-source heterogeneous data through multiple acquisition interfaces. These data may come from various devices and systems and have different formats and structures. Data governance is performed on the converged multi-source heterogeneous data to form a standard, consistent and orderly association analysis data set. Advanced algorithms such as machine learning, threshold deviation behavior detection and time series association analysis are used to fuse and analyze the governed data. Through comprehensive modeling, potential value is mined from massive data. Real-time discovery of hidden network security threats is achieved, and alarms are issued in a timely manner, thereby improving the overall protection capability of network security.

[0069] A data fusion correlation analysis method, device, equipment and medium based on multi-source heterogeneous scenarios is a method that uses machine learning, threshold deviation behavior detection, time series correlation and other advanced algorithms to first converge multi-source heterogeneous data through multiple acquisition interfaces, and form a standard, consistent and orderly correlation analysis data set through data governance, and then conduct comprehensive modeling analysis by fusing and correlating multiple data, mining the potential value of massive data, and realizing real-time discovery and alarm of hidden network security threats. Its core includes multi-source heterogeneous data convergence and data fusion correlation analysis.

[0070] Figure 1 It is a flow chart of a data fusion association analysis method according to an embodiment of the present disclosure.

[0071] First, refer to Figure 1 The present disclosure provides a data fusion association analysis method, including:

[0072] S11, obtain data from multiple heterogeneous data sources through multiple acquisition interfaces;

[0073] S12, performing data management on the acquired data to generate a managed data set;

[0074] S13. Perform multi-dimensional fusion correlation analysis on the managed data set to generate corresponding comprehensive alarm events.

[0075] In the embodiments of the present disclosure, multiple collection interfaces are provided through data collection to achieve unified aggregation of multi-source heterogeneous data, and unified data preprocessing such as data conversion, data filtering, and data supplementation is achieved through data governance. Preliminary data fusion is performed during the data connection process, ultimately achieving integrated connection and aggregation of multi-source heterogeneous data.

[0076] In the disclosed embodiments, the multi-source heterogeneous data connection and convergence may include data convergence and data governance. Through data collection, a variety of collection interfaces are provided to achieve unified convergence of multi-source heterogeneous data. Through data governance, unified data preprocessing such as data conversion, data filtering, and data supplementation is achieved. In the process of data connection, preliminary data fusion is performed, and finally, the integrated connection and convergence of multi-source heterogeneous data is achieved.

[0077] Facing the entire cyberspace, by aggregating, integrating, and preliminarily processing distributed, heterogeneous, and dynamic network security-related data, the data consistency of the data source as a whole is maintained and the efficiency of information sharing and utilization is improved. The present disclosure organizes data in a transparent way to users, and provides a subscription interface that allows businesses to subscribe to data on demand, providing data to be processed for security management, switches, security protection, security monitoring and other related systems, thereby forming an overall cyberspace security situation information. Data aggregation is mainly divided into data source and data import.

[0078] Figure 2 It is an exemplary block diagram of a multi-source heterogeneous data aggregation device according to an embodiment of the present disclosure.

[0079] In the disclosed embodiment, Figure 2 The composition and functional modules of multi-source heterogeneous data aggregation equipment are demonstrated. Figure 2 As shown, the multi-source heterogeneous data aggregation device includes: a data source 201, a log source manager 204, a log source descriptor 205, a parsing template editor 206, a JSON template 207, a data receiver 202, a receiving monitoring server 208 and a data manager 203. However, those skilled in the art should understand that Figure 2The multi-source heterogeneous data aggregation device shown is an example, and it can be modified or expanded according to user needs in actual applications.

[0080] exist Figure 2 In the example, data source 201 may include security management, switches, routers, security protection, security monitoring and other equipment, which generate and collect various logs, events and monitoring data in the network. After the data is extracted, it is input into data receiver 202 for processing.

[0081] In some embodiments, the heterogeneous data sources include: security management devices, switches, routers, security protection devices, and security monitoring devices that generate and collect various logs, events, and monitoring data in the network.

[0082] In the embodiment of the present disclosure, multi-source data used in the present disclosure can be obtained from the data source 201, and multi-source formatted data access is supported. Data in different formats from the data source 201 can include harmful program events, network attack events, information destruction events, APT events, flow monitoring data, domain name data, URL access logs, sample restoration logs, and other event data.

[0083] In some embodiments, the data obtained from the multiple heterogeneous data sources include: network security related data including harmful program events, network attack events, information destruction events, APT events, flow monitoring data, domain name data, URL access logs and sample restoration logs.

[0084] exist Figure 2 In the example, the log source manager 204 is used to configure and manage different log sources and define the sources that need to monitor and collect log data. The log source descriptor 205 generates the data transmission protocol of the log source according to the log source received from the log source manager 204. The multi-protocol reception in the data receiver 202 is ensured to receive data from different data sources according to the data transmission protocol from the log source descriptor 205.

[0085] Parsing template editor 206 is used to create or edit parsing templates (e.g., JSON templates) to meet the needs of different log sources and data formats. JSON template 207 describes the structure and parsing rules of the data. JSON template 207 contains detailed information on how to parse a data source in a specific format, such as field names, data types, hierarchical structures, etc.

[0086] The data receiver 202 includes three parts: multi-protocol receiving, templated parsing, and multiple class loading. Multi-protocol receiving receives data from different data sources. Since the data sources may use different communication protocols (such as HTTP, FTP, Syslog, etc.), multi-protocol receiving supports multiple communication protocols according to the detailed information from the log source descriptor 205 to ensure smooth data reception from different data sources; templated parsing uses predefined templates, such as JSON templates, to parse data in different formats; multiple class loading classifies and loads the parsed data into the corresponding parts of the data manager 203 for subsequent use, including data resource management, database management, and file system loading.

[0087] The receiving monitoring server 208 monitors the data collected by the data receiver 202 in real time, and can find out whether the reported data is collected, and can monitor the original data in real time through monitoring.

[0088] The data manager 203 manages the data collected by the data receiver 202 and is divided into the following sub-modules: data resource management, which classifies, stores and manages the data; database management, which stores structured data in the database for quick retrieval and analysis; and file system, which processes and stores unstructured data (such as log files, configuration files, etc.) and provides data backup and recovery functions.

[0089] In some embodiments, data is acquired from multiple heterogeneous data sources through multiple acquisition interfaces, including:

[0090] receiving the data according to a communication protocol of the heterogeneous data source;

[0091] Parsing the data using a predefined parsing template to generate standardized data;

[0092] The standardized data is classified and stored.

[0093] Figure 3 4 is a flow chart of data import according to an embodiment of the present disclosure.

[0094] In the embodiments of the present disclosure, refer to Figure 3 The present disclosure can import multi-source formatted data, support multi-level and multi-dimensional data classification, support multi-level and multi-dimensional classification by security event type and security event level, and collect data mainly including harmful program events, network attack events, information destruction events and APT events, etc. The data aggregation process supports on-demand import of data sources, and supports two data acquisition methods: offline and online real-time: real-time data push, which can provide a visual interface to configure the data source to be pushed in real time, and receive multi-source data corresponding to the configuration in real time; and offline data subscription, users can easily and flexibly select specific types, specific times and specific fields.

[0095] In some embodiments, the acquisition interface supports acquisition of offline or real-time data.

[0096] exist Figure 3 In the embodiment of the present disclosure, data import includes:

[0097] Step 1: Configure offline and real-time data receiving models.

[0098] A receiving model for processing offline and real-time data is configured in the data receiving module. The data receiving model can be configured with a data source, a specific type, a specific time, and a specific field according to specific needs.

[0099] Step 2: According to actual needs, real-time or offline data is obtained. The data includes message security events (for example, harmful program events, network attack events, APT events, information destruction events), flows, domain names, URLs, sample restoration data, etc.

[0100] Step 3: Return the required data.

[0101] Step 4: Classify by multiple dimensions such as security incident type and security incident level.

[0102] The data receiving module classifies the acquired data according to multiple dimensions such as security event type (such as harmful program events, network attack events, APT events, information destruction events, etc.) and security event level (such as low, medium, and high).

[0103] Step 5: The data is saved in the data center.

[0104] The classified data is stored in the data center.

[0105] Step 6: Execute data storage.

[0106] Data storage is performed to store the data in the data center.

[0107] Figure 3 Describes the complete process of data import to ensure that data can be effectively received, classified and stored.

[0108] In some embodiments, receiving the data according to the communication protocol of the heterogeneous data source includes:

[0109] Configure offline and real-time data receiving models;

[0110] The data is received in real time or offline according to the communication protocol of the heterogeneous data source.

[0111] In some embodiments, classifying and storing the standardized data includes:

[0112] Classifying the standardized data according to a multi-level dimension including a security event type and a security event level;

[0113] The classified data is stored in the data center.

[0114] Figure 4 It is a flowchart of data governance according to an embodiment of the present disclosure.

[0115] In the embodiments of the present disclosure, refer to Figure 4 , provides a flowchart of data governance. Those skilled in the art should understand that Figure 4 The specific data governance is exemplary and can be added, deleted and modified in various ways according to different needs of users.

[0116] exist Figure 4 The data governance process shown in the figure involves the following multiple levels: various types of raw data, cleansing and enrichment layer, summary and aggregation layer, integration service layer and dictionary data layer.

[0117] Various types of raw data include dictionary data, flow data, domain name data, message data, sample data, URL data and other logs.

[0118] In the cleaning and enrichment layer, some of the various types of original data are processed in turn according to the cleaning and filtering, data conversion, data extraction, and data enrichment specifications to generate cleaned and enriched data.

[0119] In the summary and aggregation layer, the cleaned and enriched data are processed in parallel according to the data summary, data aggregation, data merging, and data deduplication specifications to generate summary and aggregated data.

[0120] At the integration service layer, the cleaned and enriched data is processed in accordance with data blacklisting, data whitelisting, and data integration specifications to generate integrated service data.

[0121] At the dictionary data layer, the dictionary data is processed according to the audit and verification specifications and integrated with the summary and aggregated data to generate the final dictionary data.

[0122] Reference Figure 4 The data governance process of the present disclosure includes the following steps:

[0123] Develop data collection channels. Specifically, in order to realize data import, two data collection channels are developed: offline data and real-time data collection import channels. Offline data collection uses FTP, HDFS, Hive, etc. as the intermediate medium for data transmission; real-time data collection can use Kafka as the data transmission medium;

[0124] Develop data processing specifications, such as Figure 4As shown in the figure, a multi-level and multi-dimensional data processing specification has been formulated, including but not limited to data cleaning, filtering, extraction, conversion, enrichment, deduplication, merging, summarizing, aggregation, black and white identification, integration, association and other operations to ensure that the data can meet the needs of subsequent processing and analysis;

[0125] Perform cleaning and enrichment treatment, such as Figure 4 As shown, after various types of original data are imported through the data acquisition channel, they are processed in the cleaning and enrichment layer in accordance with the cleaning and filtration, data conversion, data extraction, and data enrichment specifications to generate the corresponding cleaning and enrichment layer data table;

[0126] Perform summary aggregation processing, such as Figure 4 As shown, the data of the cleaned enriched layer is processed in parallel according to the specifications of data summarization, data aggregation, data merging, and data deduplication to generate summary aggregation layer data;

[0127] Perform integration service processing, such as Figure 4 As shown, the data of the cleansed enriched layer is processed according to the specifications of data black judgment, data white consideration, and data integration to generate the corresponding integrated service layer data table;

[0128] Perform dictionary data processing, such as Figure 4 As shown, the dictionary data is processed according to the audit verification specifications and integrated with the data of the summary aggregation layer to finally generate the dictionary data;

[0129] Data quality control and alarm generation: during the data import process, non-compliant data is identified and retained according to filtering and cleaning specifications, and data quality alarm data is generated. The alarm data is aggregated and output at the integration service layer; and

[0130] Data storage and management: store the associated analysis data set consisting of integrated service data, dictionary data and alarm data in the data center (data middle platform), manage it according to specifications, and provide data middle platform services to the outside world through the data middle platform.

[0131] In some embodiments, performing data governance on the acquired data to generate a governed data set includes:

[0132] The acquired data is processed in sequence according to cleaning and filtering, data conversion, data extraction, and data enrichment specifications to generate cleaned and enriched data, wherein when data that does not conform to the specifications is identified during the cleaning and filtering process, data quality warning data is generated;

[0133] The cleaned and enriched data are processed in parallel according to data summarization, data aggregation, data merging, and data deduplication specifications to generate summary and aggregated data;

[0134] The cleaned and enriched data is processed in accordance with data black identification, data white identification, and data integration specifications to generate integrated service data;

[0135] Processing the original dictionary data according to the audit verification specification and integrating it with the summary and aggregated data to generate integrated dictionary data;

[0136] The governed data set consisting of the integrated service data, the integrated dictionary data and the data quality warning data is generated.

[0137] Reference Figure 4 The network security information data imported by the present disclosure can better meet the needs of the multi-source data aggregation system through data enrichment and cleaning. The imported data includes: harmful program events, network attack events, information destruction events, APT events, flow monitoring data, domain name data, URL access logs, sample restoration data, other event data, business data, knowledge intelligence, network security engine data, network security monitoring platform data, etc. First, the accessed data is enriched. These operations include: domain name extraction, IP type conversion, geographic positioning, security identification addition, unit filing, rule name determination, protocol type and event type identification, etc. At the same time, it supports the cleaning of accessed data, and can clean data according to conditions through flexible user configuration. For example, it can filter according to the configured IP or domain name, and clean invalid data according to the message rules. Data deduplication is achieved by configuring deduplication data tables and fields to achieve deduplication of the same network security information records. In addition, data quality audits are performed to evaluate the quality of data from multiple dimensions such as fill rate, correctness, and logical consistency of values ​​between fields. The entire preprocessing process provides a visual interface to display the data preprocessing report, ultimately achieving the preprocessing of the imported network security information data and outputting the preprocessed network security data and preprocessing report.

[0138] In some embodiments, the data enrichment includes domain name extraction, IP type conversion, geolocation, security identification addition, unit filing, rule name determination, and identification of protocol type and event type.

[0139] In some embodiments, the data deduplication is performed by configuring a deduplication data table and deduplication fields.

[0140] Based on the large amount of data generated by network, security, and cryptographic devices, a fusion correlation analysis model is constructed. The source data includes various types of security events and log data, such as harmful program events, network attack events, information destruction events, APT (advanced persistent threat) events, traffic monitoring data, domain name data, URL access logs, sample restoration logs, and other related event data. Relying on machine learning, threshold deviation behavior detection, time series association and other operator algorithms and multi-dimensional modeling technology for massive data fusion analysis, a fusion correlation analysis model based on big data is constructed to realize massive data processing, and multi-dimensional and in-depth fusion and correlation analysis of network data, security data and cryptographic data is carried out to explore the hidden relationship between independent security events, thereby discovering complex network combination attack events.

[0141] In an embodiment of the present disclosure, the multi-dimensional fusion association analysis is performed using multi-dimensional modeling based on massive data analysis.

[0142] Figure 5 It is a schematic diagram of a multi-dimensional modeling principle based on massive data analysis according to an embodiment of the present disclosure.

[0143] In the embodiments of the present disclosure, multidimensional modeling based on massive data analysis provides an architecture and tool set for processing, analyzing and correlating data of different dimensions to identify potential security threats and abnormal behaviors. Through the collaborative work of various modules in the architecture (such as analysis module library, analysis process execution engine, extended analysis module, etc.), complex data analysis tasks can be performed, and finally deep fusion correlation analysis can be achieved.

[0144] exist Figure 5 In the paper, the multidimensional modeling framework includes: extended analysis module, analysis module engine, analysis module library, big data analysis business model, process transformation engine, executable model instance, analysis process execution engine, big data cluster and data analysis tools.

[0145] exist Figure 5 In the process, the extended analysis module builds different analysis modules through the analysis module engine and stores these analysis modules in the analysis module library; then, the big data analysis business model combines the analysis modules stored in the analysis module library and builds executable model instances through the process conversion engine; these executable model instances are then processed and executed by the analysis process execution engine to perform analysis operations on the associated model; the final output result is the big data cluster after the associated analysis and the constructed data analysis tool.

[0146] In actual applications, when a specific analysis is required (such as threshold deviation behavior detection or timing correlation analysis), the system calls the corresponding analysis module from the analysis module library. These modules are integrated into the executable model instance through the process conversion engine, and are finally executed by the analysis process execution engine to output the analysis results. In addition to the threshold deviation behavior detection algorithm and the timing correlation algorithm, the system can also call other analysis modules according to specific needs to achieve more complex and diverse analysis tasks.

[0147] In an embodiment of the present disclosure, the analysis modules in the analysis module library include but are not limited to a threshold deviation behavior detection algorithm and a time series correlation algorithm.

[0148] In some embodiments, performing a multi-dimensional fusion correlation analysis on the managed data set to generate a corresponding comprehensive alarm event includes:

[0149] Build analysis module library;

[0150] Building an executable model instance in combination with the analysis modules in the analysis module library;

[0151] The executable model instance is executed to process the governed data set.

[0152] In some embodiments, the analysis modules in the analysis module library include: a threshold deviation behavior detection algorithm and a timing correlation algorithm.

[0153] The threshold deviation behavior detection algorithm is an algorithm framework widely used in security incident detection. The core is to set one or more thresholds and identify potential abnormal behaviors by detecting whether the data deviates from these thresholds. The algorithm framework is highly flexible and can use different statistical and machine learning methods to set thresholds according to specific needs to determine whether the behavior deviates from the normal range. This algorithm is suitable for extracting and detecting potential security incidents and abnormal behaviors from association analysis data sets that have undergone data governance.

[0154] The input data of the threshold deviation behavior detection algorithm is the correlation analysis data set after data governance. These data may include source data from various sources such as network, security, and cryptographic devices, such as messages, traffic data, domain name data, URL access logs, sample restoration logs, and other event data. Before entering the algorithm, all data has gone through data cleaning, enrichment, aggregation, and other governance processes, and is stored in the data center as the input of the threshold deviation behavior detection algorithm.

[0155] The output of the threshold deviation behavior detection algorithm is an accurate threshold reference value based on abnormal behavior security events. These thresholds are used for further security analysis and exception handling.

[0156] The implementation of the threshold deviation behavior detection algorithm can include multiple ways. The algorithm calculates one or more threshold reference values ​​based on the statistical characteristics of historical data and current data (such as mean, standard deviation, etc.) through modeling (such as using 3sigma model, linear regression, etc.). These thresholds define under what circumstances, the data or behavior in the system can be considered abnormal.

[0157] Threshold deviation behavior detection algorithm based on linear regression: After inputting the association analysis data set, a linear regression model is used to fit the data and calculate the residual between the predicted value and the actual value. The threshold is set by the standard deviation of the residual to identify abnormal behavior.

[0158] The threshold deviation algorithm using the linear regression equation consists of the following steps:

[0159] 1) Fitting the linear regression model:

[0160] The linear regression model is used to fit the input association analysis data set. Specifically, the algorithm uses the data within the specified learning range to calculate the linear regression equation: is the predicted dependent variable (or response variable), x is the independent variable (or explanatory variable), b is the slope (indicating the rate of change of the dependent variable when the independent variable changes), and a is the intercept (the value of the predicted dependent variable when the independent variable is zero).

[0161] 2) Calculate the residual: For each data point, calculate its residual, that is, the difference between the actual observed value and the predicted value of the linear regression model. The residual calculation formula is: Among them, e is the residual, y is the actual observation value, is the predicted value predicted by the linear regression model.

[0162] 3) Determine the threshold: According to business needs and data characteristics, set a threshold for judging anomalies. Usually, the standard deviation of the residual can be used as the initial basis for selecting this threshold. The specific threshold may need to be adjusted according to actual conditions.

[0163] 4) Detect outliers: The residual of each data point is checked to determine whether it exceeds the set threshold. If the residual exceeds the threshold, the data point is marked as an outlier, which may represent a potential security incident or abnormal behavior.

[0164] 5) Outlier processing: The identified outliers can be further analyzed. For example, the reasons for the occurrence of these outliers can be investigated in depth, or corrections can be made in the model to improve the accuracy of the prediction and the robustness of the model.

[0165] In some embodiments, the threshold deviation behavior detection algorithm includes: a threshold deviation behavior detection algorithm based on a linear regression model.

[0166] Threshold deviation behavior detection algorithm based on 3sigma model: The threshold is set according to the mean and standard deviation of the data. Usually "mean ± 3*standard deviation" is used as the benchmark for judging data anomalies. Points exceeding this threshold are considered abnormal.

[0167] In an embodiment of the present disclosure, the threshold deviation behavior detection algorithm includes: a threshold deviation behavior detection algorithm based on a 3sigma model.

[0168] In the embodiments of the present disclosure, the actual observed data is compared with the set threshold to identify possible abnormal behavior. For linear regression, the point where the residual exceeds the threshold is marked as abnormal. For the 3sigma model, data outside the set range is considered abnormal. The identified outliers can be further analyzed to deeply explore the reasons behind them, and the threshold or model can be adjusted according to the results to improve the accuracy of detection.

[0169] The threshold deviation behavior detection algorithm will generate accurate threshold reference values ​​based on abnormal behavior security events. These reference values ​​can be used for further security analysis and threat detection to help identify abnormal behaviors such as large-volume attacks, port anomalies, inflow and outflow anomalies, Web security threats, and traffic mutations.

[0170] 1) Large traffic attack behavior

[0171] Objective: To identify large-volume attack events in the network.

[0172] Threshold deviation behavior detection includes:

[0173] Collect network traffic data within a certain time range;

[0174] Calculate the mean (μ) and standard deviation (σ) of flow data;

[0175] The 3sigma model is used to take (μ+3σ) as the flow baseline, which is defined as the upper limit of normal flow;

[0176] Filter out traffic that exceeds the baseline, considering that the traffic may be a large-volume attack.

[0177] Output the corresponding time, traffic size, IP address and other attack target information of abnormal traffic.

[0178] 2) Analysis of abnormal port behavior

[0179] Objective: Detect abnormal behavior on important ports, such as brute force attacks.

[0180] Threshold deviation behavior detection includes:

[0181] Filter out traffic data related to security units and important ports (such as 445, 3389, 139, etc.) through keyword matching;

[0182] Aggregate and count traffic data based on the IP address of the security unit and important ports.

[0183] Define a threshold (e.g. 1000 times / second), and traffic exceeding this threshold is considered abnormal;

[0184] Filter out traffic that exceeds the threshold, considering that the traffic may be brute force or other abnormal port behavior;

[0185] Outputs information about abnormal port traffic to help identify and prevent attacks.

[0186] 3) Inflow and outflow monitoring

[0187] Objective: To monitor the inflow and outflow traffic of the security unit network, and also support the monitoring of the traffic of ports, protocols, and peer IP addresses.

[0188] Threshold deviation behavior detection includes:

[0189] Filter out traffic data related to the security unit through keyword matching, including protocol type, IP address, port, etc.

[0190] Based on the historical data of one week, the mean (μ) and standard deviation (σ) of inflow and outflow are calculated to establish a flow baseline model (3sigma model);

[0191] The 3sigma model is used to set the flow threshold (μ+3σ), and the flow exceeding the baseline is considered abnormal;

[0192] Filter out traffic that is larger than the baseline, considering that this traffic may represent abnormal behavior;

[0193] Output the abnormal analysis results of inflow and outflow monitoring for further analysis and processing.

[0194] 4) Web Security Analysis

[0195] Objective: Analyze the security of Web services and detect attacks such as SQL injection and XSS.

[0196] Threshold deviation behavior detection includes:

[0197] Filter out traffic data related to the Web application of the security unit through keyword matching;

[0198] Use Web attack detection models to analyze the collected data and identify potential SQL injection, XSS attacks, etc.

[0199] Set thresholds based on normal patterns of Web traffic and treat traffic exceeding the thresholds as potential attacks;

[0200] Filter out traffic that does not match the set threshold and mark it as abnormal behavior;

[0201] Output Web security analysis results to help identify and prevent network attacks.

[0202] 5) Traffic mutation analysis

[0203] Objective: To identify dramatic changes in network traffic within a short period of time, such as sudden increases or decreases.

[0204] Threshold deviation behavior detection includes:

[0205] Collect traffic data in 5-minute intervals and establish a time sliding window;

[0206] Count the traffic difference between two adjacent time points in the sliding window;

[0207] The 3sigma model is used to set the flow difference threshold (μ+3σ), and flow changes exceeding this threshold are considered abnormal;

[0208] Filter out traffic mutations that exceed a threshold, considering that these mutations may represent abnormal behavior;

[0209] Output traffic mutation analysis results for further security analysis.

[0210] The threshold deviation behavior detection algorithm can effectively identify abnormal behaviors in traffic, ports, Web services, etc. by establishing a baseline (i.e., mean) and setting a threshold (mean + 3*standard deviation). This method is particularly effective when processing large-scale and variable network data and can provide reliable security monitoring results.

[0211] In some embodiments, executing the executable model instance to process the governed data set includes:

[0212] Using the governed data set to train a threshold deviation behavior detection algorithm model;

[0213] Based on the trained threshold deviation behavior detection algorithm model, one or more threshold reference values ​​for judging abnormal behavior are set.

[0214] Figure 6 It is a schematic diagram of the timing correlation algorithm of the embodiment of the present disclosure.

[0215] Time series association algorithms are mainly used to analyze and mine the correlation and association between time series data. These algorithms can reveal the mutual influence and time series association rules between time series data, which is helpful for predicting future trends or conducting causal analysis. This algorithm processes the output data and security event data of threshold deviation anomaly detection by calculating data association on the vertical and horizontal time axes. These data include: accurate threshold reference values ​​and security events based on abnormal behavior security events (i.e., data sets after data governance). The final output result is a comprehensive alarm event obtained based on the analysis of the time series association algorithm.

[0216] In the disclosed embodiment, the time series correlation algorithm performs time series correlation analysis on the precise threshold reference value based on abnormal behavior and the data set after data governance, performs alarm processing through fine-grained data noise reduction and rule hitting on the horizontal time axis, and adjusts the threshold on the vertical time axis to finally generate a comprehensive alarm event.

[0217] Reference Figure 6 ,On the horizontal time axis, the algorithm performs fine-grained noise reduction on security and network security events based on time association rules.,The smaller the granularity of the noise reduction cycle, the higher the processing accuracy, but it will also increase the computational load of the model. Figure 6 Each "security, network data" oval in the figure represents a data noise reduction cycle. Within two data noise reduction cycles, there may be temporal overlap between the data, which is called the data overlap time baseline. Figure 6 The data overlap data baseline refers to the overlapping portion between the two "security, network data" ovals (that is, the portion indicated by the horizontal line at the tail of any of the preceding and following hit alarm arrows).

[0218] like Figure 6 As shown in , an alarm is triggered if the overlapping part of the data hits either forward or backward, that is, if the data of the overlapping part is checked forward and meets the hit rule, an alarm is triggered; if the data of the overlapping part is checked backward and meets the hit rule, an alarm is triggered; the data on the right side of the data overlapping time baseline will trigger a rule hit alarm backward; the data on the left side of the data overlapping time baseline will trigger a rule hit alarm forward.

[0219] In the embodiment of the present disclosure, within a learning cycle and a threshold validity cycle, triggered alarm events are merged to reduce duplicate alarms and improve the effectiveness of alarms.

[0220] On the vertical time axis, the algorithm calculates the mean of the threshold in the historical data as the reference parameter for the next threshold adjustment, and obtains the boundary value with a smaller deviation, thereby improving the accuracy of the judgment of abnormal deviation of the threshold. As time goes by, new data is continuously collected and the mean and deviation are recalculated to adjust the threshold to keep it consistent with the changes in the system status.

[0221] During the entire process of time series association analysis, time series data is first subjected to denoising processing based on association rules on the horizontal time axis, and alarm detection is performed by means of data overlap. Then, on the vertical time axis, new thresholds are adjusted using historical threshold data to improve the accuracy of judgment. Finally, the algorithm outputs comprehensive alarm events that have undergone time series association analysis.

[0222] In some embodiments, executing the executable model instance to process the governed data set further comprises:

[0223] Performing time series correlation analysis on the threshold reference value and the data in the treated data set by using a time series correlation algorithm;

[0224] When a network security threat event is identified from the data in the governed data set based on the time series correlation analysis, the corresponding comprehensive alarm event is generated.

[0225] In some embodiments, the temporal correlation analysis includes:

[0226] On the vertical time axis, adjusting the threshold reference value according to historical data related to the threshold reference value determination;

[0227] On the horizontal time axis, noise reduction is performed on the data in the governed data set based on time association rules, and alarm detection is performed in combination with the data overlapping time baseline.

[0228] In some embodiments, alarm detection is performed in conjunction with the data overlapping time baseline, including:

[0229] The following operations are performed during two data denoising cycles:

[0230] Perform bidirectional rule matching on data within the data overlapping time baseline to generate comprehensive alarm events;

[0231] Performing backward rule matching on the data located on the right side of the data overlapping time baseline to generate a comprehensive alarm event;

[0232] Forward rule matching is performed on the data located on the left side of the data overlapping time baseline to generate a comprehensive alarm event.

[0233] Figure 7 It is a schematic diagram of the composition of an electronic device in an embodiment of the present disclosure.

[0234] Second, refer to Figure 7 , an embodiment of the present disclosure provides an electronic device, comprising:

[0235] One or more processors 701;

[0236] A memory 702, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement the fusion association analysis method for multi-source heterogeneous data according to the first aspect of the embodiment of the present disclosure;

[0237] One or more I / O interfaces 703 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0238] Among them, the processor 701 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 702 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 703 is connected between the processor 701 and the memory 702, and can realize information interaction between the processor 701 and the memory 702, including but not limited to a data bus (Bus), etc.

[0239] In some embodiments, the processor 701 , the memory 702 , and the I / O interface 703 are connected to each other via a bus 704 , and further connected to other components of the computing device.

[0240] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for fusion association analysis of multi-source heterogeneous data described in the first aspect of the embodiment of the present disclosure.

[0241] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method for fusion association analysis of multi-source heterogeneous data described in the first aspect of the embodiment of the present disclosure.

[0242] Figure 8 It is a block diagram of a data fusion association analysis device according to an embodiment of the present disclosure.

[0243] Fifth, refer to Figure 8 The present disclosure provides a data fusion association analysis device, including:

[0244] The data acquisition module 801 is used to acquire data from multiple heterogeneous data sources through multiple acquisition interfaces;

[0245] A data management module 802 is used to perform data management on the acquired data to generate a managed data set;

[0246] The fusion correlation analysis module 803 is used to perform multi-dimensional fusion correlation analysis on the managed data set to generate corresponding comprehensive alarm events.

[0247] In some embodiments, the data acquisition module 801 includes:

[0248] A multi-protocol receiving module, used for receiving the data according to the communication protocol of the heterogeneous data source;

[0249] A templated parsing module library, used to parse the data using predefined parsing templates to generate standardized data;

[0250] Multiple class loading modules are used to classify and store the standardized data.

[0251] In some embodiments, the data governance module 802 includes:

[0252] A cleaning and enrichment unit, used to process the acquired data in accordance with cleaning and filtering, data conversion, data extraction, and data enrichment specifications in sequence to generate cleaned and enriched data, wherein when data not meeting the specifications is identified during the cleaning and filtering process, data quality warning data is generated;

[0253] A summary and aggregation unit, used to process the cleaned and enriched data in parallel according to data summary, data aggregation, data merging, and data deduplication specifications to generate summary and aggregated data;

[0254] An integration service unit, used to process the cleaned and enriched data in accordance with data blacklisting, data whitelisting, and data integration specifications in sequence to generate integrated service data;

[0255] A dictionary data unit, used to process the original dictionary data according to the audit verification specification and integrate it with the summary and aggregated data to generate integrated dictionary data;

[0256] The data generating unit generates the managed data set consisting of the integrated service data, the integrated dictionary data and the data quality warning data.

[0257] In some embodiments, the fusion association analysis module 803 includes:

[0258] Expand the analysis module to build an analysis module library;

[0259] A process conversion engine, used to construct an executable model instance in combination with the analysis modules in the analysis module library;

[0260] An analysis process execution engine is used to execute the executable model instance to process the governed data set.

[0261] In some embodiments, the analysis module comprises:

[0262] A threshold deviation detection unit, used to train a threshold deviation behavior detection algorithm model using the governed data set, and set one or more threshold reference values ​​based on the model;

[0263] A time series correlation analysis unit is used to perform time series correlation analysis on a threshold reference value and data in a managed data set through a time series correlation algorithm, and when a network security threat event is identified from the data in the managed data set based on the time series correlation analysis, generate the corresponding comprehensive alarm event.

[0264] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present disclosure, the technical solutions provided by the embodiments of the present disclosure are described in detail below through specific embodiments:

[0265] Embodiment 1

[0266] Based on multi-source data such as network data, security data, and password data, by adopting a variety of advanced algorithms such as machine learning, threshold deviation behavior detection, and time series correlation, multi-dimensional fusion correlation analysis is performed on different types of data to explore the potential relationship between multiple independent security events, thereby identifying hidden or complex combined network security events.

[0267] Fig. 9 It is a schematic diagram of data fusion association analysis according to an embodiment of the present disclosure.

[0268] According to the data fusion association analysis schematic diagram described in the embodiment of the present disclosure, the specific implementation process is as follows:

[0269] 1. Data connection and aggregation:

[0270] First, through the multi-data access and aggregation technology, password data, security data and network data are obtained and aggregated from multiple sources, and these data sources are enriched, classified, filtered and processed, and the processed data is stored;

[0271] 2. Threshold deviation anomaly detection:

[0272] Apply the threshold deviation anomaly detection model to the stored data, analyze the password data in the transmission mode, obtain the security upper and lower bounds of its threshold deviation, and monitor the abnormal deviation of these thresholds in real time. The generated threshold abnormal deviation results are used for subsequent correlation analysis;

[0273] 3. Time series correlation analysis:

[0274] Based on the time series correlation algorithm, the threshold deviation of password data and other abnormal deviation events are analyzed vertically and horizontally. At the same time, the key fields in network data, password data and security data are integrated.

[0275] 4. Security threat alert generation:

[0276] When the abnormal deviation of the threshold matches the network security event, a security threat alarm will be generated to indicate the possible hidden lateral threat attack.

[0277] 5. Comprehensive alarm output:

[0278] Finally, comprehensive alarm events are output, which include alarm details, alarm level, related logs, defense plans, and linkage strategies, etc., to help further security management and protection measures.

[0279] Through the above steps, the present disclosure solves the bottleneck problem of the convergence of diverse and dispersed heterogeneous data sources through unified standards and orderly data collection, and forms a consistent correlation analysis data set. It no longer relies on a single log data set, but integrates multiple data for comprehensive modeling and analysis, thereby achieving in-depth correlation analysis of complex and hidden network attacks, solving the shortcomings of traditional network security analysis.

[0280] In the embodiments of the present disclosure, those skilled in the art should understand that the examples specifically given above are exemplary and various modifications may be made thereto according to the methods of the present disclosure.

[0281] This paper focuses on multi-source data fusion correlation analysis, and the key technical points involved are as follows:

[0282] By providing a variety of data collection interfaces, unified aggregation of multi-source heterogeneous data is achieved. Through data governance, unified data pre-processing such as data conversion, data filtering, and data supplementation is achieved. Preliminary data integration is performed during the data connection process, ultimately achieving integrated aggregation and processing of multi-source data.

[0283] Based on the data generated by network, security, and cryptographic devices, we use machine learning, threshold deviation behavior detection, time series correlation and other algorithms, as well as multi-dimensional modeling technology for massive data fusion analysis, to build a fusion correlation analysis model based on big data. We conduct multi-dimensional and in-depth fusion correlation analysis on network data, security data, and cryptographic data, and explore the potential relationships between independent security events, thereby discovering complex network combination attack events.

[0284] The main beneficial effects of the present disclosure are as follows:

[0285] 1. Solve the bottleneck problem faced by diverse, dispersed and heterogeneous data sources in the data collection process. Through the implementation of systematic data collection governance measures, ensure the unified standardization, consistency and order of data, so as to form a reliable data set for correlation analysis. This process includes data standardization, data filtering and data supplementation to ensure that the final data set can support effective analysis and decision-making.

[0286] 2. Use multiple data sources for fusion and correlation modeling analysis, instead of relying solely on a single log data set. Through comprehensive analysis of multiple network security events, advanced technical means are used for in-depth correlation analysis. This method can identify and reveal hidden and complex network attack combinations, thereby overcoming the limitations of traditional network security analysis and improving the ability to detect and respond to complex attack patterns.

[0287] In general, this disclosure effectively improves the detection capability of network security incidents by integrating multi-source heterogeneous data and advanced modeling and analysis techniques, can identify complex and hidden attack patterns, and overcomes the limitations of traditional security analysis.

[0288] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0289] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data fusion association analysis method, include: Acquire data from multiple heterogeneous data sources through a variety of collection interfaces; Performing data governance on the acquired data to generate a governed data set; A multi-dimensional fusion correlation analysis is performed on the governed data set to generate corresponding comprehensive alarm events.

2. The data fusion association analysis method according to claim 1, in, Acquire data from multiple heterogeneous data sources through a variety of collection interfaces, including: receiving the data according to a communication protocol of the heterogeneous data source; Parsing the data using a predefined parsing template to generate standardized data; The standardized data is classified and stored.

3. The data fusion association analysis method according to claim 2, in, Receiving the data according to the communication protocol of the heterogeneous data source includes: Configure offline and real-time data receiving models; The data is received in real time or offline according to the communication protocol of the heterogeneous data source.

4. The data fusion association analysis method according to claim 3, in, Classifying and storing the standardized data includes: Classifying the standardized data according to a multi-level dimension including a security event type and a security event level; The classified data is stored in the data center.

5. The data fusion association analysis method according to claim 4, in, Performing data governance on the acquired data to generate a governed data set includes: The acquired data is processed in sequence according to cleaning and filtering, data conversion, data extraction, and data enrichment specifications to generate cleaned and enriched data, wherein when data that does not conform to the specifications is identified during the cleaning and filtering process, data quality warning data is generated; The cleaned and enriched data are processed in parallel according to data summarization, data aggregation, data merging, and data deduplication specifications to generate summary and aggregated data; The cleaned and enriched data is processed in accordance with data black identification, data white identification, and data integration specifications to generate integrated service data; Processing the original dictionary data according to the audit verification specification and integrating it with the summary and aggregated data to generate integrated dictionary data; The governed data set consisting of the integrated service data, the integrated dictionary data and the data quality warning data is generated.

6. The data fusion association analysis method according to any one of claims 1 to 5, in, Perform multi-dimensional fusion correlation analysis on the governed data set to generate corresponding comprehensive alarm events, including: Build analysis module library; Building an executable model instance in combination with the analysis modules in the analysis module library; The executable model instance is executed to process the governed data set.

7. The data fusion association analysis method according to claim 6, in, Executing the executable model instance to process the governed data set includes: Using the governed data set to train a threshold deviation behavior detection algorithm model; Based on the trained threshold deviation behavior detection algorithm model, one or more threshold reference values ​​for judging abnormal behavior are set.

8. The data fusion association analysis method according to claim 7, in, Executing the executable model instance to process the governed data set further comprises: Performing time series correlation analysis on the threshold reference value and the data in the treated data set by using a time series correlation algorithm; When a network security threat event is identified from the data in the governed data set based on the time series correlation analysis, the corresponding comprehensive alarm event is generated.

9. An electronic device, include: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion association analysis method according to any one of claims 1 to 8.

10. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data fusion association analysis method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the data fusion association analysis method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Log correlation analysis method based on time axis events

    CN108964995A

  • Network attack path tracking method and device

    CN113783896A

  • Log anomaly detection system based on process invariance

    CN116028266A

  • Multi-source heterogeneous alarm information fusion method and system based on space-time relevance

    CN116483666A

  • Abnormal event time sequence correlation analysis method and device for multi-source heterogeneous data and storage medium

    CN117909890A

Cited By

  • Data security risk management method and device based on multi-modal large model

    CN120185931A

  • AI security detection method and system combining off-line and on-line dual modes

    CN121462317A

  • An ai security detection method and system combining offline and online dual modes

    CN121462317B