Lake and warehouse integrated architecture based on threat intelligence multi-source heterogeneous data
Through the combination of Paimon and StarRocks, combined with the Flink streaming and batch integrated computing engine, the Hujian integrated architecture was built, solving the problems of high cost, low efficiency and poor real-time performance of traditional data warehouses when processing multi-source heterogeneous threat intelligence data, and realizing the Efficient data storage and real-time analysis can reduce operational costs and ensure data security.
Patent Information
- Application Number
- CN202411001472.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-03
AI Technical Summary
Traditional data warehouses have high cost, low efficiency, poor real-time performance, complex data governance, data quality problems and poor query performance when processing multi-source heterogeneous threat intelligence data, which is difficult to meet the needs of real-time analysis.
Using Paimon as the data storage engine and StarRocks as the data analysis engine, combined with Flink streaming and batch integrated computing engine, a Hucang integrated architecture is built to realize efficient storage, management and real-time analysis of threat intelligence data, and optimize data access through metadata management and materialized views. and analysis.
It realizes efficient data query and analysis, meets the needs of real-time analysis, reduces operating costs, improves data storage and management efficiency, ensures data security and privacy, and simplifies data observability.
Smart Images

Figure CN119066046A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence. Background Art
[0002] With the advent of the intelligent era of the Internet of Everything, data has become a key means of production. According to data from IDC, the world is expected to generate 216ZB of data each year by 2026. This growth in data volume has brought new challenges to data infrastructure platforms, and traditional data warehouses can no longer meet the storage and analysis needs of the surging data.
[0003] In the current network environment, threat intelligence has become an important means to ensure information security. Threat intelligence involves a large amount of multi-source heterogeneous data, which includes data from different sources, formats and structures, such as log data, network traffic data, security event data, etc. How to effectively store, manage and analyze these multi-source heterogeneous data has become a major challenge facing enterprises and organizations.
[0004] To meet these challenges, big data architecture needs to upgrade data storage and computing analysis capabilities while improving resource utilization. Traditional data warehouses and data lakes have their own advantages and disadvantages in processing multi-source heterogeneous data. Data warehouses provide efficient data query and analysis capabilities, but are less efficient when processing large-scale unstructured data; data lakes can store large-scale unstructured data, but have deficiencies in data management and query optimization.
[0005] Data warehouse is a technology used to store, manage and analyze large amounts of historical data. Its main features are centralization, non-real-time, and batch processing. Data warehouse is subject-oriented and integrated, and is built to support various businesses. The data comes from scattered operational data. Therefore, it is necessary to extract the required data from multiple heterogeneous data sources, process and integrate them, reorganize them according to themes, and finally enter the data warehouse. Data warehouses are mainly used to support enterprise decision-making analysis, and the data operations designed are mainly data queries.
[0006] A data lake is a large data warehouse that centrally stores various types of structured and unstructured data. It can store raw data from multiple data sources and multiple data types. Data can be accessed, processed, analyzed, and transmitted without undergoing structured processing. A data lake can help companies quickly complete federated analysis, mining, and exploration of data value from heterogeneous data sources. The essence of a data lake is a solution consisting of "data storage architecture + data processing tools."
[0007] However, the above prior art has the following disadvantages:
[0008] (1) The cost of building and maintaining a data warehouse is high, including hardware, software, and labor costs.
[0009] (2) Data warehouses mainly focus on structured data and have a weak ability to process unstructured data in threat intelligence.
[0010] (3) Data in data warehouses are usually updated in batch mode, which has poor real-time performance and is not suitable for application scenarios that require real-time data.
[0011] (4) Due to the wide variety of threat intelligence data stored in the data lake, data governance and security management are relatively complex.
[0012] (5) The threat intelligence data in the data lake has not been cleaned and transformed, and may have data quality issues.
[0013] (6) Due to the diverse data formats in the data lake, the query performance may not be as good as that of the data warehouse.
[0014] (7) The fusion processing efficiency of multi-source heterogeneous data is low and it is difficult to meet the needs of real-time analysis. Summary of the invention
[0015] The object of the present invention is to solve at least one of the technical drawbacks.
[0016] To this end, the purpose of the present invention is to propose a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence to solve the problems mentioned in the background technology and overcome the shortcomings of the prior art.
[0017] In order to achieve the above objectives, an embodiment of the present invention provides a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence, including:
[0018] Data storage layer, data management layer, data analysis layer and real-time alarm and automated response layer, among which:
[0019] The data storage layer is used to use Paimon as a data storage engine to write and read the collected threat intelligence data and store it in Paimon;
[0020] The data management layer records the storage location, version information, and partition information of threat intelligence data based on the metadata management function of Paimon, and realizes the location and access of data through metadata management;
[0021] The data analysis layer uses StarRocks as the data analysis engine, reads the multi-source heterogeneous threat intelligence data in Paimon through the appearance characteristics of StarRocks, establishes a materialized view as the data detail layer DWD layer, realizes the analysis of threat intelligence data, and generates threat intelligence analysis results;
[0022] The real-time alarm and automated response layer is used to generate alarm information in real time based on the threat intelligence analysis results, so as to promptly notify security personnel and respond to and handle security incidents;
[0023] The process of building a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence includes:
[0024] First, the Flink streaming-batch computing engine is used to build a data warehouse in the Paimon format on the data lake. Flink is used to complete the calculations from the data warehouse's data operation layer (ODS) layer to the data detail layer (DWD layer), the data service layer (DWS layer), and the data application layer (ADS layer). StarRocks is used to perform unified OLAP queries on each layer of the data warehouse and online analysis on the ADS layer.
[0025] Preferably, any of the above solutions includes: logs, network traffic and attack samples.
[0026] Preferably, any of the above solutions supports distributed storage and multiple copy redundancy.
[0027] Preferably, any of the above schemes stores the threat intelligence data in Paimon in the form of partitions and files, and supports version control and incremental updates of the threat intelligence data.
[0028] Preferably, any of the above schemes uses the materialized view of StarRocks in the data analysis layer as the DWD layer, and then uses the second layer of nested materialized view as the DWS layer, and finally provides it to the data service layer for data analysis to achieve instant query and analysis of threat intelligence data.
[0029] The preferred solution among the above is to use StarRocks to uniformly manage the data lake and data warehouse, and to analyze the businesses with high concurrency and real-time requirements in StarRocks.
[0030] The preferred solution among the above is to use Paimon Catalog and external tables to perform analysis on the data lake, so that data in Apache Paimon can be directly queried without performing data import.
[0031] Preferably, any of the above solutions designs a dynamic data migration mechanism to automatically adjust the storage location and format of threat intelligence data between the data lake and the data warehouse according to the usage frequency and query requirements of the threat intelligence data.
[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0033] The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence in the embodiment of the present invention solves the problems of high real-time processing requirements of multi-source heterogeneous data of threat intelligence in big data scenarios, difficult unstructured data management, and complex system operation and maintenance. It breaks down the barriers between data warehouses and data lakes and integrates the advantages of both.
[0034] 1. It realizes efficient data query and analysis, meets real-time analysis needs, and improves the timeliness and accuracy of business decisions.
[0035] 2. Improved the storage and management efficiency of multi-source heterogeneous data and solved the heterogeneity problems of data formats and structures.
[0036] 3. Provide a streamlined platform through a single data storage system to meet all business data needs. Lake warehouse integration can also reduce the amount of data moved through data pipelines to multiple systems, thereby simplifying data observability.
[0037] 4. The operating cost of the lake-warehouse integrated data analysis solution is lower than that of the data warehouse. In addition, the lake-warehouse integrated hybrid architecture does not require people to maintain multiple data storage systems, thereby reducing operating costs.
[0038] 5. The lake-warehouse integrated architecture alleviates the standard governance issues brought by the data lake. For example, when extracting and uploading data, it can ensure that the data meets the defined model requirements, thereby reducing downstream data quality issues.
[0039] 6. Through the data security module, the security and privacy of data are guaranteed and the data compliance requirements are met.
[0040] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0042] Figure 1 It is a structural diagram of a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to an embodiment of the present invention;
[0043] Figure 2 It is a schematic diagram of a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to an embodiment of the present invention;
[0044] Figure 3 The data processing flow chart of accelerating query data analysis according to an embodiment of the present invention. DETAILED DESCRIPTION
[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0046] The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence proposed in the present invention combines the advantages of data lake and data warehouse through Paimon+StarRocks, providing a unified data storage and processing platform, realizing efficient storage, management and analysis of massive multi-source heterogeneous data of threat intelligence, and significantly improving the processing efficiency and analysis capabilities of threat intelligence.
[0047] like Figure 1 and Figure 2 As shown, the lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence in an embodiment of the present invention includes: a data storage layer, a data management layer, a data analysis layer, and a real-time alarm and automated response layer.
[0048] Specifically, the data storage layer uses Paimon as the data storage engine to write and read the collected threat intelligence data and store it in Paimon. The data storage layer supports distributed storage and multi-copy redundancy. Among them, threat intelligence data is divided into structured data and unstructured data according to type.
[0049] In an embodiment of the present invention, threat intelligence data includes: logs, network traffic, attack samples, etc.
[0050] The data storage layer stores threat intelligence data in Paimon in the form of partitions and files, supporting version control and incremental updates of threat intelligence data.
[0051] The data management layer is based on Paimon's metadata management function, which records the storage location, version information, and partition information of threat intelligence data, and realizes the location and access of data through metadata management.
[0052] The data analysis layer uses StarRocks as the data analysis engine. StarRocks supports efficient OLAP (on-line Analytical Processing) queries and has fast multi-dimensional analysis capabilities. Through the appearance characteristics of StarRocks, the massive multi-source heterogeneous threat intelligence data in Paimon is read, and a layer of materialized view is established as the DWD (DataWarehouse Details) layer to analyze the threat intelligence data and generate threat intelligence analysis results.
[0053] The data analysis layer uses StarRocks as the data analysis engine. The materialized view of StarRocks has certain ETL capabilities. After it serves as the DWD layer, it uses the second layer of nested materialized views as the DWS (Data Warehouse Service) layer, and finally provides data analysis to the data service layer to achieve instant query and analysis of threat intelligence data.
[0054] The real-time alarm and automated response layer is used to generate alarm information in real time based on the results of threat intelligence analysis, and promptly notify security personnel to ensure that security incidents can be responded to and handled quickly.
[0055] The process of building a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence includes:
[0056] like Figure 3 As shown in the figure, the Flink+Paimon+StarRocks solution combines Flink, Paimon and StarRocks very closely. First, the Flink batch-stream integrated computing engine is used to build the data warehouse in the Paimon format on the lake, and Flink is used to complete the calculation of the data warehouse ODS (Operation Data Store) layer to the DWD layer, DWS and ADS. By using StarRocks, unified OLAP queries and ADS (Application Data Service) layer online analysis are performed on each layer of the data warehouse. Based on Paimon, high-throughput entry into the lake can be achieved. Based on Flink, full-link batch-stream integrated computing can be achieved, and based on StarRocks, high-performance OLAP queries can be achieved, so the entire link can achieve a relatively good balance in terms of real-time performance, timeliness and cost.
[0057] The present invention uses StarRocks to uniformly manage the data lake and data warehouse, and analyzes the business with high concurrency and real-time requirements in StarRocks. Paimon Catalog and external tables can also be used to analyze on the data lake. The data in Apache Paimon can be directly queried without executing data import.
[0058] In the data lake warehouse scenario, Flink can be used to complete complex data splicing and aggregation calculations, and meet high real-time requirements. In addition, during the use of real-time links, it is inevitable that there will be a need for data correction and data backtracking due to some data delays and other issues. Flink's stream-batch integration feature allows users to conveniently use the same job code as the real-time link to efficiently complete data correction and data backtracking requirements.
[0059] The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence in the embodiment of the present invention deeply integrates Paimon and StarRocks to form a lake-warehouse integrated architecture with a collaborative working mechanism to store and analyze multi-source heterogeneous data of threat intelligence. The present invention designs a dynamic data migration mechanism to automatically adjust the storage location and format of data between the data lake and the data warehouse according to the frequency of use of threat intelligence data and query requirements, thereby optimizing query performance. The present invention realizes seamless collaboration of storage and computing, and can automatically optimize the utilization of storage and computing resources at different stages of the data life cycle.
[0060] The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence in the embodiment of the present invention can also use the MinIO object storage system in combination with Hudi to build a lake-warehouse integrated data analysis platform for threat intelligence.
[0061] The lake-warehouse integrated architecture based on threat intelligence multi-source heterogeneous data in the embodiment of the present invention combines the flexibility of the data lake with the efficiency of the data warehouse to build a lake-warehouse integrated architecture to solve the storage, management and analysis problems of multi-source heterogeneous data. Specifically, the present invention can solve the following problems existing in the following prior arts:
[0062] 1. The present invention solves the problem of low efficiency in converting existing threat intelligence data between data lakes and data warehouses.
[0063] 2. The present invention solves the consistency, real-time and efficiency problems in existing threat intelligence data processing.
[0064] 3. The present invention solves the problems of low efficiency and poor accuracy in multi-source heterogeneous data fusion processing in existing threat intelligence.
[0065] 4. The present invention provides threat intelligence data security and privacy protection to ensure data security.
[0066] The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence in the embodiment of the present invention solves the problems of high real-time processing requirements of multi-source heterogeneous data of threat intelligence in big data scenarios, difficult unstructured data management, and complex system operation and maintenance. It breaks down the barriers between data warehouses and data lakes and integrates the advantages of both.
[0067] 1. It realizes efficient data query and analysis, meets real-time analysis needs, and improves the timeliness and accuracy of business decisions.
[0068] 2. Improved the storage and management efficiency of multi-source heterogeneous data and solved the heterogeneity problems of data formats and structures.
[0069] 3. Provide a streamlined platform through a single data storage system to meet all business data needs. Lake warehouse integration can also reduce the amount of data moved through data pipelines to multiple systems, thereby simplifying data observability.
[0070] 4. The operating cost of the lake-warehouse integrated data analysis solution is lower than that of the data warehouse. In addition, the lake-warehouse integrated hybrid architecture does not require people to maintain multiple data storage systems, thereby reducing operating costs.
[0071] 5. The lake-warehouse integrated architecture alleviates the standard governance issues brought by the data lake. For example, when extracting and uploading data, it can ensure that the data meets the defined model requirements, thereby reducing downstream data quality issues.
[0072] 6. Through the data security module, the security and privacy of data are guaranteed and the data compliance requirements are met.
[0073] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0074] It is not difficult for those skilled in the art to understand that the present invention includes any combination of the invention content and specific implementation methods of the above specification and the various parts shown in the drawings. Due to the limited space and to make the specification concise, the various schemes composed of these combinations are not described one by one. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
[0075] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence, characterized in that: include: Data storage layer, data management layer, data analysis layer and real-time alarm and automated response layer, among which: The data storage layer is used to use Paimon as a data storage engine to write and read the collected threat intelligence data and store it in Paimon; The data management layer records the storage location, version information, and partition information of threat intelligence data based on the metadata management function of Paimon, and realizes the location and access of data through metadata management; The data analysis layer uses StarRocks as the data analysis engine, reads the multi-source heterogeneous threat intelligence data in Paimon through the appearance characteristics of StarRocks, establishes a materialized view as the data detail layer DWD layer, realizes the analysis of threat intelligence data, and generates threat intelligence analysis results; The real-time alarm and automated response layer is used to generate alarm information in real time based on the threat intelligence analysis results, so as to promptly notify security personnel and respond to and handle security incidents; The process of building a lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence includes: First, the Flink streaming-batch computing engine is used to build a data warehouse in the Paimon format on the data lake. Flink is used to complete the calculations from the data warehouse's data operation layer (ODS) layer to the data detail layer (DWD layer), the data service layer (DWS layer), and the data application layer (ADS layer). StarRocks is used to perform unified OLAP queries on each layer of the data warehouse and online analysis on the ADS layer.
2. The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to claim 1 is characterized in that: The threat intelligence data includes: logs, network traffic and attack samples.
3. The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to claim 1 is characterized in that: The data storage layer supports distributed storage and multi-copy redundancy.
4. The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to claim 1 is characterized in that: The data storage layer stores threat intelligence data in Paimon in the form of partitions and files, and supports version control and incremental updates of threat intelligence data.
5. The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to claim 1 is characterized in that: After the materialized view of StarRocks in the data analysis layer is used as the DWD layer, the second layer of nested materialized view is used as the DWS layer, and finally provided to the data service layer for data analysis to achieve real-time query and analysis of threat intelligence data.
6. The lake-warehouse integrated architecture based on threat intelligence multi-source heterogeneous data as claimed in claim 1 is characterized in that: Use StarRocks to uniformly manage data lakes and data warehouses, and analyze businesses with high concurrency and real-time requirements in StarRocks.
7. The lake-warehouse integrated architecture based on multi-source heterogeneous data of threat intelligence according to claim 1 is characterized in that: Use Paimon Catalog and external tables to perform analysis on the data lake. You can directly query the data in Apache Paimon without having to import data.
8. The lake-warehouse integrated architecture based on threat intelligence multi-source heterogeneous data as claimed in claim 1 is characterized in that: Design a dynamic data migration mechanism to automatically adjust the storage location and format of threat intelligence data between the data lake and data warehouse based on the usage frequency and query requirements of threat intelligence data.
