Network traffic protocol log high-speed persistence method and device based on message queue
Through the high-speed persistence method of network traffic protocol logs based on message queues, the problems of slow loading speed of network traffic logs and low data retrieval efficiency in traditional technology are solved, and high-speed parallel loading and efficient query of large-scale network traffic logs are realized, and the network is supported. Space security business analysis.
Patent Information
- Application Number
- CN202410854253.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional network traffic log processing technology cannot meet the needs of large-scale, high-speed, and high-complex network traffic data processing, especially inefficient in loading, storage and querying, and cannot meet the needs of complex queries.
Using the high-speed persistence method of network traffic protocol logs based on message queues, read data through multi-threading, performing data partitioning, multi-column hybrid sorting and indexing, directly store data files into distributed databases, and utilize HDFS and Hive storage structures. , supports efficient loading and querying.
It realizes high-speed parallel loading of large-scale network traffic logs, improves data query access efficiency, supports network space security service analysis, and ensures high-speed loading and fast access of data.
Smart Images

Figure CN119046280A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer data processing, and in particular to a method and device for high-speed persistence of network traffic protocol logs based on a message queue. Background Art
[0002] With the rapid development and widespread application of Internet technology, network traffic has become one of the important indicators for measuring network performance and security. In order to effectively monitor, manage and optimize network traffic and ensure the security and stability of the network, the collection, storage and analysis of network traffic protocol logs have emerged. Network traffic data has shown an explosive growth trend. This growth has not only brought unprecedented pressure to the network infrastructure, but also put forward higher requirements for the loading, storage and query technology of network traffic protocol logs. Traditional log processing technology can no longer meet the current large-scale, high-speed and high-complexity network traffic data processing needs. For large-scale network traffic data, how to effectively load, store and query has become an urgent problem to be solved.
[0003] In terms of loading, traditional network traffic log loading mainly relies on simple file transfer and reading technology. However, with the continuous increase of network traffic data, traditional loading methods can no longer meet the requirements of efficiency and stability. In terms of storage, traditional relational databases often face performance bottlenecks and storage limitations when facing massive network traffic logs. In terms of query, how to quickly and accurately query the required information from massive network traffic logs is an important issue that affects the efficiency of business analysis. Traditional query methods often rely on simple text searches or database query statements, but this method is inefficient when processing large-scale data and cannot meet complex query requirements. Summary of the invention
[0004] In order to solve the defects of the prior art, the present invention provides a method and device for high-speed persistence of network traffic protocol logs based on message queues, which solves the problems of slow loading speed and low efficiency of large-scale data retrieval when loading network protocol logs under large-scale network traffic.
[0005] The present invention provides the following technical solutions:
[0006] A high-speed persistence method for network traffic protocol logs based on a message queue includes several scalable persistence threads, each of which is configured to execute the following method:
[0007] Read network traffic protocol log data from the message queue into memory;
[0008] The read network traffic protocol log data is partitioned in the memory, and the partition data belonging to the same partition is divided into the same data cache set;
[0009] In memory, multiple columns of mixed sorting are performed on each partition data in the same data cache set according to the set search fields;
[0010] Build an index for the partitioned data after mixed sorting of multiple columns in memory according to the set search fields;
[0011] The partitioned data in the same data cache collection after indexing is stored persistently in the distributed database as a data file.
[0012] Further, the reading of network traffic protocol log data from the message queue into the memory includes:
[0013] Connect with the message queue, and actively pull the network traffic protocol log data in the Topic of the message queue into the memory by calling the subscription service interface of the message queue;
[0014] The column fields of the network traffic protocol log data are identified in memory according to the data format configuration file, and the network traffic protocol log data in Avro format is deserialized.
[0015] Furthermore, the read network traffic protocol log data is partitioned in the memory, and the partitioned data belonging to the same partition is divided into the same data cache set, including:
[0016] Partitioning the network traffic protocol log data in memory according to time to obtain a plurality of partition data;
[0017] Partition data belonging to the same partition is divided into the same data cache set in memory.
[0018] Furthermore, the step of performing multi-column mixed sorting on each partition data in the same data cache set in the memory according to the set search field includes:
[0019] According to the set search fields, the partition data in the same data cache set is sorted by multiple columns based on the Morton sorting algorithm.
[0020] Furthermore, the partition data in the same data cache set after indexing is persistently stored in the distributed database as a data file, including:
[0021] Determine whether the number of partition data cached in the data cache set in the memory reaches the set partition data number threshold, or determine whether the data cache set in the memory reaches the set cache time threshold;
[0022] If the set partition data number threshold or the set cache time threshold is reached, the data cache collection in the memory is organized into a data file in HDFS format, and the constructed index is used as the metadata information of the data file;
[0023] The data file is directly and persistently stored in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the data structure of Hive on HDFS.
[0024] Furthermore, the method further comprises:
[0025] Periodically merge multiple data files whose file sizes after persistent storage are less than the set size threshold.
[0026] A high-speed persistence device for network traffic protocol logs based on a message queue includes a plurality of scalable persistence threads, each of which is configured to execute the following process:
[0027] The data reading module is used to read the network traffic protocol log data from the message queue into the memory;
[0028] A data partitioning module is used to partition the read network traffic protocol log data in the memory, and divide the partition data belonging to the same partition into the same data cache set;
[0029] The data sorting module is used to perform multi-column mixed sorting on each partition data in the same data cache set in memory according to the set search fields;
[0030] The index building module is used to build an index for the partitioned data after mixed sorting of multiple columns in memory according to the set search fields;
[0031] The persistent storage module is used to store the partitioned data in the same data cache set after indexing as a data file in a distributed database.
[0032] Furthermore, the data reading module includes:
[0033] A data extraction unit is used to connect with the message queue and actively pull the network traffic protocol log data in the Topic of the message queue into the memory by calling the subscription service interface of the message queue;
[0034] The data processing unit is used to identify the column fields of the network traffic protocol log data according to the data format configuration file in the memory, and deserialize the network traffic protocol log data in Avro format.
[0035] Furthermore, the data partitioning module includes:
[0036] A data partitioning unit, used to partition the network traffic protocol log data according to time in the memory to obtain a plurality of partition data;
[0037] The set partitioning unit is used to partition the partition data belonging to the same partition into the same data cache set in the memory.
[0038] Furthermore, the index building module is used to:
[0039] According to the set search fields, the partition data in the same data cache set is sorted by multiple columns based on the Morton sorting algorithm.
[0040] Furthermore, the persistent storage module includes:
[0041] A judging unit, used to judge whether the number of partition data cached in the data cache set in the memory reaches a set partition data number threshold, or judge whether the data cache set in the memory reaches a set cache time threshold;
[0042] A file organization unit is used to organize the data cache collection in the memory into a data file in HDFS format if the set partition data number threshold is reached or the set cache time threshold is reached, and use the constructed index as the metadata information of the data file;
[0043] The persistent storage unit is used to store the data file directly and persistently in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the data structure of Hive on HDFS.
[0044] Furthermore, the device also includes:
[0045] The merging module is used to periodically merge multiple data files whose file sizes after persistent storage are smaller than a set size threshold.
[0046] The present invention has the following beneficial effects:
[0047] The present invention is executed through multiple threads, each thread reads the protocol log data from the message queue, then performs multi-column mixed sorting on the data, builds a data index for the sorted partition data, and finally organizes the data into a data file format, and directly stores the data file in a database.
[0048] The present invention stores the massive network traffic protocol logs cached in the message queue in a database in a persistent manner, solving the problems of slow loading speed and low efficiency of large-scale data retrieval when loading network protocol logs under large-scale network traffic. The rate of a single persistence process is not less than 300MB / s, and has horizontal linear expansion characteristics, supports high-speed parallel loading of large-scale network traffic protocol logs, and solves the loading performance bottleneck of network protocol logs under large-scale network traffic. And by partitioning data and constructing common field indexes based on multi-column mixed sorting of data, the efficiency of data query access is greatly improved. It provides a basic support base for high-speed data loading and rapid access for upper-level business analysis such as situation monitoring, threat analysis, and qualitative tracing for network traffic, ensuring the smooth development of business analysis for cyberspace security. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flow chart of the method for high-speed persistence of network traffic protocol log based on message queue of the present invention;
[0050] Figure 2 An exemplary diagram of the framework of the present invention;
[0051] Figure 3 A schematic diagram of the connection relationship between the present invention and the message queue;
[0052] Figure 4 It is a schematic diagram of a high-speed persistence device for network traffic protocol logs based on a message queue of the present invention. DETAILED DESCRIPTION
[0053] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.
[0054] Due to the characteristics of high instantaneous traffic and large data scale of network traffic protocol logs, data loading and querying face bottleneck problems such as slow data loading and low query efficiency. Since the data persistence link can complete the on-site storage of data, real-time calculation of data storage partitions, and real-time construction of data indexes, the core bottlenecks of data loading and querying are in the data persistence link. It is urgent to improve the data loading speed, optimize the data storage mode, and build data indexes in the data persistence link, so as to provide a basic support base for high-speed data loading and fast access for upper-level business analysis such as situation monitoring, threat analysis, and traceability of network traffic, and ensure the smooth development of cyberspace security business analysis.
[0055] Based on this, the embodiment of the present invention provides a high-speed persistence method for network traffic protocol logs based on message queues. The method of the present invention stores the massive network traffic protocol logs cached in the message queue (such as Kafka) in a distributed database (such as Hive). The present invention is multi-threaded execution, and its framework is as follows Figure 2 As shown, it includes several scalable persistence threads, each of which is configured to execute the following Figure 1 The method shown:
[0056] S100: Read network traffic protocol log data from the message queue into the memory.
[0057] As network traffic data continues to grow, the traditional technology of loading network traffic logs that relies on simple file transfer and reading technology can no longer meet the requirements of efficiency and stability. Therefore, the present invention adopts a message queue for persistence to improve the speed and stability of log loading, and preliminarily realizes the efficient loading of large-scale network traffic logs by optimizing data transmission and processing procedures.
[0058] In an example, the specific implementation of this step includes:
[0059] S110: Connect with the message queue and actively pull the network traffic protocol log data in the Topic of the message queue into the memory by calling the subscription service interface of the message queue;
[0060] S120: Identify the column fields of the network traffic protocol log data in memory according to the data format configuration file, and deserialize the network traffic protocol log data in Avro format, laying a foundation for subsequent data partitioning, sorting, and indexing.
[0061] The present invention can meet the high-speed persistence requirements of large-scale network traffic protocol log data by deploying multiple sets, and support linear expansion. Multiple sets of the present invention can form a consumer group to jointly complete the persistence of one or more Topic data, thereby improving the data persistence rate. The Topic of the message queue and the deployed persistence program can have a one-to-one, one-to-many, many-to-one, or many-to-many relationship, such as Figure 3 shown.
[0062] S200: partitioning the read network traffic protocol log data in the memory, and dividing the partition data belonging to the same partition into the same data cache set.
[0063] Data partitioning is the process of dividing a large data set into smaller, more manageable parts. Specific methods of data partitioning include:
[0064] S210: partitioning the network traffic protocol log data in the memory according to time (for example, partitioning the data according to day or hour granularity based on the time feature of the data) to obtain a plurality of partitioned data.
[0065] S220: Divide the partition data belonging to the same partition into the same data cache set.
[0066] The main benefit of data partitioning is that it can effectively improve query efficiency. When executing a query involving a partition column, the system only needs to scan the partition containing the relevant data, rather than the entire data set. This greatly reduces the time for I / O operations and data scanning, and speeds up queries. Assuming that the protocol log repository docked by the present invention is Hive, it supports table-level partitioning, and the data cache collection of each partition is physically stored as an independent HDFS file directory.
[0067] S300: performing multi-column mixed sorting on each partition data in the same data cache set in memory according to the set search field.
[0068] For the data partition files of multiple memory caches in the previous step, the cached partition data is sorted by multiple columns before persistence operation. The data fields selected for the multi-column mixed sorting are the commonly used search fields for access protocol logs: such as source IP, source port, time, protocol, etc. The specific steps of the multi-column mixed sorting are: according to the set search fields, the multi-column mixed sorting is performed on the partition data in the same data cache set based on the Morton sorting algorithm.
[0069] The purpose of multi-column mixed sorting is to improve the acceleration effectiveness of the overall index when accessing data by constructing indexes for multiple columns separately, avoiding the situation where sorting only one column makes the indexes constructed for other columns meaningless in improving data access efficiency.
[0070] The Morton sorting algorithm is used for multi-column mixed sorting. The network traffic protocol log is two-dimensional data. The data dimensions involved in the sorting are encoded as binary numbers and intertwined to form a Z-curve code. This code represents the position in a one-dimensional space and converts the multi-dimensional spatial description of the data into a one-dimensional spatial description. By sorting the one-dimensional code, a mixed sorting result based on multi-column data is obtained.
[0071] S400: indexes are constructed in memory for the partitioned data after mixed sorting of multiple columns according to the set commonly used search fields.
[0072] Traditional query methods often rely on simple text searches or database query statements, but this method is inefficient when processing large-scale data and cannot meet complex query requirements. In order to quickly and accurately query the required information, the present invention adds indexes to the protocol log to achieve efficient query. The index of the present invention supports Max / Min, BitMap, BloomFilter and other indexes.
[0073] S500: After indexing, partition data in the same data cache set is stored as a file in a distributed database.
[0074] Traditional relational databases often face performance bottlenecks and storage limitations when facing massive network traffic logs. Therefore, the present invention adopts a storage solution of a distributed database. Through distributed storage, horizontal expansion and other technical means, it is possible to provide storage services with high reliability, high scalability and high fault tolerance. By distributing and storing data on multiple nodes, horizontal expansion and load balancing of data are achieved, so that massive network traffic log data can be stored.
[0075] A specific implementation of this step includes:
[0076] S510: Determine whether the number of partition data cached in the data cache set in the memory reaches a set partition data number threshold (eg, 20,000 entries), or determine whether the data cache set in the memory reaches a set cache time threshold (eg, 10 minutes).
[0077] S520: If the set partition data number threshold is reached or the set cache time threshold is reached, a persistence operation is triggered to organize the data cache collection in the memory into a data file in HDFS format (such as a Parquet data file), and the constructed index is used as metadata information of the data file.
[0078] The present invention saves the index of partition data into the metadata of the data file. Taking the Parquet data file as an example, a file consists of multiple Row Groups. A Row Group includes multiple column chunks. A column chunk is all data sets of a column, which are divided into multiple pages. A page is the smallest processing unit and can be encoded or compressed.
[0079] The index is written to the Column Meta Data of each Row Group in FileMetaData. When reading data files, the predicate push-down technology is used. Based on the constructed column index, the Row Group data in the Parquet file that needs further analysis and processing can be quickly filtered, so as to quickly determine whether the data file contains data records that meet the search conditions, thereby reducing the amount of data scanning and improving data retrieval efficiency.
[0080] S530: The organized data files are directly and persistently stored in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the data structure of Hive on HDFS.
[0081] In order to improve the persistence efficiency, the present invention needs to bypass the Hive storage module, organize the data to be persisted into the file storage format of Hive on HDFS, and directly perform persistent storage according to the directory structure of Hive on HDFS.
[0082] As an improvement, the method of the present invention further comprises:
[0083] S600: Periodically merge multiple data files whose file sizes after persistent storage are smaller than a set size threshold.
[0084] For persistent data files, since the files are small, in order to solve the metadata storage management pressure brought by a large number of small files, after the data is persistently stored, the small file merging operation is performed offline at a regular interval. Taking the Parquet file format as an example, the m Row Group data in multiple small files are merged into n Row Groups under one large file (default 256MB). At the same time, the column index of each Row Group after merging is calculated based on the column index value of the Row Group of the small file.
[0085] The present invention is executed through multiple threads, each thread reads the protocol log data from the message queue, then performs multi-column mixed sorting on the data, builds a data index for the sorted partition data, and finally organizes the data into a data file format, and directly stores the data file in a database.
[0086] The present invention stores the massive network traffic protocol logs cached in the message queue in a database in a persistent manner, solving the problems of slow loading speed and low efficiency of large-scale data retrieval when loading network protocol logs under large-scale network traffic. The rate of a single persistence process is not less than 300MB / s, and has horizontal linear expansion characteristics, supports high-speed parallel loading of large-scale network traffic protocol logs, and solves the loading performance bottleneck of network protocol logs under large-scale network traffic. And by partitioning data and constructing common field indexes based on multi-column mixed sorting of data, the efficiency of data query access is greatly improved. It provides a basic support base for high-speed data loading and rapid access for upper-level business analysis such as situation monitoring, threat analysis, and qualitative tracing for network traffic, ensuring the smooth development of business analysis for cyberspace security.
[0087] The embodiment of the present invention also provides a high-speed persistence device for network traffic protocol logs based on a message queue, which includes a plurality of scalable persistence threads, each of which is configured to execute the following Figure 4 The process described:
[0088] The data reading module 100 is used to read the network traffic protocol log data from the message queue into the memory.
[0089] The data partitioning module 200 is used to partition the read network traffic protocol log data in the memory, and divide the partition data belonging to the same partition into the same data cache set.
[0090] The data sorting module 300 is used to perform multi-column mixed sorting on each partition data in the same data cache set in the memory according to the set search fields.
[0091] The index building module 400 is used to build an index for the partition data after mixed sorting of multiple columns in memory according to the set search field.
[0092] The persistent storage module 500 is used to persistently store the partitioned data in the same data cache set after indexing as a data file in a distributed database.
[0093] As an improvement, the aforementioned data reading module includes:
[0094] The data extraction unit is used to connect with the message queue and actively pull the network traffic protocol log data in the message queue Topic into the memory by calling the subscription service interface of the message queue.
[0095] The data processing unit is used to identify the column fields of the network traffic protocol log data according to the data format configuration file in the memory, and deserialize the network traffic protocol log data in the Avro format.
[0096] The data partition module in the present invention includes:
[0097] The data partition unit is used to partition the network traffic protocol log data in the memory according to time to obtain a plurality of partition data.
[0098] The set partitioning unit is used to partition the partition data belonging to the same partition into the same data cache set in the memory.
[0099] Furthermore, the index building module is used to:
[0100] According to the set search fields, the partition data in the same data cache set is sorted by multiple columns based on the Morton sorting algorithm.
[0101] As another improvement, the aforementioned persistent storage module includes:
[0102] The judgment unit is used to judge whether the number of partition data cached in the data cache set in the memory reaches a set partition data number threshold, or judge whether the data cache set in the memory reaches a set cache time threshold.
[0103] The file organization unit is used to organize the data cache collection in the memory into a data file in the HDFS format if the set partition data number threshold or the set cache time threshold is reached, and use the constructed index as the metadata information of the data file.
[0104] The persistent storage unit is used to store data files directly and persistently in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the Hive data structure on HDFS.
[0105] After persistent storage, the device further includes:
[0106] The merging module is used to periodically merge multiple data files whose file sizes after persistent storage are smaller than a set size threshold.
[0107] The device provided in the above embodiment, its implementation principle and the technical effect produced correspond to the embodiment of the above method one by one. For the sake of brief description, the part not mentioned in the embodiment of the device can refer to the corresponding content in the embodiment of the above method. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the modules and units described in the device can refer to the corresponding process in the embodiment of the above method, and will not be repeated here.
[0108] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A high-speed persistence method for network traffic protocol logs based on message queues, characterized in that: It includes several scalable persistence threads, each of which is configured to execute the following methods: Read network traffic protocol log data from the message queue into memory; The read network traffic protocol log data is partitioned in the memory, and the partition data belonging to the same partition is divided into the same data cache set; In memory, multiple columns of mixed sorting are performed on each partition data in the same data cache set according to the set search fields; Build an index for the partitioned data after mixed sorting of multiple columns in memory according to the set search fields; The partitioned data in the same data cache collection after indexing is stored persistently in the distributed database as a data file.
2. According to the message queue-based network traffic protocol log high-speed persistence method of claim 1, it is characterized in that: The reading of network traffic protocol log data from the message queue into the memory includes: Connect with the message queue, and actively pull the network traffic protocol log data in the Topic of the message queue into the memory by calling the subscription service interface of the message queue; The column fields of the network traffic protocol log data are identified in memory according to the data format configuration file, and the network traffic protocol log data in Avro format is deserialized.
3. According to the message queue-based network traffic protocol log high-speed persistence method of claim 2, it is characterized in that: The data partitioning is performed on the read network traffic protocol log data in the memory, and the partition data belonging to the same partition is divided into the same data cache set, including: Partitioning the network traffic protocol log data in memory according to time to obtain a plurality of partition data; Partition data belonging to the same partition is divided into the same data cache set in memory.
4. According to the message queue-based network traffic protocol log high-speed persistence method of claim 3, it is characterized in that: The step of performing multi-column mixed sorting on each partition data in the same data cache set in the memory according to the set search field includes: According to the set search fields, the partition data in the same data cache set is sorted by multiple columns based on the Morton sorting algorithm.
5. According to the message queue-based network traffic protocol log high-speed persistence method of claim 4, it is characterized in that: The partition data in the same data cache set after indexing is persistently stored in the distributed database as a data file, including: Determine whether the number of partition data cached in the data cache set in the memory reaches the set partition data number threshold, or determine whether the data cache set in the memory reaches the set cache time threshold; If the set partition data number threshold or the set cache time threshold is reached, the data cache collection in the memory is organized into a data file in HDFS format, and the constructed index is used as the metadata information of the data file; The data file is directly and persistently stored in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the data structure of Hive on HDFS.
6. The method for high-speed persistence of network traffic protocol logs based on message queues according to any one of claims 1 to 5, characterized in that: The method further comprises: Periodically merge multiple data files whose file sizes after persistent storage are less than the set size threshold.
7. A high-speed persistence device for network traffic protocol logs based on message queues, characterized in that: It includes several scalable persistence threads, each of which is configured to execute the following process: The data reading module is used to read the network traffic protocol log data from the message queue into the memory; A data partitioning module is used to partition the read network traffic protocol log data in the memory, and divide the partition data belonging to the same partition into the same data cache set; The data sorting module is used to perform multi-column mixed sorting on each partition data in the same data cache set in memory according to the set search fields; The index building module is used to build an index for the partitioned data after mixed sorting of multiple columns in memory according to the set search fields; The persistent storage module is used to store the partitioned data in the same data cache set after indexing as a data file in a distributed database.
8. The high-speed persistence device for network traffic protocol logs based on message queues according to claim 7 is characterized in that: The data reading module comprises: A data extraction unit is used to connect with the message queue and actively pull the network traffic protocol log data in the Topic of the message queue into the memory by calling the subscription service interface of the message queue; The data processing unit is used to identify the column fields of the network traffic protocol log data according to the data format configuration file in the memory, and deserialize the network traffic protocol log data in Avro format.
9. The high-speed persistence device for network traffic protocol logs based on message queues according to claim 8 is characterized in that: The data partition module comprises: A data partitioning unit, used to partition the network traffic protocol log data according to time in the memory to obtain a plurality of partition data; The set partitioning unit is used to partition the partition data belonging to the same partition into the same data cache set in the memory.
10. The high-speed persistence device for network traffic protocol logs based on message queues according to claim 9 is characterized in that: The persistent storage module includes: A judging unit, used to judge whether the number of partition data cached in the data cache set in the memory reaches a set partition data number threshold, or judge whether the data cache set in the memory reaches a set cache time threshold; A file organization unit is used to organize the data cache collection in the memory into a data file in HDFS format if the set partition data number threshold is reached or the set cache time threshold is reached, and use the constructed index as the metadata information of the data file; The persistent storage unit is used to store the data file directly and persistently in the HDFS of the Hive database according to the Hive file mode, bypassing the Hive storage module, and according to the data structure of Hive on HDFS.
