Open-type multi-source data collection system and method

Abstract

The invention relates to an open-type multi-source data collection system and method. The system comprises a task management module, a task execution module and a data storage module, wherein the task management module is used for creating a collection task as needed, configuring collection task information and generating a corresponding data collection interface rule, the task execution module is used for loading the data collection interface rule, executing the collection task in the activated state, conducting data source monitoring and data collection on the corresponding collection mask, packaging the collected data according to a unified format, and sending the packaged data to the data storage module, and the data storage module is used for storing data output by a data output module. According to the open-type multi-source data collection system and method, comprehensive data source collection types are covered, current universal data collection modes are supported, dynamic extension of a data collection interface and simultaneous collection of multiple data sources are supported, and when a new data source has access to the system, data collection carried out on the new data source can be easily achieved only by configuring the data collection task information without the deployment of a collection agency and a data mode conversion tool.

Description

technical field [0001] The invention relates to the field of network security, in particular to an open multi-source data collection system and method. Background technique [0002] With the rapid development of computer technology and the advancement of informatization, network security issues have become increasingly prominent. In order to deal with network security threats, it is often necessary to deploy various security products, including vulnerability scanning systems, virus protection systems, Trojan horse monitoring systems, intrusion detection system, etc. Network administrators collect and correlate the log alarms generated by different security products to obtain potential security threats in the network. [0003] The log alarms generated by different security products are different in output type and data format. The network security management system needs to collect data from multi-distributed, heterogeneous and autonomous data sources, and maintain data inte...

AI Technical Summary

Problems solved by technology

However, it cannot cover all the current general data collection types, such as the way of socket communication data reception; moreover, the storage of collected data can only be defined by the user, and when the collected data is large and needs to be quickly retrieved, it cannot provide A Storage Interface Facilitating Massive Data Retrieval

[0004] To sum up, the existing multi-source data collection tools have the following deficiencies: (1) The data collection methods covered are not comprehensive, only focusing on the active collection of database and text data, ignoring the passive reception of communication data such as socket data; (2) Data interface expansion and maintenance are inconvenient. When accessing a new data interface, it is necessary to deploy a collection agent or use a data mode conversion tool; (3) The storage method cannot achieve efficient storage and retrieval of massive data

Images