Network security data management system

Abstract

The invention provides a network security data management system, and belongs to the technical field of network security. In order to solve the problems that an existing network security data processing mode is not comprehensive enough in security threat detection, not systematic enough in analysis and not rapid enough in response, the invention provides a network security data management system. Comprising a data acquisition module, a data processing module, a data storage module, a data analysis and calculation module, an application service module and a function presentation module, and the analysis and calculation module comprises three self-research technical components: HQL lite, ASE rules and Incident Cognition Engine. The system can quickly collect massive alarm logs to perform centralized correlation analysis and provide an analysis result display interface, so that the detection, analysis and response capabilities to security threats are improved, and the security of the system is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network security data management system Background Art

[0002] With the acceleration of the digitalization process, the boundaries of network security are constantly expanding, and the risks are also amplified. The existing security alert systems for traffic and boundary products based on the post-signature mechanism have insufficient correlation analysis capabilities when dealing with new types of network attacks and cannot form an effective collaborative defense. In addition, the traditional security incident handling methods rely too much on personal capabilities, the research and judgment and decision-making processes are slow, and the response to security threats is not fast enough. How to develop a collaborative and efficient network security data processing method remains an unsolved problem.

[0003] Therefore, the present invention urgently needs to provide a solution to improve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide a network security data management system, which can solve the problems that the existing network security data processing methods are not comprehensive enough in detecting security threats, not systematic enough in analysis, and not fast enough in response.

[0005] A network security data management system provided by the present invention includes a data collection module, a data processing module, a data storage module, a data analysis and calculation module, an application service module, and a function presentation module. The analysis and calculation module includes three self-developed technical components: HQL lite, ASErules, and Incident Cognition Engine.

[0006] A network security data management system provided by the present invention can quickly and uniformly collect a large amount of alarm logs through the use of various data processing, analysis, and visualization technologies for centralized correlation analysis and provide an analysis result display interface, improving the detection, analysis, and response capabilities for security threats, thereby enhancing the security of the system.

[0007] Optionally, the data collection module includes a threat intelligence application programming interface, a log data interface, a traffic data interface, an asset data interface, and a custom application programming interface.

[0008] Optionally, the data processing module includes a data bus technical component and a data processing component.

[0009] Optionally, the data storage module includes a distributed retrieval system, a distributed storage system, a distributed message queue system, and a database management system.

[0010] Optionally, the data analysis and calculation module includes a database query tool, a word segmentation and sorting tool, a data visualization tool, a big data streaming correlation analysis component, a long-term event recognition engine, and a batch analysis component.

[0011] Optionally, the database query tool uses the self-developed component HQL lite.

[0012] Optionally, the big data streaming correlation analysis component uses the self-developed component ASErules.

[0013] Optionally, the long-term event recognition engine uses the self-developed component Incident Cognition Engine.

[0014] Optionally, the application service module includes a reverse proxy server, an interface service bus, and a server architecture mode.

[0015] Optionally, the function presentation module includes a front-end framework for building a user interface and a tool for simulating back-end interface data. Description of the Drawings

[0016] Figure 1 It is a schematic diagram of the network security data management system architecture Detailed Embodiments

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein shall have the ordinary meanings understood by those of ordinary skill in the art in the field to which the present invention pertains. The words such as "including" used herein are intended to mean that the elements or items appearing before this word cover the elements or items listed after this word and their equivalents, without excluding other elements or items.

[0018] As Figure 1 shown, the present invention provides a network security data management system, including a data collection module, a data processing module, a data storage module, a data analysis and calculation module, an application service module, and a function presentation module. The analysis and calculation module includes three self-developed technical components: HQL lite, ASErules, and Incident Cognition Engine.

[0019] Among them, the data acquisition module is used to access devices, security technology prevention systems, and business systems to collect various types of original security data, including Threat Intelligence (TI) Application Programming Interface (API), log data interfaces, traffic data interfaces, asset data interfaces, and custom application programming interfaces; the TI-API is used to connect to third-party online Internet threat intelligence; the log data interface collects log data through the following professional components and protocols: system log protocol, file transfer protocol, Java Database Connectivity interface, proxy program, Apache Kafka, and distributed storage system. These components and protocols are implemented through corresponding plugins or adapters to achieve seamless integration with the log data management system; the traffic data interface uses the form of acquisition probes to connect to mirror traffic acquisition to collect network traffic data, and can capture data packets passing through network devices such as switches and routers; asset data is connected to a third-party asset platform in the form of an API, supporting offline import; the custom API is used for users to connect to third-party platforms such as work order and operation and maintenance management platforms.

[0020] Among them, the data processing module uniformly processes the collected original security data for subsequent data operation processes, including data bus technology components and data processing components; the data bus technology components usually refer to the infrastructure responsible for data transmission and communication in the software architecture, which ensures that data can flow between different systems, services, or components. Apache Kafka is used to process the original data from the data acquisition module and output it to the data processing components; the data processing components adopt a standard ETL technology framework and implement the normalization of data formats and contents based on the security data baseline specification. ETL is the abbreviation of Extract, Transform, Load, and is a data processing process for data warehouses. In this framework, the original data is first extracted, then cleaned and transformed to meet the needs of subsequent processing, and finally loaded into the data storage module.

[0021] Among them, the data storage module uses distributed cluster deployment to achieve the storage and retrieval calculation of PB-level security data, meeting the requirements of large storage and high performance for actual security services, including a distributed retrieval system, a distributed storage system, a distributed message queue system, and a database management system; the distributed retrieval system stores data inspection, event / alarm data, and consists of the following four aspects:

[0022] User interface: Provides an interface for users to interact with the system, receives users' retrieval requests, and presents the retrieval results to users.

[0023] Index Server: Responsible for storing and managing index data, which are obtained after preprocessing documents and are used to quickly locate and retrieve relevant documents.

[0024] Data Server: Stores the original document data. When the Index Server finds the index of a relevant document, it fetches the corresponding original document from the Data Server.

[0025] Coordinator: Responsible for coordinating the work among various nodes, distributing the user's retrieval requests to appropriate Index Servers and Data Servers, and integrating the returned results.

[0026] The distributed storage system stores the original data, dispersing the data across multiple nodes and connecting these nodes through a network to jointly provide storage services. It consists of the following four aspects:

[0027] Data distribution: Adopts techniques such as data sharding and hashing to disperse the data across multiple nodes to achieve load balancing and improve storage performance.

[0028] Replica mechanism: Through data replication technology, replicates the data to multiple nodes to improve data reliability and availability.

[0029] Consistency protocol: Ensures data consistency among multiple replicas. Common protocols include Paxos, Raft, etc.

[0030] Cache mechanism: Utilizes cache technology to cache frequently accessed data in memory or high-speed storage devices to improve data access speed.

[0031] The distributed message queue system uses Apache Kafka for asynchronous data transmission. It allows different systems or components to communicate through message passing without direct interaction; the database management system uses MySQL to store the foreground table structure data.

[0032] Among them, the data analysis and calculation module performs real-time, offline, and interactive analysis on the data in the storage module, supporting the upper-layer scenario models and security analysis applications, including database query tools, word segmentation and sorting tools, data visualization tools, big data streaming correlation analysis components, long-cycle event recognition components, and batch analysis components;

[0033] The database query tool uses the self-developed component HQL lite for quickly querying raw logs, events, and alarms. HQLLite can retrieve data from Hive tables, supports basic operations such as SELECT, WHERE, GROUP BY, etc., and can perform filtering and aggregation calculations on data through simple statements, such as calculating averages, sums, etc. It can be integrated with other big data tools and platforms for convenient data processing and analysis.

[0034] The word segmentation and sorting tool decomposes the text into individual words and sorts these words to help quickly locate and respond to security events. It uses the Jieba Chinese word segmentation tool, which supports multiple word segmentation modes, such as the accurate mode, the full mode, and the search engine mode. It can be called through the Python language to perform word segmentation on the text and sort the word segmentation results in combination with the sorting function of Python; The data visualization tool uses Power BI to create rich interactive reports and dashboards to visually display data and provide decision-making support;

[0035] The big data streaming correlation analysis component uses the self-developed component ASErules for real-time online correlation analysis. It consumes the original events and asset information input from Apache Kafka and outputs real-time alarms for rule matching. It collects data related to events through various channels, including sensor data and log files, and then performs preprocessing operations such as cleaning, transformation, and feature extraction on the collected data for subsequent analysis and processing. Using machine learning algorithms and deep learning models, it analyzes and models the preprocessed data to identify event patterns, trends, and correlations. According to the results of cognitive analysis, corresponding decisions and response strategies are formulated; The long-cycle event recognition engine uses the self-developed component Incident CognitionEngine, which is a system that uses artificial intelligence and machine learning technologies to perceive, understand, analyze, and make decisions on various events and is used to analyze historical long-cycle data;

[0036] The batch analysis component is used to batch analyze historical data. It queries, filters, aggregates, and correlates a large amount of data through SQL language to achieve batch analysis of data..

[0037] Among them, the application service module obtains data from data sources such as databases, processes it, and provides it to users. For example, it retrieves data from the database according to the query conditions input by users and presents the results to users. It can also store the data input by users in the database. This module includes a reverse proxy server, an interface service bus, and a server architecture mode; The reverse proxy server uses Nginx to forward client requests to multiple backend servers, thus achieving load balancing and improving performance; The interface service bus uses API Garraway as the only entry point of the system. It provides a unified interface for clients to access backend services without directly interacting with these services; The service architecture publishes applications in a mode that combines microservices and web servers

[0038] Among them, the function presentation module provides normalized log, alarm data, and metric data services externally, including a front-end framework for building a user interface and a tool for simulating back-end interface data. The front-end framework uses Vue.js and AngularJS. Vue.js is a lightweight front-end JavaScript framework for developing single-page applications, and AngularJS is an open-source Web application framework maintained by Google for building large-scale, high-performance Web applications; the back-end server uses nginx to handle HTTP requests and forward them to the back-end application for processing; the tool for simulating back-end interface data uses mock, which is convenient for development and joint debugging.

[0039] Although the embodiments of the present invention have been described in detail above, it is obvious to those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes are all within the scope and spirit of the present invention described in the claims. Moreover, the present invention described herein can have other embodiments and can be implemented or realized in various ways.

Claims

1. A network security data management system, characterized in that: It includes data collection module, data processing module, data storage module, data analysis and calculation module, application service module and function presentation module. The analysis and calculation module contains three self-developed technical components: HQL lite, ASErules and Incident Cognition Engine.

2. A network security data management system according to claim 1, characterized in that: The data collection module includes a threat intelligence application programming interface, a log data interface, a traffic data interface, an asset data interface and a custom application programming interface.

3. A network security data management system according to claim 1, characterized in that: The data processing module includes a data bus technology component and a data processing component.

4. A network security data management system according to claim 1, characterized in that: The data storage module includes a distributed retrieval system, a distributed storage system, a distributed message queue system and a database management system.

5. A network security data management system according to claim 1, characterized in that: The data analysis and calculation module includes a database query tool, a word segmentation and sorting tool, a data visualization tool, a big data streaming association analysis component, a long-period event recognition engine, and a batch analysis component.

6. A network security data management system according to claim 5, characterized in that: The database query tool adopts the self-developed component HQL lite.

7. A network security data management system according to claim 5, characterized in that: The big data streaming correlation analysis component adopts the self-developed component ASErules.

8. A network security data management system according to claim 5, characterized in that: The long-period event recognition engine adopts the self-developed component Incident Cognition Engine.

9. A network security data management system according to claim 1, characterized in that: The application service module includes a reverse proxy server, an interface service bus and a server architecture model.

10. A network security data management system according to claim 1, characterized in that: The function presentation module includes a front-end framework for building a user interface and a tool for simulating back-end interface data.

Images