A distributed full-flow analysis system based on cloud computing

Abstract

The present invention discloses a distributed full-flow analysis system based on cloud computing, including distributed collection probes, a management center based on edge cloud computing technology, a distributed flow storage retrieval balancer, a storage center, a distributed analysis center and a cloud platform orchestrator; the present invention can flexibly analyze, detect and process the entire network flow, and is convenient to deploy and maintain; it can perform high-priority key analysis on key users or application flows of concern, and can discover key user or key application risks earlier based on policies; it solves the problem that the local storage space of the collection probe is insufficient and difficult to expand, and can improve the expansion capability through cloud storage capabilities, with flexible deployment and convenient operation; the analysis and storage resources can be dynamically elastically scaled and intelligently scheduled, and can automatically expand the capacity to improve the detection capability during peak hours (such as daytime or weekdays), and automatically shrink during idle hours (such as night or holidays), thereby reducing the processing and analysis energy consumption and reducing the operating costs.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic analysis, and in particular to a distributed full-traffic analysis system based on cloud computing. Background Art

[0002] The full-flow analysis system collects and analyzes network traffic through physical probes, and collects traffic through switch mirroring or splitter deployment in bypass mode, providing full-flow storage and full-data analysis capabilities for network links. With full-flow storage analysis, you can perform multi-angle, all-round, and repeatedly traceable in-depth detection of attacks that have already occurred, including restoring the course of events, network threat analysis, file theft detection, etc., making it easier to detect potential intrusions and discover attacks that are missed by other tools. The general device deployment form is a stand-alone version or a distributed multi-probe collection storage device + management center mode.

[0003] The existing full-flow analysis system based on physical machines is oriented to cloud computing virtualization scenarios and has limited data collection capabilities for east-west flow analysis.

[0004] Due to limited storage space, physical machines are designed for storing large traffic data packets, and the monitoring and tracing time is limited. Depending on the size of the monitored traffic, they can generally only store 1-3 months of traffic data packets, and have no expansion capabilities.

[0005] The analysis capability is limited. Due to the limitation of the CPU capacity of the physical machine, the traffic detection and analysis capability is limited to a certain extent and cannot handle the analysis of ultra-large traffic data. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the defects of the background technology. The present invention provides a distributed full-flow analysis system based on cloud computing; it can flexibly analyze, detect and process the full network flow, perform high-priority key analysis on key users or application flow of concern, and discover key user or key application risks earlier based on strategies.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A distributed full-flow analysis system based on cloud computing, including distributed collection probes, a management center based on edge cloud computing technology, a distributed flow storage retrieval balancer, a storage center, a distributed analysis center and a cloud platform orchestrator;

[0009] The management center is used to manage registered acquisition probe devices, is responsible for global acquisition resource and analysis resource allocation and scheduling, and issues acquisition strategies and analysis strategies;

[0010] The distributed collection probe has two forms, physical machine and virtual machine, and collects traffic for physical environment and virtualized environment respectively. It has the ability to identify and classify DPI high-speed application traffic, and classifies or pre-filters the traffic according to the pre-traffic classification strategy issued by the management center. The physical collection probe is a hardware form deployed on the physical host, and collects traffic through physical switch mirroring traffic or splitter; the virtual collection probe is deployed on the virtual machine in the cloud computing environment, and collects traffic through the cloud platform virtual switch.

[0011] The distributed traffic storage retrieval balancer is used to collect the converged gateway resource pool of traffic, has the ability to elastically expand traffic collection access, and classifies and stores the collected traffic through the storage center;

[0012] The distributed analysis center is used to perform quasi-real-time or offline analysis on the stored data traffic pcap files, and has the capabilities of network attack, threat analysis, secret theft analysis, content analysis, etc., and reports the analysis results to the management center control brain;

[0013] The cloud platform orchestrator is used for virtual machine deployment control and elastic scaling management of computing, storage, and network resources.

[0014] Preferably, the method comprises the following steps:

[0015] Step 1: The collection probe registers the collection device with the management center, performs device authentication, and performs secure access;

[0016] Step 2: The management center allocates a distributed traffic storage retrieval balancer to the registered probes for establishing a collection channel. This step allocates traffic storage retrieval balancers according to the traffic storage retrieval balancer resource pool. If the resource pool is less than the available threshold or the workload of the distributed traffic storage retrieval balancer in operation is lower than the safety threshold, the management center applies for resources from the cloud platform, dynamically expands the processing resource pool through elastic scaling, and deploys new distributed traffic storage retrieval balancer VMs. If idle resources have not been used for a long time, the relevant VM resources are returned to the cloud platform.

[0017] Step 3: The distributed traffic storage retrieves the collection channel reported by the balancer and requests analysis resources from the management center;

[0018] Step 4: The management center pre-allocates the corresponding analysis resource information to the corresponding distributed traffic storage retrieval balancer, and the distributed traffic storage retrieval balancer and the analysis center establish a corresponding information notification channel; this step will allocate according to the analysis center's analysis engine resource pool. If the resource pool is less than the available threshold or the workload of the working analysis engine is lower than the safety threshold, the management center will apply for resources from the cloud platform, dynamically expand the processing resource pool, and deploy new analysis engine VMs; if idle resources have been unused for a long time, the relevant VM resources will be returned to the cloud platform;

[0019] Step 5: The distributed collection probe collects the corresponding traffic according to the pre-filtering strategy, and sends the traffic and traffic information to the distributed traffic storage retrieval balancer through the collection channel;

[0020] Step 6: The distributed traffic storage retrieval balancer receives the collected traffic and information and saves them to the storage center, establishes relevant retrieval indexes, and aggregates the analysis request information to notify the analysis center;

[0021] Step 7: The analysis center performs attack detection and content detection analysis on the stored traffic pcap file according to the analysis request, and reports the analysis results to the management.

[0022] Preferably, a quasi-real-time analysis mode is also included, and the specific steps are as follows:

[0023] Step 1: Distributed collection probes collect traffic and perform DPI analysis, classify packets, multiplex packets, and send traffic classification information and message data;

[0024] Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, creates a fast index, and constructs an analysis request to send to the scheduler;

[0025] Step 3: The scheduler schedules the analysis request according to the requested time. If the corresponding analysis engine resources exist, the scheduler sends the analysis request to the allocated analysis engine, carrying the traffic information and pcap file storage information. Otherwise, the scheduler requests the analysis resources from the management center, which triggers the allocation of the analysis resource pool or expands the resource pool.

[0026] Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

[0027] Preferably, an offline analysis mode processing flow is also included, and the specific steps are as follows:

[0028] Step 1: Distributed collection probes collect traffic and perform DPI analysis, classify packets, multiplex packets, and send traffic classification information and message data;

[0029] Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, classifies and stores the traffic based on the user information, creates a quick index, and stores the traffic in the database cache.

[0030] Step 3: The scheduler selects the incoming analysis request for scheduling according to the scheduling strategy. If there are corresponding analysis engine resources, the analysis request is sent to the allocated analysis engine, carrying traffic information and pcap file storage information. Otherwise, the analysis resource is requested from the management center, and the management center triggers the allocation of the analysis resource pool or expands the resource pool.

[0031] Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention provides a distributed full-flow analysis system based on cloud computing, which can flexibly analyze, detect and process the full flow of the network and is easy to deploy and maintain.

[0034] 2. The present invention provides a distributed full-flow analysis system based on cloud computing, which can perform high-priority analysis on key users or application traffic of concern, and can discover key user or key application risks earlier based on strategies.

[0035] 3. The distributed full-flow analysis system based on cloud computing provided by the present invention solves the problem of insufficient local storage space of collection probes and difficulty in expansion. It can improve expansion capabilities through cloud storage capabilities, and is flexible to deploy and easy to operate.

[0036] 4. The present invention provides a distributed full-flow analysis system based on cloud computing, in which analysis and storage resources can be dynamically and elastically scaled and intelligently scheduled. During peak hours (such as daytime or weekdays), capacity can be automatically expanded to improve detection capabilities, and during off-peak hours (such as night or holidays), capacity can be automatically contracted to reduce processing and analysis energy consumption and reduce operating costs.

[0037] 5. The present invention provides a distributed full-flow analysis system based on cloud computing, which focuses on ensuring flow collection and storage during peak periods and analysis and processing during trough periods, thereby achieving the effect of shaving peaks and flattening valleys. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 This is a diagram of the virtualized full-flow analysis system based on cloud computing architecture;

[0040] Figure 2 It is a general flow chart of the full flow analysis system based on cloud computing architecture;

[0041] Figure 3 Processing flow chart for quasi-real-time analysis mode;

[0042] Figure 4 Processing flow chart for offline analysis mode. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It mainly includes distributed virtualization / physical collection probes, a management center based on edge cloud computing technology, a distributed traffic storage retrieval balancer, a storage center, a distributed analysis center and other subsystems:

[0045] 1) Management center: manages registered collection probe devices, is responsible for the global collection resource and analysis resource allocation and scheduling, issues collection strategies and analysis strategies; and has the ability to visualize and trace traffic.

[0046] 2) Distributed collection probes, which are available in both physical and virtual machine forms, collect traffic for physical and virtual environments respectively. They have the ability to identify and classify DPI high-speed application traffic, and can classify or pre-filter traffic based on the pre-traffic classification strategy issued by the management center. Virtualization probes can be deployed based on physical hosts and divert traffic based on the virtual switches on the physical hosts.

[0047] 3) Traffic storage retrieval balancer, the aggregation gateway resource pool for collecting traffic, has the ability to elastically expand traffic collection access. The collected traffic can be classified and stored through the storage center, and an efficient index can be established.

[0048] 4) The analysis center performs quasi-real-time or offline analysis on the stored data traffic pcap files, has the capabilities of network attack, threat analysis, espionage analysis, content analysis, etc., and reports the analysis results to the management center.

[0049] 5) Cloud platform orchestrator, virtual machine deployment control, and elastic scaling management of computing, storage, and network resources.

[0050] Combination Figure 2 Introduce the general steps of the above-mentioned full-flow analysis system based on cloud computing architecture:

[0051] Step 1: The collection probe registers the collection device with the management center, performs device authentication, and performs secure access;

[0052] Step 2: The management center allocates traffic storage retrieval devices to the registered probes for establishing collection channels. This step will be allocated according to the traffic storage retrieval device resource pool. If the resource pool is less than the available threshold or the workload of the working traffic storage retrieval device is lower than the safety threshold, the management center will apply for resources from the cloud platform, elastically expand the processing resource pool, and deploy new traffic storage retrieval device VMs. If idle resources are not used for a long time, the relevant VM resources will be returned to the cloud platform to achieve energy saving and consumption reduction.

[0053] Step 3: The traffic storage retriever reports the established collection channel and requests analysis resources from the management center;

[0054] Step 4: The management center pre-allocates the corresponding analysis resource information to the corresponding traffic storage retriever, and the traffic storage retriever and the analysis center establish a corresponding information notification channel; this step will be allocated according to the analysis center's analysis engine resource pool. If the resource pool is less than the available threshold or the workload of the working analysis engine is lower than the safety threshold, the management center will apply for resources from the cloud platform, dynamically expand the processing resource pool, and deploy new analysis engine VMs. If idle resources have been unused for a long time, the relevant VM resources will be returned to the cloud platform to achieve the purpose of energy saving and consumption reduction.

[0055] Step 5. The collection probe collects the corresponding traffic according to the pre-filtering strategy, and sends the traffic and traffic information to the traffic storage retriever through the collection channel. The collection channel in this step can be a direct channel or a tunnel channel, which supports small packet multiplexing and can multiplex messages of different streams into a large data frame for packaged transmission, thereby improving the transmission bandwidth utilization.

[0056] Step 6: The traffic storage retriever receives the collected traffic and information and saves them to the storage center, establishes relevant search indexes, and aggregates the analysis request information to notify the analysis center;

[0057] Step 7: The analysis center performs attack detection, content detection and other analyses on the stored traffic pcap file according to the analysis request, and reports the analysis results to the management center / control brain.

[0058] The key processes of the full traffic analysis system solution based on cloud computing architecture are mainly divided into quasi-real-time analysis mode and offline analysis mode:

[0059] (1) Quasi-real-time analysis mode processing flow

[0060] See also Figure 3 , including the following steps:

[0061] Step 1: The collection probe collects traffic and performs DPI analysis, classifies messages, multiplexes packets, and sends traffic classification information and message data.

[0062] Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, creates a fast index, and constructs an analysis request to send to the scheduler.

[0063] Step 3: The scheduler schedules the analysis request according to the requested time. If the corresponding analysis engine resources exist, the scheduler sends the analysis request to the allocated analysis engine, carrying the traffic information and pcap file storage information. Otherwise, the scheduler requests the analysis resources from the management center, which triggers the allocation of the analysis resource pool or expands the resource pool.

[0064] Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

[0065] (2) Offline analysis mode processing flow

[0066] See also Figure 4 , including the following steps:

[0067] Step 1: The collection probe collects traffic and performs DPI analysis, classifies messages, multiplexes packets, and sends traffic classification information and message data.

[0068] Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, classifies and stores the traffic based on the user information, creates a quick index, and stores the traffic in the database cache.

[0069] Step 3: The scheduler selects the incoming analysis request for scheduling according to the scheduling strategy (application priority / user priority). If there are corresponding analysis engine resources, the analysis request is sent to the allocated analysis engine, carrying traffic information and pcap file storage information. Otherwise, the analysis resource is requested from the management center, and the management center triggers the allocation of the analysis resource pool or expands the resource pool.

[0070] Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed full-flow analysis system based on cloud computing, characterized in that: It includes distributed collection probes, a management center based on edge cloud computing technology, a distributed traffic storage retrieval balancer, a storage center, a distributed analysis center, and a cloud platform orchestrator; The management center is used to manage registered acquisition probe devices, is responsible for global acquisition resource and analysis resource allocation and scheduling, and issues acquisition strategies and analysis strategies; The distributed collection probe has two forms, physical machine and virtual machine, which collect traffic for physical environment and virtualized environment respectively, and has the ability to identify and classify DPI high-speed application traffic, and classify or pre-filter the traffic according to the pre-traffic classification strategy issued by the management center; Physical collection probes are hardware-based and deployed on physical hosts. They collect traffic through physical switch mirroring or splitters. Virtualized collection probes are deployed on virtual machines in a cloud computing environment and collect traffic through cloud platform virtual switches. The distributed traffic storage retrieval balancer is used to collect the converged gateway resource pool of traffic, has the ability to elastically expand traffic collection access, and classifies and stores the collected traffic through the storage center; The distributed analysis center is used to perform quasi-real-time or offline analysis on the stored data traffic pcap files, and has the capabilities of network attack, threat analysis, secret theft analysis, and content analysis, and reports the analysis results to the management center control brain; The cloud platform orchestrator is used for virtual machine deployment control and elastic scaling management of computing, storage, and network resources.

2. According to the distributed full-flow analysis system based on cloud computing of claim 1, it is characterized in that: The following steps are involved: Step 1: The collection probe registers the collection device with the management center, performs device authentication, and performs secure access; Step 2: The management center allocates a distributed traffic storage retrieval balancer to the registered probes for establishing a collection channel; Allocate resources based on the traffic storage and retrieval machine resource pool. If the resource pool is less than the available threshold or the workload of the distributed traffic storage and retrieval balancer in operation is lower than the safety threshold, the management center will apply for resources from the cloud platform, dynamically expand the processing resource pool through elastic scaling, and deploy new distributed traffic storage and retrieval balancer VMs. If idle resources are not used for a long time, the relevant VM resources will be returned to the cloud platform. Step 3: The distributed traffic storage retrieves the collection channel reported by the balancer and requests analysis resources from the management center; Step 4: The management center pre-allocates the corresponding analysis resource information to the corresponding distributed traffic storage retrieval balancer, and the distributed traffic storage retrieval balancer and the analysis center establish a corresponding information notification channel; the allocation is made according to the analysis center's analysis engine resource pool. If the resource pool is less than the available threshold or the workload of the working analysis engine is lower than the safety threshold, the management center applies for resources from the cloud platform, dynamically expands the processing resource pool, and deploys a new analysis engine VM; if idle resources have been unused for a long time, the relevant VM resources are returned to the cloud platform; Step 5: The distributed collection probe collects the corresponding traffic according to the pre-filtering strategy, and sends the traffic and traffic information to the distributed traffic storage retrieval balancer through the collection channel; Step 6: The distributed traffic storage retrieval balancer receives the collected traffic and information and saves them to the storage center, establishes relevant retrieval indexes, and aggregates the analysis request information to notify the analysis center; Step 7: The analysis center performs attack detection and content detection analysis on the stored traffic pcap file according to the analysis request, and reports the analysis results to the management.

3. A distributed full-flow analysis system based on cloud computing according to claim 2, characterized in that: The distributed full-flow analysis system based on cloud computing also includes a quasi-real-time analysis mode. The specific steps are as follows: Step 1: Distributed collection probes collect traffic and perform DPI analysis, classify packets, multiplex packets, and send traffic classification information and packet data; Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, creates a fast index, and constructs an analysis request to send to the scheduler; Step 3: The scheduler schedules the analysis request according to the requested time. If the corresponding analysis engine resources exist, the scheduler sends the analysis request to the allocated analysis engine, carrying the traffic information and pcap file storage information. Otherwise, the scheduler requests the analysis resources from the management center, which triggers the allocation of the analysis resource pool or expands the resource pool. Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

4. The distributed full-flow analysis system based on cloud computing according to claim 2 is characterized in that: The distributed full-flow analysis system based on cloud computing also includes an offline analysis mode processing flow, the specific steps are as follows: Step 1: Distributed collection probes collect traffic and perform DPI analysis, classify packets, multiplex packets, and send traffic classification information and packet data; Step 2: The traffic storage and retrieval component receives the collected traffic, performs storage processing based on the flow information, classifies and stores the traffic based on the user information, creates a fast index, and stores the traffic in the cache. Step 3: The scheduler selects the incoming analysis request for scheduling according to the scheduling strategy. If there are corresponding analysis engine resources, the analysis request is sent to the allocated analysis engine, carrying traffic information and pcap file storage information. Otherwise, the analysis resource is requested from the management center, and the management center triggers the allocation of the analysis resource pool or expands the resource pool. Step 4: The analysis engine receives the analysis request, performs in-depth detection and analysis based on the traffic information and pcap file storage information, and feeds the results back to the storage retriever and reports to the management center.

Images