Hadoop malicious node detection system based on network behavior analysis

Abstract

The invention provides a Hadoop malicious node detection system based on network behavior analysis for protecting the internal security of a cluster in view of a present situation that most malicious programs have certain network behaviors and even permeate inside the cluster to perform malicious behaviors. The Hadoop malicious node detection system comprises a network behavior monitoring module, a node log analyzing module, a node load analyzing module, and a training evaluation module malicious detection module. The network behavior monitoring module, the node log analyzing module, and the node load analyzing module operate on each node to acquire, monitor and preliminarily analyze information. The training evaluation module malicious detection module operates in an analysis host, performs model training and malicious detection after receiving information acquired by each node, and updates and stores the model regularly.

Description

Technical field [0001] The invention relates to the field of network behaviors, in particular to a Hadoop malicious node detection system based on network behavior analysis. Background technique [0002] Based on malicious analysis of network behavior, first establish a monitoring module in the cluster to monitor the behavior of each node, and set the monitoring center to be responsible for recording the key behaviors of these nodes, and use these behaviors to train the evaluation model. Through continuous iterative training, Maintain a good evaluation model to evaluate the status of nodes in the cluster in real time. In a cluster, if a node is attacked and a malicious program is run, a large number of behaviors different from other nodes will be generated, which is the core condition of judgment. [0003] At present, the malicious program monitoring technology based on network behavior analysis has had many phased results, such as NICTER system, TrumanBox system, AMCAS system, et...

AI Technical Summary

Problems solved by technology

The main idea of ​​these systems is almost based on network analysis, constructing a virtual network to trigger malicious programs, thereby locating malicious programs. However, there are relatively few researches on the detection of malicious nodes inside Hadoop. The current security assurance framework with relatively complete services mainly includes SecureMR. VIFA, etc., but all have certain limitations

[0004] 1) SecureMR: Secure MapReduce is a framework for enhancing and improving the mapping and simplification work in the MapReduce computing model, adding the Secure Committer and Secure Verifier modules, and designing and implementing the communication protocols of these modules, which can ensure non-collusion work The correctness and mapping of node calculation results, and the security of simplified nodes, but the framework cannot effectively detect the malicious behavior of a group of colluding malicious nodes.

[0005] 2) VIFA: Verification-based Integrity Assurance Framework. A verification-based service integrity assurance framework, which introduces high-level security and credible verification work nodes in cloud computing, and assumes that the simplified work nodes are credible. The calculation results of the mapping working nodes are verified by iH, and all tasks are copied and executed, and the calculation model of "reputation value" is introduced to effectively detect non-colluding and colluding malicious nodes, but each mapping task is Assigning two working nodes for repeated calculation will seriously affect the task processing performance of the cloud computing system; the verification node is a high-cost computing resource, which should be used more reasonably and efficiently. The calculation results are verified, and no caching mechanism is introduced to cache popular tasks, which will cause waste of resources for verification nodes

[0010] At present, the technology in this field lacks a comprehensive consideration of the tasks of the nodes in the cluster. For some Hadoop nodes, if they perform common tasks, their behavior will be very similar. However, if they perform different tasks, large differences are acceptable. , so when analyzing the malicious behavior of a node, the current task of the node must be taken into account, that is, we can start with the system log and analyze it in combination with network behavior

Images