User behavior abnormality detection method under Hadoop cluster

A hadoop cluster and anomaly detection technology, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve the problem of low model training efficiency, achieve the effect of overcoming inapplicability and improving efficiency

Inactive Publication Date: 2017-09-29
UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF3 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to overcome existing technical deficiencies, provide a kind of user behavior anomaly detection method under Hadoop cluster, not only can solve the abnormal behavior monitoring problem for user accessing HDFS data under Hadoop cluster, but also to traditional principal component The analysis algorithm is processed in parallel to solve the problem of low model training efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • User behavior abnormality detection method under Hadoop cluster
  • User behavior abnormality detection method under Hadoop cluster
  • User behavior abnormality detection method under Hadoop cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0037] like figure 1 As shown, a method for detecting abnormal user behavior under a Hadoop cluster includes the following steps:

[0038] S1: User behavior data collection, Hadoop integrates Apache's open source project Log4j by default, and obtains the HDFS audit log from the cluster NameNode through the Log4j log management service and stores it in the database;

[0039] S2: Data preprocessing. Read audit records from the database, and for each user's audit records, based on a time window, count the number of times each file operation command appears in that time, and combine them to form a feature vector, which is represented by x=(x 1 ,x 2 ,…,x 13 ) to indicate that there are 13 kinds of file operation commands in total, and the value of each dimen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a user behavior abnormality detection method under a Hadoop cluster. According to the method, user behavior data is acquired and analyzed and is recorded in a form of logs, feature vectors are formed according to behavior features of users, a feature vector set is processed by utilizing a parallel principal component analysis algorithm, behavior modes of the users are acquired efficiently, and by comparing the behavior modes of the users with historical modes, the abnormal behavior problem generated when the users access a HDFS is found and simultaneously, the hidden security threat under the Hadoop cluster is found, so that an effect of ensuring security of the HDFS is achieved. According to the user behavior abnormality detection method disclosed by the invention, not only is effective monitoring established for the data access behaviors of the users, abnormal behaviors are timely found and data security of the Hadoop cluster is ensured, but also by the parallel principal component analysis algorithm, model training efficiency is improved, and the problem of low training efficiency of a conventional model is solved.

Description

technical field [0001] The invention relates to a method for detecting abnormal user behavior, in particular to a method for detecting abnormal user behavior based on a Hadoop cluster. Background technique [0002] In recent years, Hadoop platform, as an excellent distributed computing system, plays an increasingly important role in large-scale data processing in enterprises. However, since Hadoop did not consider security factors at the beginning of its development, although some security mechanisms were added later, Hadoop's security audit mechanism, access control mechanism, and identity authentication mechanism are passive static security technologies that cannot monitor user behavior. monitoring, which makes it vulnerable to hidden security attacks. For example, illegal users steal the accounts and passwords of legitimate users, and obtain relevant permissions to illegally access data; data leakage is likely to occur during malicious intrusion, maintenance, and media l...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/06G06F17/30
Inventor 郝玉洁钟德建王芷若崔建鹏陆文斌
Owner UNIV OF ELECTRONICS SCI & TECH OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap