Service denial detection method for large data platform

Abstract

The invention discloses a service denial detection method for a large data platform. The method includes the steps that a plurality of nodes are selected in the large data platform and serve as detection nodes, and a plurality of nodes serve as calculation nodes; a user blacklist is set; secondly, a submitted application is firstly detected in the detection nodes; if any task is detected unqualified, the task and the application are both marked failed; if tasks are detected qualified, the application is allocated to the calculation nodes for continuous execution; thirdly, the nodes are detected, and when the number of the tasks marked failed on the nodes exceeds a set value P, the nodes are marked UNHEALTHY; when the number of the calculation nodes marked UNHEALTHY exceeds a set threshold, the tasks marked failed on the calculation nodes are detected, and if the tasks come from a same user or the similarity exceeds a set value, the calculation nodes are added into an exclusion list; if the ratio of the number of the nodes in the exclusion list exceeds N%, it is judged that attack of service denial exists.

Description

technical field [0001] The invention relates to the field of big data security, in particular to a denial of service detection method for a big data platform. Background technique [0002] In the context of the era of big data, its information security issues have extended to various fields, mainly including corporate data leakage, commercial espionage, privacy leakage, and hacker attacks. In the context of the gradual completion of the big data era, it can be said that the form of network information security is very severe. Whether you can protect your privacy, information security, and cluster availability has become the first problem in deploying big data. At present, common big data processing platforms such as Hadoop, Storm, Spark, etc., do not consider how to create a secure distributed computing environment during development. Since the clusters are deployed in a LAN protected by a firewall and only allow access by internal personnel of the company, the motivation ...

AI Technical Summary

Problems solved by technology

[0017] 2) The authorization and authentication mechanism only performs security verification at the system level, but it will not work for internal users or malicious users who enter the cluster through traditional attack means. Attackers can still use the node health check mechanism and the exclusion list mechanism to initiate rejection service attack

[0018] 3) The existing authorization authentication method is not enabled by default. It is generally believed that the cluster is deployed in a safe and reliable environment, so it has little effect on the security hardening of the cluster.

Images