Big data platform junk task collection system and method, and computer system

Abstract

The invention discloses a big data platform junk task collection system and a method, and a computer system, and the collection system comprises an audit log loading module which is used for loading an audit log generated by a big data platform in a data processing process; a data processing module which is connected with the auditing log loading module and is used for carrying out data processingon the auditing log; a temporary table registration module which is connected with the data processing module and is used for registering the audit log subjected to data processing as a correspondingtemporary table and storing the temporary table; a data grouping and aggregating module which is connected with the temporary table registering module and is used for grouping and aggregating the data in the temporary table to obtain a grouping and aggregating result; and a data storage module which is connected with the data grouping and aggregating module and is used for storing the grouping and aggregating result; according to the big data platform junk task processing system and method, junk tasks executed by a big data platform can be automatically recognized, storage resources are released, and the calculation cost is reduced.

Description

technical field [0001] The present invention specifically relates to the field of computer technology, in particular to a garbage task collection system and method for a big data platform, and a computer system. Background technique [0002] The cluster machines of the big data platform are huge in scale, and millions of tasks are executed every day, and it shows a trend of continuous increase. With the rapid development of business, iterative updates of products, and continuous adjustment of data structures, some of the millions of tasks have become garbage tasks with no execution value. However, if these garbage tasks are not monitored in a timely manner, the garbage tasks will still take up computing power. And storage resources, how to identify junk tasks among millions of tasks to reduce storage and computing costs has become an urgent technical problem to be solved. Contents of the invention [0003] The object of the present invention is to provide a large data pla...

AI Technical Summary

Problems solved by technology

With the rapid development of business, iterative updates of products, and continuous adjustment of data structures, some of the millions of tasks have become garbage tasks with no execution value. However, if these garbage tasks are not monitored in a timely manner, the garbage tasks will still take up computing power. and storage resources, how to identify junk tasks among millions of tasks to reduce storage and computing costs has become an urgent technical problem to be solved

Images