Data processing method and device

Abstract

The invention provides a data processing method and device. According to the method, after to-be-processed data is obtained, feature data of the to-be-processed data is extracted, wherein the featuredata comprises at least one of features of the to-be-processed data in a time dimension or a data access frequency dimension; and based on the feature data, a pre-trained data processing model is utilized to process the feature data so as to determine the life cycle state of the to-be-processed data. According to the scheme, the pre-trained data processing model is utilized to determine the life cycle state of the to-be-processed data based on at least one of the features of the to-be-processed data in the time dimension or the data access frequency dimension, therefore, when the life cycle state of the to-be-processed data is determined, the law of the corresponding relation between at least one of the features of big data in the time dimension or the data access frequency dimension and the life cycle state of the big data is utilized specifically. Compared with the prior art, labor cost is lowered, and the determined data life cycle state can have high accuracy.

Description

technical field [0001] The invention belongs to the field of machine learning, and in particular relates to a data processing method and device. Background technique [0002] In a production environment, it is often necessary to delete data whose life cycle has ended, such as tables and log files, to free up storage space and ensure the normal operation of the system. It is necessary to determine whether the life cycle of the data has ended. [0003] There are currently two mainstream methods for determining whether the data life cycle is over. One is to pre-set a threshold T, compare the total survival time t of the data from the moment of generation with the set threshold T, and set t>T The data is considered as the data whose life cycle has ended and deleted, otherwise, the life cycle of the data is not considered to be ended; the other is to manually review whether the data is useful, and the data that is useless after manual review is considered as the data whose lif...

AI Technical Summary

Problems solved by technology

[0004] Both of the above two methods have some defects. The threshold T in the first method is difficult to determine, which will lead to low accuracy in judging whether the data life cycle is over or not, which in turn will bring adverse effects on the data deletion operation. Among them, if If T is set too large, a lot of useless data may not be deleted. If T is set too small, useful data will often be deleted while deleting useless data. The second method requires manual review and judgment of each data. The problem of high labor costs

Images