Data cleaning method and device based on data popularity and storage medium

Abstract

The invention provides a data cleaning method and device based on data popularity and a storage medium. The method comprises the steps of collecting data popularity information from a target data platform; analyzing a plurality of data types in the data popularity information to obtain popularity of each data type; according to a preset popularity weight, carrying out popularity evaluation on eachdata type popularity to obtain evaluation information of the data type popularity; and determining a to-be-deleted data type according to a preset cleaning strategy and the evaluation information, and cleaning data corresponding to the to-be-deleted data type. According to the method and device, the collected data heat information can be analyzed to obtain the data type heat, the heat evaluationinformation of the data type heat is given based on the data heat evaluation model, the data type is automatically cleaned according to the evaluation information, manual processing is not needed, andthe cleaning efficiency and accuracy are improved.

Description

technical field [0001] The present invention mainly relates to the technical field of data cleaning, and in particular to a data cleaning method, device and storage medium based on data heat. Background technique [0002] With the development of big data platforms, large-scale data warehouses, data lakes and other large data centers are becoming more and more common. While data centers continue to accumulate data, they also bring pressure on storage and performance to ensure efficient operation of data centers and improve data quality. The value of central data poses new challenges to data operation and maintenance. Cleaning up low-value and low-hot data in a timely manner is an effective means to solve the above methods. However, the current evaluation methods for data popularity and value rely more on manual methods , low efficiency and high complexity, and often have the problem of low accuracy. Contents of the invention [0003] The technical problem to be solved by t...

AI Technical Summary

Problems solved by technology

[0002] With the development of big data platforms, large-scale data warehouses, data lakes and other large data centers are becoming more and more common. While data centers continue to accumulate data, they also bring pressure on storage and performance to ensure efficient operation of data centers and improve data quality. The value of central data poses new challenges to data operation and maintenance. Cleaning up low-value and low-hot data in a timely manner is an effective means to solve the above methods. However, the current evaluation methods for data popularity and value rely more on manual methods , low efficiency and high complexity, and often have the problem of low accuracy

Images