Hive-based data aging method, device and apparatus

Abstract

The invention discloses a data aging method based on hive, which comprises the following steps: when receiving read-write access to data area of hive, refreshing access time record of data area corresponding to read-write access in metadata database of hive; Judging whether the preset scanning condition is satisfied at present; If the preset scanning condition is satisfied, the access time recordsof all the data areas of the hive are scanned, and the data area satisfying the preset aging condition is taken as the aging area, and the contents of the aging area are deleted. The invention can realize automatic scanning and aging judgment of hive data area and deletion of aging area, and has high efficiency, accuracy and reliability. Another object of the present invention is to provide an apparatus, device and computer-readable storage medium based on the method described above.

Description

technical field [0001] The invention relates to the technical field of data aging processing, in particular to a hive-based data aging method. The invention also relates to a hive-based data aging device, equipment and computer-readable storage medium. Background technique [0002] Hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a simple SQL query function, which can convert SQL statements into distributed computing tasks for execution. [0003] At present, when hive is used for big data processing, due to the huge storage capacity of big data itself, especially in many actual production systems, there is a large amount of new data every day, and the increasingly expanding data has a negative impact on the storage resources of the system. huge challenge. When the physical disk on which the distributed file system depends is full, not only the new data cannot be written, but also the calculation of the o...

AI Technical Summary

Problems solved by technology

When the physical disk on which the distributed file system depends is full, not only the new data cannot be written, but also the calculation of the original data needs to occupy a certain amount of disk space due to the problem of temporary file generation, which also leads to great impact. Impact

[0004] At present, when the storage space is full, if the hardware resources cannot be increased, the space can only be released by manually selecting the storage table for drop (a deletion operation), but for a distributed file system as a big data warehouse , Manually select which tables to drop from the massive data tables, not only the screening is more complicated, but also the data that is still in use may be deleted by mistake, the accuracy and reliability are low

Images