HBase total data import method and device, computer equipment and storage medium

Abstract

The invention discloses an HBase total data import method and device, computer equipment and a storage medium, the method comprises the following steps: receiving a data import request, determining atarget data table corresponding to to-be-imported data in a target database according to the data import request, the target database comprising an HBase database, caching a historical data file corresponding to the target data table into a preset memory container, importing to-be-imported data into a target database, and deleting the historical data file from the memory container. According to the method, table cleaning operation does not need to be executed on the data table of the to-be-imported data, it is guaranteed that the data table always provides service for the outside online, no perception is given to user service, the problem of service interruption is avoided, meanwhile, merging operation in a database is reduced, IO pressure of a server is greatly reduced, and the problem ofwrite-in amplification is prevented; and service stability is ensured.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method, device, computer equipment and storage medium for importing full amount of HBase data. Background technique [0002] Currently, the offline data import function that comes with the open source HBase provides two different data import methods. The first is to directly import offline data, and the second is to import offline data after performing table cleaning operations. [0003] Among them, the first data import method, when there is a large amount of historical data in the data table to be imported or the full amount of offline data is frequently imported, will cause the server to perform data merging operations, which will lead to server write amplification problems and seriously affect The stability of the HBase cluster; the second data import method needs to import offline data after the data table is cleared, which will lead to the unavailability of the da...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problems of the prior art, the embodiment of the present invention provides a full data import method, device, computer equipment and storage medium of HBase, so as to overcome the problems in the prior art of importing offline data after the data table is cleared. The resulting unavailability of the data table during the table clearing period, and as the historical data of the table increases, the service interruption time will also become longer, etc.

Images