Data extraction method and apparatus, computer storage medium and computer device

Abstract

The invention discloses a data extraction method and device, a computing device and a computer storage medium. The method comprises the following steps: when receiving the data inquiry request of thewhole table, judging whether to switch the inquiry mode according to the total number of rows and the average length of each row of the data table; If yes, the query method is switched from full tablequery method to paging query method; The corresponding data is extracted from the data source by using paging query and stored in the distributed buffer. Based on the scheme provided by the invention, by adopting the paging inquiry mode, the data amount of the data extracted each time can be reduced, thereby reducing the frequency of the occurrence of the GC situation, enabling the GC situation to reach a normal state, effectively controlling the memory overflow error to not occur again, improving the stability, and bringing a good use experience to the user.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data extraction method, device, computing equipment and computer storage medium. Background technique [0002] In distributed computing tasks, Mysql data extraction is limited by cluster machine authorization and needs to be run in Yarn-client mode. When the amount of data extracted by Mysql is large, the pressure on the Driver side is high, and frequent garbage collection (GC (Garbage Collection)) will occur. [0003] Among them, GC means: after extracting a piece of data, it will load the data into the memory, and then create an object, which has a life cycle. Since the memory is limited, when the program no longer needs to use an object, it needs to destroy the object and release the memory resources it occupies, so that this space can be reused, and GC will occur. Due to the large amount of data to be extracted, more objects will be created, and frequent G...

AI Technical Summary

Problems solved by technology

Due to the large amount of data to be extracted, more objects will be created, and frequent GC situations are likely to occur, resulting in slow running of the data extraction task. Most of the time, it is waiting for the GC time, and the data extraction task may fail. In this situation, the main reason for the failure of the data extraction task is memory overflow (commonly understood as insufficient memory)

Images