Mass data import method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of big data, and relates to a large-batch data import method, which comprises the following steps: after receiving a data import request, obtainingto-be-imported data corresponding to the data import request; acquiring current real-time memory information of a system, wherein the real-time memory information at least carries a physical memory size, a real-time occupation memory and a real-time occupation memory ratio; obtaining a preset optimal occupation ratio of the system; according to the current real-time memory information of the system, obtaining an optimal import amount corresponding to the optimal memory occupation rate through a preset rule; acquiring real-time import data corresponding to the optimal import quantity in batches from the to-be-imported data; and importing the real-time import data acquired each time into a system database in batches. The invention further provides a large-batch data import device, computerequipment and a storage medium. The import time is greatly shortened, the data import efficiency is improved, and meanwhile, the problem of system crash caused by insufficient memory is avoided.

Description

technical field [0001] The present application relates to the technical field of big data, and in particular to a large-volume data import method, device, computer equipment and storage medium. Background technique [0002] With the advancement of computer technology and network age, batch import technology has been widely used. Especially in some systems combined with traditional industries, there is a large amount of traditional industry data that needs to be stored electronically. At this time, related technologies of batch import are generally used. [0003] There is an existing batch import method. After receiving a data import request, the data is imported into the system database at one time, so as to realize the import of small batches of data below 128M. [0004] However, the traditional batch import method is generally not smart. When the data to be imported is large, the traditional batch import method will take a long time and the import efficiency is low; at th...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the embodiments of the present application is to propose a method for importing large batches of data, aiming at solving the problem that the existing batch import method takes too long to import large batches of data, has low efficiency, easily causes insufficient system memory, and even causes the entire system to crash. question

Images