Mass data import method and device, computer equipment and storage medium

A data import and mass-volume technology, which is applied in the field of big data, can solve the problems of long time-consuming mass data, insufficient system memory, system crash, etc., and achieve the effect of improving data import efficiency, smooth operation, and smooth operation state

Active Publication Date: 2020-03-06
PING AN TECH (SHENZHEN) CO LTD
View PDF7 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the embodiments of the present application is to propose a method for importing large batches of data, aiming at solving the problem that the existing batch import method takes too long to import large batches of data, has low efficiency, easily causes insufficient system memory, and even causes the entire system to crash. question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass data import method and device, computer equipment and storage medium
  • Mass data import method and device, computer equipment and storage medium
  • Mass data import method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] figure 1 A flow chart of implementing the method for importing large batches of data according to Embodiment 1 of the present invention is shown, and for convenience of description, only parts related to the present invention are shown.

[0044] In step S101, after a data import request is received, data to be imported corresponding to the data import request is acquired.

[0045] In the embodiment of the present invention, the data import request carries the identifier of the data resource to be imported.

[0046] In the embodiment of the present invention, after the data import request is obtained, according to the above-mentioned identifier, the data resource corresponding to the identifier can be obtained as the data to be imported.

[0047] In step S102, current real-time memory information of the system is collected, and the real-time memory information at least includes physical memory size, real-time occupancy, and real-time occupancy ratio.

[0048] In the em...

Embodiment 2

[0063] figure 2 It shows the implementation flow chart of the large-volume data import method provided by Embodiment 2 of the present invention, and for the convenience of description, only the parts related to the present invention are shown.

[0064] Step S104 specifically includes step S201, step S202 and step S203.

[0065] In step S201, it is judged whether the optimal occupancy ratio is greater than the real-time occupancy ratio.

[0066] In the embodiment of the present invention, the real-time memory occupation ratio refers to the ratio of all memory occupied by the current process of the system.

[0067] In step S202, if the judgment result is yes, the optimal introduction amount D2 is expressed as:

[0068] D2=D*P0-D1;

[0069] In step S203, if the judgment result is no, the optimal introduction amount D2 is expressed as:

[0070] D2=D*[(1-P1)*P0].

[0071] Wherein, D is the size of the physical memory, D1 is the real-time occupancy, P1 is the real-time occupan...

Embodiment 3

[0080] image 3 It shows a flow chart of implementing the large-volume data import method provided by Embodiment 3 of the present invention, and for convenience of description, only the parts related to the present invention are shown.

[0081] After step S104, step S301, step S302, step S303, step S304 and step S305 are also included.

[0082] In step S301, a pre-import time is calculated according to the optimal import amount and the size of the data to be imported.

[0083] In the embodiment of the present invention, the pre-import time refers to the time required for the system to import the data to be imported according to the current optimal import amount. For example, if the size of the data to be imported is 512MB and the optimal import amount is 5.12MB, then the pre-import time is 100 seconds. It should be understood that the example of the pre-import time here is only for convenience of understanding and is not intended to limit the present invention.

[0084] As a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention belongs to the field of big data, and relates to a large-batch data import method, which comprises the following steps: after receiving a data import request, obtainingto-be-imported data corresponding to the data import request; acquiring current real-time memory information of a system, wherein the real-time memory information at least carries a physical memory size, a real-time occupation memory and a real-time occupation memory ratio; obtaining a preset optimal occupation ratio of the system; according to the current real-time memory information of the system, obtaining an optimal import amount corresponding to the optimal memory occupation rate through a preset rule; acquiring real-time import data corresponding to the optimal import quantity in batches from the to-be-imported data; and importing the real-time import data acquired each time into a system database in batches. The invention further provides a large-batch data import device, computerequipment and a storage medium. The import time is greatly shortened, the data import efficiency is improved, and meanwhile, the problem of system crash caused by insufficient memory is avoided.

Description

technical field [0001] The present application relates to the technical field of big data, and in particular to a large-volume data import method, device, computer equipment and storage medium. Background technique [0002] With the advancement of computer technology and network age, batch import technology has been widely used. Especially in some systems combined with traditional industries, there is a large amount of traditional industry data that needs to be stored electronically. At this time, related technologies of batch import are generally used. [0003] There is an existing batch import method. After receiving a data import request, the data is imported into the system database at one time, so as to realize the import of small batches of data below 128M. [0004] However, the traditional batch import method is generally not smart. When the data to be imported is large, the traditional batch import method will take a long time and the import efficiency is low; at th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F12/02
CPCG06F9/5016G06F12/0246Y02D10/00
Inventor 杨冬振刘飞
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap