Quick importing method and device of big data, terminal equipment and storage medium

A big data and fast technology, applied in the field of big data processing, can solve the problems of low data import efficiency, achieve the effects of saving processing time, improving data export efficiency, and improving processing efficiency

Active Publication Date: 2018-08-07
PING AN TECH (SHENZHEN) CO LTD
View PDF2 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a method, device, terminal device, and storage medium for quickly importing big data, so a

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Quick importing method and device of big data, terminal equipment and storage medium
  • Quick importing method and device of big data, terminal equipment and storage medium
  • Quick importing method and device of big data, terminal equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] figure 1 A flow chart showing the method for quickly importing big data in this embodiment. The method for quickly importing big data is applied to a database management system, the database management system is connected to a big data management platform, and is used to quickly import large-scale data (ie, batch data) on the big data management platform into a relational database (such as traditional relational data such as MySQL, Oracle or Postgres) to improve the efficiency of batch data import. In this embodiment, the Oracle database management system is used as an example for description, which is used to quickly import batch data in the big data management platform into the Oracle database. Such as figure 1 As shown, the method for quickly importing big data includes the following steps:

[0027] S10: Obtain at least one original batch of data sent by the big data management platform, each original batch of data corresponds to a state parameter.

[0028] Raw b...

Embodiment 2

[0076] figure 2 A functional block diagram of a device for fast big data import corresponding to the method for fast big data import in Embodiment 1 is shown. Such as figure 2 As shown, the device for quickly importing big data includes an original batch data acquisition module 10 , a first storage module 20 , a pending batch data acquisition module 30 and a second storage module 40 . Among them, the implementation functions of the original batch data acquisition module 10, the first storage module 20, the batch data acquisition module 30 to be processed, the second storage module 40 and the state parameter modification module 50 correspond to the steps of the method for fast import of big data in the embodiment One-to-one correspondence, in order to avoid redundant description, this embodiment does not describe in detail one by one.

[0077] The original batch data acquisition module 10 is configured to acquire at least one original batch data sent by the big data managem...

Embodiment 3

[0096] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for quickly importing big data in Embodiment 1 is implemented. In order to avoid repetition, it is not repeated here repeat. Alternatively, when the computer program is executed by the processor, the functions of each module / unit in the device for rapid import of big data in Embodiment 2 are realized, and in order to avoid repetition, details are not repeated here.

[0097]It can be understood that the computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal and telecommunication signal, etc.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a quick importing method and device of big data, terminal equipment and a storage medium. The method comprises: acquiring at least one piece of original batch data sent by a big data management platform, wherein each piece of original batch data corresponds to one status parameter; storing every piece of original batch data in a pre-created temporary table, and storing thecorresponding status parameters in a status table; triggering a data monitoring command to scan the status parameters in the status table, and acquiring batch data to be processed which have the status parameters being in pending status; removing duplicates in the batch data to be processed so as to obtain effective batch data, and storing the effective batch data to a target table in a relationaldatabase. The method allows batch data to be imported to the relational database more efficiently.

Description

technical field [0001] The present invention relates to the field of big data processing, in particular to a method, device, terminal equipment and storage medium for quickly importing big data. Background technique [0002] In the Internet and financial industries, the amount of data processed by the big data management platform is large, and the disk space occupied by the data will reach 500G or even T level. In order to facilitate the analysis and processing of data, it is necessary to import the large amount of data in the big data management platform (hereinafter referred to as batch data) into a relational database (such as MySQL, Oracle or Postgres, etc.). In the current big data management platform, the Sqoop tool is usually used to import batch data into a relational database. Among them, the Sqoop tool is a tool used to transfer the data in Hadoop (that is, the distributed system infrastructure) and the relational database to each other, that is, the data in the r...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/2282G06F16/284G06F16/254
Inventor 石宇李云利张文君张小敏吴春海李辉陈晶
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap