Method and system for quickly importing data, and computer storage medium

A data and fast technology, applied in the field of big data processing, can solve problems such as the inability to meet business needs, and achieve the effect of improving maintainability, improving capabilities, and reducing pressure on demand

Pending Publication Date: 2020-08-28
北京思特奇信息技术股份有限公司
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the past, this type of operation could analyze and process tens of millions of quantities overnight, but with the development of business, the response

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for quickly importing data, and computer storage medium
  • Method and system for quickly importing data, and computer storage medium
  • Method and system for quickly importing data, and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0035] The principles and features of the present invention will be described below with reference to the accompanying drawings. The examples cited are only used to explain the present invention, and are not used to limit the scope of the present invention.

[0036] Such as figure 1 with figure 2 As shown, a method for quickly importing data files includes the following steps:

[0037] S1, based on the distributed file processing mechanism, split the data file to be imported into multiple split data files distributed on multi-host nodes;

[0038] S2: Import the split data files distributed on each host node into the database based on the parallel operation of multiple host nodes;

[0039] S3: Analyze and process all the split data files in the database based on multi-host and multi-thread deployment and running operations to generate multiple target sub-files;

[0040] S4, based on the distributed file processing mechanism, gradually merge all the target sub-files through hierarchical ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method and system for quickly importing a data file and a computer storage medium, and the method comprises the steps: splitting a to-be-imported data file into a pluralityof split data files based on a distributed file processing mechanism; importing the split data files distributed on the host nodes into a database based on parallel operation of the host nodes; analyzing and processing all split data files in the database based on multi-host and multi-thread deployment and operation to generate a plurality of target sub-files; based on a distributed file processing mechanism, gradually combining all the target sub-files through layering and grading to generate a plurality of final files; and importing all the final files into a target database. According to the method, a distributed file, multi-host node, multi-process and high-concurrency processing mechanism is used for splitting big data, storing the big data into a database, analyzing and processing the big data, combining the big data and importing the big data, so that the big data can be imported quickly, and the background batch data importing speed is increased.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method, system and computer storage medium for fast data import. Background technique [0002] The promotion of new business is often accompanied by cost / benefit control, which involves the delineation of the user range, and how to quickly link up with the business after the user is delineated is currently a rule such as adding a target range to the business, but the premise is that it needs to be The scope given by the user can only be stored in the warehouse according to different marking marks for different businesses. In the past, this type of operation could analyze and process tens of millions of quantities overnight, but with the development of business, the response speed required by users must reach tens of millions of levels per hour, so the existing processing methods cannot meet the growing demand. Business needs. Contents of the invention [0003] The technic...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F16/27G06F9/50
CPCG06F16/25G06F16/27G06F9/5027G06F2209/5018
Inventor 周成刚
Owner 北京思特奇信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap