Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for quickly importing data, and computer storage medium

A data and fast technology, applied in the field of big data processing, can solve problems such as the inability to meet business needs, and achieve the effect of improving maintainability, improving capabilities, and reducing pressure on demand

Pending Publication Date: 2020-08-28
北京思特奇信息技术股份有限公司
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the past, this type of operation could analyze and process tens of millions of quantities overnight, but with the development of business, the response speed required by users must reach tens of millions of levels per hour, so the existing processing methods cannot meet the growing demand. Business needs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for quickly importing data, and computer storage medium
  • Method and system for quickly importing data, and computer storage medium
  • Method and system for quickly importing data, and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0036] Such as figure 1 and figure 2 As shown, a method for quickly importing data files includes the following steps,

[0037] S1, based on the distributed file processing mechanism, the data file to be imported is split into multiple split data files distributed on multiple host nodes;

[0038] S2, based on the parallel operation of multiple host nodes, importing the split data files distributed on each host node into the database;

[0039] S3, analyzing and processing all the split data files in the database based on multi-host and multi-thread deployment and running calculations, to generate multiple target sub-files;

[0040] S4, based on the distributed file processing mechanism, gradually merging all the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a method and system for quickly importing a data file and a computer storage medium, and the method comprises the steps: splitting a to-be-imported data file into a pluralityof split data files based on a distributed file processing mechanism; importing the split data files distributed on the host nodes into a database based on parallel operation of the host nodes; analyzing and processing all split data files in the database based on multi-host and multi-thread deployment and operation to generate a plurality of target sub-files; based on a distributed file processing mechanism, gradually combining all the target sub-files through layering and grading to generate a plurality of final files; and importing all the final files into a target database. According to the method, a distributed file, multi-host node, multi-process and high-concurrency processing mechanism is used for splitting big data, storing the big data into a database, analyzing and processing the big data, combining the big data and importing the big data, so that the big data can be imported quickly, and the background batch data importing speed is increased.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method, system and computer storage medium for fast data import. Background technique [0002] The promotion of new business is often accompanied by cost / benefit control, which involves the delineation of the user range, and how to quickly link up with the business after the user is delineated is currently a rule such as adding a target range to the business, but the premise is that it needs to be The scope given by the user can only be stored in the warehouse according to different marking marks for different businesses. In the past, this type of operation could analyze and process tens of millions of quantities overnight, but with the development of business, the response speed required by users must reach tens of millions of levels per hour, so the existing processing methods cannot meet the growing demand. Business needs. Contents of the invention [0003] The technic...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/25G06F16/27G06F9/50
CPCG06F16/25G06F16/27G06F9/5027G06F2209/5018
Inventor 周成刚
Owner 北京思特奇信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com