Method for loading data in parallel after splitting files

A post-splitting, file technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of large amount of data, large amount of information, low efficiency, etc., so as to weaken low efficiency and improve performance. Effect

Inactive Publication Date: 2013-05-01
BANK OF CHINA
View PDF1 Cites 14 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the existing technology, although the technical application of SQL Loader is in the ascendant, it is limited by the characteristics of massive data itself, such as large data volume, large amount of information, and real-time generation, so the efficiency of simple SQL Loader loading data is inevitably low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for loading data in parallel after splitting files
  • Method for loading data in parallel after splitting files
  • Method for loading data in parallel after splitting files

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] The method for parallel loading data into a database after splitting a file according to an embodiment of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, but this is not intended to limit the present invention.

[0022] Before describing the present invention, the SQL Loader and related technical applications will be described first. SQL Loader is a data loading tool provided by Oracle (such as Orancle8) and used under operating system commands, which can load data from external data files into Oracle database tables. SQL Loader needs to involve at least 2 core files: control file and data file. The control file is a text file that can be recognized by SQL Loader and written in a specific language. The control file notifies SQL Loader where to put the data, and describes various types of data that need to be loaded into the Oracle database. The data file is the data source in the co...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for loading data in parallel after splitting files, which is used for loading data by using an SQL (Structured Query Language) Loader. The method comprises the following steps of: reading control files and data files to generate log files, damaged files and waste files; configuring the parameters of the SQL Loader; judging whether the record number of files required to be loaded is larger than a record limit value of parameter configuration or not according to the configured parameters; sequentially splitting the files required to be loaded into n sub-files according to a sub-file record numerical value of the parameter configuration; and starting m sub-processes of the SQL Loader serving as a data loading tool to load the sub-files in parallel, wherein a plurality of sub-files are loaded in series in each sub-process. According to the method for loading data in parallel after splitting files disclosed by the invention, the loading performance of mass data is further optimized on the basis of the SQL Loader tool.

Description

technical field [0001] The invention relates to a performance solution for loading large amounts of data into a database in the field of financial industry, and in particular to a method for parallel loading data into a database after splitting a file. Background technique [0002] With the rapid development of information technology, the banking business field increasingly needs to solve the performance optimization problems brought about by loading and storing large amounts of data. The amount of data generated by these businesses in real time is very large. If the SQL statement is used to load the data in the application program, it will consume a lot of time, and the real-time data and system stability cannot be guaranteed. Therefore, when developing business systems with massive data, data loading tools are often used to load massive data. SQL Loader is an effective tool provided by Oracle to load the data in the file into the database. The improvement of the SQL Load...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 王党团徐胜平
Owner BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap