Method and system for quickly exporting batch data from database based on multi-concurrency technology

Abstract

The invention discloses a method and a system for quickly exporting batch data from a database based on a multi-concurrency technology, the system comprises a data exporting tool and a data exportingplug-in registered to a database server, and the data exporting plug-in comprises a file writing process and a plurality of data processing processes; the method comprises the following steps that a data export tool is responsible for reading configuration file connection and calling a data export plug-in to carry out export work; the process of receiving the connection by the data export plug-inis a file writing process, and the file writing process creates a plurality of data processing processes; each data processing process reads the data of the corresponding data block for tuple analysis; the parsed tuples are converted into data in a target format, and the data is stored in a shared memory queue; and the file writing process traverses all the shared memory queues, takes out target format data from the shared memory queues and writes the target format data into target data files in batches. The method has the beneficial effects that the expansibility is enhanced, the coupling with the database server is reduced, and the export efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of databases, in particular to a method and system for quickly exporting batch data from a database based on multi-concurrency technology, which is suitable for batch data export. Background technique [0002] With the rapid development of computer technology, the application of database is becoming more and more extensive. The efficiency of data import and export is a very important indicator of the overall performance of the database. Data export is mainly to export the data of the database into CSV, binary and text file formats, and also export the data to EXCEL or other databases. [0003] Currently, the main techniques for exporting data from a database are as follows: [0004] One is to connect to the database by implementing an external tool, store the exported data in memory and convert it into a target format according to the set data export configuration, and finally save it as a target file. For...

AI Technical Summary

Problems solved by technology

[0008] Aiming at the above-mentioned technical problems in the related art, the present invention proposes a method and system for quickly exporting batch data from a database based on multi-concurrency technology, which can solve the problem of low export speed or dependence on the framework of the database, high coupling, and lack of correlation when exporting. Exception handling, low reliability, etc.

Images