Data extraction method and device

Abstract

The invention provides a data extraction method applied to a relational database. The method includes: according to range distribution of a certain field in a selected data table, dividing the data table into M data partitions, wherein the type of the field is a numeric type or a value of the field can be converted into a numerical value; computing weight of each data partition according to number of data lines of each data partition; allocating a thread count to each data partition according to the weight of each data partition; enabling the sum of the thread counts allocated to all the data partitions to be equal to a preset total thread count N, wherein M< / =N; opening N threads, and respectively performing data extraction on each data partition by adopting the corresponding number of threads according to the allocated thread counts. By dividing the data table into the data partitions and dynamically allocating the thread counts of each data partition, the problem of nonuniform allocation data of each thread is solved, and data extraction efficiency of the relational data is improved.

Description

technical field [0001] The invention relates to the field of data extraction, in particular to a data extraction method and device for a relational database. Background technique [0002] Data integration is the logical or physical concentration of data from different sources, formats, and characteristics, so as to provide comprehensive data sharing. It is an important part of enterprise business intelligence and data warehouse systems. ETL is the main solution for enterprise data integration. The three letters in ETL represent Extract, Transform, and Load, that is, extraction, conversion, and loading. Data extraction is the process of extracting data from a data source. In practical applications, relational databases are mostly used as data sources. [0003] The methods of extracting data from relational databases can be divided into methods such as directly exporting backup data and reading data through interfaces such as JDBC. Among them, the method of reading through...

AI Technical Summary

Problems solved by technology

However, if the method of extracting data through interfaces such as ODBC or JDBC does not use multi-threaded parallelism, the efficiency will be relatively low, especially in today's era of big data, it is often necessary to extract database tables with hundreds of millions of data

Multi-thread parallel data extraction needs to pre-segment the data in the data source. If the distribution of data entries allocated by each thread is uneven, the efficiency of multi-threading will be greatly reduced; but if you want the data allocated by each thread to be very uniform, It is necessary to calculate the detailed distribution of data in the data table, which requires a lot of database operations before extracting data, which affects the efficiency of data extraction

Images