Million-level excel data quick and stable import system

Abstract

The invention relates to the technical field of excel data, and discloses a million-level excel data quick and stable import system, which comprises the following steps: step 1, after starting to read data and load an excel file, adopting a line-by-line data reading mode to avoid overflowing of a memory of the system due to the fact that the data is completely loaded into the memory; quickly and stably importing the million-level excel data into the system; through using read-write separation, one thread queue being responsible for reading data; processing data, then putting into a data queue pool; meanwhile, the other thread queue being responsible for obtaining data from the data queue pool; writing data into a database by multiple threads simultaneously; no longer executing the two steps synchronously. The data are imported in batches without influencing the importing speed along with the increase of excel content, and meanwhile, the method supports multiple threads to write the data into a database, so that the importing speed is greatly increased, and the importing speed is always kept within a relatively constant speed range value.

Description

technical field [0001] The invention relates to the technical field of excel data, in particular to a system for quickly and stably importing million-level excel data. Background technique [0002] At present, most of the excel data import methods on the market use single-thread synchronous blocking to import data. When importing millions of excel data, you will find that the speed of importing data will be very slow. When importing excel data, often use Put all the data into the memory. If you import millions of excel data into the system, it is easy to cause system memory overflow, which cannot support the business needs of millions of data imports. When the business peak period, there will be a large number of Excel data is imported into the system, resulting in a geometric increase in the number of database write operations, which puts a huge pressure on the database, and may drag down the entire database in serious cases, thus affecting the normal operation of the busin...

AI Technical Summary

Problems solved by technology

[0002] At present, most of the excel data import methods on the market use single-thread synchronous blocking to import data. When importing millions of excel data, you will find that the speed of importing data will be very slow. When importing excel data, often use Put all the data into the memory. If you import millions of excel data into the system, it is easy to cause system memory overflow, which cannot support the business needs of millions of data imports. When the business peak period, there will be a large number of Excel data is imported into the system, resulting in a geometric increase in the number of database write operations, which puts a huge pressure on the database, and may drag down the entire database in severe cases, thus affecting the normal operation of the business

Images