Method and device for importing large data volume of text files into distributed database

Abstract

The invention relates to a method and a device for importing a large data volume of text files into a distributed database, which are characterized in that the method comprises the following steps of:inquiring and acquiring file import parameter configuration; dividing the text file with the large data volume into a plurality of file reading subareas in a data volume averaging manner; splitting all the file reading sub-regions by threads with the corresponding number, and respectively splitting data in each file reading sub-region into a plurality of sub-files with the same number; distributing all the sub-files obtained after splitting to a plurality of distributed APs; and sending a file import request to each distributed AP by using a multi-thread asynchronous outbound call, initiatinga corresponding number of threads by each distributed AP according to the number of the distributed sub-files to import the file, and importing the file into each sub-table according to the number ofthe split sub-files concurrently. In addition, the invention further relates to a device for importing the text files with the large data volume into the distributed database.

Description

technical field [0001] The invention relates to a method and a device for importing a text file with a large amount of data into a distributed database. Background technique [0002] Since the promulgation of the new rules on asset management, there are a large number of wealth management products that meet the requirements of net worth, which need to be transferred from the parent bank to the name of the wealth management subsidiary for operation. The data involved in the wealth management system include liquidation information, account information, customer information, share information, investment research information, net worth information, regulatory information, etc. In order to quickly and successfully migrate these data to the large asset management system, we will face the requirements of system processing capacity, hardware facilities, and normal business operations, as well as the pressure and challenges of data security, availability, and scalability. Therefore...

AI Technical Summary

Problems solved by technology

Therefore, data security, availability, and scalability are poor

At the same time, it may be necessary to move the imported data to the corresponding sub-database and sub-table in the future, which will cause a waste of database resources

[0006] In addition, in the traditional solution, if a text file with a large amount of data is imported into the database, it is likely to cause memory overflow on the server, or occupy the CPU for a long time, affecting other transaction requests; if multiple joint query requests occur in the system at the same time, it will There is a risk of crippling the system

[0007] In addition, the large amount of data in a single table in a single database will also affect the read and write efficiency of other tables on the database, thereby affecting the efficiency of system business processing, and it is not friendly to batch data processing. The number of concurrency is limited, and it is difficult to deal with the situation of relatively large concurrency

Images