Massive file import method, device, processor and electronic device

Abstract

The application discloses a large-capacity file import method and device, a processor and electronic equipment. The method is applied to the field of financial technology or other fields. The method comprises the following steps: determining a large-capacity file to be imported into a database; splitting the large-capacity file to obtain N split files; performing line-by-line analysis on each split file to obtain M data shards corresponding to each split file; processing each data shard according to a shard algorithm to obtain routing information of each data shard; and importing the N split files into the database according to the routing information of each data shard. Through the application, the problem that the efficiency of importing a large-capacity file is poor due to the overflow of the memory of a middleware caused by too many file lines when the large-capacity file is imported through the middleware in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology or other fields, and more specifically, to a method, apparatus, processor, and electronic device for importing large-capacity files. Background Technology

[0002] Currently, middleware-based distributed database systems typically import large amounts of data by writing the data into a large file, which is then imported as a file. When importing data from this large file into the database via middleware, the entire file is loaded into memory, and the data is read line by line. Each line is then routed and parsed, and the parsing results are sent to the corresponding database nodes for execution. However, this method creates a large number of connections between the middleware and the database within a short period, causing the middleware to consume excessive memory and potentially leading to memory overflow. This significantly degrades the performance of both the middleware and the database, indirectly impacting the normal operation of business processes.

[0003] There is currently no effective solution to the problem of poor import efficiency when importing large files via middleware due to excessive file lines causing memory overflow in the middleware. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, processor, and electronic device for importing large files, in order to solve the problem in the related art where the memory of the middleware overflows due to the excessive number of lines in the file when importing large files through middleware, resulting in poor efficiency in importing large files.

[0005] To achieve the above objectives, according to one aspect of this application, a method for importing large files is provided. This method is applied to middleware and includes: determining a large file to be imported into a database, wherein the database contains S database shards, where S is an integer greater than 1; splitting the large file into N split files, where N is an integer; parsing each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; processing each data shard according to a sharding algorithm to obtain routing information for each data shard; and importing the N split files into the database based on the routing information for each data shard.

[0006] Further, parsing each split file line by line to obtain the M data fragments corresponding to each split file includes: parsing each split file line by line to obtain M lines of data for each split file; determining the target field and determining the target field value corresponding to the target field in each line of data in each split file to obtain the target field value corresponding to each line of data in each split file; and determining the M data fragments corresponding to each split file based on the M lines of data in each split file and the target field value corresponding to each line of data in each split file.

[0007] Furthermore, processing each data shard according to the sharding algorithm to obtain the routing information of each data shard includes: determining the mapping relationship between each data shard and the S database shards to obtain mapping table information; calculating the target field value corresponding to each data shard using the sharding algorithm to obtain the calculation result corresponding to each data shard; and determining the routing information of each data shard based on the mapping table information and the calculation result corresponding to each data shard.

[0008] Further, determining the mapping relationship between each data shard and the S database shards to obtain the mapping table information includes: determining a first array and a second array based on the number of database shards S, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; calculating the number of logical partitions P based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between the data shards and the database shards, and P is a positive integer not greater than S; determining the partition information of each of the P logical partitions based on the first array and the second array; and determining the mapping table information based on the partition information of the P logical partitions.

[0009] Further, determining the routing information for each data shard based on the mapping table information and the calculation results corresponding to each data shard includes: determining the target logical partition corresponding to the data shard based on the calculation results corresponding to each data shard; determining the target partition information of the target logical partition in the mapping table information; determining the target database shard corresponding to the data shard based on the target partition information; and determining the routing information for the data shard based on the target database shard.

[0010] Further, importing the N split files into the database based on the routing information of each data shard includes: writing the M data shards into Q sub-files based on the routing information of the M data shards of each split file, where Q is a positive integer not greater than S; establishing Q links between the Q sub-files and the Q database shards of the database; and sending the Q sub-files to the Q database shards based on the Q links, so as to import each split file in the N split files into the database.

[0011] Furthermore, splitting the large file into N split files includes: determining the number of rows to be split in the large file; and splitting the large file into the N split files based on the number of rows to be split.

[0012] To achieve the above objectives, according to another aspect of this application, a large-capacity file import apparatus is provided. The apparatus includes: a determining unit, configured to determine a large-capacity file to be imported into a database, wherein the database contains S database shards, where S is an integer greater than 1; a splitting unit, configured to split the large-capacity file into N split files, where N is an integer; a parsing unit, configured to parse each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; a processing unit, configured to process each data shard according to a sharding algorithm to obtain routing information for each data shard; and an import unit, configured to import the N split files into the database according to the routing information of each data shard.

[0013] Further, the parsing unit includes: a parsing subunit, used to parse each split file line by line to obtain M lines of data in each split file; a first determining subunit, used to determine a target field and determine the target field value corresponding to the target field in each line of data in each split file to obtain the target field value corresponding to each line of data in each split file; and a second determining subunit, used to determine M data fragments corresponding to each split file based on the M lines of data in each split file and the target field value corresponding to each line of data in each split file.

[0014] Further, the processing unit includes: a third determining subunit, used to determine the mapping relationship between each data shard and the S database shards, and obtain mapping table information; a calculation subunit, used to calculate the target field value corresponding to each data shard using the sharding algorithm, and obtain the calculation result corresponding to each data shard; and a fourth determining subunit, used to determine the routing information of each data shard based on the mapping table information and the calculation result corresponding to each data shard.

[0015] Further, the third determining subunit includes: a first determining module, used to determine a first array and a second array based on the number S of database shards, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; a calculation module, used to calculate the number P of logical partitions based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between the data shards and the database shards, and P is a positive integer not greater than S; a second determining module, used to determine the partition information of each of the P logical partitions based on the first array and the second array; and a third determining module, used to determine the mapping table information based on the partition information of the P logical partitions.

[0016] Further, the fourth determining subunit includes: a fourth determining module, used to determine the target logical partition corresponding to each data shard based on the calculation result corresponding to each data shard; a fifth determining module, used to determine the target partition information of the target logical partition in the mapping table information; a sixth determining module, used to determine the target database shard corresponding to the data shard based on the target partition information; and a seventh determining module, used to determine the routing information of the data shard based on the target database shard.

[0017] Further, the import unit includes: a writing subunit, configured to write the M data fragments into Q sub-files based on the routing information of the M data fragments of each split file, where Q is a positive integer not greater than S; a creation subunit, configured to establish Q links between the Q sub-files and the Q database fragments of the database; and a sending subunit, configured to send the Q sub-files to the Q database fragments based on the Q links, so as to import each split file in the N split files into the database.

[0018] Furthermore, the splitting unit includes: a fifth determining subunit, used to determine the number of split lines of the large file; and a splitting subunit, used to split the large file according to the number of split lines to obtain the N split files.

[0019] To achieve the above objectives, according to one aspect of this application, a processor is provided for running a program, wherein the program executes the large file import method described in any of the above claims during runtime.

[0020] To achieve the above objectives, according to one aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the large file import method described in any of the above.

[0021] This application employs the following steps: determining a large file to be imported into a database, wherein the database contains S database shards, where S is an integer greater than 1; splitting the large file into N split files, where N is an integer; parsing each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; processing each data shard according to a sharding algorithm to obtain routing information for each data shard; and importing the N split files into the database based on the routing information for each data shard. This solves the problem in related technologies where importing large files via middleware results in memory overflow due to excessive file lines, leading to poor import efficiency. The middleware splits large files into multiple smaller files, parses the routing information line by line in each smaller file to obtain the routing information for each data shard, and then imports the large files into the corresponding database shards in batches based on the routing information of each data shard. This avoids the memory overflow problem caused by the middleware reading large files line by line, and also reduces the database performance loss caused by frequently establishing connections between the middleware and the database. This improves the robustness of the database during the import of large files, thereby enhancing database stability and ensuring the normal operation of enterprise business. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of a large file import method according to Embodiment 1 of this application;

[0024] Figure 2 This is a schematic diagram of an optional large-capacity file import method provided in Embodiment 1 of this application. Figure 1 ;

[0025] Figure 3 This is a schematic diagram of an optional large-capacity file import method provided in Embodiment 1 of this application. Figure 2 ;

[0026] Figure 4 This is a schematic diagram of a large-capacity file import device according to Embodiment 2 of this application;

[0027] Figure 5 This is a schematic diagram of importing large files into an electronic device according to Embodiment 5 of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the processing method, apparatus, processor, and electronic device specified in this application can be used in the financial technology field to improve the stability and timeliness of importing large files. It can also be used in any field other than the financial technology field. The application fields of the processing method, apparatus, processor, and electronic device specified in this application are not limited.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, imported user information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, imported data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0034] A distributed database is a database management system that stores data across multiple physical computer nodes, communicating and coordinating them via a network. In a distributed database, data is divided into multiple parts and stored on different nodes. Each node can independently handle read and write operations and can communicate and synchronize with other nodes via network connections. This distributed architecture allows the database system to have better performance when processing large amounts of data, while also allowing the system's capacity to be expanded by adding nodes. Distributed databases also provide data redundancy and fault tolerance mechanisms; even if a node fails, the system can continue to operate, and data can be recovered using backup nodes. Furthermore, distributed databases provide support for distributed transaction processing and data consistency to ensure data integrity and consistency.

[0035] Example 1

[0036] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of a large file import method according to Embodiment 1 of this application, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S101: Determine the large file of the database to be imported, wherein the database contains S database shards, where S is an integer greater than 1.

[0038] In this first embodiment, a large file refers to a file containing a large amount of data that needs to be imported into the database. For example, a large file A contains 30,000 SQL statements for inserting data into the database (e.g., INSERT INTO table AVALUES(“1”, “Zhang San”, “22”)).

[0039] The database is a distributed database. In a distributed database, data is stored across multiple database instances, and a distributed database management system is used to manage and coordinate data access and operations between these instances, thereby improving the database's scalability and performance. Database instances can be relational databases that use a table structure to store data and support SQL for data querying and processing, such as MySQL, Oracle, and SQL Server.

[0040] Step S102: Split the large file to obtain N split files, where N is an integer.

[0041] In this embodiment, to avoid the problem of memory overflow caused by the middleware loading each line of data of a large file into memory, the large file needs to be split into N split files. Then, the middleware processes the N split files to import the large file into the database.

[0042] Step S103: Parse each split file line by line to obtain M data fragments corresponding to each split file, where M is an integer.

[0043] In this first embodiment, in order to import N split files into the database, each split file needs to be parsed line by line to obtain M data fragments corresponding to each split file.

[0044] Step S104: Process each data fragment according to the fragmentation algorithm to obtain the routing information of each data fragment.

[0045] In this first embodiment, the routing information of each data shard represents the database shard information of the database shard where the data shard is located after it is written to the distributed database.

[0046] Step S105: Import N split files into the database based on the routing information of each data shard.

[0047] In this first embodiment, in order to avoid the middleware establishing a large number of connections with the database when reading data line by line, which would affect the performance of the database and the middleware, each line of data in the split file can be processed to obtain the routing information of each data shard. Then, based on the routing information of each data shard, data with the same routing information can be imported into the database in batches.

[0048] Figure 2 This is a schematic diagram illustrating the process of importing large files into the database using middleware in this solution. For example... Figure 2 As shown, the LOAD DATA processor in the middleware front-end connection receives a large file (such as...) Figure 2 The LOAD DATA processor splits the large file (File) into N smaller files (e.g., File) to import into the database. Figure 2 File1 to FileN in the N split files). Then, the LOAD DATA processor parses each of the N split files line by line to obtain the split file (e.g., File1 to FileN). Figure 2 M data fragments (e.g., File1) in File1) Figure 2 File1-1 to File1-M in the database. Next, determine the storage location of each data shard in the database (i.e., the routing information mentioned above), that is, the database shard corresponding to each data shard (e.g., ...). Figure 2 The data shard File1-1 corresponds to the database shard MySQL-1. Finally, based on the routing information of each data shard, the large file File is imported into the MySQL database.

[0049] In summary, the large file import method provided in Embodiment 1 of this application determines a large file to be imported into the database, wherein the database contains S database shards, where S is an integer greater than 1; splits the large file into N split files, where N is an integer; parses each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; processes each data shard according to the sharding algorithm to obtain routing information for each data shard; and imports the N split files into the database based on the routing information for each data shard. This solves the problem in related technologies where importing large files through middleware results in memory overflow of the middleware due to excessive number of file lines, leading to poor efficiency in importing large files. The middleware splits large files into multiple smaller files, parses the routing information line by line in each smaller file to obtain the routing information for each data shard, and then imports the large files into the corresponding database shards in batches based on the routing information of each data shard. This avoids the memory overflow problem caused by the middleware reading large files line by line, and also reduces the database performance loss caused by frequently establishing connections between the middleware and the database. This improves the robustness of the database during the import of large files, thereby enhancing database stability and ensuring the normal operation of enterprise business.

[0050] Optionally, in the large-capacity file import method provided in Embodiment 1 of this application, parsing each split file line by line to obtain M data fragments corresponding to each split file includes: parsing each split file line by line to obtain M lines of data for each split file; determining the target field and determining the target field value corresponding to the target field in each line of data in each split file to obtain the target field value corresponding to each line of data in each split file; and determining the M data fragments corresponding to each split file based on the M lines of data in each split file and the target field value corresponding to each line of data in each split file.

[0051] Specifically, the target field refers to the field in the SQL statement used to import data into the database. For example, if table C in the database includes fields "ID", "Name", and "Age", then the "ID" field in table C can be used as the target field, and the value corresponding to the "ID" field in the SQL statement will be the target field value. It's important to note that when the same split file contains import SQL statements for multiple tables, the target field for each SQL statement can be determined based on the fields contained in the tables on which the SQL statement is applied.

[0052] After determining the target field, the target field value for each row of data can be determined based on the target field. Then, based on each row of data and its corresponding target field value, M data shards can be determined. For example, split file A and split file B contain SQL statements that insert data into table C. Split file A contains two SQL statements: SQL statement A1 "INSERT INTO table CVALUES("1", "Zhang San", "22")" and SQL statement A2 "INSERT INTO table C VALUES("2", "Li Si", "23")". If the "ID" field in table C is used as the target field, the target field value for SQL statement A1 is 1, and the target field value for SQL statement A2 is 2. Finally, based on SQL statement A1 "INSERT INTO table C VALUES("1", "Zhang San", "22")" and the target field value of SQL statement A1 (1), a data shard A1 for split file A is determined.

[0053] In this first embodiment, in order to import large files into the database in batches, each split file of the large file can be parsed line by line to obtain M lines of data in each split file and the target field value corresponding to each line of data. Then, the routing information of each line of data is determined according to each line of data and the target field value corresponding to that line of data. Finally, the large files are imported into the database in batches according to the routing information of each line of data in each split file.

[0054] By configuring the target field, the middleware can parse the split file line by line, determine the data shard corresponding to each line of data, and determine the routing information of each data shard. This facilitates the batch import of large files into the database based on the data shards of the split file, avoiding the memory overflow problem caused by the middleware needing to establish a connection with the database for each line of data imported, thus improving the stability of the database.

[0055] Optionally, in the large file import method provided in Embodiment 1 of this application, processing each data fragment according to the fragmentation algorithm to obtain the routing information of each data fragment includes: determining the mapping relationship between each data fragment and S database fragments to obtain mapping table information; calculating the target field value corresponding to each data fragment using the fragmentation algorithm to obtain the calculation result corresponding to each data fragment; and determining the routing information of each data fragment based on the mapping table information and the calculation result corresponding to each data fragment.

[0056] In this embodiment, the mapping table information refers to the mapping relationship between the target field value corresponding to each data shard and the database shard. Table 1 is an example of the mapping table information. The first row of data in Table 1 represents the calculation result obtained after calculating the target field value corresponding to the data shard using the sharding algorithm, and the second row of data in Table 1 represents the sequence number of the database shard.

[0057] The sharding algorithm is used to calculate the target field value corresponding to the data shard, and then determine the database shard to which the data shard is stored based on the calculation result, i.e., the routing information of each data shard. Specifically, a hash partitioning algorithm can be used to calculate the target field value corresponding to the data shard. For example, if the target field value corresponding to data shard A is 6, then the target field value 6 is modulo 11 (Table 1 contains 11 database shards), and the remainder is 6 (i.e., the calculation result above). The database shard corresponding to data shard A is determined based on the remainder 6.

[0058] Table 1 Mapping Table Information

[0059]

[0060] It's important to note that if a data shard must be written to a fixed database shard, the routing information for that data shard is directly determined to be the fixed database shard corresponding to it, without using a partitioning algorithm to calculate the target field value for that data shard. For example, since table B contains relatively little data, it only needs to be stored in database shard 0. When processing the import of data shard C from table B, the routing information for data shard C can be directly determined to be database shard 0.

[0061] By determining the mapping table information and using a sharding algorithm to calculate the target field value corresponding to each data shard, the database shard corresponding to each data shard can be determined based on the calculation results. This allows multiple database shards to execute data import requests in parallel, thereby improving the import efficiency of large files.

[0062] Optionally, in the large file import method provided in Embodiment 1 of this application, determining the mapping relationship between each data shard and S database shards to obtain mapping table information includes: determining a first array and a second array based on the number of database shards S, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; calculating the number of logical partitions P based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between data shards and database shards, and P is a positive integer not greater than S; determining the partition information of each of the P logical partitions based on the first array and the second array; and determining the mapping table information based on the partition information of the P logical partitions.

[0063] Specifically, Figure 3 This is a schematic diagram illustrating the process of determining the database shards corresponding to each data shard in this scheme. First, the first array is determined to be {1, 2} and the second array is determined to be {3, 4}. Then, the first and second arrays are multiplied by a dot to obtain the number of logical partitions P (such as the data in the first row of Table 1, where each data in the first row of Table 1 represents a logical partition), i.e., 1*2 + 3*4 = 11, and the number of partitions P is 11. Secondly, based on the first array {1, 2} and the second array {3, 4}, the database shard numbers corresponding to the 11 logical partitions are determined: Taking the first element of the first array and the first element of the second array, the database shard numbers corresponding to the first 3 logical partitions are determined, meaning each logical partition corresponds to one database shard number (e.g., logical partition 0 corresponds to database shard 0 in Table 1); taking the second element of the first array and the second element of the second array, the database shard numbers corresponding to the last 8 logical partitions are determined, meaning every two logical partitions correspond to one database shard number (e.g., logical partition 3 and logical partition 4 correspond to database shard 3 in Table 1). Finally, the mapping table information is determined based on the database shard numbers corresponding to the 11 logical partitions (as shown in Table 1).

[0064] In addition, when determining the first and second arrays, the elements and the number of elements in the first and second arrays can be flexibly determined based on the actual production situation, provided that the result of the dot product of the first and second arrays is less than the number of database shards.

[0065] By flexibly configuring the first and second arrays and determining the mapping table information based on the first and second arrays, each data shard is distributed as evenly as possible across all database shards in the database. This solves the problem that a single database shard may have poor data processing efficiency due to a large amount of data, and also helps to determine the routing information of the data shards.

[0066] Optionally, in the large file import method provided in Embodiment 1 of this application, determining the routing information of each data shard based on the mapping table information and the calculation results corresponding to each data shard includes: determining the target logical partition corresponding to the data shard based on the calculation results corresponding to each data shard; determining the target partition information of the target logical partition in the mapping table information; determining the target database shard corresponding to the data shard based on the target partition information; and determining the routing information of the data shard based on the target database shard.

[0067] Specifically, if the mapping table information is as shown in Table 1, the hash partitioning algorithm is used to calculate the target field value of 38 corresponding to data shard A, resulting in 5. Then, based on Table 1, the target logical partition corresponding to the calculation result 5 is determined to be logical partition 5 (as shown in the first row of data in Table 1). The database shard number corresponding to logical partition 5 is 4 (i.e., the target partition information mentioned above), meaning the database shard corresponding to data shard A is database shard 4 (i.e., the target database shard mentioned above). Finally, it is determined that data shard A should be stored in database shard 4 (i.e., the routing information mentioned above).

[0068] By determining the routing information for each data shard through mapping table information and calculation results corresponding to each data shard, multiple data shards of each split file in a large file can be imported into the database in batches, thereby improving the import efficiency of large files.

[0069] Optionally, in the large-capacity file import method provided in Embodiment 1 of this application, importing N split files into the database based on the routing information of each data fragment includes: writing M data fragments into Q sub-files based on the routing information of M data fragments of each split file, where Q is a positive integer not greater than S; establishing Q links between the Q sub-files and the Q database fragments of the database; and sending the Q sub-files to the Q database fragments based on the Q links, so as to import each split file in the N split files into the database.

[0070] In this first embodiment, to batch import N split files of a large file into the database, data shards with the same routing information in each split file can be written into a sub-file. The middleware establishes a connection between each sub-file and its corresponding database shard, and imports the N split files into the database in batches through these connections. By writing data shards with the same routing information in each split file into a sub-file, a connection can be established with the database based on the sub-file. This avoids the memory crash problem caused by the middleware establishing a large number of connections to the database when reading each row of data in the split file, thus improving the stability of the database.

[0071] Optionally, in the large file import method provided in Embodiment 1 of this application, splitting the large file to obtain N split files includes: determining the number of rows to be split in the large file; splitting the large file according to the number of rows to obtain N split files.

[0072] In this first embodiment, before splitting a large file, the number of rows to be split can be configured in the middleware's configuration file, and the middleware can be started according to this configuration file. When the middleware's LOAD DATA processor receives an import request for a large file to be imported into the database, the LOAD DATA processor will parse the statement. If the number of data rows in the large file exceeds the number of split rows, the file will be split into multiple files according to the number of split rows and saved. If the number of remaining data rows after the last split is less than the number of split rows, the remaining data can be written to the last split file, that is, it is not necessary to write the remaining data to a new split file.

[0073] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] Example 2

[0075] This application's second embodiment also provides a large-capacity file import device. It should be noted that the large-capacity file import device of this application's second embodiment can be used to execute the large-capacity file import method provided in this application's first embodiment. The large-capacity file import device provided in this application's second embodiment is described below.

[0076] Figure 4 This is a schematic diagram of a large-capacity file import device according to Embodiment 2 of this application. Figure 4 As shown, the device includes: a determining unit 401, a splitting unit 402, a parsing unit 403, a processing unit 404, and an importing unit 405.

[0077] Specifically, the determining unit 401 is used to determine the large file of the database to be imported, wherein the database contains S database shards, and S is an integer greater than 1.

[0078] Splitting unit 402 is used to split a large file into N split files, where N is an integer.

[0079] Parsing unit 403 is used to parse each split file line by line to obtain M data fragments corresponding to each split file, where M is an integer.

[0080] The processing unit 404 is used to process each data fragment according to the fragmentation algorithm to obtain the routing information of each data fragment.

[0081] Import unit 405 is used to import N split files into the database based on the routing information of each data shard.

[0082] The large file import device provided in Embodiment 2 of this application determines a large file to be imported into a database by a determining unit 401, wherein the database contains S database shards, where S is an integer greater than 1; a splitting unit 402 splits the large file into N split files, where N is an integer; a parsing unit 403 parses each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; a processing unit 404 processes each data shard according to a sharding algorithm to obtain routing information for each data shard; and an import unit 405 imports the N split files into the database according to the routing information for each data shard. This solves the problem in related technologies where importing large files through middleware results in memory overflow of the middleware due to excessive number of file lines, leading to poor efficiency in importing large files. The middleware splits large files into multiple smaller files, parses the routing information line by line in each smaller file to obtain the routing information for each data shard, and then imports the large files into the corresponding database shards in batches based on the routing information of each data shard. This avoids the memory overflow problem caused by the middleware reading large files line by line, and also reduces the database performance loss caused by frequently establishing connections between the middleware and the database. This improves the robustness of the database during the import of large files, thereby enhancing database stability and ensuring the normal operation of enterprise business.

[0083] Optionally, in the large-capacity file import device provided in Embodiment 2 of this application, the parsing unit 403 includes: a parsing subunit, used to parse each split file line by line to obtain M lines of data in each split file; a first determining subunit, used to determine a target field and determine the target field value corresponding to the target field in each line of data in each split file to obtain the target field value corresponding to each line of data in each split file; and a second determining subunit, used to determine M data fragments corresponding to each split file based on the M lines of data in each split file and the target field value corresponding to each line of data in each split file.

[0084] Optionally, in the large-capacity file import device provided in Embodiment 2 of this application, the processing unit 404 includes: a third determining subunit, used to determine the mapping relationship between each data shard and S database shards to obtain mapping table information; a calculation subunit, used to calculate the target field value corresponding to each data shard using a sharding algorithm to obtain the calculation result corresponding to each data shard; and a fourth determining subunit, used to determine the routing information of each data shard based on the mapping table information and the calculation result corresponding to each data shard.

[0085] Optionally, in the large-capacity file import device provided in Embodiment 2 of this application, the aforementioned third determining subunit includes: a first determining module, used to determine a first array and a second array based on the number S of database shards, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; a calculation module, used to calculate the first array and the second array to obtain the number P of logical partitions, wherein the logical partitions are used to determine the mapping relationship between data shards and database shards, and P is a positive integer not greater than S; a second determining module, used to determine the partition information of each of the P logical partitions based on the first array and the second array; and a third determining module, used to determine the mapping table information based on the partition information of the P logical partitions.

[0086] Optionally, in the large-capacity file import device provided in Embodiment 2 of this application, the aforementioned fourth determining subunit includes: a fourth determining module, used to determine the target logical partition corresponding to each data shard based on the calculation result corresponding to each data shard; a fifth determining module, used to determine the target partition information of the target logical partition in the mapping table information; a sixth determining module, used to determine the target database shard corresponding to the data shard based on the target partition information; and a seventh determining module, used to determine the routing information of the data shard based on the target database shard.

[0087] Optionally, in the large-capacity file import device provided in Embodiment 2 of this application, the import unit 405 includes: a writing subunit, used to write M data fragments into Q sub-files according to the routing information of M data fragments of each split file, where Q is a positive integer not greater than S; a creation subunit, used to establish Q links between the Q sub-files and the Q database fragments of the database; and a sending subunit, used to send the Q sub-files to the Q database fragments according to the Q links, so as to import each split file in the N split files into the database.

[0088] Optionally, in the large file import device provided in Embodiment 2 of this application, the splitting unit 402 includes: a fifth determining subunit, used to determine the number of split lines of the large file; and a splitting subunit, used to split the large file according to the number of split lines to obtain N split files.

[0089] The large file import device includes a processor and a memory. The aforementioned determination unit 401, splitting unit 402, parsing unit 403, processing unit 404, and import unit 405 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.

[0090] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency of importing large files.

[0091] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0092] Embodiment 3 of the present invention provides a computer-readable storage medium on which a program is stored, which, when executed by a processor, implements a method for importing large files.

[0093] Embodiment 4 of the present invention provides a processor for running a program, wherein the program executes a large file import method during runtime.

[0094] like Figure 5 As shown, Embodiment 5 of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: determining a large file to be imported into a database, wherein the database contains S database shards, where S is an integer greater than 1; splitting the large file to obtain N split files, where N is an integer; parsing each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; processing each data shard according to a sharding algorithm to obtain routing information for each data shard; and importing the N split files into the database according to the routing information for each data shard.

[0095] When the processor executes the program, it also performs the following steps: parsing each split file line by line to obtain M data fragments corresponding to each split file, including: parsing each split file line by line to obtain M lines of data for each split file; determining the target field and determining the target field value corresponding to the target field in each line of data in each split file to obtain the target field value corresponding to each line of data in each split file; and determining the M data fragments corresponding to each split file based on the M lines of data in each split file and the target field value corresponding to each line of data in each split file.

[0096] When the processor executes the program, it also performs the following steps: processing each data shard according to the sharding algorithm to obtain the routing information of each data shard, including: determining the mapping relationship between each data shard and S database shards to obtain mapping table information; calculating the target field value corresponding to each data shard using the sharding algorithm to obtain the calculation result corresponding to each data shard; and determining the routing information of each data shard based on the mapping table information and the calculation result corresponding to each data shard.

[0097] When the processor executes the program, it also performs the following steps: determining the mapping relationship between each data shard and S database shards, and obtaining the mapping table information, including: determining a first array and a second array based on the number of database shards S, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; calculating the number of logical partitions P based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between data shards and database shards, and P is a positive integer not greater than S; determining the partition information of each of the P logical partitions based on the first array and the second array; and determining the mapping table information based on the partition information of the P logical partitions.

[0098] When the processor executes the program, it also performs the following steps: determining the routing information of each data shard based on the mapping table information and the calculation results corresponding to each data shard, including: determining the target logical partition corresponding to the data shard based on the calculation results corresponding to each data shard; determining the target partition information of the target logical partition in the mapping table information; determining the target database shard corresponding to the data shard based on the target partition information; and determining the routing information of the data shard based on the target database shard.

[0099] When the processor executes the program, it also performs the following steps: importing N split files into the database based on the routing information of each data split, including: writing M data splits into Q sub-files based on the routing information of M data splits of each split file, where Q is a positive integer not greater than S; establishing Q links between the Q sub-files and the Q database splits; and sending the Q sub-files to the Q database splits based on the Q links, so as to import each split file in the N split files into the database.

[0100] When the processor executes the program, it also performs the following steps: splitting a large file into N split files, including: determining the number of lines to split the large file; splitting the large file into N split files based on the number of lines.

[0101] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0102] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: determining a large file to be imported into a database, wherein the database contains S database shards, where S is an integer greater than 1; splitting the large file into N split files, where N is an integer; parsing each split file line by line to obtain M data shards corresponding to each split file, where M is an integer; processing each data shard according to a sharding algorithm to obtain routing information for each data shard; and importing the N split files into the database according to the routing information for each data shard.

[0103] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: parsing each split file line by line to obtain M data fragments corresponding to each split file, including: parsing each split file line by line to obtain M rows of data for each split file; determining the target field and determining the target field value corresponding to the target field in each row of data in each split file to obtain the target field value corresponding to each row of data in each split file; and determining the M data fragments corresponding to each split file based on the M rows of data in each split file and the target field value corresponding to each row of data in each split file.

[0104] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: processing each data shard according to a sharding algorithm to obtain routing information for each data shard, including: determining the mapping relationship between each data shard and S database shards to obtain mapping table information; calculating the target field value corresponding to each data shard using a sharding algorithm to obtain the calculation result corresponding to each data shard; and determining the routing information for each data shard based on the mapping table information and the calculation result corresponding to each data shard.

[0105] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: determining the mapping relationship between each data shard and S database shards, and obtaining mapping table information including: determining a first array and a second array based on the number of database shards S, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; calculating the number of logical partitions P based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between data shards and database shards, and P is a positive integer not greater than S; determining the partition information of each of the P logical partitions based on the first array and the second array; and determining the mapping table information based on the partition information of the P logical partitions.

[0106] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: determining the routing information of each data shard based on the mapping table information and the calculation results corresponding to each data shard, including: determining the target logical partition corresponding to the data shard based on the calculation results corresponding to each data shard; determining the target partition information of the target logical partition in the mapping table information; determining the target database shard corresponding to the data shard based on the target partition information; and determining the routing information of the data shard based on the target database shard.

[0107] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: importing N split files into the database based on the routing information of each data split, including: writing M data splits into Q sub-files based on the routing information of M data splits of each split file, where Q is a positive integer not greater than S; establishing Q links between the Q sub-files and the Q database splits; and sending the Q sub-files to the Q database splits based on the Q links, so as to import each split file in the N split files into the database.

[0108] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: splitting a large file to obtain N split files, including: determining the number of split lines in the large file; splitting the large file according to the number of split lines to obtain N split files.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for importing large files, characterized in that, The large file import method is applied to middleware, including: Identify a large file of the database to be imported, wherein the database contains S database shards, where S is an integer greater than 1; The large file is split into N split files, where N is an integer; Each split file is parsed line by line to obtain M data fragments corresponding to each split file, where M is an integer; each data fragment is processed according to the fragmentation algorithm to obtain the routing information of each data fragment; Determine the mapping relationship between each data shard and the S database shards to obtain the mapping table information; The process of determining the mapping relationship between each data shard and the S database shards to obtain the mapping table information includes: Based on the number S of the database shards, a first array and a second array are determined, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1. The number of logical partitions P is calculated by performing calculations on the first array and the second array, wherein the logical partition is used to determine the mapping relationship between the data shard and the database shard, and P is a positive integer not greater than S; Based on the first array and the second array, determine the partition information of each of the P logical partitions; The mapping table information is determined based on the partition information of the P logical partitions; Based on the routing information of each data shard, the N split files are imported into the database; The method further includes: configuring the number of splitting lines in the configuration file of the middleware before splitting the large file, and starting the middleware according to the configuration file; Each data fragment is processed according to the fragmentation algorithm to obtain the routing information for each data fragment, including: The target field value corresponding to each data shard is calculated using the aforementioned sharding algorithm to obtain the calculation result corresponding to each data shard. The routing information for each data fragment is determined based on the mapping table information and the calculation results corresponding to each data fragment; Importing the N split files into the database based on the routing information of each data shard includes: Based on the routing information of the M data fragments of each split file, the M data fragments are written into Q sub-files, where Q is a positive integer not greater than S; Establish Q links based on the Q sub-files and the Q database shards of the database; Based on the Q links, the Q sub-files are sent to the Q database shards to import each of the N split files into the database; The method further includes, if there is a data shard that must be written to a fixed database shard, then directly determining the routing information of that data shard as the fixed database shard corresponding to that data shard.

2. The method according to claim 1, characterized in that, Each split file is parsed line by line to obtain the M data fragments corresponding to each split file, including: Each split file is parsed line by line to obtain M lines of data for each split file; Determine the target field, and determine the target field value corresponding to the target field in each row of data in each split file, to obtain the target field value corresponding to each row of data in each split file; Based on the M rows of data in each split file and the target field value corresponding to each row of data in each split file, determine the M data shards corresponding to each split file.

3. The method according to claim 1, characterized in that, The routing information for each data shard is determined based on the mapping table information and the calculation results corresponding to each data shard, including: The target logical partition corresponding to each data shard is determined based on the calculation results corresponding to each data shard. The target partition information of the target logical partition is determined from the mapping table information; The target database shard corresponding to the data shard is determined based on the target partition information; The routing information of the data shard is determined based on the target database shard.

4. The method according to claim 1, characterized in that, The large file is split into N split files, including: Determine the number of lines to split the large file; The large file is split according to the number of split lines to obtain the N split files.

5. A large-capacity file import device, the device being applied to middleware, characterized in that, include: A determining unit is used to determine a large file of the database to be imported, wherein the database contains S database shards, where S is an integer greater than 1; A splitting unit is used to split the large file into N split files, where N is an integer; The parsing unit is used to parse each split file line by line to obtain M data fragments corresponding to each split file, where M is an integer; The processing unit is used to process each data fragment according to the fragmentation algorithm to obtain the routing information of each data fragment; The processing unit includes: a third determining subunit, used to determine the mapping relationship between each data shard and S database shards, and obtain mapping table information; The third determining subunit further includes: a first determining module, used to determine a first array and a second array based on the number S of database shards, wherein the first array includes L values ​​and the second array includes L values, where L is a positive integer greater than 1; a calculation module, used to calculate the number P of logical partitions based on the first array and the second array, wherein the logical partitions are used to determine the mapping relationship between data shards and database shards, and P is a positive integer not greater than S; a second determining module, used to determine the partition information of each of the P logical partitions based on the first array and the second array; a third determining module, used to determine the mapping table information based on the partition information of the P logical partitions; and an import unit, used to import the N split files into the database based on the routing information of each data shard. The device further includes: configuring the number of split lines in the configuration file of the middleware before splitting the large file, and starting the middleware according to the configuration file; The processing unit includes: a calculation subunit, used to calculate the target field value corresponding to each data shard using a sharding algorithm to obtain the calculation result corresponding to each data shard; and a fourth determination subunit, used to determine the routing information of each data shard based on the mapping table information and the calculation result corresponding to each data shard. The import unit includes: a writing subunit, used to write M data fragments into Q subfiles based on the routing information of M data fragments of each split file, where Q is a positive integer not greater than S; a creation subunit, used to establish Q links between the Q subfiles and the Q database fragments of the database; and a sending subunit, used to send the Q subfiles to the Q database fragments based on the Q links, so as to import each split file in the N split files into the database. The device is also used to, if there is a data fragment that must be written to a fixed database fragment, directly determine the routing information of that data fragment as the fixed database fragment corresponding to that data fragment.

6. A processor, characterized in that, The processor is used to run a program, wherein the program executes the large file import method according to any one of claims 1 to 4 when it runs.

7. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the large file import method according to any one of claims 1 to 4.

Images