Database data import method and device, electronic equipment and storage medium

Abstract

The application relates to a database technology and provides a database data import method and device, electronic equipment and a storage medium. The database data import method is applied to a preset import tool, the master control thread is used to query the meta information of a table of to-be-imported data, the control information of the table of the to-be-imported data is determined, a reading thread group and a sending thread group are created, the target reading thread is used to read row data of the to-be-imported data, the distribution key and the partition key corresponding to the row data are determined according to the control information of the table, so that the row data is inserted into the corresponding partition data unit, and the target sending thread is used to send the partition data unit to a target database according to the node information and the partition information in the obtained partition data unit. The import tool does not need to be deployed and can be independently executed on the client side. Only small server computing resources are consumed in the data import process, the influence on the database service process can be reduced, and the import performance can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of database, and particularly relate to a database data import method and device, an electronic device and a storage medium. BACKGROUND

[0002] After a business system runs for a long time, TB or even PB level business data is generated and accumulated. During database migration, how to quickly import the accumulated massive raw data into a new database cluster is an important problem that needs to be faced in the migration work. Data import is an important link in the data migration process. Low data import performance not only increases the cost of migration, but also blocks the online use of the subsequent business system, and even causes online accidents. Therefore, data import is crucial to the rapid deployment and online of a business system.

[0003] The existing data import can use a streaming sending method, that is, a tool side reads original data files, sends them to a database server through a data stream, and stores them in corresponding table partitions after being parsed and converted by a service process of the server. The data import can also use an external table method, that is, an external table is created based on original data files, and the external table and an internal table are quickly imported through an INSERT INTO SELECT method. In the two methods, the processing and organization of data are completed by the service process of the database side, which not only cannot fully utilize the processing capacity of the client side, but also affects the smooth running of the business of the database side.

[0004] Therefore, how to improve the performance of data import while reducing the business impact on the database service process becomes a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a database data import method and device, an electronic device and a storage medium, which aim to solve the problem of how to improve the performance of data import while reducing the business impact on the database service process.

[0006] In a first aspect, a database data import method is provided,

[0007] The database data import method is applied to a preset import tool, the import tool is provided with a master thread and N processing threads, N is an integer greater than 1, and the database data import method comprises the following steps:

[0008] The master thread is used to query and parse the meta information of a table of to-be-imported data, determine control information of the table of to-be-imported data, and create a reading thread group and a sending thread group from the N processing threads;

[0009] Use any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information.

[0010] Use any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0011] Secondly, a database data import device is provided, which is applied to a preset import tool.

[0012] The import tool is configured with a main control thread and N processing threads, where N is an integer greater than 1. The database data import method includes:

[0013] The data preprocessing module is used to query and parse the metadata of the table containing the data to be imported using the main control thread, determine the control information of the table containing the data to be imported, and create a read thread group and a send thread group from the N processing threads.

[0014] The data reading and analysis module is used to select any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information.

[0015] The data import and sending module is used to select any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0016] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein,

[0017] Memory, used to store computer programs;

[0018] A processor is used to execute programs stored in memory to implement the database data import method described in the first aspect above.

[0019] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the database data import method in the first aspect is implemented.

[0020] The database data import method provided by the application has the following beneficial effects compared with the prior art: The database data import method is applied to a preset import tool. The import tool is provided with a master thread and N processing threads. The master thread is used to query and parse the meta information of a table of the data to be imported, and determine the control information of the table of the data to be imported. A reading thread group and a sending thread group are created from the N processing threads. Any available thread in the reading thread group is used as a target reading thread. The target reading thread is used to read the control information of the table of the data to be imported. According to the control information of the table of the data to be imported, the distribution information and the row partition information of each row of data in the data to be imported are determined. According to the distribution information and the row partition information, the row data is inserted into the corresponding partition data unit. Any available thread in the sending thread group is used as a target sending thread. The target sending thread is used to send the partition data unit to a target database according to the node information and the partition information of the partition data unit. The import tool provided by the application does not need to be deployed and can be independently executed on a client. Only a small amount of server computing resources is consumed in the data import process. The influence on the database service process is reduced, and the import performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is an import process schematic diagram of an existing copy tool provided by the application;

[0022] Figure 2 is an import process schematic diagram of an existing gpfdist service tool provided by the application;

[0023] Figure 3 is a flow schematic diagram of the database data import method provided by the first embodiment of the application;

[0024] Figure 4 is a flow schematic diagram of the database data import method provided by the second embodiment of the application;

[0025] Figure 5 is an architecture schematic diagram of the database data import provided by the second embodiment of the application;

[0026] Figure 6 is a flow schematic diagram of the database data import method provided by the third embodiment of the application;

[0027] Figure 7 is an execution flow schematic diagram of the reading thread provided by the third embodiment of the application;

[0028] Figure 8This is a schematic diagram of the execution flow of the sending thread provided in Embodiment 3 of this application;

[0029] Figure 9 This is a schematic diagram of the database data import device provided in Embodiment 4 of this application;

[0030] Figure 10 This is a structural diagram of the electronic device provided in Embodiment 5 of this application. Detailed Implementation

[0031] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] As data volumes continue to grow, batch insertion methods can no longer meet performance demands. To improve data import performance, different database developers have designed and provided various import tools. Examples include the copy tool, the GDS import tool, and the gpfdist service tool, as detailed below:

[0033] like Figure 1 The diagram illustrates the import process of the existing copy tool. PostgreSQL provides the `copy` meta-command, and based on this meta-command, a copy tool is provided to support users in copying data to remote database tables. Compared to the batch insert method of `INSERT INTO VALUES`, the `copy` meta-command uses a new copy protocol. In scenarios where the default expression on the table is constant and there is no auto-incrementing sequence, it can skip the execution layer logic processing and interact directly with the storage layer in batches, sending data directly to the storage layer for storage. This optimization significantly improves the performance of batch data import. The copy tool operates in single-threaded mode. After reading the data file, the tool does not perform any parsing or transformation processing, but sends the data to the database service process in a streaming manner. The service process then performs parsing, transformation, and partitioning calculations. This working mode not only fails to fully utilize the multi-core resources of the tool's node but also increases the processing pressure on the service process and may even affect the stable operation of other business processes.

[0034] GaussDB provides a GDS import tool, which realizes batch import of data through external tables. Based on data files, a read-only external table is created, and then data is imported from the external table to the internal table through INSERT INTO SELECT, realizing fast import of data. In a distributed deployment scenario, the CN node is responsible for planning the import task, and the specific import task is completed by the DN node. The DN nodes can be processed in parallel. If the data responsible for by DN1 belongs to DN2, DN1 needs to route the data to DN2. This working mode requires a large amount of data forwarding between DN nodes, which not only increases the processing pressure of DN, but also increases the network load inside the cluster, affecting the smooth operation of normal business. Moreover, GDS needs to deploy GDS services, increasing the consumption and complexity of resource use.

[0035] As shown in Figure 2 , it is an import process diagram of the existing gpfdist service tool. The gpfdist service tool provided by GP is a file server service based on external tables using HTTP protocol. In the data import scenario (that is, using the tool GPLOAD for import), a read-only external table is created, and the data of the original data file is loaded into GP using INSERT INTO SELECT. The INSERT statement is parsed on the master node, and then distributed to the segment node (segment node). The segment node connects the gpfdist service to retrieve data and perform parsing and calculation, and then distributes the data to the segment node. In addition, the segment nodes can be processed in parallel. Data needs to be routed between segments, which not only increases the processing pressure of the segment, but also increases the network load inside the cluster. Similarly, gpfdist service needs to be deployed independently, increasing the consumption and complexity of resource use.

[0036] In order to avoid the above problems, the present application provides a database data import method, specifically provides an import tool, which is provided with a master control thread and N processing threads, N is an integer greater than 1. As shown in Figure 3 , it is a flowchart of a database data import method provided by the first embodiment of the present application. The database data import method comprises the following steps:

[0037] Step S301, using the master control thread to query and parse the meta information of the table to be imported, determining the control information of the table to be imported, and creating a reading thread group and a sending thread group from N processing threads.

[0038] The number of threads of the corresponding thread group is set in the import tool, and specifically, the master thread can create and start the corresponding thread according to the number of the set thread groups.

[0039] The master thread queries the meta information of the table of the data to be imported, thereby determining the control information of the table of the data to be imported, wherein the control information can include but is not limited to distribution information, partition information, column attribute information and distributed routing information.

[0040] The master thread starts the import task after receiving the data to be imported, and therefore, the corresponding thread needs to be created for reading and sending. The reading thread (i.e., the reader thread) forms a reading thread group, and the sending thread (i.e., the sender thread) forms a sending thread group. It can be seen that the reading thread group includes at least one reading thread, and the sending thread group includes at least one sending thread. The threads in the reading thread group and the threads in the sending thread group can be processed in parallel, can support the setting of parallelism, can fully utilize the processing capacity of multi-core, and can improve the processing efficiency.

[0041] Optionally, the master thread is used to query and parse the meta information of the table of the data to be imported, and determine the control information of the table of the data to be imported, including:

[0042] The master thread is used to query and extract the meta information of the table of the data to be imported, and obtain the control information of the table of the data to be imported.

[0043] According to the control information of the table of the data to be imported, the distribution key, the partition key and the column attribute information of the table are determined.

[0044] The master thread queries the meta information of the table of the data to be imported, and further determines the control information of the table, especially the distribution information, the partition information, the column attribute information and the distributed routing information. The distribution information, the partition information, the column attribute information and the distributed routing information can be used for subsequent data calculation.

[0045] In step S302, any available thread in the reading thread group is used as a target reading thread, the target reading thread is used to read the control information of the table of the data to be imported, the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information.

[0046] Any thread in the read thread group can be a target read thread, and one target read thread can process one row data at a time. The target read thread reads the row data in the data to be imported, and can obtain the control information determined by the master thread. The distribution key and the partition key of the row data can be calculated from the control information, and a partition data unit can be determined according to the distribution key and the partition key, so that the row data is inserted into the partition data unit, and the subsequent data transmission is realized.

[0047] The partition data unit is a storage area segmented from a storage area constructed in the import tool, and is used to store the data to be imported. Specifically, the partition data unit corresponding to the row data needs to be determined from all the partition data units in the storage area according to the distribution key and the partition key.

[0048] Optionally, the distribution information and the row partition information of each row data in the data to be imported are determined according to the control information of the table of the data to be imported, including:

[0049] The corresponding row data is subjected to data parsing and data type conversion according to the column attribute information of the table of the data to be imported, and the converted row data is obtained.

[0050] The distribution information of the converted row data is determined according to the distribution key of the table of the data to be imported, and the row partition information of the converted row data is determined according to the partition key of the table of the data to be imported.

[0051] The row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information, including:

[0052] The converted row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information.

[0053] The target read thread reads each row data in the table of the data to be imported, and performs data parsing, type conversion, distribution key calculation and partition key calculation according to the column attribute information of the table of the data to be imported, so as to obtain the partition key and the distribution key, and obtain the corresponding converted row data, for subsequent import.

[0054] Optionally, the converted row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information, including:

[0055] The partition data unit is determined according to the distribution information and the row partition information.

[0056] The row number is allocated from the partition data unit, and the number of working threads of the partition data unit is increased. The number of working threads is used to indicate whether the partition data unit is sent to the sending queue of the target sending thread according to the number of working threads.

[0057] According to the assigned row number, the converted row data is inserted into the partition data unit, and after the insertion is completed, the working thread number of the partition data unit is reduced.

[0058] Among all the partition data units in the storage area of the import tool, according to the distribution key and the partition key, the corresponding partition data unit can be determined, and if a partition data unit is full, a new partition data unit is obtained for storage.

[0059] The row number is assigned from the partition data unit, and the working thread number of the partition data unit and the record number of the partition data unit are increased. The read thread is instructed to determine whether to send the partition data unit to the sending queue of the target sending thread according to the working thread number and the record number. In addition, after the insertion of the partition data unit is completed, the working thread number of the partition data unit is reduced, thereby being used subsequently.

[0060] In addition, if a partition data unit is being assigned a row number for other row data, i.e., the partition data unit is in use, it needs to wait until the assignment is completed before it can be used again, i.e., a partition data unit can only assign a row number to one read thread at the same time.

[0061] When assigning a row number, the working thread number of the partition data unit needs to be increased. After obtaining the row number, the converted row data is inserted into the partition data unit according to the assigned row number, and after the insertion is completed, the working thread number of the partition data unit is reduced.

[0062] In step S303, any available thread in the sending thread group is used as a target sending thread, and the target sending thread is used to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0063] Any available thread in the sending thread group can be used as a target sending thread, and one target sending thread can process one partition data unit at a time. The target sending thread reads the partition data unit in the storage area of the import tool, and can obtain the node information and partition information in the partition data unit.

[0064] The above node information is used to represent the data node of the target database to which the to-be-imported data needs to be imported, so that the import tool knows the destination of the data, thereby sending the to-be-imported data to the data node of the target database.

[0065] In addition, the partition information is used to represent the information of the partition data unit, so that the data can be accurately inserted into the target partition of the to-be-imported table when the data is sent to the target database.

[0066] The distribution key and partition key of the row data can be calculated from the control information. By combining the distribution information and partition information, a partition data unit can be determined so that the row data can be inserted into the partition data unit to enable the subsequent data transmission.

[0067] In this embodiment, the database data import method is applied to a preset import tool. The import tool has a main control thread and N processing threads. The main control thread queries and parses the metadata of the table containing the data to be imported, determines the control information of the table, creates a read thread group and a send thread group from the N processing threads, selects any available thread in the read thread group as the target read thread, and uses the target read thread to read the control information of the table containing the data to be imported. Based on the control information of the table containing the data to be imported, it determines the distribution information and row partition information of each row of data in the data to be imported. Based on the distribution information and row partition information, it inserts the row data into the corresponding partition data unit. Any available thread in the send thread group is selected as the target send thread, and uses the target send thread to send the partition data unit to the target database based on the node information and partition information of the partition data unit. The import tool provided in this application does not require deployment and can be executed independently on the client side. It consumes only a small amount of server-side computing resources during the data import process, improving import performance while reducing the impact on the database service process.

[0068] like Figure 4 As shown, this is a database data import method provided in Embodiment 2 of this application. In step S301 above, the method of using a main control thread to query and parse the metadata of the table containing the data to be imported, and to determine the control information of the table containing the data to be imported, may further include the following steps:

[0069] Step S401: According to the preset block size, the data to be imported is divided into file block information of at least one file block.

[0070] When the data to be imported is large, it needs to be split into at least one file block according to the preset block size. Each file block is processed separately. Specifically, a reading thread is allocated one file block and the reading thread continues to work until all lines of data in the file block have been processed.

[0071] Accordingly, in step S302 above, the target reading thread reads the control information of the table containing the data to be imported. Based on the control information of the table containing the data to be imported, the distribution information and row partition information of each row of data in the data to be imported are determined, which may include the following steps:

[0072] Step S402: For any file block, obtain the file block information of the file block, and use the target reading thread to read any line of data in the file block.

[0073] Step S403: Based on the control information of the table to be imported, determine the distribution information and row partition information of each row of data in the data to be imported.

[0074] In this process, a file block information is matched with a reading thread, which is marked as busy until all lines of data in the file block information have been processed.

[0075] like Figure 5 The diagram shown is a schematic of the database data import architecture provided in Embodiment 2 of this application. It is configured with N read threads and M send threads. Through the above architecture, data can be imported into the target database.

[0076] In this embodiment, the data to be imported is divided into blocks so that each file block corresponds to a reading thread, allowing different reading threads to process different file blocks simultaneously, achieving a parallel effect and improving data processing efficiency.

[0077] like Figure 6 The diagram shown is a flowchart of a database data import method provided in Embodiment 3 of this application. After inserting row data into the corresponding partition data unit based on distribution information and row partition information in step S302, the method may further include the following steps:

[0078] Step S601: Detect whether the partition data unit meets the preset transmission conditions.

[0079] The preset sending conditions are used to characterize whether a partitioned data unit can be sent to the target database. Specifically, the preset sending conditions may be whether the data storage volume has reached the preset storage space, or whether the number of rows of data has reached the preset number of rows threshold, etc.

[0080] Step S602: If the partition data unit is detected to meet the preset transmission conditions, the corresponding transmission thread of the partition data unit is determined from the transmission thread group, and the location information of the partition data unit is sent to the transmission queue of the corresponding thread.

[0081] Specifically, when the preset sending conditions are met, the corresponding thread, namely the target sending thread, is determined from the sending thread group, and the partition data unit is sent to the sending queue of the target sending thread, waiting for the target sending thread to send it.

[0082] Since the target sending thread can send more than one partition data unit, setting a sending queue allows the target sending thread to send the partition data units in the sending queue one by one until all partition data units in the sending queue have been sent.

[0083] Correspondingly, the step S303 uses the target sending thread to send the partition data unit to the target database according to the node information and the partition information of the partition data unit can include the following steps:

[0084] In step S603, the target sending thread determines the position information of the partition data unit to be sent from the sending queue, and acquires the corresponding partition data unit according to the position information of the partition data unit to be sent.

[0085] The target sending thread acquires the position information of the partition data unit to be sent after determining the partition data unit to be sent from the sending queue, so as to acquire the data in the corresponding partition data unit according to the position information.

[0086] In step S604, the partition data unit is parsed to determine the node information and the partition information of the corresponding partition data unit.

[0087] The target sending thread parses the partition data unit to determine the corresponding node information and partition information, so that the data in the partition data unit can be sent to the target database according to the node information and the partition information subsequently.

[0088] Optionally, the step of sending the partition data unit to the target database according to the node information and the partition information includes:

[0089] According to the node information, the link of the corresponding target database is determined;

[0090] According to the link and the partition information, the partition data unit is sent to the service thread of the target database, and the service thread is used to call the storage layer interface to write the data in the partition data unit to the target database after receiving the data in the partition data unit.

[0091] According to the node information, the link of the corresponding target database is determined, and the partition information is set in the link, so that the data in the partition data unit can be sent to the service thread of the target database, thereby realizing the data import of the partition data unit.

[0092] In the embodiments of the present application, the state of the partition data unit is monitored, so that the partition data unit is sent as a whole when the sending condition is met, centralized sending is realized, and frequent calling of the sending thread is avoided.

[0093] For example, the yasldr import tool provided by the present application can be executed independently on the client side without the need to deploy any service in advance. Figure 7 and 8As shown, respectively, are an execution flow schematic diagram of a reading thread and an execution flow schematic diagram of a sending thread provided by the embodiment three of the present application, when imported using a yasldr import tool, the processing flow of the yasldr is as follows:

[0094] 1) The master thread queries control information of a table of the data to be imported, including distribution information, partition information, column attribute information, distributed routing information, and the like, and initializes partition calculation context;

[0095] 2) The master thread splits the data file according to a fixed block size, for subsequent parallel processing of the reader thread;

[0096] 3) The master thread creates a reader thread group and a sender thread group, and starts the thread groups;

[0097] 4) The master thread loops to wait for the processing of the reader thread group and the sender thread group to end;

[0098] 5) The reader thread obtains the split file block;

[0099] 6) The reader thread reads a row of data, and performs data parsing, type conversion, distribution key calculation, and partition key calculation according to the column attribute information of the query;

[0100] 7) The reader thread inserts the data into a corresponding partition data unit according to the calculated distribution key and partition key;

[0101] 8) The reader thread checks whether the partition unit reaches a sending condition, and writes control information of the partition unit into a sending queue of the sender;

[0102] 9) The sender thread loops to wait on the sending queue, and obtains the partition unit to be sent;

[0103] 10) The sender thread obtains the sending unit, and parses node information and partition information to which the sending unit belongs;

[0104] 11) The sender thread uses a link corresponding to the node, sets the partition information, and sends the partition unit to a database service thread;

[0105] 12) The database service thread receives the partition unit data, reads the data, and calls a storage layer interface to write the data in batches.

[0106] The embodiment of the present application further provides a database data import device 90, which is applied to a preset import tool, the import tool is provided with a master thread and N processing threads, N is an integer greater than 1, please refer to Figure 9 , and includes:

[0107] The data preprocessing module 910 is configured to query and parse the meta information of the table of the data to be imported using the master thread, determine the control information of the table of the data to be imported, and create a reading thread group and a sending thread group from the N processing threads;

[0108] The data reading and analyzing module 920 is configured to use any available thread in the reading thread group as a target reading thread, read the control information of the table of the data to be imported using the target reading thread, determine the distribution information and the row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information.

[0109] The data importing and sending module 930 is configured to use any available thread in the sending thread group as a target sending thread, and send the partition data unit to the target database according to the node information and the partition information of the partition data unit using the target sending thread.

[0110] Optionally, the data preprocessing module 910 comprises:

[0111] The data querying unit is configured to query and extract the meta information of the table of the data to be imported using the master thread, and obtain the control information of the table of the data to be imported.

[0112] The control information extraction unit is configured to determine the distribution key, the partition key and the column attribute information of the table according to the control information of the table of the data to be imported.

[0113] Optionally, the data reading and analyzing module 920 comprises:

[0114] The data analyzing and converting unit is configured to perform data analysis and data type conversion on the corresponding row data according to the column attribute information of the table of the data to be imported, and obtain the converted row data.

[0115] The data key calculation unit is configured to determine the distribution information of the converted row data according to the distribution key of the table of the data to be imported, and determine the row partition information of the converted row data according to the partition key of the table of the data to be imported.

[0116] The data reading and analyzing module 920 comprises:

[0117] The data inserting unit is configured to insert the converted row data into the corresponding partition data unit according to the distribution information and the row partition information.

[0118] Optionally, the data inserting unit comprises:

[0119] The partition determination subunit is configured to determine the partition data unit according to the distribution information and the row partition information.

[0120] The row number allocation subunit is used to allocate row numbers from the partition data unit and increase the number of worker threads for the partition data unit. The number of worker threads is used to instruct the reading thread to determine whether to send the partition data unit into the sending queue of the target sending thread based on the number of worker threads.

[0121] The data insertion sub-unit is used to insert the transformed row data into the partition data unit according to the assigned row number, and reduces the number of working threads of the partition data unit after the insertion is completed.

[0122] Optionally, database data import methods also include:

[0123] The file splitting module is used to use the main control thread to query and parse the metadata of the table to be imported, determine the control information of the table to be imported, and then split the data to be imported according to the preset block size to obtain file block information of at least one file block.

[0124] Data reading and analysis module 920 includes:

[0125] The line data reading unit is used to obtain the file block information of any file block and use the target reading thread to read any line of data in the file block;

[0126] The thread status marker unit is used to determine the distribution information and row partition information of each row of data in the data to be imported, based on the control information of the table to be imported.

[0127] Optionally, database data import methods also include:

[0128] The sending detection module is used to detect whether the partition data unit meets the preset sending conditions after inserting the row data into the corresponding partition data unit according to the distribution information and row partition information.

[0129] The information sending module is used to determine the sending thread corresponding to the partition data unit from the sending thread group if the partition data unit is detected to meet the preset sending conditions, and send the location information of the partition data unit to the sending queue of the corresponding thread.

[0130] The data import and sending module 930 includes:

[0131] The partition data acquisition unit is used to determine the location information of the partition data unit to be sent from its own sending queue using the target sending thread, and to acquire the corresponding partition data unit based on the location information of the partition data unit to be sent.

[0132] The partition data parsing unit is used to parse the partition data unit and determine the node information and partition information of the corresponding partition data unit.

[0133] Optionally, the data import sending module 930 comprises:

[0134] a link determining unit, configured to determine a link of the target database according to the node information;

[0135] a data import sending unit, configured to send the partition data unit to a service thread of the target database according to the link and the partition information, the service thread being configured to call a storage layer interface to write the data in the partition data unit into the target database after receiving the data in the partition data unit.

[0136] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by the modules can be referred to the method embodiments part, which will not be described here.

[0137] The present application also provides an electronic device 100, please refer to Figure 10 , comprising a memory 101 and a processor 102, wherein the memory 101 is used to store computer programs; the processor 102 is used to execute the programs stored in the memory 101, and realize the database data import method introduced in any embodiment of the present application.

[0138] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the database data import method introduced in any embodiment of the present application.

[0139] In the present application, multiple means two or more than two.

[0140] In the present application, unless otherwise explicitly limited, the terms "installation", "connection", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through intermediate medium, or the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0141] The terms "first", "second", "third", "fourth" and the like (if any) in the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0142] In the present application, the term "and / or" is only used to describe the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which means that there are three kinds of situations: A alone, A and B exist at the same time, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0143] If not otherwise specified, all steps of the application can be performed in any order. For example, the method comprises steps A and B, meaning that the method can comprise steps A and B performed in sequence, or steps B and A performed in sequence. For example, the method can further comprise step C, meaning that step C can be added to the method in any order, for example, the method can comprise steps A, B and C, or steps A, C and B, or steps C, A and B, etc.

[0144] The above description is only the preferred embodiment of the present application, not to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A database data import method characterized by, The database data import method is applied to a preset import tool, the import tool is provided with a master thread and N processing threads, N is an integer greater than 1, and the database data import method comprises the following steps of: querying and parsing the meta information of the table of the data to be imported by using the master thread, determining the control information of the table of the data to be imported, creating a reading thread group and a sending thread group from the N processing threads, and the control information comprises distribution information, partition information and column attribute information; any available thread in the reading thread group is used as a target reading thread, the control information of the table of the data to be imported is read by using the target reading thread, the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information; any available thread in the sending thread group is used as a target sending thread, and the partition data unit is sent to the target database according to the node information and the partition information of the partition data unit by using the target sending thread; the control information of the table of the data to be imported is determined by querying and parsing the meta information of the table of the data to be imported by using the master thread, and the control information of the table of the data to be imported is determined by querying and parsing the meta information of the table of the data to be imported by using the master thread; the control information of the table of the data to be imported is obtained by querying and extracting the meta information of the table of the data to be imported by using the master thread; the distribution key, the partition key and the column attribute information of the table are determined according to the control information of the table of the data to be imported; the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information; the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information; the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information; the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information. the distribution information and the row partition information of each row of data in the data to be imported are determined according to the control information of the table of the data to be imported, and the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information.

2. The database data import method of claim 1, wherein, after the step of querying and parsing the meta information of the table of the data to be imported by using the master thread and determining the control information of the table of the data to be imported, the method further comprises the following steps of: ​ ​ ​ 3. The database data import method of claim 1, wherein, ​ According to a preset block size, the to-be-imported data is segmented to obtain file block information of at least one file block; The control information of the table of the to-be-imported data is read by using the target reading thread, and according to the control information of the table of the to-be-imported data, distribution information and row partition information of each row of data in the to-be-imported data are determined, including: For any file block, the file block information of the file block is obtained, and any row of data in the file block is read by using the target reading thread; According to the control information of the table of the to-be-imported data, distribution information and row partition information of each row of data in the to-be-imported data are determined.

4. The database data import method according to any one of claims 1 to 3, characterized by, After the row data is inserted into the corresponding partition data unit according to the distribution information and the row partition information, the following steps are further included: It is detected whether the partition data unit meets a preset sending condition; If it is detected that the partition data unit meets the preset sending condition, a sending thread corresponding to the partition data unit is determined from the sending thread group, and position information of the partition data unit is sent to a sending queue of the corresponding thread; The partition data unit is sent to a target database according to node information and partition information of the partition data unit by using the target sending thread, including: The position information of the to-be-sent partition data unit is determined from the sending queue of the target sending thread according to the node information and the partition information of the to-be-sent partition data unit, and the corresponding partition data unit is obtained according to the position information of the to-be-sent partition data unit; The partition data unit is parsed to determine the node information and the partition information corresponding to the partition data unit.

5. The database data import method of claim 4, wherein, The partition data unit is sent to a target database according to the node information and the partition information, including: According to the node information, a link corresponding to the target database is determined; According to the link and the partition information, the partition data unit is sent to a service thread of the target database, and the service thread is configured to call a storage layer interface to write data in the partition data unit into the target database after receiving the data in the partition data unit.

6. A database data import apparatus characterized by comprising: The database data import device is applied to a preset import tool, the import tool is provided with a master control thread and N processing threads, N is an integer greater than 1, and the database data import method includes: A data preprocessing module is configured to query and parse meta information of a table of to-be-imported data by using the master control thread, determine control information of the table of the to-be-imported data, and create a reading thread group and a sending thread group from the N processing threads, wherein the control information includes distribution information, partition information and column attribute information; A data reading and analyzing module is configured to use any available thread in the reading thread group as a target reading thread, read control information of a table of to-be-imported data by using the target reading thread, determine distribution information and row partition information of each row of data in the to-be-imported data according to the control information of the table of the to-be-imported data, and insert the row data into a corresponding partition data unit according to the distribution information and the row partition information. The data import sending module is configured to use any available thread in the sending thread group as a target sending thread, and send the partition data unit to a target database according to the node information and the partition information of the partition data unit by using the target sending thread. The data preprocessing module comprises: The data query unit is configured to query and extract the meta information of the table of the data to be imported by using the master thread, and obtain the control information of the table of the data to be imported. The control information extraction unit is configured to determine the distribution key, the partition key and the column attribute information of the table according to the control information of the table of the data to be imported. The data reading and analyzing module comprises: The data analysis and conversion unit is configured to perform data analysis and data type conversion on the corresponding row data according to the column attribute information of the table of the data to be imported, and obtain the converted row data. The data key calculation unit is configured to determine the distribution information of the converted row data according to the distribution key of the table of the data to be imported, and determine the row partition information of the converted row data according to the partition key of the table of the data to be imported. The data reading and analyzing module comprises: The data insertion unit is configured to insert the converted row data into the corresponding partition data unit according to the distribution information and the row partition information.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the database data import method of any one of claims 1-5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the database data import method of any one of claims 1-5. ​ 8. A computer-readable storage medium, characterized in that, ​

Images