Database data importing method and device, electronic equipment and storage medium

Abstract

The invention relates to a database technology, and provides a database data importing method and device, electronic equipment and a storage medium. The database data importing method is applied to a preset importing tool, and comprises the following steps: querying meta information of a table of to-be-imported data by using a main control thread, determining control information of the table of the to-be-imported data, creating a reading thread group and a sending thread group, reading row data of the to-be-imported data by using a target reading thread, and sending the row data of the to-be-imported data according to the control information of the table. And determining a distribution key and a partition key corresponding to the row data, inserting the row data into a corresponding partition data unit, and sending the partition data unit to a target database by using a target sending thread according to the obtained node information and partition information in the partition data unit. The import tool does not need to be deployed, can be independently executed at the client, only consumes small server computing resources in the data import process, and can reduce the influence on the database service process while improving the import performance.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database data import method, apparatus, electronic device, and storage medium. Background Technology

[0002] After long-term operation, business systems generate and accumulate terabytes (TB) or even petabytes (PB) of business data. During database migration, a crucial issue is how to quickly import this massive amount of raw data into the new database cluster. Data import is a vital step in the data migration process. Poor data import performance not only increases migration costs but also hinders the subsequent deployment of business systems and can even cause network incidents. Therefore, data import is essential for the rapid deployment and launch of business systems.

[0003] Data import can be done using two methods: streaming and external tables. The client reads the raw data file and sends it to the database server, where the server process parses and transforms the data before storing it in the corresponding table partition. Alternatively, an external table can be created based on the raw data file, and data can be quickly imported between the external and internal tables using INSERT INTO SELECT statements. In both of these methods, data processing and organization are handled by the database server process, which not only fails to fully utilize the client-side processing power but also impacts the stable operation of database-side services.

[0004] Therefore, how to improve the performance of data import while minimizing the business impact on the database service process has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a database data import method, apparatus, electronic device, and storage medium, aiming to solve the problem of how to improve the performance of data import while reducing the business impact on the database service process.

[0006] Firstly, it provides a method for importing data from a database. The database data import method is applied to a preset import tool, which has a main control thread and N processing threads, where N is an integer greater than 1. The database data import method includes: The main control thread is used to query and parse the metadata of the table containing the data to be imported, determine the control information of the table containing the data to be imported, and create a read thread group and a send thread group from the N processing threads. Use any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information. Use any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0007] Secondly, a database data import device is provided, which is applied to a preset import tool. The import tool is configured with a main control thread and N processing threads, where N is an integer greater than 1. The database data import method includes: The data preprocessing module is used to query and parse the metadata of the table containing the data to be imported using the main control thread, determine the control information of the table containing the data to be imported, and create a read thread group and a send thread group from the N processing threads. The data reading and analysis module is used to select any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information. The data import and sending module is used to select any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0008] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein, Memory, used to store computer programs; A processor is used to execute programs stored in memory to implement the database data import method described in the first aspect above.

[0009] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the database data import method described in the first aspect.

[0010] The advantages of this application compared to existing technologies are as follows: The database data import method of this application is applied to a preset import tool, which is equipped with a main control thread and N processing threads. The main control thread queries and parses the metadata of the table to be imported to determine the control information of the table. A read thread group and a send thread group are created from the N processing threads. Any available thread in the read thread group is selected as the target read thread, which reads the control information of the table to be imported. Based on the control information of the table, the distribution information and row partition information of each row of data in the data to be imported are determined. Based on the distribution information and row partition information, the row data is inserted into the corresponding partition data unit. Any available thread in the send thread group is selected as the target send thread, which sends the partition data unit to the target database based on the node information and partition information of the partition data unit. The import tool provided by this application does not require deployment and can be executed independently on the client side. During the data import process, it consumes only a small amount of server-side computing resources, thereby improving import performance while reducing the impact on the database service process. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the import process of an existing copy tool provided in this application; Figure 2 This is a schematic diagram of the import process of an existing gpfdist service tool provided in this application; Figure 3 This is a flowchart illustrating the database data import method provided in Embodiment 1 of this application; Figure 4 This is a flowchart illustrating the database data import method provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the database data import architecture provided in Embodiment 2 of this application; Figure 6 This is a flowchart illustrating the database data import method provided in Embodiment 3 of this application; Figure 7 This is a schematic diagram of the execution flow of the reading thread provided in Embodiment 3 of this application; Figure 8 This is a schematic diagram of the execution flow of the sending thread provided in Embodiment 3 of this application; Figure 9 This is a schematic diagram of the database data import device provided in Embodiment 4 of this application; Figure 10 This is a structural diagram of the electronic device provided in Embodiment 5 of this application. Detailed Implementation

[0012] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] As data volumes continue to grow, batch insertion methods can no longer meet performance demands. To improve data import performance, different database developers have designed and provided various import tools. Examples include the copy tool, the GDS import tool, and the gpfdist service tool, as detailed below: like Figure 1 The diagram illustrates the import process of the existing copy tool. PostgreSQL provides the `copy` meta-command, and based on this meta-command, a copy tool is provided to support users in copying data to remote database tables. Compared to the batch insert method of `INSERT INTO VALUES`, the `copy` meta-command uses a new copy protocol. In scenarios where the default expression on the table is constant and there is no auto-incrementing sequence, it can skip the execution layer logic processing and interact directly with the storage layer in batches, sending data directly to the storage layer for storage. This optimization significantly improves the performance of batch data import. The copy tool operates in single-threaded mode. After reading the data file, the tool does not perform any parsing or transformation processing, but sends the data to the database service process in a streaming manner. The service process then performs parsing, transformation, and partitioning calculations. This working mode not only fails to fully utilize the multi-core resources of the tool's node but also increases the processing pressure on the service process and may even affect the stable operation of other business processes.

[0014] GaussDB's GDS import tool enables batch data import via external tables. It creates read-only external tables based on data files, then imports data from these tables into internal tables using INSERT INTO SELECT statements, achieving rapid data import. In a distributed deployment scenario, CN nodes are responsible for planning the import tasks, while the actual import tasks are completed by DN nodes, which can process data in parallel. If data handled by DN1 belongs to DN2, DN1 needs to route the data to DN2. This workflow requires a significant amount of data forwarding between DNs, increasing not only the processing pressure on the DNs but also the network load within the cluster, impacting the smooth operation of normal business processes. Furthermore, GDS requires the deployment of a GDS service, increasing resource consumption and complexity.

[0015] like Figure 2The diagram illustrates the import process of the existing gpfdist service tool. The gpfdist service tool provided by GP is a file server service based on external tables and using the HTTP protocol. In a data import scenario (i.e., using the GPLOAD tool), a read-only external table is created, and the data from the original data file is loaded into GP using the INSERT INTO SELECT statement. The INSERT statement is parsed on the master node and then distributed to the segment nodes. After connecting to the gpfdist service, the segment nodes retrieve, parse, and compute the data, and then distribute the data to their respective segment nodes. Furthermore, segment nodes can process data in parallel. Data needs to be routed between segments, which not only increases the processing pressure on the segments but also increases the network load within the cluster. Similarly, the gpfdist service needs to be deployed independently, increasing resource consumption and complexity.

[0016] To avoid the aforementioned problems, this application provides a database data import method, specifically an import tool. This tool has a main control thread and N processing threads, where N is an integer greater than 1. For example... Figure 3 The diagram shown is a flowchart illustrating a database data import method provided in Embodiment 1 of this application. The database data import method includes the following steps: Step S301: Use the main control thread to query and parse the metadata of the table to be imported, determine the control information of the table to be imported, and create a read thread group and a send thread group from N processing threads.

[0017] Specifically, the number of threads in the corresponding thread group is set in the import tool. The main control thread can create and start the corresponding threads according to the set number of thread groups.

[0018] The main control thread queries the metadata of the table from which the data to be imported, thereby determining the control information of the table. This control information may include, but is not limited to, distribution information, partition information, column attribute information, and distributed routing information.

[0019] After receiving the data to be imported, the main control thread initiates the import task. Therefore, corresponding threads need to be created for reading and sending. Reading threads (i.e., reader threads) form a reading thread group, and sending threads (i.e., sender threads) form a sending thread group. Thus, each reading thread group includes at least one reading thread, and each sending thread group includes at least one sending thread. Threads in the reading and sending thread groups can process data in parallel, supporting adjustable parallelism to fully utilize multi-core processing power and improve processing efficiency.

[0020] Optionally, the main control thread can be used to query and parse the metadata of the table from which the data is to be imported, and determine the control information of the table from which the data is to be imported, including: The main control thread is used to query and extract the metadata of the table from which the data to be imported, and obtain the control information of the table from which the data to be imported. Based on the control information of the table from which the data is to be imported, determine the table's distribution key, partition key, and column attribute information.

[0021] The main control thread queries the metadata of the table containing the data to be imported, thereby determining the table's control information, especially distribution information, partition information, column attribute information, and distributed routing information. This distribution information, partition information, column attribute information, and distributed routing information can be used for subsequent data calculations.

[0022] Step S302: Select any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table to be imported, determine the distribution information and row partition information of each row of data in the data to be imported based on the control information of the table to be imported, and insert the row data into the corresponding partition data unit based on the distribution information and row partition information.

[0023] In this process, any available thread in the read thread group can be used as the target read thread. A target read thread can process one row of data at a time. The target read thread reads the row data in the data to be imported and can obtain the control information determined by the main control thread. From the control information, the distribution key and partition key of the row data can be calculated. Combining the distribution key and partition key, a partition data unit can be determined so that the row data can be inserted into the partition data unit to realize the subsequent data transmission.

[0024] A partitioned data unit is a storage area segmented from the storage area constructed in the import tool, used to store the data to be imported. Specifically, it is necessary to determine the partitioned data unit corresponding to the row of data from all the partitioned data units in the storage area based on the distribution key and partition key.

[0025] Optionally, based on the control information of the table to be imported, determine the distribution information and row partitioning information of each row of data in the data to be imported, including: Based on the column attribute information of the table to be imported, the corresponding row data is parsed and the data type is converted to obtain the converted row data; Based on the distribution key of the table to be imported, determine the distribution information of the converted row data; based on the partition key of the table to be imported, determine the row partition information of the converted row data. Based on the distribution information and row partition information, insert the row data into the corresponding partition data unit, including: Based on the distribution information and row partition information, the transformed row data is inserted into the corresponding partition data unit.

[0026] The target reading thread reads each row of data from the table containing the data to be imported. It then performs data parsing, type conversion, distribution key calculation, and partition key calculation based on the column attribute information of the table containing the data to be imported, thereby obtaining the partition key and distribution key, as well as the corresponding converted row data, for subsequent import.

[0027] Optionally, based on the distribution information and row partitioning information, the transformed row data is inserted into the corresponding partition data unit, including: Based on the distribution information and row partition information, determine the partition data unit; Assign row numbers from the partition data unit and increment the number of worker threads for the partition data unit. The number of worker threads is used to instruct the read thread to determine whether to send the partition data unit into the send queue of the target send thread based on the number of worker threads. Based on the assigned row number, the transformed row data is inserted into the partition data unit, and the number of worker threads in the partition data unit is reduced after the insertion is completed.

[0028] In this process, for all partition data units in the known storage area of ​​the import tool, the corresponding partition data unit can be determined based on the distribution key and partition key. If a partition data unit is full, a new partition data unit is obtained for storage.

[0029] Assign row numbers from the partition data unit, and increase the number of worker threads and records in the partition data unit. Instruct the read thread to determine whether to send the partition data unit into the send queue of the target send thread based on the number of worker threads and records. In addition, after the partition data unit is inserted, decrease the number of worker threads in the partition data unit for subsequent use.

[0030] In addition, if a partition data unit is currently allocating row numbers to other rows of data, meaning the partition data unit is in use, it can only be used again after the allocation is completed. In other words, a partition data unit can only be allocated row numbers to one read thread at a time.

[0031] When assigning row numbers, the number of worker threads for the partition data unit needs to be increased. After obtaining the row number, the converted row data is inserted into the partition data unit according to the assigned row number, and the number of worker threads for the partition data unit is reduced after the insertion is completed.

[0032] Step S303: Select any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0033] In this context, any available thread in the sending thread group can serve as a target sending thread. A target sending thread can process one partition data unit at a time. The target sending thread reads the partition data unit in the storage area of ​​the import tool and can obtain the node information and partition information in that partition data unit.

[0034] The aforementioned node information is used to characterize the data nodes of the target database into which the data to be imported needs to be imported, so that the import tool knows the destination of the data and sends the data to be imported to the data nodes of the target database.

[0035] In addition, partition information is used to characterize the information of partitioned data units, so that when sending data to the target database, the data can be accurately inserted into the target partition of the table to be imported.

[0036] The distribution key and partition key of the row data can be calculated from the control information. By combining the distribution information and partition information, a partition data unit can be determined so that the row data can be inserted into the partition data unit to enable the subsequent data transmission.

[0037] In this embodiment, the database data import method is applied to a preset import tool. The import tool has a main control thread and N processing threads. The main control thread queries and parses the metadata of the table containing the data to be imported, determines the control information of the table, creates a read thread group and a send thread group from the N processing threads, selects any available thread in the read thread group as the target read thread, and uses the target read thread to read the control information of the table containing the data to be imported. Based on the control information of the table containing the data to be imported, it determines the distribution information and row partition information of each row of data in the data to be imported. Based on the distribution information and row partition information, it inserts the row data into the corresponding partition data unit. Any available thread in the send thread group is selected as the target send thread, and uses the target send thread to send the partition data unit to the target database based on the node information and partition information of the partition data unit. The import tool provided in this application does not require deployment and can be executed independently on the client side. It consumes only a small amount of server-side computing resources during the data import process, improving import performance while reducing the impact on the database service process.

[0038] like Figure 4 As shown, this is a database data import method provided in Embodiment 2 of this application. In step S301 above, the method of using a main control thread to query and parse the metadata of the table containing the data to be imported, and to determine the control information of the table containing the data to be imported, may further include the following steps: Step S401: According to the preset block size, the data to be imported is divided into file block information of at least one file block.

[0039] When the data to be imported is large, it needs to be split into at least one file block according to the preset block size. Each file block is processed separately. Specifically, a reading thread is allocated one file block and the reading thread continues to work until all lines of data in the file block have been processed.

[0040] Accordingly, in step S302 above, the target reading thread reads the control information of the table containing the data to be imported. Based on the control information of the table containing the data to be imported, the distribution information and row partition information of each row of data in the data to be imported are determined, which may include the following steps: Step S402: For any file block, obtain the file block information of the file block, and use the target reading thread to read any line of data in the file block.

[0041] Step S403: Based on the control information of the table to be imported, determine the distribution information and row partition information of each row of data in the data to be imported.

[0042] In this process, a file block information is matched with a reading thread, which is marked as busy until all lines of data in the file block information have been processed.

[0043] like Figure 5 The diagram shown is a schematic of the database data import architecture provided in Embodiment 2 of this application. It is configured with N read threads and M send threads. Through the above architecture, data can be imported into the target database.

[0044] In this embodiment, the data to be imported is divided into blocks so that each file block corresponds to a reading thread, allowing different reading threads to process different file blocks simultaneously, achieving a parallel effect and improving data processing efficiency.

[0045] like Figure 6 The diagram shown is a flowchart of a database data import method provided in Embodiment 3 of this application. After inserting row data into the corresponding partition data unit based on distribution information and row partition information in step S302, the method may further include the following steps: Step S601: Detect whether the partition data unit meets the preset transmission conditions.

[0046] The preset sending conditions are used to characterize whether a partitioned data unit can be sent to the target database. Specifically, the preset sending conditions may be whether the data storage volume has reached the preset storage space, or whether the number of rows of data has reached the preset number of rows threshold, etc.

[0047] Step S602: If the partition data unit is detected to meet the preset transmission conditions, the corresponding transmission thread of the partition data unit is determined from the transmission thread group, and the location information of the partition data unit is sent to the transmission queue of the corresponding thread.

[0048] Specifically, when the preset sending conditions are met, the corresponding thread, namely the target sending thread, is determined from the sending thread group, and the partition data unit is sent to the sending queue of the target sending thread, waiting for the target sending thread to send it.

[0049] Since the target sending thread can send more than one partition data unit, setting a sending queue allows the target sending thread to send the partition data units in the sending queue one by one until all partition data units in the sending queue have been sent.

[0050] Accordingly, step S303 above uses a target sending thread to send the partition data unit to the target database based on the node information and partition information of the partition data unit, which may include the following steps: Step S603: The target sending thread determines the location information of the partition data unit to be sent from its own sending queue, and obtains the corresponding partition data unit based on the location information of the partition data unit to be sent.

[0051] In this process, after the target sending thread determines the partition data unit to be sent from the sending queue, it obtains the location information of the partition data unit and then obtains the data in the corresponding partition data unit based on the location information.

[0052] Step S604: parse the partition data unit to determine the node information and partition information of the corresponding partition data unit.

[0053] The target sending thread parses the partition data unit to determine the corresponding node information and partition information, so that the data in the partition data unit can be sent to the target database based on the node information and partition information.

[0054] Optionally, based on node information and partition information, the partition data units are sent to the target database, including: Based on the node information, determine the link to the corresponding target database; Based on the link and partition information, the partition data unit is sent to the service thread of the target database. After receiving the data in the partition data unit, the service thread calls the storage layer interface to write the data in the partition data unit into the target database.

[0055] Specifically, the node information can be used to determine the link to the target database. Then, by setting the partition information in the link, the data in the partition data unit can be sent to the service thread of the target database, thereby realizing the data import of the partition data unit.

[0056] In this embodiment, the status of the partitioned data unit is monitored, so that the partitioned data unit is sent as a whole after the sending conditions are met, thereby achieving centralized sending and avoiding frequent calls to the sending thread.

[0057] For example, the yasldr import tool provided in this application can be executed independently on the client side without requiring any prior service deployment. Figure 7 and 8 The figures shown are schematic diagrams of the execution flow of the reading thread and the sending thread provided in Embodiment 3 of this application, respectively. When importing using the yasldr import tool, the yasldr processing flow is as follows: 1) The main control thread queries the control information of the table containing the data to be imported, including distribution information, partition information, column attribute information, distributed routing information, etc., and initializes the partition calculation context; 2) The main control thread splits the data file into segments of fixed block size for subsequent reader threads to process in parallel; 3) The main control thread creates the reader thread group and the sender thread group, and starts the thread groups; 4) The main control thread continuously waits for the reader thread group and sender thread group to finish processing; 5) The reader thread retrieves the segmented file blocks; 6) The reader thread reads a row of data and performs data parsing, type conversion, distribution key calculation, and partition key calculation according to the queried column attribute information; 7) The reader thread inserts the data into the corresponding partition data unit based on the calculated distribution key and partition key; 8) The reader thread checks if the partition unit meets the sending condition and writes the control information of the partition unit into the sender's sending queue; 9) The sender thread loops through the send queue, waiting to acquire the partition unit to be sent; 10) After obtaining the sending unit, the sender thread parses the node information and partition information to which the sending unit belongs; 11) The sender thread uses the connection corresponding to the node to set the partition information and then sends the partition unit to the database service thread; 12) After receiving the partition unit data, the database service thread reads the data and calls the storage layer interface to write the data in batches.

[0058] This application embodiment also provides a database data import device 90, which is applied to a preset import tool. The import tool is equipped with a main control thread and N processing threads, where N is an integer greater than 1. Please refer to [reference needed]. Figure 9 ,include: The data preprocessing module 910 is used to query and parse the metadata of the table to be imported using the main control thread, determine the control information of the table to be imported, and create a read thread group and a send thread group from N processing threads. The data reading and analysis module 920 is used to select any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table to be imported, determine the distribution information and row partition information of each row of data in the data to be imported based on the control information of the table to be imported, and insert the row data into the corresponding partition data unit based on the distribution information and row partition information. The data import and sending module 930 is used to select any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

[0059] Optionally, the data preprocessing module 910 includes: The data query unit is used to use the main control thread to query and extract the metadata of the table to be imported, and obtain the control information of the table to be imported. The control information extraction unit is used to determine the distribution key, partition key, and column attribute information of the table based on the control information of the table to be imported.

[0060] Optionally, the data reading and analysis module 920 includes: The data analysis and transformation unit is used to parse and convert the corresponding row data based on the column attribute information of the table to be imported, so as to obtain the transformed row data. The data key calculation unit is used to determine the distribution information of the converted row data based on the distribution key of the table to be imported, and to determine the row partition information of the converted row data based on the partition key of the table to be imported. Data reading and analysis module 920 includes: The data insertion unit is used to insert the transformed row data into the corresponding partition data unit based on the distribution information and row partition information.

[0061] Optionally, the data insertion unit includes: The partitioning determination sub-unit is used to determine the partitioned data unit based on the distribution information and row partitioning information; The row number allocation subunit is used to allocate row numbers from the partition data unit and increase the number of worker threads for the partition data unit. The number of worker threads is used to instruct the reading thread to determine whether to send the partition data unit into the sending queue of the target sending thread based on the number of worker threads. The data insertion sub-unit is used to insert the transformed row data into the partition data unit according to the assigned row number, and reduces the number of working threads of the partition data unit after the insertion is completed.

[0062] Optionally, database data import methods also include: The file splitting module is used to use the main control thread to query and parse the metadata of the table to be imported, determine the control information of the table to be imported, and then split the data to be imported according to the preset block size to obtain file block information of at least one file block. Data reading and analysis module 920 includes: The line data reading unit is used to obtain the file block information of any file block and use the target reading thread to read any line of data in the file block; The thread status marker unit is used to determine the distribution information and row partition information of each row of data in the data to be imported, based on the control information of the table to be imported.

[0063] Optionally, database data import methods also include: The sending detection module is used to detect whether the partition data unit meets the preset sending conditions after inserting the row data into the corresponding partition data unit according to the distribution information and row partition information. The information sending module is used to determine the sending thread corresponding to the partition data unit from the sending thread group if the partition data unit is detected to meet the preset sending conditions, and send the location information of the partition data unit to the sending queue of the corresponding thread. The data import and sending module 930 includes: The partition data acquisition unit is used to determine the location information of the partition data unit to be sent from its own sending queue using the target sending thread, and to acquire the corresponding partition data unit based on the location information of the partition data unit to be sent. The partition data parsing unit is used to parse the partition data unit and determine the node information and partition information of the corresponding partition data unit.

[0064] Optionally, the data import and sending module 930 includes: The link determination unit is used to determine the link to the corresponding target database based on the node information; The data import and sending unit is used to send partition data units to the service thread of the target database based on the link and partition information. After receiving the data in the partition data unit, the service thread calls the storage layer interface to write the data in the partition data unit into the target database.

[0065] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0066] This application also provides an electronic device 100, please refer to... Figure 10 It includes a memory 101 and a processor 102, wherein the memory 101 is used to store computer programs; the processor 102 is used to execute the programs stored in the memory 101 to implement the database data import method described in any embodiment of this application.

[0067] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the database data import method described in any embodiment of this application.

[0068] In this application, "multiple" refers to two or more.

[0069] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0070] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0071] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0072] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0073] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for importing data from a database, characterized in that, The database data import method is applied to a preset import tool, which has a main control thread and N processing threads, where N is an integer greater than 1. The database data import method includes: The main control thread is used to query and parse the metadata of the table containing the data to be imported, determine the control information of the table containing the data to be imported, and create a read thread group and a send thread group from the N processing threads. Use any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information. Use any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

2. The database data import method according to claim 1, characterized in that, The step of using the main control thread to query and parse the metadata of the table containing the data to be imported, and determining the control information of the table containing the data to be imported, includes: Using the main control thread, the metadata of the table containing the data to be imported is queried and extracted to obtain the control information of the table containing the data to be imported. Based on the control information of the table containing the data to be imported, determine the table's distribution key, partition key, and column attribute information.

3. The database data import method according to claim 2, characterized in that, The step of determining the distribution information and row partition information of each row of data in the data to be imported based on the control information of the table of the data to be imported includes: Based on the column attribute information of the table containing the data to be imported, the corresponding row data is parsed and its data type is converted to obtain the converted row data. Based on the distribution key of the table containing the data to be imported, determine the distribution information of the converted row data; based on the partition key of the table containing the data to be imported, determine the row partition information of the converted row data. The step of inserting the row data into the corresponding partition data unit according to the distribution information and the row partition information includes: Based on the distribution information and the row partition information, the transformed row data is inserted into the corresponding partition data unit.

4. The database data import method according to claim 3, characterized in that, The step of inserting the transformed row data into the corresponding partition data unit according to the distribution information and the row partition information includes: Based on the distribution information and the row partition information, the partition data unit is determined; A row number is assigned from the partition data unit, and the number of working threads for the partition data unit is increased. The number of working threads is used to instruct the reading thread to determine whether to send the partition data unit into the sending queue of the target sending thread based on the number of working threads. Based on the assigned row number, the converted row data is inserted into the partition data unit, and the number of working threads in the partition data unit is reduced after the insertion is completed.

5. The database data import method according to claim 1, characterized in that, After using the main control thread to query and parse the metadata of the table containing the data to be imported, and determining the control information of the table containing the data to be imported, the process further includes: The data to be imported is divided according to a preset block size to obtain file block information of at least one file block; The step of using the target reading thread to read the control information of the table containing the data to be imported, and determining the distribution information and row partition information of each row of data in the data to be imported based on the control information of the table containing the data to be imported, includes: For any file block, obtain the file block information of the file block, and use the target reading thread to read any line of data in the file block; Based on the control information of the table containing the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported.

6. The database data import method according to any one of claims 1 to 5, characterized in that, After inserting the row data into the corresponding partition data unit according to the distribution information and the row partition information, the method further includes: Detect whether the partition data unit meets the preset transmission conditions; If the partition data unit is detected to meet the preset sending conditions, the sending thread corresponding to the partition data unit is determined from the sending thread group, and the location information of the partition data unit is sent to the sending queue of the corresponding thread. The step of using the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit includes: The target sending thread determines the location information of the partition data unit to be sent from its own sending queue, and obtains the corresponding partition data unit based on the location information of the partition data unit to be sent. The partition data unit is parsed to determine the node information and partition information corresponding to the partition data unit.

7. The database data import method according to claim 6, characterized in that, The step of sending the partition data unit to the target database based on the node information and the partition information includes: Based on the node information, determine the link to the corresponding target database; Based on the link and the partition information, the partition data unit is sent to the service thread of the target database. After receiving the data in the partition data unit, the service thread calls the storage layer interface to write the data in the partition data unit into the target database.

8. A database data import device, characterized in that, The database data import device is applied to a preset import tool, which has a main control thread and N processing threads, where N is an integer greater than 1. The database data import method includes: The data preprocessing module is used to query and parse the metadata of the table containing the data to be imported using the main control thread, determine the control information of the table containing the data to be imported, and create a read thread group and a send thread group from the N processing threads. The data reading and analysis module is used to select any available thread in the reading thread group as the target reading thread, use the target reading thread to read the control information of the table of the data to be imported, determine the distribution information and row partition information of each row of data in the data to be imported according to the control information of the table of the data to be imported, and insert the row data into the corresponding partition data unit according to the distribution information and the row partition information. The data import and sending module is used to select any available thread in the sending thread group as the target sending thread, and use the target sending thread to send the partition data unit to the target database according to the node information and partition information of the partition data unit.

9. An electronic device, characterized in that, Including processor and memory, among which, Memory, used to store computer programs; A processor for executing a program stored in memory to implement the database data import method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the database data import method according to any one of claims 1-7.

Images