A distributed database data loading method and system based on FTP server

Abstract

The present invention provides a distributed database data loading method and system based on an FTP server, which belongs to the field of file loading of a distributed database, and specifically comprises: calculating the number of data nodes participating in loading according to the file to be loaded, and slicing the file to be loaded according to the number of nodes; downloading multiple file slices of the file to be loaded in parallel by a file server through multiple data nodes corresponding to the number of nodes, and assembling them into data items and storing them in a ring buffer; reading data items from the ring buffer through multithreading and parsing the data items into database tuples; sending the database tuples to the data nodes corresponding to the storage location of the database tuples, and the corresponding data nodes writing the database tuples into the bottom storage of the distributed database. The present invention can realize fast batch loading of files on the file server, and effectively improve the data loading efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of file loading, and in particular to a distributed database data loading method and system based on FTP server. Background Art

[0002] With the rapid development of information technology, the data of enterprises has shown explosive growth. Single-node databases can no longer meet the needs of enterprises for data analysis and processing. Under this circumstance, large-scale distributed database clusters came into being.

[0003] As for the need to import large files into a distributed database, the following methods are generally adopted: (1) Import through a certain CN using a load command. This method does not fully utilize the resources of CN and DN; (2) Develop a separate tool to split the large file into multiple CNs for concurrent import, but all data must be processed by the CN once, which increases the network interaction and still has certain performance limitations; (3) Develop a loading server. The CN only needs to notify the DN of the information of the loading server. Subsequently, under the control of the loading server, the loading server and DN interact to complete the direct loading of data files and data distribution between DNs. The loading server needs to monitor the load status of the cluster, be responsible for parsing data files, and schedule and send them to each DN node in the form of data block transmission. This method is very efficient, but it is difficult to develop and maintain the loading server. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a distributed database loading method and system based on an FTP server.

[0005] The present invention provides a distributed database data loading method based on an FTP server, comprising:

[0006] S1: Calculate the number of data nodes participating in loading according to the file to be loaded, and slice the file to be loaded according to the number of nodes;

[0007] S2: The file server downloads multiple file fragments of the file to be loaded in parallel through multiple data nodes corresponding to the number of nodes, and assembles them into data items and stores them in the ring buffer;

[0008] S3: reading data items from the ring buffer through multiple threads and parsing the data items into database tuples;

[0009] S4: Send the database tuple to a data node corresponding to the database tuple storage location, and the corresponding data node writes the database tuple into the underlying storage of the distributed database.

[0010] According to a distributed database data loading method based on an FTP server provided by the present invention, in step S1, the number of slices of the file to be loaded is equal to the number of nodes.

[0011] According to a distributed database data loading method based on an FTP server provided by the present invention, step S2 further comprises:

[0012] S21: The file server downloads the file to be loaded according to the segment information of the file to be loaded, and obtains the downloaded file;

[0013] S22: Parse the downloaded file to obtain multiple data rows;

[0014] S23: adding header information to the multiple data rows respectively and assembling them to obtain data items;

[0015] S24: placing the data item at the tail of the queue of the ring buffer, so as to sequentially store the data items in the ring buffer.

[0016] According to a distributed database data loading method based on an FTP server provided by the present invention, when reading the data item in step S3, it is read from the queue head of the ring buffer.

[0017] According to a distributed database data loading method based on an FTP server provided by the present invention, in step S2, when all data items are stored in the ring buffer, a mark completion message is added to the tail of the queue of the ring buffer; in steps S3 and S4, when all data items and the mark completion message are read from the ring buffer, and the database tuples obtained by parsing all data items are sent to the corresponding data nodes, a completion message is sent.

[0018] According to a distributed database data loading method based on an FTP server provided by the present invention, step S4 also includes:

[0019] S41: When the data node receives the sending completion message, the data node encapsulates the database tuple received by the data node itself into a data message;

[0020] S42: Packing the data message and the sending completion message into a data packet, and sending the data packet to an underlying storage receiving queue for writing database tuples into a distributed database;

[0021] S43: When writing the database tuple into the underlying storage of the distributed database, the receiving queue is polled, and when a data packet appears in the receiving queue, the data packet is parsed to obtain the database tuple corresponding to the data packet, and the parsed database tuple is written into the underlying storage of the distributed database.

[0022] The present invention also provides a distributed database data loading system based on an FTP server, which is used to execute a distributed database loading method based on an FTP server as described in any one of the above, comprising:

[0023] Access control module: used to receive a loading command, and also used to calculate the number of data nodes participating in the loading according to the file to be loaded, and to slice the file to be loaded according to the number of nodes;

[0024] Overall control module: used to initialize multiple data nodes and ring buffers according to the number of shards calculated by the access control module;

[0025] Data download module: used for downloading multiple file fragments of the file to be loaded in parallel from the file server, and assembling them into data items and storing them in the ring buffer;

[0026] Data reading module: used for reading data items from the ring buffer through multiple threads and parsing the data items into database tuples;

[0027] Data distribution module: used for sending the database tuple obtained by parsing the data reading module to the data node corresponding to the database tuple storage location;

[0028] Data storage module: used to write database tuples received by multiple data nodes into the underlying storage of the distributed database in batches.

[0029] According to a distributed database data loading system based on an FTP server provided by the present invention, the data download module is implemented by a single thread, the data reading module is implemented by multiple threads, the data distribution module is implemented by multiple threads, and the data disk placement module is implemented by multiple threads.

[0030] According to a distributed database data loading system based on an FTP server provided by the present invention, the overall control module is also used to monitor the task completion status of the data download module, the data reading module, the data distribution module, and the data disk module.

[0031] The present invention provides a distributed database data loading method and system based on FTP server, which uses FTP server as file server, and the dynamic formulation and execution of data loading tasks and the download and analysis of files are all realized inside the database system, thereby improving the file loading efficiency. In addition, large files can be arbitrarily fragmented, which solves the limitation that they must be fragmented in whole.

[0032] The present invention can automatically and efficiently load data. The data loading process is directly participated by the DN node, and the data belonging to the DN can be directly stored on the disk, which effectively reduces network interaction and effectively utilizes system resources. In addition, the loading progress and system status can be monitored in real time. The entire implementation is completely realized in the database system, which makes the loading process more controllable and easy to deploy and implement.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 The present invention provides a flow chart of a distributed database data loading method based on an FTP server.

[0036] Figure 2 The present invention provides a schematic diagram of the structure of a distributed database data loading system based on an FTP server.

[0037] Reference numerals:

[0038] 100, access control module; 200, overall control module; 300, data download module; 400, data reading module; 500, data distribution module; 600, data disk module; 700, file server; 800, ring buffer. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0040] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0041] In order to better understand the present invention, the terms appearing in the present invention are explained in detail below.

[0042] FTP is a file server system. Users connect to the FTP server program on the remote host through a client program that supports the FTP protocol. Users send commands to the server program through the client program. The server program executes the commands issued by the user and returns the results of the execution to the client. For example, the user sends a command to ask the server to send a copy of a certain file to the user. The server will respond to this command and send the specified file to the user's machine. The client program receives the file on behalf of the user and stores it in the user's directory.

[0043] In addition, the distributed database system described in the present invention mainly involves multiple coordination nodes (coordinate) and data nodes (datanode) and a GTM node, wherein the coordination node (coordinate) is referred to as CN as the entrance for external applications to access the database system, and mainly formulates database query tasks. The data node (datanode) is referred to as DN, which is the computing and storage node of the distributed database and executes the query tasks of CN. GTM mainly provides a global transaction ID. The distributed database data loading method and system based on the FTP server mentioned below in the present invention mainly involve CN and DN nodes.

[0044] Combine the following Figure 1 to Figure 2 Embodiments of the present invention are described.

[0045] The present invention provides a distributed database data loading method based on an FTP server, comprising:

[0046] S1: Calculate the number of data nodes participating in loading according to the file to be loaded, and slice the file to be loaded according to the number of nodes;

[0047] In step S1, the number of shards of the file to be loaded is equal to the number of nodes.

[0048] Furthermore, in this stage, the loading command of the external application is first received, and the list and size of files to be loaded are obtained from the file server according to the configuration parameters. The number of DNs and thread concurrency required to download data from the file server are estimated. If the amount of data is not large, not all data nodes need to download data from the file server. Then the large file is sharded, and each DN participating in the download has at most one shard. Finally, the data file to be loaded or the large file shard is distributed to each data node to ensure that the load of each DN node participating in the download is as even as possible. Finally, the data loading task is generated and issued, and then the loading results are collected.

[0049] The specific sharding process is to estimate the number of data nodes that need to participate in data downloading and the number of concurrent requests for data loading; shard large files according to the number of data nodes participating in the download, and ensure that each data node participating in the download has at most one shard, and small files can be directly handed over to one data node; allocate the data files or large file shards to be loaded to the data nodes participating in the download, and allocate at least one file or large file shard.

[0050] S2: The file server downloads multiple file fragments of the file to be loaded in parallel through multiple data nodes corresponding to the number of nodes, and assembles them into data items and stores them in the ring buffer;

[0051] In this stage, we first download files or large file fragments from the file server (FTP server), parse the data, extract the data rows, assemble them into items and put them into the circular buffer. This is implemented in a single thread, and it has been proven that a single thread is sufficient.

[0052] Step S2 further comprises:

[0053] S21: The file server downloads the file to be loaded according to the segment information of the file to be loaded, and obtains the downloaded file;

[0054] S22: Parse the downloaded file to obtain multiple data rows;

[0055] S23: adding header information to the multiple data rows respectively and assembling them to obtain data items;

[0056] S24: placing the data item at the tail of the queue of the ring buffer, so as to sequentially store the data items in the ring buffer.

[0057] Furthermore, the specific process of step S2 is to start downloading data from the file server according to the file list responsible for downloading, and parse the data rows at the same time. Each data row needs to add corresponding header information to form an item (data item), and put it at the end of the shared circular queue. If the circular queue is full, the download is temporarily suspended waiting for idle notification.

[0058] S3: reading data items from the ring buffer through multiple threads and parsing the data items into database tuples;

[0059] When reading a data item in step S3, it is read from the queue head of the ring buffer.

[0060] After steps S1 and S2, we use each thread to read the data row from the head of the shared circular queue and parse it into the tuple (database tuple) finally accepted by the database. The tuple needs to be further encapsulated to form data messages and other messages. For example, if the message is completed, the message is packaged together into a packet and sent. That is, each packet contains multiple tuples or other messages.

[0061] S4: Send the database tuple to a data node corresponding to the database tuple storage location, and the corresponding data node writes the database tuple into the underlying storage of the distributed database.

[0062] Furthermore, it is determined whether the tuple belongs to the local or other data node. If it is processed locally, it is directly written to the disk. If it is processed by other data nodes, it is directly sent to the specified data node. The bottom layer reads and distributes the sent data and puts it into the corresponding receiving queue.

[0063] Then poll the receiving queue. If there is a packet, parse the tuple and put it into the batch disk cache. If the disk cache is full or the last tuple is directly written to the disk, writing to the disk means writing the data in batches to the distributed database.

[0064] In step S2, when all data items are stored in the ring buffer, a mark completion message is added to the tail of the queue of the ring buffer; in steps S3 and S4, when all data items and the mark completion message are read from the ring buffer, and the database tuples obtained by parsing all data items are sent to the corresponding data nodes, a completion message is sent.

[0065] Step S4 also includes:

[0066] S41: When the data node receives the sending completion message, the data node encapsulates the database tuple received by the data node itself into a data message;

[0067] S42: Packing the data message and the sending completion message into a data packet, and sending the data packet to an underlying storage receiving queue for writing database tuples into a distributed database;

[0068] S43: When writing the database tuple into the underlying storage of the distributed database, the receiving queue is polled, and when a data packet appears in the receiving queue, the data packet is parsed to obtain the database tuple corresponding to the data packet, and the parsed database tuple is written into the underlying storage of the distributed database.

[0069] Furthermore, starting from step S3, data is read from the head of the shared ring queue and parsed into a database tuple. The tuple needs to be further encapsulated to form a data message and other messages, such as a completion message, and then the messages are packaged together into a packet and sent, that is, each packet contains multiple tuples or other messages.

[0070] The present invention also provides a distributed database data loading system based on an FTP server, which is used to execute a distributed database loading method based on an FTP server as described in any one of the above, comprising:

[0071] Access control module 100: used to receive a loading command, and also used to calculate the number of data nodes participating in the loading according to the file to be loaded, and to slice the file to be loaded according to the number of nodes;

[0072] The overall control module 200 is used to initialize multiple data nodes and a ring buffer 800 according to the number of shards calculated by the access control module 100;

[0073] Data download module 300: used for downloading multiple file segments of a file to be loaded in parallel from file server 700, and assembling them into data items and storing them in ring buffer 800;

[0074] Data reading module 400: used to read data items from the ring buffer 800 through multiple threads and parse the data items into database tuples;

[0075] Data distribution module 500: used to send the database tuple obtained by parsing the data reading module 400 to the data node corresponding to the database tuple storage location;

[0076] Data write-to-disk module 600: used to write database tuples received by multiple data nodes into the underlying storage of the distributed database in batches.

[0077] The data download module 300 is implemented by a single thread, the data reading module 400 is implemented by multiple threads, the data distribution module 500 is implemented by multiple threads, and the data storage module 600 is implemented by multiple threads.

[0078] Furthermore, the access control module is responsible for generating file loading tasks and sending them to the master control modules of all DNs. Data nodes that do not participate in downloading data from the file server must also be sent, because all nodes must participate in data storage.

[0079] After receiving the data loading task, the overall control module extracts the file download list and file server information, initializes the data download module, data reading module, data distribution module and data disk module, initializes the circular buffer area, and passes the file download list and file server information to the data download module. Here, the data reading module, data distribution module and data disk module all start multi-threading. If the data node finds that there is no download list, it only needs to initialize the data disk module and establish the communication link between threads (within the node and between nodes);

[0080] The data download module starts downloading data from the file server according to the list of files to be downloaded, and parses the data rows at the same time. Each data row needs to add corresponding header information to form an Item and put it at the end of the shared circular queue. If the circular queue is full, the download will be temporarily suspended when waiting for idle notification;

[0081] The data reading module and the data distribution module read the data row from the head of the shared circular queue based on each thread and parse it into the tuple finally accepted by the database, and then determine whether the tuple belongs to the local or other data node. If it is processed locally, it is directly sent to the disk module. If it is processed by other data nodes, it is directly sent to the specified data node;

[0082] The data placement module will poll the receiving queue (the underlying layer will put the data sent by the data reading and distribution module into the corresponding receiving queue). If there is a packet, tuples will be parsed and placed in the batch placement cache. If the placement cache is full or the last tuple is directly placed on the disk.

[0083] The overall control module 200 is also used to monitor the task completion status of the data download module 300 , the data reading module 400 , the data distribution module 500 , and the data storage module 600 .

[0084] Furthermore, after downloading the data, the data download module marks the last item in the shared circular cache queue, and then cleans up and exits; the data reading module and the data distribution module send a completion message to the DN disk module after processing the last item through the last item mark of the shared circular cache queue, and also cleans up and exits; the data disk module marks the corresponding receiving queue as completed after receiving the completion message. If all receiving queues are completed, the last tuple is directly dropped to the disk and then exits; the main control module detects that all sub-modules have been completed and reports the loading results to CN, sends a loading completion message, and then performs the final cleanup; the access control module receives the loading completion message sent by the main control module and reports the loading results to the application module.

[0085] The data download process in some embodiments is described in detail below.

[0086] The data download module is responsible for downloading data files from the file server, parsing the data into data rows in string format, and then adding a length header to form an item and putting it at the end of the circular queue.

[0087] A key point here is the processing of large file shards. File sharding is based on length, so the first and last lines of the file slice may not be complete. The processing method here is to skip the first line of the file slice if it is incomplete. If the last line is incomplete, continue to download data until this line is complete, and then interrupt the continued download of the file. For example, if the 1st and 8th lines are incomplete, then when parsing, skip the 1st line directly, and parse the 8th line completely. In this way, from the perspective of the entire cluster, the entire data file is loaded without data loss. Therefore, in order to complete the last line, the data actually downloaded and processed by each data node should be slightly larger than the size of the shard sent.

[0088] The data distribution processing flow in some embodiments is described in detail below.

[0089] The data distribution module mainly obtains items from the shared ring queue, parses them into tuples, and then distributes them to the disk module. The sending method can directly adopt the existing database technology. Generally, a message header is added to each tuple. Multiple tuples form a larger packet and share a communication channel with other messages.

[0090] The following is a detailed description of the data storage process in some embodiments.

[0091] Each disk placement thread is independent of each other. Each thread and the corresponding thread of the distribution module has a logical receiving connection and a packet buffer queue. The packet consisting of tuples and control messages sent by the distribution subsystem is first cached in the packet buffer queue of the corresponding connection, realizing decoupling from the underlying communication.

[0092] The disk placement thread will batch poll the connection's receiving queue to parse out tuples and control messages. If it is a tuple, it will be directly placed in the disk placement cache. If it is an end message, it will mark the corresponding connection as received. After all receiving connections are completed, the final disk placement will be performed, and then clean up and exit.

[0093] The present invention provides a distributed database loading method and system based on FTP server. Through an access control module located in CN, a general control module, a data download module, a data reading module, a data distribution module and a data disk placement module located in a DN node, a common FTP server is used as a file server, and the dynamic formulation and execution of data loading tasks and the download and analysis of files are all realized inside the database system, thereby improving the file loading efficiency. In addition, large files can be arbitrarily fragmented, thus solving the limitation that they must be fragmented in full. Generally speaking, fast batch loading of files on the file server is realized, thereby effectively improving the data loading efficiency.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed database data loading method based on FTP server, characterized in that: include: S1: Calculate the number of data nodes participating in loading according to the file to be loaded, and slice the file to be loaded according to the number of nodes; S2: the file server downloads multiple file fragments of the file to be loaded in parallel through multiple data nodes corresponding to the number of nodes, and assembles them into data items and stores them in the ring buffer; wherein, when all data items are stored in the ring buffer, a mark completion message is added to the tail of the queue of the ring buffer; S3: reading data items from the ring buffer through multiple threads and parsing the data items into database tuples; S4: sending the database tuple to the data node corresponding to the database tuple storage location, and the corresponding data node writes the database tuple into the underlying storage of the distributed database; wherein, when all data items and the marking completion message are read from the ring buffer, and the database tuple obtained by parsing all data items is sent to the corresponding data node, the sending completion message is sent, specifically including: S41: When the data node receives the sending completion message, the data node encapsulates the database tuple received by the data node itself into a data message; S42: Packing the data message and the sending completion message into a data packet, and sending the data packet to an underlying storage receiving queue for writing database tuples into a distributed database; S43: When writing the database tuple into the underlying storage of the distributed database, the receiving queue is polled, and when a data packet appears in the receiving queue, the data packet is parsed to obtain the database tuple corresponding to the data packet, and the parsed database tuple is written into the underlying storage of the distributed database.

2. A distributed database data loading method based on FTP server according to claim 1, characterized in that: In step S1, the number of shards of the file to be loaded is equal to the number of nodes.

3. A distributed database data loading method based on FTP server according to claim 1, characterized in that: Step S2 further comprises: S21: The file server downloads the file to be loaded according to the segment information of the file to be loaded, and obtains the downloaded file; S22: Parse the downloaded file to obtain multiple data rows; S23: adding header information to the multiple data rows respectively and assembling them to obtain data items; S24: placing the data item at the tail of the queue of the ring buffer, so as to sequentially store the data items in the ring buffer.

4. A distributed database data loading method based on FTP server according to claim 3, characterized in that: When reading a data item in step S3, it is read from the queue head of the ring buffer.

5. A distributed database data loading system based on an FTP server, used to execute a distributed database loading method based on an FTP server as claimed in any one of claims 1 to 4, characterized in that: include: Access control module: used to receive a loading command, and also used to calculate the number of data nodes participating in the loading according to the file to be loaded, and to slice the file to be loaded according to the number of nodes; Overall control module: used to initialize multiple data nodes and ring buffers according to the number of shards calculated by the access control module; Data download module: used for downloading multiple file fragments of the file to be loaded in parallel from the file server, and assembling them into data items and storing them in the ring buffer; wherein, when all data items are stored in the ring buffer, a mark completion message is added to the tail of the queue of the ring buffer; Data reading module: used for reading data items from the ring buffer through multiple threads and parsing the data items into database tuples; Data distribution module: used for sending the database tuple obtained by parsing the data reading module to the data node corresponding to the database tuple storage location; Data disk module: used to write the database tuples received by multiple data nodes into the underlying storage of the distributed database in batches; wherein, when all data items and the marking completion message are read by the ring buffer, and the database tuples obtained by parsing all data items are sent to the corresponding data nodes, the sending completion message is sent, specifically including: S41: When the data node receives the sending completion message, the data node encapsulates the database tuple received by the data node itself into a data message; S42: Packing the data message and the sending completion message into a data packet, and sending the data packet to an underlying storage receiving queue for writing database tuples into a distributed database; S43: When writing the database tuple into the underlying storage of the distributed database, the receiving queue is polled, and when a data packet appears in the receiving queue, the data packet is parsed to obtain the database tuple corresponding to the data packet, and the parsed database tuple is written into the underlying storage of the distributed database.

6. A distributed database data loading system based on FTP server according to claim 5, characterized in that: The data download module is implemented by a single thread, the data reading module is implemented by multiple threads, the data distribution module is implemented by multiple threads, and the data disk module is implemented by multiple threads.

7. A distributed database data loading system based on FTP server according to claim 5, characterized in that: The overall control module is also used to monitor the task completion status of the data download module, the data reading module, the data distribution module, and the data storage module.

Images