File import method for distributed database

Abstract

The present specification provides a file import method of a distributed database, comprising: a first thread reading buffer segments of a database file from a buffer in response to a database import task of the database file, each buffer segment containing partial data of the database file; for each buffer segment read by the first thread: performing a delimiting operation to determine complete data and remaining incomplete data in the buffer segment, creating a corresponding database import subtask for the complete data and assigning the database import subtask to a subtask processing thread group, and recording the remaining incomplete data for use in forming complete data in a next buffer segment read by the first thread; and a second thread processing an assigned database import subtask to send complete data indicated by the database import subtask to a corresponding node in the distributed database if it is determined that the second thread belongs to the subtask processing thread group.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for importing files into a distributed database. Background Technology

[0002] Data import is a common function in databases, often used in scenarios such as data migration and regression testing. When using database files for data import in distributed databases, data from the same database file may be written to different nodes. Furthermore, as the scale of the distributed database and the size of the database files increase, the requirements for data import efficiency for each database file also become increasingly stringent.

[0003] In related technologies, database files are usually pre-splitted to generate several smaller files, and multiple clients import these smaller files to improve data import efficiency. However, splitting database files will cause additional resource consumption and storage space occupation, which greatly limits data import efficiency. Summary of the Invention

[0004] In view of this, this specification provides a method and apparatus for importing files into a distributed database to address the shortcomings of related technologies.

[0005] Specifically, this specification is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this specification, a method for importing files into a distributed database is provided, comprising:

[0007] In response to a database import task for a database file, the first thread reads buffered fragments of the database file from the buffer, each buffered fragment containing a portion of the data in the database file;

[0008] For each buffer segment read, the first thread performs a delimiting operation to determine the complete data and the remaining incomplete data in the buffer segment, creates a corresponding database import subtask for the complete data and assigns it to the subtask processing thread group, and records the remaining incomplete data to form the complete data in the next buffer segment read by the first thread.

[0009] When the second thread determines that it belongs to the subtask processing thread group, it processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

[0010] According to a second aspect of the embodiments of this specification, a method for importing files into a distributed database is provided, comprising:

[0011] In response to a database import task for a database file, the first thread reads buffered fragments of the database file from the buffer, each buffered fragment containing a portion of the data in the database file;

[0012] For each buffer segment read, the first thread performs a delimiting operation to determine the complete data and remaining incomplete data in the buffer segment; creates a corresponding database import subtask for the complete data and assigns it to a subtask processing thread group, so that the second thread, upon determining that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and records the remaining incomplete data to form the complete data in the next buffer segment read by the first thread.

[0013] According to a third aspect of the embodiments of this specification, a file import apparatus for a distributed database includes:

[0014] A buffer fragment reading unit is configured to enable a first thread to read buffer fragments of the database file from a buffer in response to a database import task for the database file, each buffer fragment containing a portion of the data in the database file;

[0015] The buffer fragment processing unit is configured to enable the first thread to perform a delimiting operation for each buffer fragment read to determine the complete data and the remaining incomplete data in the buffer fragment, create a corresponding database import subtask for the complete data and assign it to the subtask processing thread group, and record the remaining incomplete data to form the complete data in the next buffer fragment read by the first thread.

[0016] The database import subtask processing unit is used to enable the second thread to process the assigned database import subtask when it determines that it belongs to the subtask processing thread group, so as to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

[0017] According to a fourth aspect of the embodiments of this specification, a file import apparatus for a distributed database includes:

[0018] A buffer fragment reading unit is configured to enable a first thread to read buffer fragments of the database file from a buffer in response to a database import task for the database file, each buffer fragment containing a portion of the data in the database file;

[0019] The buffer fragment processing unit is configured to enable the first thread to perform a delimitation operation for each buffer fragment read, to determine the complete data and the remaining incomplete data in the buffer fragment, to create a corresponding database import subtask for the complete data and assign it to a subtask processing thread group, so that the second thread, upon determining that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and to record the remaining incomplete data for use in constructing the complete data in the next buffer fragment read by the first thread.

[0020] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first and second aspects.

[0021] According to a sixth aspect of the embodiments of this specification, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first and second aspects.

[0022] In the technical solution provided in this specification, by performing a delimitation operation on the corresponding buffer segments in the buffer, the first thread can simultaneously read the database file and split the entire database file, avoiding the resource consumption required for splitting the file separately and the disk space required for storing the split file. At the same time, the database import subtask based on the subtask processing thread group processes the corresponding complete data, which improves the efficiency of importing the data of the database file into the corresponding node of the distributed database.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a schematic diagram of the architecture of a file import system for a distributed database, as illustrated in an exemplary embodiment of this specification.

[0026] Figure 2This is a flowchart illustrating an exemplary embodiment of a file import method for a distributed database as shown in this specification.

[0027] Figure 3 This is a schematic diagram illustrating a delimitation operation in an exemplary embodiment of this specification;

[0028] Figure 4 This is a schematic diagram illustrating another delimitation operation as shown in an exemplary embodiment of this specification;

[0029] Figure 5 This is a schematic diagram illustrating an exemplary embodiment of importing a database file into a distributed database, as shown in this specification.

[0030] Figure 6 This is a flowchart illustrating another method for importing files into a distributed database, as shown in an exemplary embodiment of this specification.

[0031] Figure 7 This is a schematic structural diagram of an electronic device shown in an exemplary embodiment of this specification;

[0032] Figure 8 This is a schematic diagram illustrating the structure of a file import device for a distributed database, as shown in an exemplary embodiment of this specification.

[0033] Figure 9 This is a schematic diagram illustrating the structure of another file import device for a distributed database, as shown in an exemplary embodiment of this specification. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification.

[0035] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “described,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] In related technologies, the data import function can be manifested as a database importing relevant data based on the content of an external file. Depending on the database architecture, this data import function can be applied to both single-machine databases and distributed databases. The data import speed in these two scenarios will be affected by different factors: In the first scenario, the data import speed mainly depends on the performance of the database's write-to-storage media; in the second scenario, assuming the database uses a partitioned table design, the data in the partitioned table may be distributed across different nodes in the distributed database. This allows the data corresponding to the external file to be sharded and sent to nodes using the thread described below. Therefore, the data import speed mainly depends on the sharding efficiency of the thread. The specific definition and implementation of partitioned tables have been largely disclosed in related technologies, and will not be elaborated upon in this application.

[0038] However, regarding the second scenario mentioned above: as the scale of the distributed database gradually increases, the number of corresponding nodes and the number of partitions in the partition table will also increase accordingly. This makes the thread performing data sharding operations on the aforementioned external files a bottleneck for data import speed. Furthermore, the multiple file fragments generated after sharding operations will occupy a significant amount of additional storage space, further raising the bar for efficient data import into the distributed database. Based on this, this specification proposes the following technical solutions to address the above problems.

[0039] Figure 1 This is a schematic diagram illustrating the architecture of a file import system for a distributed database, as shown in an exemplary embodiment of this specification. Figure 1 As shown, the above system includes a distributed database 11 and a client 12.

[0040] The distributed database 11 contains M nodes to store corresponding database data. Each node can be configured as a server connected to the client 12, and each node can be a physical server containing an independent host, or a virtual server hosted in a host cluster. Here, M is a positive integer greater than 1. During system operation, any node in the distributed database 11 can act as a server, responding to instructions initiated by the client 12 to perform corresponding database data operations on any of the aforementioned nodes.

[0041] Client 12 is an electronic device that can access nodes of distributed database 11. Client 12 runs a first thread 121 and a subtask thread group 122 containing N second threads. Here, N is a positive integer greater than or equal to 1. During system operation, the first thread 121 can read a database file (i.e., the aforementioned external file) and distribute its data to the subtask thread group 122 in the form of multiple database import subtasks. This allows threads in the subtask thread group 122 to send the complete data indicated by the database import subtask to the corresponding node in distributed database 11.

[0042] Although Figure 1 The distributed database 11 and the client 12 are drawn independently, but in reality, the client 12 can be deployed on any node in the distributed database 11. Of course, the client 12 can also be deployed on an electronic device independent of all nodes. Such electronic devices can be, for example, mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), and one or more embodiments in this specification do not limit this.

[0043] The following is combined with Figure 2 The embodiments shown illustrate the technical solutions of this specification. Figure 2 This is a flowchart illustrating an exemplary embodiment of a file import method for a distributed database, as shown in this specification. Figure 2 As shown, the method may include the following steps:

[0044] S201, in response to a database import task for a database file, the first thread reads buffered fragments of the database file from a buffer, each buffered fragment containing a portion of the data in the database file.

[0045] When a user actively triggers the process or the aforementioned client automatically triggers other tasks to import data from a database file into a distributed database, the client containing the first thread can initiate a database import task for that first thread. This allows the first thread to respond to the database import task, determine the database file, and read the corresponding buffer in memory. Since the buffer is typically small, even if all the space in the buffer is used to store the database file, the buffer can only store a fragment containing a portion of the data from the database file. The database file can originate from other databases, such as being exported from another database, or it can have any other arbitrary source; this specification does not impose any restrictions on this.

[0046] S202, for each buffer segment read, the first thread performs a delimitation operation to determine the complete data and remaining incomplete data in the buffer segment. For the complete data, a corresponding database import subtask is created and assigned to the subtask processing thread group. The remaining incomplete data is recorded to form the complete data in the next buffer segment read by the first thread. Due to the uncertainty of the database file size and the fixed storage space of the buffer itself, each buffer segment read by the first thread may contain incomplete data, leading to data loss. Therefore, the first thread needs to perform the delimitation operation on the buffer segments to determine the complete data and remaining incomplete data, and assign the complete data as a separate database import subtask to the subtask processing thread group. Specifically, the delimitation operation can result in either of the following: 1. The buffer segment contains only complete data; 2. The buffer segment contains both complete and incomplete data. Here, "incomplete data" can refer to a portion of the buffer segment containing data that cannot satisfy the minimum unit of data required for database execution (e.g., a row of data in a table). Regarding the second result mentioned above, the data can be further divided based on its storage location in the buffer segment. For example, if the buffer stores data sequentially from head to tail (assuming low-order address to high-order address) according to the storage order, then the buffer segment may contain "complete data - incomplete data", "incomplete data - complete data", and "incomplete data - complete data - incomplete data". Those skilled in the art will understand that, regardless of the situation, the presence of incomplete data in one buffer segment necessarily implies the presence of incomplete data in another buffer segment. Specifically, if incomplete data A exists at the tail of the buffer corresponding to the current buffer segment, then the head of the buffer corresponding to the next buffer segment read by the first thread will also contain another incomplete data B corresponding to incomplete data A. Therefore, the first thread can record the incomplete data obtained by performing a delimitation operation in the buffer segment and use the recorded incomplete data and the incomplete data in the next buffer segment read by the first thread to form complete data.

[0047] In one embodiment, if the first thread determines that there is residual incomplete data in the read buffer segment, it can add the incomplete data to the head of the buffer corresponding to the next buffer segment read by the first thread. In this embodiment, the incomplete data that should have been at the head of the corresponding buffer in the next buffer segment will, together with the added incomplete data, constitute new complete data. Figure 3For example, the diagram shows two buffers: a first buffer and a second buffer. The first buffer stores complete data at its head, while the non-first buffer stores incomplete data at its tail. The first thread can then add the incomplete data from the tail of the first buffer to the head of the second buffer, thus ensuring that the original incomplete data in the second buffer is continuous with the added incomplete data, which facilitates the formation of complete data in the second buffer.

[0048] In real-world scenarios, due to limitations such as cost, the number of buffers is not unlimited; instead, it consists of a fixed number of buffers. Furthermore, once the database import subtask corresponding to any buffer is completed, that buffer can be reused, i.e., a new buffer segment can be written for the first thread's reading. Therefore, the buffer corresponding to the current buffer segment read by the first thread may be the same as or different from the buffer corresponding to the next buffer segment read.

[0049] In one embodiment, the buffer corresponding to the current buffer segment read by the first thread is called the first buffer, and the buffer corresponding to the next buffer segment read is called the second buffer. If the first buffer and the second buffer are the same, the remaining incomplete data is moved to the head of the current buffer; if the first buffer and the second buffer are different, the remaining incomplete data is written to the head of the second buffer and cleared from the first buffer. Figure 4 For example, in this embodiment, if the first buffer and the second buffer are the same, then the remaining incomplete data in the buffer only needs to be moved from the tail to the head of the corresponding buffer. The cases where the two are different have already been discussed in the previous embodiment. Figure 3 This will be discussed in more detail here.

[0050] As mentioned earlier, the database files described above can be manually created and obtained according to the specific rules outlined above, or actively exported from the database. These specific rules are determined by the data types and formats within the database files. Based on different data structure classifications, the data types can be categorized into three types: structured data, unstructured data, and semi-structured data. Databases inherently possess a natural regularity in their data storage and arrangement, thus qualifying as structured data. Unstructured data, with its irregular or incomplete structure and lack of a predefined data model (such as audio or video), is unsuitable for use as database files. Semi-structured data falls between fully structured and unstructured data; it does not conform to the data model structure of relational databases or other tables, but contains relevant tags to separate semantic elements and hierarchically categorize records and fields. Therefore, it is also known as a self-describing structure, where data format and content are mixed together without clear distinction, making it suitable for importing its contents into the database.

[0051] In one embodiment, the data in the aforementioned database file can be semi-structured data. The first thread can determine whether the currently read buffer segment is complete according to a preset data integrity rule. The data integrity rule includes: if the end of the buffer corresponding to the current buffer segment read by the first thread is a line separator, the data in the buffer segment is determined to be complete; otherwise, the data in the buffer segment is determined to be incomplete, and a delimiting operation is performed based on the determination result. The aforementioned integrity rule corresponds to the aforementioned specific rule: for example, a specific rule corresponding to a database file stipulates that each line of data in the database file is considered a complete data entry, then the corresponding integrity rule can be determined based on whether the end of the buffer corresponding to the current buffer segment is a line separator. Those skilled in the art will understand that the aforementioned database file can be in formats such as Comma-Separated Values ​​(CSV), JavaScript Object Notation (JSON), and Extensible Markup Language (XML). Of course, the data integrity rules for different database file formats are basically the same. The main difference lies in the form of the line separator and the way to judge the corresponding form. For example, the line separator for a CSV file can be "\n", the line separator for a JSON file can be the corresponding right bracket "}", and the line separator for an XML file can be the corresponding closing tag. This manual does not impose any restrictions on this.

[0052] In summary, since the above-mentioned delimiting operation is performed by the first thread on the buffer fragments of the buffer according to the above-mentioned data integrity rules, the semi-structured data is completely divided into multiple complete data during the reading process, and is directly processed by the second thread below to be sent to the corresponding database. Therefore, the problem that some database files containing semi-structured data cannot be directly split in related technologies, as well as the problem that the split files need to occupy additional disk space, are solved.

[0053] S203, when the second thread determines that it belongs to the subtask processing thread group, it processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

[0054] Once the first thread completes the delimitation operation for a specific buffer segment and creates a corresponding database import subtask for the complete data, assigning it to the subtask processing thread group, the database import subtask can be processed by the second thread within that group. Specifically, the second thread can send the complete data indicated by the database import subtask to the corresponding node in the distributed database. The distributed database can incorporate the partitioned table design described above, allowing the database files to be partitioned into different partitions.

[0055] In one embodiment, the database file involves multiple partitions of the distributed database on different nodes. The second thread can parse and convert the complete data, and perform partition calculations on the processed complete data to obtain partition data fragments corresponding to each partition. Simultaneously, each partition data fragment is serialized, and the serialized partition data fragments are sent to the corresponding nodes, so that each node imports the received partition data fragments into the distributed database. Since the second thread only needs to process the data indicated in the database import subtask (i.e., a portion of the database file), the number of threads in the subtask processing thread group can be appropriately increased to ensure that more threads process the data of the same database file simultaneously, thereby achieving the effect of parallel importing large-volume database files. Combined with the delimitation method for semi-structured files such as CSV described above by the first thread, the solution in this embodiment can solve the problem of parallel import of large-volume semi-structured files in related technologies, greatly improving the import efficiency of such files. Furthermore, the principles and implementation methods of data parsing, type conversion, partition calculation, and serialization techniques have been largely disclosed in related technologies and will not be repeated here.

[0056] During the process of the second thread sending the serialized partition data fragments to the corresponding nodes, the method of sending the partition data fragments by the second thread can be changed according to actual needs.

[0057] In one embodiment, the second thread can directly send the partition data fragment to the node where the corresponding partition is located, thereby ensuring the real-time nature of the data in the corresponding database.

[0058] In another embodiment, if the user does not require high real-time performance from the database, the second thread can store the partition data fragments in a preset space corresponding to the respective partition. When the preset space meets preset sending conditions, the partition data fragments in the preset space are sent in batches to the node where the corresponding partition resides. In this embodiment, the batch sending mechanism avoids overly frequent interactions between the client and the node, improving the efficiency of sending partition data fragments in a single operation. The preset sending conditions may include the amount of data already stored in the preset space reaching a preset threshold, or the storage duration of the earliest stored data in the preset space exceeding a preset duration, etc. This specification does not impose any limitations on these conditions.

[0059] The solution in this specification can be configured to change the way the local device stores the database file by setting the acquisition method of the above-mentioned buffer fragments according to the actual situation. For example, the buffer fragment can actually be the data written into the buffer after the first thread performs streaming read of the database file. The streaming read only requires reading the file on the disk or network once. Compared with the related technology, which requires multiple scans of the corresponding database file to parse semi-structured data, it has lower IO overhead and avoids the problem that IO becomes the bottleneck of data import speed when the database file is in a network NFS or object storage system.

[0060] The following is combined Figure 5 The paper then discusses the above file import method using the example of importing data from a CSV format database file into a distributed database.

[0061] In one embodiment, assume a client initiates a database import task for a CSV format database file (filenamed "databasefile.csv"). The content of the database file is:

[0062] User ID 1, User Name 1, Household Registration 1, Contact Information 1;

[0063] User ID2, User Name2, Household Registration2, Contact Information2;

[0064] User ID 3, User Name 3, Household Registration 3, Contact Information 3;

[0065] ...

[0066] User ID 99, User Name 99, Household Registration 99, Contact Information 99;

[0067] User ID 100, User Name 100, Household Registration 100, Contact Information 100;

[0068] Correspondingly, assuming the aforementioned distributed database stores a "user information table," and this distributed database splits the user information table into three partitions—"Partition 1," "Partition 2," and "Partition 3"—based on hash partitioning, with each partition corresponding to a different node. The rules for these hash partitions stipulate that records where the user ID divided by 3 leaves a remainder of 1 are stored in Partition 1; records where the user ID divided by 3 leaves a remainder of 2 are stored in Partition 2; and records where the user ID divided by 3 leaves a remainder of 0 are stored in Partition 3. In one embodiment, as... Figure 5 The first thread, in response to the database import task for the aforementioned database file, reads the first buffer segment of the database file from the first buffer to obtain: "User ID1, User Name1, Household Registration1, Contact Information1".

[0069] ...

[0070] User ID 50, User Name 50, Household Registration 50, Contact Information 50;

[0071] User ID 51, User Name 51, Account

[0072] Obviously, the first thread can determine that the data in the first buffer segment is incomplete by using the data integrity rule corresponding to the delimiting operation (assuming the rule is: if the buffer segment corresponding to the current buffer segment read by the first thread ends with a semicolon, the data in the buffer segment is determined to be complete; otherwise, the data in the buffer segment is determined to be incomplete). Based on this rule, the data from the beginning of the first buffer segment to the last semicolon is taken as complete data (i.e., "User ID1, User Name1, Household Registration1, Contact Information1; ... User ID50, User Name50, Household Registration50, Contact Information50;"), and the data after the complete data is taken as the remaining incomplete data (i.e., "User ID51, User Name51, Household Registration").

[0073] For the complete data of the first buffer segment mentioned above, the first thread can create a corresponding database import subtask and assign it to the subtask processing thread group. Assuming that there are two idle threads, Thread 1 and Thread 2, in the subtask processing thread group, and further assuming that the subtask processing thread group specifically assigns the database import subtask to Thread 1 for execution, Thread 1 can then perform a series of operations on the complete data corresponding to the database import subtask (i.e., "User ID1, User Name1, Household Registration1, Contact Information1; ... User ID50, User Name50, Household Registration50, Contact Information50;"), including parsing, data type conversion, and partition calculation, to obtain the following partition data segments corresponding to the three partitions: Partition data segment corresponding to partition 1 ( This includes data segments for user ID1, user name1, household registration1, contact information4; user ID4, user name4, household registration4, contact information4; ... user ID49, user name49, household registration49, contact information49;"), partition data segments for partition 2 (i.e., user ID2, user name2, household registration2, contact information2; user ID5, user name5, household registration5, contact information5; ... user ID50, user name50, household registration50, contact information50;) and partition data segments for partition 3 (i.e., user ID3, user name3, household registration3, contact information3; user ID6, user name6, household registration6, contact information6; ... user ID48, user name48, household registration48, contact information48;)). After serializing each partition data segment, it can be sent to the node where the corresponding partition is located.

[0074] While the second thread 1 processes the complete data, the first thread can write the remaining incomplete data in the first buffer segment to the head of the second buffer and clear it from the first buffer. On this basis, the data subsequently read by the first thread can be concatenated with the incomplete data (i.e., "User ID 51, User Name 51, Household Registration 51, Contact Information 51; ... User ID 100, User Name 100, Household Registration 100, Contact Information 100;").

[0075] For the complete data of the second buffer segment mentioned above, the first thread can create a corresponding database import subtask and assign it to the subtask processing thread group. Assuming that the second thread 1 is still processing, the subtask processing thread group can specifically assign the database import subtask to the second thread 2 for execution. At this time, the second thread 2 can perform a series of operations such as parsing, data type conversion processing, and partition calculation on the complete data corresponding to the database import subtask (i.e., "User ID 51, User Name 51, Household Registration 51, Contact Information 51; ... User ID 100, User Name 100, Household Registration 100, Contact Information 100;") to obtain the following partition data segments corresponding to the three partitions: the partition data segment corresponding to partition 1 (i.e., "User ID 52, User Name 52, Household Registration 52, Contact Information 52; User ID 55, User Name 55, Household Registration 55, Contact Information 55; ... User ID 100, User Name 10;") The data segments are: 0, Household Registration 100, Contact Information 100;”, corresponding to partition 2 (i.e., “User ID 53, User Name 53, Household Registration 53, Contact Information 53; User ID 56, User Name 56, Household Registration 56, Contact Information 56; ... User ID 98, User Name 98, Household Registration 98, Contact Information 98;”), and corresponding to partition 3 (i.e., “User ID 51, User Name 51, Household Registration 51, Contact Information 51; User ID 54, User Name 54, Household Registration 54, Contact Information 54; ... User ID 99, User Name 99, Household Registration 99, Contact Information 99;”). After serializing each partition data segment, it can be sent to the node where the corresponding partition is located.

[0076] The second thread 1 and second thread 2 described above can process the complete data from the first and second buffer segments in parallel, thus significantly improving the import efficiency for a single database file (i.e., "database file.csv"). Ultimately, this allows each partition to be correctly imported with its corresponding data (see details in...). Figure 5 bottom).

[0077] Figure 6 This is a flowchart illustrating another method for importing files into a distributed database, as shown in an exemplary embodiment of this specification. Figure 6 As shown, this method is applied to the first thread and includes the following steps:

[0078] S601, in response to a database import task for a database file, the first thread reads buffered fragments of the database file from a buffer, each buffered fragment containing a portion of the data in the database file.

[0079] As mentioned above, when a user actively triggers the process or when the aforementioned client executes other tasks, pre-defined conditions are automatically triggered to import the data contained in the database file into the distributed database. The client containing the first thread can then initiate a database import task for that first thread. This allows the first thread to respond to the database import task, determine the database file, and read the corresponding buffer in memory. Since the buffer space is typically small, even if all the space in a buffer is used to store the database file, the buffer can only store a buffer fragment containing a portion of the data from the database file. The database file can originate from other databases, such as being exported from another database, or it can have any other arbitrary source; this specification does not impose any restrictions on this.

[0080] S602, for each buffer segment read, the first thread performs a delimitation operation to determine the complete data and the remaining incomplete data in the buffer segment; creates a corresponding database import subtask for the complete data and assigns it to the subtask processing thread group, so that the second thread, upon determining that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and records the remaining incomplete data to form the complete data in the next buffer segment read by the first thread.

[0081] As mentioned earlier, due to the uncertainty of the database file size and the fixed storage space of the buffer itself, each buffer segment read by the first thread may contain incomplete data, leading to data loss. Therefore, the first thread needs to perform the aforementioned delimitation operation on the buffer segments to determine the complete data and the remaining incomplete data within the buffer segment, and then assign the complete data as a separate database import subtask to the aforementioned subtask processing thread group. Specifically, the delimitation operation can result in either of the following: 1. The buffer segment contains only complete data; 2. The buffer segment contains both complete and incomplete data. Here, "incomplete data" can refer to a portion of the buffer segment containing data that does not meet the minimum requirement for database execution (e.g., a row of records in a data table). Regarding the second result mentioned above, the data can be further divided based on its storage location in the buffer segment. For example, if the buffer stores data sequentially from head to tail (assuming low-order address to high-order address) according to the storage order, then the buffer segment may contain "complete data - incomplete data", "incomplete data - complete data", and "incomplete data - complete data - incomplete data". Those skilled in the art will understand that, regardless of the situation, the presence of incomplete data in one buffer segment necessarily implies the presence of incomplete data in another buffer segment. Specifically, if incomplete data A exists at the tail of the buffer corresponding to the current buffer segment, then the head of the buffer corresponding to the next buffer segment read by the first thread will also contain another incomplete data B corresponding to incomplete data A. Therefore, the first thread can record the incomplete data obtained by performing a delimitation operation in the buffer segment and use the recorded incomplete data and the incomplete data in the next buffer segment read by the first thread to form complete data.

[0082] As described above, in one embodiment, if the first thread determines that there is residual incomplete data in the read buffer segment, it can add the incomplete data to the head of the buffer corresponding to the next buffer segment read by the first thread. In this embodiment, the incomplete data that should have been at the head of the corresponding buffer in the next buffer segment will, together with the added incomplete data, constitute new complete data. Figure 3 For example, the diagram shows two buffers: a first buffer and a second buffer. The first buffer stores complete data at its head, while the non-first buffer stores incomplete data at its tail. The first thread can then add the incomplete data from the tail of the first buffer to the head of the second buffer, thus ensuring that the original incomplete data in the second buffer is continuous with the added incomplete data, which facilitates the formation of complete data in the second buffer.

[0083] As mentioned earlier, in real-world scenarios, due to limitations such as cost, the number of buffers is not unlimited but consists of a fixed number of buffers. Furthermore, once the database import subtask corresponding to any buffer is completed, that buffer can be reused, i.e., a new buffer segment can be written for the first thread's reading. Therefore, the buffer corresponding to the current buffer segment read by the first thread may be the same as or different from the buffer corresponding to the next buffer segment read.

[0084] As described above, in one embodiment, the buffer corresponding to the current buffer segment read by the first thread is the first buffer, and the buffer corresponding to the next buffer segment read is the second buffer. If the first buffer and the second buffer are the same, the remaining incomplete data is moved to the head of the current buffer; if the first buffer and the second buffer are different, the remaining incomplete data is written to the head of the second buffer and cleared from the first buffer. Figure 4 For example, in this embodiment, if the first buffer and the second buffer are the same, then the remaining incomplete data in the buffer only needs to be moved from the tail to the head of the corresponding buffer. The cases where the two are different have already been discussed in the previous embodiment. Figure 3 This will be discussed in more detail here.

[0085] As mentioned earlier, the database files described above can be manually created and obtained according to the specific rules outlined above, or actively exported from the database. These specific rules are determined by the data types and formats within the database files. Based on different data structure classifications, the data types can be categorized into three types: structured data, unstructured data, and semi-structured data. Databases inherently possess a natural regularity in their data storage and arrangement, thus qualifying as structured data. Unstructured data, with its irregular or incomplete structure and lack of a predefined data model (such as audio or video), is unsuitable for use as database files. Semi-structured data falls between fully structured and unstructured data; it does not conform to the data model structure of relational databases or other tables, but contains relevant tags to separate semantic elements and hierarchically categorize records and fields. Therefore, it is also known as a self-describing structure, where data format and content are mixed together without clear distinction, making it suitable for importing its contents into the database.

[0086] As mentioned above, in one embodiment, the data in the database file can be semi-structured data. The first thread can determine whether the currently read buffer segment is complete according to a preset data integrity rule. The data integrity rule includes: if the end of the buffer corresponding to the current buffer segment read by the first thread is a line separator, the data of the buffer segment is determined to be complete; otherwise, the data of the buffer segment is determined to be incomplete, and a delimiting operation is performed based on the determination result. The above integrity rule corresponds to the above specific rule: for example, a specific rule corresponding to a database file stipulates that each line of data in the database file is considered a complete data line, then the corresponding integrity rule can be determined based on whether the end of the buffer corresponding to the current buffer segment is a line separator. Those skilled in the art will understand that the database file can be in formats such as Comma-Separated Values ​​(CSV), JavaScript Object Notation (JSON), and Extensible Markup Language (XML). Of course, the data integrity rules for different database file formats are basically the same. The main difference lies in the form of the line separator and the way to judge the corresponding form. For example, the line separator for a CSV file can be "\n", the line separator for a JSON file can be the corresponding right bracket "}", and the line separator for an XML file can be the corresponding closing tag. This manual does not impose any restrictions on this.

[0087] As mentioned above, in summary, since the delimiting operation is performed by the first thread on the buffer segments according to the data integrity rules, the semi-structured data is completely divided into multiple complete data during the reading process, and then directly processed by the second thread to be sent to the corresponding database. This solves the problem in related technologies where some database files containing semi-structured data cannot be directly split, and the problem that split files require additional disk space. As can be seen from the above embodiments, the solution of this application avoids the resource consumption required for individual file splitting and the disk space required for storing the split files by performing delimiting operations on the corresponding buffer segments in the buffer. Simultaneously, it successfully splits a single database file similar to a CSV format using data integrity rules, and the second thread processes the split complete data in parallel. Furthermore, this solution reduces the number of reads of the database file based on streaming reading, improving the overall processing efficiency of the database import task.

[0088] Figure 7 This is a schematic structural diagram of an electronic device according to an exemplary embodiment. Please refer to... Figure 7At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other necessary hardware. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it, forming a file import device for a distributed database at the logical level. Of course, this specification does not exclude other implementation methods besides software implementation, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0089] Corresponding to the aforementioned embodiments of the file import method for distributed databases, this specification also provides an embodiment of a file import apparatus for distributed databases.

[0090] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of a file import device for a distributed database, as shown in an exemplary embodiment. Figure 8 As shown, in a software implementation, the device may include:

[0091] The buffer fragment reading unit 801 is used for the first thread to read buffer fragments of the database file from the buffer in response to a database import task for the database file, wherein each buffer fragment contains a portion of the data of the database file;

[0092] The buffer fragment processing unit 802 is used by the first thread for each buffer fragment read: performing a delimiting operation to determine the complete data and the remaining incomplete data in the buffer fragment, creating a corresponding database import subtask for the complete data and assigning it to the subtask processing thread group, and recording the remaining incomplete data to form the complete data in the next buffer fragment read by the first thread;

[0093] The database import subtask processing unit 803 is used by the second thread to process the assigned database import subtask when it determines that it belongs to the subtask processing thread group, so as to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

[0094] Optionally, the buffer fragment processing unit 802 is specifically used to: when the first thread determines that there is remaining incomplete data in the buffer fragment it has read, add the incomplete data to the head of the buffer corresponding to the next buffer fragment read by the first thread.

[0095] Optionally, the buffer corresponding to the current buffer segment read by the first thread is the first buffer, and the buffer corresponding to the next buffer segment read is the second buffer; the buffer segment processing unit 802 is specifically used for:

[0096] If the first buffer and the second buffer are the same, the remaining incomplete data is moved to the head of the current buffer;

[0097] If the first buffer is different from the second buffer, the remaining incomplete data is written to the head of the second buffer and cleared from the first buffer.

[0098] Optionally, the data in the database file is semi-structured data, and the database file is in any of the following formats: comma-separated value CSV format, JavaScript object notation JSON format, and Extensible Markup Language (XML) format.

[0099] Optionally, the database file involves multiple partitions of the distributed database on different nodes; the database import subtask processing unit 803 is specifically used for:

[0100] The complete data is parsed and its data type is converted.

[0101] The processed complete data is partitioned to obtain the partition data fragments corresponding to each partition in the complete data;

[0102] Each partition data fragment is serialized, and the serialized partition data fragments are sent to the corresponding nodes, so that each node imports the received partition data fragments into the distributed database.

[0103] Optionally, the database import subtask processing unit 803 is specifically used to: directly send the partition data fragment to the node where the corresponding partition is located;

[0104] or,

[0105] The partitioned data fragments are stored in a preset space corresponding to the respective partition;

[0106] When the preset space meets the preset sending conditions, the partition data fragments in the preset space are sent in batches to the node where the corresponding partition is located.

[0107] Optionally, the buffer segment is the data written into the buffer after the first thread performs streaming reads of the database file.

[0108] Please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating the structure of a file import device for a distributed database, as shown in an exemplary embodiment. Figure 9 As shown, in a software implementation, the device may include:

[0109] The buffer fragment reading unit 901 is used to enable the first thread to read buffer fragments of the database file from the buffer in response to a database import task for the database file, each buffer fragment containing a portion of the data of the database file;

[0110] The buffer fragment reading unit 902 is configured to cause the first thread to perform a delimitation operation for each buffer fragment read, to determine the complete data and the remaining incomplete data in the buffer fragment, to create a corresponding database import subtask for the complete data and assign it to the subtask processing thread group, so that the second thread, if it determines that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and to record the remaining incomplete data for use in constituting the complete data in the next buffer fragment read by the first thread.

[0111] Optionally, the buffer fragment reading unit 902 is specifically used for:

[0112] If the first thread determines that there is remaining incomplete data in the buffer segment it reads, it adds the incomplete data to the head of the buffer corresponding to the next buffer segment read by the first thread.

[0113] Optionally, the buffer corresponding to the current buffer segment read by the first thread is the first buffer, and the buffer corresponding to the next buffer segment read is the second buffer; the buffer segment reading unit 902 is specifically used for:

[0114] If the first buffer and the second buffer are the same, the remaining incomplete data is moved to the head of the current buffer;

[0115] If the first buffer is different from the second buffer, the remaining incomplete data is written to the head of the second buffer and cleared from the first buffer.

[0116] Optionally, the data in the database file is semi-structured data; the buffer fragment reading unit 902 is specifically used for:

[0117] The data integrity rules determine whether the current buffer segment being read is complete according to preset data integrity rules. The data integrity rules include: if the end of the buffer corresponding to the current buffer segment read by the first thread is a line separator, the data of the buffer segment is determined to be complete; otherwise, the data of the buffer segment is determined to be incomplete.

[0118] Perform a delimitation operation based on the judgment result.

[0119] Optionally, the database file may be in any of the following formats: comma-separated value CSV format, JavaScript object notation JSON format, and Extensible Markup Language (XML) format.

[0120] Optionally, the buffer segment is the data written into the buffer after the first thread performs a streaming read of the database file.

[0121] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0122] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0123] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0124] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0125] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0126] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0127] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0128] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0129] Therefore, specific embodiments of the subject matter have been described. Furthermore, the processes depicted in the figures are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0130] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for importing files into a distributed database, comprising: In response to a database import task for a database file, the first thread reads buffered fragments of the database file from the buffer, each buffered fragment containing a portion of the data in the database file; For each buffer segment read, the first thread performs a delimiting operation to determine the complete data and the remaining incomplete data in the buffer segment, creates a corresponding database import subtask for the complete data and assigns it to the subtask processing thread group, and records the remaining incomplete data to form the complete data in the next buffer segment read by the first thread. When the second thread determines that it belongs to the subtask processing thread group, it processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

2. The method according to claim 1, wherein recording the remaining incomplete data comprises: If the first thread determines that there is remaining incomplete data in the buffer segment it reads, it adds the incomplete data to the head of the buffer corresponding to the next buffer segment read by the first thread.

3. The method according to claim 2, wherein the buffer corresponding to the current buffer segment read by the first thread is the first buffer, and the buffer corresponding to the next buffer segment read is the second buffer; the step of adding the incomplete data to the head of the buffer corresponding to the next buffer segment read by the first thread includes: If the first buffer and the second buffer are the same, the remaining incomplete data is moved to the head of the current buffer; If the first buffer is different from the second buffer, the remaining incomplete data is written to the head of the second buffer and cleared from the first buffer.

4. The method according to claim 1, wherein the data in the database file is semi-structured data; The first thread performs a delimiting operation, including: The data integrity rules determine whether the current buffer segment being read is complete according to preset data integrity rules. The data integrity rules include: if the end of the buffer corresponding to the current buffer segment read by the first thread is a line separator, the data of the buffer segment is determined to be complete; otherwise, the data of the buffer segment is determined to be incomplete. Perform a delimitation operation based on the judgment result.

5. The method according to claim 1, wherein the database file is in any of the following formats: comma-separated value CSV format, JavaScript object notation JSON format, and Extensible Markup Language XML format.

6. The method according to claim 1, wherein the database file involves multiple partitions of the distributed database on different nodes; the second thread processes the assigned database import subtask, including: The complete data is parsed and its data type is converted. The processed complete data is partitioned to obtain the partition data fragments corresponding to each partition in the complete data; Each partition data fragment is serialized, and the serialized partition data fragments are sent to the corresponding nodes, so that each node imports the received partition data fragments into the distributed database.

7. The method according to claim 6, wherein sending the serialized partition data fragments to the corresponding nodes of the partitions comprises: The partition data fragment is sent directly to the node where the corresponding partition is located; or, The partitioned data fragments are stored in a preset space corresponding to the respective partition; When the preset space meets the preset sending conditions, the partition data fragments in the preset space are sent in batches to the node where the corresponding partition is located.

8. The method according to claim 1, wherein the buffer segment is the data written into the buffer after the first thread performs streaming reading of the database file.

9. A method for importing files into a distributed database, comprising: In response to a database import task for a database file, the first thread reads buffered fragments of the database file from the buffer, each buffered fragment containing a portion of the data in the database file; For each buffer segment read, the first thread performs a delimiting operation to determine the complete data and remaining incomplete data in the buffer segment; creates a corresponding database import subtask for the complete data and assigns it to a subtask processing thread group, so that the second thread, upon determining that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and records the remaining incomplete data to form the complete data in the next buffer segment read by the first thread.

10. A file import device for a distributed database, comprising: A buffer fragment reading unit is configured to enable a first thread to read buffer fragments of the database file from a buffer in response to a database import task for the database file, each buffer fragment containing a portion of the data in the database file; The buffer fragment processing unit is configured to enable the first thread to perform a delimiting operation for each buffer fragment read to determine the complete data and the remaining incomplete data in the buffer fragment, create a corresponding database import subtask for the complete data and assign it to the subtask processing thread group, and record the remaining incomplete data to form the complete data in the next buffer fragment read by the first thread. The database import subtask processing unit is used to enable the second thread to process the assigned database import subtask when it determines that it belongs to the subtask processing thread group, so as to send the complete data indicated by the database import subtask to the corresponding node in the distributed database.

11. A file import device for a distributed database, comprising: A buffer fragment reading unit is configured to enable a first thread to read buffer fragments of the database file from a buffer in response to a database import task for the database file, each buffer fragment containing a portion of the data in the database file; The buffer fragment processing unit is configured to enable the first thread to perform a delimitation operation for each buffer fragment read, to determine the complete data and the remaining incomplete data in the buffer fragment, to create a corresponding database import subtask for the complete data and assign it to a subtask processing thread group, so that the second thread, upon determining that it belongs to the subtask processing thread group, processes the assigned database import subtask to send the complete data indicated by the database import subtask to the corresponding node in the distributed database; and to record the remaining incomplete data for use in constructing the complete data in the next buffer fragment read by the first thread.

12. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method as claimed in any one of claims 1 to 9.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in any one of claims 1 to 9.

Images