A method and apparatus for performing a data import task

Abstract

Embodiments of the present specification provide a method for performing a data import task, the data import task comprising importing first data to a first partition of a target database, the method being applied to a first node of the target database, the first node storing a first copy of the first partition; the method comprising: reading the first data into a buffer set for the data import task in the first node. When a first condition is met, determining a buffer state of the buffer, the first condition comprising one of: an amount of data read into the buffer reaching a first threshold, the first data being all read into the buffer. Determining a target import mode, the target import mode being determined according to at least the buffer state of the first node. Importing the first data to the first copy based on the target import mode.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a method and apparatus for performing a data import task. Background Technology

[0002] In the information age, data has become a crucial factor of production and a strategic resource. As the core software for data storage, management and analysis, databases have been applied to fields such as finance, communications and the Internet of Things, becoming an indispensable data foundation for modern enterprise information management.

[0003] In the daily operation and development of databases, data import is a fundamental and frequent operation, especially in scenarios such as data migration, data warehouse construction, and batch data exchange. Users often need to perform data import tasks to load massive amounts of data into database tables. The amount of data involved in different data import tasks can vary greatly. Using a single, fixed import method for different volumes of data to be imported often fails to achieve high import efficiency. Therefore, dynamically selecting the appropriate import method based on the actual amount of data involved in the data import task is key to improving the overall execution efficiency of data import tasks.

[0004] Currently, mainstream database systems typically offer two data import methods to handle data import tasks of different scales. One is the DML (Data Manipulation Language) import method, which follows the database's standard transaction processing and logging workflow, completing data import through standardized execution of DML statements. Due to its fixed logging overhead, lock contention, and frequent dump operations between memory and persistent storage, it often encounters performance bottlenecks when importing massive amounts of data, making it more suitable for small-scale data imports. The other is the bypass import method, which bypasses the database engine's DML parsing layer, directly allocating space and writing data to the underlying data files (persistent storage). This can significantly improve data import speed; however, in small-scale data import scenarios, the initialization overhead of this method may offset its efficient writing advantage, making it even less efficient than the DML import method. Therefore, it is more suitable for large-scale data imports.

[0005] Faced with the two data import methods, each with its own advantages and disadvantages, the bottleneck of existing database technology is that when executing data import tasks, the database usually has difficulty in accurately predicting the amount of data to be imported in advance, lacks a basis for automated decision-making, and cannot determine the appropriate import method for the data import task.

[0006] Therefore, it is desirable to provide a solution that, through technical means, enables the database to automatically assess the data volume and select an appropriate import method when performing data import tasks, thereby improving the efficiency of data import. Summary of the Invention

[0007] One or more embodiments of this specification describe a method and apparatus for performing a data import task, which can monitor the amount of data during the execution of the data import task and automatically determine the appropriate import method based on the actual amount of data, thereby improving the execution efficiency of the data import task.

[0008] According to a first aspect, a method for performing a data import task is provided, the data import task including importing first data into a first partition of a target database, the method being applied to a first node of the target database, the first node storing a first copy of the first partition; the method includes:

[0009] The first data is read into the buffer set for the data import task in the first node.

[0010] When a first condition is met, the buffer state of the buffer is determined. The first condition includes one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data is read into the buffer.

[0011] The target import method is determined, which is at least based on the buffer state of the first node.

[0012] Based on the target import method, the first data is imported into the first copy.

[0013] According to a second aspect, this specification provides an apparatus for performing a data import task, the data import task including importing first data into a first partition of a target database, the apparatus being disposed on a first node of the target database, the first node storing a first copy of the first partition; the apparatus includes:

[0014] The buffer module is configured to read first data into the buffer set in the first node for the data import task.

[0015] The first determining module is configured to determine the buffer state of the buffer when a first condition is met, wherein the first condition includes one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data is read into the buffer.

[0016] The second determining module is configured to determine the target import method, wherein the target import method is determined at least based on the buffer state of the first node.

[0017] The import module is configured to import the first data into the first copy based on the target import method.

[0018] According to a third aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0019] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0020] In summary, the methods and apparatus provided in the embodiments of this specification offer a method for performing data import tasks. During the execution of the data import task, the appropriate data import method can be automatically determined by real-time sensing of the data size to be imported. This improves the performance of the data import task and avoids the complexity caused by mixing different data import methods. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 This is a schematic diagram of a typical LSM-Tree storage architecture disclosed in this specification;

[0023] Figure 2 This is a schematic diagram of an exemplary distributed database implementation architecture disclosed in this specification;

[0024] Figure 3A This is an implementation framework for a method of performing a data import task provided according to embodiments of this specification;

[0025] Figure 3B This is an implementation framework for a method of performing a data import task provided according to embodiments of this specification;

[0026] Figure 4 This is a flowchart illustrating a method for performing a data import task according to an embodiment of this specification;

[0027] Figure 5 This is a schematic diagram of an apparatus for performing a data import task according to an embodiment of this specification. Detailed Implementation

[0028] The solutions provided in the embodiments of this specification will now be described with reference to the accompanying drawings. To aid understanding, some database technology concepts involved in the various embodiments of this specification will be introduced first, especially the database system with LSM-Tree as the storage architecture.

[0029] Currently, in many enterprises, data analysis and operational activities are all based on databases as the data foundation. Databases can efficiently organize, store, and manage massive amounts of data. Through flexible query interfaces and transaction mechanisms, they provide efficient data read and write services for upper-layer applications while ensuring data consistency and reliability.

[0030] From a database architecture perspective, databases can employ an append-only write strategy to handle new data. This means that insert, update, and delete operations performed on a database instance are all stored as newly added data, and once stored, the data is not modified. For example, data insertion can be achieved by directly appending new data records, updating historical data can be achieved by appending new version data records, and deleting historical data can be achieved by appending data to delete marker records.

[0031] At the data storage level, besides organizing data in data pages and persisting it on storage media, databases can also use log-structured storage structures for data persistence. Typically, an LSM-Tree (Log Structured Merge Tree) storage architecture can be used. In this architecture, new data written to the database is first sequentially appended to a MEMTable in memory, and then persisted (also referred to as a dump in this specification) to the storage media in an immutable form (MEMTable), forming multiple SSTable (Sorted String Table) files generated based on time order. In other words, the MEMTable is stored in memory to store dynamic data and provides read and write operations; the SSTable is stored on non-volatile storage media to store static data. Non-volatile storage media differs from memory in that it features data retention after power failure, large storage capacity at relatively low cost, high sequential write bandwidth but high random read / write latency. Under this storage architecture, all data insertion, update, and deletion operations can be transformed into new data records and ultimately appended to the SSTable file, in accordance with the aforementioned append-only write strategy. In some practices, the SSTable file can be further divided into several fixed-size blocks at the logical level, with each block storing several data records.

[0032] The basic data unit for database operations can be the SSTable file described above, or the blocks that make up the SSTable file. In the embodiments of this specification, a data area will be used to represent the basic data unit for database operations, and its specific meaning may vary depending on the database architecture design in actual applications.

[0033] For a concrete implementation example of the LSM-Tree database described above, please refer to [link / reference]. Figure 1 The diagram shows a typical LSM-Tree storage architecture. Database systems using this architecture can transform random write operations into batch sequential write operations, greatly improving data writing speed.

[0034] See Figure 1 Data updates over a period of time (corresponding to write operations shown in the attached diagram) are persisted to a log file (corresponding to the Write-Ahead Log (WAL) shown in the attached diagram) in a sequential manner and written to a data structure in memory (corresponding to the MEMTable shown in the attached diagram). When the amount of data in the MEMTable exceeds a certain threshold, the MEMTable is frozen and transformed into an immutable MEMTable (corresponding to the ImmutableMEMTable shown in the attached diagram). Simultaneously, to avoid blocking database write operations, a new MEMTable is generated to respond to subsequent data writes. Next, background tasks in the database system persist the data in the Immutable MEMTable in memory (corresponding to the flush operation shown in the attached diagram) to the SSTable file on disk without blocking the processing of database foreground tasks. In this way, random write operations performed by the front-end system on different data in the database system can be batch-flushed to the SSTable on disk. This flushing process is a sequential append write to the SSTable, thus transforming the originally scattered random write operations into batch, continuous sequential write operations, significantly improving data write efficiency.

[0035] Please refer to the appendix for further details. Figure 1SSTables on disk are organized in a multi-level structure (the example in the attached diagram includes L0, L1, and L2, three levels). The number of levels can be set according to specific needs. Generally, the total capacity of the SSTables in the upper level is smaller than that in the lower level. The operation of creating SSTables is called compaction. Typically, the L0 level SSTable is generated by flushing Immutable MEMTables, while other levels (e.g., L1, L2) are generated by the SSTables in the upper level. For example, when the data capacity of the L0 SSTable reaches or approaches its limit, a compaction operation can be used to write the data from the L0 SSTable to the L1 SSTable. Similarly, when the data capacity of the L1 SSTable reaches its limit, a compaction operation can be used to write the data from the L1 SSTable to the L2 SSTable, and so on. In this way, the data stored in the upper level SSTable will be newer than that in the lower level SSTable. Overall, the newest data is stored in memory, the next newest data is stored in the L0 SSTable, persistent data is gradually migrated to the next lower level as compression and merging operations are performed, and the oldest data is stored in the lowest level SSTable.

[0036] Please refer to the appendix for further details. Figure 1 In LSM-Tree databases, the execution of read operations is also closely related to the aforementioned multi-level storage architecture. When processing a read operation, the database system first queries the MEMTable in memory. If no match is found, it continues to query the Immutable MEMTable in memory. If no matching data is found in memory, the system searches the multi-level SSTables on disk, typically starting from a certain level (e.g., L0 level) and traversing level by level down (e.g., from L0 to L1 level, then to L2 level) until matching data is found.

[0037] The above is a detailed introduction to the LSM-Tree storage architecture in databases. Database systems based on this storage architecture can convert random data writes into sequential append writes and reorganize data using background compression and merging operations, thereby achieving extremely high data write throughput. It is particularly suitable for carrying continuous writes of massive amounts of data and is widely used in data write-intensive scenarios such as time-series data recording and operation log archiving.

[0038] Next, we will introduce the implementation architecture of distributed databases. Figure 2 An exemplary distributed database implementation architecture is shown. It should be noted that this specification... Figure 2The distributed database architecture shown is only an example, and the number of nodes, partitioning, and region affiliation are not limiting factors. In practice, these can be flexibly configured according to needs.

[0039] See Figure 2 Distributed database systems typically consist of several nodes, each usually being a single physical machine. These nodes often belong to different regions. For example... Figure 2 The diagram shows region A, which contains nodes A1 and A2, and region B, which contains nodes B1 and B2. A region is a logical concept, typically representing a set of nodes with similar network conditions or geographical locations. That is, a region can have different meanings depending on the deployment model. For example, when a database system is deployed in a data center, nodes in a region can be machines belonging to the same rack or machines bridged on the same switch. When a database system is deployed in multiple data centers, each region can correspond to one data center, and the nodes in a region are the machines deployed within that data center.

[0040] In some distributed databases, data in a table can be split (horizontally partitioned) into multiple data shards according to certain partitioning rules. Each data shard is a table partition, or simply partition. Any row of data in a table belongs to one and only one partition. Several partitions of a data table can be distributed and stored across multiple nodes within a single region. Figure 2 For example, data table 1 is divided into 4 partitions, data table 2 is divided into 2 partitions, and they are distributed and stored on various nodes.

[0041] In practical applications, to achieve high availability of a database system, a partition can have multiple replicas. Typically, multiple replicas of a partition are distributed across multiple different regions. Of these replicas, only one can accept data modification operations; this replica is the primary replica, and the others are secondary replicas. Figure 2 In the example shown, the rounded rectangle represents the primary copy of the partition, and the parallelogram represents the secondary copy of the partition.

[0042] The node hosting the primary replica can be considered the master node, and the nodes hosting the secondary replicas can be considered secondary nodes. Once the node hosting the primary replica begins providing services, user data update operations will generate corresponding logs / write-ahead logs on that node. These logs are then synchronized to all secondary replica nodes based on a distributed consensus protocol (e.g., Raft, Paxos), ensuring data consistency across replicas. When the node hosting the primary replica fails, the database system can automatically initiate an election through the consensus protocol to select a new master node from the surviving secondary nodes to continue providing services, achieving automatic fault recovery and continuous service availability.

[0043] It is understandable that in a scenario where the database system is deployed on a single machine, partitions can also be stored as a single copy on a single node.

[0044] As mentioned earlier, data import is a fundamental and frequent operation in the daily operation and development of databases. Database systems in related technologies typically offer two data import methods: DML import and bypass import. Therefore, choosing the appropriate import method when performing data import tasks is a crucial technical problem. Specifically, DML import, due to its adherence to standard transaction processes, has lower initialization overhead. However, it incurs overhead such as fixed logging and memory dumping to persistent storage during the import process, and its performance decreases as the data volume increases. Therefore, it is more suitable for small-scale data imports. In contrast, bypass import can directly allocate space in the underlying data area and write the data to be imported. Although it has higher initialization overhead (e.g., requiring the construction of a bypass import thread pool, allocation of storage space, and construction of an execution context), it avoids the inherent dumping overhead of DML import, thus achieving higher performance when handling massive data imports, making it more suitable for large-scale data imports.

[0045] However, in related technologies, database systems struggle to accurately and in real-time assess the actual size of the data to be imported when performing data import tasks, thus lacking the ability to automatically determine the appropriate import method.

[0046] In related technologies, one strategy relies on explicit user declaration of the import method. This requires users to explicitly add specific commands to DML statements when submitting a data import task to specify which import method the database engine should use. This strategy not only increases user complexity but also risks using the wrong import method due to user errors, thus reducing data import performance. Another strategy estimates the size of the data to be imported based on historical statistics within the database (e.g., table size, data growth trends), and then automatically determines the import method. The reliability of this strategy is heavily dependent on the accuracy and timeliness of the statistics. Outdated or inaccurate statistics will lead to incorrect decisions, especially for data import tasks with filtering clauses (e.g., SELECT and WHERE clauses in INSERT statements). The actual amount of data imported depends on the filtering results of these clauses and may differ significantly from the historical statistics of the source table. Database systems struggle to make accurate estimates in such scenarios, causing the automated decision-making mechanism to fail.

[0047] In view of this, the inventors have proposed a method for performing a data import task in the embodiments of this specification. Figure 3A A schematic diagram of the implementation framework of this method is shown.

[0048] See Figure 3A The data import task for the target database can contain multiple sets of data to be imported, as well as partitions of the target data table corresponding to each set of data. The data import task is used to instruct that the data to be imported be stored in the corresponding partitions. The relationship between the data to be imported and the partitions contained in the data import task can be identified by the database engine or by the master node of each partition (in a target database deployed on a single machine, the master node is the node where the target database is deployed). This specification does not specifically limit this in the embodiments. The following will use any one set of data to be imported (the first set of data) as an example, combined with... Figure 3A The embodiments described in this specification are presented here.

[0049] The data import task includes importing first data into the first partition of the target database. The first node in the target database is a node that stores any copy of the first partition (the first copy). The first node also has a buffer for the data import task. It can be understood that in some implementation scenarios (e.g., a single-machine deployment of the target database), the first copy can be a unique copy of the first partition. When executing the data import task, the first data can be read into the buffer. This reading process is not limited to reading the first data all at once, but can be done in batches according to the target database's engine settings and the data size of the database engine's data reading batch, reading the first data into the buffer in stages.

[0050] During the process of reading data into the buffer, data reading can be stopped at any time according to the data reading status, and the buffer status of the buffer can be determined. For example, when the amount of data read into the buffer reaches a first threshold (at this time, the first data has not been completely read into the buffer), reading can be stopped and the buffer status can be determined as the storage space is full; when all the first data has been read into the buffer, the buffer status can be determined as the reading is complete.

[0051] The first node can determine the target import method based on the buffer status. The rules for determining the target import method can be flexibly set according to actual needs. For example, when the buffer status indicates that the buffer storage space is full, the target import method can be determined as bypass import.

[0052] Once the target import method is determined, the first node can import the first data into the first copy of the first partition based on the target import method.

[0053] Figure 3B This paper illustrates a framework for performing a data import task in a distributed deployment of a target database. It is understood that in this distributed database system, the first partition can have multiple replicas, stored on multiple (e.g., M) nodes. Based on the steps described above, the M nodes can determine the buffer state of their respective buffers. Then, based on these M buffer states, the target import method can be determined.

[0054] Correspondingly, in this embodiment, the rules for determining the target import method can be flexibly set according to actual needs. For example, when any one of the M buffer states of the M nodes is full, the target import method is determined to be bypass import. Furthermore, the target import method can be shared among the M nodes, allowing each node to perform data import tasks based on the same data import method.

[0055] The above method can perceive the scale of the data to be imported in real time during the data import task, without user intervention or reliance on historical statistics. It automatically determines the target import method based on the buffer status of the buffers set on the nodes in the data import task and executes the data import using that method. In distributed database deployment scenarios, the target import method can also be synchronized across multiple nodes, ensuring that multiple nodes executing the same data import task can complete the data import using the same method. Therefore, by selecting an appropriate import method, the execution performance of the data import task can be improved, and by synchronizing the target import status across multiple nodes, the execution path of the data import can be unified, reducing the complexity caused by mixing different data import methods.

[0056] Based on the above technical framework Figure 4 A flowchart illustrating a method for performing a data import task according to an embodiment of this specification is shown. It is understood that the method disclosed in the embodiments of this specification can be executed by any apparatus, device, platform, or cluster of devices with computing and processing capabilities.

[0057] See Figure 4 In one embodiment, the data import task includes importing first data into a first partition of a target database. The method is applied to a first node of the target database, and the first node stores a first copy of the first partition. The method includes at least the following steps: Step S401: Reading the first data into a buffer set up in the first node for the data import task. Step S403: Determining the buffer state of the buffer when a first condition is met. Step S405: Determining a target import method, the target import method being determined at least based on the buffer state of the first node. Step S407: Importing the first data into the first copy based on the target import method.

[0058] In the database field, data import refers to the operation of persistently storing data from external data sources (e.g., data files, other databases, or other data tables) into data tables within a target database through a specific processing flow. When the target database engine receives a data import task, it typically translates the task into a DML-described data import statement (e.g., converting a BULK INTO instruction into an INSERT INTO statement), or it receives a DML import statement explicitly provided by the user (e.g., an INSERT INTO SELECT statement). During the execution of a data import task, the data import statement can be parsed to determine the target data table, and then, based on the partitioning rules of the target data table, the target partition of the target data table into which the data to be imported will be determined. It's important to understand that in a distributed database, a partition can be stored distributed across different nodes of the target database as multiple replicas (e.g., one primary replica and several secondary replicas). From the perspective of the relationship between the data to be imported and the target partition, a data import task can be divided into multiple logical subtasks, each subtask corresponding to a target partition of the target data table and containing the data to be imported into that partition. This allows the target database to process data import tasks in parallel, importing different data to be imported into their respective partitions.

[0059] In this embodiment, we will take importing first data into the first partition of the target database as an example. The first partition is a logical partition of a data table in the target database, and the first data is the data to be imported that belongs to the first partition according to the partitioning rules. Based on the above, it can be understood that this import operation means that the first data needs to be imported into a replica of the first partition. In a distributed deployment scenario, the first data needs to be imported into multiple replicas of the first partition. The first node is a node in the target database that stores any replica of the first partition (referred to as the first replica in this embodiment).

[0060] It should also be noted that the embodiments in this specification will be described using a target database with a distributed deployment of multiple nodes as an example. However, this does not represent a specific limitation on the application scenarios of the present invention. The technical methods disclosed in the embodiments of this specification can also be applied to perform data import tasks in databases with other deployment methods. For example, in a single-machine deployed database system, the single node can be regarded as the master node, and this method can be applied to perform data import tasks.

[0061] When the first node executes the data import task, in step S401: first data is read into the buffer set in the first node for the data import task.

[0062] A buffer is a temporary data storage area set up by the first node for executing a data import task, whereby the first data is imported into the first replica. In other words, in the target database, each node can set up a buffer corresponding to its data import task. The buffer can have a preset storage capacity limit, which can be a default fixed value (e.g., 8MB) or a parameter that can be flexibly configured by the user or administrator according to specific needs. It can be understood that in a distributed database, each node can set its own buffer and can independently set a storage capacity limit for its own buffer.

[0063] In a specific implementation, the buffer can be placed inside the operator used to perform the data import task. In a database engine, an operator is the basic execution unit for performing specific data operations (e.g., scan, filter, join, etc.). A complex data import statement, after being parsed and optimized by the database engine, is transformed into an execution plan composed of multiple operators. By embedding the buffer within the operator responsible for data import, the first data can be read into the buffer inside the operator when the database engine schedules and executes the operator.

[0064] In this step, the process of reading the first data into the buffer is not to read all the first data at once, but to read the first data into the buffer in batches based on a preset data reading batch size (e.g., 2MB).

[0065] Next, step S403: when the first condition is met, determine the buffer state of the buffer.

[0066] During the process of reading the first data into the buffer, the reading can be stopped and the buffer state can be determined when a first condition is met. The first condition may include one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data has been read into the buffer.

[0067] In a specific application, the first threshold can be the upper limit of the buffer's storage capacity. At this point, if the amount of data read into the buffer reaches the first threshold, it indicates that the buffer is full, or that the remaining storage space in the buffer is insufficient to store the next batch of data.

[0068] In step S405: Determine the target import method, which is determined at least based on the buffer state of the first node.

[0069] As mentioned earlier, in a single-machine deployment scenario, the target database's partitioned data is stored on a single node (i.e., the first node). Based on the buffer status of the first node, the target import method can be determined. The strategy for determining the target import method will be described below.

[0070] In distributed databases, to ensure data reliability and service availability, partitioned data is typically stored in multiple replicas across multiple nodes. In a specific example, multiple replicas of the first partition are distributed across M nodes, including the first node. Any of these M nodes can independently maintain a buffer set up for the data import task and independently execute its own import task (i.e., import the first data into the replica of the first partition maintained by that node). Any of the M nodes can also determine the buffer state of its own buffer when reading the first data into the buffer.

[0071] After the M nodes have determined their respective buffer states, a suitable import method for the first data, namely the target import method, can be determined based on these M buffer states. The strategy for determining the target import method will be described below. To ensure that the data import process for each replica of the first partition remains consistent and to avoid data inconsistencies or management complexity caused by different import methods used by different nodes, in this embodiment, the target import method can be shared among the M nodes. This means that the target import method is determined by a single node (e.g., the first node) based on multiple buffer states (e.g., the M buffer states of the M nodes). After the target import method is determined, it is broadcast to multiple nodes (e.g., the M nodes) to unify the data import methods of these multiple nodes.

[0072] In a specific implementation, the node responsible for determining the target import method is called the coordinating node. The coordinating node can be a globally unique node set up for the data import task, or it can be several nodes set up for each partition. The coordinating node can be determined by multiple nodes of the target database through election based on a distributed consensus protocol (e.g., Raft protocol, Paxos, etc.).

[0073] Taking the second node in the target database as the coordinating node as an example:

[0074] First, the M nodes can send their respective buffer statuses to the second node. This process can be performed based on a streaming data service, for example, using the publish function in the DataHub service, where the M nodes can report their respective buffer statuses to the second node.

[0075] Then, after collecting the M buffer states of the M nodes, the second node determines the target import method based on these states and broadcasts a target message containing the target import method to the M nodes. The M nodes receive the target message sent by the second node, thereby obtaining the target import method. This process can also be performed based on a streaming data service, for example, based on the subscription function in the DataHub service, where the M nodes can subscribe to obtain the target message.

[0076] It should be understood that the second node mentioned above is only used as a reference for a coordinating node and is not intended to be limiting. The coordinating node can be any node in the target database. In one possible implementation, the first node can also be used as a coordinating node, and this specification does not make any specific limitation on this.

[0077] Through the above steps, the coordinating node can receive buffer states from multiple nodes based on the actual amount of data read, and thus determine the target import method. Specifically, the coordinating node collects the M buffer states corresponding to each of the M nodes. If the buffers of most nodes have read all of the first data, it can be inferred that the data size of the first data is small, and it is suitable to perform data import via DML import. If the buffer states of most nodes indicate that the amount of data read into the buffer has reached a first threshold, it can be inferred that the data size of the first data is large, and it is more suitable to perform data import via bypass import.

[0078] In a specific application, if the M buffer status indicators each contain complete first data in their respective buffers, the target import method can be determined as the first method of importing via memory (i.e., DML import method); otherwise, the target import method is determined as the second method of importing without memory (i.e., bypass import method).

[0079] In the implementation described above, the target import method for executing the data import task is determined by the M buffer states corresponding to the M nodes storing each replica of the first partition. In practice, the data import task may further include importing second data into the second partition of the target database. To unify the way nodes execute the data import task, the target import method can also be determined based on the buffer states of the N nodes storing each replica of the second partition. It can be understood that in this practice, the target import method is determined based on M+N buffer states, which are shared among the M and N nodes.

[0080] Next, returning to the main process, after determining the target import method, in step S407: the first data can be imported into the first copy based on the target import method.

[0081] When the target import method is the first method via memory, a data area (hereinafter referred to as the first data area) can be initialized in memory first. This data area is a memory region allocated for temporarily storing the first data. Then, the buffered data corresponding to the first data that has already been read into the buffer is written into the first data area. In one specific scenario, if the buffer status indicates that the amount of data read into the buffer has reached a first threshold, meaning that the first data has not been completely read into the buffer, then after writing the buffered data already read into the buffer into the first data area, the remaining data in the first data (excluding the buffered data) can be read and written into the first data area. In another specific scenario, if the buffer status indicates that the first data has been completely read into the buffer, then only the buffered data already read into the buffer can be written into the first data area.

[0082] When a database dump operation is triggered, the data in the first data area will be written to the first copy after the dump operation, thus completing the persistence of the first data. The triggering time for the dump operation can be set by the user or administrator of the target database. For example, it can be when the amount of data stored in the memory data area reaches a preset threshold, or when a preset time interval is reached, etc. This specification does not provide examples of each of these scenarios.

[0083] In a specific practice, the target database's storage architecture is a log structure merged tree (LSM-Tree). The first data area mentioned above is specifically the memory table MEMTable in the LSM-Tree architecture. The corresponding dump operation is the flush operation in the LSM-Tree architecture, which is used to persistently store the MEMTable data containing the first data in the SSTable file, thereby completing the import of the first data into the first replica.

[0084] When the target import method is the second method that does not involve memory, a data area corresponding to the first copy (hereinafter referred to as the second data area) can first be initialized in the non-volatile storage medium. This data area is used to persist the first data and corresponds to the first copy. Simultaneously, in some databases, resources such as temporary storage space (for data page swapping), thread pools (for multi-threaded copying), and execution context also need to be initialized. Then, the buffered data corresponding to the first data that has already been read into the buffer is directly written to the first data area. This process does not involve memory processing, thus achieving a direct copy of the first data to the non-volatile storage medium. In a specific scenario, if the buffer status indicates that the amount of data read into the buffer has reached a first threshold, meaning that the first data has not been completely read into the buffer, then after writing the buffered data already read into the buffer to the second data area, the remaining data in the first data (excluding the buffered data) can continue to be read and written to the first data area. In another specific scenario, if the buffer status indicates that the first data has been completely read into the buffer, then only the buffered data already read into the buffer can be written to the second data area.

[0085] In a specific implementation, the target database uses a log-structured merged tree (LSM-Tree) storage architecture. The second data area mentioned above is specifically the ordered string table (SSTable) within the LSM-Tree architecture, corresponding to the first replica, and used for persistent storage of the first replica's data. By writing the first data to the second data area without passing through memory, the first data can be directly persisted, completing the import of the first data into the first replica.

[0086] The foregoing description, based on one or more embodiments, details a method for performing a data import task. Using the method provided in the embodiments of this specification, the data scale of the data to be imported can be perceived in real time during the execution of the data import task. No user intervention is required, and there is no need to rely on historical statistical data. The target import method is automatically determined based on the buffer status of the buffers set by nodes on the data import task. In a distributed deployment scenario, the target import method can also be automatically determined based on the buffer status of the buffers set by multiple nodes on their respective data import tasks. Simultaneously, the target import method is synchronized among multiple nodes, ensuring that multiple nodes performing the same data import task can complete the data import based on the same import method. Therefore, by selecting an appropriate import method, the execution performance of the data import task can be improved. Furthermore, by synchronizing the target import status among multiple nodes, the execution path of the data import can be unified, reducing the complexity caused by mixing different data import methods.

[0087] In this specification, the terms "first" in the first node, first partition data, etc., and the corresponding terms "second" and "third" (if they exist) in the text are merely for the convenience of distinction and description, and do not have any limiting meaning.

[0088] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0089] Figure 5 This is a schematic diagram of an apparatus for performing a data import task according to an embodiment of this specification. The apparatus 500 is deployed in a computing device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. The data import task includes importing first data into a first partition of a target database. The apparatus is located on a first node of the target database, and the first node stores a first copy of the first partition. This apparatus embodiment is similar to... Figure 4 Corresponding to the method embodiment shown, the apparatus 500 includes:

[0090] The buffer module 501 is configured to read first data into the buffer set in the first node for the data import task.

[0091] The first determining module 502 is configured to determine the buffer state of the buffer when a first condition is met, wherein the first condition includes one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data is read into the buffer.

[0092] The second determining module 503 is configured to determine the target import method, wherein the target import method is determined at least based on the buffer state of the first node.

[0093] Import module 504 is configured to import the first data into the first copy based on the target import method.

[0094] According to another embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 4 The steps of the method are described.

[0095] According to yet another embodiment, this specification also provides a computing device including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the foregoing combination. Figure 4 The steps of the method are described.

[0096] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0097] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for performing a data import task, the data import task comprising importing first data into a first partition of a target database, the method being applied to a first node of the target database, the first node storing a first copy of the first partition; The method includes: Read the first data into the buffer set for the data import task in the first node; When a first condition is met, the buffer state of the buffer is determined, wherein the first condition includes one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data is read into the buffer; The target import method is determined, and the target import method is determined at least based on the buffer state of the first node; Based on the target import method, the first data is imported into the first copy.

2. The method according to claim 1, wherein, The target database has M nodes that store copies of the first partition respectively; the target import method is also determined based on the buffer status of the nodes that store other copies of the first partition; and the target import method is shared among the M nodes.

3. The method according to claim 1, wherein, The data import task also includes importing second data into the second partition of the target database; the target import method is also determined based on the buffer status of each of the N nodes storing each copy of the second partition.

4. The method according to claim 1, wherein, The methods for determining the target import method include: Send the buffer status to the second node of the target database; Receive the target message sent by the second node, including the target import method.

5. The method according to claim 2, wherein, The method for determining the target import method includes: Collect the M buffer states corresponding to each of the M nodes; If each of the M buffer status indicators contains complete first data in its corresponding buffer, then the target import method is determined to be the first method of importing via memory; otherwise, the target import method is determined to be the second method of importing without memory.

6. The method according to claim 5, further comprising: Send a target message containing the target import method to all nodes except the first node among the M nodes.

7. The method according to claim 1, wherein, The target import method is a first method via memory; the import of the first data into the first copy includes: Initialize the first data area in memory; The buffered data corresponding to the first data in the buffer is written into the first data area; the data in the first data area is written into the first copy after a dump operation.

8. The method according to claim 7, wherein, The target database uses a log structure merged tree (LSM-Tree) storage architecture; the first data area is a MEMTable; and the dump operation is a flush operation.

9. The method according to claim 7, wherein, The buffer status indicates that the amount of data read into the buffer has reached a first threshold. The step of importing the first data into the first copy further includes: writing the data in the first data other than the buffered data into the first data area.

10. The method according to claim 1, wherein, The target import method is a second method that does not involve memory; the import of the first data into the first copy includes: Initialize the second data area in the non-volatile storage medium corresponding to the first copy; The buffered data corresponding to the first data in the buffer is written directly to the second data area without going through memory.

11. The method according to claim 10, wherein, The target database uses a log-structured merged tree (LSM-Tree) storage architecture; the second data area is an SSTable.

12. The method according to claim 10, wherein, The buffer status indicates that the amount of data read into the buffer has reached a first threshold. The step of importing the first data into the first copy further includes: writing the data in the first data other than the buffered data into the second data area.

13. The method according to claim 1, wherein, The first threshold is set as the upper limit of the storage capacity of the buffer.

14. An apparatus for performing a data import task, the data import task including importing first data into a first partition of a target database, the apparatus being disposed on a first node of the target database, the first node storing a first copy of the first partition; The device includes: The buffer module is configured to read first data into the buffer set up in the first node for the data import task; The first determining module is configured to determine the buffer state of the buffer when a first condition is met, wherein the first condition includes one of the following: the amount of data read into the buffer reaches a first threshold, or all the first data is read into the buffer; The second determining module is configured to determine the target import method, wherein the target import method is determined at least based on the buffer state of the first node; The import module is configured to import the first data into the first copy based on the target import method.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-13.

16. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-13.

Images