Multi-line insertion method and device in distributed database system, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-line insertion method and device in a distributed database system, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving an insertion instruction for a target distributed table, wherein the insertion instruction carries to-be-inserted data; when the insertion instruction is a target instruction, calculating a target data node corresponding to the to-be-inserted data, and disassembling the to-be-inserted data according to the target data node to obtain a plurality of sub-tables; the target instruction is to execute a multi-row insertion operation on a target distributed table; and sending the sub-table to the target data node. By adopting the method, the processing efficiency of the distributed database can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed database technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for multi-row insertion in a distributed database system. Background Technology

[0002] With the development of distributed database technology, data is typically stored across multiple data nodes based on a distribution key to improve system scalability and write capabilities. When processing multi-row insert commands, the system needs to distribute the data to be inserted to the corresponding data nodes for writing.

[0003] In traditional technology, the coordinating node sends an execution plan containing all the data to be inserted to a data node. The data node then performs routing calculations on the entire batch of data during the execution phase and forwards data that does not belong to this node to other target nodes to complete the writing.

[0004] However, this method requires transferring the entire batch of data between multiple data nodes, resulting in repeated routing calculations, cross-node data forwarding, and multiple serialization and deserialization processes, which increases network and CPU overhead and reduces write efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for multi-row insertion in a distributed database system that can improve the write efficiency of the distributed database system, in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for multi-row insertion in a distributed database system, including:

[0007] Receive an insert instruction for a target distributed table, the insert instruction carrying the data to be inserted;

[0008] When the insertion instruction is a target instruction, the target data node corresponding to the data to be inserted is calculated, and the data to be inserted is decomposed according to the target data node to obtain multiple sub-tables; the target instruction is to perform a multi-row insertion operation on the target distributed table.

[0009] Send the sub-table to the target data node.

[0010] In one embodiment, calculating the target data node corresponding to the data to be inserted includes:

[0011] Obtain the distribution strategy information of the target distributed table;

[0012] The distribution key of each sub-data is extracted from the data to be inserted based on the distribution strategy information;

[0013] The distribution key is hashed to obtain the initial logical partition number;

[0014] The corresponding target logical segment number is queried in the preset logical segment mapping table based on the initial logical segment number;

[0015] Based on the target logical shard number, determine the target data node corresponding to each data.

[0016] In one embodiment, after receiving the insert instruction for the target distributed table, the method further includes:

[0017] The insertion instruction is parsed to determine its operation type;

[0018] When the operation type is a multi-row insert operation, the target data of the insert instruction is determined based on the metadata of the distributed database system.

[0019] When the target data is the target distributed table, determine the number of sub-data in the data to be inserted;

[0020] When the number of sub-data items meets the preset conditions, the insertion instruction is used as the target instruction.

[0021] In one embodiment, after disassembling the data to be inserted according to the target data node, the method further includes:

[0022] Obtain the execution plan tree corresponding to the target distributed table, the execution plan tree including the insertion node for performing the insertion operation and the left subtree;

[0023] For each target data node, the execution plan tree is copied, and the data to be inserted in the left subtree of the copied execution plan tree is replaced with the corresponding sub-table to generate the execution plan subtree corresponding to each target data node.

[0024] In one embodiment, sending the sub-table to the target data node further includes:

[0025] The execution plan subtree is sent to the target data node; the execution plan subtree carries the sub-table; the target data node parses the execution plan subtree, reads the sub-table from the execution plan subtree, and inserts the sub-table into the local shard.

[0026] In one embodiment, before sending the execution plan subtree to the target data node, the process includes:

[0027] The execution plan subtree is serialized, and the serialized execution plan subtree is sent to the target data node.

[0028] Secondly, this application also provides a multi-row insertion device in a distributed database system, comprising:

[0029] A receiving module is used to receive an insertion instruction for a target distributed table, the insertion instruction carrying the data to be inserted;

[0030] The splitting module is used to calculate the target data node corresponding to the data to be inserted when the insertion instruction is a target instruction, and to split the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform a multi-row insertion operation on the target distributed table;

[0031] The sending module is used to send the sub-table to the target data node.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any of the above embodiments.

[0033] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the above embodiments.

[0034] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method of any of the above embodiments.

[0035] The multi-row insertion method, apparatus, computer device, computer-readable storage medium, and computer program product in the aforementioned distributed database system coordinate the nodes to parse the data to be inserted, determine the target data nodes corresponding to each row of data to be inserted, and split the data to be inserted into sub-tables corresponding to different data nodes, and then send each sub-table to its corresponding data node. Thus, each target data node only receives the sub-tables relevant to itself, eliminating the need for cross-node parsing, routing calculations, and data forwarding of the entire batch of data to be inserted, avoiding repeated splitting and multiple serialization processes, reducing network transmission and computational overhead, and thereby improving the execution efficiency of multi-row insertion operations. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an application environment diagram of a multi-row insertion method in a distributed database system, as shown in one embodiment.

[0038] Figure 2 This is a flowchart illustrating a multi-row insertion method in a distributed database system, as shown in one embodiment.

[0039] Figure 3 This is a flowchart illustrating the multi-row insertion steps in a distributed database system in an exemplary embodiment.

[0040] Figure 4 This is a structural block diagram of a multi-row insertion device in a distributed database system according to one embodiment;

[0041] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] For example, combined Figure 1 , Figure 1 This diagram illustrates the execution of a multi-row insert operation in a distributed database in the prior art. In the prior art, after receiving an insert instruction carrying multiple rows of data to be inserted, the coordinating node sends an execution plan carrying the entire batch of data to be inserted to a certain data node (such as DN1). During the execution phase, this data node performs routing calculations on the entire batch of data to be inserted based on the distribution key, determines the target data node corresponding to each record, and forwards data that does not belong to this node laterally to other data nodes at the same level (such as DN2 and DN3) to complete the write.

[0044] In one embodiment, such as Figure 2 As shown, a method for multi-row insertion in a distributed database system is provided. This embodiment illustrates the method using a terminal as an example. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0045] Step 202: Receive an insert instruction for the target distributed table; the insert instruction carries the data to be inserted.

[0046] In a distributed database system, a target distributed table refers to a data table stored across multiple data nodes in a sharded manner. The logical structure of this table is managed uniformly by a coordinating node, while its physical storage is distributed across multiple data nodes, with each data node storing only its corresponding data shard. For example, the target distributed table logically appears as a single, complete data table, but at the physical level, it is divided into multiple shards distributed across different data nodes. The coordinating node is responsible for receiving requests from clients and scheduling write operations across the data nodes.

[0047] An insert command is an instruction used to write data to a target distributed table, and may include information such as the insert operation type, the target data table, and the data to be inserted. For example, when a client initiates an insert statement as follows, it can be considered an insert command.

[0048] The data to be inserted refers to the multiple records to be written carried by the insert command, used for incremental updates of the target distributed table. It should be noted that the data to be inserted is not directly written to the target distributed table itself. Instead, according to the distribution strategy of the target distributed table, each record to be inserted is mapped to a corresponding data node, and then the data node writes it to its corresponding local shard.

[0049] Step 204: When the insertion instruction is represented as a target instruction, calculate the target data node corresponding to the data to be inserted, and decompose the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform multiple row insertion operations on the target distributed table.

[0050] The target instruction is to perform a multi-row insert operation on the target distributed table. A multi-row insert operation refers to an instruction that carries multiple records to be inserted in a single insert operation. Since the records contained in this type of insert instruction may correspond to different data nodes, it is necessary to determine the target data node for each record separately. This embodiment focuses on processing this type of instruction to improve the insertion execution efficiency in multi-row insert scenarios.

[0051] For example, the target distributed table is stored in a sharded manner across multiple data nodes, with each data node responsible for storing its corresponding data shard. In this case, routing calculations are needed for each record in the data to be inserted to determine the data node to which the record belongs, and based on this, the data to be inserted is split into multiple subsets, each subset corresponding to a target data node.

[0052] Optionally, the target distributed table is stored according to a distribution strategy, which indicates the distribution key used for data distribution, the number of logical shards, and the mapping relationship between logical shards and data nodes. Therefore, the data to be inserted can be split according to the distribution strategy to obtain subsets corresponding to each data node.

[0053] For example, when a target distributed table is sharded using a certain field as the distribution key, the values ​​of that field in different records may map to different data nodes. Therefore, the data to be inserted can be grouped based on this field, ultimately forming multiple subsets corresponding to different data nodes. For instance, a set of data to be inserted can be divided into two subsets after the distribution strategy is applied, where the first subset corresponds to the first data node and the second subset corresponds to the second data node.

[0054] Step 206: Send the sub-table to the target data node.

[0055] After obtaining the sub-tables corresponding to the target data nodes, each sub-table is sent to the corresponding target data node so that each target data node only receives data related to itself, thus eliminating the need to further split and forward the entire batch of data to be inserted across nodes.

[0056] Optionally, sending the sub-table to the target data node includes: sending the sub-table directly as content to be sent to the corresponding target data node via the network; wherein, after receiving the sub-table, the target data node can complete the data writing based on local execution logic.

[0057] Alternatively, the sub-table can be written into the corresponding execution plan subtree, so that the execution plan subtree carries the corresponding sub-table, and the execution plan subtree is sent to the target data node. For specific implementation details, please refer to the relevant descriptions in the following embodiments.

[0058] In the above embodiments, the coordinating node parses the data to be inserted, determines the target data node corresponding to each piece of data to be inserted, and splits the data to be inserted into sub-tables corresponding to different data nodes. Then, each sub-table is sent to the corresponding data node. Thus, each target data node only receives the sub-tables related to itself, eliminating the need for cross-node parsing, routing calculations, and data forwarding of the entire batch of data to be inserted. This avoids repeated splitting and multiple serialization processes, reduces network transmission and computational overhead, and thereby improves the execution efficiency of multi-row insertion operations.

[0059] In one embodiment, the above calculation of the target data node corresponding to the data to be inserted includes: obtaining the distribution strategy information of the target distributed table; extracting the distribution key of each sub-data from the data to be inserted according to the distribution strategy information; performing hash calculation on the distribution key to obtain the initial logical shard number; querying the corresponding target logical shard number in the preset logical shard mapping table according to the initial logical shard number; and determining the target data node corresponding to each data according to the target logical shard number.

[0060] Since the target distributed table is stored according to a preset distribution strategy, and different data to be inserted may correspond to different data nodes, it is necessary to extract the distribution key from the data to be inserted based on the distribution strategy information. The distribution key is used to represent the data field used for data sharding of the distributed table, and its value is used to distinguish the data shard to which different sub-data belongs.

[0061] Optionally, the distribution key is extracted from the data to be inserted based on the distribution strategy information. This includes: locating the corresponding data column in the data to be inserted based on the distribution key field indicated in the distribution strategy information, and reading the field values ​​of each record in that data column as the distribution key for the corresponding record. The distribution key field is a pre-defined key field in the target distributed table used for data sharding, such as a user identifier or order identifier.

[0062] To avoid data concentration on a few nodes and improve data distribution balance, a hash calculation is performed on the distribution key based on the distribution strategy information. This maps different sub-data to different logical shards according to the value of the distribution key, thus obtaining an initial logical shard number. The initial logical shard number represents the logical shard position of the data to be inserted. Further, to map the logical shard to the actual physical node, a lookup is performed in a pre-defined logical shard mapping table based on the initial logical shard number to obtain the corresponding target data node. The logical shard mapping table indicates the mapping relationship between logical shards and data nodes and can be dynamically adjusted according to system scaling.

[0063] Optionally, hashing the distribution key includes: performing a hash operation on the distribution key field value based on the number of logical shards to generate an initial logical shard number; wherein the number of logical shards is used to determine the numbering range into which the hash value falls, so as to distribute different records to multiple logical shards. For example, the data to be inserted can be hashed by performing a hash algorithm on the distribution key field value to obtain a hash result, and the hash result can be modulo the number of logical shards to obtain the initial logical shard number.

[0064] For example, in one exemplary embodiment, the system configures multiple initial logical shards for the target distributed table to represent the original location results obtained based on hash calculations. To support system scaling and shard migration, the system further maintains the mapping relationship between the initial logical shards and the target logical shards. For example, data with initial logical shard number 0 is still mapped to target logical shard number 0, while data with initial logical shard numbers 1 and 3 are mapped to target logical shard number 3, and data with initial logical shard number 2 is still mapped to target logical shard number 2. Simultaneously, the system also maintains the correspondence between target logical shards and actual data nodes. For example, the data node corresponding to target logical shard number 0 is DN1, the data node corresponding to target logical shard number 2 is DN2, and the data node corresponding to target logical shard number 3 is DN3. Therefore, when the distribution key of a certain sub-data is hashed to obtain an initial logical shard number of 1, the target logical shard number corresponding to the record can be determined to be 3 according to the mapping relationship from the initial logical shard to the target logical shard. Furthermore, the target logical shard number is mapped to the data node DN3 to determine that the record should eventually be written to DN3.

[0065] In the above embodiments, the coordinating node can accurately determine the target data node for the data to be inserted based on the distribution strategy of the target distributed table, thereby reducing data forwarding and redundant calculations during the execution phase and improving the execution efficiency of multi-row insertion operations.

[0066] In other embodiments, before splitting the data to be inserted according to the preset distribution strategy, the method further includes: updating the distribution strategy of the target distributed table based on the sharding prediction model trained on the target distributed table.

[0067] Specifically, the coordinating node extracts sample records from the historical storage data of the target distributed table, using the data fields used for sharding in the sample records as clustering input features. When the sharding field is numerical data, the numerical value is directly used as the clustering input; when the sharding field is character data, the character content is first hashed or mapped using a dictionary, and the encoded result is normalized before being used as the clustering input. Subsequently, the coordinating node executes a clustering algorithm (e.g., K-means clustering) on ​​the sample features, dividing the sample records into a preset number of clusters, each cluster corresponding to a logical shard, and recording the feature value range of the cluster centers. The coordinating node writes the correspondence between cluster numbers and logical shard numbers, as well as the feature value range of the cluster centers, into the distribution strategy information, forming the updated sharding strategy.

[0068] During the runtime phase, when the coordinating node splits the data to be inserted, it first extracts the sharding field from the data and converts it into numerical features in a manner consistent with the training phase. Then, it calculates the distance between this feature and the features of each cluster center, assigning the feature to the cluster with the smallest distance, thereby determining the corresponding target logical shard number. The coordinating node determines the target data node based on the mapping relationship between the logical shard number and the data node, and divides the data into multiple sub-tables according to the target data node. The cluster center is used to represent the feature representative value corresponding to a class of data records with similar distribution key value characteristics in the target distributed table.

[0069] Using the above method, the sharding prediction model is trained based on the historical storage distribution of the target distributed table. It can adaptively delineate logical sharding boundaries according to the value characteristics of the sharding field, grouping data records with similar characteristic values ​​into the same data node. Compared with fixed hash sharding, this method can reduce the probability of data skew, making the data distribution more balanced, and also helps to improve the efficiency of local queries based on the distributed table.

[0070] In one embodiment, after receiving the insert instruction for the target distributed table, the method further includes: parsing the insert instruction to determine the operation type of the insert instruction; when the operation type is a multi-row insert operation, determining the target data of the insert instruction based on the metadata of the distributed database system; when the target data is the target distributed table, determining the number of sub-data in the data to be inserted; and when the number of sub-data meets a preset condition, using the insert instruction as the target instruction.

[0071] This embodiment is used to determine whether to perform multi-row insertion optimization processing on the current insert instruction. After receiving the insert instruction, the system first performs syntax parsing on the instruction to identify its operation type; when the operation type is identified as a multi-row insert operation, it determines whether the data table targeted by the instruction is the target distributed table based on the metadata of the distributed database system; after confirming that the target data is the target distributed table, it counts the number of sub-data in the data to be inserted, and when the number meets a preset threshold, the insert instruction is determined as the target instruction to trigger the subsequent data splitting and targeted sending process.

[0072] The preset threshold is used to indicate the minimum data size required to trigger sharding optimization, so as to avoid unnecessary splitting and node allocation calculations when the data volume is small, thereby reducing system overhead. For example, the preset threshold can be set according to the processing capacity of the distributed database system, network bandwidth and inter-node forwarding overhead, such as 100 records, 1000 records, or dynamically adjusted according to business needs.

[0073] Optionally, parsing the insert command includes: performing lexical and syntactic analysis on the SQL command according to the database query parser, identifying the operation type information of the SQL command, such as INSERT, SELECT, or COPY, to determine whether it belongs to a multi-row insert operation.

[0074] Optionally, determining the target data based on the metadata of the distributed database system includes: querying the system metadata table to obtain the storage mode information of the target data table; if the metadata indicates that the table is a sharded table or a distributed table, then it is determined to be the target distributed table; if the metadata indicates that the table is a local table or a single-node table, then multi-row insertion optimization is unnecessary. The metadata includes the table's distribution key field, sharding rules, number of logical shards, and the mapping relationship between logical shards and physical nodes.

[0075] In the above embodiments, by performing semantic parsing and target table identification on the insertion command before data distribution and splitting, the subsequent optimization process is only triggered for multi-row insertion operations on distributed tables that meet the data scale requirements, thereby avoiding unnecessary splitting and node calculation logic when inserting single rows or local tables, and reducing system resource consumption.

[0076] In one embodiment, after decomposing the data to be inserted according to the target data node, the method further includes: obtaining the execution plan tree corresponding to the target distributed table of the data to be inserted, wherein the execution plan tree of the data to be inserted includes the insertion node for performing the insertion operation and the left subtree; for each target data node of the data to be inserted, copying the execution plan tree of the data to be inserted, and replacing the data to be inserted in the left subtree of the copied execution plan tree with the corresponding sub-table to be inserted, thereby generating the execution plan subtree corresponding to each target data node of the data to be inserted.

[0077] The execution plan tree describes the hierarchical relationship between operator nodes during a database insert operation. The root node is the insert node, which indicates the execution of the insert action; the left subtree consists of data nodes, which represent the data records to be inserted. For example, when the original insert instruction contains multiple records to be inserted, the data nodes are used to carry all the records to be inserted.

[0078] Specifically, the original execution plan tree corresponding to the target distributed table is first obtained. The original execution plan tree includes the insertion node for performing the insertion operation and the left subtree for storing the data to be inserted. Then, the execution plan tree is copied for each target data node, and the data to be inserted in the left subtree of the copied execution plan tree is replaced with the sub-table corresponding to that node to generate the execution plan subtree corresponding to the target data node.

[0079] After sharding the data to be inserted, a corresponding execution plan subtree is generated for each target data node. Each execution plan subtree retains the structure of the original execution plan tree, but the data nodes within it are replaced with the sub-tables corresponding to that target data node. For example, if the original insert operation contains six records, but after sharding, only two should be written to the first data node, then the execution plan subtree generated for the first data node will only include these two records; the remaining records will not be included in this subtree. Therefore, each data node does not need to process data unrelated to itself when executing its corresponding execution plan.

[0080] Specifically, the original execution plan tree corresponding to the target distributed table is first obtained. The original execution plan tree includes the insertion node for performing the insertion operation and the left subtree for storing the data to be inserted. Then, the execution plan tree is copied for each target data node, and the data to be inserted in the left subtree of the copied execution plan tree is replaced with the sub-table corresponding to that node to generate the execution plan subtree corresponding to the target data node.

[0081] Optionally, when generating the execution plan subtree, the method further includes: differentiating different types of execution steps based on their execution scope in the execution plan tree. Specifically, for execution steps that only involve writing the data to be inserted into local storage, the step is retained and executed locally on the corresponding target data node; for execution steps involving global constraint verification, cross-node consistency verification, or distributed index maintenance, the execution step is removed from the execution plan subtree, and the execution dependencies corresponding to the execution step are recorded, so that this part of the logic is uniformly triggered and executed by the coordination node or the global service module. This approach avoids the repeated execution of global verification logic on multiple target data nodes, thereby reducing redundant computation and coordination overhead.

[0082] Optionally, the generated multiple execution plan subtrees are merged, and multiple sub-tables corresponding to the same target data node are merged into a single batch execution node. This allows the target data node to process multiple insertion batches at once during execution, thereby reducing the number of execution scheduling and storage engine calls and improving batch insertion performance.

[0083] Furthermore, in one embodiment, sending the sub-table to the target data node further includes: sending an execution plan subtree to the target data node; the execution plan subtree carrying the sub-table; the target data node parsing the execution plan subtree, reading the sub-table from the execution plan subtree, and inserting the sub-table into a local shard.

[0084] For example, after the coordinating node completes the fragmentation of the data to be inserted and constructs corresponding execution plan subtrees for each target data node, it encodes and encapsulates each execution plan subtree to form an execution request message to be sent. The execution request message includes node identification information for identifying the target data node and the corresponding execution plan subtree. Based on the node identification information, the coordinating node sends each execution request message to the corresponding target data node through the internal communication channel of the distributed database system.

[0085] When a target data node receives an execution request message carrying an execution plan subtree, it first parses the message to reconstruct the data structure of the execution plan subtree. Then, based on the structural information in the execution plan subtree, it locates the position where the subtable is stored and reads the subtable belonging to its own node. Since the execution plan subtree has limited the data to be inserted to the subtable corresponding to the current target data node, the target data node does not need to filter or route data from other nodes.

[0086] After retrieving the sub-table, the target data node executes the insertion operation sequentially according to the predefined execution steps in the execution plan subtree. For example, based on the insertion node in the execution plan subtree, the local shard corresponding to the target distributed table on this node can be determined, and each record in the sub-table can be written to that local shard. During the insertion operation, the target data node can also combine constraint checks, logging, or index maintenance steps in the execution plan subtree to complete the full write process to the local shard.

[0087] Furthermore, in one embodiment, before sending the execution plan subtree to the target data node, the method includes: serializing the execution plan subtree and sending the serialized execution plan subtree to the target data node.

[0088] Before sending the execution plan subtree to the target data node, the execution plan subtree and its carried subtables can be serialized to adapt to the internal network transmission format of the distributed database system. After receiving the corresponding execution request message, the target data node deserializes the execution plan subtree to recover the execution plan structure and its carried subtables, which can be directly used by the local execution engine. In this way, the expressive power of the execution plan can be maintained while network transmission efficiency is also taken into account.

[0089] Optionally, when inserting the sub-table into the local shard on the target data node, multiple records in the sub-table can be written in batches within a single transaction to reduce the number of transaction starts and commits, thereby further improving the write throughput in batch insertion scenarios.

[0090] In the above embodiments, by distributing the data sub-table along with the execution plan subtree, the target data node can directly execute data writing based on the local execution plan without processing the full amount of data, thereby reducing the processing overhead on the data node side. At the same time, the execution plan subtree carries the insertion logic, so that the insertion process follows the database's native execution path, avoiding the introduction of additional data format conversion due to directly sending data, and improving the overall execution efficiency of distributed batch insertion.

[0091] In one exemplary embodiment, combined with Figure 3 , Figure 3 This is a schematic diagram of a multi-row insertion method in a distributed database system according to an exemplary embodiment.

[0092] like Figure 3 As shown, after the coordinating node CN receives a multi-row insert command for the target distributed table, it first calculates the target data node corresponding to each row of data in the data to be inserted through the path distribution adjustment module (Adjust_path_distribution). Then, in the plan construction and initialization phase (ExecInitRemoteSubPlan), the set of data to be inserted (values ​​list) carried in the insert command is split into multiple data subsets, and a corresponding execution plan subtree (insert -> values ​​scan) is constructed for each target data node. Each execution plan subtree contains only the data subset belonging to the corresponding target data node. Finally, in the plan delivery phase (ExecFinishInitRemoteSubplan), CN sequentially traverses each target data node, serializes the execution plan subtree corresponding to each target data node, and sends it to the corresponding target data node.

[0093] After receiving the serialized execution plan subtree, each data node (DN1, DN2, DN3) deserializes and parses it to restore the insert->values ​​scan execution plan structure. The insert node then directly reads the data subset carried in the values ​​scan and performs the local insertion operation, so that the data falls directly into the corresponding local shard. This avoids full data parsing, cross-node forwarding, and repeated filtering, thereby improving the overall execution efficiency of multi-row insertion.

[0094] In the above embodiments, by splitting the data and placing it in front of the coordination node and sending it to the corresponding execution plan subtree, the data is directly written to disk after being distributed once, which reduces the overhead of repeated parsing and forwarding and improves the execution efficiency of large-scale multi-line insertion.

[0095] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0096] Based on the same inventive concept, this application also provides a multi-row insertion device for a distributed database system for implementing the multi-row insertion method in the distributed database system described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the multi-row insertion device in a distributed database system provided below can be found in the limitations of the multi-row insertion method in the distributed database system described above, and will not be repeated here.

[0097] In one exemplary embodiment, such as Figure 4 As shown, a multi-row insertion device for a distributed database system is provided, comprising: a receiving module 100, a splitting module 200, and a sending module 300, wherein:

[0098] The receiving module 100 is used to receive an insert instruction for the target distributed table, the insert instruction carrying the data to be inserted;

[0099] The splitting module 200 is used to calculate the target data node corresponding to the data to be inserted when the insertion instruction is a target instruction, and to split the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform a multi-row insertion operation on the target distributed table.

[0100] The sending module 300 is used to send the sub-table to the target data node.

[0101] In one embodiment, the above-mentioned splitting module includes:

[0102] The acquisition unit is used to acquire the distribution strategy information of the target distributed table.

[0103] The extraction unit is used to extract the distribution key of each sub-data from the data to be inserted based on the distribution strategy information.

[0104] The hash calculation unit is used to perform hash calculations on the distribution key to obtain the initial logical shard number.

[0105] The mapping unit is used to query the corresponding target logical segment number in the preset logical segment mapping table based on the initial logical segment number.

[0106] The node determination unit is used to determine the target data node corresponding to each data based on the target logical fragment number.

[0107] In the above embodiments, the apparatus further includes a parsing module, which includes:

[0108] The type determination unit is used to perform syntax parsing on the insert instruction and determine the operation type of the insert instruction.

[0109] The target determination unit is used to determine the target data of the insertion instruction based on the metadata of the distributed database system when the operation type is a multi-row insertion operation.

[0110] The quantity judgment unit is used to determine the number of sub-data in the data to be inserted when the target data is a target distributed table.

[0111] The instruction judgment unit is used to select the insertion instruction as the target instruction when the number of sub-data meets the preset conditions.

[0112] In one embodiment, the above-described apparatus includes a plan tree generation module, comprising:

[0113] The execution plan tree acquisition unit is used to obtain the execution plan tree corresponding to the target distributed table. The execution plan tree includes the insertion node used to perform the insertion operation and the left subtree.

[0114] The subtree generation unit is used to copy the execution plan tree for each target data node, and replace the data to be inserted in the left subtree of the copied execution plan tree with the corresponding sub-table to generate the execution plan subtree corresponding to each target data node.

[0115] In one embodiment, the sending module is used to send the execution plan subtree to the target data node; the execution plan subtree carries a subtable; the target data node parses the execution plan subtree, reads the subtable from the execution plan subtree, and inserts the subtable into its local shard.

[0116] In one embodiment, the above-mentioned sending module further includes:

[0117] The serialization unit is used to serialize the execution plan subtree and send the serialized execution plan subtree to the target data node.

[0118] The modules in the multi-row insertion device of the aforementioned distributed database system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0119] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores a target distributed table. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a multi-row insertion method in a distributed database system.

[0120] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: receiving an insertion instruction for a target distributed table, the insertion instruction carrying data to be inserted; when the insertion instruction is a target instruction, calculating the target data node corresponding to the data to be inserted, and disassembling the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform multiple row insertion operations on the target distributed table; and sending the sub-tables to the target data node.

[0122] In one embodiment, when the processor executes the computer program, it further performs the following steps: calculating the target data node corresponding to the data to be inserted, including: obtaining the distribution strategy information of the target distributed table; extracting the distribution key of each sub-data from the data to be inserted according to the distribution strategy information; performing hash calculation on the distribution key to obtain the initial logical shard number; querying the corresponding target logical shard number in the preset logical shard mapping table according to the initial logical shard number; and determining the target data node corresponding to each data according to the target logical shard number.

[0123] In one embodiment, when the processor executes the computer program, it further performs the following steps: parsing the insertion instruction to determine the operation type of the insertion instruction; when the operation type is a multi-row insertion operation, determining the target data of the insertion instruction based on the metadata of the distributed database system; when the target data is a target distributed table, determining the number of sub-data in the data to be inserted; when the number of sub-data meets a preset condition, using the insertion instruction as the target instruction.

[0124] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the execution plan tree corresponding to the target distributed table, the execution plan tree including the insertion node for performing the insertion operation and the left subtree; for each target data node, copying the execution plan tree, and replacing the data to be inserted in the left subtree of the copied execution plan tree with the corresponding sub-table, thereby generating the execution plan subtree corresponding to each target data node.

[0125] In one embodiment, when the processor executes the computer program, it further performs the following steps: sending an execution plan subtree to a target data node; the execution plan subtree carrying a subtable; the target data node parsing the execution plan subtree, reading the subtable from the execution plan subtree, and inserting the subtable into a local shard.

[0126] In one embodiment, when the processor executes the computer program, it further performs the following steps: serializing the execution plan subtree and sending the serialized execution plan subtree to the target data node.

[0127] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When the computer program is executed by a processor, it performs the following steps: receiving an insert instruction for a target distributed table, the insert instruction carrying data to be inserted; when the insert instruction is a target instruction, calculating the target data node corresponding to the data to be inserted, and disassembling the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform multiple row insertion operations on the target distributed table; and sending the sub-tables to the target data node.

[0128] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: calculating the target data node corresponding to the data to be inserted, including: obtaining the distribution strategy information of the target distributed table; extracting the distribution key of each sub-data from the data to be inserted according to the distribution strategy information; performing hash calculation on the distribution key to obtain the initial logical shard number; querying the corresponding target logical shard number in the preset logical shard mapping table according to the initial logical shard number; and determining the target data node corresponding to each data according to the target logical shard number.

[0129] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: parsing the insertion instruction to determine the operation type of the insertion instruction; when the operation type is a multi-row insertion operation, determining the target data of the insertion instruction based on the metadata of the distributed database system; when the target data is a target distributed table, determining the number of sub-data in the data to be inserted; when the number of sub-data meets a preset condition, using the insertion instruction as the target instruction.

[0130] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the execution plan tree corresponding to the target distributed table, the execution plan tree including the insertion node for performing the insertion operation and the left subtree; for each target data node, copying the execution plan tree, and replacing the data to be inserted in the left subtree of the copied execution plan tree with the corresponding sub-table, thereby generating the execution plan subtree corresponding to each target data node.

[0131] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sending an execution plan subtree to a target data node; the execution plan subtree carrying a subtable; the target data node parsing the execution plan subtree, reading the subtable from the execution plan subtree, and inserting the subtable into a local shard.

[0132] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: serializing the execution plan subtree and sending the serialized execution plan subtree to the target data node.

[0133] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: receiving an insert instruction for a target distributed table, the insert instruction carrying data to be inserted; when the insert instruction is a target instruction, calculating the target data node corresponding to the data to be inserted, and disassembling the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform multiple row insert operations on the target distributed table; and sending the sub-tables to the target data node.

[0134] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: calculating the target data node corresponding to the data to be inserted, including: obtaining the distribution strategy information of the target distributed table; extracting the distribution key of each sub-data from the data to be inserted according to the distribution strategy information; performing hash calculation on the distribution key to obtain the initial logical shard number; querying the corresponding target logical shard number in the preset logical shard mapping table according to the initial logical shard number; and determining the target data node corresponding to each data according to the target logical shard number.

[0135] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: parsing the insertion instruction to determine the operation type of the insertion instruction; when the operation type is a multi-row insertion operation, determining the target data of the insertion instruction based on the metadata of the distributed database system; when the target data is a target distributed table, determining the number of sub-data in the data to be inserted; when the number of sub-data meets a preset condition, using the insertion instruction as the target instruction.

[0136] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the execution plan tree corresponding to the target distributed table, the execution plan tree including the insertion node for performing the insertion operation and the left subtree; for each target data node, copying the execution plan tree, and replacing the data to be inserted in the left subtree of the copied execution plan tree with the corresponding sub-table, thereby generating the execution plan subtree corresponding to each target data node.

[0137] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sending an execution plan subtree to a target data node; the execution plan subtree carrying a subtable; the target data node parsing the execution plan subtree, reading the subtable from the execution plan subtree, and inserting the subtable into a local shard.

[0138] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: serializing the execution plan subtree and sending the serialized execution plan subtree to the target data node.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for inserting multiple rows in a distributed database system, characterized in that, The distributed database system includes a coordinating node; the method is applied to the coordinating node; the method includes: Receive an insert instruction for a target distributed table, the insert instruction carrying the data to be inserted; When the insertion instruction is a target instruction, the target data node corresponding to the data to be inserted is calculated, and the data to be inserted is decomposed according to the target data node to obtain multiple sub-tables; the target instruction is to perform a multi-row insertion operation on the target distributed table. Send the sub-table to the target data node.

2. The method according to claim 1, characterized in that, The calculation of the target data node corresponding to the data to be inserted includes: Obtain the distribution strategy information of the target distributed table; The distribution key of each sub-data is extracted from the data to be inserted based on the distribution strategy information; The distribution key is hashed to obtain the initial logical partition number; The corresponding target logical segment number is queried in the preset logical segment mapping table based on the initial logical segment number; Based on the target logical shard number, determine the target data node corresponding to each data.

3. The method according to claim 1, characterized in that, After receiving the insert instruction for the target distributed table, the process further includes: The insertion instruction is parsed to determine its operation type; When the operation type is a multi-row insert operation, the target data of the insert instruction is determined based on the metadata of the distributed database system. When the target data is the target distributed table, determine the number of sub-data in the data to be inserted; When the number of sub-data items meets the preset conditions, the insertion instruction is used as the target instruction.

4. The method according to claim 1, characterized in that, After the step of disassembling the data to be inserted according to the target data node, it further includes: Obtain the execution plan tree corresponding to the target distributed table, the execution plan tree including the insertion node for performing the insertion operation and the left subtree; For each target data node, the execution plan tree is copied, and the data to be inserted in the left subtree of the copied execution plan tree is replaced with the corresponding sub-table to generate the execution plan subtree corresponding to each target data node.

5. The method according to claim 4, characterized in that, Sending the sub-table to the target data node further includes: The execution plan subtree is sent to the target data node; the execution plan subtree carries the sub-table; the target data node parses the execution plan subtree, reads the sub-table from the execution plan subtree, and inserts the sub-table into the local shard.

6. The method according to claim 5, characterized in that, Before sending the execution plan subtree to the target data node, the following steps are included: The execution plan subtree is serialized, and the serialized execution plan subtree is sent to the target data node.

7. A multi-row insertion device in a distributed database system, characterized in that, The device includes: A receiving module is used to receive an insertion instruction for a target distributed table, the insertion instruction carrying the data to be inserted; The splitting module is used to calculate the target data node corresponding to the data to be inserted when the insertion instruction is a target instruction, and to split the data to be inserted according to the target data node to obtain multiple sub-tables; the target instruction is to perform a multi-row insertion operation on the target distributed table; The sending module is used to send the sub-table to the target data node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images