Table structure modification method and system for distributed database

Abstract

The embodiment of the invention discloses a method and a system for modifying a table structure of a distributed database. The method comprises the following steps: receiving a table structure modification instruction of a target user through a target computing node, splitting the table structure modification instruction to determine a plurality of task instructions, and sending the plurality of task instructions to a target storage node based on instruction priorities; modifying a table structure in a target storage node based on the received task instructions through the target storage node to obtain a target table, generating a modification completion signal based on the target table, and sending the modification completion signal to a target computing node; wherein in the process that the target storage node executes the task instruction, the target storage node does not block the data manipulation operation. Through the technical scheme of the embodiment of the invention, the data manipulation operation and the table structure modification operation can be executed in parallel, and the operation execution efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method and system for modifying the table structure of a distributed database. Background Art

[0002] In a distributed database, there is a situation where the table structure needs to be modified during the execution of a data manipulation language (DML) operation. The table structure modification operation can define the table structure in the database through a DDL statement.

[0003] At present, when modifying the table structure in a distributed database, data manipulation operations are blocked, and data manipulation operations are started again when the table structure modification is completed. This table structure modification method cannot perform data manipulation operations and table structure modification operations in parallel. It can be seen that a method that can perform data manipulation operations and table structure modification operations in parallel is urgently needed. Summary of the invention

[0004] The embodiment of the present invention provides a method and system for modifying the table structure of a distributed database, so as to realize the parallel execution of data manipulation operation and table structure modification operation, thereby improving the operation execution efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a method for modifying a table structure of a distributed database, comprising:

[0006] Receive a table structure modification instruction of a target user through a target computing node, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to a target storage node based on instruction priorities;

[0007] Modifying the table structure in the target storage node based on the received multiple task instructions by the target storage node to obtain a target table, and generating a modification completion signal based on the target table and sending it to the target computing node;

[0008] In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

[0009] Optionally, the method also includes: receiving a table structure modification operation of a target user on a target storage node through a client, converting the table structure modification operation into a table structure modification instruction, and sending the table structure modification instruction to a target computing node; generating a table structure modification success flag based on a received modification completion signal through the target computing node, and sending the table structure modification success flag to the client for display to the target user.

[0010] Optionally, the method also includes: the table structure modification instruction is generated by the target user for the target storage node corresponding to the target computing node in the distributed database; the distributed database includes multiple computing nodes; the target storage node includes at least one storage node shard; the storage node shard includes a master data node.

[0011] Optionally, the method also includes: based on the received first task instruction, the master data node creates a table structure according to the original table to obtain a temporary table structure, and sends the execution result of the first task instruction to the target computing node, so that the target computing node sends the second task instruction to the master data node; based on the received second task instruction, the master data node copies the current data in the original table to the temporary table structure to obtain the current temporary table, and sends the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node; based on the received third task instruction, the master data node deletes the original table, and modifies the table name of the current temporary table to the table name corresponding to the original table, so as to obtain the target table.

[0012] Optionally, the method further includes: the storage node shard further includes at least one backup data node.

[0013] Optionally, the method also includes: based on the received first task instruction, the master data node creates a table structure according to the original table to obtain a temporary table structure of the master data node, generates a temporary table structure creation log, and sends the temporary table structure creation log to the standby data node; based on the temporary table structure creation log, the standby data node creates a table structure to obtain the temporary table structure of the standby data node, and sends the execution result of the first task instruction to the master data node; and sends the received execution result to the target computing node through the master data node, so that the target computing node sends the second task instruction to the master data node.

[0014] Optionally, the method also includes: copying the current data in the original table to the temporary table structure based on the received second task instruction through the master data node to obtain the current temporary table of the master data node, generating a current data replication log, and sending the current data replication log to the standby data node; replicating the current data based on the current data replication log through the standby data node to obtain the current temporary table of the standby data node, and sending the execution result of the second task instruction to the master data node; sending the received execution result to the target computing node through the master data node, so that the target computing node sends the third task instruction to the master data node.

[0015] Optionally, the method also includes: deleting the original table based on the received third task instruction through the master data node, and modifying the table name of the current temporary table to the table name corresponding to the original table, obtaining the target table of the master data node, and generating a target table replacement log, and sending the target table replacement log to the standby data node; replacing the original table based on the target table replacement log through the standby data node to obtain the target table of the standby data node, and sending the execution result of the third task instruction to the master data node.

[0016] Optionally, the method further includes: obtaining the current resource occupancy rate of the server where the target storage node is located through the load monitoring module when the target storage node executes the multiple task instructions, and controlling the execution speed of the multiple task instructions based on the current resource occupancy rate and a preset occupancy threshold.

[0017] In a second aspect, an embodiment of the present invention further provides a system for modifying a table structure of a distributed database, the system comprising: a target computing node and a target storage node;

[0018] The target computing node is used to receive the table structure modification instruction of the target user, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to the target storage node based on the instruction priority;

[0019] The target storage node is used to modify the table structure in the target storage node based on the received multiple task instructions to obtain a target table, and generate a modification completion signal based on the target table and send it to the target computing node;

[0020] In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

[0021] The technical solution of the embodiment of the present invention is to receive the table structure modification instruction of the target user through the target computing node, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to the target storage node based on the instruction priority; the target storage node modifies the table structure in the target storage node based on the received multiple task instructions to obtain the target table, and generates a modification completion signal based on the target table and sends it to the target computing node; wherein, in the process of the target storage node executing the task instruction, the target storage node does not block the data manipulation operation, thereby realizing the parallel execution of the data manipulation operation and the table structure modification operation, and improving the operation execution efficiency.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flow chart of a method for modifying a table structure of a distributed database provided in Embodiment 1 of the present invention;

[0025] Figure 2 It is a flow chart of a method for modifying a table structure of a distributed database provided in Embodiment 2 of the present invention;

[0026] Figure 3 It is a flow chart of a method for modifying the table structure of a distributed database provided in Embodiment 3 of the present invention;

[0027] Figure 4 This is an example diagram of a scenario of executing a table structure modification instruction involved in Embodiment 3 of the present invention;

[0028] Figure 5 It is a structural diagram of a distributed database table structure modification system provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a method for modifying a table structure of a distributed database is provided for the first embodiment of the present invention. This embodiment is applicable to the case of modifying a table structure. The method can be executed by a system for modifying a table structure of a distributed database. The system for modifying a table structure of a distributed database can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0033] S110, receiving a table structure modification instruction of a target user through a target computing node, splitting the table structure modification instruction into multiple task instructions, and sending the multiple task instructions to a target storage node based on instruction priorities.

[0034] Among them, the distributed database may include multiple computing nodes. The target computing node may refer to the computing node that needs to execute the table structure modification instruction issued by the target user. The task instruction may correspond to each task execution stage in the table structure modification instruction. Completing a task instruction can be understood as completing a phased task. When a high-priority task instruction is executed, the task instruction of this priority level may continue to be executed until all task instructions are executed. The storage node may be used to store data in a tabular form. The target storage node may refer to the storage node that needs to modify the table structure.

[0035] Specifically, a target computing node receives a table structure modification instruction of a target user, such as a DDL table structure modification statement, and splits the table structure modification instruction to determine multiple task instructions. A session is initiated to a target storage node through the target computing node, and multiple task instructions are sent to the target storage node based on instruction priority in the session.

[0036] S120. Modify the table structure in the target storage node based on the received multiple task instructions through the target storage node to obtain a target table, and generate a modification completion signal based on the target table and send it to the target computing node.

[0037] Among them, during the process of the target storage node executing the task instruction, the target storage node does not block the data manipulation operation. The target table may refer to a table whose table structure modification is completed. The parameter values ​​in the target table are parameter values ​​obtained by performing data manipulation operations simultaneously during the execution of the table structure modification instruction. Compared with the situation in which the parameter values ​​in the table before and after the modification remain unchanged in the traditional table structure modification method, the embodiment of the present invention can execute the table structure modification instruction and the data manipulation operation in parallel, thereby improving the execution efficiency of the data manipulation operation. The modification completion signal can be used to characterize the situation in which the table structure modification in the target storage node is completed.

[0038] Specifically, the target storage node receives multiple task instructions sent by the target computing node based on the established session, and executes the task instructions in sequence according to the task priority. When the last task instruction is executed, the target table with the table structure modified is obtained, and a modification completion signal is generated based on the target table. The modification completion signal is sent to the target computing node to prompt the target computing node that the table structure modification instruction is completed.

[0039] The technical solution of the embodiment of the present invention receives the table structure modification instruction of the target user through the target computing node, splits the table structure modification instruction into multiple task instructions, and sends the multiple task instructions to the target storage node based on the instruction priority; the target storage node modifies the table structure in the target storage node based on the received multiple task instructions to obtain the target table, and generates a modification completion signal based on the target table and sends it to the target computing node; wherein, in the process of the target storage node executing the task instruction, the target storage node does not block the data manipulation operation, thereby realizing the parallel execution of the data manipulation operation and the table structure modification operation, and improving the operation execution efficiency.

[0040] Based on the above technical solution, the method also includes: receiving the table structure modification operation of the target user on the target storage node through the client, and converting the table structure modification operation into a table structure modification instruction, and sending the table structure modification instruction to the target computing node; generating a table structure modification success flag based on the received modification completion signal through the target computing node, and sending the table structure modification success flag to the client for display to the target user.

[0041] The client may refer to a device that can be directly operated by a user. The table structure modification success mark may refer to a preset icon that is displayed to the target user by the user.

[0042] Specifically, the target user creates a table structure modification operation for the target storage node through an application on the client, so that the client converts the table structure modification operation into a table structure modification instruction understandable by the target computing node and the target storage node, and sends the table structure modification instruction to the target computing node. When the target computing node receives the modification completion signal, the target computing node generates a table structure modification success flag based on the received modification completion signal, and sends the table structure modification success flag to the client for display to the target user, so as to prompt the target user that the table structure has been modified.

[0043] Based on the above technical solution, the method also includes: when executing multiple task instructions on the target storage node through the load monitoring module, obtaining the current resource occupancy rate of the server where the target storage node is located, and controlling the execution speed of multiple task instructions based on the current resource occupancy rate and the preset occupancy threshold.

[0044] The resource utilization rate may refer to the utilization rate of server resources when executing task instructions. The resource utilization rate may include but is not limited to CPU utilization rate and IO utilization rate. The server may refer to the host that hosts the target storage node.

[0045] Specifically, when executing multiple task instructions at the target storage node through the load monitoring module, the current resource occupancy rate of the server where the target storage node is located is obtained. If the current resource occupancy rate is greater than or equal to the preset occupancy threshold, the execution speed of the task instructions executed at the current moment is reduced, thereby ensuring the execution speed of the data manipulation operation. If the current resource occupancy rate is less than the preset occupancy threshold, the execution speed of the task instructions executed at the current moment can be appropriately increased, thereby further improving the table structure modification speed while ensuring the execution speed of the data manipulation operation.

[0046] Embodiment 2

[0047] Figure 2 This is a flowchart of a method for modifying the table structure of a distributed database provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the target table. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0048] S210. Receive a table structure modification instruction of a target user through a target computing node, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to a target storage node based on instruction priorities.

[0049] The table structure modification instruction is generated by the target user for the target storage node corresponding to the target computing node in the distributed database; the distributed database includes multiple computing nodes; the target storage node includes at least one storage node shard; the storage node shard includes a master data node. The advantage of this setting is that the table structure modification operation can be performed on multiple storage node shards in the target storage node in parallel, further improving the efficiency of table structure modification.

[0050] S220. Based on the received first task instruction, the master data node creates a temporary table structure according to the original table, and sends the execution result of the first task instruction to the target computing node, so that the target computing node sends the second task instruction to the master data node.

[0051] Among them, the first task instruction may refer to the task instruction corresponding to the preparation stage of the table structure modification. For example, the first task instruction may be expressed as DDL table structure_PRE (preparation stage). The first task instruction may be used to create a new table structure, that is, a temporary table structure. The original table may refer to the data table in the main data node when the table structure is not modified. The temporary table structure may refer to the table structure that the target user expects to be modified to. The execution result may include: execution success and execution failure.

[0052] Specifically, the first task instruction is received in the session by the master data node, and based on the received first task instruction, a table structure is created on the basis of the original table to obtain a temporary table structure. If the temporary table structure is successfully created, the execution result is a successful execution. If the temporary table structure is not successfully created, the execution result is an execution failure. The execution result of the first task instruction is sent to the target computing node through the master data node. The execution result is received in the session by the target computing node and the execution result is detected. If the execution result is an execution failure, the execution result is fed back to the client used by the target user. If the execution result is a successful execution, the second task instruction is sent to the master data node.

[0053] S230. Based on the received second task instruction, the master data node copies the current data in the original table to the temporary table structure to obtain the current temporary table, and sends the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node.

[0054] Among them, the second task instruction may refer to the task instruction corresponding to the execution phase of the table structure modification. For example, the second task instruction may be expressed as DDL table structure_EXEC (execution phase). The second task instruction may be used to copy the updated data in the original table to the newly created temporary table structure. The current data may refer to the parameter value obtained by performing the data manipulation operation simultaneously during the second task instruction process, that is, the last updated parameter value in the target table. The current temporary table may refer to a temporary table with parameter values ​​whose data is updated to the current moment. The table structure of the temporary table is a temporary table structure.

[0055] Specifically, when the master data node completes the execution of the first task instruction, the data manipulation operation can modify and read the parameter values ​​of the original data in the original table, thereby continuously updating the original data in the original table to obtain the current data. The master data node copies the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table, and sends the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node.

[0056] S240. Based on the received third task instruction, the master data node deletes the original table and modifies the table name of the current temporary table to the table name corresponding to the original table to obtain a target table.

[0057] The third task instruction may refer to a task instruction corresponding to the commit phase of the table structure modification. For example, the third task instruction may be expressed as DDL table structure_COMMIT (commit phase). The third task instruction may be used to replace the original table with the current temporary table. The target table may refer to a data table that can replace the original table after renaming.

[0058] Specifically, the master data node deletes the original table based on the received third task instruction, and modifies the table name of the current temporary table to the table name corresponding to the original table, so as to obtain the target table.

[0059] S250. Generate a modification completion signal based on the target table through the master data node and send it to the target computing node.

[0060] In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

[0061] The technical solution of the embodiment of the present invention is that the master data node creates a table structure based on the original table based on the received first task instruction to obtain a temporary table structure, and sends the execution result of the first task instruction to the target computing node, so that the target computing node sends the second task instruction to the master data node; the master data node copies the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table, and sends the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node; the master data node deletes the original table based on the received third task instruction, and modifies the table name of the current temporary table to the table name corresponding to the original table to obtain the target table. Thus, the table structure modification instruction is split into the first task instruction, the second task instruction and the third task instruction according to different stages, and a new table structure is created based on the first task instruction, and the data in the original table that is changed according to the data manipulation operation before the second task instruction is completed is copied by the second task instruction and used as the initial data of the new table, and the third task instruction is used to replace the original table with the new table, thereby realizing the accurate modification of the table structure while taking into account the execution of the data manipulation operation and the table structure modification operation.

[0062] Embodiment 3

[0063] Figure 3This is a flowchart of a method for modifying the table structure of a distributed database provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the target table. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 3 As shown, the method includes:

[0064] S310, receiving a table structure modification instruction of a target user through a target computing node, splitting the table structure modification instruction into multiple task instructions, and sending the multiple task instructions to a target storage node based on instruction priorities.

[0065] The storage node shard also includes at least one backup data node. Figure 4 A scenario example of executing a table structure modification instruction is given. Figure 4 In this scenario, there is an application layer, three computing nodes (CN), and two storage node shards. Each storage node shard contains a primary data node (DN primary) and three backup data nodes (DN backup). The application in the target user client belongs to the application layer. Computing node CN2 is the target computing node. The target storage nodes include: storage node shard 1 and storage node shard 2. In storage node shard 1, the primary data node is DN1 primary, and the backup nodes include: DN1 backup 1, DN1 backup 2, and DN1 backup 3. In storage node shard 2, the primary data node is DN2 primary, and the backup nodes include: DN2 backup 1, DN2 backup 2, and DN2 backup 3.

[0066] Exemplarily, the application layer initiates a DDL table structure modification instruction to the computing node CN2 of the distributed database. The computing node CN2 splits the DDL table structure statement, i.e., the table structure modification instruction, into task instructions corresponding to three stages: DDL table structure_PRE (preparation stage), DDL table structure_EXEC (execution stage), and DDL table structure_COMMIT (commitment stage). Then, the computing node CN2 initiates sessions to the DN1 master in the storage node shard 1 and the DN2 in the storage node shard 2, respectively. Each session will execute these three stages.

[0067] S320. The primary data node creates a temporary table structure of the primary data node based on the original table based on the received first task instruction, generates a temporary table structure creation log, and sends the temporary table structure creation log to the standby data node.

[0068] The log may refer to a binary log, which can be used to record and transmit operations in the primary data node to the standby data node, so that the standby data node can replicate the operations in the primary data node based on the log content.

[0069] Specifically, in Figure 4On the basis of the above, the DN1 master receives the DDL table structure _PRE (preparation phase) instruction issued by the computing node CN2, upgrades the metadata shared lock of the original table to an exclusive lock. The data in the table cannot be modified at this stage, and the data manipulation operation DML will be blocked for a very short time. A temporary table structure is created, and then the statement for creating the temporary table structure is transmitted to DN1 backup 1, DN1 backup 2, and DN1 backup 3 through the log file.

[0070] S330: Create a table structure based on the temporary table structure creation log by the standby data node to obtain a temporary table structure of the standby data node, and send the execution result of the first task instruction to the master data node.

[0071] Specifically, in Figure 4 Based on the log, DN1 backup 1, DN1 backup 2 and DN1 backup 3 receive the logs respectively, create temporary table structures according to the logs to obtain their corresponding temporary table structures, and feed back the successful results of the execution to DN1 master.

[0072] S340. Send the received execution result to the target computing node through the master data node, so that the target computing node sends the second task instruction to the master data node.

[0073] Specifically, in Figure 4 On the basis of, when the DN1 master receives the execution result, it records the execution position GTID_1 in the log at this time, and sends the execution result to the computing node CN2, so that the computing node CN2 sends the DDL table structure _EXEC (execution phase) phase instruction to the DN1 master.

[0074] It should be noted that the computing node CN2 will send the first task instruction to the master DN1 and the master DN2 at the same time, so that the master DN1 and the master DN2 execute the task instruction in parallel, further improving the efficiency of table structure modification.

[0075] S350. Based on the received second task instruction, the master data node copies the current data in the original table to the temporary table structure to obtain the current temporary table of the master data node, generates a current data replication log, and sends the current data replication log to the standby data node.

[0076] Among them, the current data replication log may include two application difference logs. For example, the first application difference log is GTID_2 minus GTID_1. The second application difference log is GTID_3 minus GTID_2. The first application difference log is used to record the execution operation of copying the original data in the original table to the temporary table structure. GTID_2 may refer to the execution position when the original data replication operation is completed. The second application difference log is used to record the execution operation of the data manipulation operation to modify the original data. GTID_3 may refer to the execution position when the data manipulation operation is blocked.

[0077] Specifically, in Figure 4 Based on the above, computing node CN2 initiates the DDL table structure_EXEC (execution phase) phase instruction to DN1 master. Upon receiving the instruction, DN1 master will downgrade the original table metadata exclusive lock to a shared lock, so that data manipulation operations are not blocked in this phase, so that the data page can be parsed at the engine layer and the original data of the original table can be copied to the temporary table. After copying the data, the GTID position point of DN1 master is recorded as GTID_2. In this phase, the difference log (GTID_2 minus GTID-1) about the original table in the application log is applied to the temporary table. At this time, after the original table metadata shared lock is upgraded to an exclusive lock, the GTID position point of DN1 master is recorded as GTID_3. During this period, many data manipulation operations can be performed, and then the difference log (GTID_3 minus GTID-2) about the original table in the application log is applied to the temporary table, thereby obtaining the current data replication log, and sending the current data replication log to DN1 backup 1, DN1 backup 2, and DN1 backup 3.

[0078] S360. The standby data node performs current data replication based on the current data replication log to obtain a current temporary table of the standby data node, and sends the execution result of the second task instruction to the master data node.

[0079] Specifically, in Figure 4 On the basis of, DN1 backup 1, DN1 backup 2 and DN1 backup 3 respectively receive the current data replication log, and replicate the execution operations in the current data replication log to copy the current data to their respective corresponding current temporary tables, and send the successful result of the execution of the second task instruction to DN1 master.

[0080] S370. Send the received execution result to the target computing node through the master data node, so that the target computing node sends the third task instruction to the master data node.

[0081] Specifically, in Figure 4 On the basis of, the DN1 master sends the received execution result to the computing node CN2, so that the computing node CN2 sends the third task instruction to the DN1 master.

[0082] It should be noted that the computing node CN2 will send the second task instruction to the main data node that completes the first task instruction first, thereby shortening the resource time occupied by the table structure modification operation and saving computing resources. And the DN1 master can copy the data from the original table to the temporary table through the concurrent number variable multi-threading. Each data node in the storage node shard monitors the host load during the execution phase. If it exceeds the pre-set CPU usage threshold and / or IO usage threshold, the speed of copying data is slowed down. If the transmission delay between each data node exceeds the set delay threshold, the speed of copying data is slowed down.

[0083] S380. Based on the received third task instruction, the master data node deletes the original table, modifies the table name of the current temporary table to the table name corresponding to the original table, obtains the target table of the master data node, generates a target table replacement log, and sends the target table replacement log to the standby data node.

[0084] The target table replacement log is the third application difference log. The third application difference log is GTID_4 minus GTID_3. The third application difference log can be used to record the execution operation of the table replacement. GTID_4 can refer to the execution position when the table replacement is completed.

[0085] Specifically, in Figure 4 Based on the above, computing node CN2 initiates the DDL table structure_COMMIT (commit phase) phase instruction to DN1 master. After receiving the instruction, DN1 master deletes the original table, replaces the new table (current temporary table) with the original table (rename), and records the GTID position of DN1 master as GTID_4, thereby subtracting GTID_3 from GTID_4 to obtain the target table replacement log, and sends the target table replacement log to DN1 backup 1, DN1 backup 2, and DN1 backup 3.

[0086] S390. The standby data node replaces the original table based on the target table replacement log to obtain the target table of the standby data node, and sends the execution result of the third task instruction to the primary data node.

[0087] Specifically, in Figure 4 Based on the data from the master, DN1 backup 1, DN1 backup 2 and DN1 backup 3 receive the target table replacement log sent by the master, replicate the operation, and perform table replacement. DN1 backup 1, DN1 backup 2 and DN1 backup 3 feed back the successful execution result to the master.

[0088] S3100. Generate a modification completion signal based on the target table through the master data node and send it to the target computing node.

[0089] In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

[0090] The technical solution of the embodiment of the present invention is that the master data node creates a table structure based on the original table based on the received first task instruction to obtain a temporary table structure of the master data node, and generates a temporary table structure creation log, so that multiple master data nodes execute the first task instruction in parallel, further improving the efficiency of table structure modification, and sending the temporary table structure creation log to the standby data node; the standby data node creates a table structure based on the temporary table structure creation log to obtain the temporary table structure of the standby data node, and sends the execution result of the first task instruction to the master data node; the master data node sends the received execution result to the target computing node, so that the target computing node sends the second task instruction to the master data node, so that the target computing node can send the second task instruction to the master data node that completes the first task instruction first, thereby shortening the resource time occupied by the table structure modification operation and saving computing resources. Based on the received second task instruction, the master data node copies the current data in the original table to the temporary table structure to obtain the current temporary table of the master data node, and generates the current data replication log. In this stage, data manipulation operations are not blocked, so that data manipulation operations and table structure modification operations are executed in parallel, which improves the efficiency of operation execution, and sends the current data replication log to the standby data node; the standby data node performs current data replication based on the current data replication log to obtain the current temporary table of the standby data node, and sends the execution result of the second task instruction to the master data node; the master data node sends the received execution result to the target computing node, so that the target computing node sends the third task instruction to the master data node. Based on the received third task instruction, the master data node deletes the original table, and modifies the table name of the current temporary table to the table name corresponding to the original table, obtains the target table of the master data node, generates the target table replacement log, and sends the target table replacement log to the standby data node; the standby data node replaces the original table based on the target table replacement log to obtain the target table of the standby data node, and sends the execution result of the third task instruction to the master data node. Compared with the traditional table structure modification method, for example, the overall table structure modification statement in the distributed database is issued by the computing node, so that each master node in each shard group of the storage node executes the modification statement first (assuming the execution time is T1). However, due to the atomicity of the DDL table structure, the backup database can only start executing after the master database completes execution (assuming the execution time of each backup database is T2). The total execution time of the table structure modification statement is T = T1 + max (T2). max (T2) is the longest time among all backup databases. Therefore, the total execution time is longer due to the existence of the backup database.The technical solution of the embodiment of the present invention can split the table structure modification statement into three-stage modification statements, and when the standby database executes the first-stage modification statement, the main database executes the second-stage execution statement. When the standby database executes the second-stage modification statement, the main database executes the third-stage execution statement, thereby achieving staggered execution, further improving the table structure modification efficiency, and taking into account both the table structure modification operation and the data manipulation operation.

[0091] The following is an embodiment of a distributed database table structure modification system provided in an embodiment of the present invention. This system and the distributed database table structure modification method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the distributed database table structure modification system, reference can be made to the embodiment of the above-mentioned distributed database table structure modification method.

[0092] Embodiment 4

[0093] Figure 5 This is a schematic diagram of the structure of a distributed database table structure modification system provided by the fourth embodiment of the present invention. Figure 5 As shown, the system includes: a target computing node 510 and a target storage node 520;

[0094] Among them, the target computing node 510 is used to receive the table structure modification instructions of the target user, and split the table structure modification instructions to determine multiple task instructions, and send the multiple task instructions to the target storage node 520 based on the instruction priority; the target storage node 520 is used to modify the table structure in the target storage node 520 based on the received multiple task instructions to obtain the target table, and generate a modification completion signal based on the target table, and send it to the target computing node 510; wherein, in the process of the target storage node 520 executing the task instructions, the target storage node 520 does not block data manipulation operations.

[0095] The technical solution of the embodiment of the present invention is to receive the table structure modification instruction of the target user through the target computing node 510, and split the table structure modification instruction to determine multiple task instructions, and send the multiple task instructions to the target storage node 520 based on the instruction priority; the target storage node 520 modifies the table structure in the target storage node 520 based on the received multiple task instructions to obtain the target table, and generates a modification completion signal based on the target table and sends it to the target computing node 510; wherein, in the process of the target storage node 520 executing the task instruction, the target storage node 520 does not block the data manipulation operation, thereby realizing the parallel execution of data manipulation operations and table structure modification operations, thereby improving the operation execution efficiency.

[0096] On the basis of the above technical solution, the system further includes: a client;

[0097] The client is used to receive the table structure modification operation of the target user on the target storage node 520, convert the table structure modification operation into a table structure modification instruction, and send the table structure modification instruction to the target computing node 510; the target computing node 510 is also used to generate a table structure modification success flag based on the received modification completion signal, and send the table structure modification success flag to the client for display to the target user.

[0098] Based on the above technical solution, the table structure modification instruction is generated by the target user for the target storage node 520 corresponding to the target computing node 510 in the distributed database; the distributed database includes multiple computing nodes; the target storage node 520 includes at least one storage node shard; the storage node shard includes a master data node.

[0099] On the basis of the above technical solution, the master data node is also used to create a temporary table structure based on the original table based on the received first task instruction, and send the execution result of the first task instruction to the target computing node 510, so that the target computing node 510 sends the second task instruction to the master data node; the master data node is also used to copy the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table, and send the execution result of the second task instruction to the target computing node 510, so that the target computing node 510 sends the third task instruction to the master data node; the master data node is also used to delete the original table based on the received third task instruction, and modify the table name of the current temporary table to the table name corresponding to the original table to obtain the target table.

[0100] Based on the above technical solution, the storage node shard also includes at least one backup data node.

[0101] On the basis of the above technical solution, the master data node is also used to create a table structure based on the original table based on the received first task instruction to obtain a temporary table structure of the master data node, generate a temporary table structure creation log, and send the temporary table structure creation log to the standby data node; the standby data node is also used to create a table structure based on the temporary table structure creation log to obtain a temporary table structure of the standby data node, and send the execution result of the first task instruction to the master data node; the master data node is also used to send the received execution result to the target computing node 510, so that the target computing node 510 sends the second task instruction to the master data node.

[0102] On the basis of the above technical solution, the master data node is also used to copy the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table of the master data node, and generate a current data replication log, and send the current data replication log to the standby data node; the standby data node is also used to replicate the current data based on the current data replication log to obtain the current temporary table of the standby data node, and send the execution result of the second task instruction to the master data node; the master data node is also used to send the received execution result to the target computing node 510, so that the target computing node 510 sends the third task instruction to the master data node.

[0103] On the basis of the above technical solution, the master data node is also used to delete the original table based on the received third task instruction, and modify the table name of the current temporary table to the table name corresponding to the original table, obtain the target table of the master data node, and generate a target table replacement log, and send the target table replacement log to the standby data node; the standby data node is also used to replace the original table based on the target table replacement log to obtain the target table of the standby data node, and send the execution result of the third task instruction to the master data node.

[0104] On the basis of the above technical solution, the system further comprises: a load monitoring module;

[0105] The load monitoring module is used to obtain the current resource occupancy rate of the server where the target storage node 520 is located when the target storage node 520 executes multiple task instructions, and control the execution speed of the multiple task instructions based on the current resource occupancy rate and a preset occupancy threshold.

[0106] The distributed database table structure modification system provided in the embodiment of the present invention can execute the distributed database table structure modification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the distributed database table structure modification method.

[0107] It is worth noting that in the above-mentioned embodiment of the table structure modification of the distributed database, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for modifying the table structure of a distributed database, characterized in that: include: Receive a table structure modification instruction of a target user through a target computing node, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to a target storage node based on instruction priorities; Modifying the table structure in the target storage node based on the received multiple task instructions by the target storage node to obtain a target table, and generating a modification completion signal based on the target table and sending it to the target computing node; In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

2. The method according to claim 1, characterized in that The method further comprises: Receiving, through a client, a table structure modification operation of a target user on a target storage node, converting the table structure modification operation into a table structure modification instruction, and sending the table structure modification instruction to a target computing node; The target computing node generates a table structure modification success flag based on the received modification completion signal, and sends the table structure modification success flag to the client for display to the target user.

3. The method according to claim 1, characterized in that The table structure modification instruction is generated by a target user for a target storage node corresponding to a target computing node in a distributed database; The distributed database includes multiple computing nodes; The target storage node includes at least one storage node shard; The storage node shard includes a master data node.

4. The method according to claim 3, characterized in that The target storage node modifies the table structure in the target storage node based on the received multiple task instructions to obtain a target table, including: The master data node creates a temporary table structure based on the original table based on the received first task instruction, and sends the execution result of the first task instruction to the target computing node, so that the target computing node sends the second task instruction to the master data node; The master data node copies the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table, and sends the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node; The master data node deletes the original table based on the received third task instruction, and modifies the table name of the current temporary table to the table name corresponding to the original table, so as to obtain the target table.

5. The method according to claim 4, characterized in that The storage node shard also includes at least one backup data node.

6. The method according to claim 5, characterized in that The master data node creates a temporary table structure based on the original table based on the received first task instruction, and sends the execution result of the first task instruction to the target computing node, so that the target computing node sends the second task instruction to the master data node, including: The primary data node creates a table structure according to the original table based on the received first task instruction to obtain a temporary table structure of the primary data node, generates a temporary table structure creation log, and sends the temporary table structure creation log to the standby data node; The standby data node creates a table structure based on the temporary table structure creation log to obtain a temporary table structure of the standby data node, and sends the execution result of the first task instruction to the primary data node; The received execution result is sent to the target computing node through the master data node, so that the target computing node sends the second task instruction to the master data node.

7. The method according to claim 6, characterized in that The method comprises: copying the current data in the original table to the temporary table structure based on the received second task instruction by the master data node to obtain the current temporary table, and sending the execution result of the second task instruction to the target computing node, so that the target computing node sends the third task instruction to the master data node, including: The primary data node copies the current data in the original table to the temporary table structure based on the received second task instruction to obtain the current temporary table of the primary data node, generates a current data replication log, and sends the current data replication log to the standby data node; The standby data node performs current data replication based on the current data replication log to obtain a current temporary table of the standby data node, and sends the execution result of the second task instruction to the primary data node; The received execution result is sent to the target computing node through the master data node, so that the target computing node sends the third task instruction to the master data node.

8. The method according to claim 6, characterized in that The master data node deletes the original table based on the received third task instruction, and modifies the table name of the current temporary table to the table name corresponding to the original table, to obtain the target table, including: The primary data node deletes the original table based on the received third task instruction, and modifies the table name of the current temporary table to the table name corresponding to the original table, thereby obtaining the target table of the primary data node, generating a target table replacement log, and sending the target table replacement log to the standby data node; The standby data node replaces the original table based on the target table replacement log to obtain the target table of the standby data node, and sends the execution result of the third task instruction to the master data node.

9. The method according to claim 1, characterized in that: The method further comprises: When the target storage node executes the multiple task instructions, the load monitoring module obtains the current resource occupancy rate of the server where the target storage node is located, and controls the execution speed of the multiple task instructions based on the current resource occupancy rate and a preset occupancy threshold.

10. A distributed database table structure modification system, characterized in that: include: target computing nodes and target storage nodes; A target computing node is used to receive a table structure modification instruction from a target user, split the table structure modification instruction into multiple task instructions, and send the multiple task instructions to a target storage node based on instruction priorities; The target storage node is used to modify the table structure in the target storage node based on the received multiple task instructions to obtain a target table, and generate a modification completion signal based on the target table and send it to the target computing node; In the process of executing the task instruction on the target storage node, the target storage node does not block the data manipulation operation.

Images