A data table structure change operation processing method, device, medium and equipment

Abstract

The specification discloses a processing method, device, medium and equipment of a data table structure change operation. In the method, a master node of a distributed database sends a table structure change command to a first node, so that the first node obtains a data table to be changed according to the received table structure change command, and performs a data table structure change operation to obtain a target data table after the table structure of the data table to be changed is changed. A checkpoint file generated by the first node in the process of performing the data table structure change operation is saved to a specified storage node, so that when the master node monitors that the first node is abnormal, the master node can reselect a second node from the distributed database to replace the first node, and continue to perform the data table structure change operation based on the checkpoint file through the second node.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and in particular to a method, apparatus, medium and device for processing data table structure change operations. Background Technology

[0002] In the database field, Data Definition Language (DDL) transactions, when executed, not only need to atomically update system metadata (such as table structure definitions), but also often involve scanning, transforming, and physically rewriting massive amounts of data in the tables to be changed.

[0003] For example, when executing a DDL transaction to add a new column with a NOT NULL constraint to an existing data table, the database system first needs to create a new data table with the new column added to the table structure, then perform a full table scan to migrate each row of data in the original data table to the new data table, and fill the fields corresponding to the new column in each row with default values ​​to obtain the target data table after the table structure change.

[0004] As is evident, executing DDL transactions in distributed databases is time-consuming and consumes significant computing resources. Therefore, executing DDL transactions on the master node continuously consumes its computing, I / O, and network resources, potentially leading to prolonged lock contention or blocking, thus significantly reducing the responsiveness of online distributed database services. To reduce operating costs, many enterprises have begun using spot instances provided by cloud service providers to execute compute-intensive or I / O-intensive background tasks. However, spot instances have a significant drawback: they can be reclaimed by the cloud platform at any time, causing ongoing tasks to be interrupted.

[0005] Therefore, how to ensure that such transactions can be quickly resumed after the auction instance is reclaimed or crashes unexpectedly is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a method for processing data table structure change operations is provided, the method being applied to a distributed database, the distributed database including a master node and a first node, the method comprising: The master node sends a table structure change command to the first node; The first node obtains the data table to be changed according to the table structure change command, and performs the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node during the execution of the data table structure change operation is saved to the designated storage node; When the master node detects an anomaly in the first node, it selects a second node from the distributed database to replace the first node, and then continues to perform the data table structure change operation through the second node based on the checkpoint file.

[0007] According to a second aspect of one or more embodiments of this specification, a processing apparatus for data table structure change operations is provided, the apparatus being applied to a distributed database, the distributed database including a master node and a first node, the apparatus comprising: The sending module is used to send a table structure change command to the first node through the master node; The execution module is used to obtain the data table to be changed through the first node according to the table structure change command, and to perform the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node during the execution of the data table structure change operation is saved to the designated storage node; The recovery module is used to select a second node from the distributed database to replace the first node when the master node detects an anomaly in the first node, and to continue the data table structure change operation through the second node based on the checkpoint file.

[0008] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the above-described data table structure change operation processing method by executing the executable instructions.

[0009] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the processing method for the data table structure change operation described above.

[0010] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the processing method for the data table structure change operation described above.

[0011] As can be seen from the above embodiments, firstly, the master node of the distributed database sends a table structure change command to the first node, so that the first node can obtain the data table to be changed according to the received table structure change command, and perform the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed. Among them, the checkpoint file generated by the first node in the process of performing the data table structure change operation is saved to the designated storage node, so that when the master node detects an anomaly in the first node, it can select a second node from the distributed database to replace the first node, and continue to perform the data table structure change operation through the second node based on the checkpoint file.

[0012] As can be seen from the above, when the master node needs to execute a DDL transaction, it can send a table structure change command to the first node to transfer the data table structure change operation corresponding to the DDL transaction from the master node to the first node. When the first node executes the data table structure change operation, it can generate a checkpoint file and save it to a designated storage node. This allows the master node to select a second node to replace the first node after detecting an anomaly in the first node, and restore the progress of the data table structure change operation performed by the first node before the anomaly occurred based on the checkpoint file saved by the first node. On this basis, the master node can continue to execute the data table structure change operation, thereby ensuring the smooth execution of the DDL transaction. Attached Figure Description

[0013] Figure 1 This is a schematic diagram illustrating the steps involved in a data table structure change operation processing method provided in an exemplary embodiment; Figure 2A This is a schematic diagram of a distributed database with an in-memory computing architecture provided in an exemplary embodiment; Figure 2B This is a schematic diagram of a distributed database with a storage-compute separation architecture provided in an exemplary embodiment; Figure 3A This is a schematic diagram of a transaction-level log structure provided in an exemplary embodiment; Figure 3B This is a schematic diagram of a table-level log structure provided in an exemplary embodiment; Figure 3C This is a schematic diagram of a table-scoped log structure provided in an exemplary embodiment; Figure 4A This is a schematic diagram of a first type of data table structure change operation recovery process provided in an exemplary embodiment; Figure 4B This is a schematic diagram of a second data table structure change operation recovery process provided in an exemplary embodiment; Figure 5This is a schematic diagram of a third data table structure change operation recovery process provided in an exemplary embodiment; Figure 6 This is a schematic diagram illustrating different types of DDL transactions provided in an exemplary embodiment; Figure 7 This is a schematic structural diagram of a device provided in an exemplary embodiment; Figure 8 This is a block diagram of a data table structure change processing apparatus provided in an exemplary embodiment. Detailed Implementation

[0014] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0015] In the database field, transactions can be categorized into Data Manipulation Language (DML) transactions and Data Definition Language (DDL) transactions based on the different objects and purposes of the operations. DML transactions, also known as Data Manipulation Language transactions, are used to perform CRUD operations on records in a data table.

[0016] For example, DML transactions can involve inserting new records into a data table using the INSERT statement, updating existing records in a data table using the UPDATE statement, deleting records from a data table using the DELETE statement, or querying records in a data table using the SELECT statement.

[0017] Data Definition Language (DDL) transactions are used to create, modify, or delete the structure of a data table or its index in a database.

[0018] For example, DDL transactions can involve using the CREATE TABLE statement to create a new data table, using the ALTER TABLE statement to modify column definitions (such as adding or deleting columns) or constraints in an existing data table, using the DROP TABLE statement to delete a data table, or using the CREATE INDEX statement to create an index for a data table.

[0019] It should be noted that in a database, a record refers to a complete row of data in a data table, also known as a row or tuple.

[0020] Building upon this, in distributed databases, multiple replicas typically exist for the same data table, distributed across different service nodes. To ensure global consistency of metadata, all DDL transactions in a distributed database are usually executed uniformly by the master node, and then the data changes resulting from each DDL transaction are synchronized to other service nodes holding replicas of the same data table. However, because DDL transactions often involve scanning, transforming, and rewriting massive amounts of data during execution, the master node experiences excessive computational load and prolonged resource consumption, ultimately impacting and reducing the responsiveness of the distributed database's online services.

[0021] To address the aforementioned issues, this specification provides a method for handling data table structure change operations. This method can offload DDL transactions that modify the table structure of the data table to be changed stored in the distributed database from the master node to a standby node or bidding node. This avoids the long-term occupation of the master node's computing resources by such DDL transactions, thereby improving the overall response performance and fault tolerance of the distributed database.

[0022] The standby node mentioned above can refer to the service node that is persistently deployed in the database. The standby node is usually used for disaster recovery against the primary node. That is, once the primary node fails, such as crashing, hardware failure, or process crash, causing the primary node to be unable to continue providing services, the standby node can automatically take over the responsibilities of the primary node based on the copies of the data tables it holds, and become the new primary node, thereby ensuring the continuous availability and data persistence of the database service.

[0023] The aforementioned bidding nodes are actually service nodes temporarily acquired from the cloud computing platform. Specifically, the cloud computing platform can auction off its idle or soon-to-be-obsolete computing resources at a significant discount. Typically, users who need these nodes can bid on them, and the platform will determine the allocation of the nodes based on each user's bid.

[0024] It's important to note that once a user wins the right to use a bidding node, they don't occupy it indefinitely. Typically, the cloud computing platform sets a usage period for bidding nodes. Once this period expires, the platform will reclaim the usage rights, effectively reclaiming the resource. If the user still needs to use the bidding node, they must re-bid to purchase the usage rights.

[0025] In addition, since bidding nodes may be computing resources that are about to be phased out in the cloud computing platform, they are also prone to anomalies such as crashes, being forcibly interrupted and reclaimed by the cloud platform, loss of local storage data, and network connection failures.

[0026] For the bidding node itself, it actually refers to a virtual machine or container running in a distributed database. It is a complete and isolated virtual computing environment, including virtualized CPU, memory, hard disk, and operating system. Therefore, the bidding node is a service-oriented encapsulation of hardware resources, which allows for the flexible provision of various specific services to users.

[0027] The following is a method for handling data table structure changes provided in this manual, which can be roughly divided into several processes, such as... Figure 1 As shown.

[0028] Figure 1 This is a schematic diagram illustrating the steps involved in a data table structure change operation processing method provided in an exemplary embodiment.

[0029] S100: The master node sends a table structure change command to the first node.

[0030] In practical applications, users may need to adjust the table structure of the data table to be changed stored in the distributed database based on their actual needs. In this case, the user can send a task request to the distributed database. After receiving the task request, the distributed database can parse the data definition statements (i.e., DDL statements used to modify the table structure of the data table to be changed, which can be written in SQL syntax) contained in the task request, so as to determine the master node to execute the task request from each service node, and the master node creates the corresponding DDL transaction based on the received task request.

[0031] Specifically, for DDL transactions used to modify the table structure of a data table stored in a distributed database, the data table structure modification operations included in the DDL transaction may include, for example, adding a new column to the data table, modifying the data type of the data table, renaming a column, or deleting a column from the data table.

[0032] The aforementioned master node can be pre-configured in the distributed database, or it can be selected by the distributed database from each service node holding a copy of the data table to be changed based on a consistency protocol.

[0033] In this specification, the entity executing the data table structure change operation can be the master node of the distributed database. The master node of the distributed database can be responsible for creating DDL transactions, determining the first node for executing the data table structure change operation contained in the DDL transaction, and sending the table structure change command to the first node. The specific execution of the data table structure change operation can be completed by the corresponding first node in the distributed database.

[0034] S102: The first node obtains the data table to be changed according to the table structure change command, and performs the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node in the process of performing the data table structure change operation is saved to the designated storage node.

[0035] As mentioned above, a distributed database can contain multiple bidding nodes provided by a cloud computing platform. Therefore, after the master node creates a DDL transaction, it can allocate at least one first node for the DDL transaction from the preset standby nodes and / or bidding nodes corresponding to the master node in the distributed database, and assign the data table structure change operations contained in the DDL transaction to the selected first node for execution. The difference between using one first node and using multiple first nodes to execute data table structure change operations will be explained in detail later.

[0036] Furthermore, after the master node determines the first node to perform the data table structure change operation, it can send a table structure change command to the first node. Upon receiving the table structure change command, the first node can obtain the data table to be changed and perform the data table structure change operation to obtain the target data table after the table structure of the data table to be changed has been changed.

[0037] As can be seen from the above, if the first node is a standby node, then the first node itself holds a copy of the data table to be changed. Therefore, it can directly perform data table structure change operations based on the copy of the data table to be changed that it holds.

[0038] Furthermore, if the first node is a bidding node, then the bidding node does not have the data table to be changed required for the data table structure change operation. Therefore, the first node needs to obtain the data table to be changed from the master node or the shared storage node contained in the distributed database.

[0039] Specifically, based on different distributed database architectures, the way the first node obtains the data table to be changed can be divided into the following two cases, as follows: Figure 2A , Figure 2B As shown.

[0040] Figure 2A This is a schematic diagram of a distributed database with an integrated storage and computing architecture provided in an exemplary embodiment.

[0041] Combination Figure 2A As can be seen, in a distributed database employing an integrated storage and computing architecture, the data table to be modified is often stored in the local storage space of the service node. In other words, under this architecture, the data table to be modified is often stored in the storage space used by the master node itself. In this case, the master node can directly transmit its stored data table to the first node, allowing the first node to perform the aforementioned data table structure modification operation based on the received data table.

[0042] Figure 2B This is a schematic diagram of a distributed database with a storage-compute separation architecture provided in an exemplary embodiment.

[0043] Combination Figure 2B As can be seen, in a distributed database that adopts a storage-compute separation architecture, the metadata and table data of the data table to be changed can be stored separately. That is, the metadata of the data table to be changed is often stored in the local storage space used by the master node, while the table data of the data table to be changed can be stored in the shared storage node.

[0044] In this system, the shared storage nodes are independent of each service node and are shared by all service nodes.

[0045] For any given data table, its metadata can refer to data describing the table's structure, attributes, and storage location. Examples include: table structure definitions (table name, column names, data types for each column), partitioning information (partition key, partitioning method, partition distribution), and version numbers (e.g., the new version number generated after each DDL transaction). The table data can refer to the individual records contained within the table.

[0046] In this scenario, the master node can directly transfer the metadata of the data table to be modified stored in its local storage space to the first node, and can also transfer the table data of the data table to be modified stored in the shared storage node to the first node through the shared storage node.

[0047] In the above content, there are multiple transmission methods, such as the master node directly transmitting the data table to be changed to the first node, or transmitting the data table to be changed to the first node through a shared storage node.

[0048] For example, if a high-speed communication network (such as RDMA) is established between the master node and the bidding node in a distributed database, the master node can send the data table to be changed directly to the bidding node through RDMA.

[0049] In addition, the master node can also transmit the data table to be modified to the bidding nodes via log synchronization. Specifically, the master node can first determine the checkpoint file associated with the data table to be modified in the distributed database as the target checkpoint file, and determine the log associated with the data table to be modified as the target log. The target log refers to the log that records data changes in the data table to be modified after the checkpoint file. Then, the master node can send the target log and the target checkpoint file to the bidding nodes, which can then replay the process data based on the target log and the target checkpoint file to replay the data table to be modified.

[0050] In the above context, the checkpoint file associated with the data table to be changed can refer to a consistent snapshot generated by the distributed database for the data table to be changed at a certain point in time (which can be understood as the table state of the data table to be changed at the current moment).

[0051] The log associated with the data table to be changed can be a log generated after the target checkpoint file that records all data change operations on the data table to be changed. Based on the target log, it can be determined which data change operations have modified the data in the data table to be changed after the target checkpoint file was created.

[0052] For example, if there is a data table A to be modified, after executing transaction 1 targeting data table A, a checkpoint file can be created. Transaction 1 modifies the account balance of user 1001 in the data table from 200 to 300. The generated checkpoint file represents a consistent snapshot of the data table A after the account balance of user 1001 has been modified to 300. If, subsequently, transaction 2 targeting data table A is executed, modifying the account balance of user 1001 from 300 to 500, the data modification operation performed by transaction 2 (i.e., modifying the account balance of user 1001 from 300 to 500) can be recorded in the target log mentioned above.

[0053] The aforementioned replay can refer to the bidding node first loading the target checkpoint file to restore the data table to be changed at the time corresponding to the target checkpoint file locally. Then, according to the order of each target log, it parses and executes the change operations recorded in the target log one by one, thereby re-executing these change operations on the locally reconstructed data table to obtain the latest data table to be changed.

[0054] As can be seen from the above, the replay process actually involves first obtaining the complete state of the data table to be changed at a certain moment through the target checkpoint file, and then using the target log to roll the state of the data table to be changed to the latest state.

[0055] Of course, in real-world applications, there may be some data tables that do not have associated checkpoint files. In this case, the master node or shared storage node can also transmit the target log of the data table to be changed to the bidding node, so that the bidding node can replay the data based on the received target log to obtain the data table to be changed.

[0056] Furthermore, after the first node has the data table to be modified, a new data table can be created according to the table structure modification command. The table structure of this new data table is obtained by applying the table structure modification operation corresponding to the table structure modification command to the table structure of the data table to be modified. This can be understood as first creating an empty table, and the table structure of this empty table is modified according to the user's requirements for the table structure of the data table to be modified.

[0057] Based on this, the first node can begin a scan operation (i.e., Scan) on the data table to be changed. This scan operation is mainly used to perform a full table scan of the data table to be changed, that is, to read all data rows in the data table to be changed, and to migrate the scanned data in the data table to be changed to a new data table to obtain the target data table. This target data table is the data table after the data table structure change operation is applied to the data table to be changed, or in other words, the data table after the table structure of the data table to be changed is modified by performing the data table structure change operation.

[0058] Furthermore, in practical applications, scanning and CDC operations are often executed in parallel. In other words, during the scanning process, the master node will still respond to user-sent business requests, performing operations such as inserting, updating, and deleting data in the data table to be changed. If these newly generated changes are not synchronized to the aforementioned new data table, the scan results will be inconsistent with the actual situation (e.g., the latest written data or newly updated data in the data table to be changed will be lost).

[0059] Therefore, in this specification, a CDC thread can be started in the first node. This thread can simultaneously obtain the change log of the data table to be changed from the preset change data capture log in real time while performing the scan operation. This change log is used as the synchronization log. By replaying the synchronization log, the data migrated to the new data table during the scan operation is updated. This applies the changes that occurred in the data table to be changed during the data table structure change operation to the new data table to obtain the target data table, thereby ensuring the consistency between the target data table and the actual data table.

[0060] The change log mentioned above is used to characterize the changes that occur in the data table to be changed during the execution of the data table structure change operation.

[0061] Furthermore, in practical applications, if the master node initially allocates only one primary node for table structure change operations, then all operations, including executing table structure changes and data replay, will be performed by a single node. However, in real-world applications, the amount of data in the table to be changed may be substantial. Using only one node to perform these operations will inevitably impact the execution efficiency of DDL transactions.

[0062] Therefore, in this specification, if the data table to be modified contains a large amount of data, the master node can allocate multiple first nodes for the data table structure change operation when initially allocating the first nodes. In this case, the master node can divide different data processing scopes for each first node, so that multiple first nodes are each responsible for the data table structure change operation within their respective scopes, thereby achieving efficient parallel execution.

[0063] Specifically, if there are multiple first nodes, the master node can further determine the data range that each first node is responsible for in the data table to be changed, which serves as the target data range for that first node. Then, through that first node, the data table to be changed within the target data range can be scanned to migrate the data from that table to a new data table created within that first node, thus obtaining the target data table within the target data range.

[0064] For example, suppose the `orders` table to be modified is partitioned by order ID (`order_id`), resulting in four data ranges: [0, 999999], [1000000, 1999999], and [2000000, 2999999]. If a user submits a DDL statement: `ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending'`, this means adding a string column `status` with a default value to the `orders` table without interrupting existing business operations. At this point, after the master node creates the DDL transaction, it can allocate three first nodes (Node A, Node B, and Node C) to process the data table structure change operations contained in the DDL transaction in parallel, based on the task load and resource conditions. Thus, [0, 999999] can be used as the target data range corresponding to the first node Node A, [1000000, 1999999] as the target data range corresponding to the first node Node B, and [2000000, 2999999] as the target data range corresponding to the first node Node C.

[0065] Once all the target data tables within each target data range have been modified and the CDC operation synchronization is complete, the master node can obtain the target data table after applying the data table structure change operation to the target data table within each target data range.

[0066] However, it should be noted that the current log structure is organized at the transaction level. That is, each transaction's change operations are centrally recorded in the thread-local log corresponding to that transaction, contained within the change data capture log. This log is organized by transaction, without distinguishing between the tables involved or specific data ranges, and each operation record is only marked with the transaction ID. Therefore, when a first node only needs to capture and process changes within a target data range (e.g., the primary key range [1000000, 1999999]), it cannot directly locate and extract the change log corresponding to that target data range from the change data capture log. Instead, it must scan all change logs contained in the change data capture log and parse the primary key or partition key in each change log's operation record to filter out the change logs belonging to its own target data range (i.e., logs representing changes to data within that target data range in the table to be changed), as the logs to be synchronized.

[0067] The aforementioned full-scan and filtering method not only significantly increases I / O and computational overhead but also reduces the execution efficiency of CDC operations. Based on this, the change log structure can be modified from transaction-level to table-level in this specification. Furthermore, based on the table-level log structure, it can be further divided into sub-log partitions according to predetermined target data ranges, i.e., a table-range-level log structure. For ease of understanding, the following will combine... Figure 3A , Figure 3B , Figure 3C Please provide a detailed explanation.

[0068] Figure 3A This is a schematic diagram of a transaction-level log structure provided in an exemplary embodiment.

[0069] Combination Figure 3A As can be seen, when the master node executes any DML transaction, if a transaction-level log structure is used, all data modification operations on multiple tables (such as T1, T2, and T3) within that transaction are recorded in the same thread-local log (TLS Log), identified only by the transaction ID. Because data modification operations from different tables are stored together in the log without being divided by table or data range, it becomes impossible to efficiently retrieve the logs to be synchronized by table or data range; the entire log must be scanned and parsed record by record.

[0070] Figure 3B This is a schematic diagram of a table-level log structure provided in an exemplary embodiment.

[0071] Combination Figure 3B As can be seen, if a table-level log structure is adopted, the master node can set up a corresponding change data capture log partition for each table (such as T1, T2, T3) to persistently store its log stream. When a DML transaction is committed, it is distributed to the corresponding change data capture log partition based on the table to which each data change operation belongs. This table-level isolation allows the first node to directly filter out the logs to be synchronized from the corresponding change data capture log partition, thereby reducing the transmission and processing of irrelevant data and significantly improving CDC efficiency.

[0072] Figure 3C This is a schematic diagram of a table-range log structure provided in an exemplary embodiment.

[0073] Combination Figure 3CAs can be seen, if a table-range log structure is adopted, the master node can divide the change data capture log partition corresponding to any table into sub-log partitions based on each predefined target data range (such as T1[0-4], T1[5-9]). When a DML transaction is committed, the master node can determine the target data range to which the table record belongs based on the primary key value or partition key value of the table record involved in each data change operation. Then, it can distribute the corresponding change log to the sub-log partition of the corresponding target data range, so that the first node can obtain the log to be synchronized from the sub-log partitions contained in the change data capture log partition of the table to be changed based on its corresponding target data range.

[0074] In addition, as can be seen from the above, for any first node used to perform the data table structure change operation, the first node can be a standby node or a bidding node. However, because bidding nodes have poor stability, for example, the cloud computing platform providing the bidding node may revoke its usage rights after the bidding node's usage period expires. Furthermore, bidding nodes themselves are soon-to-be-obsolete computing resources, and therefore are prone to machine crashes and downtime.

[0075] Therefore, in order to reduce the impact of first node anomalies on the processing of data table structure change operations, for each first node, a checkpoint file is generated during the execution of data table structure change operations and saved to a designated storage node for use in recovery after a first node anomaly occurs.

[0076] The designated storage node mentioned above can refer to a preset shared storage node, master node, etc.

[0077] The timing of the first node generating the checkpoint file for data table structure change operations can be set according to actual needs, and this manual does not impose any restrictions on it.

[0078] Based on this, the master node can also monitor at least one first node used to perform the data table structure change operation during the execution of the data table structure change operation. If any first node is detected as unavailable, a second node is selected from the distributed database to replace the first node. The second node then continues to execute the data table structure change operation based on the checkpoint file saved in the specified storage node.

[0079] The master node can monitor whether the previously assigned first node is experiencing any anomalies in several ways. For example, it can send a heartbeat signal to each first node; if it does not receive a response signal based on the heartbeat signal from the first node within a set time, an anomaly is detected. This method is primarily used to handle sudden anomalies in the first node, such as crashes or network connection failures.

[0080] For example, a first node with a recycling plan will send a recycling notification message to the master node to inform it that it will be recycled by the cloud computing platform within a specified time. If the master node receives a recycling notification message from any first node, it can detect that the first node is an unavailable node.

[0081] As can be seen from the above, the monitoring method actually addresses two different situations: one can be called unplanned recycling (i.e., sudden anomalies occur), and the other can be called planned recycling (i.e., bidding instances are recycled because the recycling time has arrived).

[0082] To address these two different scenarios, the second service instance will also employ two different methods when performing data recovery.

[0083] Specifically, if the first node is found to be abnormal based on the recycling notification message, since the recycling time has been predetermined for the first node, it can generate a new checkpoint file and store it in the designated storage node before the recycling time arrives. At this time, the second node can directly determine the checkpoint file closest to the recycling time from the checkpoint files associated with the data table to be changed, and use it as the first checkpoint file. Based on the first checkpoint file, it performs data replay locally to restore the table structure and intermediate data table consistent with the new data created by the first node up to the time the checkpoint file was created. Based on the intermediate data table, it continues to perform the data table structure change operation.

[0084] Furthermore, if the first node is determined to be unavailable because it fails to receive a response signal based on the heartbeat signal within a set time, and since the collapse of the first node is difficult to predict in advance, the checkpoint file it generates is not the latest checkpoint file. Therefore, the time when the first node is determined to have an anomaly needs to be taken as the collapse time. At this time, the second node can determine the checkpoint file closest to the collapse time from the checkpoint files associated with the data table to be changed, and take it as the second checkpoint file. It can also determine the log with the timestamp between the time of generating the second checkpoint file and the collapse time from the preset change data capture log, and take it as the recovery log. Then, based on the recovery log and the second checkpoint file, it can restore the intermediate data table with the same table structure and data as the new data created by the first node up to the collapse time, and continue to perform the data table structure change operation based on the intermediate data table.

[0085] To facilitate understanding, the following examples, using the specified storage node as the shared storage node and the above-mentioned anomaly as the planned reclamation, will detail the recovery process when a data table structure change operation is executed by a single first node, and the recovery process when a data table structure change operation is executed by multiple first nodes. Specifically... Figure 4A , Figure 4B As shown.

[0086] Figure 4A This is a schematic diagram of a first type of data table structure change operation recovery process provided in an exemplary embodiment.

[0087] Combination Figure 4A As can be seen, when a data table structure change operation is executed by a single first node, this first node can generate a checkpoint file during the operation and store it in the shared storage node. If the master node detects an anomaly in this first node, it can select a second node from the distributed database. This second node then retrieves the checkpoint file corresponding to the data table structure change operation from the checkpoint files stored in the shared storage node and continues executing the data table structure change operation based on the retrieved checkpoint file.

[0088] Figure 4B This is a schematic diagram of a second data table structure change operation recovery process provided in an exemplary embodiment.

[0089] Combination Figure 4BAs can be seen, when a data table structure change operation is executed by multiple first nodes, each first node can generate a checkpoint file corresponding to the new data table within the target data range it is responsible for during the execution of the data table structure change operation, and store the checkpoint file in the shared storage node. Based on this, if the master node detects an anomaly in the first node, it can reselect a second node from the distributed database. The second node then retrieves the checkpoint file corresponding to the target data range from the checkpoint files stored in the shared storage node, and continues to execute the data table structure change operation based on the retrieved checkpoint file.

[0090] Furthermore, in practical applications, due to different business needs, the frequency of concurrent write operations on different data tables often varies. When performing a data table structure change operation on a certain data table, if the frequency of concurrent write operations on that data table triggered by the received DML transaction is low, if the above-mentioned scan operation and CDC operation are still performed using the real-time synchronous merging method, it may cause additional overhead because there is less table data that is changed in that data table during the data table structure change operation.

[0091] Specifically, if the first node wants to perform CDC operations simultaneously while performing scanning operations, it needs to reserve fixed memory buffers, worker threads, and network connections for CDC operations. When there are few table data changes in the data table, the utilization rate of these resources will decrease.

[0092] Based on this, for data tables with low concurrent write operation frequency, when performing data table structure change operations on such data tables as the tables to be modified, the master node can simultaneously perform a scan operation on the data table to be modified through the first node, and concurrently determine the logs to be synchronized from the preset change data capture log to identify data change records. These data change records can then be stored in an intermediate index. After the scan is complete, the intermediate index is merged with the new data table to obtain the target data table. The aforementioned data change records include: the row identifier of any row in the data table to be modified that has undergone data change, and the corresponding changed data for that row, specifically as follows: Figure 5 As shown.

[0093] Figure 5 This is a schematic diagram of a third data table structure change operation recovery process provided in an exemplary embodiment.

[0094] Combination Figure 5As can be seen, during the process of the first node performing the data table structure change operation according to the above method, it can perform the scan operation and generate the checkpoint file corresponding to the new data table and store it in the specified storage node. It can also generate the corresponding index checkpoint file for the intermediate index in the above CDC operation process and store it in the specified storage node. Here, the index checkpoint file corresponding to the intermediate index is a persistent snapshot of the intermediate index stored by the first node during the process of performing the data table structure change operation.

[0095] Based on this, if the first node encounters an anomaly, the reassigned second node can obtain the checkpoint file corresponding to the new data table, as well as the index checkpoint file mentioned above, from the specified storage node to continue executing the data table structure change operation.

[0096] Furthermore, it should be understood that the aforementioned intermediate index is only a data structure used to temporarily store data changes generated during the execution of the aforementioned data table structure change operations, and is not a table index used to accelerate retrieval. It can also be replaced with other temporary data structures, such as log files. In this specification, the aforementioned data change records are stored in the form of an intermediate index because indexes are often built based on the primary key or partition key of the data table to be changed. The two are organized in the same way, which allows the corresponding row in the new data table to be quickly located based on the row identifier when merging the intermediate index with the new data table, so as to perform replacement, insertion, or deletion operations, thereby greatly improving the efficiency of merging the two.

[0097] As can be seen from the above, when performing a data table structure change operation, the generated checkpoint file is saved in a designated storage node. Therefore, if an anomaly is detected in the previously allocated first node, a second node can be reselected in the distributed database. The data is then replayed according to the checkpoint file recorded in the designated storage node, and the data table structure change operation can continue to be performed through the second node. This ensures that the data table structure change operation can continue even if it is interrupted due to anomalies or resource reclamation, thus guaranteeing the smooth execution of the data table structure change operation.

[0098] Furthermore, DDL transactions can actually be divided into two steps: data replication and constraint validation (logical verification). Data replication refers to the process of rewriting, migrating, or reconstructing existing data in a data table according to the new table structure definition.

[0099] For example, if executing a DDL transaction requires changing the data type of a column in a data table to another data type (such as changing INT to BIGINT), a new data table can be created first (that is, the data table after the data type of the above column has been changed to BIGINT), and then the data is read row by row from the data table to be changed and converted according to the new format before being written into the new data table. This process is called data copying.

[0100] During this process, the data table to be changed can respond to user requests and manage data normally. The data change operations that occur during this process can be stored in the preset change data capture log in the form of logs. In the process of migrating data from the data table to be changed into the new data table, the data changes recorded in the change data capture log can be synchronized to the new data table in parallel.

[0101] Constraint verification refers to the process of checking whether data meets the newly introduced logical constraints during structural changes. For example, if a DDL transaction requires adding a unique constraint to a column in the data table to be changed, the operation of adding a unique constraint requires ensuring that there are no duplicate values ​​in the column. In this case, it is necessary to verify whether the column contains duplicate values. This verification process is called constraint verification.

[0102] As can be seen from the above, in practical applications, not all DDL transactions require both data replication and constraint validation. Therefore, DDL transactions can be categorized into validation-only DDL transactions, replication-only DDL transactions, DDL transactions requiring both replication and validation, and DDL transactions requiring neither replication nor validation. Specifically... Figure 6 As shown.

[0103] Figure 6 This is a schematic diagram illustrating different types of DDL transactions provided in an exemplary embodiment.

[0104] Combination Figure 6 As can be seen, verifying only DDL transactions means that the physical storage format of the data table does not need to be modified (i.e., no data copying, structure adjustment and other format conversion operations are required), and only the constraint legality verification DDL operation of the existing data is required.

[0105] A copy-only DDL transaction refers to an operation that does not require additional logical constraint checks, but requires physical rewriting or format migration of existing data in the table according to the new structure.

[0106] DDL transactions that require replication and validation refer to operations that involve both physical data replication (such as column type changes, default value filling, encoding adjustments, etc.) and logical constraint validity checks on the transformed data. DDL transactions that require neither replication nor verification refer to operations that do not modify the physical data content or introduce new logical constraints, but only update the metadata.

[0107] Based on this, the Tesseract architecture can be adopted in this specification to separate the data itself (storage) from the logical view of the data (schema), thereby allowing the execution of DDL transactions to be completely unaffected by ongoing data read and write operations (i.e., DML transactions), so as to avoid the degradation of database concurrency performance and the impact on online business continuity.

[0108] Specifically, in this specification, the data table can be divided into storage blocks and schema blocks. The storage block stores at least one piece of data contained in the data table, and each piece of data is stored in the form of key-value pairs. For example: (key: 1, value: {id: 1, name: "Alice"}), (key: 2, value: {id: 2, name: "Bob"}).

[0109] It should be noted that the storage block simply stores and retrieves these key-value pairs; it does not need to determine which data table each key-value pair belongs to.

[0110] Within the schema block, for each data table, at least one version of schema data corresponding to that table is stored. Each schema data is a set of mapping rules used to map the key-value pairs in the storage block to the table structure displayed to the user. For example, the mapping rules mentioned above could be "read the key-value pair with key x from the storage layer", "map the id field in the key-value pair to the users.id column", and "map the name field in the key-value pair to the users.name column".

[0111] Based on this, when a DDL transaction needs to be executed, for example, to execute ALTER TABLE users ADDCOLUMN age When executing a DDL transaction containing an INT statement (i.e., adding an age column to the users table and setting the age column's data type to integer INT), since the data in the users table is separated from the logical view of the data, the database system does not need to modify any data itself. Instead, it quickly creates a new version of the schema data, such as: "reading the key-value pair with key x from the storage layer", "mapping the id field in the key-value pair to the users.id column", "mapping the name field in the key-value pair to the users.name column", and "if the key-value pair has an age field, mapping it to the users.age column; otherwise, returning NULL". After this, when executing a DML transaction, if the timestamp of the DML transaction indicates that it occurred before the aforementioned DDL transaction, the old version of the schema data can continue to be used. In this case, it can be considered that the age column does not exist in the users table. However, if the DML transaction indicates that it occurred after the aforementioned DDL transaction, the new version of the schema data can be used to manage the data in the users table that contains the age column. Since this process only involves changes to the mapping rules in the schema block, the execution time of this DDL transaction is extremely short.

[0112] It's important to note that while the above method can be used to execute DDL transactions directly on the master node that only require verification and do not involve replication or verification, it doesn't guarantee successful execution for DDL transactions that only require replication or verification. In other words, although the method allows DDL transactions to execute by changing mapping rules, the physical data of the tables being modified remains unchanged and will still require subsequent modifications.

[0113] Therefore, in practical applications, DDL transactions that only require verification and do not need replication or verification can be quickly executed on the master node by modifying only the mapping rules of the schema block. For DDL transactions that only require replication and verification, they can be quickly executed on the master node by modifying the mapping rules of the schema block to ensure that the stability of online services is not affected. At the same time, the data table structure change operations contained in the DDL transaction can be offloaded to the first node for execution using the above method. By combining the two execution methods, the response performance of the distributed database for online services during the execution of DDL transactions can be improved.

[0114] Figure 7This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 7 As shown, device 700 mainly consists of a communication interface 702, a user interface 704, a processor 706, and a data storage 708. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 710. The communication interface 702 enables device 700 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 702 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 702 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 702 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 702 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0115] User interface 704 includes receiving user input and providing output to the user. Therefore, user interface 704 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 704 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 704 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 700 may support remote access from other devices via communication interface 702 or another physical interface (not shown). User interface 704 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 704 may also be configured as a display device for rendering or displaying text fragments.

[0116] Processor 706 may contain one or more general-purpose processors and / or special-purpose processors.

[0117] Data storage 708 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 706. Data storage 708 may include removable and non-removable components.

[0118] Processor 706 is capable of executing program instructions 718 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 708 to perform the various functions described herein. Data storage 708 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 700, enable device 700 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 706 executing program instructions 718 may result in processor 706 using data 712.

[0119] For example, program instructions 718 may include an operating system 722 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 700 and one or more applications 720 (e.g., a browser, social application, or game application). Similarly, data 712 may include operating system data 716 and application data 714. Operating system data 716 is primarily accessible to the operating system 722, while application data 714 is primarily accessible to one or more applications 720. Application data 714 may reside in a file system visible or hidden from the user of device 700.

[0120] Application 720 can communicate with operating system 722 through one or more application programming interfaces (APIs). These APIs help application 720 read and / or write application data 714, transmit or receive information via communication interface 702, receive or display information on user interface 704, etc.

[0121] In some terminology, application 720 may be simply referred to as "app". Furthermore, application 720 can be downloaded to device 700 through one or more online app stores or app markets. However, applications can also be installed on device 700 in other ways, such as through a web browser or a physical interface on device 700 (e.g., a USB port).

[0122] Please refer to Figure 8 , Figure 8 This is an exemplary embodiment of a data table structure modification processing apparatus, which can be applied to, for example... Figure 7 The device shown, in order to implement the technical solution of this specification, includes: Sending module 801 is used to send a table structure change command to the first node through the master node; The execution module 802 is used to obtain the data table to be changed through the first node according to the table structure change command, and perform the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node during the execution of the data table structure change operation is saved to the designated storage node; The recovery module 803 is used to select a second node from the distributed database to replace the first node when the master node detects an anomaly in the first node, and to continue to execute the data table structure change operation through the second node based on the checkpoint file.

[0123] Optionally, the first node includes: a standby node and a bidding node, wherein the standby node is a service node persistently deployed in the database, and the bidding node is a service node temporarily obtained from the cloud computing platform.

[0124] Optionally, the execution module 802 is specifically used to: obtain a checkpoint file associated with the data table to be changed from the specified storage node as a target checkpoint file, and obtain a log associated with the data table to be changed as a target log; and perform data replay based on the target checkpoint file and the target log to restore the data table to be changed in the local storage space.

[0125] Optionally, the recovery module 803 is specifically used to send a heartbeat signal to the first node, and if no response signal is received from the first node based on the heartbeat signal within a set time, an abnormality is detected in the first node; or if a recycling notification message is received from the first node, an abnormality is detected in the first node, wherein the recycling notification message indicates that the first node will be recycled.

[0126] Optionally, the recovery module 803 is specifically used to, when it is determined that the first node has an anomaly according to the recycling notification message, determine the checkpoint file closest to the recycling time from each checkpoint file associated with the data table to be changed through the second node, and use it as the first checkpoint file, and continue to execute the data table structure change operation based on the first checkpoint file; the recycling time is included in the recycling notification message; the first checkpoint file is generated by the first node after sending the recycling notification message.

[0127] Optionally, the recovery module 803 is specifically configured to: when no response signal is received from the first node based on the heartbeat signal within a set time, determine the time when the first node experienced an anomaly as the crash time; through the second node, determine the checkpoint file closest to the crash time from each checkpoint file associated with the data table to be changed, as the second checkpoint file, and determine the log with the timestamp located between the time of generating the second checkpoint file and the crash time from the preset change data capture log, as the recovery log; and based on the recovery log and the second checkpoint file, continue to execute the data table structure change operation.

[0128] Optionally, the execution module 802 is specifically used to perform a data table structure change operation to create a new data table; wherein the table structure of the new data table is obtained by changing the table structure of the data table to be changed after performing the data table structure change operation; and to scan the data table to be changed to migrate the data in the data table to be changed to the new data table to obtain the target data table.

[0129] Optionally, the sending module 801 is specifically used to, if there are multiple first nodes, determine the data range that the first node is responsible for in the data table to be changed for each first node, and use it as the target data range corresponding to the first node; The execution module 802 is specifically used to scan the data tables to be changed within the target data range through the first node, so as to migrate the data in the data tables to be changed within the target data range to a new data table created in the first node, thereby obtaining the target data table within the target data range.

[0130] Optionally, the execution module 802 is specifically used to scan the data table to be changed, migrate the data in the data table to be changed to the new data table, and determine the log to be synchronized from the preset change data capture log in parallel, update the data migrated to the new data table according to the log to be synchronized, and obtain the target data table; the log to be synchronized is used to characterize the data change operations that occur during the execution of the data table structure change operation; the data change operations are used to change the data in the data table to be changed within the target data range.

[0131] Optionally, in the distributed database, logs associated with different data tables are stored in different change data capture log partitions; for each change data capture log partition, the change data capture log partition contains at least one sub-log partition divided according to the data range.

[0132] Optionally, the execution module 802 is specifically used to obtain the log to be synchronized from the sub-log partition corresponding to the target data range.

[0133] Optionally, the execution module 802 is specifically configured to: scan the data table to be changed to migrate the data in the data table to the new data table; and, in parallel, determine the data change records based on the synchronization logs determined from the preset change data capture logs, and store the data change records in an intermediate index; the data change records include: the row identifier of any row of data in the data table to be changed that has undergone data change and the changed data corresponding to that row; after the scan is completed, the intermediate index is merged with the new data table to obtain the target data table.

[0134] Optionally, the execution module 802 is specifically used to continue executing the data table structure change operation through the second node based on the checkpoint file and the index checkpoint file stored in the designated storage node; the index checkpoint file is a persistent snapshot of the intermediate index generated by the first node and saved to the designated storage node during the execution of the data table structure change operation.

[0135] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0136] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0137] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0138] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0139] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0140] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0141] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0142] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0143] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0144] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0145] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for processing data table structure change operations, the method being applied to a distributed database, the distributed database including a master node and a first node, the method comprising: The master node sends a table structure change command to the first node; The first node obtains the data table to be changed according to the table structure change command, and performs the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node during the execution of the data table structure change operation is saved to the designated storage node; When the master node detects an anomaly in the first node, it selects a second node from the distributed database to replace the first node, and then continues to perform the data table structure change operation through the second node based on the checkpoint file.

2. The method of claim 1, wherein the first node comprises: The standby node and the bidding node are service nodes that are persistently deployed in the database, and the bidding node is a service node that is temporarily obtained from the cloud computing platform.

3. The method as described in claim 1, specifically including: obtaining the data table to be changed, further comprising: Obtain the checkpoint file associated with the data table to be changed from the specified storage node as the target checkpoint file, and obtain the log associated with the data table to be changed as the target log; Data replay is performed based on the target checkpoint file and the target log to restore the data table to be changed in the local storage space.

4. The method as described in claim 1, wherein detecting an anomaly in the first node specifically includes: A heartbeat signal is sent to the first node. If no response signal is received from the first node based on the heartbeat signal within a set time, an anomaly is detected in the first node. or If a recycling notification message is received from the first node, an anomaly is detected in the first node. The recycling notification message indicates that the first node will be recycled.

5. The method as described in claim 4, wherein the second node continues to execute the data table structure change operation based on the checkpoint file, specifically including: When it is determined that the first node has encountered an anomaly based on the recycling notification message, the second node determines the checkpoint file closest to the recycling time from the checkpoint files associated with the data table to be changed, and uses it as the first checkpoint file. Based on the first checkpoint file, the data table structure change operation continues to be executed. The recycling time is included in the recycling notification message. The first checkpoint file is generated by the first node after sending the recycling notification message.

6. The method as described in claim 4, wherein the second node continues to execute the data table structure change operation based on the checkpoint file, specifically including: If a response signal based on the heartbeat signal is not received from the first node within the set time, the time when the first node becomes abnormal will be determined as the crash time. Using the second node, the checkpoint file closest to the crash time is determined from each checkpoint file associated with the data table to be changed, and is used as the second checkpoint file. Logs with timestamps between the time the second checkpoint file was generated and the crash time are determined from the preset change data capture log and are used as recovery logs. Based on the recovery logs and the second checkpoint file, the data table structure change operation continues to be performed.

7. The method as described in claim 1, performing a data table structure change operation to obtain a target data table after changing the table structure of the data table to be changed, specifically includes: Perform a data table structure change operation to create a new data table; wherein, the table structure of the new data table is obtained by changing the table structure of the data table to be changed after performing the data table structure change operation; Scan the data table to be changed to migrate the data in the data table to the new data table to obtain the target data table.

8. The method of claim 7, further comprising: If there are multiple first nodes, then for each first node, determine the data range that the first node is responsible for in the data table to be changed, and use it as the target data range corresponding to the first node; Scanning the data table to be modified to migrate data from the data table to the new data table to obtain the target data table, specifically includes: Through the first node, the data tables to be changed within the target data range are scanned to migrate the data in the data tables to be changed within the target data range to a new data table created in the first node, thereby obtaining the target data table within the target data range.

9. The method as described in claim 7, wherein scanning the data table to be changed to migrate data from the data table to the new data table to obtain the target data table, specifically includes: Scan the data table to be changed to migrate the data in the data table to the new data table, and in parallel determine the log to be synchronized from the preset change data capture log, update the data migrated to the new data table according to the log to be synchronized, and obtain the target data table; The log to be synchronized is used to characterize the data change operations that occur during the execution of the data table structure change operation; the data change operation is used to change the data in the data table to be changed within the target data range.

10. The method of claim 9, wherein in the distributed database, logs associated with different data tables are stored in different change data capture log partitions; for each change data capture log partition, the change data capture log partition contains at least one sub-log partition obtained by dividing according to the data range.

11. The method as described in claim 10, wherein determining the logs to be synchronized within the target data range from the preset change data capture logs, specifically includes: The log to be synchronized is obtained from the sub-log partition corresponding to the target data range.

12. The method of claim 7, wherein scanning the data table to be changed to migrate data from the data table to the new data table to obtain the target data table, specifically includes: Scan the data table to be changed to migrate the data in the data table to the new data table; as well as, In parallel, based on the logs to be synchronized determined from the preset change data capture log, the data change records are identified and stored in the intermediate index; The data change record includes: the row identifier of any row of data in the data table to be changed that has undergone a data change, and the corresponding changed data for that row; After the scan is complete, the intermediate index is merged with the new data table to obtain the target data table.

13. The method as described in claim 12, wherein the second node continues to execute the data table structure change operation based on the checkpoint file, specifically including: The second node continues to perform the data table structure change operation based on the checkpoint file and the index checkpoint file stored in the designated storage node; The index checkpoint file is a persistent snapshot of the intermediate index generated by the first node during the execution of the data table structure change operation and saved to the designated storage node.

14. A processing apparatus for data table structure change operations, the apparatus being applied to a distributed database, the distributed database including a master node and a first node, the apparatus comprising: The sending module is used to send a table structure change command to the first node through the master node; The execution module is used to obtain the data table to be changed through the first node according to the table structure change command, and to perform the data table structure change operation to obtain the target data table after the table structure of the data table to be changed is changed; wherein, the checkpoint file generated by the first node during the execution of the data table structure change operation is saved to the designated storage node; The recovery module is used to select a second node from the distributed database to replace the first node when the master node detects an anomaly in the first node, and to continue the data table structure change operation through the second node based on the checkpoint file.

15. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1 to 13 by executing the executable instructions.

16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 13.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 13.

Images