Distributed Database Schema Change Method Based on Delayed Migration

Abstract

The present invention discloses a method for changing the distributed database schema based on latency, which mainly solves the problem that the current online schema change of the distributed database only allows the new schema to provide services after all data of the source table has been synchronized. The implementation solution is as follows: the SQL engine of the distributed database is modified in two stages. That is, in the first stage, the schema change request is mainly processed and the new schema is initialized; in the second stage, the requests of users on the new schema are mainly processed. In this stage, the data involved in the user requests is first migrated from the old schema to the new schema, and then the new schema is used to process the user requests and provide relevant services. After receiving the schema change request, the system of the present invention can immediately use the new schema with only simple initialization, avoiding the long waiting for data migration. At the same time, special optimization processing can be carried out according to the type of schema change to improve the migration efficiency, effectively reducing the latency of user queries, and can be used for the online upgrade and update of application programs.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and particularly relates to a method for changing a distributed database schema, which can be used for online upgrade and update of application programs. Background Art

[0002] With the increasing demand for high availability, high scalability, and high reliability of data, more and more application programs choose distributed databases such as NewSQL as the backend database instances. During the update and iteration of these application programs, the requirements of the front-end applications are constantly changing, and as a formal language for describing the database structure, that is, the database schema often needs to be changed accordingly. Database schema change, also known as schema migration, is usually implemented by means of the Data Definition Language (DDL). In order to avoid the impact of table locking and offline schema change on the business, traditional mainstream relational database management systems (RDBMSs) use online schema change techniques or external migration tools to perform schema change operations.

[0003] The online schema change techniques proposed in European patents EP3213229A1 and EP2418591A1, as well as the open-source tools pt-osc and gh-osc provided by github, can all perform online schema changes, and their general processes are similar: First, the system receives the DDL statement for schema change, and after query processing, registers a new schema and its corresponding table or materialized view, which is also called a shadow table. At this time, the new schema does not provide services yet; then, the system copies the source table in the background, that is, migrates the data on the old schema to the shadow table. Since the source table is still online and can provide services at this time, in order to ensure data consistency, when the system processes requests to change the data of the source table, these modified data need to be synchronized to the shadow table, and this synchronization process is achieved by means of triggers bound to the source table or log propagation. After all data replication and migration are completed, the system locks the source table and renames the shadow table to complete the switch between the old and new schemas. However, the above-mentioned techniques or tools only allow the new schema to provide services after synchronizing all the data of the source table. If the amount of data in the table of the old schema is large, the schema change will take more time. This requires waiting for the backend database schema change to be completed before deploying the code update of the front-end application instance, which is not conducive to the online upgrade or update of application programs.

[0004] NewSQL is a class of modern relational distributed databases that integrates the features of NoSQL and RDBMS, aiming to provide the same scalability performance as NoSQL systems for OLTP read-write loads, while providing atomicity, consistency, isolation, and durability guarantees for transactions. Google's Spanner is a representative and pioneer in NewSQL, and its ideas have inspired distributed NewSQL databases such as CockroachDB, TiDB, and KaiwuDB. Their characteristics include: 1. Adopting a new architecture that is distributed, shared-nothing, and separates storage and computing, splitting the database into non-overlapping data sets, i.e., partitions or shards, to support horizontal scaling. 2. Based on distributed consistency and consensus algorithms, synchronizing redo logs among multiple replicas to ensure data consistency across multiple replicas, achieving true high availability and high reliability. 3. Mostly using the multi-version concurrency control MVCC protocol or a combination of the two-phase lock 2PL protocol and MVCC for concurrency control. However, the online schema change technology on these distributed NewSQL databases still follows the ideas of traditional mainstream RDBMS solutions, and mostly references the Google F1 paper to introduce a transitional state to ensure the availability and consistency of cluster data. Therefore, its disadvantages are the same as those of traditional mainstream RDBMS solutions.

[0005] In view of the above disadvantages of the existing technologies, researchers have proposed a technical solution that combines schema change and lazy migration for online schema change to avoid waiting for data migration during schema change, so that it can be used immediately after registering a new schema.

[0006] The paper "Non-blocking Lazy Schema Changes in Multi-Version Database Management Systems" proposes a multi-stage lazy schema migration method for schema change operations of adding columns. When making a schema change, the system adds a new logical schema without any data migration. Data for the same table may be stored in multiple physical tables with different schemas. The system processes queries on the new schema by interpreting data according to the latest schema, and only moves data from the old schema to the new schema when necessary. However, this method only targets schema change operations of adding columns, has a limited scope of application, and the characteristic of maintaining multiple physical tables also increases the complexity and overhead of the database system. At the same time, this method is only discussed in the single-machine database system Terrier and cannot be adapted to distributed database systems such as NEWSQL.

[0007] The paper "BullFrog: Online Schema Evolution via Lazy Evaluation" uses a similar idea to change the schema by delaying data migration. The difference is that the system only needs to maintain two schema versions, and the new schema can provide services immediately after registration. When the system receives a query on the new schema, it first migrates the relevant data involved in the old schema to the new schema and then processes the query on the new schema. Although this solution supports more types of schema change operations and solves some backward compatibility problems of schema changes, it can only be carried out on the single-machine database system PostgreSQL at present and has not been explored and studied on distributed databases such as NewSQL. Currently, the work on schema changes in distributed database systems focuses more on the parallel acceleration of DDL and the exception handling of multi-version data. Therefore, it is of great significance to study how to use the delayed migration strategy to change the schema of distributed databases. Summary of the Invention

[0008] The purpose of the present invention is to propose a method for changing the schema of a distributed database based on delayed migration in view of the above deficiencies of the prior art, so as to avoid long waiting for data migration and enable the new schema to provide services quickly by performing online schema changes on a NewSQL distributed database.

[0009] The technical solution of the present invention mainly modifies the SQL engine of the distributed NewSQL database in two stages, that is, the first stage mainly processes schema change requests and performs some initialization operations; the second stage mainly processes user requests on the new schema. In this process, the data involved in the user request will be migrated from the old schema to the new schema first, and then the new schema will be used to process the user request. The implementation steps are as follows:

[0010] (1) The user first submits a schema change request and describes the structure of the tables and the data involved in the new schema.

[0011] (2) After receiving the request, the distributed database system creates an empty new table for the new schema according to the type of schema change.

[0012] (3) Create a predicate filter on the old table and associate a background migration thread with the new table to complete the initialization. The user starts to issue SQL requests for insert, delete, update, and query through the created new table.

[0013] (4) The distributed database system receives the SQL request on the new table and determines its type:

[0014] If the request is an INSERT statement, execute (5).

[0015] If the request is a SELECT / UPDATE / DELETE statement, then execute (6).

[0016] For the remaining requests, no processing is performed and an error message is returned.

[0017] (5) The distributed database system directly executes the INSERT logic on the new table and returns the successfully executed result to the user.

[0018] (6) The distributed database system uses a predicate filter to locate the data on the old table required by the request statement.

[0019] (7) The distributed database system checks the located data to determine whether all of this data has been migrated to the new table:

[0020] If all have been migrated, then execute (8).

[0021] Otherwise, execute (9).

[0022] (8) The distributed database system executes the SELECT / UPDATE / DELETE logic on the new table and returns the successfully executed result to the user.

[0023] (9) Migrate the data that has not been migrated yet and return to (7).

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, since the present invention migrates relevant data on demand before processing requests for a new mode, delaying and splitting the entire migration process, the system can simply initialize and immediately use the new mode after receiving a mode change request, avoiding long waits for data migration and providing support for application upgrades or updates that need to directly deploy updates to front-end code instances and back-end database instances.

[0026] Second, since the present invention performs special optimization processing for the type of mode change, creating a special empty table with the same data distribution as the old table for mode changes where the new table contains the primary key of the old table, the number of nodes for executing distributed plan routing and distribution during data migration is reduced, the migration efficiency is improved, and the latency of user queries is effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0028] Figure 1 This is the overall flowchart for the implementation of the present invention;

[0029] Figure 2 This is the sub-flowchart for the first stage in the present invention;

[0030] Figure 3 This is the schematic diagram of the mode change request in the first stage of Embodiment 1 of the present invention;

[0031] Figure 4 This is the data distribution diagram of the old user table used in the present invention;

[0032] Figure 5 This is the sub-flowchart for the second stage in the present invention;

[0033] Figure 6 This is the schematic diagram of data migration on a special empty table in the second stage of Embodiment 1 of the present invention;

[0034] Figure 7 This is the schematic diagram of the mode change request in the first stage of Embodiment 2 of the present invention;

[0035] Figure 8 This is the schematic diagram of data migration on a normal empty table in the second stage of Embodiment 2 of the present invention.

[0036] Figure 9 This is the NewSQL database cluster diagram used in the present invention; Detailed implementation manners

[0037] The following further describes the embodiments of the present invention in detail with reference to the accompanying drawings:

[0038] A method for changing the distributed database mode based on delayed migration proposed by the present invention mainly modifies the SQL engine of the distributed NewSQL database. It is divided into two stages: the first stage is to process the mode change request, that is, to perform some initialization operations without involving specific data migration to obtain the new mode; the second stage is to process the user requests on the new mode, that is, to first migrate the data involved in the user requests from the old mode to the new mode, and then use the new mode to process the user requests.

[0039] The present invention gives the following two embodiments, which use a distributed NewSQL database with a key-value (KV) storage engine and a shared-nothing architecture, but are not limited to this type of database.

[0040] Embodiment 1, perform distributed database mode change based on the mode change request where the newly created table structure contains the primary key of the old table.

[0041] Refer to Figure 1 , the implementation steps of this example are as follows:

[0042] Step 1, process the schema change request in the first phase.

[0043] Refer to Figure 2 , the implementation steps of this phase are as follows:

[0044] 1.1) The user first submits a schema change request to the distributed database system, which is a DDL statement that describes the structure of the tables in the new schema and the data involved;

[0045] In this example, the old table user is changed to the new user_rights table. The primary key of the new user_rights table is the same as that of the old user table, which is the id column. The DDL statement corresponding to this schema change is CREATE TABLE user_rights ASSELECT id,rights FROM user, as Figure 3 shown.

[0046] 1.2) After receiving the request, the distributed database system creates an empty new table for the new schema according to the type of schema change:

[0047] If the structure of the new table to be created does not contain the primary key of the old table, a normal empty table is created according to the default policy of the current distributed database system, such as in Embodiment 2;

[0048] If the structure of the new table to be created contains the primary key of the old table, on the basis of creating a normal empty table, a special empty table with the same data distribution as the old table is created according to the metadata information of the old table, that is, execute 1.3);

[0049] In this example, considering the schema change from the old user table to the new user_rights table, the data distribution of the old user table used on a three-node database cluster is as Figure 4 shown. This table contains 3 shards, range1, range2, and range3, and each shard has three replicas. The gray area is the replica holding the lease, and strong consistent synchronization is performed between the replicas through the distributed consensus protocol Raft. For convenience, only the values of the primary key id column in the data of the cluster are marked, and the Key-Value pairs represented by it are listed in the right half of the figure. The Key is encoded by the table number, primary key, and timestamp ts, and the Value is encoded by the non-primary key data. Assuming the table number of the old user table is 51, the Key form of the old user table data is / 51 / id / ts, and the Value form is / name / age / rights. In this example, since the new user_rights table contains the primary key column id of the old user table, a special empty table needs to be created.

[0050] 1.3) The implementation steps of creating a special empty table are as follows:

[0051] 1.3.1) The distributed database system looks up the metadata information of the old table, obtains the table number of the old table and the columns and data types of the primary key of the old table, and maps them to the corresponding columns and data types in the new table, which are denoted as mapped columns;

[0052] 1.3.2) Construct the Key information of the new table: The Key of the new table data is composed of two parts, a prefix and a body. The prefix is the old table number plus the mapped column, and the body is the Key composed of the new table number and its own primary key column. That is, when storing the data of the new table, the Key of the old table is used as the prefix;

[0053] 1.3.3) Write the constructed Key information into the metadata information of the new table and use it as the basis for interpreting data when accessing the underlying key-value Key-Value storage engine;

[0054] 1.3.4) Add an additional flag bit to the metadata information of the new table to ensure that the distributed database system will perform operations of adding and deleting the Key prefix when performing read and write operations on the new table subsequently.

[0055] In this example, since the table number of the old user table is 51, the mapped column is the id column, the table number of the new user_rights table is 52, and the primary key of the new table is also the id column, the Key form of the data in the new user_rights table is / 51 / id / * / 52 / id / ts, with the symbol * dividing the prefix and the body of the Key, and the Value form is / rights.

[0056] 1.4) Create a predicate filter on the old table according to whether the distributed database system supports view resolution technology:

[0057] If it supports, the distributed database system creates a view with the same structure as the new table on the old table according to the structure of the new schema table in the schema change request, and uses it as the predicate filter;

[0058] If it does not support, then rely on the existing open-source tool LessQL for query rewriting on github, whose link address is https: / / github.com / morris / lessql and introduce the LessQL middleware as the predicate filter.

[0059] In this example, since the distributed database system supports view resolution technology, a view with the same structure as the new user_rights table is created on the old user table as the predicate filter, and the DDL statement for creating the view is CREATE VIEW user_rights_view AS SELECT id, rights FROM user;

[0060] 1.5) Associate a background migration thread with the old table:

[0061] 1.5.1) Obtain the total amount of data in the old table from the distributed database system, divide the total amount of data into small data blocks according to the data primary key or row id, and then generate multiple range SELECT statements on this basis to form a new table query script sufficient to traverse the data of the old table;

[0062] 1.5.2) Create a new thread and set a timer to periodically run the script to simulate the user's query request for the new table, so as to ensure that all the data in the old table will eventually be migrated.

[0063] In this example, there are five pieces of data in the old user table. Three range SELECT statements can be generated according to the primary key id <10, 11 <id <20, id > 20 to form a new table query script.

[0064] Step 2, deployment and use of the new mode.

[0065] The user migrates the relevant application services on the old mode to the new mode and uses the new mode to provide services for it.

[0066] Step 3, process the user's requests on the new mode in the second stage.

[0067] Refer to Figure 5 , the implementation steps of this stage are as follows:

[0068] 3.1) The user sends a request on the new mode using the application service. The distributed database system receives the SQL request on the new table and judges its type:

[0069] If the request is an INSERT statement, execute 3.2),

[0070] If the request is a SELECT / UPDATE / DELETE statement, execute 3.3),

[0071] Return error information for the remaining requests;

[0072] In this example, the gateway node node3 of the distributed database system receives a SELECT statement request on the new mode: SELECT id, rights FROM user_rights WHERE id < 10;

[0073] 3.2) The distributed database system directly executes the INSERT logic on the new table and returns the successfully executed result to the user;

[0074] 3.3) The distributed database system uses a predicate filter to locate the data on the old table required by the request statement:

[0075] 3.3.1) Judge whether the distributed database system supports view resolution technology:

[0076] If supported, the distributed database system first rewrites the user's SQL request for the new table into a request for a view on the old table, then inputs it to the query optimizer to obtain an execution plan, obtains the filtering predicate or expression on the old table according to the execution plan, and constructs a SELECT statement on the old table using the filtering predicate or expression;

[0077] If not supported, the distributed database system uses the LessQL middleware to convert the user's SQL request for the new table into a request for the old table, then inputs it to the query optimizer to obtain an execution plan, obtains the filtering predicate or expression on the old table according to the execution plan, and constructs a SELECT statement on the old table using the filtering predicate or expression;

[0078] In this example, the distributed database system rewrites the user's SQL request for the new table, SELECT id, rights FROM user_rights WHERE id < 10; into a request for a view on the old table, SELECT id, rights from user_rights_view WHERE id < 10; inputs it to the query optimizer to obtain an execution plan, obtains the filtering predicate id < 10 on the old user table according to the execution plan, and constructs the SELECT statement on the old user table using this filtering predicate as SELECT id, rights from user WHERE id < 10;

[0079] 3.3.2) Use the SELECT statement of the old table constructed in 3.3.1) as a subquery of the INSERT statement used when migrating data. The distributed database system executes the subquery part of the INSERT statement, obtains the result after execution from the storage engine, which is the data on the old table required by the new table SQL request statement.

[0080] In this example, use the SELECT statement of the constructed old user table, SELECT id, rights from user WHERE id < 10, as a subquery of the INSERT statement used when migrating data: INSERT INTO user_rights (id, rights) (SELECT id, rights from user WHERE id < 10); The distributed database system executes the subquery part of the INSERT statement, and thus determines that the relevant data in the old user table is on the shard Range1. At this time, among the three replicas of the shard Range1, the replica on node node1 holds the lease, that is, the lessee node is node1. Therefore, the system routes to node1 and obtains the relevant data, such as Figure 6As shown by the solid arrows in the figure.

[0081] 3.4) The distributed database system checks each piece of positioning data to determine whether this data has been migrated to the new table:

[0082] If the timestamp is the preset migrated flag bit, it means that this piece of data has been migrated;

[0083] If the timestamp is the preset migrating flag bit, it means that this piece of data is being migrated;

[0084] The remaining timestamps indicate that this piece of data has not been migrated;

[0085] 3.5) Determine whether all the positioning data has been fully migrated:

[0086] If each piece of positioning data has been migrated, execute 3.6);

[0087] Otherwise, execute 3.7);

[0088] In this example, the preset migrating flag bit is ts_migrating1, and the preset migrated flag bit is ts_migrated1. The distributed database system checks two pieces of data with positioning IDs 1 and 2, and their timestamps are ts, indicating that both pieces of data have not been migrated.

[0089] 3.6) The distributed database system executes the SELECT / UPDATE / DELETE logic on the new table and returns the successfully executed results to the user;

[0090] 3.7) Migrate the data that has not been migrated, and then return to 3.4), which is implemented as follows:

[0091] 3.7.1) The distributed database system executes the parent query part of the INSERT statement for migrating data constructed in 3.3.2):

[0092] For the data that is being migrated and the data that has been migrated, the system skips the migration;

[0093] For the data that has not been migrated, execute 3.7.2);

[0094] 3.7.2) The system performs the migration and inserts the data in the old table into the new table:

[0095] At the beginning of the migration, the system modifies the timestamp of the data in the old table to the preset migrated flag bit;

[0096] When the migration is completed, the system modifies the timestamp of the data in the old table to the preset migrating flag bit and returns to 3.4).

[0097] In this example, the INSERT statement used for migrating data is INSERT INTO user_rights(id,rights)(SELECT id,rights from user WHERE id<10); the distributed database system executes the parent query part of this statement for data migration. Since a special empty table was created in 1.3), the Key in the data of the old user table is used as the prefix of the Key in the data of the new user_rights table. Therefore, the data distribution of the new table and the old table is the same, and data with similar Keys is on the same shard. The distributed database system modifies the timestamps of the data of two old user tables with ids 1 and 2 to the preset migrating flag bit ts_migrating1. According to the encoding form of the Key in the new user_rights table, the data of its two rows with ids 1 and 2 should also be located on shard range1. Therefore, the distributed database system inserts data on the lessee node node1 of range1; finally, the timestamps of the data of the two old user tables with ids 1 and 2 are modified to the preset migrated flag bit ts_migrated1, as Figure 6 shown by the dashed arrow in the figure, and finally return to step 3.4).

[0098] Example 2. Based on the schema change request that the newly created table structure does not contain the primary key of the old table, perform distributed database schema change.

[0099] The implementation process of this example is the same as that of Example 1. The difference is that the request for submitting schema change to the distributed database system by the user is different. That is, in this example, the old table user is changed to the new user_new table, and the primary key of the new user_new table is the name column. The DDL statement corresponding to this schema change is CREATE TABLE user_new AS SELECT name,age FROM user, as Figure 7 shown.

[0100] The main different steps between this example and Example 1 for this request are described as follows:

[0101] In step 1.2), since the new user_new table does not contain the primary key in the old user table in this example, it is necessary to create a normal empty table.

[0102] The data distribution of the old table user used in this example on a three-node database cluster is the same as that in Example 1, as Figure 4As shown in the figure. The node in the distributed database system that receives the schema change request is the gateway node node3. A normal empty table is created on node3 according to the default policy of the current distributed database system. The table number is 52. The Key form of the new user_new table data is / 52 / name / ts, and the Value form is / age.

[0103] In step 3.1), the gateway node node3 of the distributed database system in this instance receives a SELECT statement request on the new schema: SELECT name,age FROM user_new WHERE age<20;

[0104] In steps 3.3) to 3.7), in this instance, the system locates the relevant data in the old table on the shard Range1 through the predicate filter. At this time, the leaseholder node of Range1 is node1. Therefore, it is routed to node1. After determining that the relevant data has not been migrated, the data is migrated to the node node3 where the new user_new table is located, as Figure 8 shown in the figure.

[0105] Compared with the special empty table in Embodiment 1, the normal empty table created in this instance increases the number of nodes that execute distributed plan routing and distribution during data migration.

[0106] Term Explanation

[0107] Shard: It means that the NewSQL database uses the range partitioning method to divide the globally incrementally ordered Key-Value data into blocks of a certain unit size. It is the basic unit of routing, storage, and replication, and has the ability to automatically split and merge.

[0108] Replica: It means multiple copies of a shard. The NewSQL database will replicate each shard and store each shard replica on different nodes to provide high availability.

[0109] Lease: It means that the authorizer grants a lease and commitment for a period of time in the distributed environment. The node where the replica holding the lease in the NewSQL database can provide consistent read and write of the shard KV externally.

[0110] Distributed Consensus Protocol: It refers to the protocol used by NewSQL databases to ensure data consistency among nodes. Taking the Raft distributed consensus protocol in the paper "In search of an understandable consensus algorithm" as an example, all replicas of a shard form a Raft group, which performs strongly consistent synchronization with the help of the Raft protocol, ensuring highly available and consistent read and write operations externally. One of the replicas is the long-lived leader, coordinating all read and write operations sent to this Raft group, and the other replicas are followers.

[0111] Leaser Node: It refers to the node where the replica holding the lease is located in the NewSQL database. The leaser node can provide consistent read and write of the shard KV externally. The replica holding the lease is generally also a leader in the Raft group.

[0112] Gateway Node: It refers to the node that is responsible for parsing SQL requests, acting as the coordinator of transactions, and routing KV operations to the correct leaser node. The gateway node is a concept opposite to the leaser node.

[0113] A three-node NewSQL database cluster has a structure as Figure 9 shown, with a total of three shards r1, r2, and r3. Each shard contains three replicas, and the Raft protocol is used among these replicas to ensure data consistency, with the leaser node providing consistent read and write.

Claims

1. A method for changing the distributed database schema based on delayed migration, characterized in that It includes the following: (1) The user first submits a schema change request and describes the structure of the tables and the data involved in the new schema; (2) After receiving the request, the distributed database system creates an empty new table for the new schema according to the type of schema change; (3) Create a predicate filter on the old table and associate a background migration thread with the new table to complete the initialization. The user starts to issue SQL requests for insert, delete, update, and query through the created new table; (4) The distributed database system receives the SQL request on the new table and determines its type: If the request is an INSERT statement, then execute (5); If the request is a SELECT / UPDATE / DELETE statement, then execute (6); For the rest of the requests, no processing is done and an error message is returned; (5) The distributed database system directly executes the INSERT logic on the new table and returns the successful execution result to the user; (6) The distributed database system uses the predicate filter to locate the data on the old table required by the request statement; (7) The distributed database system checks each piece of located data to determine whether all of this data has been migrated to the new table: If all have been migrated, then execute (8); Otherwise, execute (9); (8) The distributed database system executes the SELECT / UPDATE / DELETE logic on the new table and returns the successful execution result to the user; (9) Migrate the data that has not been migrated yet and return to (7).

2. The method according to claim 1, wherein: In (2), creating an empty new table is based on whether the structure of the new table to be created according to the schema change contains the primary key of the old table: If it does not contain, create an ordinary empty table according to the default policy of the current distributed database system; If it contains, on the basis of creating an ordinary empty table, create a special empty table with the same data distribution as the old table according to the metadata information of the old table.

3. The method according to claim 2, characterized in that: The realization of creating a special empty table with the same data distribution as the old table according to the metadata information of the old table is as follows: First, the distributed database system looks up the metadata information of the old table, obtains the table number of the old table and the columns and data types of the primary key of the old table, and maps them to the corresponding columns and data types in the new table, denoted as the mapped columns; Next, construct the Key information of the new table: The Key of the new table data is composed of two parts, a prefix and a body. The prefix is the old table number plus the mapped column, and the body is the Key composed of the new table number and its own primary key column. That is, when storing the data of the new table, the Key of the old table is used as the prefix; Then, write the constructed Key information into the metadata information of the new table and use it as the basis for interpreting the data when accessing the underlying key-value Key-Value storage engine; Finally, add an additional flag bit to the metadata information of the new table to ensure that the distributed database system will perform operations of adding and deleting the Key prefix during subsequent read and write operations on the new table.

4. The method according to claim 1, characterized in that: In (3), creating a predicate filter on the old table is based on whether the distributed database system supports view resolution technology: If it supports, the distributed database system creates a view with the same structure as the new table on the old table according to the structure of the new schema table in the schema change request as the predicate filter; If not supported, then with the help of the open-source tool LessQL for query rewriting on the existing GitHub, introduce the LessQL middleware as a predicate filter.

5. The method according to claim 1, characterized in that: The step (3) associates a background migration thread with the old table, and the implementation is as follows: First, the distributed database system obtains the total amount of data in the old table, splits it into small data blocks according to the data primary key or row id, and then generates multiple range SELECT statements on this basis to form a new table query script sufficient to traverse the data in the old table; Then, create a new thread and set a timer to run this script regularly to simulate the user's query request for the new table, so as to ensure that all the data in the old table will eventually be migrated.

6. The method according to claim 1, wherein: In the step (6), the distributed database system uses a predicate filter to locate the data on the old table required by the request statement, and the implementation is as follows: (6a) Determine whether the distributed database system supports view resolution technology: If supported, the distributed database system first rewrites the user's SQL request for the new table as a request for the view on the old table, then inputs it to the query optimizer to obtain an execution plan, obtains the filtering predicate or expression on the old table according to the execution plan, and constructs a SELECT statement on the old table using this filtering predicate or expression; If not supported, the distributed database system uses the LessQL middleware to convert the user's SQL request for the new table into a request for the old table, then inputs it to the query optimizer to obtain an execution plan, obtains the filtering predicate or expression on the old table according to the execution plan, and constructs a SELECT statement on the old table using this filtering predicate or expression; (6b) Use the SELECT statement of the old table constructed in (6a) as a subquery of the INSERT statement used when migrating data. The distributed database system executes the subquery part of this INSERT statement to obtain the result after execution from the storage engine, that is, the data on the old table required by the new table SQL request statement.

7. The method according to claim 1, characterized in that: In the step (7), the distributed database system checks each piece of located data by checking the timestamp of the located data to determine whether it has all been migrated to the new table: If the timestamp is the preset migrated flag bit, it means that this piece of data has been migrated; If the timestamp is the preset migrating flag bit, it means that this piece of data is being migrated; The remaining timestamps indicate that this piece of data has not been migrated yet.

8. The method according to claim 1, characterized in that: The step (9) migrates the data that has not been migrated yet, and the implementation is as follows: 9a) The distributed database system executes the parent query part of the INSERT statement for migrating data constructed in 6(b): For the data being migrated and the data that has been migrated, the system skips the migration; For the data that has not been migrated, execute 9b); 9b) The system performs the migration and inserts the data in the old table into the new table: At the beginning of the migration, the system modifies the timestamp of the data in the old table to the preset migrating flag bit; When the migration is completed, the system modifies the timestamp of the data in the old table to the preset migrated flag bit.

Images