Database migration method and apparatus, database cluster

Abstract

Embodiments of the present application provide a database migration method and device, and a database cluster. The method comprises: in response to a migration request for any database in a source cluster, determining a to-be-migrated data table corresponding to the database; acquiring storage rule information of the to-be-migrated data table in the source cluster; querying an index table associated with the to-be-migrated data table in the source cluster according to the storage rule information; and copying the to-be-migrated data table and the index table to a target cluster. Embodiments of the present application improve the database migration efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic devices, and in particular to a database migration method and device and a database cluster. BACKGROUND

[0002] A database is a data collection that stores data according to a specified data structure. A database can usually be stored in a computer cluster or a server cluster, which is a computer system. A cluster can be composed of multiple computer nodes, and the cluster can store data tables of a database. A data table can be a table that stores real data. A data table can be composed of a series of Key-Value pairs. Key is used to identify the entity meaning corresponding to the data, and Value refers to the specific data.

[0003] In the prior art, when a cluster needs to be upgraded or replaced, the data tables stored in the cluster need to be migrated to a new cluster. When migrating a data table, a migration command of the data table is usually run to copy the Key-Value pairs in the data table from a source node to a target node. In order to improve the query efficiency of data, an index directory can usually be established for each Key-Value pair. That is, each Key-Value pair of the original cluster data table is sequentially copied to the new cluster, and an index directory is established for each Key-Value pair in the new cluster every time a Key-Value pair is stored, until all Key-Value pairs of the data table are copied. Then, when data needs to be queried, the index directory of the new data table is directly queried to obtain the storage address of the data, and the data is located according to the storage address to achieve fast reading of the data.

[0004] As can be seen from the above description, when migrating a data table from a source cluster to a new cluster, all Key-Value pairs of the data table need to be traversed, and a new index directory needs to be established for each Key-Value pair, which takes a long time and results in low database migration efficiency. SUMMARY

[0005] Therefore, the embodiments of the present application provide a database migration method and device and a database cluster to solve the technical problem that in the prior art, a new index directory needs to be established for each Key-Value pair in a data table when migrating the database, which results in low database migration efficiency.

[0006] In a first aspect, the embodiments of the present application provide a database migration method, comprising:

[0007] In response to a migration request for any database in a source cluster, determining a to-be-migrated data table corresponding to the database;

[0008] Obtaining storage rule information of the to-be-migrated data table in the source cluster;

[0009] According to the storage rule information, query an index table associatedly stored with the to-be-migrated data table in the source cluster;

[0010] Copy the to-be-migrated data table and the index table to the target cluster in association.

[0011] In a second aspect, an embodiment of the present application provides a database migration method, comprising:

[0012] In response to a database migration request initiated for a database migration interface, acquire processing resources corresponding to the database migration interface;

[0013] Use the processing resources corresponding to the database migration interface to perform the following steps:

[0014] In response to a migration request for any database in a source cluster, determine a to-be-migrated data table corresponding to the database;

[0015] Acquire storage rule information of the to-be-migrated data table in the source cluster;

[0016] According to the storage rule information, query an index table associatedly stored with the to-be-migrated data table in the source cluster;

[0017] Copy the to-be-migrated data table and the index table to the target cluster in association.

[0018] In a third aspect, an embodiment of the present application provides a database migration apparatus, comprising:

[0019] A request response module, configured to, in response to a migration request for any database in a source cluster, determine a to-be-migrated data table corresponding to the database;

[0020] A rule acquisition module, configured to acquire storage rule information of the to-be-migrated data table in the source cluster;

[0021] An index query module, configured to, according to the storage rule information, query an index table associatedly stored with the to-be-migrated data table in the source cluster;

[0022] A data migration module, configured to copy the to-be-migrated data table and the index table to a target cluster in association.

[0023] In a fourth aspect, an embodiment of the present application provides a database cluster, comprising: at least one node and a database processing component;

[0024] The at least one node is configured to distribute one or more databases;

[0025] The database processing component is configured to:

[0026] In response to a migration request for any database, it is determined that the database corresponds to a data table to be migrated; storage rule information of the data table to be migrated is acquired; according to the storage rule information, an index table associated with the data table to be migrated is queried; and the data table to be migrated and the index table are associated and copied to a target cluster.

[0027] According to the embodiment of the present application, in response to a migration request for a data table in a source cluster, storage rule information of the source cluster can be acquired, so that an index table associated with the data table to be migrated in the source cluster is queried according to the storage rule information, and then the data table to be migrated and the index table are associated and copied to a target cluster. It is not necessary to establish an index directory for key-value pairs in the data table one by one, and the migration efficiency of the database is improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 A flowchart of one embodiment of a database migration method provided by the embodiment of the present application;

[0030] Figure 2 A flowchart of another embodiment of a database migration method provided by the embodiment of the present application;

[0031] Figure 3 A schematic diagram of one embodiment of a database migration method provided by the embodiment of the present application;

[0032] Figure 4 A flowchart of one embodiment of a database migration method provided by the embodiment of the present application;

[0033] Figure 5 A structural schematic diagram of one embodiment of a database migration device provided by the embodiment of the present application;

[0034] Figure 6 A structural schematic diagram of one embodiment of a computing device provided by the embodiment of the present application;

[0035] Figure 7 A structural schematic diagram of one embodiment of a database cluster provided by the embodiment of the present application;

[0036] Figure 8 A structural schematic diagram of one embodiment of a database system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] The terms used in the embodiments of the present application are only for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0039] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0040] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to identifying". Similarly, depending on the context, the phrase "if it is determined" or "if it is identified (a stated condition or event)" can be interpreted as "when it is determined" or "in response to determining" or "when it is identified (a stated condition or event)" or "in response to identifying (a stated condition or event)".

[0041] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such product or system. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the product or system including the element.

[0042] The technical solutions of the embodiments of the present application can be applied to the migration scene of a database, for example, the fast migration of a Cassandra (an open source distributed NoSQL (Not Only Structured Query Language, a non-relational database) database system) database.

[0043] In order to facilitate the understanding of the technical solutions of the present application, the technical terms that may appear in the embodiments of the present application are first explained as follows:

[0044] Database: a long-term storage in a computer, a large amount of data organized, shareable, and uniformly managed collection, a database name can be set to a database.

[0045] Data table: DataTable, a table in a database that stores real data, a piece of data in a data table is called a record, and a record is usually composed of one or more key-value pairs.

[0046] Index table: the table formed by the index directory established for each record. The index directory is composed of keys and corresponding storage locations. When querying a record in the data table, the key of the record can be queried, and the storage location corresponding to the key can be queried, and then the required data can be directly queried from the storage location without scanning the data table, thereby quickly obtaining the required data and improving the query efficiency of the data. In the Cassandra database, the index table is invisible to the user or lessee of the database.

[0047] The data table can be divided into primary and secondary indexes according to the primary and secondary relationships or the order. The primary index is the cluster index, and the leaf node saves the primary key index. The secondary index is the non-cluster index, and the leaf node saves the data row corresponding to the primary key value.

[0048] Data partitioning: a physical database design technique that splits a data table to obtain a data table that can evenly distribute the data of the database table to different storage spaces, and each data partition stores part of the data. After the data table is split into data partitions, it is actually still a data table. By partitioning, the data of the table can be evenly distributed to different places, improving the efficiency of data retrieval and reducing the access pressure of the database. When querying the database table, the query is performed on the multiple data partitions obtained by splitting. The query results of different data partitions can be aggregated to obtain the final result of the entire database table. After the data table is partitioned, it is stored in the corresponding storage space in a certain file format. For example, in the Cassandra database, a data table can obtain multiple SStable (Sorted String Table, Sorted String Table) files after being partitioned. SStable is a file format for storing key-value pairs in order on the disk, and is a data file. Multiple SStable files can be stored in different computing nodes in the cluster to achieve distributed storage.

[0049] Due to user's own reasons, the database established by the user can be migrated to another computer cluster, or the cloud server providing the database service in the background can also migrate the database stored by the user to another computer cluster. In order to improve the query efficiency of the database, after the data table of the database is migrated, the records or data in the database can be read one by one to construct a new index for each piece of data according to the construction mode of the database index directory. For example, when the SStable file, that is, the data file, on the source node needs to be copied to the target node in the Cassandra database, the noderefre command can be called to load the SStable file to the target node, and then each data key of the SStable file is traversed, the value corresponding to the key is read, and then a two-level index is constructed one by one based on the size of the Key and Kalue. Or directly copy each Key-Value data in the data table to a csv format file, and then use the copy form command to read each data in the csv file, write it to the target node, and construct a two-level index for each key-value pair written to the target node when writing to the target node. The construction mode of the above two indexes is to read the key-value pair data one by one and construct the index, which consumes a lot of time and causes low efficiency of database migration.

[0050] In the embodiment of the present application, by using the storage rule information of the source cluster, the index table associated with the data table is queried, and the data table and the index table are copied to the target cluster at the same time, so that the index directory for the key-value pair in the data table does not need to be established one by one, and the migration efficiency of the database is improved.

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] Figure 1 An embodiment of a flowchart of a database migration method provided in the embodiments of the present application can include the following steps:

[0053] 101: In response to a migration request for any database in the source cluster, determine the database corresponding to the data table to be migrated.

[0054] 102: Obtain the storage rule information of the data table to be migrated in the source cluster.

[0055] The embodiments of the present application can be applied to a computing device, which can include a cloud server, a computer, a super personal computer, a notebook computer, and the like. The specific type of the computing device is not limited in the embodiments of the present application.

[0056] The data table is a subordinate unit of data in the database, and the database is used to query the data items in the data table. When the database is migrated, the data table corresponding to the database needs to be migrated. In actual application, the data amount stored in the data table is relatively large, and occupies a large memory, thus generating a distributed storage demand. The data table can be partitioned for storage in multiple nodes of the source cluster. For example, when the storage space occupied by the data table is 1 TB (Terabyte), the data table can be partitioned and stored in multiple nodes of the cluster in a distributed manner. In order to improve the query efficiency of the data table, an index table can be established for the data table in general, and the data table and the index table corresponding thereto are stored in association.

[0057] Different data tables can be distinguished by using a data table identifier. The migration request can include an identifier of the database to determine the database to be migrated.

[0058] Optionally, the migration request can be initiated by a target user, and the method can further include: receiving the migration request initiated by the target user for any database. The target user can be a user of the database or a service provider of the database. The user of the database can be a user who queries the data table corresponding to the database. The service provider of the database can be an operation and maintenance user who provides the database storage service. When the computing device detects the migration request initiated by the target user for any data table, the migration request initiated by the target user based on the database identifier of any database can be detected.

[0059] Optionally, the migration request can also be automatically generated by the computing device when it detects that a certain database has a migration demand. In this case, the method can further include: generating the migration request of any database when the database meets the migration condition. When the computing device generates the migration request of the data table to be migrated, the migration request can be generated based on the identifier of the database to be migrated.

[0060] In actual application, the storage rule information of the data table is known. The storage rule information can be set in the local computing device for real-time reading, or can also be stored in the source cluster, and the storage rule information of the data table to be migrated can be requested from the source cluster after the data table to be migrated is determined. Alternatively, the storage rule information of the data table to be migrated can also be provided by the target user, that is, the storage rule information of the data table to be migrated can be detected after the data table to be migrated is determined.

[0061] 103: According to the storage rule information, an index table associated with the to-be-migrated data table in the source cluster is queried.

[0062] The data table is stored in the source cluster in a certain storage manner, and the index table corresponding to the data table can be stored in association with the data table in the source cluster. According to the storage rule information, the index table associated with the to-be-migrated data table in the source cluster can be queried, which can include: according to the storage rule information, the association storage relationship between the to-be-migrated data table and the index table corresponding thereto is determined, and according to the association storage relationship, the index table associated with the to-be-migrated data table is determined.

[0063] Optionally, the index table of the to-be-migrated data table can be a secondary index table. For a primary index, the data volume is small and the reconstruction cost is low. For a secondary index, the data volume is very large, so the reconstruction cost is very high, and a secondary index needs to be established for each key-value pair. Therefore, for the secondary index table of the to-be-migrated data table, the to-be-migrated data table is directly associated and migrated, and the index table of the data table does not need to be reconstructed, thereby improving the migration efficiency of the index table.

[0064] 104: The to-be-migrated data table and the index table are associated and copied to the target cluster.

[0065] The to-be-migrated data table and the index table associated with the to-be-migrated data table are associated and copied to the target cluster. After the target cluster obtains the to-be-migrated data table and the index table associated with the to-be-migrated data table, the to-be-migrated data table and the index table associated therewith can be stored in association.

[0066] In the embodiments of the present application, in response to a migration request for a data table in a source cluster, storage rule information of the source cluster can be obtained, so that an index table associated with the to-be-migrated data table in the source cluster is queried according to the storage rule information, and then the to-be-migrated data table and the index table are associated and copied to the target cluster. It is not necessary to establish an index directory for each key-value pair in the data table, thereby improving the migration efficiency of the database.

[0067] In actual applications, the data volume stored in the data table corresponding to the database is usually large. The time cost of copying the data table from the source cluster to the target cluster is very low, for example, the copying time of a 1TB data table is about 1 minute. However, in general, due to the constraints of the prior art, the index table corresponding to the data table is usually invisible. In order to continue to efficiently query data in the target cluster, a new index table can be established for the data table in the target cluster. However, if a new index is constructed for each key-value pair in the data table in the target cluster, it will take several hours, which will greatly reduce the efficiency of database migration.

[0068] In order to solve the technical problem of low migration efficiency of the database in the source cluster. The inventors found that when the data table is stored in the source cluster, in order to solve the technical problem of large data volume of the data table, which is not conducive to data storage and improvement of data query efficiency, the data table is usually distributedly stored in the source cluster, and the data table is distributedly stored in at least one node of the source cluster. When the data table is distributedly stored in different nodes, the data file stored in each node is associated with the index file corresponding to the data file.

[0069] The storage rule information of the source cluster can include storage information of the data table on at least one first node of the source cluster. Since the data file and the index file on any first node are associatedly stored, the storage path of the index file is related to the storage path of the data file, the storage path of the index file belongs to the sub-path of the storage path of the data file, and when the data file is determined, the storage path of the index file can be determined through the storage path of the data file.

[0070] As shown in FIG. 1, an embodiment of a flowchart of a database migration method provided by the embodiment of the present application can include the following steps: Figure 2

[0071] 201: In response to a migration request for any database in a source cluster, determine the data table to be migrated of the database.

[0072] It should be noted that some steps in the embodiment of the present application are the same as some steps in the foregoing embodiments, and for the sake of brevity of description, they will not be described here.

[0073] 202: Obtain storage rule information of the data table to be migrated in the source cluster.

[0074] Optionally, the storage rule information can include data partition information corresponding to the data table in at least one first node of the source cluster.

[0075] The data table to be migrated is distributedly stored in the source cluster according to the data partition information corresponding to at least one first node respectively. The key-value pairs in the data file stored in any first node satisfy the data partition information corresponding to the first node.

[0076] 203: According to the storage rule information, determine the data file corresponding to at least one first node of the data table to be migrated in the source cluster respectively.

[0077] ​The storage rule information includes data partition information corresponding to each of the at least one first node. The data table can be partitioned into a plurality of data value ranges according to the value of the target key. For example, assuming that the data value range of the target key of the data table is 0-50000, the data value range can be divided into three data intervals, i.e., 0-20000, 20001-40000, and 40001-50000. Any data value range can be configured as the data partition information of a first node, and at least one data value range can be configured to a corresponding first node as the data partition information corresponding to each of the at least one first node. At this time, the data partition information of any first data node can be the data value range configured to the first node. Generally, any data value range corresponds to a first node, and each first node stores a data file satisfying the corresponding data value range. The data table is partitioned and stored in the at least one first node, and any first node stores a data file stored by the data table.

[0078] According to the storage rule information, the data file corresponding to each of the at least one first node of the data table to be migrated in the source cluster can include: querying the data file stored in each of the at least one first node according to the data partition information corresponding to each of the at least one first node of the data table to be migrated.

[0079] 204: Query the index file associated with the data file of any first node to obtain the data file and the index file corresponding to each of the at least one first node.

[0080] The associated storage relationship between the data table and the index table can include a path-dependent relationship between the data table and the index table. There is a dependent relationship between the storage path of the data file and the storage path of the index file in any first node. The storage path of the data file is the main path of the storage path of the index file, and the storage path of the index file is the sub-path of the storage path of the data file. Assuming that the storage path of the data file is A / B, the storage path of the index file can be the sub-path A / B / C of the storage path of the data file. And in actual application, the path master-slave rule of the data file and the index file is known, so when the storage path of the data file is determined, the storage path of the index file can be determined by using the storage path of the data file and the path master-slave rule. The master-slave rule can be a path mapping relationship between the storage path of the data file and the storage path of the index file, for example, in a Cassandra cluster, the storage path of the index file is a one-level sub-path of the storage path of the data file, and when the storage path of the data file is determined, the corresponding index file is directly searched in the next level storage path.

[0081] Optionally, the querying the index file associated with the data file of any first node stored in the first node can comprise: determining the first storage path of the data file of any first node in the first node; determining the second storage path corresponding to the index file stored in the sub-path of the first storage path by using the first storage path; and obtaining the index file associated with the data file of the first node according to the second storage path.

[0082] 205: setting a corresponding second node for each of the at least one first node in the target cluster.

[0083] Since the at least one first node is set in the source cluster, and the at least one first node respectively corresponds to the data partition information, in order to facilitate the accurate migration of the data file and to ensure that the data table can continue to be queried after the migration, the at least one first node respectively corresponds to the second node in the target cluster, so as to ensure that the at least one second node respectively saves the data file and the index file of the corresponding first node, realizes the mirror migration of the data table, and ensures that the data table after the migration can be effectively used.

[0084] 206: copying the data file and the index file corresponding to any first node to the second node corresponding to the first node, so as to copy the data file and the index file corresponding to the at least one first node to the second node corresponding to each first node.

[0085] Optionally, the copying the data file and the index file corresponding to any first node to the second node corresponding to the first node can comprise: reading the data file and the index file corresponding to any first node from the first node, and sending the data file and the index file of the first node to the second node corresponding to the first node. That is, the data file and the index file corresponding to any first node are forwarded to the second node corresponding to the first node.

[0086] In the embodiments of the present application, in response to a migration request for any database in the source cluster, the to-be-migrated data table of the database is determined. After obtaining the storage rule information of the to-be-migrated data table, the data files respectively corresponding to at least one first node of the source cluster can be determined according to the storage rule information. Then, the index files stored in association with the data files of any first node are queried to obtain the data files and the index files respectively corresponding to at least one first node. Corresponding second nodes are respectively set for the first nodes in the target cluster. Then, the data files and the index files respectively corresponding to any first node are associated and copied to the second nodes corresponding to the first nodes, and the copying of the data files and the index files respectively corresponding to at least one first node to the corresponding second nodes is completed. By respectively establishing corresponding second nodes for at least one first node in the source cluster in the target cluster, the node architecture of the target cluster is made the same as the node architecture of the source cluster. When migrating the to-be-migrated data table and the index table corresponding to the migrated data table, the data files and the index files stored in each node of the to-be-migrated data table and the associated index table are mirrored and migrated according to the node distribution architecture of the source cluster, so that the data table and the index table in the target cluster are completely migrated, so as to use the migrated data table according to the reason data table query mode, improve the migration efficiency of the data table, and ensure the use effectiveness of the data table.

[0087] In actual application, when the data table is partitioned and stored, data value ranges can be respectively set for at least one first node in the source cluster. The data value range refers to the value range of the data value of the target key. The target key can be any key in the data table.

[0088] The storage rule information can include data partition information respectively corresponding to at least one first node.

[0089] As an embodiment, respectively setting corresponding second nodes for at least one first node in the target cluster can include:

[0090] Respectively associating second nodes for at least one first node in the target cluster.

[0091] For any first node, the data partition information corresponding to the first node is set as the data partition information of the second node associated with the first node, and at least one second node respectively corresponding to at least one first node is obtained.

[0092] The data table can be stored according to the data partition information respectively corresponding to at least one first node. As a possible implementation, the method can further include:

[0093] According to the data partition information corresponding to each of the at least one first node, the data table is divided and stored in the at least one first node respectively, and data files corresponding to the at least one first node are obtained.

[0094] The data table is stored according to the data partition information corresponding to each of the at least one first node.

[0095] In the target cluster, a second node corresponding to each of the at least one first node is set, that is, the data partition information of the first node is mirrored to a second node, and the second node becomes the second node corresponding to the first node.

[0096] As a possible implementation, the data partition information corresponding to each of the at least one first node can be obtained by detecting configuration parameters of the data partition information corresponding to each of the at least one first node. In addition, the data partition information corresponding to each of the at least one first node can also be obtained by input of a management user. The data partition information corresponding to each of the at least one first node input by the management user can be detected.

[0097] In order to ensure that the second node corresponding to any first node can be accurately determined, the identifier of the first node can be used as the node identifier of the second node corresponding to the first node. In some embodiments, associating the second node with each of the at least one first node in the target cluster can include:

[0098] According to the first node identifier of each of the at least one first node, a mirror identifier corresponding to each of the at least one first node identifier is determined.

[0099] At least one configuration node is determined in the target cluster, and the number of the at least one configuration node is the same as the number of the at least one first node.

[0100] The at least one mirror identifier is set as the node identifier of the at least one to-be-configured node, and the at least one second node is obtained.

[0101] Based on the mirror identifier corresponding to each of the at least one first node identifier, the second node associated with each of the at least one first node is determined.

[0102] If the first node identifier of any first node corresponds to the mirror identifier of a second node, it is determined that the second node is the second node associated with the first node.

[0103] In the embodiments of the present application, the mirror identifier is established by using the first node identifier of any first node as the second node identifier corresponding to the first node, so that the second node corresponding to the first node can be quickly queried by using the first node identifier of the first node, the association migration of the data file and the index file is realized, the accuracy of data migration is improved, and the accurate migration of the data table is realized.

[0104] As an embodiment, the file types of the data file of any first node and the index file associated with the data file are the same. Taking Cassandra, the LTS database as an example, the data file and the index file are stored in the node in the SSTable file format. In order to solve the reading difficulty problem of the index file, the reading mode of the data file can be used to read the index file. As a possible implementation manner, after copying the data file and the index file corresponding to the at least one first node to the respective second nodes, the method can further include:

[0105] On any second node, the data file and the index file in the second node are loaded using a predetermined loading mode to load the respective data file of the at least one second node and complete the migration of the data table and the index table.

[0106] The data file and the index file have the same file format, and the same loading mode can be used for file loading. Taking the SSTable file format as an example, on any second node, the data file and the index file on the second node can be loaded using the loading mode corresponding to the file format.

[0107] In actual application, the storage rule information can include, in addition to the data partition information corresponding to the at least one first node in the source cluster, the storage path information of the data table corresponding to the at least one first node. When the data table to be migrated is determined, the storage location of the data table to be migrated corresponding to the at least one first node can be directly determined through the storage path information, and the data file corresponding to the at least one first node is further obtained. The index file associated with the data file stored in any first node is found through the path dependency relationship between the data file and the index file.

[0108] Different data tables can be stored in correspondence with respective index tables. For example, in actual application, the storage path of the data table can be used as the main path of the storage path of the index table, and the storage path of the index table can be a sub-path of the storage path of the corresponding data table. For example, assuming that the data table is stored in the storage path of E:\11222\11111, the index table corresponding to the data table can be stored in the storage path of E:\11222\11111\1111. When the storage path of the data table is determined, the storage path information of the index table can be determined according to the master-slave path relationship information of the data table and the index table, according to the storage path information of the data table, and the index table associated with the data table to be migrated is queried and obtained according to the storage path information of the index table.

[0109] When the respective data files of the at least one first node are known, in order to solve the technical problem that the data table corresponding index table is invisible, the migration of the database can be converted into the data files stored on each node and the corresponding migration of the data files.

[0110] Under the constraint of the storage rule, the copying manner of the data files and the index files can be determined.

[0111] In a possible design, the method further includes:

[0112] The storage path information of the at least one data file on the corresponding first node is determined.

[0113] The copying of the data file and the index file associated with the data file corresponding to any first node to the second node corresponding to the first node can specifically include:

[0114] For any data file and the index file associated with the data file, a copying instruction is generated.

[0115] In response to the copying instruction, the data file and the index file associated with the data file are copied to the second node corresponding to the first node.

[0116] According to the storage path information of any data file on the corresponding first node, the data file and the index file associated with the data file are stored in the second node.

[0117] Taking a Cassandra distributed database as an example, it is assumed that there are three nodes A / B / C in the source cluster, the data partition information corresponding to the A node is (a, b], the data partition information corresponding to the B node is (b, c], and the data partition information corresponding to the C node is (c, a]. The data files and the index files corresponding to the A node are (sstA1, sstA2, sstA3, indexA1, indexA2), the data files and the index files corresponding to the B node are (sstB1, sstB2, sstB3, indexB1), and the data files and the index files corresponding to the C node are (sstC1, sstC2, sstC3, indexC1, indexC2). Wherein, sst is a data file, and index is an index file.

[0118] Assuming that there are three nodes in the target cluster, and the node name corresponding replication is A' / B' / C', wherein A corresponds to A', B corresponds to B', and C corresponds to C'. The data partition information (a, b] corresponding to the A node can be set as the data partition information of A', the data partition information (b, c] corresponding to the B node can be set as the data partition information of B', and the data partition information (c, a] corresponding to the C node can be set as the data partition information of C'. Then, the data file and the index file can be associated and copied, that is, (sstA1, sstA2, sstA3) of the A node is copied to the A' node, (sstB1, sstB2, sstB3) of the B node is copied to the B' node, and (sstC1, sstC2, sstC3) of the C node is copied to the C' node. After the data file and the index file are associated and copied to the target cluster, the data and the index can be quickly copied, and the data file can continue to use the original index file after the copying, thereby effectively improving the migration efficiency of the database.

[0119] To facilitate understanding, the source cluster M1 is taken as an example, which is composed of three first nodes M11-M13, and the target cluster is taken as an example, which is composed of four second nodes M21-M24, and the technical scheme of the embodiment of the present application is introduced in detail.

[0120] In the embodiment of the present application, the source cluster can store a database, and the database actually stores data tables in the original cluster. Generally, the data tables are stored in the three first nodes according to the data partition information in the source cluster, and each first node can store a data file formed by a key-value pair within a data value range constraint. Figure 3 When the computing device M3 configured with the database migration method provided by the present application detects a migration request of the database, the computing device M3 can determine the data table to be migrated in response to the migration request. Since the data table to be migrated is stored in the source cluster according to at least one data partition information corresponding to each first node, the data files stored in the three first nodes of the source cluster can be determined. Further, according to the data files corresponding to the three first nodes, for example, the storage path information of the data files, the index files associated with the data files are obtained, so as to obtain the index files respectively associated with the data files of the three first nodes.

[0121] Assuming that the computing device M3 is in the target cluster, and the corresponding second nodes are M21-M23 for the three first nodes M11-M13 respectively. That is, the second node M21 in the target cluster corresponds to the first node M11 in the source cluster, the second node M22 corresponds to the first node M12, and the second node M23 corresponds to the first node M13. In order to ensure accurate reading of the data file and the index file, the data file of the first node M11 and the associated index file thereof can be migrated to the second node M21, the data file of the first node M12 and the associated index file thereof can be migrated to the second node M22, and the data file of the first node M13 and the associated index file thereof can be migrated to the second node M23. In this way, accurate migration of the data table and the associated index table is realized, and the migration efficiency of the data table is improved.

[0122] As shown in Figure 4 FIG. 1 is a flowchart of another embodiment of a database migration method provided by the embodiment of the present application. The method can include the following steps:

[0123] 401: In response to a database migration request initiated for a database migration interface, processing resources corresponding to the database migration interface are acquired.

[0124] The following steps are performed by using the processing resources corresponding to the database migration interface:

[0125] 402: In response to a migration request for any database in a source cluster, a data table corresponding to the database to be migrated is determined.

[0126] The migration request can be obtained by using the database migration interface.

[0127] 403: Storage rule information of the data table to be migrated in the source cluster is acquired.

[0128] 404: According to the storage rule information, an index table associated with the data table to be migrated in the source cluster is queried.

[0129] 405: The data table to be migrated and the index table are associated and copied to a target cluster.

[0130] In the embodiment of the present application, in response to a migration request for a data table in a source cluster, storage rule information of the source cluster can be acquired, so that an index table associated with the data table to be migrated in the source cluster is queried according to the storage rule information, and then the data table to be migrated and the index table are associated and copied to a target cluster. It is not necessary to establish an index directory for each key-value pair in the data table, and the migration efficiency of the database is improved.

[0131] The specific steps performed by the processing resources corresponding to the database migration interface in the embodiment of the present application are the same as those in the embodiment of the present application. Figures 1-3The database migration methods shown perform the same processing steps, and the specific implementation methods and technical effects of each technical feature have been described in [the document / document / etc.]. Figures 1-3 The embodiments shown are described in detail and will not be repeated here.

[0132] The database migration method described in this application embodiment is provided to the target user as a migration service. Specific service forms include web page services, software program services, or program modules existing in the form of SDKs (Software Development Kits) or APIs (Application Programming Interfaces). The program modules can provide corresponding recommended interfaces to provide services externally. Target users obtain the database migration service provided in this application embodiment by browsing information processing interface interfaces, service software interfaces, or software module interfaces.

[0133] like Figure 5 The diagram shown is a flowchart of another embodiment of a database migration apparatus provided in this application. The apparatus may include:

[0134] Request / Response Module 501: Used to respond to migration requests for any database in the source cluster and determine the corresponding data table to be migrated.

[0135] Rule Acquisition Module 502: Used to acquire the storage rule information of the data table to be migrated in the source cluster.

[0136] Index Query Module 503: Used to query the index table of the data table to be migrated in the associated storage of the source cluster based on the storage rule information.

[0137] Data migration module 504: Used to copy the data table to be migrated and the index table to the target cluster.

[0138] In this embodiment, in response to a migration request for a data table in the source cluster, the storage rule information of the source cluster can be obtained. Then, according to the storage rule information, the index table associated with the data table to be migrated in the source cluster is queried, and the data table to be migrated and its index table are copied to the target cluster. This eliminates the need to create index directories for each key-value pair in the data table, improving database migration efficiency.

[0139] As one embodiment, the index query module may include:

[0140] The file determination unit is used to determine the data files corresponding to at least one first node of the source cluster for the data table to be migrated, based on storage rule information.

[0141] The index query unit is configured to query the index file associated with the first node to obtain the data file corresponding to each of the at least one first node and the index file corresponding to each of the at least one first node.

[0142] The data migration module can include:

[0143] The node setting unit is configured to set the second node corresponding to each of the at least one first node in the target cluster.

[0144] The file migration unit is configured to copy the data file corresponding to each of the at least one first node and the index file corresponding to each of the at least one first node to the second node corresponding to each of the at least one first node, so as to copy the data file corresponding to each of the at least one first node and the index file corresponding to each of the at least one first node to the second node corresponding to each of the at least one first node.

[0145] In some embodiments, the storage rule information includes data partition information corresponding to each of the at least one first node.

[0146] The node setting unit can include:

[0147] The node association sub-unit is configured to associate the second node corresponding to each of the at least one first node in the target cluster.

[0148] The partition setting sub-unit is configured to set the data partition information of each of the at least one first node as the data partition information of the second node associated with each of the at least one first node, to obtain the second node corresponding to each of the at least one first node.

[0149] In a possible design, the apparatus can further include:

[0150] The distributed storage module is configured to divide the data table to be migrated to the at least one first node according to the data partition information corresponding to each of the at least one first node, to obtain the data file corresponding to each of the at least one first node.

[0151] As a possible implementation manner, the node association sub-unit can be specifically configured to:

[0152] determine the mirror identifier corresponding to each of the at least one first node identifier according to the first node identifier of each of the at least one first node; determine at least one to-be-configured node in the target cluster, the number of nodes of which is the same as that of the at least one first node; set the at least one mirror identifier as the node identifier of the at least one to-be-configured node, to obtain the at least one second node; and determine the second node associated with each of the at least one first node based on the mirror identifier corresponding to each of the at least one first node identifier.

[0153] As yet another embodiment, the data file of each of the at least one first node and the index file associated with the data file have the same file type. The apparatus can further include:

[0154] a file loading module, configured to load the data files and the index files in the at least one second node by using a predetermined data loading manner, so as to load the respective data files and the index files of the at least one second node, and complete the migration of the to-be-migrated data table and the index table.

[0155] In some embodiments, the apparatus can further include:

[0156] a path determining module, configured to determine the storage path information of the at least one data file in the corresponding first node respectively;

[0157] The file migration unit can be specifically configured to:

[0158] generate a copy instruction for any data file and the index file associated with the data file, copy the data file and the index file associated with the data file to the second node corresponding to the first node in response to the copy instruction, and store the data file and the corresponding index file in the second node according to the storage path information of the data file in the corresponding first node.

[0159] Figure 5 The database migration apparatus can perform Figure 1 the database migration method of any one of the above-mentioned embodiments, and the implementation principle and technical effects thereof will not be repeated. The specific manners of the steps performed by each module and unit in the above-mentioned embodiments have been described in detail in the embodiments related to the method, and will not be described in detail here.

[0160] In actual application, Figure 5 The database migration apparatus shown in the above-mentioned embodiments can be configured as a computing device, such as a server. Figure 6 The computing device can include a storage component 601 and a processing component 602. The storage component 601 can be configured to store one or more computer instructions, which can be invoked by the processing component 602 to execute the database migration method of any one of the above-mentioned embodiments. Figure 1 The database migration apparatus can also be configured to execute the database migration method of any one of the above-mentioned embodiments.

[0161] The processing component 602 can include one or more processors to execute the computer instructions to complete all or part of the steps in the above-mentioned method. Of course, the processing component can also be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements, which are configured to execute the above-mentioned database migration method.

[0162] The storage component 601 is configured to store various types of data to support the operation of the terminal. The storage component can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0163] Of course, the server can also include other components, such as an input / output interface, a communication component, etc. The input / output interface provides an interface between the processing component and peripheral interface modules, which can be output devices, input devices, etc. The communication component is configured to facilitate wired or wireless communication between the server and other devices, such as user equipment, etc.

[0164] As shown in Figure 7 An embodiment of a structure diagram of a database cluster provided by an embodiment of the present application is shown. The database cluster can include one or more database tables distributedly stored in at least one node 701.

[0165] The database processing component can be configured to: in response to a migration request for any database, determine the database corresponding to-be-migrated data table. Obtain the storage rule information of the to-be-migrated data table. According to the storage rule information, query the index table associated with the to-be-migrated data table. Copy the to-be-migrated data table and the index table associated therewith to the target cluster.

[0166] The specific implementation of each step in the database processing component in the embodiment of the present application is the same as that of each step in the database migration method in the embodiment shown in Figure 1 For the sake of brevity of description, it will not be repeated here.

[0167] Optionally, Figure 7 The database cluster in the embodiment shown can be the source cluster in the foregoing embodiments, and the target cluster can have the same structure as the database cluster shown in Figure 7 to achieve the mirror migration of the database. That is, the target cluster can have the same number of nodes as the at least one node 701 or more nodes than the at least one node 701.

[0168] As shown in Figure 8As shown in the figure, a structural schematic diagram of one embodiment of a database system provided in the embodiment is shown, and the database system can include a source cluster 801, a computing device 802, and a target cluster 803. The source cluster 801 and the target cluster 803 can perform data transmission or communication with the computing device 802. One or more databases each have a data table stored in the source cluster 801 in a distributed manner.

[0169] The computing device 802 can be configured to: in response to a migration request for any database in the source cluster 801, determine a to-be-migrated data table corresponding to the database; acquire storage rule information of the to-be-migrated data table in the source cluster 801; query an index table associated with the to-be-migrated data table in the source cluster 801 according to the storage rule information; and copy the to-be-migrated data table and the index table to the target cluster 803 in association.

[0170] Optionally, each step performed by the computing device 802 in the embodiment is the same as that in the embodiment shown in the figure, and for the sake of brevity of the description, details are not described herein. Figure 1

[0171] In addition, the embodiment of the present application further provides a computer readable storage medium, which can store one or more computer instructions. The computer instructions are executed to implement the database migration method in the above embodiments. Figure 1

[0172] The device embodiments described above are only schematic, and the units shown as separate components can be or can not be physically separate, and the components shown as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of general hardware platform as necessary, and of course can also be realized by means of combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of computer products, and the present application can be in the form of computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0174] ​​Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A database migration method, comprising: in response to a migration request for any database in a source cluster, determining that the database corresponds to a to-be-migrated data table; obtaining storage rule information of the to-be-migrated data table in the source cluster; according to the storage rule information, querying an index table associated with the to-be-migrated data table stored in the source cluster; copying the to-be-migrated data table and the index table to a target cluster in association; the step of querying the index table associated with the to-be-migrated data table stored in the source cluster according to the storage rule information comprises: determining an associated storage relationship between the to-be-migrated data table and the index table corresponding thereto according to the storage rule information, and determining the index table associated with the to-be-migrated data table according to the associated storage relationship; wherein the index table comprises an index directory. 2.The method of claim 1, wherein the step of querying the index table associated with the to-be-migrated data table stored in the source cluster according to the storage rule information comprises: determining data files corresponding to at least one first node of the source cluster respectively according to the storage rule information; querying an index file associated with the data file of any first node stored in the first node to obtain the data file and the index file corresponding to the at least one first node respectively; the step of copying the to-be-migrated data table and the index table to the target cluster in association comprises: setting a corresponding second node for the at least one first node in the target cluster respectively; copying the data file and the index file corresponding to any first node to the second node corresponding to the first node in association to copy the data file and the index file corresponding to the at least one first node to the corresponding second node respectively.

3. The method of claim 2, said storing rule information comprising: data partition information corresponding to the at least one first node respectively; the step of setting a corresponding second node for the at least one first node in the target cluster respectively comprises: associating a second node for the at least one first node in the target cluster respectively; setting the data partition information of any first node as the data partition information of the second node associated with the first node to obtain the second node corresponding to the at least one first node respectively. 4.The method of claim 3, further comprising: dividing the to-be-migrated data table to the at least one first node for storage respectively according to the data partition information corresponding to the at least one first node to obtain the data file corresponding to the at least one first node respectively. 5.The method of claim 3, wherein the step of associating a second node for the at least one first node in the target cluster respectively comprises: determining a mirror identifier corresponding to the at least one first node identifier respectively according to the first node identifier of the at least one first node; determining at least one to-be-configured node in the target cluster with the same number of nodes as the at least one first node; setting at least one mirror identifier as the node identifier of the at least one to-be-configured node to obtain the at least one second node. Determine the second nodes associated with the at least one first node respectively based on the mirror identifiers corresponding to the at least one first node identifier respectively.

6. The method of claim 2, wherein the data file of any one of the first nodes and the index file associated with the data file have the same file type. Further comprising: Load the data file and the index file in the second nodes using a predetermined data loading manner to load the respective data file and the index file of the at least one second node, thereby completing the migration of the to-be-migrated data table and the index table.

7. The method of claim 2, further comprising: Determine the storage path information of the at least one data file in the corresponding first node respectively; The copying of the data file and the index file associated with the data file of any one of the first nodes to the second node corresponding to the first node comprises: Generate a copy instruction for any data file and the index file associated with the data file; Copy the data file and the index file associated with the data file to the second node corresponding to the first node in response to the copy instruction; Store the data file and the corresponding index file of the data file in the second node according to the storage path information of the data file in the corresponding first node.

8. A database migration method, comprising: In response to a database migration request initiated through a database migration interface, obtain the processing resources corresponding to the database migration interface; Using the processing resources corresponding to the database migration interface, perform the following steps: In response to a migration request for any database in a source cluster, determine the to-be-migrated data table corresponding to the database; Obtain the storage rule information of the to-be-migrated data table in the source cluster; According to the storage rule information, query the index table associated with the to-be-migrated data table stored in the source cluster; Copy the to-be-migrated data table and the index table to a target cluster; According to the storage rule information, query the index table associated with the to-be-migrated data table stored in the source cluster comprises: according to the storage rule information, determine the association storage relationship between the to-be-migrated data table and its corresponding index table, and according to the association storage relationship, determine the index table associated with the to-be-migrated data table; wherein the index table comprises an index directory.

9. A database migration apparatus, comprising: A request response module configured to determine the to-be-migrated data table corresponding to any database in a source cluster in response to a migration request for the database; A rule obtaining module configured to obtain the storage rule information of the to-be-migrated data table in the source cluster; An index querying module configured to query the index table associated with the to-be-migrated data table stored in the source cluster according to the storage rule information; A data migration module configured to copy the to-be-migrated data table and the index table to a target cluster; The querying, according to the storage rule information, of the index table associated with the to-be-migrated data table and stored in the source cluster comprises: determining, according to the storage rule information, an associated storage relationship between the to-be-migrated data table and the index table corresponding to the to-be-migrated data table, and determining, according to the associated storage relationship, the index table associated with the to-be-migrated data table; the index table comprises an index directory.

10. A database cluster comprising: At least one node and a database processing component; Each data table of one or more databases is distributed and stored in the at least one node. The database processing component is configured to: In response to a migration request for any database, determine a to-be-migrated data table corresponding to the database, acquire storage rule information of the to-be-migrated data table, query an index table associated with the to-be-migrated data table according to the storage rule information, and associate and copy the to-be-migrated data table and the index table to a target cluster. The querying, according to the storage rule information, of the index table associated with the to-be-migrated data table and stored in the source cluster comprises: determining, according to the storage rule information, an associated storage relationship between the to-be-migrated data table and the index table corresponding to the to-be-migrated data table, and determining, according to the associated storage relationship, the index table associated with the to-be-migrated data table; the index table comprises an index directory.

Images