Data migration method and apparatus

Abstract

The application discloses a data migration method and device, and relates to the technical field of big data, wherein the method comprises the following steps: exporting first business data in a Cassandra database in a JSON format; reading the first business data in the JSON format to obtain a data type prefix, an environment variable, a data content type, a business primary key, an addressing target value, an update time and accessory information of the first business data; generating a key in a TiKV database according to the data type prefix, the environment variable, the data content type and the business primary key, and taking the addressing target value, the update time and the accessory information as values in the TiKV database to complete migration of the first business data; and migrating second business data in the Cassandra database to the TiKV database according to preconfigured flow conditions and a plurality of preset read-write states. The application can improve the data migration efficiency.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to data migration methods and apparatus. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Cassandra, as a distributed database, offers adjustable consistency levels, ultimately achieving eventual consistency at the corresponding level. However, some of its characteristics and performance are not well-suited for internal system construction during routine maintenance and disaster recovery. TiKV, as a distributed key-value database, also provides high availability and high scalability while meeting the requirements of business systems. Existing solutions for migrating data from Cassandra to TiKV require shutting down online services and then using existing tools to export and import data between the two databases separately. Due to the large data volume, exporting and importing are time-consuming operations, and online services cannot tolerate prolonged downtime. If there is no downtime, online services still have write requests that need to be processed, and this incremental data also needs to be migrated. To support online data migration, additional tools are needed to handle incremental data in the existing export and import processes. Existing data migration methods are inefficient and cumbersome. Summary of the Invention

[0004] This invention provides a data migration method to achieve efficient data migration without system downtime, improve data migration efficiency, and simplify the data migration process. The method includes:

[0005] Export the first business data from the Cassandra database using JSON format. The first business data includes the business data written to the Cassandra database before the preset export time.

[0006] Read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information of the first business data;

[0007] Generate keys in the TiKV database based on data type prefixes, environment variables, data content types, and business primary keys. Use the target address value, update time, and supplementary information as values ​​in the TiKV database to complete the first business data migration.

[0008] Based on pre-configured transfer conditions and various preset read / write states, the second business data in the Cassandra database is migrated to the TiKV database; the second business data includes business data written to the Cassandra database after a preset export time.

[0009] This invention also provides a data migration apparatus for achieving efficient data migration without system downtime, improving data migration efficiency, and simplifying the data migration process. The apparatus includes:

[0010] The first processing module is used to export the first business data from the Cassandra database in JSON format, wherein the first business data includes business data written to the Cassandra database before a preset export time.

[0011] The second processing module is used to read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information of the first business data;

[0012] The third processing module is used to generate keys in the TiKV database based on data type prefixes, environment variables, data content types, and business primary keys, and to use the target address value, update time, and auxiliary information as values ​​in the TiKV database to complete the first business data migration.

[0013] The fourth processing module is used to migrate the second business data in the Cassandra database to the TiKV database according to the pre-configured transfer conditions and various preset read and write states; wherein the second business data includes business data written to the Cassandra database after a preset export time.

[0014] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data migration method.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data migration method.

[0016] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described data migration method.

[0017] In this embodiment of the invention, first business data in a Cassandra database is exported using JSON format. This first business data includes business data written to the Cassandra database before a preset export time. The first business data in JSON format is read to obtain its data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information. A key in a TiKV database is generated based on the data type prefix, environment variables, data content type, and business primary key. The address target value, update time, and auxiliary information are used as values ​​in the TiKV database to complete the migration of the first business data. Second business data in the Cassandra database is migrated to the TiKV database according to pre-configured transfer conditions and various preset read / write states. This second business data includes business data written to the Cassandra database after a preset export time. This allows for efficient data migration without system downtime, improving migration efficiency and simplifying the data migration process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a flowchart of a data migration method provided in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of a method for exporting first business data from a Cassandra database using JSON format, provided in an embodiment of the present invention.

[0021] Figure 3 This is a flowchart of a method for migrating second business data from a Cassandra database to a TiKV database based on pre-configured transfer conditions and multiple preset read / write states, as provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of a data migration device provided in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0025] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0026] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] In the description of this specification, the terms "comprising," "including," "having," and "containing" are open-ended terms, meaning that they include but are not limited to. The terms "an embodiment," "a specific embodiment," "some embodiments," and "for example," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and can be adjusted appropriately as needed.

[0028] Research has revealed that Cassandra, as a distributed database, offers adjustable consistency levels, ultimately achieving eventual consistency. However, some of its characteristics and performance are not well-suited for internal system construction during routine maintenance and disaster recovery. TiKV, as a distributed key-value database, also provides high availability and scalability while meeting business system requirements. Existing solutions for migrating data from Cassandra to TiKV require shutting down online services and then using existing tools to export and import data between the two databases. Due to the large data volume, exporting and importing are time-consuming operations, and online services cannot tolerate prolonged downtime. Even without shutdown, online services have write requests that need processing, and this incremental data also needs to be migrated. To support online data migration, additional tools are needed to handle incremental data processing within the existing export and import processes. Existing data migration methods are inefficient and cumbersome.

[0029] Regarding the above research, such as Figure 1 As shown, an embodiment of the present invention provides a data migration method, including:

[0030] S101: Export the first business data from the Cassandra database in JSON format, wherein the first business data includes the business data written to the Cassandra database before the preset export time;

[0031] S102: Read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information of the first business data;

[0032] S103: Generate a key in the TiKV database based on the data type prefix, environment variable, data content type, and business primary key; use the address target value, update time, and auxiliary information as the value in the TiKV database to complete the first business data migration.

[0033] S104: Based on the pre-configured transfer conditions and various preset read / write states, migrate the second business data in the Cassandra database to the TiKV database; wherein, the second business data includes business data written to the Cassandra database after the preset export time.

[0034] In this embodiment of the invention, first business data in a Cassandra database is exported using JSON format. This first business data includes business data written to the Cassandra database before a preset export time. The first business data in JSON format is read to obtain its data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information. A key in a TiKV database is generated based on the data type prefix, environment variables, data content type, and business primary key. The address target value, update time, and auxiliary information are used as values ​​in the TiKV database to complete the migration of the first business data. Second business data in the Cassandra database is migrated to the TiKV database according to pre-configured transfer conditions and various preset read / write states. This second business data includes business data written to the Cassandra database after a preset export time. This allows for efficient data migration without system downtime, improving migration efficiency and simplifying the data migration process.

[0035] The above data migration methods are explained in detail below.

[0036] For the above S101, the first business data includes, for example, the existing data in the Cassandra database, that is, the data migration that begins at the preset export time. The first business data includes the business data written into the Cassandra database before the preset export time.

[0037] Specifically, when exporting the first business data from the Cassandra database in JSON format, existing data export tools can be used, which will not be elaborated here.

[0038] Furthermore, during the initial data export process, export failures may occur. To improve data migration efficiency at any step, such as... Figure 2 The diagram shown is a flowchart of a method for exporting first business data from a Cassandra database using JSON format, according to an embodiment of the present invention. The method includes:

[0039] S201: When the export of the first business data in the Cassandra database fails, extract the scope of the first business data content that failed to be exported from the error log;

[0040] S202: Generate a new export script based on the scope of the first business data content that failed to be exported;

[0041] S203: Execute a new export script to export the first range of business data content that failed.

[0042] For S1012 to S103 above, the data type prefix, environment variables, data content type, business primary key, addressing target value, update time, and auxiliary information are read from the first business data in JSON format; the key in the TiKV database is generated based on the data type prefix, environment variables, data content type, and business primary key; and the value in the TiKV database is used based on the addressing target value, update time, and auxiliary information to complete the migration of the first business data.

[0043] In the TiKV database, data is stored in a key-value format. Therefore, after generating the corresponding key and value for the first business data in the TiKV database, the migration of the first business data can be completed.

[0044] For the above S104, the circulation conditions include, for example, the first circulation condition, the second circulation condition, the third circulation condition, the fourth circulation condition, the fifth circulation condition, the sixth circulation condition, and the seventh circulation condition.

[0045] The transition conditions are as follows: The first transition condition is that startup is successful based on the startup process log (e.g., checking the startup process logs of all applications requiring library connections; those with correct statuses are considered successful). The second transition condition is that the TiKV database cluster startup is error-free based on the startup process log. The third transition condition is that the TiKV database cluster startup is abnormal based on the startup process log. The fourth transition condition is that the consumer module has started, the consumer queue is not backed up, and the TiKV database cluster switchover log is correct based on the startup process log. The fifth transition condition is that the consumer module has started, the consumer queue is not backed up, and the TiKV database cluster switchover log is abnormal based on the startup process log. The sixth transition condition is that the consumer module has started, the consumer queue is not backed up, the switchover log is correct based on the startup process log, and the dual-source database status statistics log shows no abnormalities. The seventh transition condition is that the consumer module has started, the consumer queue is not backed up, the switchover log is correct based on the startup process log, and the dual-source database status statistics log shows abnormalities.

[0046] The consumer module refers to the functional module used to read messages from the queue that need to be written to the corresponding database, and write them to the corresponding database according to the specific operation type (such as insert, update, delete).

[0047] In addition, the preset read / write states include, for example: first preset read / write state, second preset read / write state, third preset read / write state, fourth preset read / write state, fifth preset read / write state, and sixth preset read / write state.

[0048] The first preset read / write state includes: configured single write, dual-database startup switch off; the second preset read / write state includes: configured single write, dual-database startup switch on, dual-read mode configured for read-only source database; or, configured dual write to write source database first, dual-database startup switch on, dual-read mode configured for read-only source database; the third preset read / write state includes: configured single write, dual-database startup switch on, dual-read mode configured for read source database first; or, configured dual write to write source database first, dual-read startup switch on, dual-read mode configured for read source database first, consumer module started; the fourth preset read / write state includes: the state where the third and fifth preset read / write states coexist; the fifth preset read / write state includes: configured single write, dual-database startup switch on, dual-read mode configured for read target database first; or, configured dual write to write target database first, dual-database startup switch on, dual-read mode configured for read target database first, consumer module started; the sixth preset read / write state includes: configured single write, dual-read startup switch off, consumer module not started.

[0049] In the migration operation, the source database includes a Cassandra database, and the target database includes a TiKV database.

[0050] The single-write, single-read startup described in this embodiment of the invention includes: connecting to only a single database, with both reads and writes accessing only the single database. Dual-database startup includes: connecting to both databases. Database write scenarios include: single-write and asynchronous dual-write via a queue. Database read scenarios include: reading only the source database; reading the source database first, then the target database; reading the target database first, then the source database; and reading only the target database.

[0051] like Figure 3 The diagram shows a flowchart of a method for migrating second business data from a Cassandra database to a TiKV database according to pre-configured transfer conditions and multiple preset read / write states, provided by an embodiment of the present invention. The method includes:

[0052] S301: Initially execute the first preset read / write state, and switch to the second preset read / write state when the first flow condition is met.

[0053] In one embodiment of the present invention, the initial execution of a first preset read / write state, and the switching to a second preset read / write state when a first transition condition is met, includes: initially executing the first preset read / write state; when the first transition condition is met, configuring dual write for applications requiring dual writes and enabling queue writing; turning on the dual-database startup switch, reading only the source database, restarting the effective client or switching the effective client; checking the status of all nodes in the logs; and keeping the consumer module closed.

[0054] S302: During the execution of the second preset read / write state, when the second flow condition is met, switch to the third preset read / write state.

[0055] In one embodiment of the present invention, during the execution of the second preset read / write state, when the second transition condition is met, switching to the third preset read / write state includes: during the execution of the second preset read / write state, when the second transition condition is met, exporting the second business data from the Cassandra database; importing the second business data into the TiKV database; verifying the second business data imported into the TiKV database; switching to dual-read mode, restarting the effective client or switching the effective client; and starting the consumer module.

[0056] S303: During the execution of the second preset read / write state, when the third transition condition is met, switch to the first preset read / write state.

[0057] In one embodiment of the present invention, during the execution of the second preset read / write state, when a third transition condition is met, switching to the first preset read / write state includes: during the execution of the second preset read / write state, when the third transition condition is met, closing queue writing; turning off the dual read start switch, restarting the effective client or switching the effective client; checking the status of all nodes in the log; and keeping the consumer module closed.

[0058] S304: During the execution of the third preset read / write state, when the fourth transition condition is met, switch to the fifth preset read / write state, wherein the fourth preset read / write state exists during the process of switching from the third preset read / write state to the fifth preset read / write state.

[0059] In one embodiment of the present invention, during the execution of the third preset read / write state, when the fourth transition condition is met, the system switches to the fifth preset read / write state, including: during the execution of the third preset read / write state, when the fourth transition condition is met, configuring the dual-write-first-write target library, and switching the read mode to dual-library startup-first-read target library.

[0060] S305: During the execution of the third preset read / write state, when the fifth transition condition is met, switch to the first preset read / write state.

[0061] In one embodiment of the present invention, during the execution of the third preset read / write state, when the fifth transition condition is met, switching to the first preset read / write state includes: during the execution of the third preset read / write state, when the fifth transition condition is met, closing queue writing; turning off the dual read start switch, restarting the effective client or switching the effective client; checking the status of all nodes in the log; and closing the consumer module.

[0062] S306: During the execution of the fifth preset read / write state, when the sixth flow condition is met, switch to the sixth preset read / write state.

[0063] In one embodiment of the present invention, during the execution of the fifth preset read / write state, when the sixth transition condition is met, switching to the sixth preset read / write state includes: during the execution of the fifth preset read / write state, when the sixth transition condition is met, starting the single read / single write mode, checking the log state switch; restarting the application connecting to the Cassandra database; and checking the log state and resource connection status.

[0064] S307: During the execution of the fifth preset read / write state, when the seventh flow condition is met, export all business data from the Cassandra database, use the inspection tool to filter out the second business data and import it into the TiKV database. After the dual source database data is repaired, switch to the sixth preset read / write state.

[0065] S308: During the execution of the fifth preset read / write state, if the seventh transition condition is met, the dual read source database data cannot be repaired, and the system switches to the first preset read / write state.

[0066] In another embodiment of the present invention, during incremental data migration, synchronous dual-write can be performed directly, without relying on a queue. During full data import, records are not written if they exist. This method requires separate marking for deletion operations to prevent deleted data from reappearing during full import.

[0067] In one embodiment of the present invention, when the seventh transition condition is met during the execution of the fifth preset read / write state, the dual-read source database data cannot be repaired, and the system switches to the first preset read / write state. This includes: when the seventh transition condition is met during the execution of the fifth preset read / write state, closing queue writing; turning off the dual-read start switch; restarting the effective client or switching the effective client; checking the status of all nodes in the log; and closing the consumer module.

[0068] This invention also provides a data migration apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the data migration method, the implementation of this apparatus can be referred to the implementation of the data migration method, and repeated details will not be elaborated further.

[0069] like Figure 4 The diagram shown is a schematic representation of a data migration apparatus provided in an embodiment of the present invention, comprising:

[0070] The first processing module 401 is used to export the first business data in the Cassandra database in JSON format, wherein the first business data includes business data written into the Cassandra database before a preset export time.

[0071] The second processing module 402 is used to read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information of the first business data;

[0072] The third processing module 403 is used to generate keys in the TiKV database based on data type prefix, environment variables, data content type, and business primary key, and to use the address target value, update time, and auxiliary information as values ​​in the TiKV database to complete the first business data migration.

[0073] The fourth processing module 404 is used to migrate the second business data in the Cassandra database to the TiKV database according to the pre-configured transfer conditions and various preset read and write states; wherein, the second business data includes business data written to the Cassandra database after a preset export time.

[0074] In one possible implementation, the first processing module is specifically configured to, when the export of the first business data in the Cassandra database fails, extract the range of the first business data content that failed to be exported from the error log; generate a new export script based on the range of the first business data content that failed to be exported; and execute the new export script to export the range of the first business data content that failed to be exported.

[0075] In one possible implementation, there are a first transition condition, a second transition condition, a third transition condition, a fourth transition condition, a fifth transition condition, a sixth transition condition, and a seventh transition condition; wherein, the first transition condition includes: confirming successful startup based on the startup process log; the second transition condition includes: confirming no abnormalities in the TiKV database cluster startup based on the startup process log; the third transition condition includes: confirming an abnormal startup of the TiKV database cluster based on the startup process log; the fourth transition condition includes: the consumer module has started, there is no backlog in the consumer queue, and the TiKV database cluster switchover log is correct based on the startup process log; the fifth transition condition includes: the consumer module has started, there is no backlog in the consumer queue, and the TiKV database cluster switchover log is abnormal based on the startup process log; the sixth transition condition includes: the consumer module has started, there is no backlog in the consumer queue, the switchover log is correct based on the startup process log, and the dual-read source database status statistics log shows no abnormalities; the seventh transition condition includes: the consumer module has started, there is no backlog in the consumer queue, the switchover log is correct based on the startup process log, and the dual-read source database status statistics log shows abnormalities.

[0076] In one possible implementation, the preset read / write states include: a first preset read / write state, a second preset read / write state, a third preset read / write state, a fourth preset read / write state, a fifth preset read / write state, and a sixth preset read / write state; wherein, the first preset read / write state includes: configuring single write, with the dual-database startup switch off; the second preset read / write state includes: configuring single write, with the dual-database startup switch on, and configuring the dual-read mode as a read-only source database; or, configuring dual write to first write the source database, with the dual-database startup switch on, and configuring the dual-read mode as a read-only source database; the third preset read / write state includes: configuring single write, with the dual-database startup switch on, and dual read... The mode is configured to read the source library first; or, configured for dual write to read the source library first, dual read start switch is enabled, dual read mode is configured to read the source library first, and the consumer module starts; the fourth preset read / write state includes: the state where the third preset read / write state and the fifth preset read / write state coexist; the fifth preset read / write state includes: configured for single write, dual library start switch is enabled, dual read mode is configured to read the target library first; or, configured for dual write to read the target library first, dual library start switch is enabled, dual read mode is configured to read the target library first, and the consumer module starts; the sixth preset read / write state includes: configured for single write, dual read start switch is disabled, and the consumer module does not start.

[0077] In one possible implementation, the fourth processing module is specifically configured to initially execute a first preset read / write state, and switch to a second preset read / write state when a first transition condition is met; during the execution of the second preset read / write state, switch to a third preset read / write state when a second transition condition is met; during the execution of the second preset read / write state, switch to the first preset read / write state when a third transition condition is met; during the execution of the third preset read / write state, switch to a fifth preset read / write state when a fourth transition condition is met, wherein a fourth preset read / write state exists during the transition from the third preset read / write state to the fifth preset read / write state; during the execution of the third preset read / write state... During the execution of the preset read / write state, if the fifth transition condition is met, switch to the first preset read / write state; during the execution of the fifth preset read / write state, if the sixth transition condition is met, switch to the sixth preset read / write state; during the execution of the fifth preset read / write state, if the seventh transition condition is met, export all business data from the Cassandra database, use an inspection tool to filter out the second business data and import it into the TiKV database, and after the dual source database data is repaired, switch to the sixth preset read / write state; during the execution of the fifth preset read / write state, if the seventh transition condition is met but the dual source database data cannot be repaired, switch back to the first preset read / write state.

[0078] In one possible implementation, the fourth processing module is specifically used to initially execute the first preset read / write state. When the first transition condition is met, for applications that require dual writing, dual writing is configured and queue writing is enabled; the dual-database startup switch is turned on, the source database is read-only, and the effective client is restarted or the effective client is switched; the log checks the status of all nodes; and the consumer module remains closed.

[0079] In one possible implementation, the fourth processing module is specifically used to, during the execution of the second preset read / write state, export the second business data from the Cassandra database when the second flow condition is met; import the second business data into the TiKV database; verify the second business data imported into the TiKV database; switch the dual-read mode and restart or switch the effective client; and start the consumer module.

[0080] In one possible implementation, the fourth processing module is specifically used to, during the execution of the second preset read / write state, close queue writing when the third transition condition is met; turn off the dual read start switch, restart the effective client or switch the effective client; check the status of all nodes in the log; and keep the consumer module closed.

[0081] In one possible implementation, the fourth processing module is specifically used to configure the dual-write target library first and switch the read mode to dual-library startup - read target library first when the fourth flow condition is met during the execution of the third preset read-write state.

[0082] In one possible implementation, the fourth processing module is specifically used to, during the execution of the third preset read / write state, close queue writing when the fifth transition condition is met; close the dual read start switch, restart the effective client or switch the effective client; check the status of all nodes in the log; and close the consumer module.

[0083] In one possible implementation, the fourth processing module is specifically used to, during the execution of the fifth preset read / write state, start the single read / write mode when the sixth transition condition is met, check the log state switch, restart the application connecting to the Cassandra database, and check the log state and resource connection status.

[0084] In one possible implementation, the fourth processing module is specifically used to, during the execution of the fifth preset read / write state, close queue writing when the seventh transition condition is met; close the dual read start switch, restart the effective client or switch the effective client; check the status of all nodes in the log; and close the consumer module.

[0085] Based on the aforementioned inventive concept, such as Figure 5 As shown, the present invention also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the aforementioned data migration method.

[0086] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data migration method.

[0087] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described data migration method.

[0088] In this embodiment of the invention, first business data in a Cassandra database is exported using JSON format. This first business data includes business data written to the Cassandra database before a preset export time. The first business data in JSON format is read to obtain its data type prefix, environment variables, data content type, business primary key, address target value, update time, and auxiliary information. A key in a TiKV database is generated based on the data type prefix, environment variables, data content type, and business primary key. The address target value, update time, and auxiliary information are used as values ​​in the TiKV database to complete the migration of the first business data. Second business data in the Cassandra database is migrated to the TiKV database according to pre-configured transfer conditions and various preset read / write states. This second business data includes business data written to the Cassandra database after a preset export time. This allows for efficient data migration without system downtime, improving migration efficiency and simplifying the data migration process.

[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that, include: Export the first business data from the Cassandra database using JSON format. The first business data includes the business data written to the Cassandra database before the preset export time. Read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time and auxiliary information of the first business data; The keys in the TiKV database are generated based on the data type prefix, environment variables, data content type, and business primary key. The values ​​in the TiKV database are then used based on the address target value, update time, and auxiliary information to complete the first business data migration. Based on pre-configured migration conditions and multiple preset read / write states, the second business data in the Cassandra database is migrated to the TiKV database. The multiple preset read / write states refer to the read / write modes of the Cassandra and TiKV databases. The second business data includes business data written to the Cassandra database after a preset export time. The migration conditions include: a first migration condition, a second migration condition, a third migration condition, a fourth migration condition, a fifth migration condition, a sixth migration condition, and a seventh migration condition. The first migration condition includes: confirming successful startup based on the startup process log; the second migration condition includes: confirming no anomalies in the TiKV database cluster startup based on the startup process log; the third migration condition... The process includes the following conditions: 1) Determining an abnormal startup of the TiKV database cluster based on the startup process log; 2) The consumer module has started, the consumer queue has no backlog, and the TiKV database cluster switchover log is correct based on the startup process log; 3) The consumer module has started, the consumer queue has no backlog, and the TiKV database cluster switchover log is abnormal based on the startup process log; 4) The consumer module has started, the consumer queue has no backlog, the switchover log is correct based on the startup process log, and the dual-read source database status statistics log shows no abnormalities; 5) The consumer module has started, the consumer queue has no backlog, the switchover log is correct based on the startup process log, and the dual-read source database status statistics log shows no abnormalities.

2. The data migration method as described in claim 1, characterized in that, Export the first business data from the Cassandra database in JSON format, including: When the export of the first business data in the Cassandra database fails, extract the scope of the first business data content that failed to be exported from the error log. Generate a new export script based on the scope of the first business data that failed to export; Execute the new export script to export the first range of business data content that failed.

3. The data migration method as described in claim 1, characterized in that, The preset read / write states include: First Preset Read / Write State, Second Preset Read / Write State, Third Preset Read / Write State, Fourth Preset Read / Write State, Fifth Preset Read / Write State, and Sixth Preset Read / Write State; The First Preset Read / Write State includes: Configured for single write, dual-database startup switch off; The Second Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-only source database; or, Configured for dual write to write to source database first, dual-database startup switch on, dual-read mode configured for read-only source database; The Third Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-first source database; or, Configured for dual write to write to source database first, dual-read startup switch on, dual-read mode configured for read-first source database, consumer module started; The Fourth Preset Read / Write State includes: The Third Preset Read / Write State and the Fifth Preset Read / Write State coexist; The Fifth Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-first target database; Alternatively, configure dual write to write to the target library first, enable dual library startup switch, configure dual read mode to read the target library first, and start the consumer module; the sixth preset read / write state includes: configure single write, disable dual read startup switch, and do not start the consumer module.

4. The data migration method as described in claim 3, characterized in that, Based on pre-configured transfer conditions and various preset read / write states, the second business data in the Cassandra database is migrated to the TiKV database, including: Initially, the system executes the first preset read / write state. When the first transition condition is met, it switches to the second preset read / write state. During the execution of the second preset read / write state, when the second transition condition is met, switch to the third preset read / write state; During the execution of the second preset read / write state, when the third transition condition is met, switch to the first preset read / write state; During the execution of the third preset read / write state, when the fourth transition condition is met, the process switches to the fifth preset read / write state. The fourth preset read / write state exists during the process of switching from the third preset read / write state to the fifth preset read / write state. During the execution of the third preset read / write state, when the fifth transition condition is met, switch to the first preset read / write state; During the execution of the fifth preset read / write state, when the sixth transition condition is met, switch to the sixth preset read / write state; During the execution of the fifth preset read / write state, when the seventh flow condition is met, all business data in the Cassandra database is exported. Using the inspection tool, the second business data is filtered out and imported into the TiKV database. After the dual source database data is repaired, the system switches to the sixth preset read / write state. During the execution of the fifth preset read / write state, if the seventh transition condition is met, the dual source database data cannot be repaired, and the system switches to the first preset read / write state.

5. The data migration method as described in claim 4, characterized in that, Initially, the system executes a first preset read / write state. Upon meeting a first transition condition, it switches to a second preset read / write state, including: Initially, the first preset read / write state is executed. When the first transition condition is met, dual writing is configured for applications that require dual writing, and queue writing is enabled. With the dual-repository startup switch enabled, the source repository is read-only. Restart the client that takes effect or switch to the client that takes effect. Log checks of all node statuses; The consumer module remains closed.

6. The data migration method as described in claim 4, characterized in that, During the execution of the second preset read / write state, when the second transition condition is met, the system switches to the third preset read / write state, including: During the execution of the second preset read / write state, when the second flow condition is met, the second business data in the Cassandra database is exported. Import the second business data into the TiKV database; Verify the second business data imported into the TiKV database; Switch to dual-read mode, restart the client that is now active, or switch to a different client that is now active. The consumer module is started.

7. The data migration method as described in claim 4, characterized in that, During the execution of the second preset read / write state, when the third transition condition is met, the system switches to the first preset read / write state, including: During the execution of the second preset read / write state, when the third transition condition is met, queue writing is closed; The dual-read startup switch is off. Restart the client that is active or switch to a different client. Log checks of all node statuses; The consumer module remains closed.

8. The data migration method as described in claim 4, characterized in that, During the execution of the third preset read / write state, when the fourth transition condition is met, the system switches to the fifth preset read / write state, including: During the execution of the third preset read / write state, when the fourth flow condition is met, the dual-write target library is configured to be written first, and the read mode is switched to dual-library startup - read target library first.

9. The data migration method as described in claim 4, characterized in that, During the execution of the third preset read / write state, when the fifth transition condition is met, the system switches to the first preset read / write state, including: During the execution of the third preset read / write state, when the fifth transition condition is met, queue writing is closed; The dual-read startup switch is off. Restart the client that is active or switch to a different client. Log checks of all node statuses; The consumer module is closed.

10. The data migration method as described in claim 4, characterized in that, During the execution of the fifth preset read / write state, when the sixth transition condition is met, the system switches to the sixth preset read / write state, including: During the execution of the fifth preset read / write state, when the sixth transition condition is met, the single read / write mode is started, and the log state is checked after switching. Restart the application that is connecting to the Cassandra database; Check the log status and resource connection status.

11. The data migration method as described in claim 4, characterized in that, During the execution of the fifth preset read / write state, if the seventh transition condition is met, the data in the dual-source database cannot be repaired, and the system switches to the first preset read / write state, including: During the execution of the fifth preset read / write state, when the seventh transition condition is met, queue writing is closed; The dual-read startup switch is off. Restart the client that is active or switch to a different client. Log checks of all node statuses; The consumer module is closed.

12. A data migration device, characterized in that, include: The first processing module is used to export the first business data from the Cassandra database in JSON format, wherein the first business data includes business data written to the Cassandra database before a preset export time. The second processing module is used to read the first business data in JSON format and obtain the data type prefix, environment variables, data content type, business primary key, address target value, update time and auxiliary information of the first business data; The third processing module is used to generate keys in the TiKV database based on data type prefixes, environment variables, data content types, and business primary keys, and to use the address target value, update time, and auxiliary information as values ​​in the TiKV database to complete the first business data migration. The fourth processing module is used to migrate the second business data from the Cassandra database to the TiKV database according to pre-configured migration conditions and multiple preset read / write states; the multiple preset read / write states are the read / write modes of the Cassandra database and the TiKV database; wherein, the second business data includes business data written to the Cassandra database after a preset export time; the migration conditions include: a first migration condition, a second migration condition, a third migration condition, a fourth migration condition, a fifth migration condition, a sixth migration condition, and a seventh migration condition; wherein, the first migration condition includes: confirming successful startup based on the startup process log; the second migration condition includes: confirming that the TiKV database cluster startup is without anomalies based on the startup process log; the third... The transition conditions are as follows: First, the TiKV database cluster startup is confirmed to be abnormal based on the startup process logs. Second, the consumer module has started, the consumer queue has no backlog, and the TiKV database cluster switchover log is confirmed to be correct based on the startup process logs. Third, the consumer module has started, the consumer queue has no backlog, and the TiKV database cluster switchover log is confirmed to be abnormal based on the startup process logs. Fourth, the consumer module has started, the consumer queue has no backlog, the switchover log is confirmed to be correct based on the startup process logs, and the dual-read source database status statistics logs show no abnormalities. Fifth, the consumer module has started, the consumer queue has no backlog, the switchover log is confirmed to be correct based on the startup process logs, and the dual-read source database status statistics logs show no abnormalities. Sixth, the consumer module has started, the consumer queue has no backlog, the switchover log is confirmed to be correct based on the startup process logs, and the dual-read source database status statistics logs show no abnormalities.

13. The data migration apparatus as described in claim 12, characterized in that, The first processing module is specifically used to extract the scope of the first business data that failed to be exported from the error log when the first business data export fails; and to generate a new export script based on the scope of the first business data that failed to be exported. Execute the new export script to export the first range of business data content that failed.

14. The data migration apparatus as described in claim 12, characterized in that, The preset read / write states include: First Preset Read / Write State, Second Preset Read / Write State, Third Preset Read / Write State, Fourth Preset Read / Write State, Fifth Preset Read / Write State, and Sixth Preset Read / Write State; The First Preset Read / Write State includes: Configured for single write, dual-database startup switch off; The Second Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-only source database; or, Configured for dual write to write to source database first, dual-database startup switch on, dual-read mode configured for read-only source database; The Third Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-first source database; or, Configured for dual write to write to source database first, dual-read startup switch on, dual-read mode configured for read-first source database, consumer module started; The Fourth Preset Read / Write State includes: The Third Preset Read / Write State and the Fifth Preset Read / Write State coexist; The Fifth Preset Read / Write State includes: Configured for single write, dual-database startup switch on, dual-read mode configured for read-first target database; Alternatively, configure dual write to write to the target library first, enable dual library startup switch, configure dual read mode to read the target library first, and start the consumer module; the sixth preset read / write state includes: configure single write, disable dual read startup switch, and do not start the consumer module.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

Images