Method and device for processing data, medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: in response to a data migration instruction, synchronizing target data in a source database to a target database by performing asynchronous data processing; newly added data generated in the client is written into the source database and the target database, and database operation is carried out in the target database by adopting a first data processing strategy; after incremental data in the source database are all synchronized to the target database, asynchronous data processing is stopped; and writing newly added data generated in the client into the source database and the target database respectively, and performing database operation in the target database by adopting a second data processing strategy. According to the data migration method and device, the consistency and rollback performance of the data are ensured in the migration process, and smooth migration and rollback of the data are achieved under the condition that service is not interrupted.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium and computer program product for processing data. Background Art

[0002] Based on the solution of the prior art, when migrating data from a database, taking the MySQL database as an example, data migration usually includes two methods: physical migration and logical migration. Physical migration refers to data migration by copying a physical copy of the database file, while logical migration completes the migration process by performing data insertion, update, and deletion operations. In the heterogeneous data migration process of MySQL, the usual practice is to scan the entire table starting from the primary key, and then write the data to the new database after modifying the structure. When all data scans are completed, the service will be briefly interrupted so that the data source can be switched to a heterogeneous database such as a key-value storage system and the service can continue to be provided. This method will cause users to be unable to access the database during the MySQL service shutdown, and requires that the MySQL version and configuration must be consistent with the source end during the shutdown migration, which limits flexibility and thus affects the user experience. Summary of the invention

[0003] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for processing data.

[0004] In one aspect of the present application, a method for processing data is provided, wherein the method comprises:

[0005] In response to the data migration instruction, synchronizing the target data in the source database to the target database by performing asynchronous data processing;

[0006] The newly added data generated in the client is written into the source database and the target database respectively, and a first data processing strategy is used in the target database to perform database operations, wherein the first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix;

[0007] After all incremental data in the source database is synchronized to the target database, asynchronous data processing is stopped;

[0008] The newly added data generated in the client is written into the source database and the target database respectively, and a second data processing strategy is adopted in the target database to perform database operations, wherein the second data processing strategy makes the key name of the data in the target database no longer contain the target prefix.

[0009] In one aspect of the present application, a device for processing data is provided, wherein the device comprises:

[0010] means for synchronizing target data in a source database to a target database by performing asynchronous data processing in response to a data migration instruction;

[0011] A device for writing newly added data generated in a client into a source database and a target database respectively, and using a first data processing strategy in the target database to perform database operations, wherein the first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix;

[0012] A device for stopping asynchronous data processing after all incremental data in the source database is synchronized to the target database;

[0013] A device for writing newly added data generated in a client into a source database and a target database respectively, and adopting a second data processing strategy in the target database to perform database operations, wherein the second data processing strategy makes the key name of the data in the target database no longer contain a target prefix.

[0014] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiment of the application.

[0018] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the application.

[0019] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0020] The solution provided by the embodiment of the present application is that during the migration process, all data changes are written to the source database and the target database at the same time, so that both databases maintain the latest data status during the migration process, ensuring that users can access the latest data even during the migration process; by adopting the data processing strategy of the embodiment of the present application to manage and synchronize the data in the two database systems, the consistency and rollability of the data are ensured, and smooth data migration and rollback are achieved without interrupting the service. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0023] Figure 1 A schematic diagram of a process flow of a method for processing data provided by an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram showing an exemplary data migration process according to an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram showing the structure of a device for processing data provided by an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0027] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0030] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0031] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0032] Figure 1 A schematic flow chart of a method for processing data provided in an embodiment of the present application is shown. The method at least includes step S101, step S102, step S103 and step S104.

[0033] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0034] In the embodiment of the present application, in the process of gradually migrating data from a source database to a target database, the data processing strategy of the embodiment of the present application is adopted to manage and synchronize the data in the two database systems, thereby ensuring the consistency and rollability of the data and achieving smooth migration and rollback of the data without interrupting the service.

[0035] The source database and the target database of the embodiment of the present application are heterogeneous data. In some embodiments, the source data is a relational database, and the target database is a key-value (KV) database.

[0036] Reference Figure 1 In step S101, in response to a data migration instruction, target data in a source database is synchronized to a target database by performing asynchronous data processing.

[0037] The target data includes snapshot data corresponding to a specific time point and incremental data generated after the time point.

[0038] Specifically, step S101 includes step S1011 to step S1014.

[0039] In step S1011, the change data of the source database after the target time point is obtained.

[0040] The target time point is a specific time point used to mark the start of data synchronization.

[0041] Optionally, the change data is obtained by starting the CDC (Change Data Capture) service. CDC is a technology for capturing database changes. In MySQL, CDC can be implemented by parsing binlog (binary log). The binary log is used to record the change operations on the data and save it in binary form on the disk of the device.

[0042] For example, determine a time point time_1 and start the CDC service in MySQL before T1 to obtain incremental data in MySQL and send the data to Kafka for storage. By starting the CDC service before time_1, all data changes in MySQL starting from time_1 can be captured.

[0043] In step S1012, a snapshot is taken of the data in the source database at the target time point, and the snapshot data is sent to the message middleware for storage.

[0044] The message middleware includes but is not limited to Kafka. The core of Kafka is a distributed message queue that allows applications to publish and subscribe to message streams.

[0045] In step S1013, the asynchronous service is started, the snapshot data is read from the message middleware, and the read snapshot data is written into the target database.

[0046] In step S1014, after all the snapshot data are written to the target database, incremental data are read from the message middleware, and the read incremental data are written to the target database.

[0047] For the new data continuously generated by the client, the new data is written into the target database by executing the above steps S1011 to S1014. This process is continuous to ensure that the data in the target database is always up to date, thereby achieving real-time synchronization of data.

[0048] Continue to refer to Figure 1 To illustrate, in step S102, the newly added data generated in the client is written into the source database and the target database respectively, and the first data processing strategy is adopted in the target database to perform database operations.

[0049] The first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix.

[0050] According to one embodiment, the method in step S102 performs database operations in the source database and the target database in a dual-write and dual-read manner.

[0051] Specifically, through the dual-write strategy, all data changes during the migration process are written to the source database and the target database at the same time. Based on the dual-read strategy, when dual-reading is performed in the source database and the target database, the data reading result in the source database shall prevail. The purpose of reading data in the target database is to compare it with the data in the source database to determine whether the data migration is correct. Since the target database may contain some erroneous or inconsistent data in the initial stage of migration, in this stage, by comparing the data in the target database with the data in the source database, possible problems in the data migration process can be discovered in a timely manner.

[0052] The method may start executing the operation of S102 based on the operation instruction from the user to write the newly added data generated in the client into the source database and the target database at the same time. Alternatively, the operation of step S102 may be executed when a preset time point is reached.

[0053] According to one embodiment, based on the first data processing strategy, the following requirements are met when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database:

[0054] 1) When inserting new data, add the target prefix to the original key name inserted;

[0055] The target prefix is ​​used to identify data of a new format or a new version, which can facilitate subsequent data scanning and migration operations because the key to be processed can be quickly identified through the target prefix.

[0056] For example, suppose the original key name is "key", and the new key "prefix_key" is formed by adding the target prefix "prefix_".

[0057] 2) When updating data, if the key name corresponding to the data does not contain the target prefix, read the latest data corresponding to the key name from the source database and insert it into the target database with the key name after adding the target prefix; if the key name corresponding to the data contains the target prefix, directly update the data corresponding to the key name according to the latest change;

[0058] 3) When deleting data, the data whose key name contains the target prefix is ​​marked for deletion;

[0059] By marking the data for deletion instead of immediately deleting the data whose key name contains the target prefix from the target database, a buffer period is provided before the data is completely deleted so that the data can be restored when needed or data loss can be prevented during data synchronization.

[0060] 4) When reading data, if there is no key name containing the target prefix, the data is obtained in the target database based on the original key name, and the obtained data is compared with the corresponding data in the source database to determine whether the data migration is accurate.

[0061] For example, prefix_key and key are read in the target database. If prefix_key does not exist, the data corresponding to the key is obtained, and then the data is compared with the data in the source data to determine whether the migration is correct.

[0062] It should be noted that since the target database may contain some erroneous or inconsistent data in the initial stage of migration, the data read by the original key name in the source database and the target database may be inconsistent in this stage. As time goes by, the data in the target database will be gradually processed and consumed until the data in the target database is completely consistent with the source data. At this time, when performing double reading, double data verification can be performed.

[0063] In step S103, after all incremental data in the source database is synchronized to the target database, asynchronous data processing is stopped.

[0064] In the target database, the key names of some of the incremental data contain the target prefix, and the key names of the rest do not contain the target prefix.

[0065] In step S104, the newly added data generated in the client is written into the source database and the target database respectively, and the second data processing strategy is adopted in the target database to perform database operations.

[0066] The second data processing strategy ensures that the key name of the data in the target database no longer contains the target prefix.

[0067] The method may execute the operation of step S104 in response to an instruction to stop asynchronous data processing. Alternatively, the method may execute the operation of step S104 based on an operation instruction from a user. Alternatively, the method may execute the operation of step S104 when a preset time point is reached.

[0068] According to one embodiment, based on the second data processing strategy, the following requirements are met when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database:

[0069] 1) When inserting new data, use the original key name to insert;

[0070] 2) When updating data, if the key name corresponding to the data contains the target prefix, it is inserted into the target database with the original key name after the data is updated, and the key containing the target prefix is ​​deleted; if the key name corresponding to the data does not contain the target prefix, the data corresponding to the key name is directly updated;

[0071] 3) When deleting data, delete the original key and the key containing the target prefix;

[0072] 4) When reading data, the data is read based on the original key name. When performing dual read operations on the source database and the target database, the read result of the source database shall prevail.

[0073] Based on the embodiments of the present application, during the migration process, all data changes are written to the source database and the target database at the same time, so that both databases maintain the latest data status during the migration process, ensuring that users can access the latest data even during the migration process. Before stopping asynchronous data processing, the first data processing strategy is used to add a target prefix to the data key name of the target database, so that the newly written data to the target database can be distinguished from the original data. After gradually stopping asynchronous data processing, the second data processing strategy is used to delete the keys containing the target prefix in the target database and switch to data reading based on the original key. Since this process is carried out step by step, the service will not be suddenly interrupted.

[0074] According to one embodiment, the method further includes step S105.

[0075] In step S105, by scanning the target database based on the keys containing the target prefix, the data whose key names contain the target prefix remaining in the target database are deleted.

[0076] Specifically, for data whose key names are marked for deletion and contain the target prefix, delete the data and its corresponding key containing the target prefix; for other data whose key names contain the target prefix, read their key values ​​and modify them to the original keys, and delete the keys containing the target prefix.

[0077] According to one embodiment, the method further includes step S106 and step S107 after step S104.

[0078] In step S106, it is confirmed whether there is no new data operation request in the source database.

[0079] The method can determine whether there are no new data operation requests by monitoring the log of the application program, the query mode of the database, etc., that is, all data operation requests have been switched to the target database.

[0080] In step S107, if it is confirmed that there is no new data operation request in the source database, the data operation in the source database is stopped.

[0081] Once it is confirmed that the source database no longer needs to process any business requests, the operation of the source database can be safely stopped and the database service can be stopped to complete the data migration.

[0082] According to one embodiment, when the source database is not offline, if a problem occurs in the target database or data consistency verification is required, a rollback operation may be performed, and the method further includes step S108.

[0083] In step S108, in response to the data rollback instruction, a corresponding rollback operation is performed on the data that needs to be rolled back in the target database.

[0084] The rollback operation includes, but is not limited to, deleting the data that has been migrated to the target database or overwriting the data that needs to be rolled back with the corresponding data in the source database.

[0085] Since the embodiment of the present application writes data changes to the source database and the target database in a dual-write manner during the data migration process, there is no need to perform additional backup operations on the source database when rolling back the data.

[0086] According to the method of the embodiment of the present application, during the migration process, all data changes are written to the source database and the target database at the same time, so that both databases maintain the latest data status during the migration process, ensuring that users can access the latest data even during the migration process; by adopting the data processing strategy of the embodiment of the present application to manage and synchronize the data in the two database systems, the consistency and rollability of the data are ensured, and smooth data migration and rollback are achieved without interrupting the service.

[0087] The data migration process of the embodiment of the present application is explained below with reference to an example.

[0088] Reference Figure 2 The exemplary data migration scenario shown in FIG. 1 is a data migration scenario in which the source database in this example is MySQL and the target database is a KV database. In addition, this example implements data synchronization and update through the MySQL change data capture (CDC) service, the Kafka message queue, and the asynchronous service.

[0089] Below Figure 2 The names that appear in the description are:

[0090] client: client;

[0091] async: asynchronous synchronization of data operations;

[0092] kv: KV database, a simple key-value data storage system;

[0093] MySQL CDC: Change data capture is a way to back up the database, often used to back up large amounts of data;

[0094] Kafka: a distributed message queue system based on publish / subscribe mode;

[0095] curd: CURD represents the four basic actions of database operations. C stands for Create, which refers to the operation of inserting new data into the database; U stands for Update, which refers to the operation of modifying existing data in the database; R stands for Retrieve, which refers to the operation of reading data from the database; and D stands for Delete, which refers to the operation of deleting data from the database.

[0096] The data migration process in this example includes four stages. The following describes the process through steps P1 to P10, where steps P1 to P5 correspond to Figure 2 Steps P6 to P8 correspond to stage 3, and steps P9 and 10 correspond to stage 4.

[0097] Steps P1 to P10 are described below:

[0098] P1: Select a time T1, start the CDC service in MySQL to obtain incremental data in MySQL, and send the data to Kafka for storage.

[0099] Among them, time T1 is a specific time point used to mark the start of data synchronization. By starting the CDC service before T1, it is ensured that all data changes in MySQL from T1 onwards can be captured. Figure 2 As shown in the figure, the topic named incremental in Kafka is used to store incremental data.

[0100] P2: Take a snapshot of the MySQL data at time T1 and send the snapshot data to Kafka for storage. Figure 2 As shown in the figure, the topic named snapshot in Kafka is used to store snapshot data.

[0101] P3: Start the asynchronous service, read the snapshot data from Kafka, and write the read snapshot data to the KV database.

[0102] like Figure 2 As shown in the figure, by consuming the snapshot topic, the stored snapshot data is written to the KV database.

[0103] P4: After all snapshot data is written to the KV database, incremental data is read from Kafka and written to the KV database.

[0104] like Figure 2 As shown in the figure, after completing the snapshot topic consumption, the incremental topic messages are consumed to write the incremental data into the KV database.

[0105] P5: After a period of time, the client continuously generates new data, which is written into the KV database by executing the above steps P1 to P4. This process is continuous, ensuring that the data in the KV database is always up-to-date, thus achieving real-time synchronization of data.

[0106] P6 adopts a dual-write and dual-read strategy, writing new data in the client to both MySQL and the KV database, and reading data from both systems.

[0107] Through the dual-write strategy, during the migration process, all data changes are written to MySQL and KV databases at the same time. Based on the dual-read strategy, when dual-reading is performed in MySQL and KV databases, the data reading result of MySQL shall prevail. The purpose of reading data in the KV database is to compare it with the data in MySQL to determine whether the data migration is correct.

[0108] When performing database operations in the KV database, the following requirements must be met so that the data in the KV database at this stage is mainly v2_key data:

[0109] When inserting data, add the prefix "v2_" before the original key name "key" to form v2_key, which is convenient for subsequent scanning operations;

[0110] When updating data, if v2_key does not exist, read the latest value from TiDB (the distributed version of MySQL) and insert it into the KV database with v2_key; if v2_key already exists, directly update the v2_key data in the KV database;

[0111] When deleting data, mark v2_key for deletion instead of deleting it from the KV database immediately;

[0112] When reading data, v2_key and the original key are read at the same time. If v2_key does not exist, the data corresponding to the key is obtained, and then the data is compared with the data in MySQL to determine whether the migration is correct.

[0113] P7. After all MySQL incremental change data is synchronized to the KV database, stop the asynchronous service;

[0114] After a period of time, ensure that all MySQL incremental change data has been synchronized to the KV database and exists in the KV database in the form of key values ​​v2_key and key.

[0115] P8. Use a dual-write and dual-read strategy to write new data in the client into the MySQL and KV databases at the same time.

[0116] When performing database operations in a KV database, the following requirements must be met so that the key of the data in the KV database at this stage no longer contains v2_key:

[0117] When inserting data, use the original key key for insertion;

[0118] When updating data, if v2_key exists, rewrite it with key after the update and delete v2_key; if v2_key does not exist, directly update the key value;

[0119] When deleting data, delete v2_key and key value;

[0120] When reading data, the key value is read, but in the double-read process, the MySQL reading result is the final result.

[0121] P9: Delete the remaining v2_key data by scanning the v2_key in the KV data. For the v2_key data marked as deleted, delete the data and its corresponding key; for other data prefixed with v2_key, read its key value and modify it to key insertion, and delete v2_key.

[0122] P10: Dual write and dual read data in MySQL and KV databases. When necessary, data can be switched between MySQL and KV databases, or data can be rolled back.

[0123] After a period of time, confirm whether there are no new data operation requests in MySQL, that is, all data operation requests are switched to the KV database. Once it is confirmed that MySQL no longer needs to process any business requests, you can safely stop the operation of the MySQL database and stop the database service to complete the data migration.

[0124] The above method in this example allows data to be migrated from MySQL to the KV database without interrupting services, while ensuring data consistency and the ability to roll back when necessary.

[0125] In addition, the present application also provides a device for processing data, the structure of which is as follows: Figure 3 shown.

[0126] The device includes: a device for synchronizing target data in a source database to a target database by performing asynchronous data processing in response to a data migration instruction (hereinafter referred to as "asynchronous synchronization device 101"), a device for synchronizing target data in a source database to a target database by performing asynchronous data processing in response to a data migration instruction (hereinafter referred to as "first processing device 102"), a device for stopping asynchronous data processing after all incremental data in the source database is synchronized to the target database (hereinafter referred to as "asynchronous stopping device 103"), and a device for writing the newly added data generated in the client into the source database and the target database respectively, and adopting a second data processing strategy in the target database to perform database operations (hereinafter referred to as "second processing device 104").

[0127] Reference Figure 3 The asynchronous synchronization device 101 responds to the data migration instruction and synchronizes the target data in the source database to the target database by performing asynchronous data processing.

[0128] The target data includes snapshot data corresponding to a specific time point and incremental data generated after the time point.

[0129] Specifically, the asynchronous synchronization device 101 performs the following operations:

[0130] The asynchronous synchronization device 101 obtains the change data of the source database after the target time point.

[0131] The target time point is a specific time point used to mark the start of data synchronization.

[0132] Optionally, the asynchronous synchronization device 101 obtains the change data by starting the CDC (Change Data Capture) service. CDC is a technology for capturing database changes. In MySQL, CDC can be implemented by parsing binlog (binary log). The binary log is used to record the change operations on the data and save it in binary form on the disk of the device.

[0133] For example, determine a time point time_1 and start the CDC service in MySQL before T1 to obtain incremental data in MySQL and send the data to Kafka for storage. By starting the CDC service before time_1, all data changes in MySQL starting from time_1 can be captured.

[0134] Next, the asynchronous synchronization device 101 takes a snapshot of the data in the source database at the target time point, and sends the snapshot data to the message middleware for storage.

[0135] The message middleware includes but is not limited to Kafka. The core of Kafka is a distributed message queue that allows applications to publish and subscribe to message streams.

[0136] Next, the asynchronous synchronization device 101 starts the asynchronous service, reads the snapshot data from the message middleware, and writes the read snapshot data into the target database.

[0137] Next, after the asynchronous synchronization device 101 writes all the snapshot data into the target database, it reads the incremental data from the message middleware and writes the read incremental data into the target database.

[0138] For the new data continuously generated by the client, the asynchronous synchronization device 101 writes the new data into the target database by executing the above search sequence. This process is continuous to ensure that the data in the target database is always the latest, thereby achieving real-time synchronization of data.

[0139] Continue to refer to Figure 3 To illustrate, the first processing device 102 writes the newly added data generated in the client into the source database and the target database respectively, and adopts the first data processing strategy to perform database operations in the target database.

[0140] The first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix.

[0141] According to one embodiment, the first processing device 102 performs database operations in the source database and the target database in a dual-write and dual-read manner.

[0142] Specifically, through the dual-write strategy, all data changes during the migration process are written to the source database and the target database at the same time. Based on the dual-read strategy, when dual-reading is performed in the source database and the target database, the data reading result in the source database shall prevail. The purpose of reading data in the target database is to compare it with the data in the source database to determine whether the data migration is correct. Since the target database may contain some erroneous or inconsistent data in the initial stage of migration, in this stage, by comparing the data in the target database with the data in the source database, possible problems in the data migration process can be discovered in a timely manner.

[0143] The operation of the first processing device 102 may be started based on the operation instruction from the user to write the newly added data generated in the client into the source database and the target database at the same time. Alternatively, the operation of the first processing device 102 may be executed when a preset time point is reached.

[0144] According to an embodiment, the first processing device 102 satisfies the following requirements when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database based on the first data processing strategy:

[0145] 1) When inserting new data, add the target prefix to the original key name inserted;

[0146] The target prefix is ​​used to identify data of a new format or a new version, which can facilitate subsequent data scanning and migration operations because the key to be processed can be quickly identified through the target prefix.

[0147] For example, suppose the original key name is "key", and the new key "prefix_key" is formed by adding the target prefix "prefix_".

[0148] 2) When updating data, if the key name corresponding to the data does not contain the target prefix, read the latest data corresponding to the key name from the source database and insert it into the target database with the key name after adding the target prefix; if the key name corresponding to the data contains the target prefix, directly update the data corresponding to the key name according to the latest change;

[0149] 3) When deleting data, the data whose key name contains the target prefix is ​​marked for deletion;

[0150] By marking the data for deletion instead of immediately deleting the data whose key name contains the target prefix from the target database, a buffer period is provided before the data is completely deleted so that the data can be restored when needed or data loss can be prevented during data synchronization.

[0151] 4) When reading data, if there is no key name containing the target prefix, the data is obtained in the target database based on the original key name, and the obtained data is compared with the corresponding data in the source database to determine whether the data migration is accurate.

[0152] For example, prefix_key and key are read in the target database. If prefix_key does not exist, the data corresponding to the key is obtained, and then the data is compared with the data in the source data to determine whether the migration is correct.

[0153] It should be noted that since the target database may contain some erroneous or inconsistent data in the initial stage of migration, the data read by the original key name in the source database and the target database may be inconsistent in this stage. As time goes by, the data in the target database will be gradually processed and consumed until the data in the target database is completely consistent with the source data. At this time, when performing double reading, double data verification can be performed.

[0154] The asynchronous stopping device 103 stops asynchronous data processing after all incremental data in the source database are synchronized to the target database.

[0155] In the target database, the key names of some of the incremental data contain the target prefix, and the key names of the rest do not contain the target prefix.

[0156] The second processing device 104 writes the newly added data generated in the client into the source database and the target database respectively, and adopts the second data processing strategy to perform database operations in the target database.

[0157] The second data processing strategy ensures that the key name of the data in the target database no longer contains the target prefix.

[0158] The operation of the second processing device 104 may be executed in response to an instruction to stop asynchronous data processing. Alternatively, the operation of the second processing device 104 may be executed based on an operation instruction from a user. Alternatively, the operation of the second processing device 104 may be executed when a preset time point is reached.

[0159] According to one embodiment, based on the second data processing strategy, the following requirements are met when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database:

[0160] 1) When inserting new data, use the original key name to insert;

[0161] 2) When updating data, if the key name corresponding to the data contains the target prefix, it is inserted into the target database with the original key name after the data is updated, and the key containing the target prefix is ​​deleted; if the key name corresponding to the data does not contain the target prefix, the data corresponding to the key name is directly updated;

[0162] 3) When deleting data, delete the original key and the key containing the target prefix;

[0163] 4) When reading data, the data is read based on the original key name. When performing dual read operations on the source database and the target database, the read result of the source database shall prevail.

[0164] Based on the embodiments of the present application, during the migration process, all data changes are written to the source database and the target database at the same time, so that both databases maintain the latest data status during the migration process, ensuring that users can access the latest data even during the migration process. Before stopping asynchronous data processing, the first data processing strategy is used to add a target prefix to the data key name of the target database, so that the newly written data to the target database can be distinguished from the original data. After gradually stopping asynchronous data processing, the second data processing strategy is used to delete the keys containing the target prefix in the target database and switch to data reading based on the original key. Since this process is carried out step by step, the service will not be suddenly interrupted.

[0165] According to one embodiment, the device further comprises a residual deleting device.

[0166] The residual deleting device deletes the data whose key names contain the target prefix in the target database by scanning the target database based on the keys containing the target prefix.

[0167] Specifically, for data whose key names are marked for deletion and contain the target prefix, delete the data and its corresponding key containing the target prefix; for other data whose key names contain the target prefix, read their key values ​​and modify them to the original keys, and delete the keys containing the target prefix.

[0168] According to one embodiment, the device further comprises a request confirmation device and a data switching device.

[0169] The request confirmation device confirms whether there is no new data operation request in the source database.

[0170] The request confirmation device can determine whether there are no new data operation requests by monitoring the log of the application program, the query mode of the database, etc., that is, all data operation requests have been switched to the target database.

[0171] If it is confirmed that there is no new data operation request in the source database, the data switching device stops the data operation in the source database.

[0172] Once it is confirmed that the source database no longer needs to process any business requests, the operation of the source database can be safely stopped and the database service can be stopped to complete the data migration.

[0173] According to one embodiment, when the source database is not offline, if a problem occurs in the target database or data consistency verification is required, a rollback operation can be performed, and the device further includes a data rollback device.

[0174] The request confirmation device responds to the data rollback instruction and performs a corresponding rollback operation on the data that needs to be rolled back in the target database.

[0175] The rollback operation includes, but is not limited to, deleting the data that has been migrated to the target database or overwriting the data that needs to be rolled back with the corresponding data in the source database.

[0176] Since the embodiment of the present application writes data changes to the source database and the target database in a dual-write manner during the data migration process, there is no need to perform additional backup operations on the source database when rolling back the data.

[0177] According to the device of the embodiment of the present application, during the migration process, all data changes are written to the source database and the target database at the same time, so that both databases maintain the latest data status during the migration process, ensuring that users can access the latest data even during the migration process; by adopting the data processing strategy of the embodiment of the present application to manage and synchronize the data in the two database systems, the consistency and rollability of the data are ensured, and smooth data migration and rollback are achieved without interrupting the service.

[0178] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for processing data in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0179] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0180] Figure 4 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0181] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0182] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0183] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0184] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0185] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0186] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0187] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0188] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0195] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for processing data, wherein: The method comprises: In response to the data migration instruction, synchronizing the target data in the source database to the target database by performing asynchronous data processing; The newly added data generated in the client is written into the source database and the target database respectively, and a first data processing strategy is used in the target database to perform database operations, wherein the first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix; After all incremental data in the source database is synchronized to the target database, asynchronous data processing is stopped; The newly added data generated in the client is written into the source database and the target database respectively, and a second data processing strategy is adopted in the target database to perform database operations, wherein the second data processing strategy makes the key name of the data in the target database no longer contain the target prefix.

2. The method according to claim 1, wherein: Writing the newly added data generated in the client into the source database and the target database respectively comprises: Use dual-write and dual-read methods to perform database operations in the source and target databases; Among them, through the double-write strategy, all data changes in the migration process are written to the source database and the target database at the same time; based on the double-read strategy, when double-reading is performed in the source database and the target database, the data reading result in the source database shall prevail, and the purpose of reading data in the target database is to compare with the data in the source database to determine whether the data migration is correct.

3. The method according to claim 1, wherein: The method, based on the first data processing strategy, meets the following requirements when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database: When inserting new data, add the target prefix to the original key name inserted; When updating data, if the key name corresponding to the data does not contain the target prefix, the latest data corresponding to the key name is read from the source database and inserted into the target database with the key name after adding the target prefix; When deleting data, the data whose key name contains the target prefix will be marked for deletion; When reading data, if there is no key name containing the target prefix, the data is obtained in the target database based on the original key name, and the obtained data is compared with the corresponding data in the source database to determine whether the data migration is accurate.

4. The method according to claim 1, wherein: The method, based on the second data processing strategy, meets the following requirements when performing four database operations of inserting new data, updating data, reading data and deleting data in the target database: When inserting new data, use the original key name to insert; When updating data, if the key name corresponding to the data contains the target prefix, it will be inserted into the target database with the original key name after the data is updated, and the key containing the target prefix will be deleted; if the key name corresponding to the data does not contain the target prefix, the data corresponding to the key name will be directly updated; When deleting data, delete the original key and the key containing the target prefix; When reading data, the data is read based on the original key name. When performing dual read operations on the source database and the target database, the read result of the source database shall prevail.

5. The method according to claim 4, wherein: The method further comprises: By scanning the target database based on the key containing the target prefix, the remaining data in the target database whose key name contains the target prefix is ​​deleted.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Confirm whether there are no new data operation requests in the source database; If it is confirmed that there are no new data operation requests in the source database, stop the data operation in the source database.

7. The method according to any one of claims 1 to 5, wherein: The method further comprises: In response to the data rollback instruction, a corresponding rollback operation is performed on the data that needs to be rolled back in the target database.

8. The method according to claim 1, wherein: The step of synchronizing the target data in the source database to the target database by performing asynchronous data processing includes: Obtain the change data of the source database after the target time point; Take a snapshot of the source database data at the target time point and send the snapshot data to the message middleware for storage; Start the asynchronous service, read the snapshot data from the message middleware, and write the read snapshot data to the target database; After all snapshot data is written to the target database, incremental data is read from the message middleware and written to the target database.

9. A device for processing data, wherein: The device comprises: means for synchronizing target data in a source database to a target database by performing asynchronous data processing in response to a data migration instruction; A device for writing newly added data generated in a client into a source database and a target database respectively, and using a first data processing strategy in the target database to perform database operations, wherein the first data processing strategy performs tagging processing on the key name of the newly added data in the target database so that the key name of the newly added data in the target database has a target prefix; A device for stopping asynchronous data processing after all incremental data in the source database is synchronized to the target database; A device for writing newly added data generated in a client into a source database and a target database respectively, and adopting a second data processing strategy in the target database to perform database operations, wherein the second data processing strategy makes the key name of the data in the target database no longer contain a target prefix.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be used by a processor to execute the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images