Data migration method and device, storage medium, electronic equipment and program product

Abstract

The invention discloses a data migration method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of cloud computing. The method comprises the steps that data in a source data system is scanned according to a scanning strategy, the scanning strategy comprises the scanning duration, the scanning frequency, the scanning range and the scanning partition of the scanning operation, and the source data system is a distributed memory key value storage system. And converting data obtained after the scanning operation into target data meeting the storage format of the target database. And migrating the target data to the target database by adopting an asynchronous mode according to the concurrent write-in operation state of the target database. According to the data migration method and device, the technical problem that in the prior art, if the stock data lacks the database interface input parameter data, the stock data cannot be migrated to the database, migration can be completed only after the database interface input parameter data is determined again, and the data migration efficiency is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, in particular to a data migration method and device, a storage medium, an electronic device and a program product. BACKGROUND

[0002] Data migration is a key link in the process of upgrading, optimizing and integrating enterprise information systems, and its importance is increasingly prominent. The purpose of data migration is to efficiently and accurately transfer data stored in the original database to a new database system to meet the changing business requirements and technical upgrade requirements.

[0003] At present, a common data migration method is to convert the input parameters of the calling source interface to the input parameters of the calling target interface, and then write the data again through the calling interface to achieve data migration. This method can meet the basic needs of data migration to some extent, but has significant limitations in actual application.

[0004] Specifically, the prior art relies on the input data of the inventory write interface. If the interface input data is missing, the data cannot be migrated through the above method. This makes the method only applicable to application scenarios that save transaction input data. However, in many actual situations, the interface input information of the inventory data may not be available due to various reasons, such as data loss, system upgrade, historical data cleaning, etc. In this case, to complete data migration, the database interface input data needs to be determined again, which not only increases the complexity of data migration, but also may result in low data migration efficiency, or even unable to complete the migration task within a reasonable time.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] The embodiments of the present application provide a data migration method, device, storage medium, electronic device and program product to at least solve the technical problem of low data migration efficiency caused by the need to determine the database interface input data again after the inventory data is missing the database interface input data in the prior art. The present application does not rely on the database interface input data of the inventory transaction data, but directly scans the stored data of the source data system, and then converts to the database for storage through an asynchronous manner.

[0007] According to an aspect of the embodiments of the present application, a data migration method is provided, comprising: performing a scanning operation on data in a source data system according to a scanning strategy, wherein the scanning strategy comprises determining a scanning time length, a scanning frequency, a scanning range and a scanning partition of the scanning operation, and the source data system is a distributed in-memory key-value storage system; converting the data obtained after the scanning operation into target data satisfying a storage format of a target database, wherein the target database is a key-value storage database; and migrating the target data to the target database in an asynchronous manner according to a concurrent write operation state of the target database.

[0008] Optionally, during the process of performing the scanning operation on the data in the source data system according to the scanning strategy, the data migration method further comprises: detecting a transaction data amount of the source data system in a target time period, wherein the target time period is any time period; and determining an adjustment mode of the scanning operation according to the transaction data amount of the source data system in the target time period.

[0009] Optionally, determining the adjustment mode of the scanning operation according to the transaction data amount of the source data system in the target time period comprises: reducing the scanning frequency of the scanning operation and / or reducing the scanning range of the scanning operation in a case where it is detected that the transaction data amount of the source data system in the target time period is greater than or equal to a first threshold value; increasing the scanning frequency of the scanning operation and / or expanding the scanning range of the scanning operation in a case where it is detected that the transaction amount of the source data system in the target time period is less than or equal to a second threshold value, wherein the second threshold value is less than the first threshold value; and maintaining the scanning frequency and the scanning range of the scanning operation unchanged in a case where it is detected that the transaction data amount of the source data system in the target time period is greater than the second threshold value and less than the first threshold value.

[0010] Optionally, in a case where it is detected that the transaction data amount of the source data system in the target time period is greater than or equal to the first threshold value, if it is detected that the source data system comprises a primary database and a standby database, the scanning range of the scanning operation is reduced to the standby database.

[0011] Optionally, during the process of performing the scanning operation on the data in the source data system according to the scanning strategy, the data migration method further comprises: dividing a database of the source data system into N partitions, wherein N is an integer greater than 1; and performing a round-robin scanning operation on the N partitions, wherein the round-robin scanning operation is used to restrict each scanning operation to scan one partition, and N continuous scanning operations correspond to one round of data scanning of the N partitions respectively.

[0012] Optionally, the target data is migrated to the target database in an asynchronous manner according to a concurrent write operation state of the target database, including: determining a migration time of the target data according to the concurrent write operation state of the target database, wherein a concurrent write data amount of the target database at the migration time is lower than a concurrent write data amount of the target database at a non-migration time.

[0013] Optionally, the target data is migrated to the target database in an asynchronous manner according to a concurrent write operation state of the target database, including: if it is detected that the target data is data supporting repeated writing, determining a data conversion rule between the target data and a data write parameter of the target database, and migrating the target data to the target database in an asynchronous manner according to the data conversion rule and a data write interface of the target database; if it is detected that the target data is data not supporting repeated writing, calling a data write function of the target database, and migrating the target data to the target database in an asynchronous manner according to the data write function.

[0014] Optionally, in the process of performing the scanning operation on the data in the source data system according to the scanning strategy, data sampling is performed on the scanned data; the success rate of the scanning operation is detected according to a result of the data sampling; performance information of the source data system in the process of the data sampling is collected; and the scanning strategy is adjusted according to the success rate of the scanning operation and the performance information of the source data system in the process of the data sampling.

[0015] According to another aspect of the embodiments of the present application, a data migration apparatus is further provided, including: a scanning unit, configured to perform a scanning operation on data in a source data system according to a scanning strategy, wherein the scanning strategy includes determining a scanning time length, a scanning frequency, a scanning range and a scanning partition of the scanning operation, and the source data system is a distributed in-memory key-value storage system; a conversion unit, configured to convert data obtained after the scanning operation into target data satisfying a storage format of a target database, wherein the target database is a key-value storage database; and a migration unit, configured to migrate the target data to the target database in an asynchronous manner according to a concurrent write operation state of the target database.

[0016] According to another aspect of the embodiments of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores a computer program, wherein when the computer program runs, the computer readable storage medium causes a device where the computer readable storage medium is located to perform the data migration method.

[0017] According to a further aspect of the embodiments of the present application, an electronic device is provided, which includes one or more processors and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data migration method described above.

[0018] According to a further aspect of the embodiments of the present application, a computer program product is provided, which optionally includes a computer program or instructions, which, when executed by a processor, implement the data migration method described above.

[0019] In the embodiments of the present application, first, data in a source data system is scanned according to a scanning strategy, wherein the scanning strategy includes determining a scanning time length, a scanning frequency, a scanning range and a scanning partition of the scanning operation, and the source data system is a distributed in-memory key-value storage system; then, data obtained after the scanning operation is converted into target data satisfying a storage format of a target database, wherein the target database is a key-value storage database; and finally, the target data is migrated to the target database in an asynchronous manner according to a concurrent write operation state of the target database.

[0020] From the above, it can be seen that the technical solution of the present application solves the migration difficulty problem caused by missing interface parameter data in the prior art by directly scanning the stored data in the source data system and performing format conversion, without relying on the interface parameter data of the inventory data. Specifically, first, the source data system (distributed in-memory key-value storage system) is scanned according to a scanning strategy, and the scanning strategy includes a scanning time length, a frequency, a range and a partition, which makes the scanning process more flexible and has less impact on the source system. Then, the scanned data is converted into a storage format required by the target database (key-value storage database), ensuring that the data can be correctly stored and accessed by the target database. Finally, the data is migrated to the target database in an asynchronous manner according to the concurrent write operation state of the target database, which not only improves the migration efficiency but also reduces the impact on the performance of the target database.

[0021] Specifically, the present application directly scans the stored data in the source data system without the need to re-determine the interface parameter data, reducing the preparation work before migration and reducing the risk of migration failure caused by missing interface parameter data. Moreover, the present application can quickly complete the data migration task by optimizing the scanning strategy and using an asynchronous migration manner, while reducing the impact on the performance of the source database and the target database. The asynchronous migration manner allows the data migration to be completed gradually in the background, and even if part of the data migration fails, it will not affect the entire migration process, and the failed task can be re-tried later, ensuring the integrity of the data migration.

[0022] In summary, the technical solution not only solves the migration difficulty caused by missing interface input parameter data in the prior art, but also significantly improves the overall efficiency and reliability of data migration by improving migration efficiency, reducing the impact on database performance, enhancing migration reliability, and optimizing resource utilization. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0024] Figure 1 is a flowchart of an optional data migration method according to an embodiment of the application;

[0025] Figure 2 is a schematic diagram of an optional data migration method according to an embodiment of the application;

[0026] Figure 3 is a schematic diagram of an optional data migration device according to an embodiment of the application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected by the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards in the relevant region, necessary security measures are taken, it does not violate public order and good customs, and provides corresponding operation portal for user to choose authorization or refusal. For example, an interface is provided between the system and the related users or institutions, and before obtaining the relevant information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the relevant information is obtained.

[0030] According to an embodiment of the present application, an embodiment of a data migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] According to an embodiment of the present application, a data migration system can be used as an execution subject of the cross-system processing method of transaction data of the embodiment of the present application, wherein the system can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiment of the present application can also be other forms of execution subjects, such as devices and equipment. Those skilled in the art should know that the specific form of the execution subject of the present application is not particularly limited.

[0032] Figure 1 is a flowchart of an optional data migration method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0033] Step S101, performing a scanning operation on the data in the source data system according to a scanning strategy.

[0034] The scanning strategy includes determining the scanning time, scanning frequency, scanning range and scanning partition of the scanning operation, and the source data system is a distributed in-memory key-value storage system. The scanning strategy can refer to a series of rules and parameters followed when scanning the data in the source data system, which can include scanning time, scanning frequency, scanning range and scanning partition. By reasonably setting the scanning strategy through scanning time, scanning frequency, scanning range and scanning partition, the scanning process can be optimized, the impact on the performance of the source database can be reduced, and the comprehensiveness and accuracy of data scanning can be improved. The source data system can refer to a database system that stores raw data.

[0035] Optionally, the scan duration can refer to the length of time that a single scan operation lasts. By controlling the execution time of the scan operation, the excessive pressure on the source system caused by a long scan process can be avoided.

[0036] Optionally, the scan frequency can refer to the number of scan operations performed per unit time. By adjusting the scan frequency, the performance impact on the source system in a high-concurrency scenario can be reduced while ensuring the timeliness of the data.

[0037] Optionally, the scan range can refer to the data range covered by each scan operation, such as a specific set of key-value pairs or a data partition. By limiting the scan range, the performance problems caused by full-scan can be avoided, and the targeting of the scan can be improved.

[0038] Optionally, the scan partition can refer to dividing the data in the source data system into multiple partitions, and each scan operation is only performed on one of the partitions. Partition scanning can distribute the scan tasks and reduce the pressure of a single scan, while facilitating the monitoring and management of the scan process.

[0039] In step S102, the data obtained after the scan operation is converted into target data that meets the storage format of the target database, wherein the target database is a key-value storage database.

[0040] In step S102, the data obtained after the scan operation can be the original data obtained from the source data system through the scan strategy. The target database storage format can refer to the data storage format required by the target database. For example, the target database is a key-value storage database, which usually requires data to be stored in the form of key-value pairs, and the key is used to quickly retrieve the corresponding value, with high read-write performance, suitable for scenarios that require fast access and update of data. The target data can be data that meets the storage format of the target database after conversion, and the conversion process can include format adjustment, type conversion, encoding optimization, etc.

[0041] Optionally, the data migration system can convert the scanned data into the storage format required by the target database. This conversion process enables correct storage and access of data in the target database, avoiding migration failure due to format mismatch. By directly converting the data format, the complex process of re-determining the interface parameter data is also avoided, not only reducing the preparation work before migration, but also reducing errors caused by inconsistent or missing interface parameter data, improving the migration efficiency.

[0042] In step S103, according to the concurrent write operation state of the target database, the target data is migrated to the target database in an asynchronous manner.

[0043] In step S103, the concurrent write operation status of the target database may refer to the number and frequency of write operations being performed by the target database at a certain moment. The concurrent write operation status reflects the current load situation of the target database and may include the number of write tasks being executed, the write speed, and the response time of the system. By monitoring the concurrent write operation status, the real-time load situation of the target database can be understood, so as to reasonably arrange the timing and method of data migration to avoid excessive performance pressure on the target database. The asynchronous method may mean that during the data migration process, the reading and writing operations of the data are not performed synchronously, but are decoupled through queues, message middleware, etc., allowing the data to be migrated step by step in the background. The asynchronous method can disperse the data migration tasks to the background and complete them step by step, avoiding blocking the normal operation of the source database and reducing the impact on the performance of the target database.

[0044] Migrating target data to the target database asynchronously is independent of interface input data. Even if interface input data is missing, data migration can still be completed through asynchronous writes, improving data migration reliability and reducing the risk of migration interruption due to single migration task failures. By monitoring the status of concurrent write operations on the target database, the timing and speed of data migration can be dynamically adjusted based on the target database's real-time load. For example, when the target database load is low, the data migration speed can be accelerated, while when the load is high, the migration speed can be slowed down to avoid negative impacts on the target database's performance and ensure the normal operation of the target database during the migration process.

[0045] Figure 2 is a schematic diagram of an optional data migration method according to an embodiment of the present application, such as Figure 2 As shown in the figure, for example, system A is first scanned to retrieve stored data one by one. If the program writing to system A has been offline, the scanning program can be started directly. However, if the program writing to system A needs to run synchronously with the program writing to system B for a period of time, dynamic variables are needed to control the scanning time, frequency, scope, clusters, and partitions when scanning system A to minimize the performance impact on the normal writing operation of system A.

[0046] For example, if peak trading hours occur between 8:00 AM and 6:00 PM, you can schedule asynchronous scans at night to minimize the impact on container performance and prevent disruption to high-concurrency writers. Additionally, you can choose to scan only the backup cluster, rather than the primary cluster. This reduces the performance impact on System A servers and limits potential impact to the backup cluster. Even if the backup cluster fails, this will not affect high-concurrency writers from writing data to the primary cluster.

[0047] At night, the scanning frequency can not be limited, but during the day, the scanning frequency should be limited through interval sleep and other methods, so as to leave as many CPU resources of the A system as possible for normal high-concurrency write operations, and ensure that high-timeliness real-time services are not affected. Due to a large amount of stored data, full scanning may take more than a day. In order to avoid the problem of scanning interruption or failure due to too long scanning time, local scanning can be performed through partitioning and range division. During the scanning process, the scanning data information can be sampled and printed for analysis of the scanning frequency and data location. At the same time, monitoring is started to check the performance of the container where the program is located and the A system server. If it is found that the CPU load is too heavy, the scanning frequency can be reduced through dynamic variables.

[0048] For each piece of data obtained by scanning, if it can be conveniently converted into a write parameter required by the B system storage, it is directly converted and reused for asynchronous writing through the existing B system transaction interface. If the condition is not met, it is directly converted into the format of the B system storage data, and the write method of the B system is called for asynchronous writing.

[0049] For data that can be repeatedly written, repeated scanning and writing can be used to ensure integrity and avoid migration omission. For data that cannot be repeatedly written, it can be extracted separately, and through multiple local scans, it is ensured that the data is successfully written at one time as much as possible, and the migration data error is reduced. If scanning failure occurs, the transaction time of the last piece of scanning data can be located through the log, and the data after the transaction time is migrated and written in the next scanning.

[0050] In an optional embodiment, during the scanning operation of the data in the source data system according to the scanning strategy, the data migration method further comprises: detecting, by the data migration system, the transaction data amount of the source data system in a target time period, wherein the target time period is any time period, and determining an adjustment mode of the scanning operation according to the transaction data amount of the source data system in the target time period.

[0051] Optionally, the data migration system can monitor the transaction data amount of the source data system in real time within a preset target time period. The transaction data amount refers to the number of transaction operations processed by the source data system in the time period, which includes but is not limited to: reading, writing, updating, etc. of data. By monitoring the transaction data amount, the data migration system can obtain the load condition of the source data system in the time period.

[0052] Optionally, based on the monitored transaction data volume, the data migration system can dynamically adjust the scanning operation strategy. For example, if the transaction data volume is high, it indicates that the source data system is in a high load state, at which time the scanning frequency can be reduced, the scanning range can be narrowed, or the scanning operation can be postponed to a period of low load to reduce the performance pressure on the source system. Conversely, if the transaction data volume is low, the source data system is in a low load state, then the scanning frequency can be increased, the scanning range can be expanded, so as to complete the data migration task faster.

[0053] It should be noted that the advantage of this dynamic adjustment mechanism is that the data migration system can flexibly optimize the scanning operation according to the real-time load of the source data system, ensuring that the data migration is completed efficiently without affecting the normal operation of the source data system. Compared with the traditional fixed scanning strategy, this method is more intelligent and flexible, and can better adapt to the dynamic changes of the source data system, thereby improving the overall efficiency and reliability of data migration.

[0054] In an optional embodiment, the adjustment manner of the scanning operation is determined according to the transaction data volume of the source data system in a target time period, including: in the case that it is detected that the transaction data volume of the source data system in the target time period is greater than or equal to a first threshold, the data migration system can reduce the scanning frequency of the scanning operation and / or narrow the scanning range of the scanning operation; in the case that it is detected that the transaction volume of the source data system in the target time period is less than or equal to a second threshold, the data migration system can increase the scanning frequency of the scanning operation and / or expand the scanning range of the scanning operation, wherein the second threshold is less than the first threshold; in the case that it is detected that the transaction data volume of the source data system in the target time period is greater than the second threshold and less than the first threshold, the data migration system maintains the scanning frequency and scanning range of the scanning operation unchanged.

[0055] Optionally, according to the technical solution of the present application, the data migration system can set two thresholds, which are a first threshold and a second threshold, wherein the second threshold is less than the first threshold. These two thresholds are used to judge the load state of the source data system in a target time period, and the scanning operation is adjusted accordingly.

[0056] Optionally, when it is detected that the transaction data volume of the source data system in a target time period is greater than or equal to a first threshold, it indicates that the source data system is in a high load state. At this time, the data migration system can reduce the scanning frequency of the scanning operation and / or narrow the scanning range. The purpose of this is to reduce the performance pressure of the scanning operation on the source data system, and to ensure that the source system can still operate stably in a high load state.

[0057] Conversely, when it is detected that the number of transactions of the source data system in the target time period is less than or equal to the second threshold value, it indicates that the source data system is in a low load state. In this case, the data migration system can increase the scanning frequency and / or expand the scanning range of the scanning operation. In this way, the speed of data migration can be accelerated, the idle resources of the source system during the low load period can be fully utilized, and the migration efficiency can be improved.

[0058] Optionally, if the amount of transaction data of the source data system in the target time period is between the second threshold value and the first threshold value, the data migration system can maintain the current scanning frequency and scanning range unchanged. In this case, the load of the system is at a medium level, and there is no need to adjust the scanning strategy to maintain the stability and continuity of the scanning operation.

[0059] From the above, it can be seen that through this threshold-based dynamic adjustment mechanism, the data migration system can intelligently adapt to the load changes of the source data system in different time periods, thereby efficiently completing the data migration task without affecting the normal operation of the source system. This method not only improves the flexibility and reliability of data migration, but also optimizes resource utilization, ensuring the efficiency and stability of the entire migration process.

[0060] In an optional embodiment, the data migration method further comprises, in the case where it is detected that the amount of transaction data of the source data system in the target time period is greater than or equal to the first threshold value, if it is detected that the source data system comprises a primary database and a standby database, reducing the scanning range of the scanning operation to the standby database.

[0061] Optionally, the source data system can comprise a primary database and a standby database, the primary database being used for daily high-concurrency write operations, and the standby database being used for backup and read operations. In the case where it is detected that the amount of transaction data of the source data system in the target time period is greater than or equal to the first threshold value, if the source data system contains a primary database and a standby database, the scanning range of the scanning operation is reduced to the standby database, which can avoid negative effects on high-concurrency write operations of the primary database, while enabling smooth data migration.

[0062] This embodiment can avoid scanning operations on the primary database in a high load state by limiting the scanning range to the standby database, thereby reducing performance degradation and potential migration failure risks caused by scanning operations. The standby database generally has a lower load and is more suitable for data migration operations, improving the reliability of data migration and ensuring smooth data migration.

[0063] In an optional embodiment, during the process of performing the scanning operation on the data in the source data system according to the scanning strategy, the data migration method further comprises: the data migration system divides the database of the source data system into N partitions, where N is an integer greater than 1. Then, a round-robin scanning operation is performed on the N partitions, where the round-robin scanning operation is used to restrict each scanning operation to scan one partition, and the N continuous scanning operations correspond to one round of data scanning of the N partitions respectively.

[0064] Optionally, the database partitioning of the source data system is to divide the database of the source data system into N partitions, where N is an integer greater than 1. Each partition contains a part of the data in the database. The partitioning can decompose a large-scale data migration task into multiple small tasks, reduce the pressure of a single scan, improve the scanning efficiency, and facilitate the monitoring and management of the scanning process. The round-robin scanning operation means to scan each partition in turn according to a certain order. Each scanning operation is only performed on one partition, and the N continuous scanning operations correspond to one round of data scanning of the N partitions respectively. Through the round-robin scanning, the scanning task of the entire database can be completed gradually, avoiding the performance problems and potential scanning failure risks caused by full scanning.

[0065] This embodiment can avoid the performance pressure caused by full scanning by dividing the database into multiple partitions and performing round-robin scanning on each partition. Each scanning is only performed on one partition, reducing the occupation of system resources, reducing the impact of scanning operation on the performance of the source data system, reducing the negative impact on the source data system, and ensuring that the source data system can operate normally during data migration, especially in the scenario of high-concurrency writing, without significant interference to business operations. The partition scanning can decompose a large-scale data migration task into multiple small tasks, and each scanning only processes the data of one partition, reducing the complexity of a single scan. Through the round-robin scanning, the scanning task of the entire database can be completed gradually, improving the scanning efficiency.

[0066] In an optional embodiment, according to the concurrent write operation state of the target database, the target data is migrated to the target database in an asynchronous manner, comprising: according to the concurrent write operation state of the target database, determining a migration time of the target data, wherein the concurrent write data amount of the target database at the migration time is lower than the concurrent write data amount of the target database at a non-migration time.

[0067] Optionally, the concurrent write operation state of the target database can refer to the number and frequency of write operations being performed by the target database at a certain time. The concurrent write operation state reflects the current load of the target database, including the number of write tasks being executed, the write speed, and the response time of the system. By monitoring the concurrent write operation state, the load of the target database at a specific time can be understood, thereby providing a basis for dynamically adjusting the timing of data migration. The migration timing can refer to the specific time point at which data migration is selected to be performed when the concurrent write operation state of the target database is low. By selecting to migrate at a time when the amount of concurrent write data of the target database is low, the impact on the performance of the target database can be reduced, ensuring that the target database can operate normally during the migration process.

[0068] By selecting to migrate at a time when the amount of concurrent write data of the target database is low, the impact on the performance of the target database can be reduced. The target database can operate normally during the migration process, reducing the performance degradation caused by large-scale write operations, reducing the negative impact on the target database, and improving the high availability and stability of the target database. By migrating data asynchronously, the data migration task can be dispersed and completed gradually in the background, avoiding blocking the normal operation of the source database. This approach can fully utilize system resources, improve the efficiency of data migration, shorten the total time of data migration, improve the overall efficiency of data migration, and also improve the reliability of data migration, reducing data omission or errors caused by migration failure.

[0069] In an optional embodiment, according to the concurrent write operation state of the target database, the target data is migrated to the target database in an asynchronous manner, including: if it is detected that the target data is data supporting repeated write, determining a data conversion rule between the target data and the data write parameter of the target database, and according to the data conversion rule and the data write interface of the target database, migrating the target data to the target database in an asynchronous manner.

[0070] The data supporting repeated writing can refer to data that can be written multiple times to the target database without causing data conflicts or errors. Such data usually has idempotency, that is, multiple writes of the same data will not change the final result. By determining the data conversion rule between the target data and the data write parameter of the target database, it is ensured that the data format meets the requirements of the target database. The embodiment migrates data in an asynchronous manner, which can disperse data migration tasks to the background to complete step by step, avoid large-scale write operations when the target database is under high load, reduce the impact on the performance of the target database, reduce the negative impact on the target database, and ensure the high availability and stability of the target database. The asynchronous manner can disperse data migration tasks to the background to complete step by step, avoid blocking the normal operation of the source database, fully utilize system resources, improve the efficiency of data migration, shorten the total time of data migration, and improve the overall efficiency of data migration.

[0071] If it is detected that the target data is data not supporting repeated writing, a data write function of the target database is called, and the target data is migrated to the target database in an asynchronous manner according to the data write function.

[0072] The data not supporting repeated writing refers to data that cannot be written multiple times to the target database, otherwise data conflicts or errors will be caused. Such data usually needs to ensure uniqueness. For data not supporting repeated writing, direct repeated writing can cause data duplication, overwrite, or violate the uniqueness constraint, thereby causing errors. Therefore, special processing is required for such data to ensure the correctness and integrity of data migration. During data migration, it is necessary to detect whether the target data supports repeated writing, which can be achieved by checking the attributes, fields, or business logic of the data. For example, some fields can be marked as unique keys, such as primary keys, unique indexes, etc. The data of these fields usually does not support repeated writing. The data write function of the target database can refer to the interface provided by the target database for writing data to the database. These functions usually perform data validation, uniqueness checking, etc. to ensure the correctness and integrity of the data. By calling the data write function of the target database, the database's own verification mechanism can be used to ensure that data not supporting repeated writing will not be written repeatedly, thereby avoiding data conflicts or errors. The asynchronous manner can disperse data migration tasks to the background to complete step by step, avoid blocking the normal operation of the source database, and reduce the impact on the performance of the target database. Moreover, the asynchronous manner can also handle failed write operations through a retry mechanism to ensure the reliability of data migration.

[0073] In an alternative embodiment, the data migration method further comprises: during the scanning operation on the data in the source data system according to the scanning strategy, sampling the scanned data, detecting the success rate of the scanning operation according to the sampling result, collecting performance information of the source data system during the sampling, and adjusting the scanning strategy according to the success rate of the scanning operation and the performance information of the source data system during the sampling.

[0074] Optionally, the data sampling refers to selecting part of the data for analysis randomly or according to certain rules during the scanning. The sampled data is used to evaluate the overall effect of the scanning operation and the performance of the source data system. Through data sampling, the success rate of the scanning operation and the performance status of the source data system can be quickly understood without processing all the data, thereby saving time and resources. The scanning success rate refers to the ratio of the successfully scanned data amount to the total amount of the sampled data. By detecting the scanning success rate, the reliability of the scanning operation can be evaluated. The scanning success rate directly affects the integrity and accuracy of the data migration. If the success rate is low, the scanning strategy may need to be adjusted to improve the efficiency. The performance information may include, but is not limited to, the central processing unit usage rate and the memory usage rate. These indicators reflect the running status of the source data system during the scanning. By collecting the performance information, the influence of the scanning operation on the source data system can be evaluated to ensure that the scanning operation does not cause negative impact on the normal operation of the source data system.

[0075] During the scanning, part of the data is selected as a sample randomly or according to certain rules. For example, a batch of data can be extracted at certain time intervals, or sampling can be performed according to the proportion of the data amount. The sampled data is analyzed to calculate the ratio of the successfully scanned data amount to the total amount of the sampled data. For example, if 100 pieces of data are extracted and 95 pieces are successfully scanned, the scanning success rate is 95%. During the data sampling, the performance information of the source data system is collected in real time. The monitoring tools or the performance monitoring function of the system itself can be used to collect the information. According to the scanning success rate and the performance information, the scanning strategy is dynamically adjusted. For example, if the scanning success rate is low, the scanning parameters can be adjusted, such as increasing the number of scanning retries, optimizing the scanning logic, etc. If the performance information shows that the load of the source data system is too high, the scanning frequency can be reduced, the scanning range can be narrowed, or the scanning time can be adjusted to a period with lower system load. If the scanning success rate is high and the performance information is normal, the current scanning strategy can be maintained, or the scanning frequency can be appropriately increased to speed up the data migration.

[0076] The embodiment can quickly evaluate the effect of the scanning operation by data sampling and dynamically adjusting the scanning strategy, and optimize the scanning parameters according to the actual situation, thereby improving the scanning efficiency, shortening the total time of data migration, and improving the overall efficiency of data migration. By collecting performance information and dynamically adjusting the scanning strategy according to the performance state, it can be ensured that the scanning operation will not cause negative impact on the normal operation of the source data system, reducing the negative impact on the source data system, and ensuring the high availability and stability of the source data system. By detecting the scanning success rate, problems that may occur during the scanning process can be found in time, and the scanning strategy can be optimized by adjusting the scanning strategy, thereby improving the reliability of data migration, reducing data omission or errors caused by scanning failure, and ensuring the integrity and accuracy of data migration.

[0077] According to another aspect of the embodiment of the present application, a data migration device is also provided, wherein, Figure 3 is a schematic diagram of a data migration device according to an embodiment of the present application, as Figure 3 shown, the device comprises a scanning unit 31, a conversion unit 32 and a migration unit 33.

[0078] The scanning unit 31 is configured to perform a scanning operation on the data in the source data system according to a scanning strategy, wherein the scanning strategy includes determining the scanning duration, scanning frequency, scanning range and scanning partition of the scanning operation, and the source data system is a distributed in-memory key-value storage system. The conversion unit 32 is configured to convert the data obtained after the scanning operation into target data that meets the storage format of the target database, wherein the target database is a key-value storage database. The migration unit 33 is configured to migrate the target data to the target database in an asynchronous manner according to the concurrent write operation state of the target database.

[0079] Optionally, the scanning unit 31 comprises a detection subunit configured to detect the transaction data amount of the source data system in a target time period, wherein the target time period is any time period; and a first determination subunit configured to determine the adjustment mode of the scanning operation according to the transaction data amount of the source data system in the target time period.

[0080] Optionally, the first determining subunit comprises: a first processing module configured to reduce the scanning frequency and / or reduce the scanning range of the scanning operation if it is detected that the transaction data volume of the source data system in the target time period is greater than or equal to a first threshold value; a second processing module configured to increase the scanning frequency and / or increase the scanning range of the scanning operation if it is detected that the transaction data volume of the source data system in the target time period is less than or equal to a second threshold value, wherein the second threshold value is less than the first threshold value; and a third processing module configured to maintain the scanning frequency and the scanning range of the scanning operation unchanged if it is detected that the transaction data volume of the source data system in the target time period is greater than the second threshold value and less than the first threshold value.

[0081] Optionally, the data migration apparatus further comprises a fourth processing module configured to reduce the scanning range of the scanning operation to the standby database if it is detected that the source data system comprises a primary database and a standby database, and if it is detected that the transaction data volume of the source data system in the target time period is greater than or equal to the first threshold value.

[0082] Optionally, the scanning unit 31 further comprises a partition subunit configured to divide the database of the source data system into N partitions, wherein N is an integer greater than 1; and a round-robin scanning subunit configured to perform a round-robin scanning operation on the N partitions, wherein the round-robin scanning operation is configured to restrict each scanning operation to scan one partition, and N continuous scanning operations correspond to one round of data scanning on the N partitions.

[0083] Optionally, the migration unit 33 comprises a second determining subunit configured to determine a migration time of the target data according to the concurrent write operation state of the target database, wherein the concurrent write data volume of the target database at the migration time is lower than the concurrent write data volume of the target database at a non-migration time.

[0084] Optionally, the migration unit 33 further comprises a third determining subunit configured to determine a data conversion rule between the target data and a data write parameter of the target database if it is detected that the target data is data supporting repeated write, and to migrate the target data to the target database in an asynchronous manner according to the data conversion rule and a data write interface of the target database; and a fourth determining subunit configured to call a data write function of the target database if it is detected that the target data is data not supporting repeated write, and to migrate the target data to the target database in an asynchronous manner according to the data write function.

[0085] Optionally, the data migration apparatus further comprises: a sampling unit, configured to sample the data scanned from the source data system according to the scanning strategy; a detection unit, configured to detect a success rate of the scanning operation according to the sampling result; a collection unit, configured to collect performance information of the source data system during the sampling; and an adjustment unit, configured to adjust the scanning strategy according to the success rate of the scanning operation and the performance information of the source data system during the sampling.

[0086] According to a further aspect of the embodiments of the present application, a computer readable storage medium is provided, wherein the computer readable storage medium stores a computer program, and the computer program, when executed, causes a device where the computer readable storage medium is located to perform the data migration method.

[0087] According to a further aspect of the embodiments of the present application, an electronic device is provided, wherein the electronic device comprises one or more processors and a memory, and the memory is configured to store one or more programs, and the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data migration method.

[0088] According to a further aspect of the embodiments of the present application, a computer program product is provided, wherein the computer program product comprises a computer program or instructions, and the computer program or instructions, when executed by a processor, implement the data migration method.

[0089] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above-described embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0091] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit division in the above-described device embodiment is only a logical function division, and there can be another division manner during actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the logical couplings or communication connections between different units, can be implemented in electronic, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0093] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0094] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0095] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A data migration method, characterized in that: include: Performing a scanning operation on data in a source data system according to a scanning strategy, wherein the scanning strategy includes determining a scanning duration, a scanning frequency, a scanning range, and a scanning partition of the scanning operation, and the source data system is a distributed memory key-value storage system; Converting the data obtained after the scanning operation into target data that meets the storage format of a target database, wherein the target database is a key-value storage database; The target data is migrated to the target database in an asynchronous manner according to the concurrent write operation status of the target database.

2. The data migration method according to claim 1, characterized in that: During the process of scanning the data in the source data system according to the scanning strategy, the data migration method further includes: Detecting the transaction data volume of the source data system in a target time period, wherein the target time period is any time period; The adjustment method of the scanning operation is determined according to the transaction data volume of the source data system in the target time period.

3. The data migration method according to claim 2, wherein: Determining an adjustment method for the scanning operation based on the transaction data volume of the source data system in the target time period includes: When detecting that the transaction data volume of the source data system in the target time period is greater than or equal to a first threshold, reducing the scanning frequency of the scanning operation and / or narrowing the scanning range of the scanning operation; increasing the scanning frequency of the scanning operation and / or expanding the scanning range of the scanning operation when detecting that the number of transactions of the source data system in the target time period is less than or equal to a second threshold, wherein the second threshold is less than the first threshold; When it is detected that the transaction data volume of the source data system in the target time period is greater than the second threshold and less than the first threshold, the scanning frequency and the scanning range of the scanning operation are maintained unchanged.

4. The data migration method according to claim 3, wherein: The data migration method further includes: When it is detected that the transaction data volume of the source data system in the target time period is greater than or equal to the first threshold, if it is detected that the source data system includes a primary database and a standby database, the scanning scope of the scanning operation is narrowed to the standby database.

5. The data migration method according to claim 1, wherein: During the process of scanning the data in the source data system according to the scanning strategy, the data migration method further includes: Dividing the database of the source data system into N partitions, where N is an integer greater than 1; A patrol scanning operation is performed on the N partitions, wherein the patrol scanning operation is used to constrain each scanning operation to scan one partition, and the consecutive N scanning operations complete a data scan corresponding to the N partitions respectively.

6. The data migration method according to claim 1, wherein: Migrating the target data to the target database in an asynchronous manner according to the concurrent write operation status of the target database includes: The migration time of the target data is determined according to the concurrent write operation status of the target database, wherein the concurrent write data volume of the target database at the migration time is lower than the concurrent write data volume of the target database at the non-migration time.

7. The data migration method according to claim 1, wherein: Migrating the target data to the target database in an asynchronous manner according to the concurrent write operation status of the target database includes: If it is detected that the target data is data that supports repeated writing, determining a data conversion rule between the target data and the data writing parameter of the target database, and migrating the target data to the target database in an asynchronous manner according to the data conversion rule and the data writing interface of the target database; If it is detected that the target data is data that does not support repeated writing, a data writing function of the target database is called, and the target data is migrated to the target database in an asynchronous manner according to the data writing function.

8. The data migration method according to any one of claims 1 to 7, characterized in that: The data migration method further includes: In the process of performing the scanning operation on the data in the source data system according to the scanning strategy, performing data sampling on the scanned data; detecting a success rate of the scanning operation according to a result of the data sampling; Collecting performance information of the source data system during the data sampling process; The scanning strategy is adjusted according to the success rate of the scanning operation and the performance information of the source data system during the data sampling process.

9. A data migration device, characterized in that: include: a scanning unit, configured to perform a scanning operation on data in a source data system according to a scanning strategy, wherein the scanning strategy includes determining a scanning duration, a scanning frequency, a scanning range, and a scanning partition of the scanning operation, and the source data system is a distributed memory key-value storage system; a conversion unit, configured to convert the data obtained after the scanning operation into target data that meets the storage format of a target database, wherein the target database is a key-value storage database; The migration unit is configured to migrate the target data to the target database in an asynchronous manner according to a concurrent write operation state of the target database.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the data migration method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the data migration method described in any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or instructions, which implements the data migration method according to any one of claims 1 to 8 when executed by a processor.

Images