Data migration method and device, storage medium, and electronic device

Abstract

The present disclosure relates to the field of data processing, and provides a data migration method, a data migration device, a computer storage medium, and an electronic device, wherein the data migration method includes: establishing a communication connection with a migration data source and a receiving data source; sharding the full amount of data in the migration data source to obtain multiple data shards; when a data migration instruction is received, distributing the multiple data shards to the receiving data source through a migration cluster; obtaining incremental data from the migration data source, and judging whether the incremental data is new data according to the creation timestamp of the incremental data; when the incremental data is new data, migrating the incremental data to the receiving data source through the migration cluster; when the incremental data is not new data, replacing the target data in the receiving data source with the incremental data through the migration cluster. The present disclosure can improve the data migration speed and ensure the orderly progress of the data migration process without affecting the read and write business of the migration data source.

Description

Background Art

[0002] Elasticsearch (ES cluster for short) is a search engine based on the Lucene library. It provides a distributed, multi-tenant full-text search engine with high query efficiency, good full-text search performance, and good support for distribution. The cluster-based environment can be easily expanded horizontally, and massive amounts of data can be distributed to multiple clusters for storage and retrieval. However, with the version upgrade of Elasticsearch, all users face a problem, which is how to quickly and accurately migrate data from a low-version cluster to a high-version cluster without affecting business read and write.

[0003] In the related technology, during the process of migrating data from a low-version cluster to a high-version cluster, the data can only be migrated gradually and cannot cross major versions (for example, directly migrating from ES2.1 to ES6.3). After the migration is completed, the cluster needs to be restarted. If the migration fails midway, the migration needs to be started again from the beginning. Therefore, the migration time span is large and the data migration efficiency is low.

[0004] In view of this, there is an urgent need in the art to develop a new data migration method and device.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data migration method, a data migration device, a computer storage medium and an electronic device, thereby at least to a certain extent avoiding the defects of the related technology that affect business reading and writing and have low migration efficiency.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data migration method is provided, including: establishing a communication connection with a migration data source and a receiving data source; sharding the full amount of data in the migration data source to obtain multiple data shards; when a data migration instruction is received, migrating the multiple data shards to the receiving data source in a distributed manner through a migration cluster; obtaining incremental data from the migration data source, and determining whether the incremental data is newly added data based on a creation timestamp of the incremental data; when the incremental data is newly added data, migrating the incremental data to the receiving data source through the migration cluster; when the incremental data is not newly added data, replacing target data in the receiving data source with the incremental data through the migration cluster; wherein the target data has the same creation timestamp as the incremental data.

[0009] In an exemplary embodiment of the present disclosure, the slicing of the full amount of data in the migration data source to obtain multiple data slicings includes: obtaining the initial running time of the migration data source, and determining the time span between the current time and the initial running time as the target time period; slicing the target time period according to a preset time interval to obtain multiple time slices; and taking the data stored in the migration data source in each of the time slices as a data slicing.

[0010] In an exemplary embodiment of the present disclosure, each data shard corresponds to a migration subtask; when a data migration instruction is received, the multiple data shards are migrated to the receiving data source through a migration cluster, including: when a data migration instruction is received, the migration subtask is distributedly executed through a migration cluster to migrate the multiple data shards to the receiving data source.

[0011] In an exemplary embodiment of the present disclosure, the method also includes: when the amount of data to be migrated corresponding to the target data shard is greater than a data amount threshold, splitting the migration subtask corresponding to the target data shard into multiple target subtasks so that the amount of data to be migrated corresponding to each target subtask is less than the data amount threshold.

[0012] In an exemplary embodiment of the present disclosure, replacing the target data in the receiving data source with the incremental data through the migration cluster includes: comparing the update timestamps of the incremental data and the target data according to an optimistic locking mechanism; when the update timestamp of the incremental data is greater than the update timestamp of the target data, replacing the target data in the receiving data source with the incremental data.

[0013] In an exemplary embodiment of the present disclosure, after replacing the target data in the receiving data source with the incremental data through the migration cluster, the method further includes: querying the number of data items stored in the migration data source; if the number of data items is greater than the number of data items in the receiving data source, obtaining the difference data between the migration data source and the receiving data source; and updating the difference data to the receiving data source through the migration cluster.

[0014] In an exemplary embodiment of the present disclosure, the method also includes: if the number of data items is equal to the number of data items in the receiving data source, obtaining the field values ​​of all data in the migration data source, and obtaining the field values ​​of all data in the receiving data source; comparing whether the field values ​​of the same data are consistent; marking the data in the migration data source with inconsistent field values; and updating the marked data to the receiving data source.

[0015] In an exemplary embodiment of the present disclosure, the method further includes: when data migration is interrupted, recording the interruption time; and performing breakpoint resume according to the interruption time.

[0016] In an exemplary embodiment of the present disclosure, the method also includes: creating a first standby server for the migration data source, and creating a second standby server for the receiving data source; regularly migrating the data in the migration data source to the first standby server through the migration cluster; and regularly migrating the data in the receiving data source to the second standby server through the migration cluster.

[0017] According to a second aspect of the present disclosure, a data migration device is provided, comprising: a communication module, used to establish a communication connection with a migration data source and a receiving data source; a data sharding module, used to shard the full data in the migration data source to obtain multiple data shards; a full data migration module, used to distribute and migrate the multiple data shards to the receiving data source through a migration cluster when a data migration instruction is received; an incremental data migration module, used to obtain incremental data from the migration data source, and determine whether the incremental data is new data according to a creation timestamp of the incremental data; when the incremental data is new data, migrate the incremental data to the receiving data source through the migration cluster; when the incremental data is not new data, use the incremental data to replace the target data in the receiving data source through the migration cluster; wherein the target data has the same creation timestamp as the incremental data.

[0018] According to a third aspect of the present disclosure, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data migration method described in the first aspect is implemented.

[0019] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data migration method described in the first aspect above by executing the executable instructions.

[0020] It can be seen from the above technical solutions that the data migration method, data migration device, computer storage medium and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0021] In the technical solutions provided by some embodiments of the present disclosure, on the one hand, after establishing a communication connection with the migration data source and the receiving data source, the full amount of data in the migration data source is sharded to obtain multiple data shards, so that the full amount of data can be migrated concurrently through the cluster, providing a general, efficient, simple to operate, rich migration scenarios, and accurate migration data migration solution. Further, when a data migration instruction is received, the multiple data shards are distributed and migrated to the receiving data source through the migration cluster, which can solve the technical problems of business suspension, long operation time, and complex migration process of the migration data source in the related technology, and can improve the data migration speed without affecting the read and write business of the migration data source, and ensure the orderly progress of the data migration process. On the other hand, incremental data is obtained from the migration data source, and the incremental data is judged whether it is new data according to the creation timestamp of the incremental data; when the incremental data is new data, the incremental data is migrated to the receiving data source through the migration cluster, which can avoid data omission and ensure data integrity; when the incremental data is not new data, the incremental data is replaced by the migration cluster with the incremental data to replace the target data in the receiving data source, which can ensure data validity and save storage space of the receiving data source.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1A A schematic diagram showing a data migration method in the related art;

[0025] Figure 1B A schematic diagram showing another data migration method in the related art;

[0026] Figure 2 A schematic diagram showing a flow chart of a data migration method in an exemplary embodiment of the present disclosure;

[0027] Figure 3 A diagram showing a deployment architecture of a data migration program package in an exemplary embodiment of the present disclosure is shown;

[0028] Figure 4 A deployment architecture diagram of a data migration program package in another exemplary embodiment of the present disclosure is shown;

[0029] Figure 5 A schematic diagram showing a data migration method in an exemplary embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram showing a data migration method in another exemplary embodiment of the present disclosure;

[0031] Fig. 7A A sub-schematic diagram showing a data migration method in yet another exemplary embodiment of the present disclosure;

[0032] Figure 7B A sub-schematic diagram showing a data migration method in yet another exemplary embodiment of the present disclosure;

[0033] Figure 8 A schematic diagram showing the overall process of a data migration method in an exemplary embodiment of the present disclosure is shown;

[0034] Fig. 9 A schematic diagram showing a data migration method in an exemplary embodiment of the present disclosure;

[0035] Fig.10 A schematic diagram of a sub-process of a data migration method in an exemplary embodiment of the present disclosure is shown;

[0036] Fig.11 A schematic diagram of a sub-process of a data migration method in an exemplary embodiment of the present disclosure is shown;

[0037] Fig.12 A schematic diagram showing a data migration method in an exemplary embodiment of the present disclosure;

[0038] Fig.13 A schematic diagram of a sub-process of a data migration method in an exemplary embodiment of the present disclosure is shown;

[0039] Fig.14 A schematic diagram showing a synchronization task table in an exemplary embodiment of the present disclosure;

[0040] Fig.15 A schematic diagram showing the structure of a data migration device in an exemplary embodiment of the present disclosure is shown;

[0041] Fig.16 A schematic diagram showing the structure of a computer storage medium in an exemplary embodiment of the present disclosure;

[0042] Fig.17 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0043] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0044] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0045] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0046] In the related technology, there are generally the following data migration solutions:

[0047] ①Elasticsearch clusters above version 5.0 support rolling upgrades (Rolling upgrade is a way to upgrade multiple replica services), but versions below 5.0 cannot be upgraded across major versions. For example, you can refer to Figure 1A , Figure 1A A schematic diagram of a data migration method in the related art is shown, specifically a schematic diagram of a rolling upgrade supported by ES cluster version 5.0 and above in the related art. Therefore, the disadvantages of this solution are: the version can only be upgraded step by step, and the cluster needs to be restarted after the upgrade. If a failure occurs, the migration needs to be restarted from the beginning, the operation time is long and the process is too complicated, and the business needs to stop reading and writing.

[0048] ② Rebuild the index (re_index) through the official ES cluster. Only after the index creation is completed can the data be migrated. The disadvantage of this solution is that the lower version index cannot be created on the higher version cluster.

[0049] ③Write the data in the low-version ES cluster to MySQL and directly read the data in MySQL to the high-version ES cluster. The disadvantage of this solution is that all data must be backed up in MySQL, and the solution is not universal.

[0050] ④ Deploy the low-version migration program and the high-version migration program separately, that is, deploy the low-version migration program on client 2 and the high-version migration program on client 6, and then write the data into the high-version ES cluster through jmq asynchronous consumption. The data migration process is: ES2.1-client2-jmq-client6-ES6. The disadvantage of this solution is that the two sets of migration programs, client 2 and client 6, are deployed separately, and there are many input and output ports, which makes the migration efficiency low.

[0051] ⑤ Migration method of big data platform: JAVA-kafka-JRC (java real-time computing platform Storm / Spark / Flink)-ES cluster. The disadvantages of this solution are: it relies on the big data environment and is not universal, the process is difficult to monitor, and failure compensation and monitoring alarms are difficult to do.

[0052] ⑥Spark scripts perform data migration based on Elasticsearch for Apache Hadoop by scheduling tasks. For example, you can refer to Figure 1B , Figure 1B A schematic diagram of another data migration method in the related art is shown, specifically showing that the Spark script in the related art needs to rely on a big data environment. Therefore, the disadvantages of this solution are: it relies on a big data environment, and the solution is imperfect and not universal enough.

[0053] In summary, the solutions in the related technologies and their drawbacks and characteristics can be referred to in the following Table 1:

[0054] Table 1

[0055]

[0056] Therefore, the related technology cannot achieve fast and accurate data migration from a low-version ES cluster to a high-version ES cluster without affecting business reading and writing.

[0057] In the embodiments of the present disclosure, firstly, a data migration method is provided, which at least to some extent overcomes the defects of the related art that the business reading and writing are affected and the migration efficiency is low.

[0058] Figure 2A flow chart of a data migration method in an exemplary embodiment of the present disclosure is shown. The execution subject of the data migration method may be a Tomcat server that deploys a data migration tool.

[0059] refer to Figure 2 According to an embodiment of the present disclosure, a data migration method includes the following steps:

[0060] Step S210, establishing a communication connection with a migration data source and a receiving data source;

[0061] Step S220, sharding the full amount of data in the migration data source to obtain multiple data shards;

[0062] Step S230, when a data migration instruction is received, multiple data shards are migrated to a receiving data source through a migration cluster distribution;

[0063] Step S240, obtaining incremental data from the migration data source, and determining whether the incremental data is newly added data according to the creation timestamp of the incremental data;

[0064] Step S250, when the incremental data is newly added data, the incremental data is migrated to the receiving data source through the migration cluster;

[0065] Step S260, when the incremental data is not newly added data, the incremental data is used to replace the target data in the receiving data source through the migration cluster; wherein the target data and the incremental data have the same creation timestamp.

[0066] exist Figure 2 In the technical solution provided by the illustrated embodiment, on the one hand, after establishing a communication connection with the migration data source and the receiving data source, the full amount of data in the migration data source is sharded to obtain multiple data shards, so that the full amount of data can be concurrently migrated through the cluster, providing a general, efficient, simple to operate, rich migration scenarios, and accurate migration data migration solution. Further, when a data migration instruction is received, the multiple data shards are distributed and migrated to the receiving data source through the migration cluster, which can solve the technical problems of business suspension, long operation time, and complex migration process of the migration data source in the related technology, and can improve the data migration speed without affecting the read and write business of the migration data source, and ensure the orderly progress of the data migration process. On the other hand, incremental data is obtained from the migration data source, and the incremental data is determined according to the creation timestamp of the incremental data. When the incremental data is new data, the incremental data is migrated to the receiving data source through the migration cluster, which can avoid data omission and ensure data integrity; when the incremental data is not new data, the incremental data is replaced by the migration cluster with the incremental data to replace the target data in the receiving data source, which can ensure data validity and save storage space of the receiving data source.

[0067] The following Figure 2 The specific implementation process of each step is described in detail:

[0068] The migration data source in the present disclosure can be a low-version storage cluster, and the receiving data source can be a high-version storage cluster. Among them, multiple distributed storage nodes are deployed in the storage cluster, and these multiple storage nodes can adopt a consistent replication protocol, such as the Raft protocol, to ensure data consistency. It should be noted that the storage cluster in the present disclosure can be an ES cluster, a cluster composed of at least two or more Mysql database servers, a cluster composed of at least two or more Mongo database servers, a cluster composed of at least two or more hbase database servers, etc., which can be set according to actual conditions and belong to the protection scope of the present disclosure.

[0069] In the following embodiments, a migration data source is an ES cluster of a lower version and a receiving data source is an ES cluster of a higher version.

[0070] For example, a data migration package (hereinafter referred to as SDK package) can be developed, and the SDK package can be deployed in a migration cluster (Tomcat cluster) to obtain a data migration tool. The full English name of SDK is software development kit, which means software development kit in Chinese. In layman's terms, it refers to a toolkit provided by a third-party service provider to implement a certain function of a software product, generally in the form of a collection of API (Application Programming Interface, referred to as API, application program interface) and documents, examples, and tools.

[0071] For example, please refer to Figure 3 , Figure 3 FIG. 1 shows a deployment architecture diagram of an SDK package in an exemplary embodiment of the present disclosure. Figure 3 , you can configure a task monitoring platform through either Web config or yaml config.json. The task monitoring platform can perform task management (including task grouping, task configuration, and task start or stop) and task monitoring and statistics (including task failure compensation and alarm, migration progress bar display, and data consistency comparison). Specifically, you can create a migration subtask in the migration data source of the ES2.1 version, and then the migration cluster (Tomcat cluster) can read the migration subtask from the task configuration table. Then, you can start the migration task through Web config or yaml config.json, so that the migration cluster starts migrating data shards.

[0072] Among them, Web config is an XML (Extensible Markup Language, extensible markup language, referred to as: XML) text file, used to store the configuration information of ASP.NET Web applications (such as the most commonly used authentication method for ASP.NET Web applications), it can be used for dynamic configuration of pages, such as: monitoring pages, cluster information, index information, type information, migration time period, time slice size, number of thread pools and setting migration buttons, etc. yamlconfig.json is a static configuration file, which can be used for static configuration, such as: database connection information, user name and password, table structure, mapping relationship, processing class and other configurations.

[0073] For example, you can also refer to Figure 4 , Figure 4 FIG. 1 shows a deployment architecture diagram of an SDK package in another exemplary embodiment of the present disclosure. Figure 4 Each server node in the migration cluster can be registered in the configuration center (Zookeeper, Spring cloudconfig, Nacos or Apollo). After registration, the migration cluster executes the migration subtask in a distributed manner when receiving the data migration instruction. The configuration center can perform task management (grouping, configuring, starting and stopping tasks), global configuration (registration center, configuration API, dynamic expansion of related storage units, etc.), log management (task operation log, task configuration log, etc.), monitoring and statistics (task failure compensation and alarm, migration progress bar display, data consistency comparison) and so on through the task monitoring platform.

[0074] Continue to refer Figure 2 In step S210, a communication connection is established with the migration data source and the receiving data source.

[0075] When you need to migrate data in the current ES cluster of version 2.1 to the ES cluster of version 6.3, the migration data source is the ES cluster of version 2.1, and the receiving data source is the ES cluster of version 6.3.

[0076] The user (the person who performs data migration) can enter the cluster name, user name and password of the migration data source, as well as the cluster name, user name and password of the receiving data source in the configuration page of the data migration tool. After submitting the above information, the data migration tool can establish a communication connection with the migration data source based on the cluster name, user name and password of the migration data source, and establish a communication connection with the receiving data source based on the cluster name, user name and password of the receiving data source.

[0077] Furthermore, the user can enter and submit relevant configuration parameters in the configuration page. For example, the parameters entered by the user may include: a preset time interval (for example, 1 hour), the number of server nodes (for example, 10), the number of thread pools on each server node (for example, 16 threads), and the duration of each server node executing each time slice (for example, 5 minutes).

[0078] Furthermore, the data migration tool can obtain the initial running time of the migration data source (for example, it can be the time when the migration data source performs business reading and writing for the first time, or the time when the first data is written to the migration data source), and then, the time span between the current time and the initial running time can be determined as the target time period (for example, the target time period can be N years, where N is a rational number greater than 0), and the target time period can be sliced ​​according to the above-mentioned preset time interval (for example: 1 hour) to obtain multiple time slices (for example: M, where M is a positive integer greater than 0).

[0079] Furthermore, the data migration tool can estimate the migration duration of the migration task based on the number of server nodes (for example, 10), the number of thread pools on each server node (for example, 16 threads), and the time for each server node to execute each time slice (for example, 5 minutes). Then, the migration duration can be added to the obtained task start time to generate the task end time. For example, the start time can be set to January 1, 2019, and then the end time can be determined to be January 1, 2020. Thus, the execution time of the migration task can be determined to be from January 1, 2019 to January 1, 2020.

[0080] Then, a migration task A can be created, and the execution time of migration task A is from January 1, 2019 to January 1, 2020. Then, the data migration tool can create two temporary task tables (including a main task table A for recording execution information related to the migration task and a subtask table B for recording execution information related to multiple migration subtasks) in the migration data source to store the above migration task A and migration subtasks.

[0081] The field names in the main task table A may include cluster (cluster name), index_name (index name), ES_type (cluster type), begin_time (task start time), end_time (task end time), shardtask_num (number of shards), complete_task_num (number of completed tasks), and progress. The specific data structure of the main task table A can be referred to in Table 2 below:

[0082] Table 2

[0083] cluster(cluster name) Coupon index_name(index name) index_name ES_type (index type) type begin_time (start time) 2019-01-01 end_time(end time) 2020-01-01 shardtask_num (number of shard tasks) 100 complete_task_num (number of completed tasks) 20 progress 20％

[0084] The field names in subtask table B may include: system_id (system identifier), cluster (cluster name), source_index (source index), type (source index type), query_date (query date), query_hour (query time), query_start_time (query start time), query_end_time (query end time), data_count (data volume), used time (migration time), refer_field_name (related field name), status (migration status), target_cluster (receiving data source), target_index (target index), target_type (target index type), ip (Internet Protocol address), retrys (number of re-migration times), msg (prompt message). The specific data structure of subtask table B can be referred to in Table 3 below:

[0085] Table 3

[0086]

[0087] Continue to refer Figure 2 In step S220, the full amount of data in the migration data source is sharded to obtain multiple data shards.

[0088] Referring to the explanation of step S210 above, after the time slices are divided, the data stored in the migration data source in each time slice can be taken as a data slice. Referring to the explanation of the steps above, each time slice is 1 hour, and the explanation is given as an example. Then, one of the data slices i can be all the data stored in the migration data source from 0:00 on January 1, 2018 to 1:00 on January 1, 2018. Thus, illustratively, the data in the migration data source can be divided into M data slices.

[0089] It should be noted that when the data is sharded, the data volume corresponding to each data shard (each data shard is a migration subtask) can also be obtained. When the amount of data to be migrated corresponding to the target data shard is greater than the data volume threshold, the migration subtask corresponding to the target data shard can be split into multiple target subtasks so that the amount of data to be migrated corresponding to each target subtask is less than the data volume threshold. For example, when the target data shard (for example: the data shard corresponding to 23:00-24:00 on November 11, 2018) is greater than the data volume threshold, the migration subtask T corresponding to the target data shard can be split to split the migration subtask T into the migration subtask T1 corresponding to 23:00-23:30, and the migration subtask T2 corresponding to 23:30-24:00.

[0090] Furthermore, it can be determined whether the amount of data to be migrated of the data shard corresponding to the migration subtask T1 is less than the data amount threshold, and whether the amount of data to be migrated of the data shard corresponding to the migration subtask T2 is less than the data amount threshold. If both are less than the data amount threshold, the splitting can be stopped. Furthermore, exemplarily, the migration subtask T1 can be executed by the server node S1 of the data migration cluster, and the migration subtask T2 can be executed by the server node S2 of the data migration cluster.

[0091] If the amount of data to be migrated corresponding to any migration subtask is greater than the data volume threshold, the migration subtask can be further split. For example, if the amount of data to be migrated of migration subtask T2 (23:30-24:00) is still greater than the data volume threshold, the migration subtask T2 can be split again to split T2 into target subtasks T2 corresponding to 23:30-23:45. 21 The target subtask T corresponding to 23:45-24:00 22 Similarly, if the amount of data to be migrated of a certain target subtask is still greater than the data amount threshold, the target subtask can continue to be iteratively split until the amount of data to be migrated corresponding to each target subtask is less than the data amount threshold.

[0092] Referring to the above explanation, it can be known that by splitting the migration subtasks whose data volume is greater than the data volume threshold, the migration subtask T can be split into migration subtask T1 and split task T2. 21 And split task T 22 , so that the migration subtask T can be executed alternately by the three server nodes in the data migration cluster (for example, the migration subtask T1 can be executed by the server node S1 of the data migration cluster, and the migration subtask T2 can be executed by the server node S2 of the data migration cluster). 21 , the migration subtask T is executed by the server node S2 of the data migration cluster 22), to avoid the phenomenon of excessive task volume, overload and node crash caused by a single server node executing the migration subtask T, and to ensure the normal operation of each server node in the Tomcat cluster.

[0093] In step S230, when a data migration instruction is received, multiple data shards are migrated to a receiving data source through a migration cluster distribution.

[0094] After the data migration tool is initialized, the user can click the Start Migration button on the configuration page (i.e., send the data migration instruction). Then, the data migration tool can receive the data migration instruction and migrate multiple data shards in the migration data source to the receiving data source through the Tomcat cluster without affecting the normal read and write services of the migration data source. Figure 5 , Figure 5 A schematic diagram showing a data migration method in an exemplary embodiment of the present disclosure, specifically showing a schematic diagram of opening a message queue for a migration data source during data migration, with reference to Figure 5 It can be seen that during the migration of the entire data, the MQ of the migration data source (ES2.1) is always in the open state. This can solve the technical problem in related technologies that requires stopping the read and write business of the migration data source and affecting the normal business processing process, ensure the normal data reading and writing of the migration data source, and optimize the user experience.

[0095] Specifically, when a data migration instruction is received, the data migration tool can call the Tomcat cluster. Multiple server nodes in the Tomcat cluster can concurrently preempt migration subtasks. After preempting the migration subtask T1, the migration status of the data shard corresponding to the migration subtask T1 can be updated to migrating (status=1). Then, it can be queried whether the data already exists in the receiving data source (for example: ES6.3). If the data already exists in the receiving data source, the data can be deleted. Then, the data shard can be migrated to the receiving data source. After the migration is completed, the migration status of each data in the data shard can be updated to migration success (status=2).

[0096] After completing the data migration, you can modify the data migration status. For example, when server node A migrates all data A in a low-version cluster (e.g., ES2.1) to a high-version cluster (e.g., ES6.3), you can modify the migration status of the full data A (e.g., status=2) to confirm that the full data A has been successfully migrated.

[0097] It should be noted that the data in the migrated data source is not deleted. In the actual environment, almost all online data cannot be deleted and can only be archived regularly. Only data older than a few years can be physically deleted to free up space. Therefore, after backing up the data in the migrated data source and migrating it to the receiving data source, the migrated data source still stores all the previous full data.

[0098] Continue to refer Figure 2 In step S240, incremental data is obtained from the migration data source, and it is determined whether the incremental data is newly added data according to the creation timestamp of the incremental data.

[0099] Exemplarily, the creation timestamp and update timestamp of all data in the migration data source may be obtained. When the creation timestamp or update timestamp of a certain data B is after the task start time, it may be determined that the data is incremental data.

[0100] After completing the data migration of the full amount of data, you can find out the incremental data written to the migration data source during the process of migrating the full amount of data. For example, you can determine the data whose creation timestamp is after the start time of the migration task as incremental data, and determine the data whose update timestamp is after the task start time as incremental data.

[0101] After the incremental data is determined, the data volume of the incremental data can be counted or estimated, and then a migration task B can be created (the migration task of the incremental data can only set the start time timeflag, without setting the end time).

[0102] Furthermore, when the amount of incremental data is greater than the first data volume threshold (i.e., if the writing of the migration data is stopped, the writing will be stopped for a long time, affecting the normal operation of the business), the incremental data can be sharded to obtain multiple data shards, and under the premise of maintaining the normal reading and writing business of the migration data source, the migration task is distributedly executed through the data migration cluster to migrate the multiple data shards corresponding to the incremental data to the receiving data source. Therefore, when the amount of data is too large, it is possible to avoid stopping the writing business for too long and affecting the normal query business of users, and maintain the normal operation of the storage cluster during data migration.

[0103] When the amount of incremental data is less than the data volume threshold (that is, if the migration data is stopped, the time required to stop writing is very short, and the impact on users is very small and can be ignored), there is no need to shard the incremental data and stop writing the data read and write business of the migration data source (see Figure 6 , Figure 6A schematic diagram of a method for migrating database data in another exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of closing a message queue of a migration data source during incremental data migration to stop data reading and writing services of the migration data source, and executing a migration task to migrate the incremental data to a receiving data source.

[0104] Furthermore, it can be determined whether the incremental data is new data (i.e., data written to the migration data source during the migration of the full data). Specifically, if the creation timestamp of the incremental data is after the task start time, it can be determined that the incremental data is new data.

[0105] Continue to refer Figure 2 In step S250, when the incremental data is newly added data, the incremental data is migrated to the receiving data source through the migration cluster.

[0106] When it is determined that the incremental data is new data, the incremental data can be directly migrated to the receiving data source through the migration cluster, thereby avoiding omission of the new data and ensuring the integrity of the data.

[0107] In step S260, when the incremental data is not newly added data, the incremental data is used to replace the target data in the receiving data source through the migration cluster.

[0108] When it is determined that the incremental data is not new data (i.e., the creation timestamp of the data is before the task start time, and the update timestamp of the data is after the task start time), it can be determined that the data is not new data, but historical data that has been modified during the migration of the full data. Therefore, the target data that has been migrated to the receiving data source and has the same creation timestamp as the incremental data can be replaced according to the incremental data, and the target data can be deleted. In this way, invalid data that has been changed can be deleted to ensure the validity of the data.

[0109] For example, during the data replacement process, the update timestamps of the incremental data and the target data can be compared according to the optimistic locking mechanism; when the update timestamp of the incremental data is greater than the update timestamp of the target data, the target data in the receiving data source can be replaced by the incremental data. If the update timestamp of the incremental data is less than the update timestamp of the target data, it can be determined that the data replacement has failed.

[0110] It should be noted that when the data migration process is abnormally terminated (for example, the network is disconnected when transmitting data Y), the current interruption time can be recorded, for example, the update timestamp of data Y. When the network connection is restored normally, the data Y in the data shard can be located according to the interruption time, and the data can be migrated from data Y onwards, thereby enabling breakpoint resumption.

[0111] It should be noted that, during the data migration process, the number of data shards that have completed the migration (status = 2) can also be obtained in real time, and the migration progress can be calculated based on the ratio of this number to the total number of data shards. Then, a migration progress bar can be displayed in the configuration page (for example, the executed part and the unexecuted part can be represented by different colors, for example: the executed part is represented by red, and the unexecuted part is represented by white. The specific color can be set according to the actual situation and falls within the scope of protection of this disclosure), or the execution progress of the migration task (for example: 20%) can be displayed in the configuration page, so that relevant users can understand the data migration progress intuitively and in real time, ensuring that the migration process is transparent and monitorable.

[0112] When data migration is interrupted due to system abnormalities (for example, program abnormalities, network abnormalities), an alarm message can also be displayed on the configuration page to facilitate relevant users to quickly troubleshoot the causes of the abnormalities, shorten the problem troubleshooting time, and improve the troubleshooting efficiency and migration efficiency.

[0113] For example, please refer to Fig. 7A and Figure 7B , Fig. 7A A schematic diagram showing a data migration method in another exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of a data migration process when the storage cluster is an ES cluster, and Figure 7B A schematic diagram showing a data migration method in another exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of the data migration process when the storage cluster is Mysql, ES, Mongo, and hbase. Fig. 7A and Figure 7B Specific implementation methods are explained.

[0114] Exemplarily, you can create migration tasks and migration subtasks through Web-config, and set the execution time period of the migration task to be from January 1, 2019 to January 1, 2020, so that each hour's corresponding data shard corresponds to a migration subtask. Then, after the data migration tool establishes a communication connection with the migration data source (ES 2.1) and the receiving data source (ES 6.3), it can communicate with the migration data source through the Transport API interface, and communicate with the receiving data source through the REST API interface. Furthermore, you can first synchronize the history (i.e., synchronize the full amount of historical data), then synchronize the newly added data in the incremental data, and then synchronize the non-newly added data (modified data) in the incremental data.

[0115] For example, Figure 8 The overall flow diagram of the data migration method in an exemplary embodiment of the present disclosure is shown below. Figure 8Specific implementation methods are explained.

[0116] refer to Figure 8 , you can migrate all the data first (i.e. Figure 8 Specifically, you can prepare to receive the data source (i.e. Figure 8 Then, create a migration task (whose execution time is from January 2019 to October 2019) and migration subtasks, and execute the migration task and migration subtasks through the data migration cluster to migrate all data to the receiving data source.

[0117] After migrating all the data, you can compare the creation timestamp and update timestamp of each data in the migration data source to determine the incremental data and migrate the incremental data to the receiving data source. Specifically, you can first prepare a new index for the receiving data source, and then shard the incremental data to create a migration task (the execution time is November 2019, and the termination time is determined by the actual data writing situation in the migration data source), and execute the migration task through the data migration cluster to achieve the migration of incremental data to the receiving data source. After the migration is completed, you can switch the traffic to the receiving data source through the system switch (that is, turn on the MQ of the receiving data source).

[0118] After the incremental data in the migration data source is migrated to the receiving data source, the data reading and writing service of the migration data source can be stopped, and a message MQ (Message Queue, message queue, abbreviated as: MQ) is opened to the receiving data source (exemplarily, Fig. 9 A schematic diagram of a method for migrating a database in an exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of closing a message queue of a migration data source and opening a message queue of a receiving data source during an incremental data migration process to start data reading and writing services of the receiving data source).

[0119] For example, Fig.10 A schematic diagram of a sub-process of a data migration method in an exemplary embodiment of the present disclosure is shown, and the schematic diagram of the sub-process specifically shown includes steps S1001-S1003. Fig.10 Specific implementation methods are explained.

[0120] In step S1001, the number of data items stored in the migration data source is queried.

[0121] Furthermore, under the premise of maintaining the data reading and writing service of the receiving data source and stopping writing the data in the receiving data source (the data migrated from the migrating data source), the number of data items stored in the migrating data source can be queried.

[0122] In step S1002, if the number of data items is greater than the number of data items in the receiving data source, the difference data between the migration data source and the receiving data source is obtained.

[0123] If the number of data items is greater than the current number of data items in the receiving data source, the difference data in the migration data source can be determined based on the data update timestamp. Exemplarily, referring to the explanation of the above step S240, when the creation timestamp or update timestamp of a certain data item C is after the start time timeflag of the migration task of the above incremental data, the data can be determined to be difference data.

[0124] In step S1003, the differential data is updated to the receiving data source by migrating the cluster.

[0125] After determining the above difference data, the difference data can be updated to the receiving data source through the migration cluster. Specifically, it can be determined whether the difference data is new data. If the difference data is new data, the difference data can be directly migrated to the receiving data source and the update timestamp of the data can be changed. If the difference data is not new data, the difference data can be used to replace the data in the receiving data source with the same creation timestamp as the difference data.

[0126] For example, you can also refer to Fig.11 , Fig.11 A schematic diagram of a sub-process of a data migration method in an exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of a sub-process of comparing field values ​​of data and updating inconsistent data in field values ​​when the number of data items in the migration data source is equal to the number of data items in the receiving data source, including steps S1101 to S1104. Fig.11 Specific implementation methods are explained.

[0127] In step S1101, if the number of data items is equal to the number of data items in the receiving data source, the field values ​​of all data in the migration data source are obtained, and the field values ​​of all data in the receiving data source are obtained.

[0128] If the number of data items in the migration data source is equal to the number of data items in the receiving data source, the field value of each data item in the migration data source can be obtained, and the field value corresponding to the field name of each data item in the receiving data source can be obtained.

[0129] In step S1102, field values ​​of the same data are compared to see if they are consistent.

[0130] In this step, the field values ​​of the same data may be compared to see whether they are the same, for example, to see whether the field values ​​of the data M in the migration data source and the data M' in the receiving data source are the same.

[0131] In step S1103, data in the migration data source with inconsistent field values ​​are marked.

[0132] Furthermore, when the field values ​​of data M and data M' are inconsistent, data M can be marked, for example, the migration status of data M is updated to migration failure, that is, status = 3. This facilitates the subsequent rapid retrieval of marked data and the execution of relevant update operations on the data.

[0133] In step S1104, the marked data is updated to the receiving data source.

[0134] Furthermore, the marked data may be updated in the receiving data source. Specifically, the corresponding data in the receiving data source (data with the same creation timestamp as the marked data) may be replaced according to the data in the marked migration data source.

[0135] Exemplarily, a statistical value of the number of the above-mentioned marked data may also be obtained and displayed to monitor the effect of data migration.

[0136] For example, Fig.12 A schematic diagram of a method for migrating a database in an exemplary embodiment of the present disclosure is shown, specifically showing a schematic diagram of an overall process of performing data comparison and updating difference data after completing incremental data migration. Fig.12 Specific implementation methods are explained.

[0137] refer to Fig.12 During the data comparison, the normal read and write services of the receiving data source (ES6.3) can be maintained (MQ is enabled), but the writing of the data to be compared needs to be stopped because the changing data cannot be compared.

[0138] Furthermore, the data in the migration data source (including the creation timestamp and the update timestamp) exists in the form of data shards, and each data shard corresponds to a migration subtask. Furthermore, the Compare task can read a migration subtask, and read the data a contained in the migration subtask from the migration data source, and read the data b contained in the migration subtask from the receiving data source, and compare the field values ​​of data a and data b through equals(a,b) to see if they are exactly the same. If they are not the same, the data a in the migration data source is marked (the migration status is updated to migration failure, that is, status = 3), and the data a is updated to the receiving data source.

[0139] For example, Fig.13 A schematic diagram of a sub-process of a method for migrating a database in an exemplary embodiment of the present disclosure is shown, specifically a schematic diagram of a sub-process of creating a disaster recovery cluster, including steps S1301 to S1303. Fig.13Specific implementation methods are explained.

[0140] In step S1301, a first standby server for migrating a data source is created, and a second standby server for receiving a data source is created.

[0141] In an exemplary embodiment of the present disclosure, a first standby server for migrating a data source and a second standby server for receiving a data source may also be created to enhance the fault tolerance of the system.

[0142] In step S1302, the data in the migration data source is migrated to the first standby server in a timely manner through the migration cluster.

[0143] The data in the migration data source can be migrated to the first standby server regularly through the migration cluster. Specifically, a synchronization task table can be created. For example, refer to Fig.15 , Fig.15 A schematic diagram showing a synchronization task table created in an exemplary embodiment of the present disclosure is shown, with reference to Fig.15 The synchronization task table may include: the index of the data in the migration data source (_index), the type of data (_type), the identification number of the data (_id), the relevance of the data to the query condition (_score), the identification of the first backup server (dest_client), the last synchronization field of the data (last_backup_time), the index of the data in the first backup server (dest_index), and the index type of the data in the first backup server (dest_index_type). Furthermore, the data migration progress can be queried in real time through the data index or other query conditions set by oneself.

[0144] Each task has a last_backup_time field (last synchronization time field), so that the scheduled task synchronizes some of this part of incremental data to the first backup server at regular intervals, and then writes the synchronization start time into this last_backup_time field, and so on and so forth to update this field the next time. In this way, the problem of data being unable to be repaired and obtained when the migration data source goes down can be avoided, ensuring data security.

[0145] In step S1303, the data in the received data source is migrated to the second standby server by the migration cluster in a timely manner.

[0146] You can migrate the data in the receiving data source to the second standby server regularly through the migration cluster. Specifically, only one synchronization task table is needed, which can be continued for reference. Fig.14 , Fig.14A schematic diagram showing a synchronization task table created in an exemplary embodiment of the present disclosure is shown, with reference to Fig.14 The synchronization task table may include: the index of the data in the receiving data source (_index), the type of data (_type), the identification number of the data (_id), the relevance of the data to the query condition (_score), the identification of the second backup server (dest_client), the last synchronization field of the data (last_backup_time), the index of the data in the second backup server (dest_index), and the index type of the data in the second backup server (dest_index_type). Furthermore, the data migration progress can be queried in real time through the data index or other query conditions set by oneself.

[0147] Each task has a last_backup_time field (last synchronization time field), so that the scheduled task synchronizes some of this part of incremental data to the second backup server at regular intervals, and then writes the synchronization start time into this last_backup_time field, and so on and so forth to update this field the next time. In this way, it can avoid the problem that the data cannot be repaired and obtained when the receiving data source crashes, and ensure data security.

[0148] Based on the above technical solutions, on the one hand, the present disclosure enables the concurrent migration of all data through a cluster, providing a general, efficient, simple to operate, rich migration scenarios, and accurate migration data migration solution. Furthermore, it can solve the technical problems in the related technologies that cause the business of migrating data sources to stop writing, the operation time is long, and the migration process is complicated. It can improve the data migration speed without affecting the read and write business of the migration data source, and ensure the orderly progress of the data migration process. Furthermore, it can avoid data omissions and ensure data integrity and validity. On the other hand, it can also avoid the problem that data cannot be repaired or obtained when the receiving data source goes down, thereby ensuring data security.

[0149] The present disclosure also provides a data migration device, Fig.15 A schematic diagram showing the structure of a data migration device in an exemplary embodiment of the present disclosure is shown; Fig.15 As shown, the data migration device 1500 may include a communication module 1501, a data sharding module 1502, a full data migration module 1503, and an incremental data migration module 1504. Among them:

[0150] The communication module 1501 is used to establish a communication connection with the migration data source and the receiving data source.

[0151] The data sharding module 1502 is used to perform sharding processing on the full amount of data in the migration data source to obtain multiple data shards.

[0152] In an exemplary embodiment of the present disclosure, a data slicing module is used to obtain the initial running time of a migration data source, and determine the time span between the current time and the initial running time as a target time period; slice the target time period according to a preset time interval to obtain multiple time slices; and use the data stored in the migration data source in each time slice as a data slice.

[0153] The full data migration module 1503 is used to migrate multiple data shards to the receiving data source through the migration cluster distribution when receiving the data migration instruction.

[0154] In an exemplary embodiment of the present disclosure, each data shard corresponds to a migration subtask; the full data migration module is used to execute the migration subtask in a distributed manner through the migration cluster when receiving a data migration instruction, so as to migrate multiple data shards to a receiving data source.

[0155] In an exemplary embodiment of the present disclosure, the full data migration module is used to split the migration subtask corresponding to the target data shard into multiple target subtasks when the amount of data to be migrated corresponding to the target data shard is greater than the data amount threshold, so that the amount of data to be migrated corresponding to each target subtask is less than the data amount threshold.

[0156] The incremental data migration module 1504 is used to obtain incremental data from the migration data source and determine whether the incremental data is new data based on the creation timestamp of the incremental data; when the incremental data is new data, the incremental data is migrated to the receiving data source through the migration cluster; when the incremental data is not new data, the incremental data is used to replace the target data in the receiving data source through the migration cluster.

[0157] In an exemplary embodiment of the present disclosure, the incremental data migration module is used to compare the update timestamps of incremental data and target data according to the optimistic locking mechanism; when the update timestamp of the incremental data is greater than the update timestamp of the target data, the incremental data is used to replace the target data in the receiving data source.

[0158] In an exemplary embodiment of the present disclosure, the incremental data migration module is used to query the number of data items stored in the migration data source; if the number of data items is greater than the number of data items in the receiving data source, the difference data between the migration data source and the receiving data source is obtained; and the difference data is updated to the receiving data source through the migration cluster.

[0159] In an exemplary embodiment of the present disclosure, the incremental data migration module is used to obtain the field values ​​of all data in the migration data source if the number of data items is equal to the number of data items in the receiving data source, and to obtain the field values ​​of all data in the receiving data source; compare whether the field values ​​of the same data are consistent; mark the data in the migration data source with inconsistent field values; and update the marked data to the receiving data source.

[0160] In an exemplary embodiment of the present disclosure, the incremental data migration module is used to record the interruption time when data migration is interrupted; and to perform breakpoint resume according to the interruption time.

[0161] In an exemplary embodiment of the present disclosure, the incremental data migration module is used to create a first standby server for migrating a data source, and to create a second standby server for receiving a data source; to migrate the data in the migrating data source to the first standby server in a scheduled manner through a migration cluster; and to migrate the data in the receiving data source to the second standby server in a scheduled manner through a migration cluster.

[0162] The specific details of each module in the above data migration device have been described in detail in the corresponding data migration method, so they will not be repeated here.

[0163] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0164] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0165] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0166] In the exemplary embodiments of the present disclosure, a computer storage medium capable of implementing the above method is also provided. A program product capable of implementing the above method of this specification is stored thereon. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0167] refer to Fig.16 As shown, a program product 1600 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0168] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0169] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0171] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0172] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0173] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0174] Refer to the following Fig.17 1700 according to this embodiment of the present disclosure is described. Fig.17 The electronic device 1700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0175] like Fig.17 As shown, the electronic device 1700 is in the form of a general computing device. The components of the electronic device 1700 may include, but are not limited to: the at least one processing unit 1710, the at least one storage unit 1720, a bus 1730 connecting different system components (including the storage unit 1720 and the processing unit 1710), and a display unit 1740.

[0176] The storage unit stores program codes, which can be executed by the processing unit 1710, so that the processing unit 1710 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1710 can perform the following steps: Figure 2As shown in: Step S210, establishing a communication connection with a migration data source and a receiving data source; Step S220, sharding the full amount of data in the migration data source to obtain multiple data shards; Step S230, when receiving a data migration instruction, distributively migrating the multiple data shards to the receiving data source through the migration cluster; Step S240, obtaining incremental data from the migration data source, and determining whether the incremental data is new data based on the creation timestamp of the incremental data; Step S250, when the incremental data is new data, migrating the incremental data to the receiving data source through the migration cluster; Step S260, when the incremental data is not new data, replacing the target data in the receiving data source with the incremental data through the migration cluster; wherein the target data has the same creation timestamp as the incremental data.

[0177] The storage unit 1720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 17201 and / or a cache storage unit 17202 , and may further include a read-only storage unit (ROM) 17203 .

[0178] The storage unit 1720 may also include a program / utility 17204 having a set (at least one) of program modules 17205, such program modules 17205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0179] Bus 1730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0180] The electronic device 1700 may also communicate with one or more external devices 1800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1700, and / or communicate with any device that enables the electronic device 1700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1750. Furthermore, the electronic device 1700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1760. As shown, the network adapter 1760 communicates with other modules of the electronic device 1700 via a bus 1730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0181] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0182] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0183] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data migration method, characterized in that: include: Establishing communication connection with migration data source and receiving data source; Slice the entire amount of data in the migration data source to obtain multiple data slicings; When receiving the data migration instruction, the plurality of data shards are migrated to the receiving data source through the migration cluster distribution; Acquire incremental data from the migration data source, and determine whether the incremental data is newly added data according to a creation timestamp of the incremental data; When the incremental data is newly added data, migrating the incremental data to the receiving data source through the migration cluster; When the incremental data is not newly added data, the incremental data is used by the migration cluster to replace the target data in the receiving data source; wherein the target data has the same creation timestamp as the incremental data; When data migration is interrupted, the interruption time is recorded; and breakpoint resuming is performed according to the interruption time; After replacing the target data in the receiving data source with the incremental data through the migration cluster, the method further includes: Query the number of data items stored in the migration data source; If the number of data items is greater than the number of data items in the receiving data source, obtaining difference data between the migration data source and the receiving data source; The difference data is updated to the receiving data source through the migration cluster.

2. The method according to claim 1, characterized in that The sharding of the full amount of data in the migration data source to obtain multiple data shards includes: Acquire the initial running time of the migration data source, and determine the time span between the current time and the initial running time as the target time period; Slice the target time period according to a preset time interval to obtain multiple time slices; The data stored in the migration data source in each of the time slices is regarded as a data slice.

3. The method according to claim 1, characterized in that Each data shard corresponds to a migration subtask; When receiving the data migration instruction, migrating the multiple data shards to the receiving data source through the migration cluster includes: When a data migration instruction is received, the migration subtask is distributedly executed through the migration cluster to migrate the multiple data shards to the receiving data source.

4. The method according to claim 3, characterized in that The method further comprises: When the amount of data to be migrated corresponding to the target data shard is greater than the data amount threshold, the migration subtask corresponding to the target data shard is split into multiple target subtasks so that the amount of data to be migrated corresponding to each target subtask is less than the data amount threshold.

5. The method according to claim 1, characterized in that The replacing the target data in the receiving data source with the incremental data through the migration cluster includes: Comparing the update timestamps of the incremental data and the target data according to an optimistic locking mechanism; When the update timestamp of the incremental data is greater than the update timestamp of the target data, the incremental data is used to replace the target data in the receiving data source.

6. The method according to claim 1, characterized in that The method further comprises: If the number of data items is equal to the number of data items in the receiving data source, then obtaining the field values ​​of all data in the migration data source, and obtaining the field values ​​of all data in the receiving data source; Compare the field values ​​of the same data to see if they are consistent; Marking the data in the migration data source with inconsistent field values; The marked data is updated into the receiving data source.

7. The method according to claim 1, characterized in that The method further comprises: Creating a first standby server for migrating the data source, and creating a second standby server for receiving the data source; Migrating the data in the migration data source to the first standby server regularly through the migration cluster; and The data in the receiving data source is migrated to the second standby server in a timely manner through the migration cluster.

8. A data migration device, characterized in that: include: A communication module, used to establish a communication connection with a migration data source and a receiving data source; A data sharding module, used to perform sharding processing on the full amount of data in the migration data source to obtain multiple data shards; A full data migration module, configured to migrate the multiple data shards to the receiving data source through a migration cluster distribution when receiving a data migration instruction; An incremental data migration module, used to obtain incremental data from the migration data source, and determine whether the incremental data is newly added data according to the creation timestamp of the incremental data; when the incremental data is newly added data, migrate the incremental data to the receiving data source through the migration cluster; When the incremental data is not newly added data, the incremental data is used by the migration cluster to replace the target data in the receiving data source; wherein the target data has the same creation timestamp as the incremental data; When data migration is interrupted, the interruption time is recorded; and breakpoint resuming is performed according to the interruption time; After the migration cluster uses the incremental data to replace the target data in the receiving data source, the incremental data migration module is further used to: Query the number of data items stored in the migration data source; If the number of data items is greater than the number of data items in the receiving data source, obtaining difference data between the migration data source and the receiving data source; The difference data is updated to the receiving data source through the migration cluster.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data migration method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data migration method according to any one of claims 1 to 7 by executing the executable instructions.

Images