Data migration method and device

Abstract

The embodiment of the present invention discloses a data migration method and device, which relates to the field of computer technology. A specific implementation of the method includes: constructing a data migration task, the data migration task indicates a target database; obtaining at least two data to be migrated from a source database; dividing the at least two data to be migrated into blocks to form at least two data blocks; according to the at least two data blocks, using one or more threads corresponding to multiple instances in the distributed cluster to execute the data migration task, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database. This implementation adopts a distributed cluster with multiple instances and multiple threads to execute the data migration task, thereby realizing high-concurrency and high-throughput data migration, thereby shortening the migration time and improving the migration efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data migration method and device. Background Art

[0002] Data migration is the process of moving the required data from the source database to the target database.

[0003] In the prior art, data migration is usually achieved by developers through single-machine tasks based on migration requirements. Due to the limited hardware performance of a single machine, it is difficult to achieve efficient data migration when the amount of data to be migrated is large, thereby reducing the efficiency of data migration. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data migration method and device, which uses multi-instance and multi-threading in a distributed cluster to execute data migration tasks, thereby achieving high-concurrency and high-throughput data migration, shortening the migration time, and improving the migration efficiency.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data migration method is provided, which is applied to a distributed cluster and includes:

[0006] Building a data migration task, wherein the data migration task indicates a target database;

[0007] Obtain at least two pieces of data to be migrated from the source database;

[0008] Dividing the at least two data to be migrated into blocks to form at least two data blocks;

[0009] According to the at least two data blocks, the data migration task is performed using one or more threads corresponding to the multiple instances in the distributed cluster, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database.

[0010] Optionally, the number of instances used to execute the data migration task is determined according to the amount of data to be migrated and the load capacity of the distributed cluster.

[0011] Optionally, the duration required to execute the data migration task is determined based on the data volume and the load capacity; when the duration is greater than a preset threshold, an instance for executing the data migration task is dynamically added in the distributed cluster.

[0012] Optionally, the number of threads in each of the instances for executing the data migration task is determined according to the computing capacity of each of the instances.

[0013] Optionally, the number of threads is dynamically adjusted according to the number of data blocks and the computing power of the instance.

[0014] Optionally, dividing the at least two data to be migrated into blocks includes:

[0015] The data to be migrated is divided into blocks according to the data identifier of the data to be migrated, the timestamp of the data to be migrated, the data type of the data to be migrated and / or the business scenario corresponding to the data to be migrated.

[0016] Optionally, before dividing the at least two data to be migrated into blocks, the method further includes:

[0017] According to the source table structure to which the data to be migrated belongs and the target table structure of the target database, the fields corresponding to the data to be migrated are converted.

[0018] According to another aspect of an embodiment of the present invention, a data migration device is provided, comprising: a task construction module, a data acquisition module, a data segmentation module, and a data migration module; wherein:

[0019] A task construction module, used to construct a data migration task, wherein the data migration task indicates a target database;

[0020] A data acquisition module, used for acquiring at least two data to be migrated from a source database;

[0021] A data block module, used for dividing the at least two data to be migrated into blocks to form at least two data blocks;

[0022] The data migration module is used to execute the data migration task according to the at least two data blocks using one or more threads corresponding to the multiple instances in the distributed cluster, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database.

[0023] According to another aspect of an embodiment of the present invention, there is provided an electronic device for data migration, comprising:

[0024] one or more processors;

[0025] a storage device for storing one or more programs,

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the data migration method provided by the present invention.

[0027] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the data migration method provided by the present invention is implemented.

[0028] An embodiment of the above invention has the following advantages or beneficial effects: by dividing at least two data to be migrated obtained from the source database into blocks to form at least two data blocks, using one or more threads corresponding to multiple instances in the distributed cluster to execute the constructed data migration task, the data to be migrated corresponding to the at least two data blocks are migrated to the target data, thereby realizing data migration. By executing the data migration task in a distributed cluster multi-instance multi-threaded manner, high-concurrency and high-throughput data migration is realized, thereby shortening the migration time and improving the migration efficiency.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 is a schematic diagram of the main process of a data migration method according to an embodiment of the present invention;

[0032] Figure 2 is a main flow chart of another data migration method according to an embodiment of the present invention;

[0033] Figure 3 is a main flow chart of another data migration method according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of the architecture of a data migration method according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of main modules of a data migration device according to an embodiment of the present invention;

[0036] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0037] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0039] Figure 1 is a schematic diagram of the main process of a data migration method according to an embodiment of the present invention. Figure 1 As shown, when the data migration method is applied to a distributed cluster, it mainly includes the following steps:

[0040] Step S101: construct a data migration task, wherein the data migration task indicates a target database.

[0041] In an embodiment of the present invention, a data migration task is constructed and deployed in a distributed cluster. The data migration task may include multiple nodes, such as multiple nodes for extracting data, processing and cleaning data, writing the data to be migrated into a target database, and verifying the migration results. Thus, the distributed cluster can be used to execute the data migration task to achieve data migration. The data migration task indicates the target database, which is the database that receives the data to be migrated. Accordingly, the source database in the embodiment of the present invention is the extraction source of the data to be migrated.

[0042] Step S102: obtaining at least two data to be migrated from the source database.

[0043] In the embodiment of the present invention, the data to be migrated may be all the data in the source database, or may be part of the data in the source database. The data volume of the source database may be in the billions, and the data migration method provided by the embodiment of the present invention can achieve the migration of data volume in the billions.

[0044] The data migration method provided by the embodiment of the present invention can be applied to the data migration of the e-commerce retail business system. For example, when the overall architecture of the e-commerce retail business system is upgraded, the data in the many business tables of the business system (source database) before the upgrade needs to be cleaned and processed and migrated to the upgraded new database (target database). In addition, the data migration is sensitive to the migration time and needs to be completed quickly in the shortest possible time to reduce the impact on users during the system architecture upgrade. Therefore, the data migration method provided by the embodiment of the present invention can be used to implement data migration with multi-instance multi-threading in a distributed cluster, thereby improving the data migration efficiency as much as possible.

[0045] In this application scenario, at least two data to be migrated can be obtained according to the set migration conditions. The migration conditions are set according to actual business needs. For example, an order table in the source database has 10 million data. Then the migration conditions can be set to obtain data from one year ago. The migration conditions can also be set to obtain data from one or several specific suppliers. According to the set migration conditions, at least two data to be migrated are obtained from the source database, so that data migration can be achieved using a distributed cluster.

[0046] In order to facilitate the storage of the migration data in the target database and its subsequent use, in an embodiment of the present invention, as Figure 2 As shown, the data migration method may further include the following step S201 after step S102 and before step S103:

[0047] Step S201: converting fields corresponding to the data to be migrated according to the source table structure of the data to be migrated and the target table structure of the target database.

[0048] When the source table structure of the source database is different from the target table structure of the target database, in order to facilitate the storage and subsequent use of the migrated data, the structure of the migrated data needs to be converted during data migration. For example, if the source table structure has 20 fields and the target table structure has 15 fields, the fields corresponding to the data to be migrated in the source table need to be converted into the target table fields during data migration. In addition, before migrating the data to be migrated to the target database, in addition to the above-mentioned structural conversion steps, other steps of processing and cleaning the data may also be included, such as data grouping, data sorting, and data deduplication and gap filling.

[0049] Step S103: Divide the at least two data to be migrated into blocks to form at least two data blocks.

[0050] In the embodiment of the present invention, in order to enable the multi-instance multi-thread parallel migration of the data to be migrated in a distributed cluster with high concurrency, the data to be migrated may be divided into multiple data blocks.

[0051] In the embodiment of the present invention, Figure 3 As shown, the specific implementation of step S103 may include the following steps S301:

[0052] Step S301: Divide the data to be migrated into blocks according to the data identifier of the data to be migrated, the time identifier of the data to be migrated, the data type of the data to be migrated and / or the business scenario corresponding to the data to be migrated.

[0053] For example, the data identifier may be a data ID segment or the name of the data table to which the data to be migrated belongs, the time identifier may be a timestamp of the data, the data type may be a text type, a number type, a date type, or a string type, etc., and the business scenario may be a sales scenario, a supplier supply source scenario, or a cost accounting scenario. Among them, any one of the data identifier, time identifier, data type, and business scenario of the data to be migrated can be used as a segmentation condition.

[0054] For example, when the data to be migrated is divided into blocks according to the data identifier (data ID number segment), multiple data to be migrated corresponding to multiple consecutive ID number segments can be divided into one data block. When the data to be migrated is divided into blocks according to the time identifier, the data to be migrated with the timestamp in the same preset time period is divided into one data block. When the data to be migrated is divided into blocks according to the data type, the data to be migrated of the same data type is divided into one data block. When the data to be migrated is divided into blocks according to the business scenario, the data to be migrated of the same application scenario is divided into one data block. For example, the data to be migrated is divided into blocks according to different suppliers, and the data of supplier A and supplier B correspond to different data blocks respectively. It can be understood that since the amount of data in the data migration process is large, the amount of data corresponding to the same data type or the same application scenario may be large. Therefore, when the data is divided into blocks, a combination of multiple block conditions is generally used to divide the data to be migrated, such as using the combination of data identifier and data type as a block condition, or using the combination of time identifier and business scenario as a block condition, or using the combination of data identifier and business scenario as a block condition, or using the combination of data identifier, data type, time identifier and business scenario as a block condition.

[0055] In an embodiment of the present invention, the specific block conditions can be determined according to the performance parameters of the source database and the size of the data to be migrated. For example, the amount of data in each data block formed after block division is not greater than a preset threshold value, and the preset threshold value is determined according to the performance parameters of the source database. For example, according to different source databases, the preset threshold value of the amount of data in the data block formed after block division can be 2000, 5000 or 10,000. Thus, after determining the amount of data in each data block according to actual needs, combined with the amount of data to be migrated, it can be determined which block condition to use to block the data to determine the number of data blocks. For example, the preset threshold value of the amount of data in the data block after block division of the source database is 2000, and the amount of data to be migrated is 1 million. Then, according to the data id number segment, 2000 consecutive data to be migrated can be divided into one data block, so that the amount of data in each data block formed after segmentation is 2000, thereby segmenting the data to be migrated into 500 data blocks.

[0056] Optionally, at least two data to be migrated are divided into blocks by using a binary method to form at least two data blocks. Specifically, the data volume of the data to be migrated is compared with a set threshold value. If the data volume is greater than the set threshold value, the data to be migrated is divided into two data blocks, and the data volume in each data block is continuously compared with the set threshold value. If the data volume of a data block is still greater than the set threshold value, the data block is continuously divided into two data blocks, and so on, until the data volume of each data block is no greater than the set threshold value.

[0057] When the data to be migrated is divided into blocks, the amount of data in each data block should be made uniform as much as possible. For example, when a data block contains an even number of data pieces, it should be divided into two data blocks with the same amount of data when it is divided. When a data block contains an odd number of data pieces, the data block should be divided into two data blocks with a difference of one data piece. Alternatively, after the total number of data blocks after division is determined according to the performance parameters of the source database, the quotient of the amount of data to be migrated and the total number of data blocks can be used as the amount of data corresponding to each data block according to the amount of data to be migrated, so as to make the amount of data in each data block as uniform as possible. Alternatively, after the set threshold value of the amount of data in each data block after division is determined according to the performance parameters of the source database, a data amount less than the set threshold value is selected as the amount of data corresponding to each data block when the data is divided, so as to divide the data. When there is remaining data at the end, the remaining data can be evenly divided into each of the divided data blocks, so that the amount of data to be migrated in each data block is relatively uniform, for example, as equal as possible or with a difference of less than 2 data pieces, or the remaining data at the end can also be used as a data block.

[0058] In the embodiment of the present invention, the data to be migrated can be segmented according to the ID number segment of the data to be migrated, and after determining the data amount of the data to be migrated in each data block, the data to be migrated with the smallest ID number segment is obtained from the data to be migrated, and a certain amount of data to be migrated is obtained from the data to be migrated in sequence as the segmented data block 1, and then the ID number segment of the data to be migrated with the largest ID number segment in data block 1 is taken as the minimum value, and the data to be migrated with the same amount as data block 1 is obtained from the remaining data to be migrated after removing data block 1 according to the order of ID number segments, as the segmented data block 2, and so on. Among them, the number of data blocks 1 and data blocks 2 is equal or differs by 1 data.

[0059] For example, the amount of data to be migrated is 1000, and the ID number range is 1-1000. It is determined that the amount of data to be migrated in each data block is 80. The data to be migrated with the smallest ID number range is obtained from the data to be migrated, and 80 data to be migrated are obtained in sequence. Then data block 1 corresponds to the data to be migrated with an ID number range of 1-80. Then, the maximum ID number 80 is determined from data block 1. 80 more data to be migrated are obtained in sequence after ID number 80 to form data block 2. Then data block 2 corresponds to the data to be migrated with an ID number range of 81-160. And so on. After obtaining 12 data blocks, there are 40 data to be migrated with an ID number range of 961-1000. The remaining 40 data can be used as data block 13, or the remaining 40 data can be evenly divided into the 12 data blocks that have been segmented. Each data block can be divided into 3 or 4 data, so that the amount of data in every two data blocks differs by at most 1 data.

[0060] In an embodiment of the present invention, in the process of segmenting the data to be migrated, the data to be migrated can be marked according to the data block, that is, the data to be migrated that are segmented into the same data block have the same data block identifier. For example, 1 million data to be migrated are segmented into 500 data blocks, each data block contains 2000 data to be migrated, and the data block identifiers of the 2000 data to be migrated that are segmented into the same data block are the same, and different data blocks have different data block identifiers. The 2000 data to be migrated in data block 1 can be marked as data block 1, and the 2000 data to be migrated in data block 2 can be marked as data block 2.

[0061] In an embodiment of the present invention, after at least two data blocks are obtained by block division, the id number segment range of the data to be migrated corresponding to each data block is recorded in the task table, that is, the task table includes the id number segment range of the data to be migrated corresponding to each data block, and the task table may also include a block condition. When block division is performed according to different business scenarios, each task table includes a block condition and a data id number segment range. For example, when the data to be migrated of supplier A is divided into blocks according to the id number, the block condition and the id number segment range with the migrated data corresponding to each data block are recorded in the task table.

[0062] Step S104: according to the at least two data blocks, use one or more threads corresponding to the multiple instances in the distributed cluster to execute the data migration task, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database.

[0063] In an embodiment of the present invention, the distributed cluster includes multiple instances, each instance corresponds to one or more threads, and each instance corresponds to a Docker.

[0064] In the embodiment of the present invention, the configuration of each instance may be different. For example, when the configuration of the instance is low, it may be a Docker with 1 or 2 CPU cores. Corresponding to the difference in the configuration of each instance, the corresponding number of threads may also be different. For example, a high-configuration instance may have more corresponding threads than a low-configuration instance.

[0065] In an embodiment of the present invention, the number of instances for executing the data migration task is determined according to the amount of data to be migrated and the load capacity of the distributed cluster. The load capacity of the distributed cluster may be the sum of the computing power of each instance in the distributed cluster. For example, when the load capacity of the distributed cluster is high and the amount of data to be migrated is also large, it may be determined that the number of instances for executing the data migration task is large.

[0066] In the embodiment of the present invention, the number of threads used to execute the data migration task in each instance is determined according to the computing capacity of each instance.

[0067] Optionally, the computing power of each instance can be determined by the number of CPU cores of the instance, and the number of threads used to execute the data migration task in the instance can be determined based on the number of CPU cores of the instance. Furthermore, the number of threads that can be enabled in each instance = 2*number of CPU cores + 1. For example, for an instance with 2 CPU cores, the number of threads that can be enabled in the instance is 2*2+1=5.

[0068] In the embodiment of the present invention, the number of threads is dynamically adjusted according to the number of data blocks and the computing power of the instance. When executing a data migration task, the computing power of each instance may be different, and the efficiency of migration may be different. When the number of data blocks is small, the number of threads can be reduced, and when the amount of data is large, the number of threads can be increased to achieve dynamic adjustment of the number of threads.

[0069] In an embodiment of the present invention, the computing capacity of each instance is the maximum amount of data that can be migrated by the instance per unit time; the load capacity of a distributed cluster can be the maximum amount of data that can be migrated by the distributed cluster per unit time, or it can be the sum of the maximum amount of data that can be migrated by each instance in the distributed cluster per unit time.

[0070] If the computing power of each instance in the distributed cluster is known, the load capacity of the distributed cluster can be determined. Then, according to the amount of data to be migrated, the number of instances used to perform the data migration task can be determined.

[0071] Furthermore, the duration required to execute the data migration task is determined based on the data volume and the load capacity; when the duration is greater than a preset threshold, an instance for executing the data migration task is dynamically added in the distributed cluster.

[0072] In an embodiment of the present invention, when performing data migration, it is generally necessary to shorten the migration time as much as possible to reduce the impact on users during data migration. By setting a preset threshold for the migration time, when the data migration time is less than the preset threshold, the impact on users is small. After the load capacity of the distributed cluster and the amount of data to be migrated are determined, the time required to perform the data migration task can be determined. When the time required to perform the data migration task is less than the preset threshold, the use of a distributed cluster can achieve efficient data migration; when the required time is greater than the preset threshold, the data migration task can be performed by dynamically adding instances to the distributed cluster to shorten the time required for data migration as much as possible.

[0073] For example, the time required to perform the data migration task can be determined based on the computing power corresponding to all instances in the distributed cluster, or the time required to perform the data migration task can be determined based on the computing power corresponding to some instances in the distributed cluster. When the time calculated based on some instances in the distributed cluster is less than the preset threshold, the instances that perform the data migration task can be some or all instances in the distributed cluster; when the required time is greater than the preset threshold, and the throughput of using the distributed cluster to perform the data migration task is low, high-concurrency and high-throughput data migration can be achieved by dynamically adding one or more instances to the distributed cluster. Among them, the added instances can be one or more low-configuration instances to reduce the cost of expansion. Among them, the low-configuration instance can be a Docker with a CPU core and 256M memory.

[0074] When hardware configuration is limited, high-concurrency and high-throughput data migration can be achieved by simply expanding the capacity of low-configuration instances in the distributed cluster, thereby shortening the migration time and improving the migration performance.

[0075] In the embodiment of the present invention, the distributed cluster uses a multi-instance and multi-threaded approach to execute data migration tasks, thereby achieving high-concurrency and high-throughput data migration, shortening the migration time, and improving the migration performance.

[0076] In an embodiment of the present invention, the data migration task is deployed in a distributed cluster and can be deployed in each instance in the distributed cluster. The data migration task includes: extracting the data to be migrated; processing and cleaning the extracted data to be migrated; writing into the target database; and verifying the migration effect.

[0077] By building the above four nodes into a general data migration task and deploying it in a distributed cluster, repetitive development work can be reduced when migrating different business tables or for different migration needs. Specifically, when migrating different business tables, you can directly configure the field mapping, table structure format, target database and other task parameters of the source database and target database in the data migration task, and then execute the data migration task according to the data to be migrated to achieve data migration, without having to write scripts for each business table, each business scenario or different hardware configurations, thereby avoiding repetitive development work during the data migration process and improving data migration efficiency.

[0078] In one embodiment of the present invention, when executing a data migration task, each instance extracts the data to be migrated from the source database according to the task table. Each task in the task table corresponds to the data to be migrated with the same data block identifier. Each instance extracts the data to be migrated with the same data block identifier according to the task in the task table. Each instance can extract one or more tasks. The number of threads can be dynamically adjusted according to the number of extracted tasks. For example, the first task in the task table corresponds to the data to be migrated with the ID number range of 1-10000 in the order table A, and the second task corresponds to If the id number range of order table A is 10001-20000, instance 1 can extract the data to be migrated with an id number range of 1-10000 according to the task table, and instance 2 can extract the data to be migrated with an id number range of 10001-20000 according to the task table. Of course, if multiple threads in instance 2 are active and instance 1 has idle threads, instance 1 can extract the data to be migrated with an id number range of 1-10000 and the data to be migrated with an id number range of 10001-20000 according to the task table.

[0079] like Figure 4The figure shows a schematic diagram of the architecture of a data migration method according to an embodiment of the present invention. The data to be migrated in the source database is divided into blocks according to different block conditions. For example, when dividing the data into blocks according to the suppliers in the business scenario, the block conditions 1, 2 and 3 are the order table data of suppliers A, B and C respectively. Then, the order table data of suppliers A, B and C are divided into blocks according to the data id segment to obtain data blocks. Each data block corresponds to the data to be migrated in an id segment range. For example, data block 1 of order table A corresponds to the data to be migrated in an id segment range of 1-10000, data block 2 corresponds to the data to be migrated in an id segment range of 10001-20000 of order table A, data block 3 corresponds to the data to be migrated in an id segment range of 20001-30000..., Docker 1, Docker 2, Docker 3, Docker 4, Docker 5...Docker in the distributed cluster n (n is a natural number greater than 5) uses a round-robin method to extract the data to be migrated in a certain ID number range after block division, and each Docker uses one or more threads to perform data migration tasks to migrate the data to the target database.

[0080] The process of executing the data migration task may include: each instance uses one or more threads to process and clean the extracted data to be migrated (such as converting the fields of the data to be migrated into the fields of the target database), and then writes the processed and cleaned data to be migrated into the target database. By comparing the data to be migrated of the data block with the migrated data of the target database, the verification result of the data to be migrated of the data block can be obtained.

[0081] The data migration method provided in the embodiment of the present invention constructs the data migration task as a general task, and does not need to be repeatedly developed for different business tables and migration requirements. The data migration task is executed by distributed cluster multi-instance multi-threading, so as to achieve high-concurrency and high-throughput data migration, shorten the migration time, and improve the migration performance. For example, 20 Dockers with 2 cores each are used to migrate more than 500 million data in the source database. The number of business form table fields is more than 90, the amount of data in each data block is 2,000, the thread pool size is 5, and the CPU usage rate of single-thread batch processing is 90%. In the case of complex data cleaning logic, the migration of more than 500 million data is completed within 4 hours. The migration time is short, which significantly improves the migration efficiency. After adding low-configuration instances or increasing the number of threads, the migration efficiency can be further improved.

[0082] Figure 5 is a schematic diagram of main modules of a data migration device according to an embodiment of the present invention, such as Figure 5As shown, the data migration device 500 includes: a task construction module 501, a data acquisition module 502, a data block module 503, and a data migration module 504; wherein,

[0083] A task construction module 501 is used to construct a data migration task, wherein the data migration task indicates a target database;

[0084] A data acquisition module 502 is used to acquire at least two data to be migrated from a source database;

[0085] A data block module 503 is used to block the at least two data to be migrated to form at least two data blocks;

[0086] The data migration module 504 is used to execute the data migration task according to the at least two data blocks using one or more threads corresponding to the multiple instances in the distributed cluster to migrate the data to be migrated corresponding to the at least two data blocks to the target database.

[0087] In the embodiment of the present invention, the task construction module 501 is further used to: convert the fields corresponding to the data to be migrated according to the source table structure to which the data to be migrated belongs and the target table structure of the target database.

[0088] In an embodiment of the present invention, the data segmentation module 503 is further used to segment the data to be migrated according to the data identifier of the data to be migrated, the time identifier of the data to be migrated, the data type of the data to be migrated and / or the business scenario corresponding to the data to be migrated.

[0089] In the embodiment of the present invention, the data migration module 504 is further used to: determine the number of instances used to perform the data migration task according to the amount of data to be migrated and the load capacity of the distributed cluster. It is further used to: determine the time required to perform the data migration task according to the amount of data and the load capacity; when the time is greater than a preset threshold, dynamically increase the instance used to perform the data migration task in the distributed cluster.

[0090] In the embodiment of the present invention, the data migration module 504 is further used to: determine the number of threads for executing the data migration task in each of the instances according to the computing capacity of each of the instances.

[0091] In the embodiment of the present invention, the data migration module 504 is further used to dynamically adjust the number of threads according to the number of data blocks and the computing capacity of the instance.

[0092] Figure 6An exemplary system architecture 600 is shown to which the data migration method or data migration apparatus according to the embodiment of the present invention can be applied.

[0093] like Figure 6 As shown, the system architecture 600 may include multiple servers 601, 602, 603, 604 and a network 605. The network 605 is used to provide a medium for communication links between the multiple servers 601, 602, 603, 604. The network 605 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0094] Servers 601, 602, 603, and 604 may be servers that provide various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices. The backend management server may analyze and process received data such as data acquisition requests, and feed back corresponding data to the terminal device.

[0095] It should be noted that the data migration device provided in the embodiment of the present invention is generally set in the servers 601 , 602 , 603 , and 604 .

[0096] It should be understood that Figure 6 The number of networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required.

[0097] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing an embodiment of the present invention. Figure 7 The computer shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0098] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0100] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0101] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a task construction module, a data acquisition module, a data segmentation module, and a data migration module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the task construction module may also be described as a "module for constructing a data migration task".

[0104] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: constructing a data migration task, the data migration task indicates a target database; obtaining at least two data to be migrated from a source database; dividing the at least two data to be migrated into blocks to form at least two data blocks; according to the at least two data blocks, using one or more threads corresponding to multiple instances in the distributed cluster to execute the data migration task, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database.

[0105] According to the technical solution of the embodiment of the present invention, a data migration task is first constructed, at least two data to be migrated obtained from the source database are divided into blocks to form at least two data blocks, and the constructed data migration task is executed by using one or more threads corresponding to multiple instances in the distributed cluster, and the data to be migrated corresponding to the at least two data blocks are migrated to the target database, thereby realizing data migration. The embodiment of the present invention realizes high-concurrency and high-throughput data migration by executing the data migration task in a distributed cluster multi-instance multi-threaded manner, thereby shortening the migration time and improving the migration performance.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data migration method, characterized in that: Applicable to distributed clusters, including: Building a data migration task, wherein the data migration task indicates a target database; Obtain at least two pieces of data to be migrated from the source database; Dividing the at least two data to be migrated into blocks to form at least two data blocks; According to the at least two data blocks, one or more threads corresponding to the multiple instances in the distributed cluster are used to execute the data migration task, so as to migrate the data to be migrated corresponding to the at least two data blocks to the target database; The method further includes: determining the number of instances for executing the data migration task according to the amount of the data to be migrated and the load capacity of the distributed cluster; According to the amount of data to be migrated and the load capacity of the distributed cluster, the number of instances used to execute the data migration task is determined, including: according to the amount of data and the load capacity, the duration required to execute the data migration task is determined; when the duration is greater than a preset threshold, the instances used to execute the data migration task are dynamically added in the distributed cluster.

2. The method according to claim 1, characterized in that The number of threads in each of the instances for executing the data migration task is determined according to the computing capability of each of the instances.

3. The method according to claim 2, characterized in that The number of threads is dynamically adjusted according to the number of data blocks and the computing capability of the instance.

4. The method according to claim 1, characterized in that: The dividing the at least two data to be migrated into blocks includes: The data to be migrated is divided into blocks according to the data identifier of the data to be migrated, the time identifier of the data to be migrated, the data type of the data to be migrated and / or the business scenario corresponding to the data to be migrated.

5. The method according to claim 1, characterized in that Before dividing the at least two data to be migrated into blocks, the method further includes: According to the source table structure to which the data to be migrated belongs and the target table structure of the target database, the fields corresponding to the data to be migrated are converted.

6. A data migration device, characterized in that: include: Task construction module, data acquisition module, data segmentation module, data migration module; among them, A task construction module, used to construct a data migration task, wherein the data migration task indicates a target database; A data acquisition module, used for acquiring at least two data to be migrated from a source database; A data block module, used for dividing the at least two data to be migrated into blocks to form at least two data blocks; A data migration module, configured to execute the data migration task according to the at least two data blocks by using one or more threads respectively corresponding to the multiple instances in the distributed cluster, so as to migrate the data to be migrated respectively corresponding to the at least two data blocks to the target database; The data migration module is further used to: determine the number of instances used to execute the data migration task according to the amount of the data to be migrated and the load capacity of the distributed cluster; The data migration module is further used to: determine the time required to execute the data migration task based on the data volume and the load capacity; when the time is greater than a preset threshold, dynamically add an instance for executing the data migration task in the distributed cluster.

7. An electronic device for data migration, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images