Data migration method and device

Abstract

The present invention discloses a data migration method and device, and relates to the field of computer technology. A specific implementation of the method includes: calculating the stacking information of the data to be migrated stored on the first storage medium, the stacking information indicating the data identifier of one or more data to be migrated that the first server will collect; storing the stacking information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue; using one or more second servers to obtain data from the second queue and store the data in the second storage medium. This implementation not only realizes the automatic stacking of the data to be migrated, but also realizes the decoupling of data collection and data storage during the data migration process, thereby improving the efficiency of data migration.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data migration method and device. Background Art

[0002] With the rapid development of computer technology and the replacement of old and new technologies, data storage methods are constantly upgraded with the changes in technology. Therefore, it is often necessary to consider migrating the original data to a new storage medium or using a new data structure to store data. At present, the commonly used data migration solutions mainly include the following steps: scanning the original data; converting the original data into a new data structure; storing the original data in a new storage medium. In the process of data migration, only one server is often used for data migration, or the original data is manually divided into piles and then multiple servers are used for data migration based on the divided data.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: when only one server is used for data migration, the data migration speed is slow and time-consuming; when multiple servers are used for data migration, the migration data is manually divided into piles, which easily causes uneven division and wastes server resources; in addition, whether one or multiple servers are used for data migration, it is impossible to dynamically adjust the data migration rate during the data migration process. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data migration method and device, which can not only automatically divide the data to be migrated into piles based on the number of servers and the data scale, but also dynamically adjust the data migration rate during the data migration process.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data migration method is provided, comprising:

[0006] Calculating heap information of the data to be migrated stored on the first storage medium, where the heap information indicates data identifiers of one or more data to be migrated that will be collected by the first server;

[0007] The stacking information is stored in a first queue, so that the first server collects the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and stores the collected data in a second queue;

[0008] One or more second servers are used to obtain data from the second queue and store the data in a second storage medium.

[0009] Optionally, the heap division information corresponding to the data to be migrated is calculated according to the amount of the data to be migrated and the number of the first servers.

[0010] Optionally, the calculating, according to the amount of the data to be migrated and the number of the first servers, the heap information corresponding to the data to be migrated includes:

[0011] Calculating a quotient of the amount of the to-be-migrated data and the number of the first servers to obtain a sub-heap size, where the sub-heap size indicates the amount of the to-be-migrated data to be collected by the first server;

[0012] According to the size of the divided pile, a maximum data identifier and a minimum data identifier corresponding to the to-be-migrated data collected by the first server are determined.

[0013] Optionally, it also includes:

[0014] Determine whether the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers;

[0015] When the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers, the number of the first servers is increased.

[0016] Optionally, using the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data in the second queue includes:

[0017] Using the first server to obtain the sub-pile size and the maximum data identifier and the minimum data identifier corresponding to the to-be-migrated data collected by the first server from the first queue;

[0018] Traversing the data to be migrated to collect the data to be migrated from the first storage medium according to the size of the sub-heap and the maximum data identifier and the minimum data identifier;

[0019] The collected data is converted into message data and pushed to the second queue.

[0020] Optionally, when the stacking information is stored in the first queue, one or more of the first servers are notified by broadcast, so that the first servers obtain the data identifier of the to-be-migrated data from the first queue when receiving the broadcast notification.

[0021] Optionally, it also includes:

[0022] When the first server receives the broadcast notification, it is determined whether the first server is in a data collection state, and the data identifier of the data to be migrated is obtained from the first queue using the first server only when the first server is not in a data collection state.

[0023] Optionally, it also includes:

[0024] After using one or more second servers to obtain data from the second queue, the obtained data structure is converted into a data structure that conforms to the second storage medium to store the data in the second storage medium.

[0025] Optionally, it also includes:

[0026] The data collection rate of the first server is monitored based on the second queue, so as to increase the number of the first servers and recalculate the heap division information when the data collection rate is lower than a threshold collection rate.

[0027] Optionally, it also includes:

[0028] The data storage rate of the second server is monitored based on the second queue, so as to increase the number of the second servers if the data storage rate is lower than a threshold storage rate.

[0029] To achieve the above object, according to another aspect of an embodiment of the present invention, a data migration device is provided, comprising: a heap information calculation module, a heap information storage module, and a data migration module; wherein:

[0030] The heap information calculation module is used to calculate the heap information of the data to be migrated stored on the first storage medium, wherein the heap information indicates the data identifier of one or more migration data to be collected by the first server;

[0031] The heap division information storage module is used to store the heap division information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue;

[0032] The data migration module is used to use one or more second servers to obtain data from the second queue and store the data in a second storage medium.

[0033] Optionally, the heap information calculation module is used to calculate the heap information corresponding to the data to be migrated according to the amount of the data to be migrated and the number of the first servers.

[0034] Optionally, the calculating, according to the amount of the data to be migrated and the number of the first servers, the heap information corresponding to the data to be migrated includes:

[0035] Calculating a quotient of the amount of the to-be-migrated data and the number of the first servers to obtain a sub-heap size, where the sub-heap size indicates the amount of the to-be-migrated data to be collected by the first server;

[0036] According to the size of the divided pile, a maximum data identifier and a minimum data identifier corresponding to the to-be-migrated data collected by the first server are determined.

[0037] Optionally, the heap information calculation module is further used to:

[0038] Determine whether the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers;

[0039] When the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers, the number of the first servers is increased.

[0040] Optionally, using the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data in the second queue includes:

[0041] Using the first server to obtain the sub-pile size and the maximum data identifier and the minimum data identifier corresponding to the to-be-migrated data collected by the first server from the first queue;

[0042] Traversing the data to be migrated to collect the data to be migrated from the first storage medium according to the size of the sub-heap and the maximum data identifier and the minimum data identifier;

[0043] The collected data is converted into message data and pushed to the second queue.

[0044] Optionally, the stacking information storage module is further used to broadcast and notify one or more of the first servers when storing the stacking information in the first queue, so that the first server obtains the data identifier of the data to be migrated from the first queue when receiving the broadcast notification.

[0045] Optionally, it also includes: an acquisition status monitoring module; wherein,

[0046] The collection status monitoring module is used to determine whether the first server is in a data collection state when the first server receives a broadcast notification, and use the first server to obtain the data identifier of the data to be migrated from the first queue only when the first server is not in a data collection state.

[0047] Optionally, the data migration module is further used to, after acquiring data from the second queue using one or more second servers, convert the acquired data structure into a data structure that conforms to the second storage medium, so as to store the data in the second storage medium.

[0048] Optionally, it also includes: a data migration monitoring module; wherein,

[0049] The data migration monitoring module is used to monitor the data collection rate of the first server based on the second queue, so as to increase the number of the first servers and recalculate the heap information when the data collection rate is lower than a threshold collection rate.

[0050] Optionally, the data migration monitoring module is further used to:

[0051] The data storage rate of the second server is monitored based on the second queue, so as to increase the number of the second servers if the data storage rate is lower than a threshold storage rate.

[0052] To achieve the above object, according to another aspect of an embodiment of the present invention, a data migration system is provided, comprising: a data migration device, a first storage medium, a first queue, one or more first servers, a second queue, one or more second servers, and a second storage medium; wherein,

[0053] The data migration device is used to calculate the heap information of the data to be migrated stored on the first storage medium, the heap information indicating the data identifier of one or more data to be migrated to be collected by the first server, store the heap information in a first queue, use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in a second queue, use one or more second servers to obtain data from the second queue and store the data in a second storage medium;

[0054] The first storage medium is used to store the data to be migrated;

[0055] The first queue is used to store the heap division information;

[0056] The first server is configured to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue;

[0057] The second queue is used to store data collected by the first server;

[0058] The second server is used to obtain data from the second queue and store the data in a second storage medium;

[0059] The second storage medium is used to store the data acquired by the second server from the second queue.

[0060] To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided an electronic device for data migration, comprising: one or more processors;

[0061] a storage device for storing one or more programs,

[0062] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the data migration methods described above.

[0063] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the methods described above in the data migration method is implemented.

[0064] The above invention has the following advantages or beneficial effects: the stacking information is calculated based on the number of first servers that collect data and the amount of data to be migrated, thereby realizing automatic stacking of the data to be migrated and the average distribution of the data to be migrated on multiple servers, thereby optimizing the utilization rate of the server; by pushing the data collected by the first server to the message queue, and using the second service to store the data obtained from the message queue on the second storage medium, the decoupling of data collection and data migration is realized; in addition, by adjusting the number of first servers and second servers, the data collection rate and the data migration rate are dynamically adjusted during the entire data migration process, thereby improving the data migration efficiency.

[0065] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0067] Figure 1 is a schematic diagram of the main process of the data migration method according to an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of the main process of another data migration method according to an embodiment of the present invention;

[0069] Figure 3 is a schematic diagram of the main process of another data migration method according to an embodiment of the present invention;

[0070] Figure 4 is a schematic diagram of main modules of a data migration device according to an embodiment of the present invention;

[0071] Figure 5is a schematic diagram of the main structure of a data migration system according to an embodiment of the present invention;

[0072] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0073] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0075] Figure 1 is a schematic diagram of the main process of the data migration method according to an embodiment of the present invention; Figure 1 As shown, the data migration method may specifically include the following steps:

[0076] Step S101: Calculate the heap information of the data to be migrated stored in the first storage medium, where the heap information indicates the data identifiers of one or more data to be migrated that will be collected by the first server.

[0077] The purpose of calculating the heap information is to distribute the data to be migrated to one or more first servers to improve the overall data collection efficiency of the first servers. It is understandable that the data to be migrated can be evenly distributed to one or more first servers, or can be distributed according to actual conditions such as the performance of the first servers.

[0078] In an optional embodiment, the heap information corresponding to the data to be migrated is calculated according to the number of the data to be migrated and the number of the first servers. This embodiment is preferably explained by taking the example of evenly distributing the data to be migrated to one or more first servers. Specifically, the quotient of the number of the data to be migrated and the number of the first servers is calculated to obtain the heap size, and the heap size indicates the number of data to be migrated that the first server will collect; according to the heap size, the maximum data identifier and the minimum data identifier corresponding to the data to be migrated that the first server will collect are determined. That is to say, the number of heaps is determined according to the number of first servers, and the heap size is calculated according to the following formula:

[0079] Heap size = the amount of data to be migrated / the number of first servers - remainder

[0080] The remainder refers to the remainder generated by dividing the amount of data to be migrated by the number of first servers.

[0081] For example, if the number of first servers is 10 and the number of data to be migrated is 10,000, the size of the heap calculated according to the above formula or the amount of data to be collected by each first server is 1,000. On this basis, the data labels of the 10,000 data to be migrated stored on the first storage medium are evenly divided into 10 piles and each pile has 1,000 data identifiers, then the corresponding heap information indicates the data identifier and heap size of each pile of data to be migrated (specifically, see Table 1). It can be understood that, in the case where the data identifier of the data to be migrated is a gradually increasing or gradually decreasing value, in order to simplify the heap information, the heap information corresponding to the data to be migrated can only indicate the maximum data identifier, the minimum data identifier and the heap size in each pile of data to be migrated, without including the data identifier of each pile of data to be migrated, such as the maximum data identifier, the minimum data identifier and the heap size corresponding to the heap information with the heap number 2 in Table 1 below are 1001, 2000, and 1000 respectively.

[0082] Table 1 Example of heap information for data to be migrated

[0083] Pile number Data Identification Heap size 1 1-1000 1000 2 1001-2000 1000 3 2001-3000 1000 …… …… …… 10 9001-10000 1000

[0084] It can be understood that when the remainder in the above formula for calculating the heap size is not zero, when allocating the collected data to be migrated to the first server according to the heap size, there will be a remainder of data to be migrated that cannot be allocated. In this case, consider increasing the number of first servers to collect the remaining data to be migrated.

[0085] Specifically, before using the first server to perform the data collection task, determine whether the amount of data to be migrated is greater than the product of the sub-pile size and the number of the first servers; if the amount of data to be migrated is greater than the product of the sub-pile size and the number of the first servers, increase the number of the first servers. In other words, when the remainder is not zero, one or more first servers can be added to allocate the remaining data to be migrated that cannot be evenly distributed to the first servers to the newly added first servers. For example, if the number of first servers is 10 and the number of data to be migrated is 10008, then according to the above formula, it can be calculated that the sub-pile size or the amount of data to be collected by the first server is 1000, and the corresponding remainder is 8. On this basis, after the data identifiers of the 10008 data to be migrated stored on the first storage medium are divided into 10 piles and each pile has 1000 data identifiers, it can be seen that there are still 8 data identifiers of data to be migrated, and the current 10 first servers all have corresponding 1000 data identifiers of data to be collected. In order to complete the collection of the remaining 8 data to be migrated, it is considered to add a new first server to collect the remaining 8 data to be migrated. In this way, by adding a new first server, the synchronous collection of all data to be migrated is achieved, and the data collection efficiency is guaranteed. It can be understood that in addition to adding a first server, in the actual execution process, the collection of the 8 fresh data to be migrated can also be achieved by allocating the 8 fresh data to be migrated to the existing 10 first servers.

[0086] It is understandable that the first storage medium includes but is not limited to any of the following: hard disk, magnetic disk, USB flash drive, file, database, floppy disk, CD, flash memory, SD card, SM card, memory stick, etc.

[0087] Step S102: storing the partition information into a first queue, using the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data into a second queue.

[0088] The first queue is only used to store the heap information containing the data identifier but not the data to be migrated itself, while the second queue is used to store the data to be migrated collected by the first server from the first storage medium. The first queue or the second queue can be a data queue or a message queue implemented by a message middleware such as Kafka, RabbitMQ, RocketMQ, ActiveMQ, etc. Among them, the data queue refers to a special linear list, which is special in that it only allows deletion operations at the front end of the table and insertion operations at the back end of the table.

[0089] In an optional embodiment, the use of the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data to the second queue includes: using the first server to obtain the sub-pile size and the maximum data identifier and the minimum data identifier corresponding to the data to be migrated that the first server will collect from the first queue; traversing the data to be migrated to collect the data to be migrated from the first storage medium according to the sub-pile size and the maximum data identifier and the minimum data identifier; converting the collected data into message data and pushing it to the second queue. It can be understood that in the case where the sub-pile information indicates all the data identifiers collected by each first server, the corresponding data to be collected can be obtained from the first storage medium according to all the data identifiers.

[0090] When using multiple first servers to perform data collection to improve data collection efficiency, in order to ensure that the first server synchronously receives the message notification of obtaining the data identifier from the first queue, when the stacking information is stored in the first queue, a broadcast notification is sent to one or more of the first servers so that the first server obtains the data identifier of the data to be migrated from the first queue when receiving the broadcast notification.

[0091] On this basis, in order to avoid the same first server processing multiple divided data at the same time, thereby causing uneven distribution of collection tasks, the first server is restricted by recording the working status of the first server, that is, recording whether the first server is in the data collection state. Specifically, when the first server receives a broadcast notification, it is determined whether the first server is in the data collection state, and the data identifier of the data to be migrated is obtained from the first queue using the first server only when the first server is not in the data collection state.

[0092] Step S103: Use one or more second servers to obtain data from the second queue and store the data in a second storage medium.

[0093] In this way, by storing the collected data in the second queue by the first server so that the second server obtains the data from the second queue and stores it in the second storage medium, not only the migration of the data to be migrated between the first storage medium and the second storage medium is achieved, but also the decoupling between data collection and data storage in the entire data migration process is achieved.

[0094] It is understandable that the second storage medium includes but is not limited to any of the following: hard disk, magnetic disk, USB flash drive, file, database, floppy disk, CD, flash memory, SD card, SM card, memory stick, etc., and the types of the first storage medium and the second storage medium can be the same or different, such as the first storage medium and the second storage medium can both be databases, or the first storage medium can be a file and the second storage medium can be a database, etc.

[0095] Based on this, when the data storage structures used by the first storage medium and the second storage medium are different, after using one or more second servers to obtain data from the second queue, the obtained data structure needs to be converted into a data structure that conforms to the second storage medium in order to store the data in the second storage medium.

[0096] Furthermore, in order to dynamically adjust the data acquisition rate and the data storage rate during the data migration process so that the data acquisition rate and the data storage rate match or are as consistent as possible, thereby improving the data migration efficiency, the data acquisition rate of the first server is monitored based on the second queue, so that when the data acquisition rate is lower than the threshold acquisition rate, the number of the first servers is increased and the stacking information is recalculated. Correspondingly, the data storage rate of the second server is monitored based on the second queue, so that when the data storage rate is lower than the threshold storage rate, the number of the second servers is increased. The threshold acquisition rate and the threshold storage rate are pre-set based on the actual number of first servers, the performance of the first server, the number of second servers, the performance of the second server, and the amount of data to be migrated.

[0097] Specifically, taking the second queue as a message queue implemented using message middleware such as ActiveMQ and RabbitMQ as an example, during the data migration process, the data collection rate of the first server can be determined by the message backlog; at the same time, the data storage efficiency of the second server can be determined by the consumption speed of the backlog messages. When the data collection rate is lower than the threshold data collection rate, the number of first servers is increased and the heap information is recalculated to distribute the data to be migrated to more first services; when the data storage rate is lower than the threshold data collection rate, the number of second servers is increased to use more second servers at the same time to obtain data from the second queue and store the obtained data in the second storage medium, so as to achieve data migration between the first storage medium and the second storage medium.

[0098] It is worth noting that in addition to dynamically adjusting the data acquisition rate and the data storage rate by increasing the number of first servers and / or the number of second servers, the data acquisition rate and the data storage rate can also be dynamically adjusted by gradually reducing the number of first servers and / or the number of second servers.

[0099] Based on the above embodiment, the heap division information is calculated based on the number of first servers that collect data and the amount of data to be migrated, thereby realizing automatic heap division of the data to be migrated and the average distribution of the data to be migrated on multiple servers, thereby optimizing the utilization rate of the server; by pushing the data collected by the first server to the message queue, and using the second service to store the data obtained from the message queue on the second storage medium, the decoupling of data collection and data migration is realized; in addition, by adjusting the number of first servers and second servers, the data collection rate and the data migration rate are dynamically adjusted during the entire data migration process, thereby improving the data migration efficiency.

[0100] See also Figure 2 Based on the above embodiment, the embodiment of the present invention provides another data migration method, which may specifically include the following steps:

[0101] Step S201: Calculate the heap information of the data to be migrated stored in the first storage medium, where the heap information indicates the data identifiers of one or more data to be migrated that will be collected by the first server.

[0102] Step S202: storing the stacking information into a first queue.

[0103] Step S203: broadcast and notify one or more of the first servers.

[0104] Step S204, when the first server receives the broadcast notification, it is determined whether the first server is in the data collection state; if it is in the data collection state, the first server is not used to obtain the data identifier of the data to be collected and migrated from the first queue; if it is not in the data collection state, it continues to point to the following step S205. In this way, it can avoid the uneven distribution of collection tasks caused by the same first server processing multiple divided data at the same time, so as to ensure the data migration rate.

[0105] Step S205: Use the first server to obtain the data identifier of the data to be migrated from the first queue, collect the data to be migrated from the first storage medium according to the data identifier, and store the collected data in the second queue.

[0106] Step S206: Use one or more second servers to obtain data from the second queue and store the data in a second storage medium.

[0107] See also Figure 3 Based on the above embodiment, the embodiment of the present invention provides another data migration method, which may specifically include the following steps:

[0108] Step S301: Calculate the heap information of the data to be migrated stored in the first storage medium, where the heap information indicates the data identifiers of one or more data to be migrated that will be collected by the first server.

[0109] Step S302: storing the stacking information into a first queue.

[0110] Step S303: Use the first server to obtain the data identifier of the data to be migrated from the first queue, collect the data to be migrated from the first storage medium according to the data identifier, and store the collected data in the second queue. At the same time, in the process of using the first server to store the collected data in the second queue, perform the following step S304.

[0111] Step S304, monitor the data collection rate of the first server based on the second queue, and determine whether the data collection rate is lower than the threshold collection rate; if the data collection rate is lower than the threshold collection rate, continue to step S305; if the data collection rate is not lower than the threshold collection rate, continue to step S306.

[0112] Step S305, when the data collection rate is lower than the threshold collection rate, increase the number of the first servers and recalculate the heap information. In this way, the data to be migrated can be distributed to more first servers based on the new heap information to increase the rate at which the first servers collect data from the first storage medium.

[0113] Step S306: Use one or more second servers to obtain data from the second queue and store the data in a second storage medium. Meanwhile, in the process of using the second server to store the obtained data in the second storage medium, the following step S307 is performed.

[0114] Step S307, monitor the data storage rate of the second server based on the second queue, and determine whether the data storage rate is lower than the threshold storage rate; if the data storage rate is lower than the threshold storage rate, continue to step S308; if the data storage rate is not lower than the threshold storage rate, continue to step S306.

[0115] Step S308: when the data storage rate is lower than the threshold storage rate, increase the number of the second servers. In this way, more second servers can be used to obtain data from the second queue and store the obtained data in the second storage medium to improve data storage efficiency.

[0116] See also Figure 4 Based on the above embodiment, the embodiment of the present invention provides a data migration device 400, including: a heap information calculation module 401, a heap information storage module 402, and a data migration module 403; wherein,

[0117] The heap information calculation module 401 is used to calculate the heap information of the data to be migrated stored on the first storage medium, where the heap information indicates the data identifier of one or more migration data to be collected by the first server;

[0118] The heap information storage module 402 is used to store the heap information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue;

[0119] The data migration module 403 is configured to use one or more second servers to obtain data from the second queue and store the data in a second storage medium.

[0120] In an optional implementation, the heap information calculation module 401 is used to calculate the heap information corresponding to the data to be migrated according to the amount of the data to be migrated and the number of the first servers.

[0121] In an optional implementation, the calculating, according to the amount of the data to be migrated and the number of the first servers, the heap information corresponding to the data to be migrated includes:

[0122] Calculating a quotient of the amount of the to-be-migrated data and the number of the first servers to obtain a sub-heap size, where the sub-heap size indicates the amount of the to-be-migrated data to be collected by the first server;

[0123] According to the size of the divided pile, a maximum data identifier and a minimum data identifier corresponding to the to-be-migrated data collected by the first server are determined.

[0124] In an optional implementation, the heap information calculation module 401 is further used to:

[0125] Determine whether the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers;

[0126] When the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers, the number of the first servers is increased.

[0127] In an optional implementation, using the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data in the second queue includes:

[0128] Using the first server to obtain the sub-pile size and the maximum data identifier and the minimum data identifier corresponding to the to-be-migrated data collected by the first server from the first queue;

[0129] Traversing the data to be migrated to collect the data to be migrated from the first storage medium according to the size of the sub-heap and the maximum data identifier and the minimum data identifier;

[0130] The collected data is converted into message data and pushed to the second queue.

[0131] In an optional embodiment, the stack information storage module 402 is also used to broadcast and notify one or more of the first servers when the stack information is stored in the first queue, so that the first server obtains the data identifier of the data to be migrated from the first queue when receiving the broadcast notification.

[0132] In an optional implementation, it further includes: an acquisition status monitoring module 404; wherein,

[0133] The collection status monitoring module 404 is used to determine whether the first server is in a data collection state when the first server receives a broadcast notification, and to use the first server to obtain the data identifier of the data to be migrated from the first queue only when the first server is not in a data collection state.

[0134] In an optional embodiment, the data migration module 403 is further used to convert the acquired data structure into a data structure that conforms to the second storage medium after acquiring data from the second queue using one or more second servers, so as to store the data in the second storage medium.

[0135] In an optional implementation, it further includes: a data migration monitoring module 405; wherein,

[0136] The data migration monitoring module is used to monitor the data collection rate of the first server based on the second queue, so as to increase the number of the first servers and recalculate the heap information when the data collection rate is lower than a threshold collection rate.

[0137] In an optional implementation, the data migration monitoring module 405 is further configured to:

[0138] The data storage rate of the second server is monitored based on the second queue, so as to increase the number of the second servers if the data storage rate is lower than a threshold storage rate.

[0139] See also Figure 5 Based on the above embodiments, an embodiment of the present invention provides a data migration system 500, including a data migration device 400, a first storage medium 501, a first queue 502, one or more first servers 503, a second queue 504, one or more second servers 505, and a second storage medium 506; wherein,

[0140] The data migration device 400 is used to calculate the heap information of the data to be migrated stored on the first storage medium, the heap information indicating the data identifier of one or more data to be migrated to be collected by the first server, store the heap information in a first queue, use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in a second queue, use one or more second servers to obtain data from the second queue and store the data in a second storage medium;

[0141] The first storage medium 501 is used to store the data to be migrated;

[0142] The first queue 502 is used to store the heap division information;

[0143] The first server 503 is configured to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue;

[0144] The second queue 504 is used to store the data collected by the first server 503;

[0145] The second server 505 is used to obtain data from the second queue and store the data in a second storage medium;

[0146] The second storage medium 506 is used to store the data obtained by the second server from the second queue.

[0147] Figure 6 An exemplary system architecture 600 is shown to which the data migration method or data migration apparatus according to the embodiment of the present invention can be applied.

[0148] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0149] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0150] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0151] Server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results to the terminal device.

[0152] It should be noted that the data migration method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data migration device is generally disposed in the server 605 .

[0153] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0154] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0155] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0156] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0157] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0158] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be arranged in a processor. For example, they may be described as follows: a processor includes a heap information calculation module, a heap information storage module, and a data migration module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the heap information calculation module may also be described as a "module for calculating the heap information of the data to be migrated stored on the first storage medium."

[0161] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: calculating the stacking information of the data to be migrated stored on the first storage medium, the stacking information indicating the data identifier of one or more data to be migrated to be collected by the first server; storing the stacking information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue; using one or more second servers to obtain data from the second queue and store the data in the second storage medium.

[0162] According to the technical solution of the embodiment of the present invention, the heap division information is calculated based on the number of first servers for collecting data and the amount of data to be migrated, thereby realizing automatic heap division of the data to be migrated and the average distribution of the data to be migrated on multiple servers, thereby optimizing the utilization rate of the servers; by pushing the data collected by the first server to the message queue, and using the second service to store the data obtained from the message queue on the second storage medium, the decoupling of data collection and data migration is realized; in addition, by adjusting the number of first servers and second servers, the data collection rate and the data migration rate are dynamically adjusted during the entire data migration process, thereby improving the data migration efficiency.

[0163] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data migration method, characterized in that: include: Calculating heap information of the data to be migrated stored on the first storage medium, the heap information indicating data identifiers of one or more data to be migrated collected by the first server; wherein the data to be migrated is evenly distributed to the one or more first servers, or the data to be migrated is distributed according to the performance of each first server; The stacking information is stored in a first queue, so that the first server collects the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and stores the collected data in a second queue; One or more second servers are used to obtain data from the second queue and store the data in a second storage medium.

2. The data migration method according to claim 1, characterized in that: According to the amount of the data to be migrated and the number of the first servers, heap division information corresponding to the data to be migrated is calculated.

3. The data migration method according to claim 2, characterized in that: The calculating, according to the amount of the data to be migrated and the amount of the first servers, the heap information corresponding to the data to be migrated includes: Calculating a quotient of the amount of the to-be-migrated data and the number of the first servers to obtain a sub-heap size, where the sub-heap size indicates the amount of the to-be-migrated data to be collected by the first server; According to the size of the divided pile, a maximum data identifier and a minimum data identifier corresponding to the to-be-migrated data collected by the first server are determined.

4. The data migration method according to claim 3, characterized in that: Also includes: Determine whether the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers; When the amount of the to-be-migrated data is greater than the product of the sub-heap size and the number of the first servers, the number of the first servers is increased.

5. The data migration method according to claim 4, characterized in that: The using the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and storing the collected data in the second queue includes: Using the first server to obtain the sub-pile size and the maximum data identifier and the minimum data identifier corresponding to the to-be-migrated data collected by the first server from the first queue; Traversing the data to be migrated to collect the data to be migrated from the first storage medium according to the size of the sub-heap and the maximum data identifier and the minimum data identifier; The collected data is converted into message data and pushed to the second queue.

6. The data migration method according to claim 1, characterized in that: When the heap division information is stored in the first queue, one or more of the first servers are notified by broadcast, so that the first servers obtain the data identifier of the to-be-migrated data from the first queue when receiving the broadcast notification.

7. The data migration method according to claim 6, characterized in that: Also includes: When the first server receives the broadcast notification, it is determined whether the first server is in a data collection state, and the data identifier of the data to be migrated is obtained from the first queue using the first server only when the first server is not in a data collection state.

8. The data migration method according to claim 1, characterized in that: Also includes: After using one or more second servers to obtain data from the second queue, the obtained data structure is converted into a data structure that conforms to the second storage medium to store the data in the second storage medium.

9. The data migration method according to claim 2, characterized in that: Also includes: The data collection rate of the first server is monitored based on the second queue, so as to increase the number of the first servers and recalculate the heap division information when the data collection rate is lower than a threshold collection rate.

10. The data migration method according to claim 9, characterized in that: Also includes: The data storage rate of the second server is monitored based on the second queue, so as to increase the number of the second servers if the data storage rate is lower than a threshold storage rate.

11. A data migration device, characterized in that: include: Heap information calculation module, heap information storage module, data migration module; wherein, The heap information calculation module is used to calculate the heap information of the data to be migrated stored on the first storage medium, wherein the heap information indicates the data identifier of one or more migration data to be collected by the first server; wherein the data to be migrated is evenly distributed to the one or more first servers, or the data to be migrated is distributed according to the performance of each first server; The heap division information storage module is used to store the heap division information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue; The data migration module is used to use one or more second servers to obtain data from the second queue and store the data in a second storage medium.

12. A data migration system, characterized in that: include: A data migration device, a first storage medium, a first queue, one or more first servers, a second queue, one or more second servers, and a second storage medium; wherein, The data migration device is used to calculate the heap information of the data to be migrated stored on the first storage medium, the heap information indicates the data identifier of one or more data to be migrated to be collected by the first server, and store the heap information in the first queue, so as to use the first server to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue, and use one or more second servers to obtain data from the second queue and store the data in the second storage medium; wherein the data to be migrated is evenly distributed to one or more first servers, or the data to be migrated is distributed according to the performance of each first server; The first storage medium is used to store the data to be migrated; The first queue is used to store the heap division information; The first server is configured to collect the data to be migrated from the first storage medium according to the data identifier obtained from the first queue, and store the collected data in the second queue; The second queue is used to store data collected by the first server; The second server is used to obtain data from the second queue and store the data in a second storage medium; The second storage medium is used to store the data acquired by the second server from the second queue.

13. An electronic device for data migration, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images