Data migration method, device and electronic equipment

Abstract

The present invention provides a data migration method, device and electronic device, which can be applied to the field of cloud computing. The method includes: analyzing the primary key of each data to be migrated in the source database, determining the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated; when the maximum primary key is not equal to the minimum primary key, determining the block algorithm according to the data type of the data to be migrated; using the block algorithm, the data to be migrated in the data set to be migrated is divided into blocks according to a preset expected number of blocks, and multiple block boundary values ​​are obtained, wherein the field type of the block boundary value corresponds to the data type; the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value is sent to the message queue, so as to use the insert statement to migrate the data in the message queue to the distributed database.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and more specifically, to a data migration method, device and electronic device. Background Art

[0002] As the business develops, the amount of data generated by applications is getting larger and larger. The source database cannot support the business with large data sets, so it is necessary to migrate the data in the source database. In the related technology, the table data that needs to be migrated in the source database is sorted and then migrated. For each data migration, the sorting field and some characteristic fields that can identify the record are recorded.

[0003] In the process of realizing the concept of the present invention, the inventors found that there are at least the following problems in the related art: during data migration, each migrated data must be recorded once, which greatly increases the time consumption of migration and has low migration efficiency. Summary of the invention

[0004] In view of this, the present invention provides a data migration method, apparatus, device, medium and program product.

[0005] One aspect of the present invention provides a data migration method, comprising: analyzing the primary key of each data to be migrated in a source database, and determining the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated; when the maximum primary key is not equal to the minimum primary key, determining a block algorithm according to the data type of the data to be migrated; using the block algorithm, dividing the data to be migrated in the data set to be migrated according to a preset expected number of blocks, and obtaining multiple block boundary values, wherein the field type of the block boundary value corresponds to the data type; sending the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary values ​​to a message queue, so as to migrate the data in the message queue to a distributed database using an insert statement.

[0006] According to an embodiment of the present invention, the above method also includes: when the above maximum primary key is equal to the above minimum primary key, using the above maximum primary key as the above block boundary value; constructing the data to be migrated corresponding to the above block boundary value in the above data to be migrated into a block array with a field type of large integer type; sending the above block array to the above message queue, so as to use the above insert statement to migrate the data in the above message queue to the above distributed database.

[0007] According to an embodiment of the present invention, the above-mentioned data type includes an integer type, and the above-mentioned block algorithm is used to block the data to be migrated in the above-mentioned data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: determining the step size according to the difference between the above-mentioned maximum primary key and the above-mentioned minimum primary key and the above-mentioned preset expected number of blocks; taking the preset expected number of blocks as a quotient of the above-mentioned difference to determine the target remainder; when it is determined that the above-mentioned step size is the first target value, blocking the data to be migrated in the above-mentioned data set to be migrated based on the above-mentioned target remainder and the preset step size to obtain multiple above-mentioned block boundary values; when it is determined that the above-mentioned step size is not the above-mentioned first target value, blocking the data to be migrated in the above-mentioned data set to be migrated according to the above-mentioned target remainder and the above-mentioned preset expected number of blocks to obtain multiple above-mentioned block boundary values.

[0008] According to an embodiment of the present invention, the data to be migrated in the data set to be migrated is divided into blocks according to the target remainder and the preset expected number of blocks to obtain multiple block boundary values, including: defining an integer variable i, wherein the initial value of the integer variable i is assigned to a first variable value; when it is determined that the first variable value is less than the preset expected number of blocks, summing the i-1th block boundary value with the step size to obtain the i-th block boundary value, wherein when i is equal to 1, the i-1th block boundary value is the minimum primary key; when the target remainder is determined, the initial value of the integer variable i is assigned to a first variable value; when the first variable value ... When the target remainder is greater than the first variable value, the i-th block boundary value is incremented to obtain a new i-th block boundary value; when it is determined that the target remainder is less than or equal to the first variable value, the i-th block boundary value is determined to remain unchanged; the integer variable i is incremented, wherein the integer variable i after increment is assigned a second variable value, and the second variable value is greater than the first variable value; when it is determined that the second variable value is less than the preset expected number of blocks, a plurality of i-th block boundary values ​​corresponding to the integer variable i after increment are iteratively determined.

[0009] According to an embodiment of the present invention, the data type also includes a string type. The block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: determining the maximum string length of multiple data to be migrated in the data set to be migrated; using the maximum string length as the number of loops, iteratively converting the characters at each position of the string in the data to be migrated into an integer type to obtain an integer value at each position, wherein the integer value of the position in the string that exceeds the maximum string length is set to a second target value; based on the position of each character in the string, the data to be migrated is converted into an integer type. The character block result is divided into blocks to obtain the character block result of the maximum string length; for each of the above character block results, the minimum integer value and the maximum integer value in the above character block result are determined, wherein the above minimum integer value is the integer value of the position of the character corresponding to the above minimum primary key in the above character block result, and the above maximum integer value is the integer value of the position of the character corresponding to the above maximum primary key in the above character block result; based on the above minimum integer value, the above maximum integer value and the above preset expected number of blocks, the above character block result is divided into blocks to obtain multiple sub-boundary values; the multiple sub-boundary values ​​corresponding to each of the above character strings are combined to obtain multiple block boundary values.

[0010] According to an embodiment of the present invention, the above-mentioned data type also includes a date and time type. The above-mentioned block algorithm is used to block the data to be migrated in the above-mentioned data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: converting the data type of the above-mentioned data to be migrated into a millisecond value of a long integer type to obtain the data to be blocked; blocking the above-mentioned data to be blocked according to the millisecond value corresponding to the above-mentioned maximum primary key and the millisecond value corresponding to the above-mentioned minimum primary key, as well as the above-mentioned preset expected number of blocks to obtain multiple millisecond boundary values; converting the data format of each of the above-mentioned millisecond boundary values ​​into a date and time type to obtain multiple of the above-mentioned block boundary values.

[0011] According to an embodiment of the present invention, the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value is sent to the message queue, so as to migrate the data in the message queue to the distributed database by using the insert statement, including: for each of the block data, determining the block migration information of each of the block data; using the block migration information to determine the insert statement corresponding to the block data; sending the block data to the message queue, so as to call the database connection interface associated with the source database to extract the data in the message queue when migrating the block data; and migrating the block data extracted from the message queue to the distributed database by using the insert statement.

[0012] According to an embodiment of the present invention, the above method also includes: in the process of migrating the above block data to the above distributed database, updating the block migration status in the above block migration information in real time to transmit the above block migration status to the above distributed database; in the above distributed database, deleting the data whose block migration status is migration failure and migration interruption.

[0013] Another aspect of the present invention provides a data migration device, including: a primary key determination module, used to analyze the primary key of each data to be migrated in the source database to determine the maximum primary key and the minimum primary key; an algorithm determination module, used to determine the blocking algorithm according to the data type of the above-mentioned data to be migrated when the above-mentioned maximum primary key is not equal to the above-mentioned minimum primary key; a data blocking module, used to use the above-mentioned blocking algorithm to block the data to be migrated in the above-mentioned data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, wherein the field type of the above-mentioned block boundary value corresponds to the above-mentioned data type; and a data migration module, used to send the block data obtained by dividing the data to be migrated in the above-mentioned data set to be migrated based on the above-mentioned block boundary values ​​to a message queue, so as to migrate the data in the above-mentioned message queue to the distributed database using an insert statement.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0016] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0017] According to an embodiment of the present invention, by determining the maximum primary key and the minimum primary key of the data to be migrated, and using a block algorithm to block the data to be migrated, since the block algorithm is determined according to the data type, the above-mentioned method of blocking the data to be migrated can be applied to different databases. In the process of blocking the data to be migrated, the block boundary value is determined by the block algorithm to send the block data to the message queue. By using the message queue to migrate the block data, the parallel transmission of the block data is realized, which effectively improves the efficiency of the distributed database data migration and makes the data migration have high availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture to which the data migration method and apparatus of the present invention can be applied is shown.

[0020] Figure 2 A flow chart of a data migration method according to an embodiment of the present invention is shown.

[0021] Figure 3 A flow chart of dividing the shaped data into blocks in the data migration method according to an embodiment of the present invention is shown.

[0022] Figure 4 Another flow chart of dividing the shaped data into blocks in the data migration method according to an embodiment of the present invention is shown.

[0023] Figure 5 A schematic diagram of block-by-block parallel migration in a data migration method according to an embodiment of the present invention is shown.

[0024] Figure 6 A schematic diagram of distributed database migration in a data migration method according to an embodiment of the present invention is shown.

[0025] Figure 7 A block diagram of a data migration device according to an embodiment of the present invention is shown.

[0026] Figure 8 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, invention, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0032] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0033] The embodiment of the present invention provides a data migration method, device, equipment, medium and program product. The method includes: analyzing the primary key of each data to be migrated in the source database, determining the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated; when the maximum primary key is not equal to the minimum primary key, determining the block algorithm according to the data type of the data to be migrated; using the block algorithm, the data to be migrated in the data set to be migrated is divided into blocks according to a preset expected number of blocks, and multiple block boundary values ​​are obtained, wherein the field type of the block boundary value corresponds to the data type; the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value is sent to the message queue, so as to use the insert statement to migrate the data in the message queue to the distributed database.

[0034] Figure 1 An exemplary system architecture to which the data migration method and device of the present invention can be applied is shown. It should be noted that: Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0035] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the data migration method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data migration device provided in the embodiment of the present invention can generally be set in the server 105. The data migration method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the data migration device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the data migration method provided in the embodiment of the present invention can also be executed by the terminal device 101, 102, or 103, or can also be executed by other terminal devices different from the terminal device 101, 102, or 103. Correspondingly, the data migration apparatus provided in the embodiment of the present invention may also be arranged in the terminal device 101 , 102 , or 103 , or in other terminal devices different from the terminal device 101 , 102 , or 103 .

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0041] Figure 2 FIG. 2 shows a flow chart of a data migration method according to an embodiment of the present invention. Figure 2 As shown, the method includes operations S210 to S240.

[0042] In operation S210 , the primary key of each to-be-migrated data in the source database is analyzed to determine the maximum primary key and the minimum primary key of the to-be-migrated data set consisting of the plurality of to-be-migrated data.

[0043] In operation S220 , when the maximum primary key is not equal to the minimum primary key, a block partitioning algorithm is determined according to the data type of the data to be migrated.

[0044] In operation S230, a block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain a plurality of block boundary values, wherein the field type of the block boundary value corresponds to the data type.

[0045] In operation S240, the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value is sent to the message queue, so as to migrate the data in the message queue to the distributed database by using an insert statement.

[0046] According to an embodiment of the present invention, in the process of migrating the data to be migrated in the source database to the distributed database, the data to be migrated is divided into blocks, and the divided data is migrated in parallel, thereby achieving high performance of data migration. Specifically, in the process of dividing the data to be migrated into blocks, the primary key of each data to be migrated is analyzed, so as to use the maximum value function to determine the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated. Among them, the maximum primary key and the minimum primary key can be obtained by the maximum value function max() and min() respectively, and the primary key can be represented by id.

[0047] According to an embodiment of the present invention, it is determined whether the maximum primary key is equal to the minimum primary key. If the two are not equal, it indicates that the data to be migrated in the data set to be migrated needs to be segmented. The segmentation algorithm is determined according to the data type of the data to be migrated, where different data types correspond to different segmentation algorithms, but each segmentation algorithm needs to convert the data into an integer for processing.

[0048] According to an embodiment of the present invention, a preset expected number of blocks specified by a user is obtained, and a block algorithm is used to block the data to be migrated in the data set to be migrated according to the preset expected number of blocks, so as to obtain a plurality of block boundary values. Among them, the field type of the block boundary value corresponds to the data type, which can be an integer type or a string type. The data to be migrated in the data set to be migrated is divided based on the block boundary value to obtain block data. The filter conditions are assembled according to the block boundary values ​​of the block data, and finally the block migration information is constructed, so that the block data is stored in the distributed database using the message queue.

[0049] According to an embodiment of the present invention, by determining the maximum primary key and the minimum primary key of the data to be migrated, and using a block algorithm to block the data to be migrated, since the block algorithm is determined according to the data type, the above-mentioned method of blocking the data to be migrated can be applied to different databases. In the process of blocking the data to be migrated, the block boundary value is determined by the block algorithm to send the block data to the message queue. By using the message queue to migrate the block data, the parallel transmission of the block data is realized, which effectively improves the efficiency of the distributed database data migration and makes the data migration have high availability.

[0050] According to an embodiment of the present invention, the data migration method also includes: when the maximum primary key is equal to the minimum primary key, using the maximum primary key as the block boundary value; constructing the data to be migrated corresponding to the block boundary value in the data to be migrated into a block array with a field type of large integer type; sending the block array to the message queue, so as to use the insert statement to migrate the data in the message queue to the distributed database.

[0051] According to an embodiment of the present invention, when the maximum primary key is equal to the minimum primary key, it means that there may be only one piece of data to be migrated, so the maximum primary key can be directly used as the block boundary value. The data to be migrated is constructed into a block array with a field type of large integer type, for example: new BigInteger[]{left,right}. Among them, BigInteger represents a large integer type, left represents the minimum primary key, and right represents the maximum primary key.

[0052] According to an embodiment of the present invention, since there is only one piece of data to be migrated, it is not necessary to divide the data to be migrated into blocks, and the block array is directly sent to the message queue. The block array in the message queue is migrated to the distributed database using an insert statement, thereby completing the migration of the data to be migrated in the source database. By judging the maximum primary key and the minimum primary key, the appropriate data migration method can be selected when facing different data scenarios, effectively improving the data migration efficiency.

[0053] Figure 3 FIG. 2 shows a flow chart of dividing the shaped data into blocks in the data migration method according to an embodiment of the present invention. Figure 3 As shown, the shaping data is divided into blocks including operations S310 to S370.

[0054] In operation S310, it is determined whether the preset expected number of blocks is greater than a preset value.

[0055] In operation S320, a step size is determined according to a difference between the maximum primary key and the minimum primary key and a preset expected number of blocks.

[0056] In operation S330, a preset expected number of blocks is divided by the difference value to determine a target remainder.

[0057] In operation S340, it is determined whether the step size is equal to the first target value.

[0058] In operation S350, the data to be migrated in the data set to be migrated is divided into blocks based on the target remainder and the preset step size to obtain a plurality of block boundary values.

[0059] In operation S360, the data to be migrated in the data set to be migrated is divided into blocks according to the target remainder and the preset expected number of blocks, to obtain a plurality of block boundary values.

[0060] In operation S370, an exception is thrown.

[0061] According to an embodiment of the present invention, the preset value can be set to 1 to determine whether the preset expected number of blocks is greater than 1. If it is not greater than 1, it means that the preset expected number of blocks specified by the user is abnormal, and the data to be migrated cannot be segmented based on the preset expected number of blocks. Therefore, when it is determined that the preset expected number of blocks is not greater than 1, an exception is thrown to prompt the user to specify a preset expected number of blocks that meets the standard. The difference between the maximum primary key and the minimum primary key is obtained, and the step length is calculated according to the preset expected number of blocks, and the step length is the difference divided by the preset expected number of blocks. The specific calculation formula is: (maximum primary key-minimum primary key) / preset expected number of blocks. Determine whether the step length is equal to the first target value, where the first target value is 0. If the step length is equal to 0, it means that the preset expected number of blocks is greater than the difference, so it can be determined that the current preset expected number of blocks is large, and the data to be migrated cannot be segmented according to the current preset expected number of blocks.

[0062] According to an embodiment of the present invention, when the step size is equal to the first target value, the preset expected number of blocks needs to be adjusted. The preset expected number of blocks is divided by the difference to determine the target remainder. The specific calculation formula is: (maximum primary key-minimum primary key)% preset expected number of blocks. The preset expected number of blocks is adjusted to the target remainder so that the migration data can be divided into blocks. Since the target remainder is actually equal to the difference, when the migration data is divided into blocks based on the target remainder, the default step size is the preset step size, that is, 1.

[0063] For example, assuming that the maximum primary key is 21, the minimum primary key is 1, and the preset expected number of blocks is 30, the calculated step length = (21-1) / 30 = 0, so the preset expected number of blocks needs to be adjusted. The calculated target remainder = (21-1)%30 = 20, and the target remainder is used as the new preset expected number of blocks. Therefore, the data to be migrated between the maximum primary key 21 and the minimum primary key 1 is divided into 20 blocks, and the step length of the division is 1.

[0064] According to an embodiment of the present invention, when the step size is not equal to the first target value, it means that the preset expected number of blocks is less than or equal to the difference. Specifically, when it is equal to the difference, it means that there is no target remainder, and the target remainder is 0. Therefore, the data to be migrated between the maximum key value and the minimum key value can be directly partitioned by the step size. When it is less than the difference, it means that there is a target remainder. Therefore, in the process of partitioning the data to be migrated between the maximum key value and the minimum key value based on the step size, the data to be migrated corresponding to the target remainder needs to be added to each block data, so that the data to be migrated is split more completely and meets the preset expected number of blocks.

[0065] Figure 4Another flow chart of dividing the shaped data into blocks in the data migration method according to an embodiment of the present invention is shown. Figure 4 As shown, the shaping data segmentation also includes operations S410 to S480.

[0066] In operation S410, a minimum primary key, a maximum primary key, a preset expected number of blocks, and a target remainder are obtained.

[0067] In operation S420, an integer variable i is defined, wherein an initial value of the integer variable i is assigned as a first variable value. According to an embodiment of the present invention, this can be represented as int i=1 in a programming language.

[0068] In operation S430, it is determined whether the first variable value is less than a preset expected number of blocks.

[0069] In operation S440, the i-1th block boundary value and the step length are summed to obtain the i-th block boundary value, wherein the i-1th block boundary value is the minimum primary key when i is equal to 1. According to an embodiment of the present invention, it can be expressed in a program language as the i-th block boundary value = the i-1th block boundary value + the step length.

[0070] In operation S450, it is determined whether the target remainder is greater than the first variable value.

[0071] In operation S460, the i-th block boundary value is incremented to obtain a new i-th block boundary value. According to an embodiment of the present invention, this can be expressed as i-th block boundary value += 1 in a programming language.

[0072] In operation S470, it is determined that the i-th block boundary value is unchanged. According to an embodiment of the present invention, it can be expressed as i-th block boundary value += 0 in a program language.

[0073] In operation S480, the integer variable i is incremented, wherein the integer variable i after the increment is assigned a second variable value, and the second variable value is greater than the first variable value.

[0074] According to an embodiment of the present invention, it can be expressed as i++ in a programming language. When it is determined that the value of the first variable is less than the preset expected number of blocks, the block boundary value is calculated normally according to the compensation. It is further determined whether the target remainder is greater than the first variable. If it is greater, it means that the target remainder needs to be added to the block boundary value. The i-th block boundary value is recalculated, the integer variable i is incremented, and multiple i-th block boundary values ​​corresponding to the integer variable i after the increment are iteratively determined. That is, the remainders are added one by one starting from the first block boundary value.

[0075] For example: Assuming the maximum primary key is 21, the minimum primary key is 1, and the preset expected number of blocks is 8, the calculated step length = (21-1) / 8=2, and the calculated target remainder = (21-1)%8=4. When partitioning the data to be migrated, first partition according to the calculated step length, that is, partition 1 to 21 into blocks 1, 3, 5, 7, 9, 11, 13, 15, 17. However, there are still four data to be migrated, 18, 19, 20, and 21, so it is necessary to start from the first block boundary value and add these four values ​​by changing the step length. Therefore, the multiple block boundary values ​​after partitioning are: 1, 4, 7, 10, 13, 15, 17, 19, 21.

[0076] According to an embodiment of the present invention, if the target remainder is not greater than the first variable, it means that the target remainder has been added and there is no need to modify the i-th block boundary value. The step size is changed based on the target remainder, thereby adjusting the block boundary value, so that the multiple block data after block division are both complete and meet the preset expected number of blocks.

[0077] According to an embodiment of the present invention, the data type also includes a string type, and a block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: when it is determined that the preset expected number of blocks is greater than the preset value, determining the maximum string length of multiple data to be migrated in the data set to be migrated; using the maximum string length as the number of loops, iteratively converting the characters at each position of the string in the data to be migrated into an integer type to obtain an integer value at each position, wherein the integer value of the position in the string that exceeds the maximum string length is set to a second target value; based on each The data to be migrated is divided into blocks according to the positions of characters, and a character block result with the maximum number of character string lengths is obtained; for each character block result, the minimum integer value and the maximum integer value in the character block result are determined, wherein the minimum integer value is the integer value of the position of the character corresponding to the minimum primary key in the character block result, and the maximum integer value is the integer value of the position of the character corresponding to the maximum primary key in the character block result; based on the minimum integer value, the maximum integer value and the preset expected number of blocks, the character block result is divided into blocks to obtain multiple sub-boundary values; the multiple sub-boundary values ​​corresponding to each character string are combined to obtain multiple block boundary values.

[0078] According to an embodiment of the present invention, when it is determined that the preset expected number of blocks is greater than the preset value, the total number of characters in each string is compared to determine the maximum string length. The maximum string length is used as the number of loops, and the integer values, that is, int values, of the characters at each position of the strings of the minimum primary key and the maximum primary key are taken respectively. The position exceeding the maximum string length is set to a second target value, and the second target value can be 0. As shown in Table 1, the integer value of the last position character in the minimum primary key strings lc1, lc2, lc3, lc4, ..., lcm is set to 0.

[0079] Table 1

[0080]

[0081] According to an embodiment of the present invention, based on the position of each character in the string, the data to be migrated is divided into blocks to obtain a character block result with a maximum number of string lengths. As shown in Table 1, the first character block result is: n11-n12-n13-…, and the second character block result is: n21-n22-n23,…. Determine the minimum integer value and the maximum integer value in each character block result. As can be seen from Table 1, the minimum integer value and the maximum integer value corresponding to the first character block result are the int values ​​of the lc1 and rc1 characters, respectively.

[0082] According to an embodiment of the present invention, based on the minimum integer value, the maximum integer value and the preset expected number of blocks, the character segmentation result is segmented to obtain multiple sub-boundary values. As shown in Table 1, the multiple sub-boundary values ​​in the first character segmentation result are n11, n12, n13, .... After the character segmentation result is segmented, the multiple sub-boundary values ​​corresponding to the character string are combined. As shown in Table 1, the sub-boundary values ​​are assembled by row, and the multiple sub-boundary values ​​are assembled to obtain: n11, n21, n31, n41, ...nm1, ...nn1. The int of the assembled sub-boundary value is converted into char (array) and spliced ​​to obtain the corresponding block boundary value. By splitting and reassembling the character string, the final block boundary value is more reasonable and accurate.

[0083] According to an embodiment of the present invention, the data type also includes a date and time type, and a block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: when it is determined that the preset expected number of blocks is greater than the preset value, the data type of the data to be migrated is converted into a millisecond value of a long integer type to obtain the data to be blocked; according to the millisecond value corresponding to the maximum primary key and the millisecond value corresponding to the minimum primary key, and the preset expected number of blocks, the data to be blocked is blocked to obtain multiple millisecond boundary values; the data format of each millisecond boundary value is converted into a date and time type to obtain multiple block boundary values.

[0084] According to an embodiment of the present invention, when the data type is a date and time type, the minimum primary key and the maximum primary key of the date and time type are converted into millisecond values ​​of a long integer type, that is, the number of milliseconds of a Long type. Then, data whose data type is an integer type is used for block division, and the millisecond boundary value obtained after the block division is converted back to the date and time type, thereby obtaining the block boundary value. By using the same block division method as the integer type for block division, the block boundary value is determined more accurately.

[0085] According to an embodiment of the present invention, block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value is sent to a message queue, so as to migrate the data in the message queue to a distributed database by using an insert statement, including: for each block data, determining the block migration information of each block data; using the block migration information to determine the insert statement corresponding to the block data; sending the block data to the message queue, so as to call the database connection interface associated with the source database to extract the data in the message queue when migrating the block data; and using the insert statement to migrate the block data extracted from the message queue to the distributed database.

[0086] According to an embodiment of the present invention, after the data to be migrated is divided into blocks based on the block boundary value, when the specified block data is migrated, the specific migration process is as follows: the block data is read by the reader and sent to the message queue, and the other end of the message queue writes the acquired data into the table of the specified target distributed database through the writer. When the migration of the specified block data is successful without exception, its status changes to completed. When an exception occurs during the migration process, its status changes to abnormal. The block migration information is written to the distributed database in real time to ensure its high availability.

[0087] According to an embodiment of the present invention, taking the migration of the user table (primary key is id, type is integer type) of the MySQL database (relational database management system) as an example, the specific process of breakpoint-resume data migration is as follows: first, the user table is divided into blocks, and the data of the user table is split with id as the block field. First, the minimum value min(id) and the maximum value max(id) of the id field are obtained, and then the blocks are divided according to the block algorithm corresponding to the integer type. The block boundary value is obtained, and the filtering condition is assembled according to the block boundary value, and finally the block migration information is constructed and stored in the distributed database.

[0088] For example, if the block boundary value of the block field id in the user table is: [a, b, c, …, m, n, …], the constructed block filtering conditions are: a <= id and id < b, b <= id and id < c, …m <= id and id < n, …. Taking the block m <= id and id < n as an example, the block migration information is as follows: {"id": "block id", "splitPk": "id", "srcTable": "test.user", "tarTable": "TEST.USER", "srcColumns": ["id", "username"], "tarColumns": ["ID", "USERNAME"], "srcSplit": "m <= id and id <n","tarSplit": "m <= ID and ID < n","state": "unstart"}。

[0089] The meanings of the fields in the above-mentioned block migration information are as follows: id: the unique identifier of the current block, the primary keyword. splitPk: the block column name of the table from which the source database data is to be migrated out. srcTable: the table name from which the source database data is to be migrated out, in the format of schema name.table name. tarTable: the table name from which the distributed database data is to be migrated in, in the format of schema name.table name. srcColumns: the column name of the table from which the source database data is to be migrated out. tarColumns: the column name of the table from which the distributed database data is to be migrated in. srcSplit: the block migration information of the table from which the source database data is to be migrated out (that is, the where condition). tarSplit: the block migration information of the table from which the distributed database data is to be migrated in, used when re-migrating and cleaning up the block data when the block migration fails. state: the block migration status, there are four states: unstart, running, finished and failed, which are: not started, running, completed and failed.

[0090] According to an embodiment of the present invention, a thread pool thread is started to migrate data for blocks, and each thread of the thread pool corresponds to a block of data in the above table. The thread pool has a fixed number of threads. Whenever a thread completes its block migration task, it will continue to drive the next block task. The detailed process of migrating blocks by a single thread is: first change the state of the block to be migrated to running and write it to distributed storage. The sql for reading data constructed according to the block migration information has the format: select ${srcColumns} from ${srcTable} where ${srcSplit}, where ${srcColumns}, ${srcTable} and ${srcSplit} are variables, corresponding to specific values ​​in the block migration information. The sql for reading related data in the above blocks is: select id, username from test.user where m <= id and id < n.

[0091] According to an embodiment of the present invention, a writer and a reader are constructed through the source database information and the distributed database information. The reader connects to the source database through JDBC (database connection interface) to execute the above SQL query, and the queried data is sent to the message queue. The writer connects to the distributed database through JDBC to retrieve relevant data from the message queue, and calls the SQL statement constructed to insert data for batch insertion. The SQL format for inserting data is: insert into ${tarTable}(${tarColumns}) values ​​(${placeHolders}), where ${tarTable}, ${tarColumns} and ${placeHolders} are variables, ${tarTable} and ${tarColumns} correspond to specific values ​​in the block migration information, and ${placeHolders} is a field placeholder, and there are as many "?" as there are fields. The specific SQL for inserting data in the above block is: insert into TEST.USER(ID, USERNAME) values ​​(?, ?).

[0092] According to an embodiment of the present invention, if the entire read-write insertion is correctly executed, the state is changed to finished and written to the distributed storage. If an exception occurs during the block migration process, the state is changed to failed and written to the distributed storage. When the migration is unexpectedly interrupted, the state of the blocks being migrated is running and written to the distributed storage, indicating that the migration is interrupted. The block migration information is written to the distributed database in real time to ensure its high availability.

[0093] Figure 5 FIG. 2 schematically shows a block-by-block parallel migration diagram of a data migration method according to an embodiment of the present invention. Figure 5 As shown, Table 1, Table 2, Table ... to Table N in the source database are divided into blocks to obtain block data block 1, block 2, block 3 ... block N. Each block data is read and written by the thread pool to complete data synchronization, thereby migrating the data in the source database to the distributed database.

[0094] According to an embodiment of the present invention, the data migration method also includes: in the process of migrating block data to a distributed database, updating the block migration status in the block migration information in real time to transmit the block migration status to the distributed database; in the distributed database, deleting the data whose block migration status is migration failure and migration interruption.

[0095] According to an embodiment of the present invention, the high availability of the migration service is ensured by the master election mechanism of the distributed database. When the standby migration service is started, the blocks with the status of running and failed are obtained, and the writer is called to clean up the incomplete migration block data of the distributed database. The SQL format of the cleanup is: delete from ${tarTable}where ${tarSplit}, where ${tarTable} and {tarSplit} are variables, corresponding to the specific values ​​in the block information. The above block cleanup SQL is: delete from TEST.USER where m <= ID and ID < n. After cleaning this type of data, the database with the block migration status of unstart, running and failed is migrated again.

[0096] Figure 6 FIG. 2 shows a schematic diagram of distributed database migration in a data migration method according to an embodiment of the present invention. Figure 6 As shown, in order to ensure the availability of the data migration service, the migration service is deployed in a distributed data cluster 610 with one master and multiple slaves. The Leader service provides external services and writes relevant data to the distributed database. When the Leader service fails, a new service needs to be elected from the Follower service to be the Leader, which is achieved through the distributed lock of the distributed key-value storage system.

[0097] According to an embodiment of the present invention, all Follower services compete for the same distributed lock and set an expiration time for the distributed lock. Figure 6As shown, each Follower service is deployed on a host, including host 620 and host 630. Only one Follower service will obtain the distribution lock, and the value of this distribution lock is its identifier, and it becomes the Leader service. After other Follower services fail in the competition, they obtain the current Leader service identifier obtained by obtaining the distribution lock and communicate with it. The Leader service needs to continuously renew the distribution lock before it expires to prove that it is healthy. All Follower services monitor whether the lock is still held by the Leader service. If not, they jump to step 1 and continue to let all Follower services compete for the same distribution lock.

[0098] Figure 7 FIG. 4 is a block diagram of a data migration device according to an embodiment of the present invention. Figure 7 As shown, the data migration device 700 includes a primary key determination module 710 , an algorithm determination module 720 , a data block module 730 , and a data migration module 740 .

[0099] The primary key determination module 710 is used to analyze the primary key of each to-be-migrated data in the source database, and determine the maximum primary key and the minimum primary key of the to-be-migrated data set composed of multiple to-be-migrated data.

[0100] The algorithm determination module 720 is used to determine a block partitioning algorithm according to the data type of the data to be migrated when the maximum primary key is not equal to the minimum primary key.

[0101] The data block module 730 is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks using a block algorithm to obtain multiple block boundary values, wherein the field type of the block boundary value corresponds to the data type.

[0102] The data migration module 740 is used to send the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value to the message queue, so as to migrate the data in the message queue to the distributed database by using the insert statement.

[0103] According to an embodiment of the present invention, the data migration device 700 further includes a boundary determination module, an array construction module, and an array migration module.

[0104] The boundary determination module is used to use the maximum primary key as the block boundary value when the maximum primary key is equal to the minimum primary key.

[0105] The array construction module is used to construct the data to be migrated corresponding to the block boundary value in the data set to be migrated into a block array with a field type of a large integer type.

[0106] The array migration module is used to send the block array to the message queue so as to migrate the data in the message queue to the distributed database by using the insert statement.

[0107] According to an embodiment of the present invention, the data partitioning module 730 includes a step size determination submodule, a remainder determination submodule, a first boundary determination submodule and a second boundary determination submodule.

[0108] The step size determination submodule is used to determine the step size according to the difference between the maximum primary key and the minimum primary key and the preset expected number of blocks.

[0109] The remainder determination submodule is used to determine the target remainder by dividing the preset expected number of blocks by the difference.

[0110] The first boundary determination submodule is used to divide the data to be migrated in the data set to be migrated into blocks based on the target remainder and the preset step length when the step length is determined to be the first target value, so as to obtain multiple block boundary values.

[0111] The second boundary determination submodule is used to divide the data to be migrated in the data set to be migrated into blocks according to the target remainder and a preset expected number of blocks when the step size is not the first target value, so as to obtain multiple block boundary values.

[0112] According to an embodiment of the present invention, the second boundary determination submodule includes a variable definition unit, a boundary determination unit, a boundary update unit, a boundary maintenance unit, a variable increment unit and a boundary iteration unit.

[0113] The variable definition unit is used to define an integer variable i, wherein an initial value of the integer variable i is assigned as a first variable value.

[0114] The boundary determination unit is used to sum the i-1th block boundary value and the step size to obtain the i-th block boundary value when it is determined that the first variable value is less than the preset expected number of blocks, wherein when i is equal to 1, the i-1th block boundary value is the minimum primary key.

[0115] The boundary updating unit is used to increment the i-th block boundary value to obtain a new i-th block boundary value when it is determined that the target remainder is greater than the first variable value.

[0116] The boundary maintaining unit is used to determine that the i-th block boundary value remains unchanged when it is determined that the target remainder is less than or equal to the first variable value.

[0117] The variable increment unit is used to increment the integer variable i, wherein the integer variable i after the increment is assigned a second variable value, and the second variable value is greater than the first variable value.

[0118] The boundary iteration unit is used to iteratively determine multiple i-th block boundary values ​​corresponding to the self-incremented integer variable i when it is determined that the second variable value is less than the preset expected number of blocks.

[0119] According to an embodiment of the present invention, the data segmentation module 730 further includes a length determination submodule, a character conversion submodule, a result determination submodule, a shaping determination submodule, a sub-boundary determination submodule, and a boundary combination submodule.

[0120] The length determination submodule is used to determine the maximum string length of multiple data to be migrated in the data set to be migrated.

[0121] The character conversion submodule is used to iteratively convert the characters at each position of the string in the data to be migrated into an integer type using the maximum string length as the number of loops to obtain the integer value of each position, wherein the integer value of the position in the string exceeding the maximum string length is set as the second target value.

[0122] The result determination submodule is used to divide the data to be migrated into blocks based on the position of each character in the string, and obtain the character block result with the maximum string length.

[0123] The integer determination submodule is used to determine the minimum integer value and the maximum integer value in the character block result for each character block result, wherein the minimum integer value is the integer value of the character position corresponding to the minimum primary key in the character block result, and the maximum integer value is the integer value of the character position corresponding to the maximum primary key in the character block result.

[0124] The sub-boundary determination submodule is used to divide the character segmentation result into blocks based on the minimum integer value, the maximum integer value and the preset expected number of blocks to obtain multiple sub-boundary values.

[0125] The boundary combination submodule is used to combine multiple sub-boundary values ​​corresponding to each character string to obtain multiple block boundary values.

[0126] According to an embodiment of the present invention, the data segmentation module 730 further includes a type conversion submodule, a millisecond determination submodule, and a millisecond conversion submodule.

[0127] The type conversion submodule is used to convert the data type of the data to be migrated into a millisecond value of a long integer type to obtain the data to be divided into blocks.

[0128] The millisecond determination submodule is used to divide the data to be divided into blocks according to the millisecond value corresponding to the maximum primary key and the millisecond value corresponding to the minimum primary key, and the preset expected number of blocks, to obtain multiple millisecond boundary values.

[0129] The millisecond conversion submodule is used to convert the data format of each millisecond boundary value into a date and time type to obtain multiple block boundary values.

[0130] According to an embodiment of the present invention, the data migration module 740 includes an information determination submodule, a statement determination submodule, an interface calling submodule, and a statement execution submodule.

[0131] The information determination submodule is used to determine the block migration information of each block data for each block data.

[0132] The statement determination submodule is used to determine the insert statement corresponding to the block data by using the block migration information.

[0133] The interface calling submodule is used to send the block data to the message queue so that when the block data is migrated, the database connection interface associated with the source database is called to extract the data in the message queue.

[0134] The statement execution submodule is used to migrate the block data extracted from the message queue to the distributed database using the insert statement.

[0135] According to the embodiment of the present invention, the data migration device 700 further includes a status updating module and a data deleting module.

[0136] The status update module is used to update the block migration status in the block migration information in real time during the process of migrating the block data to the distributed database, so as to transmit the block migration status to the distributed database.

[0137] The data deletion module is used to delete data in the distributed database whose block migration status is migration failure or migration interruption.

[0138] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0139] For example, any multiple of the primary key determination module 710, the algorithm determination module 720, the data block module 730, and the data migration module 740 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the primary key determination module 710, the algorithm determination module 720, the data block module 730, and the data migration module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, at least one of the primary key determination module 710, the algorithm determination module 720, the data segmentation module 730, and the data migration module 740 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0140] It should be noted that the data migration device part in the embodiment of the present invention corresponds to the data migration method part in the embodiment of the present invention. The description of the data migration device part specifically refers to the data migration method part, which will not be repeated here.

[0141] Figure 8 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present invention is shown. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0143] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0144] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0145] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0147] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0148] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 802 and / or the RAM 803 described above and / or one or more memories other than the ROM 802 and the RAM 803 .

[0149] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data migration method provided by the embodiment of the present invention.

[0150] When the computer program is executed by the processor 801, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0151] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0152] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0154] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data migration method, characterized in that: include: Analyze the primary key of each data to be migrated in the source database to determine the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated; When the maximum primary key is not equal to the minimum primary key, determining a block partitioning algorithm according to the data type of the data to be migrated; Using the block algorithm, the data to be migrated in the data set to be migrated is divided into blocks according to a preset expected number of blocks to obtain a plurality of block boundary values, wherein the field type of the block boundary value corresponds to the data type; Sending the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value to the message queue, so as to migrate the data in the message queue to the distributed database by using an insert statement; In the case where the data type is a string type, the block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: Determine the maximum character string length of multiple data to be migrated in the data set to be migrated; Taking the maximum string length as the number of loops, iteratively converting the characters at each position of the string in the to-be-migrated data into an integer type to obtain an integer value at each position, wherein the integer value at the position in the string exceeding the maximum string length is set as a second target value; Based on the position of each character in the character string, the data to be migrated is divided into blocks to obtain a character block result having the maximum character string length; For each of the character block results, determine the minimum integer value and the maximum integer value in the character block result, wherein the minimum integer value is the integer value of the position of the character corresponding to the minimum primary key in the character block result, and the maximum integer value is the integer value of the position of the character corresponding to the maximum primary key in the character block result; Based on the minimum integer value, the maximum integer value and the preset expected number of blocks, the character block result is divided into blocks to obtain a plurality of sub-boundary values; The multiple sub-boundary values ​​corresponding to each of the character strings are combined to obtain multiple block boundary values.

2. The method according to claim 1, characterized in that: The method further comprises: When the maximum primary key is equal to the minimum primary key, the maximum primary key is used as the block boundary value; The data to be migrated corresponding to the block boundary value in the data set to be migrated is constructed into a block array whose field type is a large integer type; The block array is sent to the message queue, so as to migrate the data in the message queue to the distributed database by using the insert statement.

3. The method according to claim 1, characterized in that The data type includes an integer type, and the block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: Determine a step size according to a difference between the maximum primary key and the minimum primary key and the preset expected number of blocks; Taking the preset expected number of blocks as a quotient of the difference value, and determining a target remainder; When the step length is determined to be the first target value, the data to be migrated in the data set to be migrated is divided into blocks based on the target remainder and the preset step length to obtain a plurality of block boundary values; When it is determined that the step length is not the first target value, the data to be migrated in the data set to be migrated is divided into blocks according to the target remainder and the preset expected number of blocks to obtain a plurality of the block boundary values.

4. The method according to claim 3, characterized in that: The step of dividing the data to be migrated in the data set to be migrated into blocks according to the target remainder and the preset expected number of blocks to obtain a plurality of the block boundary values ​​includes: defining an integer variable i, wherein an initial value of the integer variable i is assigned to a first variable value; When it is determined that the first variable value is less than the preset expected number of blocks, summing the i-1th block boundary value and the step size to obtain the i-th block boundary value, wherein when i is equal to 1, the i-1th block boundary value is the minimum primary key; When it is determined that the target remainder is greater than the first variable value, the i-th block boundary value is incremented to obtain a new i-th block boundary value; In the case where it is determined that the target remainder is less than or equal to the first variable value, determining that the i-th block boundary value remains unchanged; Incrementing the integer variable i, wherein the integer variable i after the increment is assigned a second variable value, and the second variable value is greater than the first variable value; When it is determined that the second variable value is less than the preset expected number of blocks, a plurality of i-th block boundary values ​​corresponding to the self-incremented integer variable i are iteratively determined.

5. The method according to claim 1, characterized in that The data type also includes a date and time type. The block algorithm is used to block the data to be migrated in the data set to be migrated according to a preset expected number of blocks to obtain multiple block boundary values, including: Convert the data type of the data to be migrated into a millisecond value of a long integer type to obtain the data to be divided into blocks; According to the millisecond value corresponding to the maximum primary key and the millisecond value corresponding to the minimum primary key, and the preset expected number of blocks, the data to be divided into blocks to obtain multiple millisecond boundary values; The data format of each of the millisecond boundary values ​​is converted into a date and time type to obtain a plurality of the block boundary values.

6. The method according to claim 1, characterized in that The sending of the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value to the message queue, so as to migrate the data in the message queue to the distributed database by using an insert statement, includes: For each of the block data, determine the block migration information of each of the block data; Determine an insert statement corresponding to the block data using the block migration information; The block data is sent to the message queue, so that when the block data is migrated, a database connection interface associated with the source database is called to extract the data in the message queue; The insert statement is used to migrate the block data extracted from the message queue to the distributed database.

7. The method according to claim 6, characterized in that The method further comprises: In the process of migrating the block data to the distributed database, updating the block migration status in the block migration information in real time to transmit the block migration status to the distributed database; In the distributed database, data whose block migration status is migration failure or migration interruption is deleted.

8. A data migration device, characterized in that: include: A primary key determination module, used to analyze the primary key of each data to be migrated in the source database, and determine the maximum primary key and the minimum primary key of the data set to be migrated composed of multiple data to be migrated; an algorithm determination module, configured to determine a block partitioning algorithm according to a data type of the data to be migrated when the maximum primary key is not equal to the minimum primary key; a data segmentation module, configured to segment the data to be migrated in the data set to be migrated according to a preset expected number of segments by using the segmentation algorithm, and obtain a plurality of segmentation boundary values, wherein the field type of the segmentation boundary value corresponds to the data type; and A data migration module, used for sending the block data obtained by dividing the data to be migrated in the data set to be migrated based on the block boundary value to the message queue, so as to migrate the data in the message queue to the distributed database by using an insert statement; The data segmentation algorithm includes: The length determination submodule is used to determine the maximum string length of multiple data to be migrated in the data set to be migrated; A character conversion submodule, used to use the maximum string length as the number of loops, iteratively convert the characters at each position of the string in the data to be migrated into an integer type, and obtain the integer value of each position, wherein the integer value of the position in the string exceeding the maximum string length is set as the second target value; The result determination submodule is used to divide the data to be migrated into blocks based on the position of each character in the string, and obtain the character block result with the maximum string length; The integer determination submodule is used to determine the minimum integer value and the maximum integer value in the character block result for each character block result, wherein the minimum integer value is the integer value of the position of the character corresponding to the minimum primary key in the character block result, and the maximum integer value is the integer value of the position of the character corresponding to the maximum primary key in the character block result; The sub-boundary determination sub-module is used to divide the character segmentation result into blocks based on the minimum integer value, the maximum integer value and the preset expected number of blocks to obtain multiple sub-boundary values; The boundary combination submodule is used to combine multiple sub-boundary values ​​corresponding to each character string to obtain multiple block boundary values.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images