Data migration method, device, electronic device and readable storage medium

Abstract

The present application provides a data migration method, device, electronic device and readable storage medium, which relate to the field of computer technology. The method includes: obtaining data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records; based on a balanced allocation method, according to the preset number of threads, the size of occupied space of each data table to be migrated and the number of records, multiple export file blocks are determined, wherein the number of multiple export file blocks is an integer multiple of the preset number of threads, and the number of data records in each export file block is an integer; through concurrent preset number of threads, the data records corresponding to the multiple export file blocks are migrated. In this way, by using a concurrent balanced processing strategy and making full use of the concurrent performance of the database, the data migration efficiency can be greatly improved and a large amount of migration time can be reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data migration method, device, electronic device and readable storage medium. Background Art

[0002] As the system runs longer and longer, the amount of data in the database becomes increasingly large. When data migration is required, it takes a long time. After testing a database with a capacity of 100G, even in a mainstream efficient database environment, it often takes about 10 hours to perform a single migration. In an operating environment that is highly sensitive to server maintenance time, a fast and efficient migration strategy is particularly important. Therefore, how to quickly migrate data has become a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0003] The embodiments of the present application provide a data migration method, device, electronic device and readable storage medium, which can improve data migration efficiency.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data migration method, including:

[0006] Obtaining data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records;

[0007] Based on a balanced allocation method, multiple export file blocks are determined according to the preset number of threads, the space occupied by each data table to be migrated, and the number of records, wherein the number of the multiple export file blocks is an integer multiple of the preset number of threads, and the number of data records in each export file block is an integer;

[0008] The data records corresponding to the multiple export file blocks are migrated through concurrent threads of the preset number of threads.

[0009] In a second aspect, an embodiment of the present application provides a data migration device, including:

[0010] An information acquisition module, used to obtain data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records;

[0011] A processing module, configured to determine a plurality of export file blocks based on a balanced allocation method according to a preset number of threads, a space occupied by each data table to be migrated, and a number of records, wherein the number of the plurality of export file blocks is an integer multiple of the preset number of threads, and the number of data records in each of the export file blocks is an integer;

[0012] The migration module is used to migrate the data records corresponding to the multiple export file blocks through the concurrent preset number of threads.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the data migration method described in any one of the aforementioned embodiments.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data migration method as described in any one of the aforementioned embodiments is implemented.

[0015] The data migration method, device, electronic device and readable storage medium provided in the embodiments of the present application are based on a balanced allocation method, according to the preset number of threads, the occupied space size of each data table to be migrated and the number of records, the data records in the above data table to be migrated are divided into multiple export file blocks, the number of export file blocks is an integer multiple of the preset number of threads, and the number of data records in each export file block is an integer; then, the data records corresponding to each of the multiple export file blocks are migrated through concurrent preset number of threads. In this way, the data records corresponding to each export file block can be determined in a balanced and automatic manner without manual operation, and then the multiple export file blocks are migrated concurrently using threads, thereby improving the data migration efficiency by utilizing the concurrent performance of the database, and saving a lot of migration time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a data migration method provided in an embodiment of the present application;

[0019] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S120;

[0020] Figure 4 for Figure 3 A schematic flow chart of the sub-steps included in sub-step S121;

[0021] Figure 5 A schematic diagram of data migration provided in an embodiment of the present application;

[0022] Figure 6 A block diagram of a data migration device provided in an embodiment of the present application.

[0023] Icons: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - data migration device; 210 - information acquisition module; 220 - processing module; 230 - migration module. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0026] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0027] There are two existing data migration solutions: The first is to use hardware to mirror the source database and use the mirror database as a relay to migrate the database; the second is to split the migration file of a single database into small pieces of data for migration.

[0028] The cost of using a mirror database for migration is too high, and as the database capacity grows exponentially, the migration cost will also increase.

[0029] Migrating a single database migration file in blocks requires manual intervention in the file block size. For example, some data tables are manually grouped as one block, and other data tables are manually grouped as one block. Moreover, when the database grows dynamically, the data tables are still manually grouped into several blocks based on the previous export situation, which cannot be processed automatically. For example, when performing an export, the staff will combine the export time of each block last time and the manually estimated growth of the database from the last export to the current export, and manually distribute the data tables to be exported into several blocks again.

[0030] In response to this situation, the embodiments of the present application provide a data migration method, device, electronic device and readable storage medium, which can flexibly and adaptively split the data records in the data table that needs to be migrated into multiple export files in a balanced manner, and perform data migration on the export file blocks through multiple concurrent threads. In this way, there is no need for human intervention in the file block size. At the same time, multiple threads can be used to migrate multiple export file blocks determined based on a balanced distribution method, which can improve the data migration speed, save migration time, and save a lot of hardware migration costs.

[0031] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0032] Please refer to Figure 1 , Figure 1 A block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be, but is not limited to, a computer, a server, etc. The electronic device 100 may include a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0033] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0034] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, the memory 110 stores a data migration device 200, and the data migration device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the data migration device 200 in the embodiment of the present application, that is, implementing the data migration method in the embodiment of the present application.

[0035] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0036] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0037] Please refer to Figure 2 , Figure 2 The following is a flow chart of a data migration method provided in an embodiment of the present application. The method can be applied to the electronic device 100. The specific flow of the data migration method is described in detail below. The method may include steps S110 to S130.

[0038] Step S110: obtaining data table information of each data table to be migrated in at least one data table to be migrated.

[0039] In this embodiment, the data table to be migrated is a data table that needs to be migrated, that is, the object for data migration. The data table to be migrated can be determined by receiving an input operation, or according to a specific rule, or by other methods, which are not specifically limited here.

[0040] The number of the data tables to be migrated may be at least one, which may be determined by actual conditions. When the data tables to be migrated are determined, the data table information of each data table to be migrated may be obtained. The data table information of a data table to be migrated may include the data table name, the size of the occupied space, and the number of records. The occupied space size in the data table information indicates the size of the space occupied by the corresponding data table when it is stored; the number of records in the data table information indicates the total number of data records included in the corresponding data table.

[0041] Step S120 , based on a balanced allocation method, a plurality of export file blocks are determined according to a preset number of threads, the space occupied by each data table to be migrated, and the number of records.

[0042] When the data tables to be migrated are determined and the data table information of each data table to be migrated is obtained, the data records included in the determined data tables to be migrated can be evenly distributed to multiple export file blocks based on a balanced distribution method according to the preset number of threads, the size of the space occupied by each data table to be migrated, and the number of records. The number of the multiple export file blocks determined is an integer multiple of the preset number of threads, so that it is convenient to use the concurrent threads of the preset number of threads to perform data migration and improve the efficiency of data migration. The number of data records in each of the export file blocks is an integer, which is convenient for ensuring the integrity of the data records.

[0043] The preset number of threads may be set according to the maximum number of concurrent threads allowed by the target database to which the data is to be migrated. The preset number of threads may be less than or equal to the maximum number of concurrent threads allowed by the target database. The preset number of threads may be greater than or equal to 2. The preset number of threads may be manually configured.

[0044] Step S130: Migrate data records corresponding to the multiple exported file blocks through concurrent threads of the preset number of threads.

[0045] When a plurality of export file blocks are automatically determined, the preset number of concurrent threads of the target database may be used to migrate data records corresponding to the plurality of export file blocks.

[0046] The embodiment of the present application is based on a balanced allocation method, and automatically determines multiple export file blocks according to the preset number of threads, the size of the space occupied by each data table to be migrated, and the number of records, and then uses concurrent multi-threading to migrate data for the multiple export file blocks, thereby reducing the time of manual intervention and avoiding the situation where the export time is long due to the large difference in the size of each export file block set during manual intervention. The embodiment of the present application can also flexibly adapt to database growth, without the need to manually intervene in the file block size based on historical export conditions and estimated database growth conditions when the database grows.

[0047] Optionally, in this embodiment, when the data table to be migrated is determined, the data table name of the data table to be migrated can be obtained, and then based on the data table name of each data table to be migrated, the occupied space size and number of records of each data table to be migrated can be obtained from the metadata database where the data table to be migrated is located. In this way, the data table information of each data table to be migrated can be obtained.

[0048] As a possible implementation method, the table names of the data tables to be migrated can be stored in TableNameList, and then the TableNameList is traversed to retrieve and obtain the occupied space size, number of records, etc. of each data table to be migrated from the source database to obtain the table information of each data table to be migrated.

[0049] When the data table information of each data table to be migrated is obtained, Figure 3 The method shown in the figure determines a plurality of export file blocks. The plurality of export file blocks include the first file block and / or the second file block. Figure 3 , Figure 3 for Figure 2 Schematic diagram of the flow of sub-steps included in step S120. In this embodiment, step S120 may include sub-steps S121 to S123.

[0050] Sub-step S121: determining a first data table and / or a second data table in the at least one data table to be migrated.

[0051] In this embodiment, the determined data table to be migrated can be divided into a first data table and a second data table. The space occupied by each of the first data tables is larger than the space occupied by each of the second data tables, and the data table to be migrated other than the first data table in the at least one data table to be migrated is the second data table. It can be understood that the determined data table to be migrated may include the first data table and the second data table, or may only include the first data table or the second data table, which is determined by the division strategy used and the actual situation of the data table to be migrated.

[0052] Alternatively, as a possible implementation, Figure 4 The first data table is determined in the manner shown. Figure 4 , Figure 4 for Figure 3 Schematic diagram of the flow of sub-steps included in sub-step S121. In this embodiment, sub-step S121 may include sub-steps S1211 to S1213.

[0053] Sub-step S1211, obtaining the total occupied space size of the at least one data table to be migrated.

[0054] Sub-step S1212, calculating an average value according to the total occupied space size and the preset number of threads.

[0055] Sub-step S1213, comparing the occupied space size of each to-be-migrated data table with the average value, and taking the to-be-migrated data table whose occupied space size is larger than the average value as the first data table.

[0056] In this method, the total occupied space size of at least one data table to be migrated can be calculated based on the occupied space size of each data table to be migrated. Then, the total occupied space size is divided by the preset number of threads to obtain an average value. Next, the occupied space size of each data table to be migrated is compared with the average value in turn. If the occupied space size of the data table to be migrated is greater than the average value, the data table to be migrated can be used as the first data table; if the occupied space size of the data table to be migrated is not greater than the average value, the data table to be migrated is not used as the first data table, but the data table to be migrated is used as the second data table. In this way, the first data table and the second data table can be determined in combination with the actual situation of the data table to be migrated this time.

[0057] Sub-step S122, based on a balanced allocation method, determines a plurality of first file blocks according to the preset number of threads and the number of records in each of the first data tables.

[0058] In this embodiment, for each first data table, the data records in the first data table can be evenly distributed to the first file blocks of the preset number of threads according to the number of records in the first data table. That is, the determined first data tables are traversed, and when a first data table is traversed, the number of records in the first data table is divided by the preset number of threads, so as to calculate the number of records of the first data table in each first file block, and then the corresponding number of data records are distributed to the corresponding first file blocks. In this way, a first data table that occupies a lot of space can be distributed to multiple first file blocks, and a first data table can be migrated using multiple threads later, thereby improving the migration efficiency of the first data table.

[0059] Sub-step S123, based on a balanced allocation method, a plurality of second file blocks are determined according to the preset number of threads, the occupied space size of each second data table, and the number of records.

[0060] In this embodiment, the data table information may further include the size of space occupied by a single record. A second file block may be determined in sequence according to the preset number of threads, the size of space occupied by each second data table, and the number of records. That is, based on the determined second data table, a part of the data is first divided into a second file block, and a second file block can be determined; and then the remaining part of the data is divided into another second file block, and a second second file block can be determined, and so on, multiple second file blocks are determined in sequence.

[0061] As a possible implementation, in order to achieve balanced allocation, a sum calculation may be performed based on the occupied space size of each second data table, and then the calculated result is divided by the preset thread data, and the obtained result is used as the second file block capacity.

[0062] For each second data table, the occupied space size of the second data table can be compared with the remaining space size of the current second file block. That is, the determined second data tables are traversed, and when a second data table is traversed, the occupied space size of the second data table is compared with the remaining space size of the current second file block. The current second file block is the second file block currently used to allocate data records in the order of 1-N. For example, when allocating the first second data table, the current second file block is the second file block 1; when allocating the second data table other than the first one, the current second file block may be the second file block 1 or the second file block 2.

[0063] The remaining space size of the current second file block is calculated based on the capacity of the second file block, the number of data records stored in the current second file block, and the corresponding space size occupied by a single record. For example, in the case where data records have been allocated in the current second file block, the data table name of the data table where the existing data records in the current second file block are located can be obtained, and then the space size occupied by a single record corresponding to the existing data records in the current second file block can be obtained in combination with the data table name; then, the occupied space size of the current second file block is calculated based on the space size occupied by a single record of each data record and the number of stored data records; finally, the remaining space size of the current second file block is calculated based on the occupied space size of the current second file block and the capacity of the second file block.

[0064] In the case that the occupied space size of the second data table is not greater than the remaining space size of the current second file block, the data records in the second data table may be allocated to the current second file block.

[0065] In the case where the occupied space size of the second data table is larger than the remaining space size of the current second file block, the number of allocatable records can be calculated based on the occupied space size of a single record of the second data table and the remaining space size; the data records of the allocatable number of records in the second data table are allocated to the current second file block. After allocatable number of data records in the second data table are allocated to the current second file block, if the second data table includes data records that have not been allocated, switch to the next second file block, and repeat the above allocation processing operation to allocate the data records that have not been allocated in the second data table to the next second file block.

[0066] This makes it easier to export the second data table if there are any problems later.

[0067] In this embodiment, when the multiple exported file blocks include the first file block and the second file block, the first file block and the second file block corresponding to the thread can be migrated simultaneously through each thread. In the migration process, the first file blocks corresponding to each of the preset number of threads are concurrently migrated to achieve concurrent migration of the first data table. The first file block is used to export to a large-capacity first data table, and the second file block is used to export to a small-capacity second data table.

[0068] Optionally, the preset number of threads is the maximum number of concurrent threads allowed by the target database. In this way, the concurrent performance of the target database can be fully utilized to greatly improve the data migration efficiency.

[0069] Optionally, the first file block and the second file block may include an export command for exporting specific data records. A preset number of threads execute the export command to migrate corresponding data records to the target database.

[0070] Combine the following Figure 5 , the above data migration method is illustrated by examples.

[0071] First, the data table name of the data table to be migrated is stored in TableNameList. TableNameList includes the data table name of the data table to be migrated.

[0072] Configure the maximum number of concurrent threads ThreadN allowed by the target database (that is, the preset number of threads mentioned above). This value is used to determine the number of blocks for the migration data export file later.

[0073] The adaptive data processing module traverses TableNameList. When it traverses to a data table name, it retrieves the space occupied by the table corresponding to the data table name, the number of records (i.e., the number of records), etc. from the source database, and stores the information in TableVolumeList. It can also obtain the space occupied by a single record of each data table to be migrated and save it in TableVolumeList. It can also calculate the total space occupied by the migrated data based on the space occupied by each data table to be migrated, and save it in TableVolumeList.

[0074] like Figure 5 As shown, the data table information of each data table to be migrated can be obtained, and the data table information of each data table to be migrated can be saved in TableVolumeList. The data table information of a data table to be migrated can include the data table name (TableName), the space occupied (TableSize), the number of records (RecordCount), the space occupied by a single record (RecordSize), etc. For example, the data table information of the data table TableName1 to be migrated can include the data table name TableName1, the space occupied by the data table TableName1 to be migrated TableSize, the number of records (RecordCount), the space occupied by a single record (RecordSize), etc. The total space occupied is Figure 5 TotalSize in.

[0075] Calculate the average value: DumpSize1 = total space occupied by migration data / ThreadN.

[0076] Traverse TableVolumeList to find the table to be migrated whose single table occupies a space larger than DumpSize1, store the table name of the table to be migrated in BigTableList, and mark the migrated table as BigTable in TableVolumeList. For example, in TableVolumeList, add BigTableTag to the table information of the table to be migrated.

[0077] For example, if the space occupied by table1 is larger than DumpSize1, you can Figure 5As shown, the data table name of table1 is stored in BigTableList, and BigTableTag is added to the data table information of table1 in TableVolumeList. The BigTableList includes the data table name of the data table to be migrated of BigTable, that is, the data table name of the first data table included in BigTableList.

[0078] Next, traverse the BigTableList, obtain the number of records in each BigTable from TableVolumeList, and then divide the blocks based on the number of records. When traversing to a BigTable, obtain the number of records in the BigTable from TableVolumeList, then divide the number of records by ThreadN to calculate the number of records in each export file block of the table; then generate an export command based on the calculated number of records. The export command is used to store the table data in independent ThreadN export file blocks.

[0079] For example, for the data table BigTable1 to be migrated, divide the number of records in BigTable1 by ThreadN, and the value obtained is the number of data records of BigTable1 included in each export file block. Then generate an export command to divide the corresponding number of data records in BigTable1 into the corresponding export file blocks: BigTable1SqlFile1, BigTable1SqlFile2, ..., BigTable1SqlFileN. BigTable1SqlFile1 means that the corresponding number of data records of BigTable1 are stored in export file block 1. Similarly, for the data table BigTable2 to be migrated, the following export commands can be generated: BigTable2SqlFile1, BigTable2SqlFile2, ..., BigTable2SqlFileN. Through the above processing, the file segmentation of BigTable can be completed, and the BigTable export file can be obtained, and the BigTable export file includes the above export command.

[0080] Traverse TableVolumeList again to obtain the space occupied by the remaining tables, that is, the space occupied by the tables in TableVolumeList that do not carry BigTableTag. Sum the space occupied by the remaining tables and divide the sum by ThreadN to obtain the average value DumpSize2.

[0081] In the order of export file blocks 1-ThreadN, one export file block is used as the current file block. Traverse TableVolumeList. When traversing to a table that does not carry BigTableTag, compare the occupied space size of the data table to be migrated with the remaining space size of the current file block. Among them, the remaining space size of the current file block is calculated based on DumpSize2 and the occupied space size. The occupied space size is calculated based on the number of data records of the non-BigTable table stored in the current file block and the corresponding space size occupied by a single record.

[0082] When the occupied space size of the data table to be migrated is not greater than the remaining space size of the current file block, an export command for allocating all data records of the data table to be migrated to the current file block is generated.

[0083] If it is greater than, the number of allocable records is calculated according to the space occupied by a single record of the data table to be migrated and the remaining space size, and an export command is generated to allocate the data records of the allocable number in the data table to be migrated to the current file block, and when the data table to be migrated includes unallocated data records, the next file block is switched to the current file block, and then the above process is repeated to generate an export command to allocate the unallocated data records in the second data table to the next file block.

[0084] That is, when it is greater than, the remaining space size of the current file block is counted, the number of records of the table that can be stored in the remaining space size is calculated, and the export command of the partial records is generated. The export command is used to allocate the partial records to the current export block, and switch to the next export file block, and generate the export command to allocate the remaining records to it. And so on, until the export commands of all non-BigTable data tables in TableVolumeList are completed. In this way, a small-capacity Table export file can be obtained, which includes the generation commands for allocating data records of non-BigTable data tables to each file block: TableSqlFile1, TableSqlFile2, ..., TableSqlFileN. TableSqlFile1 means that some data records are allocated to file block 1.

[0085] When the target database needs to import data, that is, when the data in the data table to be migrated is to be migrated to the target database, the adaptive data processing module can automatically read the TableVolumeList and BigTableList, and deliver each balanced exported block file (that is, the above-mentioned small-capacity Table export file and BigTable export file) to all threads of the target database, so that all threads of the target database can concurrently and efficiently import the data records of the data table to be migrated into the target database.

[0086] like Figure 5 As shown, DBThread1 imports the corresponding data records of BigTable1, ..., BigTableN and the small-capacity table into the target database based on BigTable1SqlFile1, ..., BigTableNSqlFile1, and TableSqlFile1. In this way, multiple threads simultaneously execute the import of a large-capacity table, which can improve the migration efficiency of the large-capacity table.

[0087] In this embodiment, the data table to be migrated is divided into a large-capacity table and a small-capacity table according to the size of the space occupied by the data table to be migrated, and then different strategies are adopted to generate a small-capacity export file TableSqlFile and a large-capacity export file BigTableSqlFile, so as to improve the export efficiency of the large-capacity table. In addition, when generating the export file based on the data table to be migrated, the capacity of the file block and the data records of the data table corresponding to the file block are automatically and evenly allocated according to the maximum number of threads of the target database, without human intervention. Finally, the export file is delivered to all threads of the target database, so that all threads of the target database execute the import synchronously, and the large batch migration data is imported with maximum performance, balance, and concurrency. Through the above-mentioned concurrent balancing processing strategy, the concurrent performance of the target database is fully utilized, the data migration efficiency is greatly improved, and a lot of migration time is saved. After actual online formal data testing, the time efficiency can be improved by about 60%, and a lot of hardware migration costs are saved for the data migration business, bringing potential economic benefits.

[0088] In order to execute the corresponding steps in the above embodiments and various possible methods, a data migration device 200 is provided below. Optionally, the data migration device 200 may adopt the above Figure 1 The device structure of the electronic device 100 is shown. Figure 6 , Figure 6The block diagram of the data migration device 200 provided in the embodiment of the present application. It should be noted that the basic principle and technical effect of the data migration device 200 provided in the present embodiment are the same as those of the above embodiment. For the sake of brief description, for the parts not mentioned in the present embodiment, reference can be made to the corresponding contents in the above embodiment. The data migration device 200 may include: an information acquisition module 210, a processing module 220 and a migration module 230.

[0089] The information acquisition module 210 is used to obtain data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records.

[0090] The processing module 220 determines a plurality of export file blocks based on a balanced allocation method according to a preset number of threads, the occupied space size of each data table to be migrated, and the number of records, wherein the number of the plurality of export file blocks is an integer multiple of the preset number of threads, and the number of data records in each of the export file blocks is an integer.

[0091] The migration module 230 migrates the data records corresponding to the multiple exported file blocks through concurrent threads of the preset number of threads.

[0092] Optionally, in this embodiment, the multiple exported file blocks include a first file block and / or a second file block, and the processing module 220 is specifically used to: determine the first data table and / or the second data table in the at least one data table to be migrated, wherein the space occupied by each of the first data tables is larger than the space occupied by each of the second data tables, and the data table to be migrated other than the first data table in the at least one data table to be migrated is the second data table; based on a balanced distribution method, determine multiple first file blocks according to the preset number of threads and the number of records of each of the first data tables; based on a balanced distribution method, determine multiple second file blocks according to the preset number of threads, the space occupied by each of the second data tables and the number of records.

[0093] Optionally, in this embodiment, the processing module 220 is specifically used to: for each first data table in turn, according to the number of records in the first data table, evenly distribute the data records in the first data table to the preset number of thread first file blocks.

[0094] Optionally, in this embodiment, the data table information also includes the space occupied by a single record, the number of the plurality of second file blocks is the preset number of threads, and the processing module 220 is specifically used to: calculate the capacity of the second file block according to the preset number of threads and the space occupied by each second data table; sequentially for each second data table, when the space occupied by the second data table is not greater than the remaining space of the current second file block, allocate the data records in the second data table to the current second file block, wherein the remaining space of the current second file block is calculated according to the capacity of the second file block, the number of data records stored in the current second file block, and the space occupied by the corresponding single record; when the space occupied by the second data table is greater than the remaining space of the current second file block, calculate the number of allocatable records according to the space occupied by a single record of the second data table and the remaining space, allocate the data records of the allocatable number in the second data table to the current second file block, and switch to the next second file block when the second data table includes data records that have not been allocated, so as to allocate the data records that have not been allocated in the second data table to the next second file block.

[0095] Optionally, in this embodiment, the processing module 220 is specifically used to: obtain the total occupied space size of the at least one data table to be migrated; calculate an average value based on the total occupied space size and the preset number of threads; compare the occupied space size of each data table to be migrated with the average value, and use the data table to be migrated whose occupied space size is greater than the average value as the first data table.

[0096] Optionally, in this embodiment, the multiple exported file blocks include a first file block and a second file block, and the migration module 230 is specifically used to: simultaneously migrate the first file block and the second file block corresponding to the thread through each thread, wherein, during the migration process, the preset number of threads are used to concurrently migrate the corresponding first file blocks to achieve concurrent migration of the first data table.

[0097] Optionally, in this embodiment, the preset number of threads is the maximum number of concurrent threads allowed by the target database.

[0098] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0099] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data migration method is implemented.

[0100] In summary, the embodiments of the present application provide a data migration method, device, electronic device and readable storage medium. Based on a balanced distribution method, according to the preset number of threads, the occupied space size of each data table to be migrated and the number of records, the data records in the above data table to be migrated are divided into multiple export file blocks, the number of export file blocks is an integer multiple of the preset number of threads, and the number of data records in each export file block is an integer; then, the data records corresponding to each of the multiple export file blocks are migrated through concurrent preset number of threads. In this way, the data records corresponding to each export file block can be determined in a balanced and automatic manner without manual operation, and then the multiple export file blocks can be migrated concurrently using threads, thereby improving the data migration efficiency by utilizing the concurrent performance of the database, and saving a lot of migration time.

[0101] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0103] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0104] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data migration method, characterized in that: include: Obtaining data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records; Based on a balanced allocation method, multiple export file blocks are determined according to the preset number of threads, the space occupied by each data table to be migrated, and the number of records, wherein the number of the multiple export file blocks is an integer multiple of the preset number of threads, and the number of data records in each export file block is an integer; Migrating data records corresponding to the multiple export file blocks through concurrent threads of the preset number of threads; The multiple export file blocks include the first file block and / or the second file block, and the balanced allocation method is based on the preset number of threads, the occupied space size of each data table to be migrated, and the number of records to determine the multiple export file blocks, including: Determine a first data table and / or a second data table in the at least one data table to be migrated, wherein the space occupied by each of the first data tables is greater than the space occupied by each of the second data tables, and the data tables to be migrated other than the first data tables in the at least one data table to be migrated are the second data tables; Based on a balanced allocation method, a plurality of first file blocks are determined according to the preset number of threads and the number of records in each of the first data tables; Based on a balanced allocation method, a plurality of second file blocks are determined according to the preset number of threads, the occupied space size of each second data table, and the number of records; Among them, determining the first data table among the at least one data table to be migrated includes: obtaining the total occupied space size of the at least one data table to be migrated; calculating an average value based on the total occupied space size and the preset number of threads; comparing the occupied space size of each data table to be migrated with the average value, and taking the data table to be migrated whose occupied space size is greater than the average value as the first data table.

2. The method according to claim 1, characterized in that The method based on balanced allocation, according to the preset number of threads and the number of records in each of the first data tables, determines a plurality of first file blocks, including: For each first data table in turn, the data records in the first data table are evenly distributed to the first file blocks of the preset number of threads according to the number of records in the first data table.

3. The method according to claim 1, characterized in that The data table information also includes the space occupied by a single record, the number of the plurality of second file blocks is the preset number of threads, and the method based on balanced allocation, according to the preset number of threads, the space occupied by each of the second data tables, and the number of records, determines the plurality of second file blocks, including: Calculating the second file block capacity according to the preset number of threads and the space occupied by each of the second data tables; For each of the second data tables in turn, when the occupied space size of the second data table is not greater than the remaining space size of the current second file block, the data records in the second data table are allocated to the current second file block, wherein the remaining space size of the current second file block is calculated according to the capacity of the second file block, the number of data records stored in the current second file block, and the space size occupied by a corresponding single record; In the case that the occupied space size of the second data table is larger than the remaining space size of the current second file block, the number of allocatable records is calculated based on the occupied space size of a single record of the second data table and the remaining space size, and the data records of the allocatable number in the second data table are allocated to the current second file block, and when the second data table includes unallocated data records, the next second file block is switched to allocate the unallocated data records in the second data table to the next second file block.

4. The method according to any one of claims 1 to 3, characterized in that: The multiple export file blocks include a first file block and a second file block, and the migrating data records corresponding to the multiple export file blocks by concurrently executing the preset number of threads includes: At the same time, each thread migrates the first file block and the second file block corresponding to the thread. During the migration process, the preset number of threads concurrently migrate the first file blocks corresponding to each other to achieve concurrent migration of the first data table.

5. The method according to any one of claims 1 to 3, characterized in that: The preset number of threads is the maximum number of concurrent threads allowed by the target database.

6. A data migration device, characterized in that: include: An information acquisition module, used to obtain data table information of each data table to be migrated in at least one data table to be migrated, wherein the data table information includes the size of occupied space and the number of records; A processing module, configured to determine a plurality of export file blocks based on a balanced allocation method according to a preset number of threads, a space occupied by each data table to be migrated, and a number of records, wherein the number of the plurality of export file blocks is an integer multiple of the preset number of threads, and the number of data records in each of the export file blocks is an integer; A migration module, configured to migrate data records corresponding to the plurality of export file blocks through concurrent threads of the preset number of threads; The multiple exported file blocks include a first file block and / or a second file block, and the processing module is specifically used for: Determine a first data table and / or a second data table in the at least one data table to be migrated, wherein the space occupied by each of the first data tables is greater than the space occupied by each of the second data tables, and the data tables to be migrated other than the first data tables in the at least one data table to be migrated are the second data tables; Based on a balanced allocation method, a plurality of first file blocks are determined according to the preset number of threads and the number of records in each of the first data tables; Based on a balanced allocation method, a plurality of second file blocks are determined according to the preset number of threads, the occupied space size of each second data table, and the number of records; Among them, the processing module is specifically used to: obtain the total occupied space size of at least one data table to be migrated; calculate an average value based on the total occupied space size and the preset number of threads; compare the occupied space size of each data table to be migrated with the average value, and use the data table to be migrated whose occupied space size is greater than the average value as the first data table.

7. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the data migration method described in any one of claims 1-5.

8. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data migration method according to any one of claims 1 to 5 is implemented.

Images