Data migration methods, source devices, target devices, equipment, media and products

Abstract

The present disclosure provides a data migration method, a source device, a target device, equipment, a medium and a product. The data migration method comprises: obtaining the data to be migrated in the source database; preprocessing the data to be migrated to obtain export data, wherein the export data comprises a plurality of different data tables, and the data tables comprise a plurality of different objects; exporting the plurality of different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported; and sending the transmission data to the target end through the first transit storage.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a data migration method, a source device, a target device, equipment, a medium and a product. Background Art

[0002] With the advancement of localization, the localization transformation and innovation of applications have become urgent issues facing the current banking business system. Since banking business requires high stability and timeliness, it is determined that it needs to be smoothly migrated between different physical layer architectures in a stable, fast and minimally impacted manner. The difficulty in solving the problem lies in the uncertainty of the time and complexity of the problems encountered during the cross-architecture application reconstruction and migration process, as well as the compatibility verification after the migration is completed. Summary of the invention

[0003] In view of the above technical problems, the present disclosure provides a data migration method, a source device, a target device, equipment, medium and product.

[0004] A first aspect of the present disclosure provides a method for data migration, the method comprising: acquiring the data to be migrated in the source database; preprocessing the data to be migrated to obtain export data, the export data comprising a plurality of different data tables, the data tables comprising a plurality of different objects; exporting the plurality of different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported; and sending the transmission data to a target end through the first transit storage.

[0005] According to an embodiment of the present disclosure, the preprocessing of the data to be migrated to obtain the exported data includes: obtaining a static table; combing the static table to obtain invalid data; and cleaning the invalid data in the data to be migrated to obtain the exported data.

[0006] According to an embodiment of the present disclosure, the objects with empty data blocks in the data table are optimized and then exported, including: obtaining a first export data table, the first export data table including multiple first data objects, the first data objects including multiple empty database blocks; cleaning up the multiple empty data blocks in the first export data table until the first export data is less than a preset storage threshold, to obtain a second export data table; and exporting the second export data table to the first transit storage.

[0007] According to an embodiment of the present disclosure, the cleaning of multiple empty data blocks in the first export data table is performed until the first export data is less than a preset storage threshold to obtain a second export data table, wherein the first export data table includes a table structure and table data, and the method includes: obtaining a second export data table, wherein the second export data table includes an empty table; after the first export data table completes cleaning of its own empty data blocks, updating the table structure of the first export data table to the second export data table; updating the table data to the second export data table; and after all data in the first data table is updated, deleting the first export data table.

[0008] A second aspect of the present disclosure provides a method for data migration, which is applied to a target end, wherein the target end includes a target database and a second transit storage, and the method includes: receiving transmission data by the second transit storage; parsing the transmission data to obtain an import data table; verifying the import data table to verify whether the data in the import data table is recognizable; and when unrecognizable data exists, processing the import data table and then formally importing it into the target database.

[0009] According to an embodiment of the present disclosure, when unrecognizable data exists, the imported data table is processed and then formally imported into the target database, wherein the unrecognizable data in the imported data table includes at least annotation data, and the method comprises: obtaining the annotation data in the imported data table; performing compatibility processing on the annotation data; and importing the imported data table after the compatibility processing into the target database.

[0010] The third aspect of the present disclosure provides a source-side device, the device comprising: an acquisition module, a preprocessing module, an export module and a sending module, wherein the acquisition module is used to acquire the data to be migrated in the source database; the preprocessing module is used to preprocess the data to be migrated to obtain exported data, the exported data includes a plurality of different data tables, and the data tables include a plurality of different objects; the export module is used to export the plurality of different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data table are optimized and then exported; and the sending module is used to send the transmission data to the target side through the first transit storage.

[0011] The fourth aspect of the present disclosure provides a device at the target end, which includes: a receiving module, a parsing module, a verification module and an importing module, wherein the receiving module is used to receive the transmission data from the second transit storage; the parsing module is used to parse the transmission data to obtain an imported data table; the verification module is used to verify the imported data table to verify whether the data in the imported data table is recognizable; and the importing module is used to process the imported data table and then formally import it into the target database when there is unrecognizable data.

[0012] The five aspects of the present disclosure provide an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method according to the above-mentioned data migration.

[0013] The sixth aspect of the present disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data migration method.

[0014] A seventh aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned data migration method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0016] Figure 1 The application scenario diagram of data migration according to an embodiment of the present disclosure is schematically shown.

[0017] Figure 2 A flow chart of a data migration method applied to a source end according to an embodiment of the present disclosure is schematically shown.

[0018] Figure 3 A flowchart of a method for data export processing according to an embodiment of the present disclosure is schematically shown.

[0019] Figure 4 A flowchart of data optimization processing according to an embodiment of the present disclosure is schematically shown.

[0020] Figure 5 A flowchart of a data migration processing method applied to a target end according to an embodiment of the present disclosure is schematically shown.

[0021] Figure 6A flowchart of a method for data import processing according to an embodiment of the present disclosure is schematically shown.

[0022] Fig. 7A The structure block diagram of a source device according to an embodiment of the present disclosure is schematically shown.

[0023] Figure 7B The structure block diagram of a target end device according to an embodiment of the present disclosure is schematically shown.

[0024] Figure 8 A block diagram of an electronic device suitable for implementing a method for data migration according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] Before describing the technical solution of the present invention, the technical terms in the art are explained as follows:

[0030] Data Pump: A method of exporting (EXPDP) and importing (IMPDP) data in Oracle database. It can achieve the following functions:

[0031] 1) Implement logical backup and logical recovery.

[0032] 2) Move objects between database users.

[0033] 3) Move objects between databases.

[0034] 4)Realize table space migration.

[0035] SAN: Storage Area Networks, is a dedicated storage system that connects one or more storage devices and servers through a dedicated high-speed network. Generally speaking, SAN includes three networking methods, namely:

[0036] 1) Direct connection networking;

[0037] 2) Single switch networking;

[0038] 3) Dual switching networking.

[0039] FTP / SFTP protocol: File Transfer Protocol / Secure File Transfer Protocol, used to complete file transfer between different devices.

[0040] SSH protocol: Secure Shell, is a protocol that provides security for remote login sessions and other network services. It can perform encrypted file transmission.

[0041] SCP: Secure Copy, a command for secure remote file copying based on SSH protocol login.

[0042] LOB object: Large Objects. In Oracle database, LOB is a data type used to store large amounts of binary and text data.

[0043] In the prior art, the following steps are generally included:

[0044] In operation S01, a high-availability cluster is built, and a new X86 virtual machine system is created. A Linux operating system (such as different versions of Redhat, SUSE, Ubuntu, etc.) is installed on each virtual machine, and related vmtools tools are installed, and the corresponding IP is configured.

[0045] In operation S02, the business software on the original minicomputer platform is deployed to the newly constructed virtual machine by exporting and importing, and the installation and debugging are completed.

[0046] In operation S03, a database virtual machine is built, storage space in the SAN is allocated, and the database data is waiting to be migrated to the virtual machine.

[0047] In operation S04, a corresponding script is executed in the original AIX minicomputer platform to send the data to the target virtual machine.

[0048] In operation S05, a mount operation is performed and data verification is completed.

[0049] In the related prior art, for different cross-architecture data migration methods, no solution is generated that combines the downtime window period in actual production.

[0050] An embodiment of the present disclosure provides a method for data migration, the method comprising: acquiring the data to be migrated in the source database; preprocessing the data to be migrated to obtain export data, the export data comprising a plurality of different data tables, the data tables comprising a plurality of different objects; exporting the plurality of different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported; and sending the transmission data to a target end via the first transit storage.

[0051] The embodiments of the present disclosure reduce the amount of exported data through preprocessing, and optimize the objects with a large number of empty blocks, so that the export efficiency of the data tables where such objects are located is improved. At the same time, the optimized empty database blocks also reduce the amount of data in the data tables where the corresponding objects are located, effectively reducing the amount of transmitted data and enabling data migration to be completed within the window period.

[0052] Figure 1 The application scenario diagram of data migration according to an embodiment of the present disclosure is schematically shown.

[0053] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a source end 110, a network 104, and a target end 120. The source end 110 includes: a source database 101 and a first transit storage 102; the target end 120 includes: a target database 105 and a second transit storage 106. The network 104 is used to provide a medium for a communication link between the source database 101 and the target database 105, or the network 104 is used to provide a medium for a communication link between the first transit storage 102 and the second transit storage 103. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0054] The user can use the source database 101 to perform data migration with the target database 105 through the first transit storage 102, the second transit storage 103 and the network 104. Various database software can be installed on the source database 101 and the target database 105 to store relevant business data.

[0055] The source end 110 and the target end 120 may be servers that provide various services, or may be a small network consisting of servers that provide various services and / or network devices.

[0056] The first network transit storage 102 may be a transit storage mounted on one end of the source database 101 , and the second transit storage 103 may be a transit storage mounted on one end of the target database 105 .

[0057] It should be noted that the data migration method provided in the embodiment of the present disclosure can generally be performed by the source end 110 and the target end 120. Accordingly, the data migration system provided in the embodiment of the present disclosure can generally be set in the source end 110 and the target end 120.

[0058] It should be understood that Figure 1 The number of source databases, networks, first network transit storage, second network transit storage, and target databases in the example are only illustrative. Any number of source databases, networks, first network transit storage, second network transit storage, and target databases may be provided as required.

[0059] It should be noted that the method of the embodiment of the present disclosure can be applied to a production environment with cross-network segments and heterogeneous migration. Therefore, it is necessary to mount the transit storage on the source database side and the target database side respectively. Among them, the transit storage on the source database side includes the first transit storage, and the transit storage mounted on the target database side includes the second transit storage. The first transit storage and the second transit storage are used for cross-network segment transmission. Of course, the number of the first transit storage and the second transit storage can be customized based on the specific implementation situation, and is not limited here.

[0060] The embodiment of the present disclosure sets the first transit storage and the second transit storage, so that the data migration solution of the present disclosure is applicable to the environment of cross-segment transmission. And the source end and the target end mount the transit storage together, so that data transmission can be based on both synchronous transmission and asynchronous transmission, which improves the application scenario range of the actual application of the present disclosure.

[0061] The following will be passed Figure 2-Figure 6 A method for data migration according to an embodiment of the present disclosure is described in detail.

[0062] Figure 2 A flow chart of a data migration method applied to a source end according to an embodiment of the present disclosure is schematically shown.

[0063] like Figure 2 As shown, this embodiment provides a method for data migration, the method comprising operations S210 to S240, specifically as follows:

[0064] In operation S210, the data to be migrated in the source database is obtained.

[0065] According to an embodiment of the present disclosure, the source database includes at least: a relational database such as Mysql and Oracle.

[0066] In operation S220, the data to be migrated is preprocessed to obtain exported data, where the exported data includes a plurality of different data tables.

[0067] According to an embodiment of the present disclosure, the data to be migrated is preprocessed to obtain exported data, and the exported data includes multiple different data tables. The method includes: obtaining a static table; combing the static table to obtain invalid data; cleaning the invalid data in the data to be migrated to obtain the exported data.

[0068] For example, the static table is combed to clean up unnecessary data, and the data volume of the exported data is obtained, and the export method is formulated according to the data volume of the exported data.

[0069] According to an embodiment of the present disclosure, the export method at least includes: traditional method, data pump, and third-party tools, etc. For example, the traditional method includes exp export and imp import; the data pump includes: expdp export and impdp import; the third-party tool includes: PL / sql Developer, etc.

[0070] Taking Oracle database as an example, when the exported data is stored in Oracle database, the exported data at least includes: XXYYUSER user data and NDTM user data. The multiple different data tables include: data tables in XXYYUSER user data and data tables in NDTM data.

[0071] In operation S230, a plurality of different data tables are exported to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported.

[0072] According to an embodiment of the present disclosure, the export method uses a data pump method to export.

[0073] Specifically, after combing the static table, it is found that the amount of data to be exported is best exported by means of a data pump, and the first transit storage can be mounted on the source database side by means of a SAN technology.

[0074] Of course, the storage technology of the first transit storage also includes commonly used storage technologies such as DAS, NAS, HBA and iSCSI.

[0075] Because a large number of empty data blocks will seriously affect the performance of data export during the migration process, it is necessary to clean up the empty data blocks in each data table.

[0076] In actual applications, taking Oracle database as an example, the empty data blocks exist in most LOB objects. In the data table in the NDTM user data, there are a large number of LOB objects, so these LOB objects need to be processed. In XXYYUSER, there are fewer LOB object data, and the impact on the export efficiency during the export process can be ignored. Therefore, the data table in the XXYYUSER user data is not processed for the time being.

[0077] According to an embodiment of the present disclosure, exporting the exported data to the first transit storage through different export methods to obtain transmission data also includes: when the exported data is exported to the first transit storage, converting the data format to obtain transmission data.

[0078] It should be noted that with the advancement of localization, more and more equipment needs to be replaced with domestic ones. Therefore, when performing data migration, most of the time, the source end is an AIX minicomputer and the target end is a domestic X86 architecture server. This situation belongs to heterogeneous migration. When data is transmitted between the first transit storage and the second transit storage that perform the data transmission function, format conversion is required. For example, in the process of exporting from the source database to the first transit storage, the data can exist in the form of a data table. During the data transmission process, the transmission data can exist in the form of a DUMP file. Therefore, format conversion is required after the data table is imported into the first transit storage.

[0079] In operation S240, the transmission data is sent to the target end through the first transit storage.

[0080] According to an embodiment of the present disclosure, the transmission protocols used in the transmission process include: FTP, SFTP and SSH.

[0081] It is worth mentioning that in actual application, initially only the OPEN SSH protocol was selected to perform concurrent transmission through the SCP process. However, during the transmission process, the process may appear to be suspended. In this case, the suspended process not only occupies the system's processing resources, but also causes data migration to be terminated. The reason is that there is a BUG in the memory of the operating system, and the existing OPEN SSH protocol needs to be upgraded. However, there may be uncontrollable risks in the upgrade. Therefore, the more secure FTP\SFTP protocol is used as the main transmission protocol to send the transmission data.

[0082] According to an embodiment of the present disclosure, the transmission process includes concurrent transmission.

[0083] For another example, the transmission process may be based on an SCP process, with 10 DUMP files being transmitted concurrently each time.

[0084] The embodiments of the present disclosure reduce the amount of exported data through preprocessing, and optimize the objects with a large number of empty data blocks, so that the export efficiency of the data table where such objects are located is improved. At the same time, the optimized empty data blocks also reduce the amount of data in the data table where the corresponding objects are located, effectively reducing the amount of transmitted data, thereby ensuring that the data can be migrated within the window period.

[0085] Figure 3 A flowchart of a method for data export processing according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 3 As shown, this embodiment provides a method for data export processing, the method includes operations S310 to S330, which are specifically as follows:

[0087] In operation S310, a first derived data table is obtained, where the first derived data table includes a plurality of first data objects, and the first data objects include a plurality of empty database blocks.

[0088] For example, in an Oracle database, the first exported data table may include a data table containing a large number of LOB objects.

[0089] It should be noted that the reason why the export of LOB objects takes a long time is that when the first exported data table stores LOB objects. If the size of the LOB object is less than 4000 bytes, Oracle will store the LOB object in the same segment as the table, but in fact, the maximum size of the LOB object stored in the row can only be 3964 bytes, so when the LOB object is larger than 3964 bytes, the LOB object will be stored in a separate LOB segment (also called out-of-row). Therefore, if the LOB object is larger than 3964 bytes, the control data of the LOB object is stored at 38 to 84 bytes in the LOB column. Storing larger LOB objects in the row causes row migration. For example, if a 3900-byte LOB object is stored in a 2k data block, a LOB object will be stored in 2 or more blocks.

[0090] Furthermore, the smallest segment allocation unit for LOB data stored outside the row is the database block (DB_BLOCK_SIZE), such as 8K; so even if the LOB object is very small, it will occupy at least one data block; if the chink size is set to be larger than 1 times the block size, such as 10K, a LOB data will occupy at least 2 data blocks. Therefore, there will be a large number of empty data blocks in the LOB table.

[0091] In summary, if there are a large number of LOB objects in a data table, the read and write performance of the data table will be greatly affected, thereby extending the import and export time.

[0092] In operation S320, a plurality of empty data blocks in the first export data table are cleared until the first export data is smaller than a preset storage threshold, thereby obtaining a second export data table.

[0093] For example, when the first exported data table is a LOB table, historical data cleaning is performed on the LOB table to reduce the storage size of each LOB object to less than 4000 bytes, wherein the preset storage threshold can be set to 4000 bytes. It should be noted that the LOB table maps the LOB object, and most of the LOB tables are stored in the NDTM user data in the source database.

[0094] Of course, the preset storage threshold can also be set according to the actual situation of the current database, and no further limitation is made here.

[0095] In operation S330, the second exported data table is exported to the first transit storage.

[0096] The embodiment of the present disclosure adopts an optimized cleaning processing method, taking the table as the unit, cleaning the empty data blocks in each table, thereby improving the read and write performance and the cleaning efficiency. In addition, for different databases and different objects, different preset storage thresholds are selected for cleaning, thereby improving the adaptability of the method.

[0097] Figure 4 A flowchart of data optimization processing according to an embodiment of the present disclosure is schematically shown.

[0098] like Figure 4 As shown, this embodiment provides a method for data export optimization processing, the method includes operations S410 to S440, which are specifically as follows:

[0099] In operation S410, a second export data table is obtained, where the second export data table is an empty table.

[0100] According to an embodiment of the present disclosure, the second exported data table is an empty table for reserving update space.

[0101] In operation S420, after the first export data table completes clearing of its own empty data blocks, the table structure of the first export data table is updated to the second export data table.

[0102] According to an embodiment of the present disclosure, the second exported data table is used to reserve space for updating.

[0103] According to an embodiment of the present disclosure, after the first exported data table completes cleaning of its own empty data blocks, the method further includes: traversing all first exported data tables, marking them, and obtaining a data table list.

[0104] Specifically, the data table list may be a LOB table list. After traversing all LOB tables, marking is completed and a LOB table list is listed. The LOB table list is used to guide the reserved space size and record the update process during subsequent updates. For example, after a certain LOB table is updated, the LOB table list is marked as updated. Thus, the update process is sorted and recorded.

[0105] In operation S430, the table data is updated to the second exported data table.

[0106] According to an embodiment of the present disclosure, updating the table data to the second exported data table further includes: compressing the table data before updating.

[0107] In operation S440, after all data in the first data table is updated, the first exported data table is deleted.

[0108] The embodiments of the present disclosure complete the space reserved for table updates by acquiring the second export data table in advance. After cleaning the original first export data table, the table structure is first updated, and then the table data is updated to the second export data table with reserved space, which is beneficial to saving update time and, further, improving the efficiency of data migration as a whole.

[0109] Figure 5 A flowchart of a data migration processing method applied to a target end according to an embodiment of the present disclosure is schematically shown.

[0110] like Figure 5 As shown, this embodiment provides a method for data migration processing, the method includes operations S510 to S540, which are specifically as follows:

[0111] In operation S510, the second transit storage receives transmission data.

[0112] For example, the second transit storage receives the DUMP file from the first transit storage.

[0113] For another example, the second transit storage may be SAN storage.

[0114] In operation S520, the transmission data is parsed to obtain an import data table.

[0115] According to an embodiment of the present disclosure, when the target database capacity is smaller than the imported data, capacity expansion processing is performed.

[0116] In operation S530, the imported data table is verified to verify whether the data in the imported data table is identifiable.

[0117] In operation S540, when unrecognizable data exists, the imported data table is processed and then formally imported into the target database.

[0118] According to an embodiment of the present disclosure, when all data are identifiable, they are directly and formally imported into the database.

[0119] According to an embodiment of the present disclosure, the unrecognizable data at least includes annotation data.

[0120] Figure 6 A flowchart of a method for data import processing according to an embodiment of the present disclosure is schematically shown.

[0121] like Figure 6 The embodiment provides a method for data import processing, the method comprising operations S610 to S630, which are as follows:

[0122] In operation S610, the annotation data in the imported data table is obtained.

[0123] For example, the annotation data includes comment, which is a keyword of the Oracle database system and is used to add comments to tables and fields for understanding the table structure and the meaning of database objects.

[0124] In operation S620, compatibility processing is performed on the annotation data.

[0125] In operation S630, the imported data table after the compatibility processing is imported into the target database.

[0126] According to an embodiment of the present disclosure, the compatibility processing includes: renaming the annotation, modifying the character type, and cutting the annotation data.

[0127] It should be noted that in a certain data migration process of an embodiment of the present disclosure, the software version of Oracle of the source database is 11g, and the software version of Oracle of the target database is 19C. When exporting from Oracle to the first technical secondary storage SAN, the file format is DUMP format, and there is no verification. When importing from the second transit storage SAN to Oracle 19C, the target database will convert the DUMP file into recognizable characters before importing. When the character sets of the source database and the target database are different, the character set conversion will occur when the target database receives the DUMP file. When the converted character sets are incompatible, garbled characters will occur. During the import, if a compatibility error (ORA error) occurs, Oracle will stop importing and parsing because it records the detailed information of the current error, thereby affecting the reading and writing speeds.

[0128] Therefore, compatibility processing is required during the import process. For example, the compatibility processing includes: renaming the comments with garbled characters in the comment line where the parsing error is reported, and naming the character set that cannot be parsed as English, etc.; or changing the character type to varchar or nvarchar, which will save database storage space; or deleting useless comment fields that exceed the maximum length of 30 characters limited by Oracle.

[0129] According to an embodiment of the present disclosure, the annotation data may also be trimmed when the source database is exported to reduce the data volume of the annotation data, thereby reducing the overhead in the process of compiling and decompiling the annotation data.

[0130] In the embodiment of the present disclosure, the annotation data is stored after being processed for compatibility, which reduces the probability of error reporting by the target database for imported data and speeds up the import process of process imported data. In general, the time consumption is further reduced, and sufficient time is reserved for handling uncontrollable events.

[0131] Fig. 7A The structure block diagram of a source device according to an embodiment of the present disclosure is schematically shown.

[0132] like Fig. 7AAs shown, the source device 710 of this embodiment includes an acquisition module 711, a pre-processing module 712, an export module 713 and a sending module 714.

[0133] in,

[0134] The acquisition module 711 is used to acquire the data to be migrated in the source database. In one embodiment, the acquisition module 711 can be used to perform the operation S210 described above, which will not be described in detail here.

[0135] The preprocessing module 712 is used to preprocess the data to be migrated to obtain the exported data, wherein the exported data includes a plurality of different data tables, and the data tables include a plurality of different objects. In one embodiment, the preprocessing module 712 can be used to perform the operation S220 described above, which will not be described in detail here.

[0136] The export module 713 is used to export multiple different data tables to the first transit storage to obtain transmission data, wherein objects with empty data blocks in the data table are optimized and then exported. In one embodiment, the export module 713 can be used to perform the operation S230 described above, which will not be repeated here.

[0137] The sending module 714 is used to send the transmission data to the target end through the first transit storage. In one embodiment, the sending module 714 can be used to perform the operation S240 described above, which will not be described in detail here.

[0138] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 711, the preprocessing module 712, the export module 713 and the sending module 714 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 711, the preprocessing module 712, the export module 713 and the sending module 714 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 711, the preprocessing module 712, the export module 713 and the sending module 714 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be executed.

[0139] The embodiments of the present disclosure reduce the amount of exported data through preprocessing, and optimize the objects with a large number of empty blocks, so that the export efficiency of the data tables where such objects are located is improved. At the same time, the optimized empty database blocks also reduce the amount of data in the data tables where the corresponding objects are located, effectively reducing the amount of transmitted data and enabling data migration to be completed within the window period.

[0140] Figure 7B The structure block diagram of a target end device according to an embodiment of the present disclosure is schematically shown.

[0141] like Figure 7B As shown, the target end device 720 of this embodiment includes a receiving module 721, a parsing module 722, a checking module 723 and an importing module 724.

[0142] in,

[0143] The receiving module 721 is used for receiving the transmission data by the second transit storage. In one embodiment, the receiving module 721 can be used to perform the operation S510 described above, which will not be described in detail here.

[0144] The analysis module 722 is used to analyze the transmission data to obtain an import data table. The analysis module 722 can be used to perform the operations S5 and 20 described above, which will not be described in detail here.

[0145] The verification module 723 is used to perform verification processing on the imported data table to verify whether the data in the imported data table is identifiable. The verification module 723 can be used to perform the operation S530 described above, which will not be described in detail here.

[0146] The import module 724 is used to process the imported data table and then formally import it into the target database when there is unrecognizable data. The import module 724 can be used to perform the operation S540 described above, which will not be described in detail here.

[0147] According to an embodiment of the present disclosure, any multiple modules of the receiving module 721, the parsing module 722, the verification module 723 and the importing module 724 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the receiving module 721, the parsing module 722, the verification module 723 and the importing module 724 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the receiving module 721, the parsing module 722, the checking module 723 and the importing module 724 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be executed.

[0148] Figure 8 A block diagram of an electronic device suitable for implementing a method for data migration according to an embodiment of the present disclosure is schematically shown.

[0149] like Figure 8As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0150] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0151] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 808. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 708 as needed.

[0152] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0153] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0154] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0155] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0156] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0157] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0158] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0159] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0161] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for data migration, It is characterized in that The method is applied to a source end, wherein the source end includes a source database and a first transit storage. The method comprises: Acquire the data to be migrated from the source database; Preprocessing the data to be migrated to obtain exported data, wherein the exported data includes a plurality of different data tables, and the data tables include a plurality of different objects; Exporting multiple different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported; and Sending the transmission data to the target end through the first transit storage; The preprocessing of the data to be migrated to obtain the derived data includes: obtaining a static table; combing the static table to obtain invalid data; and cleaning invalid data in the data to be migrated to obtain the derived data; The method of performing export processing after optimizing objects with empty data blocks in a data table includes: obtaining a first export data table, the first export data table including multiple first data objects, the first data objects including multiple empty database blocks; cleaning multiple empty data blocks in the first export data table until the first export data is less than a preset storage threshold, thereby obtaining a second export data table; and exporting the second export data table to the first transit storage.

2. The method according to claim 1, It is characterized in that The step of cleaning up a plurality of empty data blocks in the first export data table until the first export data is less than a preset storage threshold value, thereby obtaining a second export data table. Wherein, the first exported data table includes a table structure and table data, The method comprises: Acquire a second export data table, wherein the second export data table includes an empty table; After the first export data table completes clearing of its own empty data blocks, updating the table structure of the first export data table to the second export data table; updating the table data to the second exported data table; and After all data in the first export data table is updated, the first export data table is deleted.

3. A method for data migration, It is characterized in that The method is applied to a target end, wherein the target end includes a target database and a second transit storage. The method comprises: The second transfer storage receives the transmission data; Parsing the transmitted data to obtain an imported data table; Performing verification processing on the imported data table to verify whether the data in the imported data table is identifiable; and When there is unrecognizable data, the imported data table is processed and then formally imported into the target database; When unrecognizable data exists, the imported data table is processed and then formally imported into the target database, wherein the unrecognizable data in the imported data table at least includes annotation data, and the method comprises: obtaining the annotation data in the imported data table; performing compatibility processing on the annotation data; and importing the imported data table after compatibility processing into the target database.

4. A source device, It is characterized in that The source end includes a source database and a first transit storage, and the device includes: an acquisition module, a preprocessing module, an export module and a sending module. in, The acquisition module is used to acquire the data to be migrated in the source database; The preprocessing module is used to preprocess the data to be migrated to obtain exported data, wherein the exported data includes a plurality of different data tables, and the data tables include a plurality of different objects; The export module is used to export a plurality of different data tables to a first transit storage to obtain transmission data, wherein objects with empty data blocks in the data tables are optimized and then exported; and The sending module is used to send the transmission data to the target end through the first transit storage; The preprocessing of the data to be migrated to obtain the derived data includes: obtaining a static table; combing the static table to obtain invalid data; and cleaning invalid data in the data to be migrated to obtain the derived data; The method of performing export processing after optimizing objects with empty data blocks in a data table includes: obtaining a first export data table, the first export data table including multiple first data objects, the first data objects including multiple empty database blocks; cleaning multiple empty data blocks in the first export data table until the first export data is less than a preset storage threshold, thereby obtaining a second export data table; and exporting the second export data table to the first transit storage.

5. A device at a target end, It is characterized in that The target end includes a target database and a second transit storage, and the device includes: a receiving module, a parsing module, a verification module and an import module. in, The receiving module is used to receive the transmission data from the second transit storage; The parsing module is used to parse the transmission data to obtain an import data table; The verification module is used to perform verification processing on the imported data table to verify whether the data in the imported data table is identifiable; and The import module is used to process the import data table and then formally import it into the target database when there is unrecognizable data; When unrecognizable data exists, the imported data table is processed and then formally imported into the target database, wherein the unrecognizable data in the imported data table at least includes annotation data, and the method comprises: obtaining the annotation data in the imported data table; performing compatibility processing on the annotation data; and importing the imported data table after compatibility processing into the target database.

6. An electronic device, include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 3.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 3.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images