A data migration method, device, electronic equipment and storage medium

Abstract

The application relates to the field of data analysis, in particular to a data migration method and device, electronic equipment and a storage medium, which are used for improving the efficiency of data migration. In the application, based on an uploaded configuration file, a preset data structure of each row in a source database and a preset corresponding data structure of each row in a target database are determined; any row of data of the source database is taken as a to-be-migrated row, and any row in the target database is taken as a target row; for each to-be-migrated row, the following is executed: the to-be-migrated row is matched with each target row based on the data structure, so that a target row matched with the to-be-migrated column is obtained; and data in the to-be-migrated row is migrated to the matched target row. A mapping relationship table between the source database and the target database does not need to be constructed in advance, various data migration scenarios between different databases can be covered, the application is more universal, and a large amount of reading and writing is not needed in the process of data migration, so that resources consumed in the process of data migration are saved.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a data migration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data development typically involves different technology stacks and databases, often requiring data migration from one database to another. Distributed storage databases are commonly used for batch data processing, while relational databases are typically used for online real-time queries. Different databases use different data structures, making it impossible to directly create new tables using the source table's structure. Instead, the target table must be created first, followed by a data migration process. However, this method involves numerous read and write operations, consuming significant resources. Summary of the Invention

[0003] The purpose of this application is to provide a data migration method, apparatus, electronic device, and storage medium for improving the efficiency of data migration.

[0004] In a first aspect, embodiments of this application provide a data migration method, the method comprising:

[0005] Based on the user-uploaded configuration file, the preset data structure of each row in the source database and the corresponding preset data structure of each row in the target database are determined; wherein the data structure of each row in the source database is the same and the data structure of each row in the target database is the same, and the row is a row or column;

[0006] Take any row of data from the source database as the row to be migrated, and take any row from the target database as the target row. Perform the following data migration process for each row to be migrated:

[0007] Based on the data structure of the column to be migrated and the data structure of each target column, the column to be migrated is matched with each target column to obtain at least one target column that matches the column to be migrated.

[0008] Migrate the data in the queue to be migrated to the matching target queue.

[0009] In this application, data migration is performed based on the data structure of each row of data in the source database and the corresponding data structure of each row in the target database. There is no need to pre-build a mapping table between the source database and the target database. This can basically cover various data migration scenarios between different databases, making this application more universal. In addition, this application does not require a lot of reading and writing during the data migration process, saving the resources consumed in the data migration process.

[0010] In some possible embodiments, after migrating the data in the queue to be migrated to the matching target queue, the method further includes:

[0011] Determine the memory usage of the data stored in each target row;

[0012] The target sort with the least memory usage is used as the final target sort;

[0013] Remove all target rows except the final target row from the target database.

[0014] In this application, the final target row is determined by the memory occupied by each row of data, thus saving storage space.

[0015] In some possible embodiments, after deleting the target rankings other than the final target ranking from the target database, the method further includes:

[0016] The data in the final target row is verified against the data in the row to be migrated.

[0017] If the verification is successful, the data migration is considered successful.

[0018] If the verification fails, the data migration is determined to have failed. An error is reported, the data in the final target row is deleted, and the data migration process is repeated for the row to be migrated.

[0019] In this application, the accuracy and usability of the data migration are ensured through a verification process.

[0020] In some possible embodiments, the step of verifying the data in the final target row after data migration with the data in the row to be migrated includes:

[0021] The data structure of the final target row is compared with the data structure of the row to be migrated;

[0022] Then, the sum of the values ​​corresponding to the data in the final target row is compared with the sum of the values ​​corresponding to the data in the row to be migrated.

[0023] In this application, the accuracy of the data migration process is further ensured by comparing the data structure and data content.

[0024] In some possible embodiments, after determining the data structure corresponding to each row of data in the source database, the method further includes:

[0025] The timeframe for data migration is determined based on the configuration file.

[0026] The source database is filtered based on the time range to obtain the list of databases to be migrated.

[0027] In this application, a method of filtering the source database based on the time range of data migration avoids the waste of resources caused by migrating the entire amount of data.

[0028] In some possible embodiments, after determining the data structure corresponding to each row of data in the source database and the data structure of each row in the target database, the method further includes:

[0029] A remote dictionary technique is used to establish a link between the source database and the target database, and it is confirmed that the network environment of the source database and the target database is normal.

[0030] In this application, the speed of data migration is effectively improved by establishing a link between the source database and the target database using remote dictionary technology.

[0031] Secondly, this application also provides a data migration apparatus, the apparatus comprising:

[0032] The data structure determination module is used to determine the preset data structure of each row in the source database and the corresponding preset data structure of each row in the target database based on the configuration file uploaded by the user; wherein the data structure of each row in the source database is the same and the data structure of each row in the target database is the same, and the row is a row or column;

[0033] The migration module is used to take any row of data in the source database as the row to be migrated, and any row in the target database as the target row, and perform the following data migration process for each row to be migrated:

[0034] Based on the data structure of the column to be migrated and the data structure of each target column, the column to be migrated is matched with each target column to obtain at least one target column that matches the column to be migrated.

[0035] Migrate the data in the queue to be migrated to the matching target queue.

[0036] In some possible embodiments, after the migration module performs the migration of data in the queue to be migrated to the matching target queue, it is further configured to:

[0037] Determine the memory usage of the data stored in each target row;

[0038] The target sort with the least memory usage is used as the final target sort;

[0039] Remove all target rows except the final target row from the target database.

[0040] In some possible embodiments, after the migration module performs the deletion of the target rankings other than the final target ranking from the target database, it is further configured to:

[0041] The data in the final target row is verified against the data in the row to be migrated.

[0042] If the verification is successful, the data migration is considered successful.

[0043] If the verification fails, the data migration is determined to have failed. An error is reported, the data in the final target row is deleted, and the data migration process is repeated for the row to be migrated.

[0044] In some possible embodiments, when the migration module performs the verification between the data in the final target row after data migration and the data in the row to be migrated, it is specifically used for:

[0045] The data structure of the final target row is compared with the data structure of the row to be migrated;

[0046] Then, the sum of the values ​​corresponding to the data in the final target row is compared with the sum of the values ​​corresponding to the data in the row to be migrated.

[0047] In some possible embodiments, after the data structure determination module determines the data structure corresponding to each row of data in the source database, it is further configured to:

[0048] The timeframe for data migration is determined based on the configuration file.

[0049] The source database is filtered based on the time range to obtain the list of databases to be migrated.

[0050] In some possible embodiments, after the data structure determination module determines the data structure corresponding to each row of data in the source database and the data structure of each row in the target database, it is further configured to:

[0051] A remote dictionary technique is used to establish a link between the source database and the target database, and it is confirmed that the network environment of the source database and the target database is normal.

[0052] Thirdly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods provided in the first aspect embodiment of this application.

[0053] Fourthly, another embodiment of this application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for causing a computer to perform any of the methods provided in the first aspect of this application.

[0054] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0055] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram illustrating an application scenario of a data migration method provided in an embodiment of this application;

[0058] Figure 2A This is a schematic diagram of the overall process of a data migration method provided in an embodiment of this application;

[0059] Figure 2B A schematic diagram of the source database and target database for a data migration method provided in an embodiment of this application;

[0060] Figure 2C This application provides a schematic diagram of a data migration process for a data migration method.

[0061] Figure 3 A flowchart illustrating the determination of the final target row in a data migration method provided in this application embodiment;

[0062] Figure 4 This application provides a schematic diagram of the data verification process for a data migration method.

[0063] Figure 5 This application provides a schematic diagram of data verification for a data migration method.

[0064] Figure 6A schematic diagram of an apparatus for a data migration method provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of an electronic device for a data migration method provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0067] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses. The term "multiple" in this application can mean at least two, for example, two, three, or more, and is not limited by the embodiments of this application.

[0068] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0069] The inventors discovered that different technology stacks and databases are typically used in data development. Data migration from one database to another is often necessary; distributed storage databases are commonly used for batch data processing, while relational databases are typically used for online real-time queries. In data development, different databases use different data structures, making it impossible to directly create new data tables using the source table's data structure. Instead, the target table must be created first, followed by a data migration process. However, this method involves numerous read and write operations, consuming significant resources.

[0070] In view of this, this application proposes a data migration method, apparatus, electronic device, and storage medium to solve the above problems. The inventive concept of this application can be summarized as follows: Based on a user-uploaded configuration file, a preset data structure for each row in the source database and a preset data structure for each row in the target database are determined; wherein the data structure for each row in the source database is the same, and the data structure for each row in the target database is the same, with rows being either rows or columns; any row of data in the source database is taken as the row to be migrated, and any row in the target database is taken as the target row; for each row to be migrated, the following data migration process is performed: based on the data structure of the row to be migrated and the data structure of each target row, the row to be migrated is matched with each target row to obtain at least one target row that matches the row to be migrated; the data in the row to be migrated is migrated to the matched target row.

[0071] For ease of understanding, the data migration method provided in this application embodiment will be described in detail below with reference to the accompanying drawings:

[0072] like Figure 1 The diagram shown illustrates an application scenario of a data migration method according to an embodiment of this application. The diagram includes: a server 10, a storage device 20, and a terminal device 30; wherein:

[0073] Server 10 determines the preset data structure of each row in the source database and the corresponding preset data structure of each row in the target database based on the configuration file uploaded by the user in the terminal device 30; wherein the data structure of each row in the source database is the same and the data structure of each row in the target database is the same, and the row is a row or column; taking any row of data in the source database as the row to be migrated and any row in the target database as the target row, the following data migration process is performed for each row to be migrated: based on the data structure of the row to be migrated and the data structure of each target row, the row to be migrated is matched with each target row to obtain at least one target row that matches the row to be migrated; the data in the row to be migrated is migrated to the matched target row and stored in the memory 20.

[0074] The description in this application details only a single server 10, memory 20, and terminal device 30. However, those skilled in the art should understand that the illustrated server 10, memory 20, and terminal device 30 are intended to illustrate the operation of the server 10, memory 20, and terminal device 30 involved in the technical solutions of this application, and are not intended to imply any limitation on the number, type, or location of the server 10, memory 20, and terminal device 30. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application. Furthermore, although for ease of explanation... Figure 1The diagram shows a bidirectional arrow from memory 20 to server 10, but those skilled in the art will understand that the sending and receiving of the aforementioned data also needs to be achieved through a network.

[0075] It should be noted that the memory in the embodiments of this application can be, for example, a cache system, hard disk storage, memory storage, etc. Furthermore, the data migration method proposed in this application is not only applicable to… Figure 1 The application scenarios shown are also applicable to any device that has data migration needs.

[0076] like Figure 2A The diagram shown is a schematic representation of the overall process of a data migration method provided in an embodiment of this application, wherein:

[0077] In step 201: Based on the configuration file uploaded by the user, determine the preset data structure of each row in the source database and the corresponding preset data structure of each row in the target database; wherein the data structure of each row in the source database is the same and the data structure of each row in the target database is the same, and the row is either a row or a column;

[0078] For example: Figure 2B As shown, the data in source database 1 is arranged vertically, with each column having the same data structure; therefore, each row of data in source database 1 is equivalent to each column of data. Similarly, the data in source database 2 is arranged horizontally, with each row having the same data structure; therefore, each row of data in source database 2 is equivalent to each row of data. Likewise, if target database 1 is arranged vertically, then each row of data in target database 1 is equivalent to each column of data; and if target database 2 is arranged horizontally, then each row of data in target database 2 is equivalent to each column of data.

[0079] In step 202: any row of data in the source database is taken as the row to be migrated, and any row in the target database is taken as the target row. The data migration process is performed for each row to be migrated.

[0080] In this application, the data migration process can be implemented as follows: Figure 2C As shown, where:

[0081] In step 211: Based on the data structure of the row to be migrated and the data structure of each target row, the row to be migrated is matched with each target row to obtain at least one target row that matches the row to be migrated;

[0082] In step 212: migrate the data in the queue to be migrated to the matching target queue.

[0083] In this application, data migration is performed based on the data structure of each row of data in the source database and the corresponding data structure of each row in the target database. There is no need to pre-build a mapping table between the source database and the target database. This can basically cover various data migration scenarios between different databases, making this application more universal. In addition, this application does not require a lot of reading and writing during the data migration process, saving the resources consumed in the data migration process.

[0084] To facilitate a further understanding of the data migration process provided in the embodiments of this application, the following describes... Figure 2A and Figure 2C The steps are explained in detail below:

[0085] In some possible embodiments, when matching the data structure of the row to be migrated with each target row based on the data structure of each target row, there may be multiple target rows that match the data structure of the row to be migrated. In this application, to save storage space, after migrating the data in the row to be migrated to the matched target rows, the following can be implemented: Figure 3 The steps shown are as follows:

[0086] In step 301: Determine the memory occupied by the data stored in each target row;

[0087] In step 302: the target sort with the smallest memory usage is taken as the final target sort;

[0088] In step 303: Delete the target rankings, except for the final target ranking, from the target database.

[0089] For example: For row 1 to be migrated, target rows 1, 2, and 3 are identified as matching the data structure of row 1. The data in row 1 to be migrated is migrated to target rows 1, 2, and 3. Then, the memory corresponding to target rows 1, 2, and 3 is determined respectively. Target row 2 is identified as the target row with the smallest memory usage. Target rows 1 and 3 are then deleted from the target database.

[0090] In this application, the final target row is determined by the memory occupied by each row of data, thus saving storage space.

[0091] In some possible embodiments, to reduce errors during the data migration process, after deleting the target ranking (excluding the final target ranking) from the target database, the following steps may be implemented: Figure 4 The steps shown are for validating the migrated data:

[0092] In step 401: the data in the final target row is verified against the data in the row to be migrated;

[0093] In some possible embodiments, in order to ensure the accuracy of data verification, data verification in this application includes, but is not limited to: comparing the data structure of the final target row with the data structure of the row to be migrated; and comparing the sum of the values ​​corresponding to the data in the final target row with the sum of the values ​​corresponding to the data in the row to be migrated.

[0094] For example: Figure 5 As shown, the source database is a vertically arranged database, including columns to be migrated: column 1, column 2, and column 3. The target database is also a vertically arranged database, and the migrated target database includes: the final target column 1 corresponding to column 1, the final target column 2 corresponding to column 2, and the final target column 3 corresponding to column 3. For column 1 to be migrated, the data structure of column 1 to be migrated is compared with the data structure of the final target column 1, and the sum of the data in column 1 to be migrated is compared with the sum of the data in the final target column 1. To further ensure data accuracy, each row of data in column 1 to be migrated is compared with each row of data in the final target column 1, specifically: the first row of data in column 1 to be migrated is compared with the first row of data in the final target column 1, the second row of data in column 1 to be migrated is compared with the second row of data in the final target column 1, and the third row of data in column 1 to be migrated is compared with the third row of data in the final target column 1.

[0095] In step 402: If the verification is successful, the data migration is considered successful;

[0096] In step 403: if the verification is inconsistent, the data migration is determined to have failed. Error handling is performed to delete the data in the final target row, and the data migration process is performed again for the row to be migrated.

[0097] For example: the source database contains columns to be migrated: column 1, column 2, and column 3; the target database after migration contains: the final target column 1 corresponding to column 1, the final target column 2 corresponding to column 2, and the final target column 3 corresponding to column 3; then the final target column 1, the final target column 2, and the final target column 3 need to be validated respectively.

[0098] In this embodiment, to ensure data migration efficiency while reducing resource consumption during the migration process, after determining the data structure corresponding to each row of data in the source database, the following steps can be implemented: determining the time range for data migration based on the configuration file; filtering the source database based on the time range to obtain the rows to be migrated. This method of filtering the source database based on the time range for data migration avoids the waste of resources caused by migrating the entire dataset.

[0099] For example, if a user sets the time range in the configuration file to 2022.01.02-2022.02.02, then the data in the database will be filtered based on this time range to obtain the data within the time period of 2022.01.02-2022.02.02.

[0100] In this application, the timing of the above steps is not limited. They can be performed after receiving the user's configuration file or after determining the data structure of each row of data. This application does not limit the timing of the execution. The execution timing given above is only one embodiment.

[0101] In this embodiment of the application, in order to ensure the speed of data migration, after determining the source database and the target database based on the configuration file uploaded by the user, it is necessary to use remote dictionary technology (Redis) to establish a link between the source database and the target database, and to ensure that the network environment of the source database and the target database is normal.

[0102] like Figure 6 As shown, based on the same inventive concept, a data migration device 600 is proposed, comprising:

[0103] The data structure determination module 6001 is used to determine the preset data structure of each row in the source database and the preset data structure of each row in the target database based on the configuration file uploaded by the user; wherein the data structure of each row in the source database is the same and the data structure of each row in the target database is the same, and the row is a row or a column.

[0104] Migration module 6002 is used to take any row of data in the source database as the row to be migrated, and any row in the target database as the target row, and perform the following data migration process for each row to be migrated:

[0105] Based on the data structure of the column to be migrated and the data structure of each target column, the column to be migrated is matched with each target column to obtain at least one target column that matches the column to be migrated.

[0106] Migrate the data in the queue to be migrated to the matching target queue.

[0107] In some possible embodiments, after the migration module 6002 performs the migration of data in the queue to be migrated to the matching target queue, it is further configured to:

[0108] Determine the memory usage of the data stored in each target row;

[0109] The target sort with the least memory usage is used as the final target sort;

[0110] Remove all target rows except the final target row from the target database.

[0111] In some possible embodiments, after the migration module 6002 performs the deletion of the target rankings other than the final target ranking from the target database, it is further configured to:

[0112] The data in the final target row is verified against the data in the row to be migrated.

[0113] If the verification is successful, the data migration is considered successful.

[0114] If the verification fails, the data migration is determined to have failed. An error is reported, the data in the final target row is deleted, and the data migration process is repeated for the row to be migrated.

[0115] In some possible embodiments, when the migration module 6002 performs the verification of the data in the final target row after data migration with the data in the row to be migrated, it is specifically used for:

[0116] The data structure of the final target row is compared with the data structure of the row to be migrated;

[0117] Then, the sum of the values ​​corresponding to the data in the final target row is compared with the sum of the values ​​corresponding to the data in the row to be migrated.

[0118] In some possible embodiments, after the data structure determination module 6001 determines the data structure corresponding to each row of data in the source database, it is further configured to:

[0119] The timeframe for data migration is determined based on the configuration file.

[0120] The source database is filtered based on the time range to obtain the list of databases to be migrated.

[0121] In some possible embodiments, after the data structure determination module 6001 determines the data structure corresponding to each row of data in the source database and the data structure of each row in the target database, it is further configured to:

[0122] A remote dictionary technique is used to establish a link between the source database and the target database, and it is confirmed that the network environment of the source database and the target database is normal.

[0123] Having described the data migration method and apparatus according to exemplary embodiments of this application, we will now describe an electronic device according to another exemplary embodiment of this application.

[0124] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0125] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of the data migration methods according to the various exemplary embodiments of this application described above.

[0126] The following reference Figure 7 To describe an electronic device 130 according to this embodiment of the present application. Figure 7 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0127] like Figure 7 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0128] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0129] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0130] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0131] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used by electronic device 130 via bus 133. It should be understood that, although... Figure 7 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] In some possible implementations, various aspects of the data migration method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in a data migration method according to various exemplary embodiments of this application as described above.

[0133] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0134] The program product for data migration according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0135] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0137] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0138] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0139] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data migration method, characterized by, The method comprises: determining, based on a configuration file uploaded by a user, a preset data structure of each row in a source database and a corresponding data structure of each row in a target database; wherein the data structure of each row in the source database is the same, the data structure of each row in the target database is the same, and the row is a row or a column; taking any row of data in the source database as a to-be-migrated row and taking any row in the target database as a target row, performing the following data migration process for each to-be-migrated row: matching the to-be-migrated row with each target row based on the data structure of the to-be-migrated row and the data structure of each target row to obtain at least one target row matched with the to-be-migrated row; migrating data in the to-be-migrated row to the matched target row; when a plurality of target rows are matched with the data structure of the to-be-migrated row, after the data in the to-be-migrated row is migrated to the matched target row, the method further comprises: determining memory occupied by data stored in each target row; taking a target row occupying the least memory as a final target row; deleting target rows other than the final target row from the target database.

2. The method of claim 1, wherein, after the target rows other than the final target row are deleted from the target database, the method further comprises: verifying data in the final target row with data in the to-be-migrated row; if the verification is consistent, determining that the data migration is successful; if the verification is inconsistent, determining that the data migration fails, performing error processing, deleting data in the final target row, and performing the data migration process again for the to-be-migrated row.

3. The method of claim 2, wherein, verifying data in the final target row after data migration with data in the to-be-migrated row comprises: comparing the data structure of the final target row with the data structure of the to-be-migrated row; and comparing a sum of values corresponding to data in the final target row with a sum of values corresponding to data in the to-be-migrated row. after determining the data structure corresponding to each row of data in the source database, the method further comprises:

4. The method of claim 1, wherein, determining a time range of data migration based on the configuration file; filtering the source database based on the time range to obtain a to-be-migrated row. after determining the data structure corresponding to each row of data in the source database and the data structure of each row in the target database, the method further comprises:

5. The method according to any one of claims 1 to 4, characterized in that, establishing a link between the source database and the target database by using a remote dictionary technology and determining that a network environment in which the source database and the target database are located is normal. The device comprises:

6. A data migration apparatus, characterized by comprising: a data structure determination module configured to determine, based on a configuration file uploaded by a user, a preset data structure of each row in a source database and a corresponding data structure of each row in a target database; wherein the data structure of each row in the source database is the same, the data structure of each row in the target database is the same, and the row is a row or a column; a migration module configured to take any row of data in the source database as a to-be-migrated row and take any row in the target database as a target row, perform the following data migration process for each to-be-migrated row: ​ match the to-be-migrated row with each target row based on the data structure of the to-be-migrated row and the data structure of each target row, to obtain at least one target row matched with the to-be-migrated row; migrate data in the to-be-migrated row to the matched target row; when multiple target rows are all matched with the data structure of the to-be-migrated row, the migration module executes the migration of the data in the to-be-migrated row to the matched target row, and then is further configured to: determine memory occupied by data stored in each target row; determine a target row with the smallest occupied memory as a final target row; delete target rows other than the final target row from the target database.

7. The apparatus of claim 6, wherein, after the migration module executes the deletion of target rows other than the final target row from the target database, the migration module is further configured to: verify data in the final target row with data in the to-be-migrated row; if the verification is consistent, it is determined that the data migration is successful; if the verification is inconsistent, it is determined that the data migration fails, error processing is performed, the data in the final target row is deleted, and the data migration process is performed again for the to-be-migrated row.

8. The apparatus of claim 7, wherein, when the migration module verifies the data in the final target row after the data migration, the migration module is specifically configured to: compare the data structure of the final target row with the data structure of the to-be-migrated row; and compare the sum of values corresponding to the data in the final target row with the sum of values corresponding to the data in the to-be-migrated row.

9. The apparatus of claim 6, wherein, after the data structure determination module determines the data structure corresponding to each row of data in the source database, the data structure determination module is further configured to: determine a time range for data migration based on the configuration file; filter the source database based on the time range to obtain a to-be-migrated row.

10. The apparatus of any of claims 6-9, wherein, after the data structure determination module determines the data structure corresponding to each row of data in the source database and the data structure of each row of data in the target database, the data structure determination module is further configured to: establish a link between the source database and the target database using a remote dictionary technology, and determine that the network environment in which the source database and the target database are located is normal.

11. An electronic device, comprising: The apparatus includes at least one processor and a memory connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the method of any one of claims 1-5.

12. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method of any one of claims 1-5.

13. A computer program product, characterised in that, The computer program product includes computer program code, which, when executed on a computer, causes the computer to execute the method of any one of claims 1-5.

Images