Data migration method and device, electronic equipment and storage medium
By obtaining the compensation data table and table type, querying the target database, determining whether the data to be compensated exists, and updating or writing the data based on the timestamp, the problem of data inconsistency loss during heterogeneous database migration is solved, and a simple and effective compensation for data consistency is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2023-01-03
- Publication Date
- 2026-07-07
AI Technical Summary
When migrating data between heterogeneous databases, the lack of data consistency may lead to unpredictable risks, especially when migrating an Oracle database to a MySQL database. For databases used 24/7, maintaining data consistency requires huge costs, and the inability to effectively maintain data consistency will lead to the loss of data consistency.
A data migration method is provided, which obtains the compensation data table and table type, queries the target database to determine whether the data to be compensated exists, and updates or writes the data according to the timestamp to ensure data consistency. Parallel processing and thread processing mechanisms are used to achieve data compensation.
It ensures data consistency between heterogeneous databases, avoids data loss during data migration, and achieves simple and effective data compensation processing.
Smart Images
Figure CN115982129B_ABST
Abstract
Description
Technical Field
[0001] This application relates to big data processing technology, which can be used in the field of fintech or other related fields, and in particular to a data migration method, apparatus, electronic device and storage medium. Background Technology
[0002] Currently, during data migration between heterogeneous databases, the lack of data consistency may pose unpredictable risks to transactions. Therefore, data consistency between heterogeneous databases is extremely important.
[0003] For example, when migrating existing data from an Oracle database to a MySQL database, it is necessary to invest a huge amount of maintenance costs to maintain data consistency for a database that is used 24 / 7. If data consistency cannot be effectively maintained, it will lead to a loss of data consistency. Summary of the Invention
[0004] This application provides a data migration method, apparatus, electronic device, and storage medium to provide a data migration compensation scheme that can ensure data consistency between heterogeneous databases.
[0005] On the one hand, this application provides a data migration method applied to a unitized architecture, the method comprising:
[0006] Obtain the compensation data table and the table type of the compensation data table, wherein the compensation data table contains the missing data to be compensated during the data migration process from the database in the unitized architecture to the target database;
[0007] If the above table type indicates that the above compensation data table is a predetermined type of data table, then the above target database is queried according to the data number of the above data to be compensated in order to determine whether the above data to be compensated already exists in the above target database;
[0008] If the aforementioned data to be compensated already exists in the aforementioned target database, and the first timestamp of the aforementioned data to be compensated in the compensation data table is greater than the second timestamp of the aforementioned data to be compensated in the aforementioned target database, then update the second data table in the aforementioned target database that contains the aforementioned data to be compensated.
[0009] If the data to be compensated does not exist in the target database, then the data to be compensated will be written into the second data table of the target database.
[0010] Furthermore, the aforementioned acquisition of the compensation data table, and the table type of the compensation data table, includes:
[0011] The migration data from the aforementioned database to the aforementioned target database within a predetermined time period is obtained to obtain a first data table, wherein the aforementioned database and the target database are heterogeneous databases, and the aforementioned migration data includes the aforementioned data to be compensated.
[0012] Obtain a predetermined number of data to be compensated from the first data table mentioned above to obtain a compensation data table;
[0013] Based on whether the data to be compensated in the above compensation data table contains a timestamp, the table type of the above compensation data table is determined. The table type corresponding to the timestamp is the above reservation data table, which includes at least one of the following: customer table, agreement table.
[0014] Furthermore, before obtaining the table type of the aforementioned compensation data table, the above method also includes:
[0015] Obtain predetermined compensation configuration parameters, wherein the compensation configuration parameters include: predetermined number of processes per session;
[0016] If the amount of data to be compensated in the above compensation data list is less than the predetermined number of data to be processed in a single session, then the flag of the data compensation loop process will be marked as ended.
[0017] If the amount of data to be compensated in the above-mentioned compensation data list is greater than or equal to the predetermined number of data to be processed in a single session, then multiple threads of a predetermined size are created to process the above-mentioned compensation data list in parallel.
[0018] Furthermore, before obtaining a predetermined number of data to be compensated from the first data table to obtain the compensation data table, the method further includes:
[0019] Obtain the flag of the data compensation loop process to determine whether the data compensation loop process has ended;
[0020] If it is determined that the above data compensation loop process has ended, the compensation processing results corresponding to each of the above multiple threads are recorded in the first data table above to obtain the processing log. If the above processing log does not report any errors, it indicates that the data compensation is successful.
[0021] If it is determined that the above data compensation loop process has not ended, then the step of obtaining a predetermined number of data to be compensated from the first data table to obtain the compensation data table is executed.
[0022] Furthermore, before querying the target database based on the data number of the data to be compensated in the aforementioned compensation data table, the method further includes:
[0023] If it is determined that the data number of the data to be compensated in the above compensation data table is empty, then the compensation processing result corresponding to the above compensation data table is determined to be compensation processing failure.
[0024] If it is determined that the data number of the data to be compensated in the above compensation data table is not null, then the above steps of querying the target database based on the above data number will be executed.
[0025] Furthermore, the above method also includes:
[0026] If the above table type indicates that the above compensation data table is a data table other than the above-predetermined data table, then obtain the string inserted or updated in the above first data table;
[0027] If it is determined that the above string already exists in the above second data table, then the above second data table is updated based on the above string; if it is determined that the above string does not exist in the above second data table, then the above string is inserted into the above second data table.
[0028] On the other hand, this application provides a data migration apparatus applied to a unitized architecture, the apparatus comprising:
[0029] The acquisition module is used to acquire the compensation data table and the table type of the compensation data table. The compensation data table contains the missing data to be compensated during the data migration process from the database in the unitized architecture to the target database.
[0030] The query module is used to query the target database based on the data number of the data to be compensated if the table type indicates that the compensation data table is a predetermined type of data table, so as to determine whether the data to be compensated already exists in the target database.
[0031] The update module is used to update the second data table in the target database that contains the data to be compensated if the data to be compensated already exists in the target database and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database.
[0032] The writing module is used to write the data to be compensated into the second data table of the target database if the data to be compensated does not exist in the target database.
[0033] Furthermore, the aforementioned acquisition module includes:
[0034] The first acquisition unit is used to acquire migration data from the aforementioned database to the aforementioned target database within a predetermined time period to obtain a first data table, wherein the aforementioned database and the target database are heterogeneous databases, and the aforementioned migration data includes the aforementioned data to be compensated.
[0035] The second acquisition unit is used to acquire a predetermined number of data to be compensated from the first data table to obtain a compensation data table.
[0036] The determining unit is used to determine the table type of the compensation data table based on whether the data to be compensated in the compensation data table contains a timestamp. The table type corresponding to the timestamp is the pre-ordered data table, which includes at least one of the following: customer table and agreement table.
[0037] Furthermore, the aforementioned device also includes:
[0038] The parameter acquisition module is used to acquire predetermined compensation configuration parameters, wherein the compensation configuration parameters include: the predetermined number of processes per session;
[0039] The marking module is used to mark the end of the data compensation loop process if the amount of data to be compensated in the above-mentioned compensation data list is less than the above-mentioned predetermined number of single processing.
[0040] The processing module is used to create multiple threads of a predetermined size to process the data list to be compensated in parallel if the data to be compensated in the above-mentioned compensation data list is greater than or equal to the predetermined number of processing times per session.
[0041] Furthermore, the aforementioned device also includes:
[0042] The flag acquisition module is used to acquire the flags of the data compensation loop process to determine whether the data compensation loop process has ended.
[0043] The recording module is used to record the compensation processing results of each of the multiple threads in the first data table if it is determined that the above data compensation loop process has ended, so as to obtain the processing log. If the above processing log does not report any errors, it indicates that the data compensation is successful.
[0044] The execution module is used to execute the second acquisition unit if it is determined that the above data compensation loop process has not ended.
[0045] Furthermore, the aforementioned device also includes:
[0046] The first judgment module is used to determine whether the data number of the data to be compensated in the above compensation data table is empty;
[0047] The first determining module is used to determine that if the above data number is empty, the compensation processing result corresponding to the above compensation data table is a compensation processing failure.
[0048] The second determining module is used to determine to execute the above query module if the above data number is not empty.
[0049] Furthermore, the aforementioned device also includes:
[0050] The string acquisition module is used to acquire the string inserted or updated in the first data table if the table type indicates that the compensation data table is a data table other than the predetermined data table.
[0051] The second judgment module is used to determine whether the above string already exists in the above second data table;
[0052] The string processing module is used to update the second data table based on the string if the string already exists, and to insert the string into the second data table if the string does not exist.
[0053] On the other hand, this application provides an electronic device, including: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement any of the methods described above.
[0054] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods described above.
[0055] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.
[0056] The data migration method, apparatus, electronic device, and storage medium provided in this application are applied to a unitized architecture. By acquiring a compensation data table and its table type, and since the compensation data table contains missing data to be compensated during the data migration process from the database in the unitized architecture to the target database, when the table type indicates that the compensation data table is a predetermined type of data table, the target database is queried according to the data number of the data to be compensated to determine whether the data to be compensated already exists in the target database. If the data to be compensated already exists in the target database, and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database, the second data table in the target database containing the data to be compensated is updated. If the data to be compensated does not exist in the target database, the data to be compensated is written into the second data table of the target database.
[0057] By providing a data migration compensation scheme, data compensation is performed for data loss or missing data during the data migration process; and data update processing is performed for data updates in the database that are not synchronized to the target database in a timely manner. This can ensure data consistency between heterogeneous databases, avoid the loss of migrated data, and is simple to implement. Attached Figure Description
[0058] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0059] Figure 1 A flowchart illustrating a data migration method provided in an embodiment of this application;
[0060] Figure 2 A flowchart illustrating an optional data migration method provided in an embodiment of this application;
[0061] Figure 3 This is a flowchart illustrating an optional data migration method provided in an embodiment of the present invention;
[0062] Figure 4 This is a flowchart illustrating an optional data migration method provided in an embodiment of the present invention;
[0063] Figure 5 This is a flowchart illustrating an optional data migration method provided in an embodiment of the present invention;
[0064] Figure 6 A structural block diagram of a data migration device provided in an embodiment of this application;
[0065] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0066] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0068] First, let me explain the terms used in this application:
[0069] Unitization: An architectural pattern where a unit is a self-contained set capable of performing all business operations. This set contains all the services required by the business and the data allocated to that unit. Unitized architecture uses units as the basic deployment unit, deploying multiple units across all data centers nationwide. The number of units in each data center is not fixed. Each unit deploys all the applications required by the system, and the data is a portion of the full dataset divided according to a certain dimension.
[0070] Heterogeneous databases are collections of multiple related database systems that enable data sharing and transparent access. These database systems already existed before being added to the heterogeneous database system and each has its own database management system. The various components of the heterogeneous database have their own autonomy. While achieving data sharing, each database system still has its own application characteristics, integrity controls, and security controls.
[0071] During the transformation of some applications to a modular architecture, taking asynchronous databases under a modular architecture as an example, existing data is migrated from an Oracle database to a MySQL database. The hundreds of millions of data points in a single database need to be split into 128 shards. During the daily data synchronization and migration process, data errors, untimely data synchronization, and data loss may occur.
[0072] To address the potential issues of data errors, untimely synchronization, and data loss during daytime data migration or synchronization of heterogeneous databases, this invention proposes a data migration method. This method aims to solve the aforementioned technical problems in existing technologies and provides a compensation mechanism for batch compensation after daytime synchronization, ensuring the consistency of the migrated data.
[0073] It should be noted that the data migration method and apparatus disclosed herein can be used in the financial sector, but also in any sector other than finance. The application areas of the data migration method and apparatus disclosed herein are not limited.
[0074] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0075] This invention provides a data migration method applied to a unitized architecture. Figure 1 This is a flowchart illustrating a data migration method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:
[0076] S101, obtain the compensation data table and the table type of the compensation data table.
[0077] S102, if the table type indicates that the compensation data table is a predetermined type of data table, then query the target database according to the data number of the data to be compensated to determine whether the data to be compensated already exists in the target database.
[0078] S103, if the data to be compensated already exists in the target database, and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database, then update the second data table in the target database that contains the data to be compensated.
[0079] S104, if the data to be compensated does not exist in the target database, then the data to be compensated is written into the second data table of the target database.
[0080] Optionally, the aforementioned compensation data table includes data to be compensated for that was missing during the migration of data from the database in the unitized architecture to the target database.
[0081] In this embodiment, the aforementioned database and target database are heterogeneous databases within a unitized architecture. For example, the database can be an Oracle database, and the target database can be a MySQL database. This embodiment can be applied to scenarios where applications are undergoing unitized architecture transformation. Taking an asynchronous database under a unitized architecture as an example, it involves exporting existing data from an Oracle database to a MySQL database, i.e., importing existing data into a MySQL database.
[0082] Optionally, the migration data from the database to the target database may be, but is not limited to, existing data generated during the transaction process, such as transaction information and customer agreement parameters. Furthermore, the migration data may include any missing data to be compensated for during the migration process.
[0083] For these different types of existing data, different types of compensation data tables can be used to perform compensation processing operations. It should be noted that in the embodiments of this disclosure, the compensation rules configured for different types of compensation data tables are different, and they can be compatible with different compensation and data benchmark rules. Moreover, the rule format is consistent, and the configuration is convenient and simple.
[0084] In one optional embodiment, the table type corresponding to the timestamp is the aforementioned reservation data table, which includes at least one of the following: a customer table and an agreement table.
[0085] For the reservation data table, the MySQL database is queried based on the data number of the data to be compensated in the reservation data table to determine whether the data to be compensated already exists in the MySQL database.
[0086] If the data to be compensated is found to exist in the MySQL database, the first timestamp of the data to be compensated in the compensation data table is compared with the second timestamp of the data to be compensated in the MySQL database to determine whether the data to be compensated in the compensation data table is the updated data.
[0087] To ensure data consistency, if the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the MySQL database, then the data to be compensated in the compensation data table will be used to update the official data table in the MySQL database that contains the data to be compensated.
[0088] If the data to be compensated is not found in the MySQL database, it indicates that the data may be missing or lost and compensation is required. In this case, the data to be compensated is directly written into the official data table of the MySQL database.
[0089] In one example, if the compensation data table is a protocol table, the MySQL database can be queried based on the protocol number of the data to be compensated in the protocol table to determine whether the data to be compensated already exists in the MySQL database.
[0090] If it does not exist, the data to be compensated in the agreement table will be directly written to the official data table in the MySQL database.
[0091] If it exists, to ensure data consistency, and to ensure that the first timestamp of the data to be compensated in the protocol table is greater than the second timestamp of the data to be compensated in the MySQL database, the data to be compensated in the protocol table will be used to update the formal data table in the MySQL database that contains the data to be compensated.
[0092] If the data to be compensated exists, and it is determined that the first timestamp of the data to be compensated in the customer table is greater than the second timestamp of the data to be compensated in the MySQL database, then it can be ignored and no further processing is required.
[0093] In another example, if the compensation data table is a customer table, the MySQL database can be queried based on the customer ID of the data to be compensated in the customer table to determine whether the data to be compensated in the customer table already exists in the MySQL database.
[0094] If it does not exist, the data to be compensated in the customer table will be directly written to the official data table in the MySQL database.
[0095] If it exists, to ensure data consistency, and to ensure that the first timestamp of the data to be compensated in the protocol table is greater than the second timestamp of the data to be compensated in the MySQL database, the data to be compensated in the customer table will be used to update the official data table in the MySQL database that contains the data to be compensated.
[0096] If the data to be compensated exists, and it is determined that the first timestamp of the data to be compensated in the protocol table is greater than the second timestamp of the data to be compensated in the MySQL database, then it can be ignored directly without further processing.
[0097] In one optional embodiment, after the compensation process is completed, the processing result of the temporary data table is updated to "processing successful". In other abnormal cases, the processing result of the temporary data table is updated to "processing failed" and the failure information is recorded. After all records are processed, the compensation process is determined to be over.
[0098] In this embodiment of the disclosure, the processing results are recorded to facilitate data verification of the compensation process. The temporary data table and the formal data table can be checked in both directions. After the check is completed, the processing results need to be recorded, and the formal data table (data compensation) and the temporary data table (updating status and compensation processing results) are updated synchronously, which can effectively maintain the data consistency between heterogeneous databases.
[0099] Therefore, the data migration method provided in this application provides a data migration compensation scheme to compensate for data loss or missing data during the data migration process; and performs data update processing to address the issue of data updates in the Oracle database not being synchronized to the MySQL database in a timely manner. This ensures data consistency between heterogeneous databases and is simple to implement.
[0100] In one example, such as Figure 2 As shown, in step S101 above, the compensation data table and the table type of the compensation data table are obtained, including:
[0101] S201, Obtain migration data from the aforementioned database to the aforementioned target database within a predetermined time period to obtain a first data table.
[0102] S202, Obtain a predetermined number of data to be compensated from the first data table to obtain a compensation data table.
[0103] S203. Determine the table type of the compensation data table based on whether the data to be compensated in the compensation data table contains a timestamp.
[0104] Optionally, the above-mentioned scheduled time period can be determined based on the scheduled migration cycle or the database usage period. For example, the scheduled time period can be set to 24 hours, 3*24 hours, or 7*24 hours, etc.
[0105] Taking a predetermined time period of 24 hours as an example, the migration data from the Oracle database to the MySQL database within 24 hours, i.e., within one working day of data migration, is obtained to obtain the first data table. It can be understood that in this embodiment of the disclosure, the first data table is a temporary data table used to implement data compensation. For example, taking an asynchronous database under a unitized architecture as an example, the existing data table can be exported from the Oracle database to a file first, and then the file import / export component of the unitized architecture can be used to import the file into the temporary data table.
[0106] In this embodiment of the disclosure, by setting up a temporary data table, all migration data from the database to the target database within 24 hours can be obtained at once, avoiding repeated data retrieval, which would result in data loss and tedious operations.
[0107] Optionally, the temporary data table can be deleted after the current data compensation process is completed, or after a predetermined number of data compensation processes are completed.
[0108] In one example, the number and type of data to be compensated each time can be preset. In order to facilitate serial or parallel compensation processing of different types of data to be compensated, a predetermined number of data to be compensated can be obtained from a temporary data table according to the predetermined compensation rules, the predetermined number of data to be compensated each time and the data type of the data to be compensated, so as to obtain a compensation data table.
[0109] For example, in this embodiment of the present disclosure, a predetermined number of data to be compensated in a temporary data table can be obtained according to the predetermined number of single processing operations and the data type of the data to be compensated in the predetermined compensation configuration parameters, so as to obtain a compensation data table.
[0110] In this way, since the data types of the data to be compensated in the compensation data table are consistent, the table type of the compensation data table is determined. Furthermore, the number of data items to be compensated in the compensation data table is fixed, which ensures data consistency and security during the compensation process and prevents the loss of data to be compensated.
[0111] Then, the table type of the compensation data table is determined based on whether the data to be compensated in the compensation data table contains a timestamp.
[0112] In one optional embodiment, the table type corresponding to the timestamp is the aforementioned reservation data table, which includes at least one of the following: a customer table and an agreement table.
[0113] In addition, it should be noted that other data that include timestamp-containing compensation data tables can also be identified as pre-defined data tables, and the data compensation mechanism for such pre-defined data tables provided in the embodiments of this application can also be used. The embodiments of this application do not specifically limit the number and type of pre-defined data tables.
[0114] In another example, such as Figure 3 As shown, before executing S203, the above method further includes:
[0115] S301, Obtain the predetermined compensation configuration parameters, wherein the compensation configuration parameters include: the predetermined number of processes per session.
[0116] S302, if the amount of data to be compensated in the above-mentioned compensation data list is less than the predetermined number of data to be processed in a single operation, then the flag of the data compensation loop process is marked as ended.
[0117] S303, if the data to be compensated in the above-mentioned compensation data list is greater than or equal to the predetermined number of data to be processed in a single operation, then multiple threads of a predetermined size are created to process the above-mentioned data to be compensated list in parallel.
[0118] Optionally, the above compensation configuration parameters may also include, but are not limited to: Uniform Resource Locator URL address, username, password, temporary data table name, and official data table name.
[0119] In one optional embodiment, if it is determined that the amount of data to be compensated in the compensation data list is less than the predetermined number of data to be processed in a single operation, indicating that the amount of data to be compensated is less than the minimum number of data to be compensated in a single operation, then there is no need to continue the compensation process, and the flag of the data compensation loop process can be directly marked as ended.
[0120] It should be noted that the order of S202 and S301 can be replaced without affecting the specific implementation of the solution in the embodiments of this disclosure. The same applies to other method steps. The implementation methods described in the exemplary embodiments do not represent all implementation methods consistent with this disclosure.
[0121] Since the transaction table data of a large database system is very large, often in the hundreds of millions, for the compensation processing of different data tables, in this embodiment of the disclosure, if it is determined that the data to be compensated in the compensation data list is greater than or equal to the predetermined number of processing times per session, then multiple threads of a predetermined size are created to process the above-mentioned data list to be compensated in parallel.
[0122] Furthermore, the embodiments disclosed herein can also support slice processing, serial processing, and parallel processing to meet the user's flexible calling needs during the compensation processing process.
[0123] In another example, such as Figure 4As shown, before executing S202 to obtain a predetermined number of data to be compensated from the first data table to obtain the compensation data table, the method further includes:
[0124] S401, obtain the flag of the data compensation loop process to determine whether the data compensation loop process has ended.
[0125] If it is determined that the above data compensation cycle has ended, then S402 is executed. If it is determined that the above data compensation cycle has not ended, then S202 is executed, which is the step of obtaining a predetermined number of data to be compensated from the first data table to obtain the compensation data table.
[0126] S402, record the compensation processing results corresponding to each of the above multiple threads in the first data table to obtain the processing log. If no errors are reported in the processing log, it indicates that the data compensation was successful.
[0127] Before obtaining a predetermined number of data to be compensated from the first data table to obtain a compensation data table, the flag of the data compensation loop process can be used to determine whether the data compensation loop process has ended. If it is determined that the data compensation loop process has not ended, the steps described above for obtaining a predetermined number of data to be compensated from the first data table to obtain a compensation data table are executed.
[0128] If it is determined that the above data compensation loop process has ended, the compensation processing results corresponding to each of the above multiple threads are recorded in the first data table above to obtain the processing log.
[0129] After obtaining the processing log, it can be output and displayed. If the processing log does not report any errors, it indicates that the data compensation is successful. If the processing log reports an error, it indicates that the data compensation has failed. It is necessary to query the compensation processing thread or the thread that encountered the error based on the processing log in order to continue the data compensation processing to ensure the consistency of the migrated data.
[0130] In one example, before querying the target database based on the data number of the data to be compensated in the compensation data table, the method further includes:
[0131] S501, determine whether the data number of the data to be compensated in the above compensation data table is a null value.
[0132] S502, if the above data number is empty, then the compensation processing result corresponding to the above compensation data table is determined to be compensation processing failure.
[0133] S503, if the above data number is not empty, then determine to execute the above step of querying the target database based on the above data number.
[0134] Optionally, before querying the target database based on the data number of the data to be compensated in the compensation data table, it can be determined whether the data number of the data to be compensated in the compensation data table is null. If it is determined that the data number of all the data to be compensated is null, the compensation processing result corresponding to the compensation data table is directly recorded as compensation processing failure. Only when the data number is not null will the above steps of querying the target database based on the data number continue to be executed.
[0135] In one example, such as Figure 5 As shown, the above method also includes:
[0136] S601, if the table type indicates that the compensation data table is a data table of a type other than the predetermined data table, then obtain the string inserted or updated in the first data table.
[0137] S602, determine whether the above string already exists in the above second data table.
[0138] S603, if the string already exists in the second data table, then update the second data table based on the string; if the string does not exist in the second data table, then insert the string into the second data table.
[0139] In an optional embodiment, if the table type indicates that the compensation data table is a data table other than the predetermined data table, i.e. a data table that does not contain timestamps, such as a system parameter table, then the string inserted or updated in the temporary data table can be obtained. By inserting or updating the string inserted or updated in the temporary data table into the target database in the same way, it can be determined whether the string already exists in the second data table.
[0140] If the string already exists in the second data table, then update the second data table based on the string; if the string does not exist in the second data table, then insert the string into the second data table.
[0141] Furthermore, data that exists in the formal data tables of the target database but not in the temporary data tables can be directly deleted without further updates or synchronization.
[0142] Through the embodiments of this disclosure, different data compensation rules are configured for different types of data tables. Furthermore, in the embodiments of this disclosure, the heterogeneous databases under the unit architecture can be compatible with different data compensation rules between heterogeneous databases, and the data compensation rule format for the same type of data tables is consistent, making configuration simple and convenient to implement.
[0143] According to one or more embodiments of this application, a data migration apparatus is provided, applied to a unitized architecture. Figure 6A structural block diagram of a data migration device provided in an embodiment of this application is shown below. Figure 6 As shown, the above-mentioned device includes:
[0144] The acquisition module 600 is used to acquire the compensation data table and the table type of the compensation data table, wherein the compensation data table contains the missing data to be compensated during the data migration process from the database in the unitized architecture to the target database;
[0145] The query module 601 is used to query the target database based on the data number of the data to be compensated if the table type indicates that the compensation data table is a predetermined type of data table, so as to determine whether the data to be compensated already exists in the target database.
[0146] The update module 602 is used to update the second data table in the target database that contains the data to be compensated if the data to be compensated already exists in the target database and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database.
[0147] The writing module 603 is used to write the data to be compensated into the second data table of the target database if the data to be compensated does not exist in the target database.
[0148] According to one or more embodiments of this application, the above-mentioned acquisition module includes:
[0149] The first acquisition unit is used to acquire migration data from the aforementioned database to the aforementioned target database within a predetermined time period to obtain a first data table, wherein the aforementioned database and the target database are heterogeneous databases, and the aforementioned migration data includes the aforementioned data to be compensated.
[0150] The second acquisition unit is used to acquire a predetermined number of data to be compensated from the first data table to obtain a compensation data table.
[0151] The determining unit is used to determine the table type of the compensation data table based on whether the data to be compensated in the compensation data table contains a timestamp. The table type corresponding to the timestamp is the pre-ordered data table, which includes at least one of the following: customer table and agreement table.
[0152] According to one or more embodiments of this application, the above-described apparatus further includes:
[0153] The parameter acquisition module is used to acquire predetermined compensation configuration parameters, wherein the compensation configuration parameters include: the predetermined number of processes per session;
[0154] The marking module is used to mark the end of the data compensation loop process if the amount of data to be compensated in the above-mentioned compensation data list is less than the above-mentioned predetermined number of single processing.
[0155] The processing module is used to create multiple threads of a predetermined size to process the data list to be compensated in parallel if the data to be compensated in the above-mentioned compensation data list is greater than or equal to the predetermined number of processing times per session.
[0156] According to one or more embodiments of this application, the above-described apparatus further includes:
[0157] The flag acquisition module is used to acquire the flags of the data compensation loop process to determine whether the data compensation loop process has ended.
[0158] The recording module is used to record the compensation processing results of each of the multiple threads in the first data table if it is determined that the above data compensation loop process has ended, so as to obtain the processing log. If the above processing log does not report any errors, it indicates that the data compensation is successful.
[0159] The execution module is used to execute the second acquisition unit if it is determined that the above data compensation loop process has not ended.
[0160] According to one or more embodiments of this application, the above-described apparatus further includes:
[0161] The first judgment module is used to determine whether the data number of the data to be compensated in the above compensation data table is empty;
[0162] The first determining module is used to determine that if the above data number is empty, the compensation processing result corresponding to the above compensation data table is a compensation processing failure.
[0163] The second determining module is used to determine to execute the above query module if the above data number is not empty.
[0164] According to one or more embodiments of this application, the above-described apparatus further includes:
[0165] The string acquisition module is used to acquire the string inserted or updated in the first data table if the table type indicates that the compensation data table is a data table other than the predetermined data table.
[0166] The second judgment module is used to determine whether the above string already exists in the above second data table;
[0167] The string processing module is used to update the second data table based on the string if the string already exists, and to insert the string into the second data table if the string does not exist.
[0168] In an exemplary embodiment, this application also provides an electronic device, including: a processor, and a memory connected to the processor;
[0169] The aforementioned memory stores instructions executed by the computer;
[0170] The processor executes computer execution instructions stored in the memory to implement any of the methods described above.
[0171] In an exemplary embodiment, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described above.
[0172] In an exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.
[0173] To implement the above embodiments, this application also provides an electronic device.
[0174] refer to Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of this application. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0175] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0176] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0177] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.
[0178] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0179] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0180] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.
[0181] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof. These programming languages include object-oriented programming languages—such as Java, SmallTalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0183] The units described in the embodiments of this application can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".
[0184] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0185] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
Claims
1. A data migration method, characterized in that, Applied to a modular architecture, the method includes: Obtain the compensation data table and the table type of the compensation data table, wherein the compensation data table contains the missing data to be compensated during the data migration process from the database in the unitized architecture to the target database; If the table type indicates that the compensation data table is a predetermined type of data table, then the target database is queried according to the data number of the data to be compensated to determine whether the data to be compensated already exists in the target database; If the data to be compensated already exists in the target database, and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database, then update the second data table in the target database that contains the data to be compensated. If the data to be compensated does not exist in the target database, then the data to be compensated is written into the second data table of the target database.
2. The method according to claim 1, characterized in that, The acquisition of the compensation data table, and the table type of the compensation data table, includes: Migration data from the database to the target database within a predetermined time period is obtained to obtain a first data table, wherein the database and the target database are heterogeneous databases, and the migration data includes the data to be compensated; Obtain a predetermined number of data to be compensated from the first data table to obtain a compensation data table; The table type of the compensation data table is determined based on whether the data to be compensated in the compensation data table contains a timestamp. The table type corresponding to the timestamp is the reservation data table, which includes at least one of the following: customer table and agreement table.
3. The method according to claim 1, characterized in that, Before obtaining the table type of the compensation data table, the method further includes: Obtain predetermined compensation configuration parameters, wherein the compensation configuration parameters include: a predetermined number of processes per session; If the amount of data to be compensated in the compensation data table is less than the predetermined number of data to be processed in a single session, then the flag of the data compensation loop process will be marked as ended. If the amount of data to be compensated in the compensation data table is greater than or equal to the predetermined number of data to be processed in a single session, then multiple threads of a predetermined size are created to process the list of data to be compensated in parallel.
4. The method according to claim 2, characterized in that, Before obtaining a predetermined number of data items to be compensated from the first data table to obtain a compensation data table, the method further includes: Obtain the flag of the data compensation loop process to determine whether the data compensation loop process has ended; If it is determined that the data compensation loop process has ended, the compensation processing results corresponding to each of the multiple threads are recorded in the first data table to obtain the processing log. If the processing log does not report any errors, it indicates that the data compensation is successful. If it is determined that the data compensation loop process has not ended, then the step of obtaining a predetermined number of data to be compensated from the first data table to obtain a compensation data table is executed.
5. The method according to any one of claims 1 to 4, characterized in that, Before querying the target database based on the data number of the data to be compensated in the compensation data table, the method further includes: If it is determined that the data number of the data to be compensated in the compensation data table is empty, then the compensation processing result corresponding to the compensation data table is determined to be compensation processing failure. If it is determined that the data number of the data to be compensated in the compensation data table is not null, then the step of querying the target database based on the data number is executed.
6. The method according to claim 2, characterized in that, The method further includes: If the table type indicates that the compensation data table is a data table of a type other than the predetermined data table, then obtain the string of the first data table being inserted or updated; If it is determined that the string already exists in the second data table, then the second data table is updated based on the string; if it is determined that the string does not exist in the second data table, then the string is inserted into the second data table.
7. A data migration device, characterized in that, Applied to a modular architecture, the device includes: The acquisition module is used to acquire the compensation data table and the table type of the compensation data table, wherein the compensation data table contains the missing data to be compensated during the data migration process from the database in the unitized architecture to the target database; The query module is used to query the target database based on the data number of the data to be compensated if the table type indicates that the compensation data table is a predetermined type of data table, so as to determine whether the data to be compensated already exists in the target database; The update module is used to update the second data table in the target database that contains the data to be compensated if the data to be compensated already exists in the target database and the first timestamp of the data to be compensated in the compensation data table is greater than the second timestamp of the data to be compensated in the target database. The writing module is used to write the data to be compensated into a second data table of the target database if the data to be compensated does not exist in the target database.
8. An electronic device, characterized in that, include: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.