Cross-heterogeneous database data migration method and system

Abstract

The invention discloses a cross-heterogeneous database data migration method and system, and the method comprises the steps: building a multi-data-source connection among multiple data sources, and enabling the multi-data-source connection to be used for representing the incidence relation between a source database and a target database; obtaining a source data table based on the migration sql, and querying source data corresponding to the source data table from a source database; converting the source data based on a preset model mapping rule to obtain target data; determining a target database into which the source data is migrated according to the multi-data source connection, and writing the target data into the target database to realize data migration; and performing data verification on the target database according to the checking rule to obtain a migration result, and generating a result report according to the migration result. According to the method, the source data is converted into the target data through the preset model mapping rule, migration obstacles caused by structural differences among heterogeneous databases are effectively solved, the consistency of the data is ensured through the checking rule, and the accuracy of data migration is improved.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and specifically to a data migration method and system for heterogeneous databases. Background Technology

[0002] As banks advance their digital transformation through IT innovation, the process of building new systems necessitates processing and transforming the business data from the old systems into new ones, and then migrating it to the new systems. To ensure the complete and flawless migration of business data to the new systems, meet their rules and requirements, and guarantee the accuracy of the final data, a development platform is needed to facilitate the data migration process, thereby helping banks complete their digital transformation and upgrade.

[0003] Data migration presents numerous challenges. Firstly, the old and new systems often exhibit heterogeneity in database structure, such as differences in data types, table structures, and field naming, increasing the complexity and difficulty of the migration. Secondly, business data frequently contains multi-modal information, such as text, numerical values, and images. Different modalities require corresponding conversion and processing during the migration process to ensure data consistency and accuracy. Furthermore, data validation is necessary during migration to ensure that the migrated data complies with the rules and requirements of the new system, avoiding business risks caused by data errors or omissions.

[0004] Therefore, it is particularly important to develop a multimodal, configurable data migration and verification system that supports domestic substitution. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data migration method and system across heterogeneous databases, aiming to achieve data migration between heterogeneous databases and improve the accuracy of migration.

[0006] This invention discloses a data migration method across heterogeneous databases, comprising: Establish multi-data source connections between multiple data sources. These connections are used to represent the relationship between the source database and the target database. The source data table is obtained based on the migration SQL, and the source data corresponding to the source data table is queried from the source database. The source data is transformed based on the preset model mapping rules to obtain the target data; The target database to which the source data is migrated is determined based on the connection to multiple data sources, and the target data is written to the target database to achieve data migration; The target database is validated according to the verification rules to obtain the migration results, and a result report is generated based on the migration results.

[0007] Preferably, establishing multi-data source connections between multiple data sources includes: Data nodes are generated based on the connection configuration information of the source database and the target database; data nodes are used to represent the communication channel and data mapping relationship between the source database and the target database. Assign source data tables to data nodes; The routing rules for the source data tables are determined based on a predefined database sharding algorithm. These routing rules indicate the data transmission path between the source and target databases.

[0008] Preferably, the process involves obtaining the source data table based on the migration SQL and querying the source data corresponding to the source data table from the source database, including: Parse the migration SQL to obtain the source data tables contained in the migration SQL; The source database corresponding to the source data table is determined based on the data node; Obtain the number of source databases, and generate migration failure information if the number of source databases is greater than 1.

[0009] Preferably, the source data is transformed based on a preset model mapping rule to obtain the target data, including: Generate preset model mapping rules based on the mapping configuration dimensions; The source data is converted into intermediate data according to the preset model mapping rules. The intermediate data is in a data format compatible with the target database. The intermediate data is cleaned and deduplicated to obtain the target data.

[0010] Preferably, the mapping configuration dimensions include: Source table name, used to indicate the data source table for the target field; Source data table field name, used to indicate the data source field of the target field; The conversion rule types include: T1-shift, T2-code value conversion, T3-default value, T4-multi-field concatenation, T5-custom processing implementation, T6-arithmic operation, and T7-date format conversion. The conversion rule types are used to characterize the preset data processing logic. The target table name is used to indicate the target table where the transformed data will be written. The target table field is used to indicate the target field where the transformed data will be written.

[0011] Preferably, writing the target data into the target database includes: Get the target table name corresponding to the target data; A subtask is generated based on the target table name. The subtask is used to represent writing the target data into the target table corresponding to the target table name. Configure the execution parameters for the subtask, including the number of parallel operations, the number of batch submissions, the number of pages, and the execution interface; An execution plan is generated based on the subtasks, and target data is written to the target database according to the execution plan.

[0012] Preferably, the verification rules include vertical verification rules, which are used to verify the accuracy of the target data in the target table; Vertical check rules are generated based on vertical configuration dimensions, which include: Stage identifiers include SOU (source database) and TRG (target database); The rule description is used to describe the content of the vertical inspection rules; The check SQL is used to perform queries and validations on the target data. Rule levels are used to indicate the importance of vertical check rules, including C1-Critical, C2-Severe, and C3-Warning; Rule status is used to identify the enabled status of vertical check rules, including A - executable and E - disabled and not executed; The execution result is used to record the verification result after the vertical check rule is executed, including pass, failure, or abnormal status.

[0013] Preferably, the verification rules include horizontal verification rules, which are used to verify the consistency of the aggregated business volume between the target database and the source database; Horizontal verification rules are generated based on horizontal configuration dimensions, which include: The database is summarized and categorized, including SOU (source database) and TRG (target database). Business category, used to distinguish the data aggregation scope of different business types; Summary SQL is used to execute aggregate queries on data of different business categories in the target database; The summary results are used to record the aggregated data results after the execution of the horizontal inspection rules; Difference checking is used to represent the difference between the summarized results of the source database and the target database.

[0014] Preferably, a results report is generated based on the migration results, including: Business reports are generated based on the migration results. These business reports are configurable. Configure the save path, organization code, and report logic for business reports to generate result reports. The organization code is used to indicate the organization to which the result report belongs, and the report logic is used to indicate the generation rules and display dimensions of the report data.

[0015] This invention discloses a data migration system across heterogeneous databases, used to perform a data migration method across heterogeneous databases. The data migration system across heterogeneous databases includes: The data source connection module is configured to establish multi-data source connections between multiple data sources, which are used to represent the relationship between the source database and the target database. The data reading module is configured to: obtain the source data table based on the migration SQL, and query the source data corresponding to the source data table from the source database; The data transformation module is configured to transform the source data based on preset model mapping rules to obtain the target data; The data migration module is configured to: determine the target database to which the source data is to be migrated based on multiple data source connections, and write the target data into the target database to achieve data migration; The report generation module is configured to: perform data verification on the target database according to the verification rules to obtain the migration results, and generate a result report based on the migration results.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: By establishing multi-data source connections between multiple data sources, the association between the source database and the target database is determined. The source data is transformed using preset model mapping rules to ensure that the transformed target data conforms to the format requirements of the target database. Then, the target data is accurately written into the target database through a data migration module, achieving data migration across heterogeneous databases. Simultaneously, verification rules are introduced to comprehensively validate the migrated data, ensuring its accuracy and consistency and effectively avoiding business risks caused by data errors or omissions. Furthermore, configurable business reports and result reports are generated based on the migration results, providing strong data support and decision-making basis, improving the accuracy and efficiency of data migration. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the data migration method across heterogeneous databases provided by this invention; Figure 2 This is a schematic diagram of the data migration system across heterogeneous databases provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings.

[0020] like Figure 1 As shown in the figure, this embodiment of the invention provides a data migration method across heterogeneous databases, including the following steps.

[0021] S1. Establish multi-data source connections between multiple data sources.

[0022] In this embodiment of the invention, multiple data source connections are used to characterize the association between the source database and the target database. By configuring database connection parameters, a communication link between the source database and the target database is established, ensuring the security and stability of the data channel.

[0023] Specifically, in establishing connections between multiple data sources, a data node is first generated based on the connection configuration information of the source and target databases. This data node not only serves as a communication channel between the source and target databases but also defines the data mapping relationship between them, ensuring accurate data correspondence during migration. Next, source data tables are assigned to the data nodes, and routing rules for these tables are determined based on a predefined database sharding algorithm. These routing rules indicate the data transmission path between the source and target databases. In this way, data can be migrated efficiently and accurately between heterogeneous databases along predetermined paths. The database sharding algorithm is designed to consider both the balance of data distribution and migration efficiency, reducing network latency and resource consumption during data migration through intelligent routing selection.

[0024] It's important to note that database sharding algorithms are strategies that dynamically adjust routing based on data characteristics, database load, and network conditions. For example, they may use methods like hash modulo, range partitioning, or consistent hashing to distribute data, ensuring that different source tables are evenly distributed across the corresponding shards in the target database. Simultaneously, the algorithm records version information for routing rules, enabling data flow tracking during subsequent migrations and providing a basis for troubleshooting. Furthermore, the database sharding algorithm supports custom extension interfaces, allowing users to adjust routing logic according to actual business scenarios, such as prioritizing low-load nodes or avoiding faulty areas, thereby further improving the flexibility and reliability of data migration.

[0025] S2. Obtain the source data table based on the migration SQL, and query the source data corresponding to the source data table from the source database.

[0026] In this embodiment of the invention, the migration SQL is pre-written by the user based on the source data table structure and migration requirements, specifying the data range to be migrated and the field mapping relationships. The migration SQL is parsed to obtain the source data tables contained within it. Specifically, during the parsing process, syntax tree parsing technology is used to identify table names, field lists, and filtering conditions in the SQL statement, extracting metadata information from the source data tables, and combining this with predefined mapping relationships in the data nodes to determine the table structure corresponding to the target database.

[0027] Furthermore, based on the previously established data nodes, the source database containing the source data table is located. The number of source databases is obtained, and if the number of source databases is greater than one, a migration failure message is generated. This effectively avoids data duplication or inconsistency issues caused by source database configuration errors or multi-source conflicts, and prevents complex problems caused by multiple source databases from affecting migration quality.

[0028] S3. Based on the preset model mapping rules, perform transformation on the source data to obtain the target data.

[0029] In this embodiment of the invention, after obtaining the source data, preset model mapping rules are generated according to the mapping configuration dimensions. The preset model mapping rules cover multiple aspects, including the source data table name, source data table field names, transformation rule type, target table name, and target table fields, ensuring the comprehensiveness and accuracy of the data transformation. The source data is then converted into intermediate data according to the preset model mapping rules; the intermediate data is in a data format compatible with the target database.

[0030] In some embodiments, intermediate data may be a temporary cache of records to be migrated in the source database, stored in a high-throughput memory buffer, and submitted to the target shard in batches.

[0031] Finally, the intermediate data is cleaned and deduplicated to obtain the target data. This effectively eliminates the risk of duplicate writes caused by data redundancy in the source system or transmission interruptions, ensuring data consistency in the target database.

[0032] Specifically, the mapping configuration dimensions of the preset model mapping rules include source data table name, source data table field name, transformation rule type, target table name, and target table fields. The source data table name indicates the data source table for the target field, the source data table field name indicates the data source field for the target field, and the transformation rule type characterizes the preset data processing logic. Transformation rule types include T1-shift, T2-code value conversion, T3-default value extraction, T4-multi-field concatenation, T5-custom processing implementation, T6-arithmic operation, and T7-date format conversion. The target table name indicates the target table where the transformed data is written, and the target table fields indicate the target fields where the transformed data is written.

[0033] It should be understood that T1 indicates a direct mapping between the source field and the target field, requiring no processing; T2 achieves field value conversion through a preset code value lookup table; T3 fills in a predetermined default value when the source field is empty; T4 concatenates multiple source fields into a single target field according to a specified delimiter; T5 calls an external script or function to complete complex logic processing; T6 performs addition, subtraction, multiplication, and division operations on numeric fields; and T7 uniformly converts the source date string into the target system's standard time format.

[0034] By flexibly combining the above mapping rules, various migration scenarios between heterogeneous databases can be adapted, improving the automation and accuracy of the migration process. In practical applications, multiple transformation rule types can be used in combination to address the characteristics of different types of heterogeneous databases.

[0035] For example, when migrating data from one relational database to another, for simple field mappings, the T1 translation rule is used to directly map source and target fields, quickly completing the migration of basic data. If there are different data encoding methods, the T2 code value conversion rule is used, employing a pre-defined code value lookup table to accurately convert the source data into an encoding that meets the requirements of the target database. When a source field contains null values, and the target database does not allow null values ​​for that field, the T3 default value rule is used to fill in pre-defined default values, ensuring data integrity. For situations where multiple field information needs to be integrated into one field, the following rule is used: The T4 multi-field concatenation rule combines multiple source fields into a single target field using a specified delimiter. For complex data processing logic, such as calculations or judgments involving specific business rules, the T5 custom processing rule is invoked to complete the complex logic processing through external scripts or functions. When processing numeric fields, if numerical calculations such as statistics and summarization are required, the T6 arithmetic operation rule performs addition, subtraction, multiplication, and division operations on the numeric fields. When migrating date fields, the T7 date format conversion rule is used to uniformly convert the source date string into the target system's standard time format, ensuring accurate transfer of date data between different databases. Through this flexible combination, data migration tasks between various heterogeneous databases can be completed efficiently and accurately.

[0036] S4. Determine the target database to which the source data will be migrated based on the multiple data source connections, and write the target data into the target database to achieve data migration.

[0037] In this embodiment of the invention, after determining the target database to which the source data will be migrated based on multiple data source connections, a refined data writing operation needs to be performed to ensure migration quality. First, the target table name corresponding to the target data is obtained. Then, a subtask is generated based on the target table name. The subtask represents writing the target data into the target table corresponding to the target table name. Next, the execution parameters of the subtask are configured, including the number of parallel operations, the number of batch submissions, the number of pages, and the execution interface. Finally, an execution plan is generated based on the subtask, and the target data is written to the target database according to the execution plan.

[0038] The parallelism parameter controls the number of concurrent threads for data writing, improving write efficiency; the batch commit parameter defines the number of data entries committed each time, balancing memory usage and write performance; the page count determines the shard size of the source data, facilitating incremental migration and exception recovery; and the execution interface specifies the database driver protocol used for write operations. By properly configuring these parameters and dynamically adjusting them based on the target database's load capacity and network conditions, a stable and efficient data migration process is ensured.

[0039] In this embodiment of the invention, a target table name corresponds to a subtask, and multiple subtasks can be combined into a subtask group. The execution plan may include multiple subtask groups and / or multiple subtasks, thereby executing the subtasks in parallel to improve the overall migration speed.

[0040] S5. Perform data verification on the target database according to the verification rules to obtain the migration results, and generate a result report based on the migration results.

[0041] In this embodiment of the invention, the work is not immediately completed after data migration. Instead, a comprehensive data verification of the target database is performed according to pre-defined verification rules. These verification rules include two main categories: vertical verification rules and horizontal verification rules, which are used to verify the accuracy of the target data and the consistency of the aggregated business volume between the target database and the source database, respectively. During the verification process, the execution result of each rule is recorded, including pass, failure, or abnormal status. Finally, a migration result report is generated based on these verification results, which reflects the overall quality of the data migration and provides strong support for subsequent data analysis and decision-making. Furthermore, this embodiment of the invention also supports the generation of configurable business reports based on the migration results, further meeting the diverse data display needs of users.

[0042] Specifically, vertical verification rules are generated based on a vertical configuration dimension, which includes a stage identifier, rule description, verification SQL, rule level, rule status, and execution result. The stage identifier marks the migration process stage of the data transformation, including SOU (source database) and TRG (target database). The rule description describes the content of the vertical verification rule, outlining its specific purpose and logic. The verification SQL executes queries and validations on the target data. The rule level indicates the importance of the vertical verification rule, including C1 (fatal), C2 (critical), and C3 (alert). The rule status indicates the enabled status of the vertical verification rule, including A (executable) and E (disabled). The execution result records the validation result after the vertical verification rule is executed, including pass, failure, or exception status.

[0043] For example, when the stage is identified as SOU-Source Database, the rule description can be set to check whether the value range of a specific field in the source data meets the requirements. The corresponding check SQL is written to extract the data of that field from the source database and make a judgment. The rule level is set to C2-Severe. If there are a large number of values ​​outside the range in the source data, it may affect the accuracy of subsequent data conversion and migration. The rule status is set to A-Executable. After execution, the execution result is recorded as pass, failure or abnormal status according to the actual verification situation.

[0044] For example, when the stage is identified as TRG-Target Database, the rule description is set to verify whether the value of a key field in the target database is consistent with the value of the corresponding field in the source database. Write a check SQL to retrieve the data of that field from the target database and compare it with the corresponding data in the source database. The rule level is set to C1 - Fatal, because inconsistency of key fields may lead to errors in business data. The rule status is set to A - Executable, and the corresponding execution result is recorded after execution.

[0045] Vertical verification rules are used to precisely check the consistency of key fields and the range of values ​​in the source and target databases. If data is found to be inconsistent with the standards set by the vertical verification rules, it is immediately marked as a failure or anomaly, and the location and details of the problematic data are recorded. This allows for the timely detection and correction of data deviations caused by data conversion, writing, and other operations during the data migration process, ensuring that the data migrated to the target database is highly consistent with the source database in key dimensions, providing a reliable data foundation for subsequent business processing.

[0046] In this embodiment of the invention, the verification rules also include horizontal verification rules, which are generated based on horizontal configuration dimensions. These horizontal configuration dimensions include summary categories, business categories, summary SQL queries, summary results, and difference verification. The summary categories are used to mark the migration process stage of the data transformation, and include SOU (Source Database) and TRG (Target Database). Business categories are used to distinguish the data summary scope of different business types, such as order categories, user categories, or transaction categories. The summary SQL queries are used to execute aggregate queries on data of different business categories in the target database. The summary results are used to record the aggregated data results after the execution of the horizontal verification rules. The difference verification is used to represent the difference between the summary results of the source database and the target database, thereby comparing whether there is a deviation in the summary data of the source database and the target database under the same business category.

[0047] For example, when the summary category is SOU-Source Database, order-related business data is selected for summary. The corresponding summary SQL is written to extract key indicators such as order amount and order quantity from the order-related data in the source database and perform aggregation calculations to obtain the summary result for the source database. Simultaneously, the corresponding indicators for order-related data are extracted from the target database in the same way and performed aggregation calculations to obtain the summary result for the target database. Then, a difference check is performed to calculate the difference between the two. If the difference is within a reasonable range, the source database and the target database are considered consistent in the summary of order-related business data; if the difference exceeds a reasonable range, the check fails, and the specific difference data is recorded.

[0048] For example, when the summary category is TRG-target database, a horizontal check is performed on user-related business data. Summary SQL is written to extract metrics such as user count and user activity from the target database and aggregate them to obtain the summary results for the target database. These results are then compared with the corresponding summary results for user-related data in the source database, and discrepancies are checked to determine if there are any deviations. If deviations exist, it indicates that there may have been a problem with the summary of user-related business data during the data migration process, requiring further investigation.

[0049] By comprehensively comparing the aggregated data of different business categories in the source and target databases through horizontal verification rules, inconsistencies in aggregated data caused by potential data loss or duplicate calculations during data migration can be effectively identified. This helps ensure that the aggregated data in the target database remains highly consistent with the source database, providing accurate and reliable data for business decisions and data analysis. Furthermore, the migration result report generated based on the execution results of vertical and horizontal verification rules provides a detailed reflection of the quality at each stage of the data migration process, offering clear directions for subsequent data optimization and improvement.

[0050] In this embodiment of the invention, the process of generating a result report based on the migration results includes: generating a business report based on the migration results. The business report is configurable, and the save path, organization code, and report logic of the business report are configured to generate the result report. The organization code indicates the organization to which the result report belongs, the report logic indicates the generation rules and display dimensions of the report data, and the save path specifies the storage location of the result report. Business reports can be automatically aggregated according to organization codes, facilitating data governance and audit tracking across multiple branches. Through preset report logic, key information such as verification items, difference data, and consistency status in the migration results are structured and organized to generate a result report containing an overall overview and detailed analysis.

[0051] In this way, users can quickly locate the problematic parts of the data migration process based on the results report, find migration issues such as data inconsistencies at the field level or data deviations at the business summary level from the results report. At the same time, the configurable business report feature allows different organizations to flexibly adjust the report content and display format according to their own needs, meeting diverse data governance and decision support needs.

[0052] For example, large enterprise groups can set different organization codes to generate data migration result reports for each branch and conduct centralized management and comparative analysis; while small and medium-sized enterprises can customize simple and clear reporting logic according to their own business characteristics.

[0053] like Figure 2 As shown, the present invention also provides a data migration system across heterogeneous databases for performing a data migration method across heterogeneous databases. The data migration system across heterogeneous databases includes a data source connection module 201, a data reading module 202, a data conversion module 203, a data migration module 204, and a report generation module 205.

[0054] The data source connection module 201 is configured to establish multi-data source connections between multiple data sources, which represent the relationship between the source database and the target database. The data reading module 202 is configured to retrieve the source data table based on the migration SQL and query the corresponding source data from the source database. The data transformation module 203 is configured to perform transformations on the source data based on preset model mapping rules to obtain the target data. The data migration module 204 is configured to determine the target database to which the source data will be migrated based on the multi-data source connections and write the target data into the target database to achieve data migration. The report generation module 205 is configured to perform data verification on the target database according to verification rules to obtain the migration results and generate a result report based on the migration results.

[0055] As can be seen from the above technical solution, this application provides a data migration method and system across heterogeneous databases. The data migration method includes: establishing multi-data source connections between multiple data sources, whereby the multi-data source connections represent the relationship between the source database and the target database; obtaining the source data table based on migration SQL and querying the source data corresponding to the source data table from the source database; performing transformation on the source data based on preset model mapping rules to obtain target data; determining the target database to which the source data is migrated based on the multi-data source connections and writing the target data into the target database to achieve data migration; performing data verification on the target database according to verification rules to obtain migration results, and generating a result report based on the migration results. This invention effectively solves the migration obstacles caused by structural differences between heterogeneous databases by converting source data into target data through preset model mapping rules, and ensures data consistency through verification rules, thereby improving the accuracy of data migration.

[0056] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data migration method across heterogeneous databases, characterized in that, include: Establish multi-data source connections between multiple data sources, wherein the multi-data source connections are used to characterize the association between the source database and the target database; The source data table is obtained based on the migration SQL, and the source data corresponding to the source data table is queried from the source database. The source data is transformed based on a preset model mapping rule to obtain the target data; The target database to which the source data is to be migrated is determined based on the multi-source data connection, and the target data is written into the target database to achieve data migration; The target database is validated according to the verification rules to obtain the migration results, and a result report is generated based on the migration results.

2. The data migration method across heterogeneous databases according to claim 1, characterized in that, The establishment of multi-data source connections between multiple data sources includes: Based on the connection configuration information of the source database and the target database, data nodes are generated; the data nodes are used to represent the communication channel and data mapping relationship between the source database and the target database. Assign the source data table to the data node; The routing rules for the source data table are determined based on a predefined database sharding algorithm. These routing rules are used to indicate the transmission path of data between the source database and the target database.

3. The data migration method across heterogeneous databases according to claim 2, characterized in that, The process of obtaining the source data table based on migration SQL and querying the source data corresponding to the source data table from the source database includes: Parse the migration SQL to obtain the source data table contained in the migration SQL; The source database corresponding to the source data table is determined based on the data node; Obtain the number of source databases, and if the number of source databases is greater than 1, generate migration failure information.

4. The data migration method across heterogeneous databases according to claim 1, characterized in that, The transformation of the source data based on preset model mapping rules to obtain target data includes: The preset model mapping rules are generated based on the mapping configuration dimensions; The source data is converted into intermediate data according to the preset model mapping rules, and the intermediate data is in a data format compatible with the target database. The intermediate data is cleaned and deduplicated to obtain the target data.

5. The data migration method across heterogeneous databases according to claim 4, characterized in that, The mapping configuration dimensions include: Source table name, used to indicate the data source table for the target field; Source data table field name, used to indicate the data source field of the target field; The conversion rule types include: T1-shift, T2-code value conversion, T3-default value, T4-multi-field concatenation, T5-custom processing implementation, T6-arithmic operation, and T7-date format conversion. These conversion rule types are used to characterize the preset data processing logic. The target table name is used to indicate the target table where the transformed data will be written. The target table field is used to indicate the target field where the transformed data will be written.

6. The data migration method across heterogeneous databases according to claim 5, characterized in that, The step of writing the target data into the target database includes: Obtain the target table name corresponding to the target data; A subtask is generated based on the target table name, and the subtask is used to represent writing the target data into the target table corresponding to the target table name. Configure the execution parameters of the subtask, including the number of parallel operations, the number of batch submissions, the number of pages, and the execution interface; An execution plan is generated based on the sub-task, and the target data is written to the target database according to the execution plan.

7. The data migration method across heterogeneous databases according to claim 6, characterized in that, The verification rules include vertical verification rules, which are used to verify the accuracy of the target data in the target table. The vertical check rules are generated based on the vertical configuration dimension, which includes: Stage identifiers include SOU (source database) and TRG (target database); The rule description is used to describe the content of the vertical inspection rules; The check SQL is used to perform queries and verifications on the target data. Rule levels are used to identify the importance of the vertical check rules, including C1-Critical, C2-Severe, and C3-Warning; Rule status is used to identify the enabled status of the vertical inspection rule, including A - executable and E - disabled and not executed; The execution result is used to record the verification result after the vertical check rule is executed, including pass, failure, or abnormal status.

8. The data migration method across heterogeneous databases according to claim 6, characterized in that, The verification rules include horizontal verification rules, which are used to verify the consistency of the aggregated business volume between the target database and the source database. The horizontal verification rules are generated based on the horizontal configuration dimension, which includes: The database is summarized and categorized, including SOU (source database) and TRG (target database). Business category, used to distinguish the data aggregation scope of different business types; The summary SQL is used to execute aggregate queries on data of different business categories in the target database; The summary results are used to record the aggregated data results after the execution of the horizontal inspection rules; Difference verification is used to represent the difference between the summarized results of the source database and the target database.

9. The data migration method across heterogeneous databases according to claim 1, characterized in that, The step of generating a result report based on the migration results includes: A business report is generated based on the migration results, and the business report is configurable. Configure the save path, organization code, and report logic of the business report to generate the result report. The organization code is used to indicate the organization to which the result report belongs, and the report logic is used to indicate the generation rules and display dimensions of the report data.

10. A data migration system across heterogeneous databases, used to perform the data migration method across heterogeneous databases as described in any one of claims 1 to 9, characterized in that, include: The data source connection module is configured to: establish multi-data source connections between multiple data sources, wherein the multi-data source connections are used to characterize the association between the source database and the target database; The data reading module is configured to: obtain the source data table based on the migration SQL, and query the source data corresponding to the source data table from the source database; The data conversion module is configured to: perform conversion on the source data based on a preset model mapping rule to obtain the target data; The data migration module is configured to: determine the target database to which the source data is to be migrated based on the multi-data source connection, and write the target data into the target database to achieve data migration; The report generation module is configured to: perform data verification on the target database according to the verification rules to obtain the migration results, and generate a result report based on the migration results.

Images