Data migration method, device, storage medium and program product across heterogeneous databases

Abstract

Embodiments of the present application provide a data migration method and device across heterogeneous databases, a storage medium and a program product, relating to the technical field of databases. The method comprises: in response to a migration task, obtaining source data information from at least one source database through a source connector; mapping the source data information into standardized data according to a preset unified data model; inputting the standardized data into a schema mapping engine, and generating target mapping configuration information corresponding to a target database according to a preset mapping rule by the schema mapping engine; converting the standardized data into data in the format of the target database based on the target mapping configuration information to obtain target-ready data; and writing the target-ready data into the target database through a target connector. The method of the present application increases the compatibility of heterogeneous databases and improves the automation degree and processing efficiency of data migration.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data migration method, device, storage medium, and program product for cross-heterogeneous databases. Background Technology

[0002] As enterprises continue their digital transformation, their IT architectures are becoming increasingly complex, and data storage systems typically include various heterogeneous systems such as relational databases, non-relational databases, search engines, and object storage systems. Against this backdrop, the need for cross-system data migration is becoming increasingly frequent due to factors such as business integration and technology upgrades.

[0003] Currently, technical solutions for handling such needs typically rely on dedicated migration tools for specific combinations of source and target databases. The basic implementation process of this solution is as follows: first, the table structure definitions and data content of the source database are parsed and extracted; then, according to preset or configurable mapping and transformation rules, the data model and content of the source end are converted into command statements that the target database can recognize and execute; finally, by executing these statements in the target database, the reconstruction of the data structure and content is completed.

[0004] However, the solutions described above are difficult to adapt flexibly to diverse database models and syntactic differences. Summary of the Invention

[0005] This application provides a data migration method, device, storage medium, and program product across heterogeneous databases to solve the technical problem of difficulty in flexibly adapting to diverse database models and syntax differences.

[0006] In a first aspect, embodiments of this application provide a data migration method across heterogeneous databases, the method comprising:

[0007] In response to the migration task, source data information is obtained from at least one source database through the source connector, wherein the source connector is a component selected and loaded from the plugin library according to the type of the source database;

[0008] Based on a pre-defined unified data model, source data information is mapped to standardized data, wherein the unified data model is configured with standardized data types that are independent of the underlying database type;

[0009] Standardized data is input into the pattern mapping engine, which then generates target mapping configuration information corresponding to the target database based on preset mapping rules.

[0010] Based on the target mapping configuration information, the standardized data is converted into data in the target database format to obtain target ready data;

[0011] The target-ready data is written to the target database through the target connector. The target connector is a component selected from the plugin library according to the type of the target database and encapsulates the batch write operation for the target database.

[0012] In this embodiment, a three-layer decoupled architecture—comprising a pluggable connector layer, a unified data model middleware layer, and an automated schema mapping engine—effectively shields the underlying differences between the source and target databases in terms of data types, storage models, and query syntax. This method achieves automated structural migration and efficient data conversion from heterogeneous data sources to multiple target databases, reducing the costs of manual configuration and script development, and improving the execution efficiency and reliability of large-scale data migration tasks. This allows it to adapt to the data flow needs of complex and ever-changing enterprise IT architectures.

[0013] In one possible implementation, the source data information is mapped to standardized data, including:

[0014] Based on predefined type mapping relationships, the field types in the source data information are converted into standardized field types defined in the unified data model.

[0015] In this implementation, by introducing an automated field type conversion mechanism based on a predefined rule base, this embodiment transforms the traditional type adaptation work, which relies on manual judgment and writing, into a standardized process executed automatically by the system. This design makes the type mapping process from heterogeneous data sources to a unified intermediate model accurate, efficient, and repeatable, avoiding mapping errors and inconsistencies that may be caused by manual intervention, and improving the reliability and overall efficiency of the initial stage of data migration.

[0016] In one possible implementation, target mapping configuration information corresponding to the target database is generated, including:

[0017] The schema mapping engine performs compatibility analysis on the field types in the standardized data and the native data types supported by the target database to generate field type mapping rules; the target mapping configuration information contains the field type mapping rules.

[0018] In this implementation, by performing rule-based automated compatibility analysis through a schema mapping engine, this embodiment achieves precise adaptation from a unified data model to a specific target database. This mechanism can automatically handle complex type semantic differences between heterogeneous databases and generate optimal field mapping rules, transforming the traditional structure mapping work, which requires extensive database expertise and manual trial and error, into an efficient and reliable automated system process, thus improving the accuracy and efficiency of structure transformation during data migration.

[0019] In one possible implementation, before converting standardized data into data in the target database format based on target mapping configuration information, the following is also included:

[0020] The standardized data is compared with the target database to detect at least one of the risks of field name conflicts, data type incompatibility, or loss of data precision, and corresponding data repair rules are generated based on the detection results.

[0021] The data repair rules include at least one of the following: field renaming rules, type weakening rules, or data truncation rules, and the target mapping configuration information also includes data repair rules.

[0022] In this implementation, by adding an automated risk pre-detection and remediation rule generation step before the final data conversion, this embodiment achieves defense and handling of potential compatibility issues during the migration process. This mechanism can identify and avoid risks such as field name conflicts, type incompatibility, and data truncation in advance, and transforms solutions into automatically executable remediation rules. This transforms post-event manual investigation and remediation into pre-event systematic prevention, enhancing the security of the data migration process and ensuring the quality of the final data.

[0023] In one possible implementation, the plugin library serves as a centralized management module for providing multiple connector components;

[0024] Select and load source connectors from the plugin library based on the type of the source database, including: matching the type identifier of the source database with the connector components registered in the plugin library, and loading the matched connector components.

[0025] In this implementation, by adopting a centralized and standardized plugin library architecture and a runtime dynamic matching and loading mechanism, this embodiment decouples the data migration system from specific database technologies. This design allows adding or updating support for a database simply by registering a new connector component with the plugin library, without requiring modification or a restart of the core migration engine. This improves the system's scalability and maintainability, and reduces the development and maintenance costs of adapting the system to new data environments.

[0026] In one possible implementation, target-ready data is written to the target database via the target-end connector, including:

[0027] The target-end connector organizes the target-ready data into a data stream that conforms to the target database's native batch import protocol format and transmits it to the target database for writing.

[0028] In this implementation, by encapsulating and calling the database's native batch import protocol using a target-side connector, this embodiment optimizes the data write path from inefficient single-statement execution to high-speed batch data stream transmission. This mechanism reduces the interaction overhead between the application layer and the database layer and fully utilizes the database engine's own loading capabilities, thereby improving write performance when facing large-scale data migration tasks and ensuring the timeliness requirements of enterprise-level data migration.

[0029] In one possible implementation, the method further includes:

[0030] By modifying the data capture module, incremental data changes can be obtained in real time from at least one source database.

[0031] Based on a pre-defined unified data model, incremental data changes are mapped to incremental standardized data;

[0032] Based on the target mapping configuration information, the incremental standardized data is converted into incremental target data in the target database format;

[0033] Incremental target data is written to the target database in batches through the target connector.

[0034] In this implementation, by reusing the core data model, mapping rules, and write links established during the full migration, and introducing a change data capture module, this embodiment constructs a real-time incremental synchronization capability consistent with the full migration architecture and unified data semantics without requiring repeated development of conversion logic. This solution extends from one-time data migration to continuous data reuse, enabling low-latency responses to real-time changes in the source system while ensuring data consistency and processing efficiency, thus meeting the requirements for timeliness and integrity of data flow in real-world migration scenarios.

[0035] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0036] The memory stores instructions that the computer executes;

[0037] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0038] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0039] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 A flowchart illustrating a data migration method across heterogeneous databases provided in this application embodiment;

[0042] Figure 2 A schematic diagram of an architecture representing a unified data model is provided for an embodiment of this application;

[0043] Figure 3 A flowchart illustrating a data writing path provided in an embodiment of this application;

[0044] Figure 4 A schematic diagram of the architecture representing a unified migration engine is provided for embodiments of this application;

[0045] Figure 5 A flowchart illustrating an automated structure mapping provided in this application embodiment;

[0046] Figure 6 This is a schematic diagram illustrating a scenario representing a Connector plugin system, provided as an embodiment of this application.

[0047] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0049] It should be noted that the data migration method, device, storage medium and program product across heterogeneous databases provided in this application can be used in the field of database technology, or in any field other than database technology. This application does not limit the application field of the data migration method, device, storage medium and program product across heterogeneous databases.

[0050] Specific application scenarios for this application include data migration across heterogeneous data storage systems within enterprise IT systems. Typical application examples include: migrating data from existing relational databases to different types of databases or storage systems during enterprise database upgrades or architecture adjustments; migrating unstructured data from document-based databases to relational databases during business system integration; achieving data migration between object storage systems from different cloud service providers in hybrid or multi-cloud environments; extracting data from multiple heterogeneous source databases and converging it to a unified target data platform when building an enterprise-level data warehouse; and achieving smooth and complete migration of historical data during comprehensive system upgrades or database product replacements.

[0051] In the aforementioned application scenarios, migration systems need to be capable of handling large-scale data volumes while being compatible with various data models, including structured, semi-structured, and unstructured data. Furthermore, these application scenarios also include complex migration requirements such as merging and migrating data from multiple source databases to a unified target, and distributing data from a single data source to multiple heterogeneous target databases.

[0052] For the above application scenarios, existing technologies typically employ the following implementation methods for data migration:

[0053] A typical implementation approach is to use specialized migration tools based on database combinations. These tools are specifically designed and developed for a particular combination of source and target databases. The specific implementation process includes: First, establishing a connection with the source database through the database's native protocol to extract complete metadata (such as table structure, field attributes, and constraints) and the actual data content; second, calling a dedicated mapping module pre-developed for this specific database combination to perform structured conversion operations, including data type conversion, SQL syntax rewriting, and database feature adaptation (such as indexes and partitioning strategies); finally, executing the reconstructed data definitions and data manipulation instructions through the target database's standard interface to complete the physical migration of the data.

[0054] Another typical implementation approach involves a customized migration solution based on scripts and manual configuration. This solution is usually implemented manually by technical personnel according to specific migration requirements. The technical implementation process generally includes: first, extracting metadata and actual data from the source database using database management tools or custom scripts; second, analyzing the differences in data models between the source and target systems by technical personnel, and manually writing detailed data type mapping tables, format conversion scripts, and business logic processing code; finally, executing these custom scripts to import the processed data into the target system in stages. This method offers high flexibility in implementation, with the technical implementation details entirely defined by manually written script logic.

[0055] In addition, another typical implementation approach is to use a migration scheme based on general extraction, transformation, and loading (ETL) tools. This type of scheme utilizes a mature ETL platform to achieve cross-system data flow. Specifically, the ETL tool connects the source and target databases through a connector or driver to perform data extraction; during the transformation phase, the ETL tool provides a graphical interface or scripting environment, allowing users to configure a series of transformation steps (such as field mapping, calculation, aggregation, filtering, etc.), which are executed within the tool's built-in engine; finally, the tool loads the transformed data onto the target system.

[0056] The core technology of this solution lies in a configurable workflow engine that coordinates the entire migration process according to a predefined data processing pipeline. Some advanced ETL tools also provide native support modules for various data formats and a limited number of database types.

[0057] However, the solution described above has the following technical problems:

[0058] First, existing solutions lack versatility and scalability. Whether it's a dedicated migration tool for a specific database combination or a general ETL tool, their architecture is typically tightly coupled with the implementation mechanism or fixed transformation rules of the specific database. This makes it difficult to flexibly extend and iterate on different database types without modifying the core code of the migration engine.

[0059] Secondly, existing solutions have a low degree of automation in data structure mapping and transformation. The migration process heavily relies on pre-built, limited rule sets or extensive manual configuration and script writing. This is especially true when dealing with scenarios with significant differences in data models, such as migrating from a document database to a relational database. Existing technologies struggle to automatically and accurately complete data type conversion, table structure mapping, constraint matching, and index strategy generation, resulting in inefficient and error-prone pre-migration preparation.

[0060] Furthermore, existing technologies have limited ability to provide unified support for heterogeneous data models. Relational, document-oriented, and search engine systems employ different data organization methods, while existing migration tools lack an intermediate representation layer capable of uniformly describing and supporting various data structures. This makes it impossible for the migration process to handle structured table data, semi-structured JSON, and complex nested documents in a consistent and efficient manner, potentially leading to the loss of data semantics during the conversion process.

[0061] Finally, when facing large-scale enterprise-level data migration tasks, the batch processing and write mechanisms of existing tools often suffer from performance bottlenecks. There is a lack of optimization strategies tailored to the storage characteristics of different target databases; for example, the data sharding scheduling and parallel write control are not finely designed, making it difficult to meet the strict migration time window requirements and impacting overall migration efficiency.

[0062] The data migration method across heterogeneous databases provided in this application aims to solve the aforementioned technical problems of existing technologies. By constructing a layered technical architecture that includes a multi-source heterogeneous adaptation layer and a unified data model, and combining a plug-in extension mechanism and an automated mapping engine, a general, efficient, and easily scalable cross-database data migration solution is achieved.

[0063] Specifically, the first step is to adopt a plug-in design concept, encapsulating the unique connection protocols, data access interfaces and basic operations of different databases into standardized source and target connectors, thereby decoupling the heterogeneity of the underlying database from the core migration process.

[0064] Secondly, by defining a unified data model with broad expressive capabilities, serving as an intermediate representation layer for data and metadata during the migration process, the solution abstracts the specific differences between different databases in terms of data types, storage models, and query semantics. Based on this, the solution integrates an automated schema analysis, mapping, and transformation engine. Using predefined rules and learnable strategies, it generates transformation logic from the source structure to the target structure, reducing the need for manual intervention and improving transformation accuracy.

[0065] Meanwhile, to meet the performance requirements of large-scale data migration, a batch write optimization mechanism was designed, including distributed task scheduling, data sharding parallel processing, and batch write optimization based on the characteristics of the target database, to ensure overall migration efficiency.

[0066] Ultimately, through the aforementioned layered abstraction and modular design, this architecture enables the system to flexibly adapt to new database types by adding or removing plugins without modifying the core engine, thereby addressing technical issues such as poor compatibility with heterogeneous databases, complex structure mapping, insufficient migration performance, and difficulty in expansion.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] First, combine Figures 1 to 6 This application introduces a data migration method across heterogeneous databases provided in its embodiments. Figure 1This is a flowchart illustrating a data migration method across heterogeneous databases provided in an embodiment of this application. Figure 2 This application provides a schematic diagram of an architecture representing a unified data model. Figure 3 This is a flowchart illustrating a data writing path provided in an embodiment of this application. Figure 4 This application provides an embodiment of an architecture diagram representing a unified migration engine. Figure 5 This application provides a schematic diagram of a process for representing automated structure mapping. Figure 6 This diagram illustrates a scenario representing a Connector plugin architecture, as provided in an embodiment of this application. The method can be executed by a data migration system deployed on a server, which is compatible with multiple data sources and targets through a plug-in architecture. For example... Figure 1 As shown, the method includes the following steps:

[0069] S101. In response to the migration task, obtain source data information from at least one source database through the source connector.

[0070] Specifically, a user or upstream system initiates a migration task to the Unified Migration Engine via API or configuration interface. This task specifies the connection information of the source database (e.g., MySQL, Oracle, MongoDB, etc.), the range of data to be migrated (e.g., table names, query conditions), and the information of the target database.

[0071] In response to this task, the unified migration engine dynamically loads the corresponding source connector from a pre-built plugin library based on the source database type identifier (e.g., "mysql" or "oracle"). This source connector is a software component that encapsulates specific database access protocols (e.g., JDBC, MongoDB driver protocols) and data extraction logic.

[0072] Once loading is complete, the source connector connects to the specified source database according to the task configuration and performs data extraction. This operation obtains two types of key information: first, metadata, namely schema information describing the data structure, including table names, field names, field data types, primary keys, indexes, constraints, etc.; and second, the actual stored data records. These two parts together constitute the source data information.

[0073] S102. Based on the preset unified data model, map the source data information into standardized data.

[0074] This embodiment presupposes a strictly regulated Unified Data Model (UDM) as the core technology layer to shield the heterogeneity of the underlying databases. See also Figure 2 This illustrates the core structure of the unified data model in this application embodiment. The UDM model is an intermediate semantic layer that uses standardized "tables" as organizational units and unified "columns" as the description basis, thereby forming a data representation that is independent of any specific database implementation.

[0075] Specifically, the model first defines a comprehensive data type system:

[0076] Basic types: used to represent common scalar values, such as UDM_STRING (string), UDM_INT (integer), UDM_DATE (date and time), UDM_NUMERIC (high-precision number), etc.

[0077] Complex types: used to support semi-structured and unstructured data, including UDM_JSON (JSON object), UDM_ARRAY (array), and UDM_MAP (key-value pair collection).

[0078] Structured types: used to describe records with a fixed set of fields, such as UDM_RECORD, allowing nesting of the above basic types and complex types, thus enabling the expression of diverse data structures from simple tables to complex nested objects.

[0079] During data migration, the Transformer in the heterogeneous adaptation layer performs the transformation from the source to the UDM according to the predefined type mapping relationship. For example... Figure 2 As shown, the original table structure from the source database (e.g., MySQL, Oracle, MongoDB) is mapped to a table in the UDM, where each field is abstracted into a column. Each column contains standardized metadata such as the field name, a uniform UDM data type, precision, and whether it is nullable. For example, the native types NUMBER, TEXT, and Document from the source database are mapped to standardized types such as UDM_NUMERIC, UDM_STRING, and UDM_JSON, respectively.

[0080] This mapping process also includes data normalization. The converter performs data verification and cleaning simultaneously, including unified control of string length, standardization of numerical precision and scale, cleanup and escaping of special characters, and conversion of various date and time formats to internal standard formats (such as ISO 8601), thereby ensuring the semantic and format consistency of the generated intermediate data.

[0081] Ultimately, the converter, by executing the aforementioned mapping and normalization logic, transforms the raw, heterogeneous source data into an intermediate representation that conforms to the unified data model specification—that is, standardized data. This standardized data provides a unified and clear semantic foundation for subsequent conversion to the target database.

[0082] S103. Input the standardized data into the pattern mapping engine, which then generates target mapping configuration information corresponding to the target database based on the preset mapping rules.

[0083] The standardized data output by S102 and the associated UDM-formatted metadata are input into the schema mapping engine (also known as the schema mapper).

[0084] The schema mapping engine has a built-in or linked mapping rule base, which contains pre-defined knowledge about the conversion and compatibility between different database systems in terms of data types, constraints, and indexing mechanisms. Based on this rule base, the engine automatically performs compatibility comparison and analysis between the input standardized data and the target database's own type system and constraint characteristics. The analysis includes, but is not limited to: whether field types can be directly mapped (e.g., UDM_INT -> PostgreSQL's INTEGER), whether there is a risk of precision loss (e.g., UDM_DECIMAL(38,10) -> MySQL's DECIMAL(20,5)), and how to adapt to the target database's unique constraints (e.g., NOT NULL constraints, unique indexes) and index types.

[0085] After the analysis is complete, the engine generates a structured target mapping configuration. This information includes at least field type mapping rules, i.e., instructions on which native field type each UDM field should be converted to in the target database; it may also include data repair rules generated to handle conflicts or incompatibilities, such as field renaming rules (to avoid conflicts with target database keywords), type weakening strategies (e.g., downgrading JSON type to long text), or data truncation rules.

[0086] S104. Based on the target mapping configuration information, the standardized data is converted into data in the target database format to obtain target ready data.

[0087] This step is performed again by the converter. The converter reads the target mapping configuration information generated in S103 and performs the final transformation on the standardized data generated in S102 accordingly.

[0088] Specifically, the converter transforms each field in the standardized data from UDM format to the native data type required by the target database, based on field type mapping rules. Simultaneously, if the configuration information includes data repair rules (such as renaming or truncation), these are also applied in this step.

[0089] After this step, the data is fully adapted to the target database in terms of both structure and content, generating target-ready data that can be directly received and understood by the target database.

[0090] S105. Write the target ready data into the target database through the target connector.

[0091] The unified migration engine loads the corresponding target connector from the plugin library based on the target database type (e.g., PostgreSQL, Elasticsearch) specified in the migration task.

[0092] Similar to the source connector, the target connector is a component that encapsulates the specific target database's write protocol and optimization strategies. Furthermore, the target connector also encapsulates efficient batch write operations for that target database, such as the PostgreSQL `COPY FROM STDIN` command, the MySQL `LOAD DATA` statement, and the Elasticsearch Bulk API. The target connector receives the target-ready data generated by S104, calls the encapsulated batch write interface, and efficiently and accurately loads the data into the target database, completing the core data flow of the migration task.

[0093] To more clearly explain the transformation and flow of data in the above process, Figure 3 The data writing path in this embodiment is illustrated. From the perspective of component functionality, this path clarifies the core processing stages that data undergoes from the source to the target.

[0094] like Figure 3 As shown, the data migration write path is mainly completed by the sequential collaboration of three core components:

[0095] Reader: Serves as the starting point of the path, corresponding to step S101. It extracts raw data from source databases such as MySQL, Oracle, and MongoDB via the source connector.

[0096] Transformer: As the core processing step, it corresponds to steps S102 and S104. First, the data provided by the reader is converted into the system's internal unified data model (UDM) format to mask differences from the source (corresponding to S102). Then, based on mapping rules, the UDM data is converted into the final format required by the target database (corresponding to S104).

[0097] Target Connector: As the endpoint of the path, corresponding to step S105. It is responsible for efficiently writing the data processed by the converter to target databases such as PostgreSQL, SQL Server, and Elasticsearch using optimized write strategies (e.g., batch operations).

[0098] This path design embodies the concept of layered decoupling, with different components handling the three stages of reading, transformation, and writing, facilitating independent optimization and expansion. Meanwhile, the UDM, acting as the transformation hub, ensures data consistency and semantic integrity as it flows through each stage.

[0099] Further, see Figure 4 The diagram illustrates the system architecture upon which the embodiments of this application are based. This architecture, through layered decoupling and plug-in design, specifically carries and implements the method flow described in S101 to S105 above.

[0100] The Unified Migration Engine serves as the core control layer, responsible for responding to migration tasks and scheduling, monitoring status, and managing anomalies throughout the entire process.

[0101] The Heterogeneous Adapter Layer, located beneath the engine, is the core processing layer responsible for data transformation and mapping. It primarily comprises two key modules:

[0102] Schema Mapper: Corresponding to step S103, it is responsible for performing automated structure mapping and rule generation.

[0103] Transformer: Corresponding to steps S102 and S104, it is responsible for performing data standardization mapping and target-oriented final format conversion.

[0104] The system's input and output ends are flexibly expandable through a plug-in mechanism. The source connector, corresponding to step S101, is responsible for extracting data from various source databases; the target connector, corresponding to step S105, is responsible for efficiently writing data to various target databases. For example... Figure 4 As shown, by loading different connector plugins, this system can be compatible with multiple source databases such as MySQL, Oracle, and MongoDB, as well as multiple target databases such as PostgreSQL, SQL Server, and Elasticsearch.

[0105] To further reveal the core mapping logic, Figure 5The detailed workflow of the Schema mapper performing automated structure mapping in this embodiment is shown.

[0106] like Figure 5 As shown, the process begins with the source schema extracted from the source database, which contains complete metadata such as table structure, field types, and constraints. The core of the process is the schema analyzer, which performs deep parsing of the source schema and automatically identifies key differences between the source schema and the target schema (i.e., the type system and constraint rules of the target database). These differences include, but are not limited to: compatibility of field type semantics, differences in numerical precision and string length, differences in timezone and format representation of date and time types, and differences in the support for constraints (such as primary keys and unique constraints) and indexes across different databases.

[0107] Based on this analysis, the schema analyzer will generate a structured recommended mapping scheme, which mainly includes the following:

[0108] Automatic mapping rules: For example, automatically mapping a specific field type on the source side to the most compatible type on the target side.

[0109] Target table data definition language statements: The system can automatically generate data definition language statements used to create table structures in the target database.

[0110] List of incompatible items: Clearly list the fields or constraints that cannot be processed automatically and require manual review or intervention, such as complex type conversions that pose a risk of semantic loss.

[0111] Ultimately, the process outputs a complete, mapped schema compatible with the target database, which serves as a core component of the target mapping configuration information and guides subsequent data transformation and table structure creation.

[0112] In summary, through Figure 5 The automated analysis and intelligent recommendation mechanism shown in this embodiment, through the schema mapper, realizes the transformation from human experience-driven to rule-based intelligent driving, improving the accuracy and efficiency of structural migration between heterogeneous databases.

[0113] In addition, to achieve flexible support for multi-source, multi-target databases, Figure 6 The Connector plugin system and its data flow used in this embodiment are further illustrated.

[0114] like Figure 6 As shown, the system clearly distinguishes three roles: Source, Plugin, and Target, and achieves decoupling through UDM.

[0115] On the data extraction side, the system provides corresponding Source Connector plugins for different types of source databases, such as MySQL, Oracle, and MongoDB. Each plugin encapsulates a private protocol and logic for establishing a connection with a specific source database and performing data extraction (such as full scan or change data capture). The core function is to uniformly convert the raw data read from the source database into the system's internal UDM format within the plugin, thereby shielding the differences between the source databases.

[0116] On the data writing side, the system provides corresponding Target Connector plugins for target databases such as PostgreSQL, SQL Server, and Elasticsearch. Each plugin encapsulates an efficient write interface for the target database (such as batch import commands). Its core function is to convert UDM format data into a format natively compatible with the target database and perform the write operation, thereby adapting to the characteristics of the target database.

[0117] The core workflow of this system is characterized by a unidirectional data flow: data extraction (Source→Plugin) → unified transformation (Plugin→UDM) → target writing (UDM→Target). All data is transferred and processed through the UDM intermediate layer.

[0118] In summary, through Figure 6 The plug-in architecture shown in this embodiment encapsulates the database interaction logic in pluggable components, decoupling the core migration engine from specific databases. When new database types need to be supported, only corresponding plug-ins implementing the standard interfaces need to be developed, without modifying the core engine, thus improving the system's scalability and maintainability.

[0119] The data migration method across heterogeneous databases provided in this embodiment effectively shields the underlying differences in data types, storage models, and query syntax between the source and target databases by constructing a three-layer decoupled architecture consisting of a pluggable connector layer, a unified data model middleware layer, and an automated schema mapping engine. This method achieves automated structural migration and efficient data conversion from heterogeneous data sources to various target databases, reducing the cost of manual configuration and script development, and improving the execution efficiency and reliability of large-scale data migration tasks. This allows it to adapt to the data flow needs of complex and ever-changing enterprise IT architectures.

[0120] Optionally, the process of mapping source data information to standardized data as described in step S102 is specifically accomplished by a converter performing automated field type conversion.

[0121] The converter accesses a pre-built type mapping rule base, which defines the correspondence between native field types from various source databases and standardized field types in the unified data model. Based on this, the converter identifies the native type of each field in the source data and converts it into the corresponding standardized field type defined in the unified data model according to the mapping relationship.

[0122] For example, the system will automatically perform the following conversions according to rules: converting NUMBER type fields from Oracle databases to UDM_NUMERIC type, converting VARCHAR or TEXT type fields from MySQL databases to UDM_STRING type, and converting Document structures from MongoDB to UDM_JSON type. This process is completed in a unified, batch manner, without the need for manual configuration of individual fields. This ensures accurate type semantic conversion while improving the efficiency and consistency of data standardization processing, providing a reliable type foundation for all subsequent processing steps based on a unified data model.

[0123] By introducing an automated field type conversion mechanism based on a predefined rule base, this embodiment transforms the traditional type adaptation work, which relies on manual judgment and writing, into a standardized process executed automatically by the system. This design makes the type mapping process from heterogeneous data sources to a unified intermediate model accurate, efficient, and repeatable, avoiding mapping errors and inconsistencies that may be caused by manual intervention, and improving the reliability and overall efficiency of the initial stage of data migration.

[0124] Optionally, the process of generating target mapping configuration information corresponding to the target database in step S103 is implemented by the pattern mapping engine through intelligent compatibility analysis.

[0125] Specifically, the schema mapping engine receives standardized data from step S102, where the fields are uniformly expressed as standardized types in a unified data model. The engine then accesses a pre-built mapping rule base and, in conjunction with the native data types, constraint characteristics, and best practices supported by the target database, performs automated compatibility analysis. The core of this analysis lies in finding the best-matching native type in the target database's type system that is semantically compatible and meets performance requirements for each standardized field type in the standardized data.

[0126] Based on this analysis, the schema mapping engine generates a set of field type mapping rules. For example, the engine may decide to map the UDM_NUMERIC type in standardized data to the NUMERIC type in PostgreSQL, or to map the UDM_JSON type to the JSON type in MySQL (if the target library supports it), and mark cases of incompatibility or risk of precision loss.

[0127] Ultimately, these automatically generated field type mapping rules are integrated into the target mapping configuration information, providing a precise and reliable set of instructions for subsequent data transformation steps, thereby achieving adaptation from a unified model to a specific target database.

[0128] This embodiment achieves precise adaptation from a unified data model to a specific target database by performing automated compatibility analysis based on a rule base using a schema mapping engine. This mechanism can automatically handle complex type semantic differences between heterogeneous databases and generate optimal field mapping rules, transforming the traditional structure mapping work, which requires extensive database expertise and manual trial and error, into an efficient and reliable automated process, thus improving the accuracy and efficiency of structure transformation during data migration.

[0129] Optionally, to further ensure the integrity and reliability of data migration, the system adds a pre-inspection and repair rule generation step before performing the final data transformation based on the target mapping configuration information.

[0130] Specifically, after obtaining the configuration information of the standardized data and the target database, the system will perform an automated, in-depth comparative analysis of both. This analysis aims to proactively identify potential risks, mainly including: detecting field name conflicts caused by keywords or reserved words with the same names as source fields in the target database; assessing whether there are data type incompatibilities between the standardized types in the unified data model and the native types in the target database that cannot be directly mapped; and determining whether there is a risk of data precision loss in numeric or string fields due to precision or length limitations of the target fields.

[0131] Based on the above detection results, the system will automatically generate corresponding data repair rules to mitigate risks. For example, for field name conflicts, the system generates field renaming rules to suggest or automatically adopt new field names; for cases of incompatible types that can be downgraded, it generates type weakening rules (e.g., weakening UDM_JSON to UDM_STRING); for potentially excessively long data, it generates data truncation rules and specifies a safe truncation strategy.

[0132] These data repair rules are ultimately incorporated into the target mapping configuration information generated in the aforementioned steps, enabling subsequent data transformation processes to be carried out in accordance with best practices and security constraints. This improves the level of migration automation while preventing data loss or write failures due to unforeseen compatibility issues.

[0133] By adding an automated risk pre-detection and remediation rule generation step before the final data transformation, this embodiment achieves defense and handling of potential compatibility issues during the migration process. This mechanism can identify and avoid risks such as field name conflicts, type incompatibility, and data truncation in advance, and transform the solutions into automatically executable remediation rules. This transforms post-event manual investigation and remediation into pre-event systematic prevention, enhancing the security of the data migration process and ensuring the quality of the final data.

[0134] Optionally, the system achieves flexible adaptation to diverse databases through a centralized and scalable plug-in library architecture. This plug-in library, as the core service module of the system, maintains a connector registry internally, storing all available source-end connectors and target-end connector components in a structured manner (e.g., configuration files or a metadata database).

[0135] Each connector component must undergo standardized registration upon being added to the library. Registration information must include at least: a unique connector identifier, supported database type identifiers (e.g., "mysql", "oracle:12c", "mongodb:4.2"), the storage path of the connector implementation file or the name of the loadable class, and a version compatibility description. This plugin library provides a unified query interface.

[0136] When a migration task is triggered and a source database type is specified (e.g., "mysql:8.0"), the unified migration engine calls the query interface of the plugin library to perform an exact or fuzzy match between the specified database type identifier and the supported types declared by all source connector components in the registry. The matching algorithm prioritizes matching identifiers that are exactly the same; if no exact match is found, the most suitable version is selected according to version compatibility rules.

[0137] Upon successful matching, the engine dynamically loads the corresponding connector implementation code from the specified file system path or network repository using a class loader or dynamic link library loading mechanism, based on the path or class name recorded in the registration information. After loading, the system instantiates and initializes the component, configuring its connection parameters (e.g., address, port, credentials), thereby generating a source-side connector instance in a ready state. The entire process is automatically scheduled by the system, requiring no service interruption or modification of core code. This achieves plug-and-play and runtime scalability for data source access, providing a foundation for cross-heterogeneous database compatibility across the entire migration system.

[0138] By adopting a centralized and standardized plugin library architecture and a runtime dynamic matching and loading mechanism, this embodiment decouples the data migration system from specific database technologies. This design allows adding or updating support for a new database simply by registering a new connector component with the plugin library, without requiring modification or a restart of the core migration engine. This improves the system's scalability and maintainability, and reduces the development and maintenance costs of adapting the system to new data environments.

[0139] Optionally, the target connector is responsible for efficiently persisting the transformed target-ready data to the target database, which is based on utilizing the database's native high-performance batch import mechanism.

[0140] Specifically, after receiving the target-ready data, the target connector invokes its internally encapsulated batch data organization logic, optimized for the specific target database, based on the type of database being connected. For example, for PostgreSQL, the connector organizes the data into a text or binary data stream conforming to the format required by the COPY FROM STDIN command; for MySQL, it generates a LOAD DATA statement containing multiple rows of values ​​or an optimized batch INSERT statement; and for Elasticsearch, it assembles the data into a JSON payload conforming to the Bulk API format.

[0141] Once the data is organized, the target connector transmits the assembled batch data stream directly to the target database server via a persistent database connection. Upon receiving this data stream, the database server performs batch write operations internally. This process reduces network round trips and SQL statement parsing overhead, and fully utilizes the database engine's own batch transaction processing capabilities, thereby achieving throughput and execution efficiency far exceeding traditional line-by-line write methods. This is particularly beneficial in scenarios involving massive data migration, as it significantly shortens the overall migration time window.

[0142] By employing a target-side connector to encapsulate and invoke the database's native batch import protocol, this embodiment optimizes the data write path from inefficient single-statement execution to high-speed batch data stream transmission. This mechanism reduces the interaction overhead between the application layer and the database layer and fully utilizes the database engine's own loading capabilities, thereby improving write performance when facing large-scale data migration tasks and ensuring the timeliness requirements of enterprise-level data migration.

[0143] Optionally, after completing the full data migration, the system can initiate an independent but architecturally consistent incremental data synchronization process to achieve data consistency between the source database and the target database.

[0144] The core of this process is the Change Data Capture (CDC) module. As an independent sub-service of the system, this module employs appropriate log parsing or event monitoring technologies for different types of source databases. For example, for relational databases that support binary logs, such as MySQL or PostgreSQL, the CDC module connects to the source database as a slave or through a parsing tool, reading and parsing the binary logs (binlog or WAL) in real time to extract detailed change events of committed transactions. For Oracle databases, it can read the redo log based on the LogMiner or XStream interface. For NoSQL databases such as MongoDB, it monitors the oplog or change stream API.

[0145] Each captured change event is standardized into a record containing the following fields: operation type (INSERT / UPDATE / DELETE), complete row data or document content (update operations include before and after mirroring), the name of the table / collection to which it belongs, primary key information, and a strictly incrementing logical timestamp (such as LSN or transaction ID) to ensure the ordering and replayability of the events.

[0146] After acquiring the incremental change event stream, the system integrates it into the existing data processing pipeline. First, the converter receives the event stream and, based on the same predefined type mapping relationships as the full migration, maps the changed data contained in the events (whether newly inserted rows, updated fields, or pre-deletion mirror images) from the source database's native format to incrementally normalized data conforming to UDM specifications. This step ensures that incremental data and full data have completely consistent semantic representations within the system.

[0147] Subsequently, the system directly reuses the target mapping configuration information (including field type mapping rules, constraint adaptation rules, etc.) generated and verified during the full migration phase. Based on this configuration, the converter transforms the incremental standardized data into incremental target data in the native format of the target database. Because the mapping rules are consistent, this ensures the compatibility between the data definition of incremental synchronization and the full migration results.

[0148] Finally, the incremental target data is delivered to the target connector. The connector uses the same batch write strategy as the full write (e.g., COPY for PostgreSQL, LOAD DATA for MySQL) to write the accumulated incremental changes to the target database in batches. The entire incremental process is monitored and scheduled by a unified migration engine, and real-time performance and throughput can be balanced by configuring parameters such as consumption latency thresholds and batch size.

[0149] By reusing the core data model, mapping rules, and write chain established during the full migration, and introducing a change data capture module, this embodiment constructs a real-time incremental synchronization capability consistent with the full migration architecture and unified data semantics without requiring repeated development of conversion logic. This solution extends from one-time data migration to continuous data reuse, enabling low-latency responses to real-time changes in the source system while ensuring data consistency and processing efficiency, thus meeting the requirements for timeliness and integrity of data flow in real-world migration scenarios.

[0150] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0151] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0152] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0153] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0154] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0158] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0159] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0160] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0161] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0162] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0163] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0164] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data migration method across heterogeneous databases, characterized in that, The method includes: In response to a migration task, source data information is obtained from at least one source database through a source connector, wherein the source connector is a component selected and loaded from a plugin library according to the type of the source database; Based on a preset unified data model, the source data information is mapped to standardized data, wherein the unified data model is configured with standardized data types that are independent of the underlying database type; The standardized data is input into the pattern mapping engine, which generates target mapping configuration information corresponding to the target database according to preset mapping rules. Based on the target mapping configuration information, the standardized data is converted into target database format data to obtain target ready data; The target ready data is written to the target database through the target connector, wherein the target connector is a component selected and loaded from the plugin library according to the type of the target database and encapsulates batch write operations for the target database.

2. The method according to claim 1, characterized in that, The process of mapping the source data information to standardized data includes: Based on the predefined type mapping relationship, the field types in the source data information are converted into the standardized field types defined in the unified data model.

3. The method according to claim 1, characterized in that, The generation of target mapping configuration information corresponding to the target database includes: The pattern mapping engine performs compatibility analysis on the field types in the standardized data and the native data types supported by the target database to generate field type mapping rules; wherein, the target mapping configuration information contains the field type mapping rules.

4. The method according to any one of claims 1-3, characterized in that, Before converting the standardized data into data in the target database format based on the target mapping configuration information, the method further includes: The standardized data is compared with the target database to detect at least one of the risks of field name conflicts, data type incompatibility, or data precision loss, and corresponding data repair rules are generated based on the detection results. The data repair rules include at least one of field renaming rules, type weakening rules, or data truncation rules, and the target mapping configuration information also includes the data repair rules.

5. The method according to claim 1, characterized in that, The plug-in library is a centralized management module for providing multiple connector components; The step of selecting and loading the source connector from the plugin library according to the type of the source database includes: matching the type identifier of the source database with the connector components registered in the plugin library, and loading the matched connector component.

6. The method according to claim 1, characterized in that, The step of writing the target ready data into the target database via the target connector includes: The target-ready data is organized into a data stream conforming to the target database's native batch import protocol format via the target connector and transmitted to the target database for writing.

7. The method according to claim 1, characterized in that, The method further includes: By changing the data capture module, incremental data changes are obtained in real time from the at least one source database; Based on the preset unified data model, the incremental data change is mapped to incremental standardized data; Based on the target mapping configuration information, the incremental standardized data is converted into incremental target data in the target database format; The incremental target data is written to the target database in batches through the target connector.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images