Method and system for data integration and migration based on model mapping and plug-in extension

Abstract

The application relates to the technical field of data integration and migration, in particular to a data integration and migration method and system based on model mapping and plug-in extension. The system comprises eight standardized links of system registration, plug-in binding, combination configuration, model mapping configuration, data mapping configuration, data conversion, data inspection and data import; the dynamic expansion access of the system is realized through definition of standard plug-in interfaces of model information reading, data analysis, data import, attribute value conversion, attribute inspection and relationship inspection; flexible conversion of heterogeneous data models is completed through a three-level mapping structure of "model-attribute-value"; and the automatic execution of data migration is realized through a task scheduling management mechanism. Through the plug-in architecture and the standardized process, the development and maintenance cost of the data integration and migration of the heterogeneous systems is greatly reduced, and the configuration flexibility, the automation degree and the data reliability are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing and information system integration technology, and more specifically, to a data integration and migration method and system based on model mapping and plug-in extension. Background Technology

[0002] Current dataset expansion and neighborhood migration technologies are gradually developing towards intelligence, but existing technical solutions are mostly single-point breakthrough designs, lacking the ability to integrate the entire process, and are difficult to meet the needs of efficient application in complex scenarios.

[0003] On the one hand, existing technologies fail to achieve deep coupling between semantic-level retrieval models and multi-dimensional analysis models. Data sources lack unified standards, and the formats of raw and expanded data are mixed, hindering standardized storage and efficient reuse. This results in a disconnect between expanded and raw data, creating "technology silos." On the other hand, technical methods are limited to single stages. Semantic retrieval, dimensional analysis, and data generation modules operate independently, failing to achieve closed-loop linkage between knowledge reasoning, pattern recognition, and sample output. Furthermore, expanded data often relies on manually pre-set rules, resulting in insufficient accuracy and incomplete coverage, making it difficult to meet the practical needs of automated and intelligent data processing. In addition, existing technologies lack integration and adaptation to risk classification standards and structured question databases, failing to fully leverage the value of existing technological achievements. This leads to insufficient analytical accuracy, weak decision support capabilities, and an inability to comprehensively improve the overall efficiency of dataset expansion and neighborhood migration.

[0004] Therefore, there is an urgent need to build an integrated intelligent data expansion and neighborhood migration method system to solve problems such as module fragmentation, data disconnection, and inefficient processes, and to achieve integrated application of technical models and closed-loop optimization of data processing. Summary of the Invention

[0005] This application provides a data integration and migration method and tool based on model mapping and plug-in extension. By standardizing the model description and mapping configuration of structured data between heterogeneous information systems, and realizing the ability to dynamically access systems based on an extensible plug-in mechanism, it enables cross-system integration and migration of data between heterogeneous information systems, forming a complete data processing closed loop.

[0006] The present application is described below from different aspects. It should be understood that the different implementation methods and beneficial effects described below can be referenced from each other.

[0007] Firstly, this application provides a data integration and migration method based on model mapping and plugin extensions. This method includes eight standardized steps: system registration, plugin binding, combination configuration, model mapping configuration, data mapping configuration, data transformation, data inspection, and data import. The system registration step registers the system to be integrated with the tool, designating the system as either a data exporter or importer, and entering basic system information, including system identifier, access address, authentication information, and model information description file. The plugin binding step binds corresponding functional plugins to the registered system according to its role. These functional plugins include model information reading plugins, data parsing plugins, data import plugins, attribute value converter plugins, attribute inspector plugins, and relationship inspector plugins. The tool defines standard interfaces for each plugin. Plugins from registered systems can dynamically integrate with the tool by implementing these standard interfaces without modifying the tool's core code. The combination configuration step selects a source system and a target system from the registered systems, generating a unique combination configuration identifier. All subsequent mapping configurations, data processing, and task management are based on this combination configuration identifier. The model mapping configuration stage configures the model correspondence, attribute correspondence, and attribute value correspondence between the source and target systems in a visual interface. The tool dynamically obtains and displays the model structures of both systems through the model information reading plugins bound to the source and target systems. The data mapping configuration stage configures the correspondence between data instances in the source system and existing data instances in the target system. This correspondence is used to correct references to related data during data conversion. The data conversion stage, based on the model mapping configuration, attribute mapping configuration, and attribute value mapping configuration, sequentially completes data model label conversion, attribute label conversion, and attribute value conversion. For data containing relationships, it automatically converts the reference relationships in the source system to the corresponding reference relationships in the target system according to the data mapping configuration. The data inspection stage performs integrity checks, attribute validity checks, and relationship checks before data import. Attribute validity checks are implemented through the attribute inspector plugin, and relationship checks are implemented through the relationship inspector plugin, generating data inspection results and details of abnormal data. The data import stage calls the data import plugin bound to the target system, writing the converted and checked data into the target system. The import plugin returns the identifier of the newly created data in the target system, and the tool records the correspondence between the source and target data.

[0008] In the above solution, the eight standardized steps are dynamically expanded through standard plug-in interfaces. New systems can be added by simply developing the corresponding plug-in, without modifying the core code of the tool, which greatly reduces the development and maintenance costs of data integration and migration of heterogeneous systems.

[0009] In one possible implementation, each combined configuration stored in the combined configuration module includes a unique identifier, a source system identifier, a target system identifier, source system bound plugin information, target system bound plugin information, and custom parameters for import plugin extensions. The tool queries all of the above information using the unique identifier during data processing. The data import plugin interface defines a method for retrieving combined configuration form items. The tool dynamically loads the configuration parameters of the import plugin extensions using this method and collects these parameters in the combined configuration interface. Different import parameter values ​​can be set for the same system in different combined configurations. The extended configuration parameters include tenant identifiers, organization identifiers, user identifiers, etc. The tool associates and stores these extended parameters with the combined configuration identifiers and passes them to the data import plugin during data import.

[0010] In the above implementation, the combination configuration is separated from the system registration, which realizes differentiated parameter configuration of the same system in different migration scenarios. For example, when importing data to different tenants or different company codes, it is only necessary to create a new combination configuration and set the corresponding parameters, without having to re-register the system and bind the plugin.

[0011] In one possible implementation, the model mapping configuration adopts a three-level mapping structure of "model-attribute-value". The first level configures a one-to-one relationship between the source system's data model and the target system's data model. This is used to dynamically obtain the model structures of both systems and display them in a tree structure by calling the model information reading plugins bound to the source and target systems. The second level configures a one-to-one relationship between the source system's model attributes and the target system's model attributes. The third level configures a one-to-one relationship between the source system's attribute values ​​and the target system's attribute values. It supports binding attribute value converter plugins to any attribute mapping. These plugins implement custom attribute value conversion logic and come in three types: target system custom converter plugins, data mapping converter plugins, and imported data converter plugins. The mapping template management unit is used to export the configured model mapping relationships, attribute mapping relationships, and attribute value mapping relationships as templates, and supports importing and reusing them in combined configurations of similar systems.

[0012] In the above implementation, the three-level mapping structure supports flexible conversion of complex heterogeneous data models. The attribute value converter plugin can realize complex attribute value conversion logic such as enumeration value conversion, unit conversion, and formula calculation. The mapping template management unit realizes configuration reuse and reduces repetitive configuration work.

[0013] In one possible implementation, the data mapping list comprises two parts. The first part is the imported data mapping, which dynamically obtains a list of data types requiring mapping through the target system's data import plugin and generates a corresponding data mapping interface based on this list. The generated data mapping result is stored in association with the combined configuration identifier and migration task identifier. The second part is the input data mapping, where configurations are entered through the combined configuration data mapping interface. A data mapping repository is used to persistently store the mapping relationships between source system data instances and target system data instances, as well as the configuration information for model mapping and data mapping. The data in the data mapping repository can be automatically read and used by the data transformation module.

[0014] In the above implementation, the tool persists the mapping relationship through a data correspondence repository, supports the automatic correction of the source system reference relationship to the target system reference relationship, and ensures the integrity of the associated data.

[0015] In one possible implementation, the main data transformation process includes reading data files, converting data model tags, converting data attribute tags, and converting attribute values. Converting attribute values ​​requires an attribute value converter. The attribute value converter has three use cases: If an attribute value in the source data is a reference to another data instance, and the referenced data instance has a corresponding relationship established in the data mapping configuration, the tool automatically converts the reference value into the identifier of the corresponding data instance in the target system through the import data converter plugin; if an attribute value in the source data is transformed based on the data mapping input from the user interface, the tool automatically converts the source attribute value into the input target attribute value through the data mapping converter plugin; if an attribute value in the source data needs to be transformed according to the specific logic of the target system, the tool supports custom extended converters for the target system, and the tool automatically calls the methods in the converter to convert the attribute value into the target value.

[0016] In the above implementation, the tool covers the attribute value conversion needs in different scenarios through three types of attribute value converters, and realizes flexible processing of complex business logic.

[0017] In one possible implementation, the data inspection process first checks the completeness of the data model and attribute mapping, then performs data transformation based on this. Finally, the transformed data is inspected using attribute inspector plugins and relational object inspector plugins. The attribute inspector plugins are returned by the target system's data import plugin based on the data type. Each data type corresponds to several attribute inspectors, and each inspector corresponds to one attribute, responsible for returning the valid value range, data type, length limit, and regular expression rules for that attribute. The tool verifies the validity of the attribute values ​​based on the valid values ​​or rules returned by the plugins. The relational inspector plugins are also returned by the target system's data import plugin based on the data type. Each relational inspector corresponds to a pair of related data, responsible for checking whether a given list of data identifiers exists in the target system. The tool determines the validity of the relationships based on the inspection results returned by the plugins. The tool records the data inspection results and details of abnormal data, generating an inspection report containing the error type, location of the erroneous data, and reason for the error. If the inspection fails, the subsequent data import process is terminated, and the user is prompted to correct the data according to the error log.

[0018] In the above implementation, the data inspection mechanism is implemented through an extensible plugin, which allows for the dynamic expansion of inspection items based on the business rules of the target system, thus ensuring the legality and consistency of the imported data.

[0019] In one possible implementation, the tool groups the inspected data according to the data types configured in the migration task. Each group of data obtains the import interface of the target system through a data import plugin, performs data import, and records the correspondence between the source and target data. The correspondence between the source and target data recorded by the tool can serve as the basis for subsequent data mapping configuration and is automatically used to correct related data references in subsequent migration tasks.

[0020] In the above implementation, the tool reduces the frequency of interface calls through a group import strategy, while the automatically recorded data correspondence provides a basis for correcting related data references in subsequent migration tasks.

[0021] One possible implementation requires unified management of migration tasks. Data transformation, data inspection, and data import processes are encapsulated as executable tasks, supporting two data acquisition methods: uploading source data files or retrieving data via a data parsing plugin that calls the source system's interface. Task creation, execution, pause, termination, and status monitoring functions are provided. Scheduled tasks can be configured for daily, weekly, and monthly automatic execution. Batch creation and execution of multiple migration tasks are supported, along with an automatic retry mechanism for failed data imports and the ability to export, correct, and re-import failed data.

[0022] In the above implementation, the task management mechanism realizes the automated scheduling of data integration and migration, improves execution efficiency through scheduled tasks and batch processing, and ensures the reliability of data import through the automatic retry mechanism.

[0023] In one possible implementation, the tool includes a set of built-in standard plugins, including a built-in model information reading plugin that supports parsing standard model information files in XML or JSON format as defined by the tool; a built-in data parsing plugin that supports parsing standard data exchange files in CSV, XML, or JSON format as defined by the tool; a built-in data converter plugin that provides basic data type conversion, string case conversion, and date format conversion functions; a built-in inspector plugin that includes an attribute value inspector to check the legality and validity of attribute values ​​and an associated object inspector to check whether a given data identifier exists in the target system; and a built-in data import plugin that provides data import interfaces for various business objects in the target system.

[0024] In the above implementation method, the built-in standard plugin set provides out-of-the-box plugin implementation for basic functions, reducing the complexity of system integration.

[0025] In one possible implementation, the standard interfaces defined by the plugin include: a model information reading interface that defines a method for obtaining the system data model structure; a data parsing interface that defines a method for reading the data to be migrated from the source system; a data import interface that defines a method for writing data into the target system and returning a newly created data identifier; an attribute value conversion interface that defines a method for customizing attribute value conversion logic; an attribute checking interface that defines a method for returning attribute validation rules and validating the validity of attribute values; and a relationship checking interface that defines a method for checking the existence of related data in the target system.

[0026] In the above implementation method, the definition of the standard interface provides a unified specification for plugin development, ensuring the compatibility and interchangeability of plugins from different systems.

[0027] Secondly, embodiments of this application provide a data integration and migration tool based on model mapping and plugin extensions. This tool includes a system registration unit, a plugin management unit, a combination configuration unit, a model mapping unit, a data mapping unit, a data conversion unit, a data inspection unit, a data import unit, and a task scheduling and management unit. The system registration unit manages the registration information of systems to be accessed, including system identifiers, access addresses, authentication information, and model information description files. The plugin management unit manages the binding relationships of functional plugins, assigning model information reading plugins, data parsing plugins, data import plugins, attribute value converter plugins, attribute inspector plugins, and relationship inspector plugins to the registered system according to system roles. The combination configuration unit creates and manages the combination configurations of the source and target systems, generating unique combination configuration identifiers and storing extended parameters. The model mapping unit configures and displays the three-level mapping structure, dynamically obtains the model structure by calling plugins, and supports the export and import of mapping templates. The data mapping unit configures the data instance correspondence and manages the data correspondence repository. The data conversion unit performs data model tag conversion, attribute tag conversion, and attribute value conversion, calling the attribute value converter to complete complex conversion logic. The data inspection unit performs integrity checks, attribute validity checks, and relationship checks, generating inspection reports and anomaly details. The data import unit invokes the data import plugin to write data, recording the correspondence between source and target data. The task scheduling and management unit encapsulates executable tasks, manages their lifecycle and execution status, and supports scheduled and batch task configurations.

[0028] In the above solution, the tool standardizes and breaks down the entire process of data integration and migration through modular unit design. Each unit works together through standard interfaces, realizing the dynamic expansion capability of plug-in.

[0029] Thirdly, this application provides a plugin-based data conversion method applied to a data conversion unit. The method includes acquiring an original data file and generating a buffer of original data to be converted; reading and loading a model mapping configuration from external storage to perform data model marking conversion on the original data based on the configuration information; reading and loading an attribute mapping configuration from external storage to perform attribute marking conversion on the original data fields of the marked model according to the configuration; reading and loading a value mapping configuration from external storage to convert attribute values ​​according to the configuration; if the corresponding value mapping configuration does not exist, the original data carrying double-marked model attributes is sent to the attribute value converter plugin cluster to execute field value conversion logic; the attribute values ​​output by the plugin cluster are received; and a complete converted dataset is assembled.

[0030] Fourthly, this application provides a plugin-based data inspection method applied to a data inspection unit. This method includes loading the upstream original business dataset to be imported as the input source for the entire verification process; verifying whether the data model definition, field mapping relationships are missing, and whether field matching is complete; generating exception records directly for missing mappings; converting the original data into a system standard data structure for preprocessing subsequent verification; checking whether data attribute values ​​are valid; calling the attribute inspector plugin to perform field value range verification; calling the attribute inspector plugin to retrieve the list of valid attribute values ​​to pull the legal enumeration or value range of each field; determining whether the current field value is within the legal value range; writing illegal values ​​to the exception database; checking whether referenced data exists; calling the related object inspection plugin to perform foreign key or association verification; calling the related object inspector plugin to query the target system to verify whether the related business object truly exists; generating association exceptions if the associated ID does not exist or the object has been deleted; and transferring the standardized dataset that passes all verifications to the downstream data import stage. All exceptions intercepted by verifications are uniformly stored in the data inspection result exception information details for persistent storage. If the check fails, the subsequent data import process is terminated, and the user is prompted to correct the data according to the error log.

[0031] Compared with the prior art, the present invention has the following beneficial effects: Through system registration and plugin binding mechanisms, this solution enables plug-in-based expansion of data integration and migration tools. Adding a new system only requires developing the corresponding plugin to complete the system integration, without modifying the core tool code, thus solving the problems of high customization and poor scalability in existing technologies. Through combined configuration and dynamic parameter expansion mechanisms, it achieves flexible pairing between source and target systems and differentiated parameter configurations for the same system in different migration scenarios, solving the problem of insufficient configuration flexibility in existing technologies. Through a three-level "model-attribute-value" mapping framework, it supports flexible conversion of complex heterogeneous data models. The attribute value converter plugin implements complex attribute value conversion logic such as enumeration value conversion, unit conversion, and formula calculation, solving the problem of weak model adaptability in existing technologies. Through an extensible data inspection mechanism, it allows dynamic expansion of inspection items based on the business rules of the target system. The attribute inspector plugin and relationship inspector plugin ensure the legality and consistency of imported data, solving the problem of inadequate data quality assurance in existing technologies. Through end-to-end task management, data conversion, inspection, and import are encapsulated into executable tasks, supporting scheduled and batch task configuration, achieving automated data integration and migration, solving the problem of low automation in existing technologies. This invention significantly reduces the development and maintenance costs of data integration and migration in heterogeneous systems, and improves configuration flexibility, automation, and data reliability. Attached Figure Description

[0032] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of a data processing flow based on a modal interpolation control to expand the dataset and a neighborhood migration method, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the semantic retrieval and multi-dimensional analysis collaborative processing architecture in an intelligent data augmentation and neighborhood migration method provided in an embodiment of this application; Figure 3 This is a schematic diagram of an interactive interface provided in an embodiment of this application for achieving standardized storage and efficient reuse of raw data and extended data; Figure 4 This is a schematic diagram of a processing flow that realizes closed-loop linkage between knowledge reasoning, pattern recognition and sample output, provided by an embodiment of this application; Figure 5 This is an interactive schematic diagram provided in an embodiment of this application for performing risk semantic retrieval and multi-dimensional analysis; Figure 6 This is a schematic diagram of system interaction for realizing closed-loop control and automated data processing throughout the entire process, provided by an embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] In this invention, "send" and "receive" indicate the direction of signal transmission. For example, "send information to XX" can be understood as the destination of the information being XX, which can include sending directly through a network interface or sending indirectly through other units or modules via a bus. "Receive information from YY" can be understood as the source of the information being YY, which can include receiving directly from YY through a network interface or receiving indirectly from YY through middleware. "Send" can also be understood as the "output" of a chip interface, and "receive" can also be understood as the "input" of a chip interface. In other words, sending and receiving can occur between devices, for example, through buses, traces, or interfaces between components, modules, chips, software modules, or hardware modules within a device.

[0035] The technical solutions of this invention can be applied to data integration and migration scenarios between heterogeneous information systems, which can employ various data exchange protocols. These protocols include, but are not limited to, HTTP / HTTPS, JDBC, ODBC, RESTful API, SOAP, or custom binary protocols, etc., which are not listed here.

[0036] The technical solutions of this invention can be applied to data interaction scenarios between the data exporting system and the data importing system, as well as to interaction scenarios between the data conversion module and the data checking module, and to interaction scenarios between the task scheduling module and various functional modules. In this invention, the term "communication" can also be described as "data exchange," "information flow," or "signaling transmission." In this invention, the term "transmission" can also be described as "uploading" and / or "downloading."

[0037] like Figure 1 As shown, the data integration and migration tool based on model mapping and plugin extension includes: a system registration unit, a plugin management unit, a combination configuration unit, a model mapping unit, a data mapping unit, a data transformation unit, a data inspection unit, a data import unit, and a task scheduling and management unit.

[0038] In one possible implementation, the system registration unit can be a device with system information management functions, supporting communication via standard interfaces and having the ability to communicate with the plug-in management unit or the combined configuration unit. This device with system registration functions can be a complete machine, or it can be a software plug-in or logic circuit installed in the complete machine. Devices with these plug-ins or circuits installed can implement the methods and functions of the embodiments of the present invention under the control of a processor. The system registration unit can be deployed on an application server, a cloud platform, or a local server. The system registration unit can be understood as a bridge connecting heterogeneous systems and a unified configuration platform, its main function being to standardize and uniformly manage the access information of different heterogeneous systems. For example, the system registration unit can be a web application or desktop client with a management interface.

[0039] In one possible implementation, the plug-in management unit can be a device with plug-in lifecycle management capabilities, supporting plug-in loading, plug-in binding, and plug-in unloading operations, and having the ability to communicate with the system registration unit or the combination configuration unit. This device with plug-in management capabilities can be a full-machine server, a computing board installed in a rack-mount device, or a parallel processing system. The plug-in management unit can also be a high-performance computing cluster, an edge computing node, or a cloud virtual machine. For example, the plug-in management unit can be an application server that supports dynamic loading, a container platform that supports hot-swapping of plug-ins, or a plug-in management system integrated into a data integration tool.

[0040] The functional plugins typically include model information reading plugins (e.g., XML model parsers or JSON model parsers), data parsing plugins (e.g., CSV parsers or XML parsers), data import plugins (e.g., JDBC batch import plugins or API import plugins), attribute value converter plugins (e.g., enumeration value converters or formula converters), attribute checker plugins (e.g., format validators or range validators), and relationship checker plugins (e.g., foreign key validators or association validators). These will not be listed individually here. In embodiments of this invention, the system including these functional plugins can be referred to as a plugin library. In practical applications, this invention does not limit the number and type of plugins in the system.

[0041] The system architecture of the embodiments of the present invention will be described below. It should be noted that the system architecture and application scenarios described in the embodiments of the present invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. As those skilled in the art will know, as the system architecture or application scenarios evolve, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0042] The data integration and migration tool based on model mapping and plug-in extensions provided by this invention is applicable to various heterogeneous system data integration scenarios, including Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Supply Chain Management (SCM), and Office Automation (OA) systems. The tool is suitable for different data migration scenarios, such as single-batch migration (e.g., historical data import), incremental synchronization (e.g., real-time data synchronization), and multi-system parallel migration. The following explanation uses a multi-system parallel migration scenario as an example.

[0043] Please see Figure 2 , Figure 2 This is a schematic diagram of a data integration and migration tool system architecture based on model mapping and plug-in extensions provided by an embodiment of the present invention. Figure 1 .like Figure 2 As shown, this tool can include a system registration unit, a plugin management unit, a combination configuration unit, a model mapping unit, a data mapping unit, a data conversion unit, a data inspection unit, a data import unit, and a task scheduling and management unit. The number of functional plugins can be one or more; here, we can take six as an example. Specifically, these could include a model information reading plugin, a data parsing plugin, a data import plugin, an attribute value converter plugin, an attribute inspector plugin, and a relationship inspector plugin. This can be understood as... Figure 2 This is just an illustration; the tool could also include other units, such as a mapping template management unit and a data mapping repository. Figure 2 It is not shown in the middle.

[0044] The data storage method used by this tool can be exemplified by a relational database. This data correspondence repository mainly refers to a standardized database deployed on the application server, used to store system registration information, combined configuration information, model mapping configuration information, data mapping configuration information, and the correspondence between source data and target data. It achieves data interoperability with various units through standardized interfaces.

[0045] Understandably, each unit in this tool can communicate with a data mapping repository, which mainly refers to a database system (e.g., MySQL, Oracle, or PostgreSQL) used for persistent storage of various configuration information and mapping relationships.

[0046] by Figure 2 Taking the data conversion unit shown as an example, it can be used to communicate with the data inspection unit. For instance, the data conversion unit can send the converted data to the data inspection unit for integrity checks and legality verification. Figure 2As shown, the data conversion unit in this embodiment of the invention can establish three different types of data processing channels, specifically including data channel P1, data channel P2, and data channel P3, as detailed below: Data channel P1: refers to the communication connection between the data conversion unit and the data inspection unit achieved by the internal process call method.

[0047] Data Channel P2: This refers to the asynchronous message queue established between the data transformation unit and the data mapping unit. It is mainly used for the transmission of transformed data (e.g., using message queue middleware such as RabbitMQ or Kafka). For example, the data transformation unit can provide a message publishing interface for the data inspection unit to subscribe to. Correspondingly, the data inspection unit can receive transformed data by subscribing to the message queue, thereby establishing a communication connection with the data transformation unit.

[0048] Data Channel P3: This refers to the direct database access channel established by the data transformation unit with the data mapping repository. It can be used to read model mapping configuration information, attribute mapping configuration information, and value mapping configuration information. It should be understood that the data mapping repository here can include: a relational database server, a distributed caching cluster, or a key-value store system.

[0049] In other words, the aforementioned data conversion unit can not only be used to achieve direct communication connection with the data inspection unit through data channel P1, but also to achieve data interaction with the data mapping unit or the data correspondence repository through data channel P2 or data channel P3.

[0050] In one possible implementation, please see Figure 3 , Figure 3 This is a schematic diagram of a data integration and migration tool system architecture based on model mapping and plug-in extensions provided by an embodiment of the present invention. Figure 2 .like Figure 3 As shown, this tool can include a system registration unit, a plugin management unit, a combination configuration unit, a model mapping unit, a data mapping unit, a data transformation unit, a data inspection unit, a data import unit, a task scheduling and management unit, and a mapping template management unit. The system registration unit can manage one or more systems; here, we'll use two as an example, specifically source system A and target system B. This is understandable. Figure 3 This is just an illustration; the tool could also include other devices, such as a visual configuration interface and a task monitoring panel. Figure 3 It is not shown in the middle.

[0051] Compared to Figure 2The tool shown also includes an additional mapping template management unit. This unit can perform mapping template export and import operations in one stop. That is, this unit can be used to export the configured model mapping relationship, attribute mapping relationship, and attribute value mapping relationship as template files, and supports importing and reusing them in the combined configuration of similar systems. This unit can be a dedicated template management server to provide template reuse functionality for configuration personnel.

[0052] like Figure 3 As shown, the mapping template management unit communicates with the model mapping unit via data channel P1, and can also synchronize data with the combination configuration unit via data channel P2. For a detailed description of the communication of the mapping template management unit, please refer to the above. Figure 2 The description of the data correspondence repository shown will not be repeated here.

[0053] Understandably, because the tool has a mapping template management unit, mapping configurations for similar systems in different combinations of configurations (e.g., customer master data migration between two different ERP systems) can be quickly generated through the template reuse function, which greatly reduces repetitive configuration work.

[0054] It should be understood that in this invention, the management device (e.g., an application server or a database server) can perform some or all of the steps in the various embodiments. These steps or operations are merely examples, and the embodiments of this invention can also perform other operations or variations thereof. Furthermore, the steps can be performed in different orders as presented in the various embodiments, and it is not necessary to perform all the operations in the embodiments of this invention. For ease of understanding, the management device in this invention can be exemplified by an application server to illustrate specific implementations of how to perform system registration and plugin binding.

[0055] In one possible implementation, during system operation, the application server can synchronize the system registration information held by both parties with the data mapping repository at preset intervals (e.g., 5 minutes) to facilitate unified management of heterogeneous systems. This system registration information may include system identifiers, access addresses, authentication information, and model information description files. The model information description files, which describe the system model information, can be dynamically updated according to the technical characteristics of the heterogeneous systems. For example, if a system subsequently upgrades its data model structure, the corresponding model information description file will be updated in the system registration information. In other words, after a synchronization, the system registration information held by the application server may include system identifiers, access addresses, authentication information, and model information description files.

[0056] The combination of system identifier and model information file can be used to represent different system access methods. As shown in Table 1 above, the authentication information and model information files held by the application server differ across different systems. This means that the system registration configuration is not fixed during system operation but can be dynamically adjusted according to actual business needs, which improves the flexibility of system access. Furthermore, in the case of parallel migration of multiple systems, the system registration information from the last synchronization will participate in the subsequent plugin binding process, thereby accurately identifying the system type and binding the corresponding functional plugins.

[0057] For ease of explanation, in this embodiment of the invention, the currently activated system registration information can be referred to as the first registration information, and the historically synchronized system registration information can be referred to as the second registration information. The historical synchronization occurred earlier than the current activation time.

[0058] As shown in Table 1 above, the model information file formats differ across systems. This means that the system model parsing rules are not fixed during system operation but can be dynamically adjusted according to actual business needs, thus improving the flexibility of model mapping configuration. Furthermore, in the case of parallel migration across multiple systems, the model information file from the last synchronization will participate in the subsequent process of building model mapping relationships, thereby accurately identifying the system data model structure.

[0059] To facilitate understanding of the above plugin binding process, please refer to Table 2, which is a plugin binding strategy mapping table stored in a tool provided in an embodiment of the present invention. Specifically, as shown in Table 2: Table 2 Data exporter Model information reading plugin, data parsing plugin Attribute Value Converter Plugin V1.0 Data importer Data import plugin Attribute Value Converter Plugin, Attribute Inspector Plugin, Relationship Inspector Plugin V1.0 For the same system role, there can be multiple required plugins. This can be understood as the system role's functionality requiring multiple plugins to work together.

[0060] Step S402: The combined configuration unit sends combined configuration information to the model mapping unit.

[0061] In one possible implementation, after receiving a system selection request, the combination configuration unit can directly determine the types of plugins that the source system and the target system need to be bound to according to the preset plugin binding rules, and then generate combination configuration information based on the plugin list stored in the plugin management unit.

[0062] For example, the combined configuration unit can combine the selected source system and target system to generate a unique combined configuration identifier, and store the plugin binding information in association with the combined configuration identifier. In other words, the combined configuration information here can include the combined configuration identifier (i.e., the index set) or the plugin binding information.

[0063] In one possible implementation, to ensure configuration integrity, the composite configuration unit needs to verify the completeness of plugin bindings after generating the composite configuration information. If it is determined that the plugin bindings contain the necessary plugin types, the composite configuration unit can send the aforementioned composite configuration information to the model mapping unit. If it is determined that the plugin binding information is missing, an error notification is directly returned to the user.

[0064] For example, after system initialization, the combined configuration unit can obtain the plugin interface standard library and establish a mapping relationship between plugin types and standard interfaces. For ease of understanding, please refer to Table 3, which is a plugin interface standard mapping table stored in a tool provided by an embodiment of the present invention. Specifically, as shown in Table 3: Table 3

[0065] For the same plugin type, it can correspond to multiple standard interface methods. This can be understood as: the implementation of this plugin type needs to support multiple interface methods to meet different business needs.

[0066] In step S403, the model mapping unit sends the model mapping configuration result to the data mapping unit. Correspondingly, the data mapping unit can receive the aforementioned model mapping configuration result. This configuration result is obtained by the model mapping unit through performing a three-level mapping configuration on the data models of the source system and the target system.

[0067] In one possible implementation, the model mapping unit can construct a three-level mapping structure of "model-attribute-value". When a mapping belongs to the model level at a given time, the element is set. Otherwise set By performing weighted aggregation operations on the matrix according to its hierarchical dimensions, the model mapping configuration result is generated.

[0068] In step S404, when the data mapping unit determines that the mapping configuration is complete, it generates a data mapping configuration scheme based on the model mapping configuration result. This data mapping configuration scheme may include an imported data mapping configuration scheme. For example, the data mapping configuration scheme is determined by the following formula: ,in For model hierarchical mapping, For attribute hierarchy mapping, Value hierarchy mapping.

[0069] The following sections will elaborate on different implementation scenarios for data integration and migration: Implementation method 1: For example, please see Figure 5 , Figure 5This is an interactive illustration of static data migration provided by an embodiment of the present invention. Figure 1 .like Figure 5 As shown, this method can be executed jointly by the system registration unit and related units in the tool. The method may include at least steps S501 to S507: Step S501: The combined configuration unit obtains the combined configuration parameters. For example, the combined configuration unit downloads the latest system registration information and plugin binding configuration from the data mapping repository.

[0070] In step S502, the data conversion unit receives a data file from the source system. This data file contains structured business data records.

[0071] In step S503, the data conversion unit sends a data conversion request to the data inspection unit.

[0072] In step S504, the data inspection unit sends standardized data to be imported to the data import unit. This data is obtained by the data conversion unit based on the model mapping configuration and the data mapping configuration.

[0073] In step S505, the data import unit performs a data import operation. Accordingly, the data import unit calls the target system's data import plugin to write the data into the target system.

[0074] In step S506, the data import unit sends import result statistics to the task scheduling and management unit. Correspondingly, the task scheduling and management unit receives the number of records successfully imported.

[0075] In step S507, the task scheduling and management unit generates a migration task execution report based on the statistical data of the import results.

[0076] In this embodiment of the invention, batch migration of heterogeneous system data is achieved by standardizing and transforming source system data according to model mapping configuration and attribute mapping configuration. The entire process replaces manual data transformation, outputs objective migration results, and improves the accuracy of data migration.

[0077] Implementation Method Two: For example, please see Figure 6 , Figure 6 This is a schematic diagram of an implementation interaction including incremental synchronization provided by an embodiment of the present invention. Figure 2 The method may include at least steps S601 to S609: Step S601: The task scheduling and management unit obtains the configuration parameters of the scheduled task.

[0078] Step S602: The data parsing plugin receives incremental data from the source system.

[0079] In step S603, the data parsing plugin sends incremental data to the data conversion unit.

[0080] In step S604, the data conversion unit performs data model label conversion.

[0081] In step S605, the data inspection unit receives the transformed incremental data and performs data inspection. For example, the data inspection unit performs integrity checks and validity verification on the incremental data using an attribute inspector plugin and a relationship inspector plugin.

[0082] In step S606, if the check passes, the data check unit generates a check pass signal.

[0083] In step S607, the data inspection unit sends the data that has passed the inspection to the data import unit.

[0084] Step S608: The data import unit calculates the incremental import success rate. For example, the incremental import success rate is determined by the following formula: ,in The number of successfully imported items. This is the total number to import.

[0085] In step S609, when the data import unit determines that the import success rate is lower than the threshold, it generates an abnormal alarm and triggers a retry mechanism.

[0086] In this embodiment of the invention, the system invokes preset data checking rules while performing data transformation. When the import success rate is lower than a threshold, a retry operation is performed, thus ensuring the reliability of data synchronization and enhancing the quality assurance of data integration.

[0087] Understandably, the data processing methods described above can be applied not only to static migration scenarios but also to dynamic synchronization scenarios, such as calculating the data synchronization success rate over different time periods. In other words, with a closed-loop data inspection system, the system can directly perform quality assessments without manual intervention.

[0088] For example, regardless of whether the data migration involves multiple table joins, the task scheduling and management unit can send execution instructions containing data mapping information to the data transformation unit. The data transformation unit generates detailed execution logs for the data transformation. When the data inspection unit determines that data anomalies exist, the data import unit calculates the percentage of abnormal data. For example, the percentage of abnormal data is determined by the following formula: ,in The number of abnormal data. This represents the total number of data points.

[0089] In the various embodiments of the present invention, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0090] It is understood that the various numerical designations used in the embodiments of the present invention are merely for descriptive convenience and are not intended to limit the scope of the embodiments of the present invention. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0091] The above are merely preferred embodiments of the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0092] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below in conjunction with specific application scenarios of heterogeneous system data integration and migration.

[0093] Step 1: During system registration and plugin binding, administrators log in to the data integration tool through the management interface, enter the system registration module, and input system registration information including the system identifier "ERP System A" and access address. After obtaining this registration information, the system registration unit stores the data in the data correspondence repository via data channel P1. The plugin management unit automatically binds the corresponding functional plugins based on the system role (data exporter or data importer), including model information reading plugins, data parsing plugins, data import plugins, attribute value converter plugins, attribute inspector plugins, and relationship inspector plugins, and accesses the data correspondence repository to obtain the currently active plugin binding configuration. Based on the interface version information in the plugin binding configuration, the plugin management unit verifies the integrity of the plugin implementation and generates a successfully bound system registration record.

[0094] Step 2: During the combined configuration and model mapping process, the combined configuration unit selects one source system and one target system from the registered systems, generating a unique combined configuration identifier. The model mapping unit dynamically retrieves the model structures of both the source and target systems through the model information reading plugin, displaying them in a tree structure. Configuration personnel configure the model correspondence, attribute correspondence, and attribute value correspondence between the source and target systems in the visual interface, using a three-level mapping structure of "model-attribute-value". The model mapping unit stores the mapping configuration information in a data correspondence repository and also supports exporting it as a mapping template for reuse.

[0095] Step 3: During data transformation and data inspection, the data transformation unit reads the original data file, completes data model label transformation according to the model mapping configuration, completes attribute label transformation according to the attribute mapping configuration, and calls the corresponding attribute value converter plugin to complete attribute value transformation according to the value mapping configuration. The data transformation unit automatically converts the reference relationships of the source system to the corresponding reference relationships of the target system through the import data converter plugin. The data inspection unit loads the transformed data, performs integrity checks, attribute validity checks, and relationship checks. It obtains a list of valid attribute values ​​through the attribute inspector plugin and verifies the validity of attribute values. It also verifies the existence of associated data in the target system through the relationship inspector plugin, generating data inspection results and details of abnormal data.

[0096] Step 4, during data import and task management, the data import unit calls the data import plugin bound to the target system to write the transformed and checked data into the target system. The import plugin returns an identifier for the newly created data in the target system. The data import unit records the correspondence between source and target data in a data correspondence repository for subsequent correction of related data references in migration tasks. The task scheduling and management unit encapsulates the data transformation, data checking, and data import processes into executable tasks. It supports scheduled task configuration for periodic automatic execution, batch creation and execution of multiple migration tasks, automatic retry mechanism after data import failure, and the function of exporting and correcting failed data for re-import.

[0097] All content not described in detail in this specification belongs to the prior art known to those skilled in the art. The application server, database server and other hardware devices involved in this technical solution all adopt standard industrial-grade servers, and their internal logical operations are executed by the processor through pre-stored software programs.

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data integration and migration system based on model mapping and plug-in extension, characterized in that, include: The system registration unit is used to register systems to be accessed, and to determine the data exporter or data importer based on the system role, and to enter the system identifier, access address, authentication information and model description file; The plugin management unit communicates with the system registration unit. The plugin management unit is used to bind functional plugins according to the registered system roles. Functional plugins include model reading plugins, data parsing plugins, data import plugins, attribute conversion plugins, attribute checking plugins, and relationship checking plugins. Each plugin is dynamically connected through a standard interface. The combined configuration unit communicates with the plug-in management unit. The combined configuration unit is used to select the source system and the target system, establish combined configuration relationships, and generate configuration identifiers. The model mapping unit communicates with the combined configuration unit. The model mapping unit is used to obtain the model structure of the source system and the target system, and to configure the mapping relationship between the model, attributes and attribute values. The data mapping unit communicates with the model mapping unit and is used to establish the correspondence between source system data instances and target system data instances. The data conversion unit is communicatively connected to the data mapping unit. The data conversion unit is used to perform model conversion, attribute conversion, and attribute value conversion according to the mapping relationship, and to correct the associated reference relationship. The data inspection unit communicates with the data transformation unit and is used to call the inspection plugin to perform integrity, attribute validity, and relationship checks. The data import unit communicates with the data inspection unit. The data import unit is used to call the target system import plugin to write the data that has passed the inspection and record the corresponding relationship. The task scheduling and management unit is connected to the data conversion unit, data inspection unit, and data import unit, respectively. The task scheduling and management unit is used to generate migration tasks and perform task management.

2. The data integration and migration system based on model mapping and plug-in extension as described in claim 1, characterized in that, The combined configuration unit is used to select the source system and the target system, establish a combined configuration relationship, and generate a configuration identifier, including: The combined configuration information includes a unique configuration identifier, a source system identifier, a target system identifier, source system bound plugin information, target system bound plugin information, and data import plugin extended parameters; The combined configuration unit is also used to store combined configuration information, and the combined configuration unit is also used to query the corresponding combined configuration information according to the unique configuration identifier, so as to call the corresponding system information and plug-in configuration during data processing; The data import plugin is provided with a configuration parameter acquisition interface. The combined configuration unit dynamically acquires the extended configuration items corresponding to the data import plugin by calling the configuration parameter acquisition interface, and loads the extended configuration items in the combined configuration interface to complete the parameter configuration. The combined configuration unit is used to store the extended parameters corresponding to different combined configurations respectively, so that the same system can be configured with different data import parameters under different migration combination relationships. The extended parameters include tenant identifier, organization identifier, and user identifier. The combined configuration unit associates and stores the extended parameters with the corresponding unique configuration identifiers, and passes the extended parameters to the corresponding data import plugin of the target system when executing the data import task.

3. The data integration and migration system based on model mapping and plug-in extension as described in claim 1, characterized in that, The model mapping unit adopts a three-level mapping structure of model, attribute, and attribute value; The model mapping unit is used to configure the one-to-one correspondence between the source system data model and the target system data model, and dynamically obtains the model structure of both parties and displays it in a tree structure by calling the model information reading plugins bound to the source system and the target system. The model mapping unit is also used to configure a one-to-one correspondence between the source system model attributes and the target system model attributes, and to configure a one-to-one correspondence between the source system attribute values ​​and the target system attribute values; The model mapping unit is also used to bind an attribute value converter plugin to the attribute mapping relationship. The attribute value converter plugin is used to convert the source system attribute value to the target system attribute value according to a preset conversion logic. The attribute value converter plugin includes a target system custom converter plugin, a data mapping converter plugin, and an import data converter plugin. The target system custom converter plugin is used to perform attribute value conversion according to the target system's business rules. The data mapping converter plugin is used to perform attribute value conversion according to a preset data mapping relationship. The import data converter plugin is used to perform attribute value conversion according to the association relationship during the data import process.

4. The data integration and migration system based on model mapping and plug-in extension as described in claim 3, characterized in that, It also includes a mapping template management unit, which is communicatively connected to the model mapping unit, wherein: The mapping template management unit is used to obtain the model mapping relationship, attribute mapping relationship and attribute value mapping relationship configured by the model mapping unit, and encapsulate the mapping relationship to generate a mapping template; The mapping template management unit is also used to store, export, and manage the version of the mapping template. When establishing a combined configuration relationship between systems of the same type, it calls the corresponding mapping template to import into the model mapping unit to automatically generate the corresponding model mapping configuration, attribute mapping configuration, and attribute value mapping configuration.

5. The data integration and migration system based on model mapping and plug-in extension as described in claim 1, characterized in that, The data mapping unit includes a data mapping list, which includes imported data mappings and input data mappings. The imported data mapping is also used to dynamically obtain the list of data types to be mapped in the target system through the data import plugin bound to the target system, generate the corresponding data mapping configuration interface according to the list of data types, and associate and store the generated data mapping result with the combined configuration identifier and the migration task identifier. The input data mapping is also used to receive the data instance correspondence input by the user through the combined configuration interface, and to generate the mapping relationship between the source system data instance and the target system data instance.

6. The data integration and migration system based on model mapping and plug-in extension as described in claim 5, characterized in that, The data mapping unit also includes a data correspondence repository, which is used to persistently store the correspondence between source system data instances and target system data instances, as well as model mapping configuration and data mapping configuration information. The data correspondence repository is communicatively connected to the data conversion unit. The data conversion unit is used to read the mapping information in the data correspondence repository during the data conversion process and complete the conversion of associated data reference relationships based on the mapping information.

7. The data integration and migration system based on model mapping and plug-in extension as described in claim 6, characterized in that, When the data conversion unit performs data conversion, it includes: The data conversion unit is also used to perform data conversion processing, which includes reading source system data files, converting data model tags, converting data attribute tags, and converting attribute values. When performing attribute value conversion, the data conversion unit calls the attribute value converter plugin to execute the corresponding attribute value conversion logic according to different conversion scenarios. Specifically, when the attribute value in the source system data is a reference to another data instance, and the referenced data instance has already established a corresponding relationship in the data mapping unit, the data conversion unit calls the import data converter plugin to convert the reference value in the source system into the identifier of the corresponding data instance in the target system. When attribute values ​​in the source system data are transformed according to the data mapping relationship configured by the user, the data transformation unit calls the data mapping converter plugin to convert the source attribute values ​​into target attribute values ​​according to the input data mapping relationship; When attribute values ​​in the source system data need to be transformed according to the business rules of the target system, the data transformation unit calls the target system's custom converter plugin and transforms the source attribute values ​​into target attribute values ​​according to the target system's preset transformation logic.

8. The data integration and migration system based on model mapping and plug-in extension as described in claim 7, characterized in that, When the data inspection unit performs data inspection, it includes: The data inspection unit is also used to perform inspection processing on the transformed data before data import. The inspection processing includes model and attribute mapping integrity check, attribute validity check, and association relationship check. The data inspection unit is also used to call the data conversion unit to complete the data conversion after confirming that the model mapping relationship and attribute mapping relationship are complete, and to call the data import plugin corresponding to the target system to return the attribute inspector plugin and the relationship inspector plugin. The attribute inspector plugin is also used to obtain the corresponding attribute validation rules according to the data type. The validation rules include the valid value range of the attribute, data type, length limit and regular expression rules. The data inspection unit performs legality verification on the converted attribute value according to the validation rules. The relationship checker plugin is also used to obtain the related data verification rules according to the data type, and to detect whether the data instance corresponding to the specified data identifier exists in the target system, so as to determine the validity of the data relationship; The data inspection unit is also used to record data inspection results and details of abnormal data, generate an inspection report including error type, error location and error cause, and terminate the data import process and output abnormal information when the inspection results do not meet the preset import conditions.

9. The data integration and migration system based on model mapping and plug-in extension as described in claim 8, characterized in that, The data import unit is used to group the data that has undergone data inspection according to the data type configured in the migration task, including: The data import unit is also used to call the corresponding data import plugin of the target system to obtain the data import interface for different types of data. The data import unit is also used to write the grouped migration data into the target system through the data import interface, and after the data import is completed, obtain the data identifier generated by the target system and establish a data correspondence between the source system data instance and the target system data instance. The data import unit is used to store the data correspondence to a data correspondence repository, wherein the data correspondence includes a mapping relationship between source data identifiers and target data identifiers; The data conversion unit and the data mapping unit are respectively connected to the data correspondence repository to read the data correspondence and update the data mapping configuration in subsequent migration tasks according to the data correspondence, so as to automatically complete the correction of the associated data reference relationship.

10. A data integration and migration method based on model mapping and plug-in extension, applicable to the data integration and migration system based on model mapping and plug-in extension as described in any one of claims 1-9, characterized in that, include: Obtain the registration information of the system to be accessed, determine the source system and target system according to the system role, and record the system identifier, access address, authentication information and model description file; Based on the system role, corresponding functional plugins are bound to the source system and the target system. The functional plugins include model reading plugin, data parsing plugin, data import plugin, attribute conversion plugin, attribute checking plugin, and relationship checking plugin. Each functional plugin is dynamically connected through a standard interface. Establish the combined configuration relationship between the source system and the target system, and generate configuration identifiers; The model reading plugin is invoked to obtain the data model structure of the source system and the target system, and to establish model mapping relationships, attribute mapping relationships, and attribute value mapping relationships. Obtain the correspondence between source system data instances and target system data instances, and establish a data instance mapping relationship; Based on the mapping relationship, perform model transformation, attribute transformation, and attribute value transformation on the source system data, and correct the related data reference relationships; The inspection plugin is invoked to perform integrity, attribute validity, and relationship checks on the transformed data; After the check is passed, the target system import plugin is invoked to complete the data writing and the correspondence between the source data and the target data is recorded; The data transformation, data inspection, and data import processes are encapsulated as migration tasks, and task management is performed.

Images