A method for cross-data space data synchronization based on linker construction

Abstract

This application discloses a cross-data space data synchronization method based on connector construction, belonging to the field of data integration. The method includes: determining a source connector and a target connector based on a user-submitted task definition file; the source connector is the connector corresponding to the source data space, and the target connector is the connector corresponding to the target data space; each connector includes a connection configuration unit, a data pattern descriptor, semantic mapping rules, a data reading interface, and a data writing interface; determining a data synchronization task graph based on the semantic mapping rules of the source connector and the target connector; reading the original data from the source data space through the source connector; mapping the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics; mapping the general data of the general data pattern and general semantics to the target data of the target data pattern and target semantics through the target connector; and writing the target data into the target data space.

Description

Technical Field

[0001] This application belongs to the field of data integration, specifically relating to a cross-data space data synchronization method based on connector construction. Background Technology

[0002] In today's enterprise data architecture, data is typically not centrally stored in a single system, but rather distributed across various systems that constitute multiple "data spaces." These data spaces may include traditional relational databases (such as MySQL and Oracle), massively parallel processing data warehouses (such as Apache Hive and Snowflake), data lakes (such as Apache Hudi, Delta Lake, and Iceberg), NoSQL databases (such as MongoDB and Cassandra), search engines (such as Elasticsearch), and cloud storage services (such as AWS S3). Each of these data spaces is a logically independent data environment, possessing its own unique data model, storage format, access protocol, metadata management system, and, most importantly, business semantic context.

[0003] Synchronizing data from one data space to another is a fundamental requirement for building data warehouses, data lakes, real-time data platforms, and enabling cross-business line data analysis.

[0004] Currently, there are various data synchronization tools and solutions available in the industry: 1. Traditional ETL tools and custom scripts: such as using tools like DataX, Sqoop, and Kettle, or directly writing Spark / Flink jobs. A common problem with these approaches is: Cumbersome configuration: Users need to write a lot of configurations or code in the source, target and transformation stages, resulting in scattered focus and high learning costs.

[0005] Semantic Gap: Tools typically only handle basic data type mappings (such as INT to BIGINT), but cannot effectively address semantic differences at the business level. For example, the `customer_id` field in the source system and the `user_code` field in the target system have the same meaning but different names; the monetary unit in the source system is "cents," while the target system requires "yuan"; status codes 1, 2, and 3 in the source system need to be mapped to 'ACTIVE', 'INACTIVE', and 'PENDING' in the target system. This logic is usually scattered in custom scripts, making it difficult to maintain and standardize.

[0006] Poor reusability: The connection, authentication, and basic field mapping logic for the same data source cannot be easily reused in different teams and projects, leading to redundant development and "siloed" integration.

[0007] Governance difficulties: Data lineage and transformation logic are hidden in scripts and scattered configurations, making it difficult to conduct global metadata management and impact analysis.

[0008] 2. Modern data integration platforms: such as Apache SeaTunnel and Apache NiFi. These platforms offer a rich set of connectors and visual data flow design interfaces, significantly improving ease of use and performance. However, there is still room for improvement. Connectors are coupled with transformation logic: In SeaTunnel's configuration, although the source, transformation, and sink are separate, the transformation logic is usually defined at the task level and is not strongly bound to specific source / target connectors. This means that standardized mapping rules for a specific scenario like "synchronizing user data from MySQL to Elasticsearch" cannot be encapsulated into a reusable, monolithic component.

[0009] Lack of semantic layer abstraction: Existing connectors mainly focus on reading and writing physical data, lacking an explicit, configurable "semantic mapping" layer to specifically handle semantic inconsistencies across systems.

[0010] Loose component management: Connectors are usually released with engine versions, and the development, distribution, version management and sharing mechanisms for custom connectors are not perfect, which is not conducive to the formation of a unified integration standard within the enterprise. Summary of the Invention

[0011] The purpose of this application is to provide a cross-data space data synchronization method based on connector construction, which can solve the above-mentioned problems.

[0012] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a cross-dataspace data synchronization method based on connector construction, the method comprising: Based on the task definition file submitted by the user, source connectors and target connectors are determined. The source connector is the connector corresponding to the source data space, and the target connector is the connector corresponding to the target data space. The connector includes a connection configuration unit, a data pattern descriptor, semantic mapping rules and data reading interface, and data writing interface. Based on the semantic mapping rules of the source connector and the semantic mapping rules of the target connector, a data synchronization operation graph is determined; According to the data synchronization operation diagram, the original data in the source data space is read through the source connector, and the original data pattern and original semantics corresponding to the original data are mapped to a general data pattern and general semantics. The general data of the general data pattern and general semantics are mapped to target data of the target data pattern and target semantics through the target connector, and the target data is written into the target data space.

[0013] Optionally, if the task definition file further includes additional transformation logic, determining the data synchronization job graph based on the semantic mapping rules of the source connector and the semantic mapping rules of the target connector includes: The additional transformation logic is integrated with the semantic mapping rules of the source connector and the semantic mapping rules of the target connector to obtain the data synchronization operation graph; The step of reading raw data from the source data space through the source connector, mapping the raw data pattern and raw semantics corresponding to the raw data to a general data pattern and general semantics, and mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics through the target connector, and writing the target data into the target data space, includes: The source connector reads the original data in the source data space and maps the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics. Based on the additional transformation logic, the general data of the general data pattern and general semantics are processed to obtain intermediate data of the general data pattern and general semantics; The intermediate data of the general data pattern and general semantics are mapped to the target data of the target data pattern and target semantics through the target connector, and the target data is written into the target data space.

[0014] Optionally, determining the source connector and target connector based on the task definition file submitted by the user includes: Parse the task definition file to determine the source linker identifier and the target linker identifier; Based on the source connector identifier and the target connector identifier, a request is sent to the connector repository to obtain the implementation package and dependencies of the source connector and the target connector. The implementation packages and dependencies of the source connector and the target connector are dynamically loaded and instantiated to determine the source connector and the target connector.

[0015] Optionally, the step of reading the original data in the source data space through the source connector and mapping the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics includes: A connection is established with the source data space through the connection parameters of the source data space recorded in the connection configuration unit of the source connector; The original data in the source data space is read through the data reading interface of the source connector; The original data pattern and original semantics of the original data are identified by the data pattern descriptor of the source connector; The original data pattern and original semantics are mapped to a general data pattern and general semantics through the semantic mapping rules of the source connector.

[0016] Optionally, the step of mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics and writing it into the target data space through the target connector includes: A connection is established with the target data space through the connection parameters of the target data space recorded in the connection configuration unit of the target connector; The semantic mapping rules of the target connector are used to map the general data pattern and general semantics to the target data pattern and target semantics. The target data is written into the target data space through the data writing interface of the target connector.

[0017] Optionally, after writing the target data into the target data space, the method further includes: The data synchronization process from the source data space to the target data space includes the mapping processes from the original data to the general data, from the general data to the intermediate data, and from the intermediate data to the target data. These processes are then integrated to generate a data lineage map, which is stored in the connection sub-warehouse. In the event of abnormalities in the data quality, data security, or data compliance of the target data in the target data space, the mapping process of the target data is traced through the data lineage map to determine the cause of the abnormality.

[0018] Optionally, the method further includes: Upload each developed connector to the connector repository; After reviewing the code quality, security, standardization, and accuracy of semantic mapping of the connector, the connector is released and a connector identifier is set. Store the metadata of the connector and set its access permissions.

[0019] Optionally, the method further includes: If the connection parameters of the data space change or the semantic mapping rules change, the version of the corresponding connector in the connection sub-repository of the data space is updated so that the task definition text referencing the connector can take effect without modification.

[0020] Optionally, the process of creating the connector includes: Configure the connection configuration unit, wherein the connection parameters of the connection configuration unit include at least the host address, port, database name, username, password, table name, and Topic name of the data space; Configure a data schema descriptor, which is used to define the structure, type and constraint relationships of data in the data space in a self-describing manner; Configure semantic mapping rules, which are pluggable components containing a rule engine, used to map and convert the data patterns and semantics of predefined or runtime data parsing data with common data patterns and semantics in terms of field naming, data types, data formats, business codes, and units of measurement; the semantic mapping rules include at least: field renaming rules, data type conversion rules, value conversion rules, unit of measurement conversion rules, and structure transformation rules; Configure a data reading interface, which is used to read data in parallel from the corresponding data space; Configure a data write interface, which is used to write data in parallel to the corresponding data space.

[0021] Secondly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps in the method described in the first aspect.

[0022] This application provides a cross-data space data synchronization method based on connector construction. The method includes: determining a source connector and a target connector based on a user-submitted task definition file; the source connector being a connector corresponding to a source data space, and the target connector being a connector corresponding to a target data space; each connector including a connection configuration unit, a data pattern descriptor, semantic mapping rules, a data reading interface, and a data writing interface; determining a data synchronization task graph based on the semantic mapping rules of the source connector and the target connector; reading raw data from the source data space through the source connector according to the data synchronization task graph, mapping the raw data pattern and raw semantics corresponding to the raw data to a general data pattern and general semantics, and mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics through the target connector, and writing the target data into the target data space.

[0023] The technical solution of this application proposes the concept of a connector, which encapsulates the scattered connection configuration, schema definition, semantic mapping and read / write logic in traditional data integration into a standardized software component. This realizes the transformation from original manual coding to declarative configuration task definition files, thereby solving the technical bottlenecks faced by existing tools when synchronizing across heterogeneous data spaces, such as configuration complexity, semantic gap, poor reusability and governance difficulties. Specifically, by using a modular design of connection configuration units, data pattern descriptors, semantic mapping rules, and read / write interfaces, the connector hides the complexity of accessing specific data spaces behind standardized interfaces. Users only need to reference the corresponding connector identifier in the task definition file to reuse all the technical capabilities within the constructed connector, eliminating the need to repeatedly write connection code and conversion logic. This significantly reduces the development threshold and configuration complexity. Furthermore, the semantic mapping rules within the connector explicitly define a large number of semantic mapping rules and ensure semantic consistency of data across the source data space → general format → target data space through a standardized process. This eliminates the semantic gap problem caused by the mixing of business semantics and technical implementation in traditional scripts. Simultaneously, managing the connector as an independent component enables high logic reuse and global synchronization of changes, avoiding governance challenges such as scattered scripts, redundant development, and difficult changes in traditional methods. Finally, by dynamically assembling connectors through parsing declarative tasks to generate job graphs, the distributed engine can schedule and execute them in parallel, decoupling physical data transmission from logical semantic conversion across data spaces. This comprehensively improves the development efficiency, operational standardization, semantic consistency, and system scalability of cross-data-space synchronization tasks. Attached Figure Description

[0024] Figure 1This is a flowchart illustrating a cross-data space data synchronization method based on connector construction provided in an embodiment of this application; Figure 2 This is an overall architecture diagram of a cross-data space data synchronization method based on connector construction provided in an embodiment of this application; Figure 3 This is an internal structural diagram of a connector provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the core technology of cross-data space data synchronization provided in an embodiment of this application; Figure 5 This is a schematic diagram of the framework of a cross-data space data synchronization device based on a connector structure provided in an embodiment of this application; Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The primary objective of this application is to overcome the shortcomings of existing data synchronization technologies and provide a cross-dataspace data synchronization method based on connector construction, which can significantly simplify the development and maintenance of synchronization tasks across heterogeneous data spaces. Secondly, it aims to fundamentally solve the semantic mismatch problem in the data synchronization process by introducing an explicit semantic mapping layer. Furthermore, it hopes to promote the reuse and standardization of data synchronization logic and improve team collaboration efficiency through component-based design and centralized repository management. Finally, it aims to provide a solid technical foundation for enterprise data governance and achieve clear and traceable data lineage.

[0028] Inspired by the concept of "connector" in differential geometry—a concept used to describe the connection between different tangent spaces on a manifold, thus allowing vector translation and comparison in curved spaces—this application proposes to introduce this idea of ​​connector into the field of data integration.

[0029] Specifically, we view each data space as a "data manifold," whose internal data structure and usage semantics constitute a local coordinate system. Synchronization across data spaces requires the establishment of communication rules between the two local coordinate systems through precisely defined "connectors," ensuring that the core semantics of the data are preserved during the synchronization process.

[0030] The following description, in conjunction with the accompanying drawings, details a cross-data space data synchronization method based on connector construction provided by the embodiments of this application through specific implementations and application scenarios.

[0031] Figure 1 This is a flowchart illustrating a cross-data space data synchronization method based on connector construction according to an embodiment of this application. Figure 2 This is an overall architecture diagram of a cross-data space data synchronization method based on connector construction provided in an embodiment of this application.

[0032] refer to Figure 1 This application provides a cross-data space data synchronization method based on connector construction, the method comprising steps S11 to S13: Step S11: Based on the task definition file submitted by the user, determine the source connector and the target connector. The source connector is the connector corresponding to the source data space, and the target connector is the connector corresponding to the target data space. The connector includes a connection configuration unit, a data pattern descriptor, semantic mapping rules and data reading interface, and data writing interface.

[0033] In this embodiment, at the initial stage of the cross-data space data synchronization process, a task definition file written in declarative syntax by the user is first received. This task definition file is a configuration file describing the user's data synchronization requirements, typically in YAML or JSON format. It explicitly declares the source and target information for data synchronization. The task definition file is then parsed by task parser and compiler. For example, by identifying the connector identifiers in the task definition file, the specific implementations of the two connectors are located from the connector repository, thereby determining the source and target connector components specified in the task definition file. For instance, the task definition file concisely describes "from where (source connector) to where (target connector)" and "what transformations are performed."

[0034] Example task definition file: Synchronize order data from MySQL to Elasticsearch and perform real-time metric calculations: job: name: "MySQL_to_ES_Order_Sync" source: connector: "connector-repo.company.com / source / mysql-connector:2.1.0" config: host: "prod-mysql.db.com" port: 3306 database: "ecommerce" table: "orders" username: "${ENCRYPTED_USERNAME}" password: "${ENCRYPTED_PASSWORD}" # Referencing predefined semantic mappings within the connector: mapping database field names to more generic names semantic_mapping_profile: "order_standard_v1" transform: # Users can add additional transformations at the task level to work in conjunction with the mappings within the connective. - sql: "SELECT *, CAST(update_time AS TIMESTAMP) as event_time FROMsource_table WHERE amount>0" - filter: "status = 'COMPLETED'" sink: connector: "connector-repo.company.com / sink / elasticsearch-connector:1.5.0" config: hosts: ["es-cluster.company.com:9200"] index: "order_index_v2" The Elasticsearch join also internally defines how to map common field names to their index mappings. semantic_mapping_profile: "es_order_index_v2" The connection sub-repository is a centralized management service used to store, version, and distribute connection sub-components. The connection sub-repository provides the following functionalities: Storage and versioning: Managing different versions of the same data space's connectors.

[0035] Metadata index: Stores metadata information for each connector, such as name, type (source / target), description, compatible dataspace version, author, input / output mode examples, etc.

[0036] Dependency management: Manages the third-party libraries that each connector depends on.

[0037] Review and Distribution: Provides the upload, review, and publication process for connectors, and provides downloads for task parsing and compilers via protocols such as HTTP / HTTPS.

[0038] Access control: Controls the access and modification permissions of different users or teams to the connector.

[0039] Among them, the source connector is a connector specifically corresponding to the source data space, which refers to the original storage system from which the data flows out, such as MySQL database, Oracle database, Apache Hive data warehouse, AWS S3 object storage, etc. The target connector is a connector specifically corresponding to the target data space, which refers to the target storage system from which the data flows in, such as Elasticsearch search engine, MongoDB document database, Snowflake cloud data warehouse, etc. The source connector encapsulates the access capability to the source data space, and the target connector encapsulates the access capability to the target data space.

[0040] In other words, in this application, a connector is pre-created for each data space to enable access to that data space during cross-data space data synchronization. Specifically, a connector is a software component that conforms to a specific interface specification and fully represents the ability to access a certain type of data space. When this data space is used as the source data space, the connector corresponding to that data space is used as the source connector. When this data space is used as the target data space, the connector corresponding to that data space is used as the target connector.

[0041] Figure 3 This is an internal structural diagram of a connector provided in an embodiment of this application, with reference to... Figure 3 connector ( Figure 3 The SourceConnector is the source connector, and the SinkConnector is the target connector. This includes a connection configuration unit and a data mode descriptor (corresponding to...). Figure 3 Data Schema), semantic mapping rules (corresponding) Figure 3 SystemMapper) and data reading and writing interfaces.

[0042] The process of creating the connector for each data space is shown in steps S001 to S005: Step S001: Configure the connection configuration unit. The connection parameters of the connection configuration unit include at least the host address, port, database name, username, password, table name, and Topic name of the data space.

[0043] In this embodiment, the connection parameters stored in the connection configuration unit are used to establish a physical connection to the data space corresponding to the connector.

[0044] Step S002: Configure the data pattern descriptor, which is used to define the structure, type and constraint relationship of the data in the data space in a self-describing manner.

[0045] In this embodiment, the data schema descriptor is used to identify and describe the structural information of the data in the data space corresponding to the connector, such as static table structures (field names, types, primary keys), descriptions of semi-structured data (such as JSONSchema), and streaming data schemas (such as Apache Avro Schema). In some implementations, it also has the ability to automatically infer the data schema from the data source.

[0046] Step S003: Configure semantic mapping rules. The semantic mapping rules are pluggable components containing a rule engine, used to predefine or parse data patterns and semantics at runtime, and to map and convert between data patterns and semantics and general data patterns and semantics in terms of field naming, data types, data formats, business codes, and units of measurement. The semantic mapping rules include at least: field renaming rules, data type conversion rules, value conversion rules, unit of measurement conversion rules, and structure transformation rules.

[0047] In this embodiment, the semantic mapping rule is encapsulated in a pluggable rule engine or configuration unit. The semantic mapping rule is used to define the mapping rules between the specific data pattern and specific semantics of the data in the data space corresponding to the connector and the internally predefined general standards (i.e., general data pattern and general semantics). The semantic mapping rule may include: Field renaming rules (e.g., src_field -> tgt_field).

[0048] Data type conversion rules (e.g., STRING to TIMESTAMP, specifying the format).

[0049] Value transformation rules (such as using lookup tables, regular expressions, Groovy / Python scripts for value mapping: status: 1 -> 'ACTIVE').

[0050] Unit conversion (e.g., amount / 100 converts minutes to yuan).

[0051] Structural transformations (such as row to column, column to row, flattening of nested structures, etc.).

[0052] Step S004: Configure the data reading interface, which is used to read data in parallel from the corresponding data space.

[0053] In this embodiment, the data reading interface is responsible for reading data from the source data space in a distributed parallel manner and generating the SeaTunnelRow data structure inside the distributed plan execution engine.

[0054] Step S005: Configure the data writing interface, which is used to write data in parallel to the corresponding data space.

[0055] In this embodiment, the data writing interface is responsible for writing the data processed by the distributed plan execution engine into the target data space in a distributed parallel manner.

[0056] Step S12: Determine the data synchronization operation graph based on the semantic mapping rules of the source connector and the semantic mapping rules of the target connector.

[0057] In this embodiment, the semantic mapping rules in the source connector and the semantic mapping rules in the target connector determined in step S11 are integrated, and an optimized data synchronization job graph that can be directly executed by the distributed plan execution engine is generated through the execution plan builder.

[0058] Step S13: According to the data synchronization operation diagram, read the original data in the source data space through the source connector, and map the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics. Then, map the general data of the general data pattern and general semantics to the target data pattern and target semantics through the target connector, and write the target data into the target data space.

[0059] In this embodiment, reference Figure 2The distributed execution engine (including the execution engine and the computing engine) receives the data synchronization job graph generated by the execution plan builder, and then transforms the data synchronization job graph into distributed tasks on the underlying computing framework (such as Apache Flink, Apache Spark, and SeaTunnel's self-developed engine), schedules and executes them, and completes high-throughput, low-latency data synchronization.

[0060] Specifically, Figure 4 This is a schematic diagram illustrating the core technology of cross-data space data synchronization provided in an embodiment of this application. The distributed execution engine schedules and executes data according to the data synchronization job graph to complete data synchronization from the source data space to the target data space. In this stage, firstly, a physical connection with the source data space is established using the connection configuration unit through the data reading interface of the source connector. Then, the structural information of the data in the source data space is identified according to the data pattern descriptor, and the original data is read from the source data space in a distributed parallel manner. Subsequently, referencing... Figure 4 The source connector applies semantic mapping rules to map the original data schema and semantics to a general data schema and semantics, converting the data into a standard format for unified processing within the system. Next, the target connector applies its semantic mapping rules to map the general data schema and semantics to the target data schema and semantics, ensuring it conforms to the specific structure and business requirements of the target data space. Finally, a physical connection is established with the target data space through the target connector's data write interface, and the converted target data is written to the target data space in a distributed parallel manner, thus completing the entire cross-data space data synchronization process.

[0061] The technical solutions described above, by proposing the concept of connectors, encapsulate the scattered connection configurations, schema definitions, semantic mappings, and read / write logic in traditional data integration into standardized software components. This achieves a transformation from original manual coding to declarative configuration task definition files, thereby solving the technical bottlenecks faced by existing tools when synchronizing across heterogeneous data spaces, such as complex configuration, semantic gaps, poor reusability, and governance difficulties. Specifically, by using a modular design of connection configuration units, data pattern descriptors, semantic mapping rules, and read / write interfaces, the connector hides the complexity of accessing specific data spaces behind standardized interfaces. Users only need to reference the corresponding connector identifier in the task definition file to reuse all the technical capabilities within the constructed connector, eliminating the need to repeatedly write connection code and conversion logic. This significantly reduces the development threshold and configuration complexity. Furthermore, the semantic mapping rules within the connector explicitly define a large number of semantic mapping rules and ensure semantic consistency of data across the source data space → general format → target data space through a standardized process. This eliminates the semantic gap problem caused by the mixing of business semantics and technical implementation in traditional scripts. Simultaneously, managing the connector as an independent component enables high logic reuse and global synchronization of changes, avoiding governance challenges such as scattered scripts, redundant development, and difficult changes in traditional methods. Finally, by dynamically assembling connectors through parsing declarative tasks to generate job graphs, the distributed engine can schedule and execute them in parallel, decoupling physical data transmission from logical semantic conversion across data spaces. This comprehensively improves the development efficiency, operational standardization, semantic consistency, and system scalability of cross-data-space synchronization tasks.

[0062] In conjunction with the above embodiments, an embodiment of this application also provides another cross-dataspace data synchronization method based on connector construction. In this method, when the task definition file further includes additional transformation logic, step S12, "determining the data synchronization job graph according to the semantic mapping rules of the source connector and the semantic mapping rules of the target connector," specifically includes: Step S12-1: Integrate the additional transformation logic with the semantic mapping rules of the source connector and the target connector to obtain the data synchronization operation graph.

[0063] In this embodiment, in actual business scenarios of cross-dataspace data synchronization, relying solely on the standardized semantic mapping rules encapsulated within the connector often fails to meet the complex and ever-changing business needs of users. The semantic mapping rules of the connector mainly address fundamental semantic alignment issues such as field naming, data types, measurement units, and encoded values ​​between different data spaces. For example, mapping MySQL's cust_id to the generic name customer_id, dividing amount_cents by 100 to convert it to amount_yuan, and mapping status code 2 to 'COMPLETED'. These rules are universal and stable, and therefore suitable for encapsulation in the semantic mapping rules of reusable connectors.

[0064] However, in actual business operations, data synchronization tasks often require personalized data processing based on specific scenarios. For example, this might involve filtering specific records based on business conditions (synchronizing only orders with amounts greater than 100 yuan), performing field calculations and derivations (calculating discount rates based on amount and date), data cleaning and filtering (removing test data or outliers), and multi-table join queries (joining customer tables to obtain customer level information). These personalized transformation logics are highly business-specific and time-sensitive, making it difficult and unsuitable to encapsulate them all within the connector. Otherwise, it would lead to bloated connectors, version explosions, and an inability to flexibly respond to business changes. Therefore, this embodiment introduces additional transformation logic in the task definition file, allowing the user client to flexibly express personalized business processing needs—i.e., additional transformation logic—using declarative syntax (such as SQL, Filters, etc.) on top of referencing standardized connectors. This allows the connector to focus on solving the standardization problem of cross-space semantic alignment, while the user client focuses on solving the personalized problems specific to the business scenario. The two are integrated to form complete data processing requirements, thus achieving a balance between standardization and flexibility, significantly improving the practicality and applicability of the method.

[0065] Therefore, after the task parser and compiler parse the task definition file, they extract the user-defined additional transformation logic. The additional transformation logic refers to the data processing operations customized by the user according to specific business needs. It is usually expressed in a declarative way, such as SQL query statements for field calculation, type conversion, and data association, or filter conditions for data filtering, cleaning, and deduplication. The additional transformation logic represents the personalized processing needs under specific business scenarios.

[0066] Specifically, during the integration of data processing logic, the conversion logic from the original data pattern to the general data pattern defined in the semantic mapping rules of the source connector is first extracted, including field renaming, conversion from the original type to the general type, and mapping from the original semantics to the general semantics. Then, the additional conversion logic declared by the user in the task definition file is parsed to identify the field names, operation types, and conditional expressions referenced therein. Next, the conversion logic from the general data pattern to the target data pattern defined in the semantic mapping rules of the target connector is extracted. Finally, the semantic mapping rules of the source connector, the additional conversion logic, and the semantic mapping rules of the target connector are integrated according to the natural order of data flow. Then, the additional transformation logic is integrated with the semantic mapping rules of the source connector and the target connector to ensure that the execution order of the distributed plan execution engine is as follows: first, the source connector semantic mapping rules are applied to transform the raw data into a general format; then, the user-declared additional transformation logic is applied to perform business processing on the general data; finally, the target connector semantic mapping rules are applied to transform the processed general data into the target format. At the same time, the system will perform optimization processing, such as predicate pushdown to advance the filtering conditions to the data source end to reduce data transmission, or merging adjacent transformation operations to improve execution efficiency. Finally, a complete data synchronization job graph is generated, which includes Source nodes (corresponding to the semantic mapping rules of the source connector), Transform nodes (corresponding to the user-declared additional transformation logic), Sink nodes (corresponding to the semantic mapping rules of the target connector), and data flow edges.

[0067] The step S13, "reading the original data in the source data space through the source connector, mapping the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics, and mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics through the target connector, and writing the target data into the target data space," includes steps S13-1 to S13-3: Step S13-1: Read the original data in the source data space through the source connector, and map the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics.

[0068] In this embodiment, the first stage of the data synchronization mapping process is to complete the data reading and preliminary conversion from the source data space through the source connector. In this process, the data reading interface of the source connector uses the connection parameters such as host address, port, and authentication information stored in its connection configuration unit to establish a physical connection with the source data space. Then, according to the data structure of the original data identified by the data pattern descriptor, the original data is read in a distributed parallel manner. For each piece of original data read, the semantic mapping rules of the source connector are applied to map the original data pattern corresponding to the original data to a general data pattern, including operations such as field renaming, data type conversion, and structure flattening. At the same time, the original semantics are converted into general semantics, including operations such as unit conversion, value mapping, and format standardization, so that the data is converted into a general format that is uniformly processed within the system and encapsulated as general data.

[0069] Step S13-2: According to the additional conversion logic, process the general data of the general data pattern and the general semantics to obtain the intermediate data of the general data pattern and the general semantics.

[0070] In this embodiment, in the second stage of the data synchronization mapping process, based on the user-defined additional transformation logic in the job graph integrated in step S12-1, in-depth data processing at the business level is performed on the general data that has completed the source join sub-mapping. Specifically, according to the SQL statements, filter conditions, calculation rules, etc. declared by the user in the task definition file, corresponding data processing operations are performed on the general data with general data patterns and general semantics. For example, SQL is used for field filtering, aggregation calculation, data association, and type conversion; filters are used for condition filtering, data cleaning, and record deduplication; or custom scripts are used for complex business logic processing. After processing, intermediate data with general data patterns and general semantics is obtained.

[0071] Step S13-3: Map the intermediate data of the general data pattern and general semantics to the target data of the target data pattern and target semantics through the target connector, and write the target data into the target data space.

[0072] In this embodiment, the third stage of the data synchronization mapping process involves data conversion and writing to the target data space via the target connector. Specifically, the semantic mapping rules of the target connector are applied to the intermediate data obtained in step S13-2 to convert the general data pattern into the target data pattern. This includes renaming fields to target system-specific names, converting types to target system-supported formats, and adjusting the structure to the structure required by the target system. Simultaneously, the general semantics are converted into target semantics, including converting units to target system units, mapping values ​​to target system codes, and adjusting the format to the target system standard. Subsequently, the data writing interface of the target connector establishes a physical connection with the target data space using parameters in its connection configuration unit, and writes the converted target data to the target data space in batches in a distributed parallel manner, thereby completing the entire cross-data space data synchronization process.

[0073] It should be noted that if the task definition file does not include additional transformation logic, then only the first and third stages described above need to be executed.

[0074] The technical solutions described above deeply integrate user-defined additional conversion logic with standardized semantic mapping rules for connectors, achieving an organic combination of personalized business needs and standardized technical capabilities, thereby further enhancing the flexibility and applicability of data synchronization across data spaces.

[0075] In conjunction with the above embodiments, an embodiment of this application also provides another cross-data space data synchronization method based on connector construction. In this method, step S11, "determining the source connector and the target connector according to the task definition file submitted by the user," includes steps S11-1 to S11-3: Step S11-1: Parse the task definition file to determine the source connector identifier and the target connector identifier.

[0076] In this embodiment, after receiving the task definition file submitted by the user, which is written in declarative syntax, it is structured and parsed to identify and determine the source connector identifier and the target connector identifier. The source and target information for data synchronization are explicitly declared. The source connector identifier is a string that uniquely identifies the connector corresponding to the source data space, containing information such as the repository address, connector name, and version number, used to accurately locate and obtain the connector for processing a specific source data space. The target connector identifier is a string that uniquely identifies the connector corresponding to the target data space, used to accurately locate and obtain the connector for processing a specific target data space. By parsing the task definition file and extracting these two connector identifiers, the user-declared source and target connectors are determined.

[0077] Step S11-2: Based on the source connector identifier and the target connector identifier, send a request to the connector repository to obtain the implementation package and dependencies of the source connector and the target connector.

[0078] In this embodiment, based on the source connector identifier and target connector identifier determined in step S11-1, a request is sent to the connector repository to download the required connector implementation package. The implementation package is an executable file containing the specific implementation code of the connector, typically in JAR or Python Wheel format. It contains code for five core components: a connector configuration unit, a data schema descriptor, semantic mapping rules, a data read interface, and a data write interface. Dependencies refer to third-party libraries or frameworks required for the connector to run, such as a JDBC driver for a MySQL connector and an HTTP client library for an ES connector. A connector request is sent to the repository, carrying the name from the identifier, and if necessary, the connector version information. After verifying the request's permissions, the connector repository returns the corresponding implementation package file and its dependency list, and then downloads and caches these files locally.

[0079] Step S11-3: Dynamically load and instantiate the implementation package and dependencies of the source connector and the implementation package and dependencies of the target connector to determine the source connector and the target connector.

[0080] In this embodiment, the system uses a Java class loader or similar mechanism to dynamically load the implementation package and dependencies obtained in step S11-2 into the runtime memory space. Dynamic loading refers to the process of loading external code into the JVM or other runtime environment during program execution rather than compilation, enabling the system to flexibly acquire and update components without restarting. Instantiation refers to creating a concrete object of the connector through reflection or dependency injection, allocating memory, and initializing its internal state, transforming it from static code into a runnable component. First, the implementation package of the source connector and its dependencies are loaded, and after instantiation, a complete source connector object is obtained, in which the internal connection configuration unit, data pattern descriptor, semantic mapping rules, and data reading interface are all in a usable state. Then, the implementation package of the target connector and its dependencies are loaded, and after instantiation, the target connector object is obtained, finally determining the required source connector and target connector.

[0081] In conjunction with the above embodiments, an embodiment of this application also provides another cross-data space data synchronization method based on connector construction. In this method, step S13, "reading the original data in the source data space through the source connector and mapping the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics," includes steps S13-2-1 to S13-2-4: Step S13-2-1: Establish a connection with the source data space through the connection parameters of the source data space recorded in the connection configuration unit of the source connector.

[0082] In this embodiment, a reliable communication channel is established with the source data space through the connection parameters recorded in the connection configuration unit encapsulated within the source connector. The connection configuration unit is a structured object within the source connector used to store all connection-related parameters, including necessary information such as host address, port number, database name, table name or Topic name, username, and password. Sensitive information such as passwords is encrypted for security, and some connection parameters support dynamic injection to adapt to different environments. The source data space is the original data storage system from which the data flows, and can be various heterogeneous systems such as relational databases, data warehouses, NoSQL databases, and cloud storage services. The system extracts these connection parameters from the connection configuration unit and uses the corresponding protocol driver or client library to establish a network connection and perform authentication to access the source data space.

[0083] Step S13-2-2: Read the original data in the source data space through the data reading interface of the source connector.

[0084] In this embodiment, the execution phase of actually acquiring raw data through the data reading interface of the source connector involves calling the data reading interface within the source connector to read raw data from the source data space in a distributed parallel manner after a successful connection is established. The data reading interface encapsulates the underlying details of interaction with the source data space and supports data reading strategies such as sharding, partitioning, and parallel reading. Raw data refers to the data records stored in the source data space. Based on the structural information identified by the data pattern descriptor, a query statement or read request is constructed, and multiple parallel read tasks are initiated. Each read task is responsible for reading a specific range of data shards, and raw data records are acquired in batches through a network connection.

[0085] Step S13-2-3: Identify the original data pattern and original semantics of the original data through the data pattern descriptor of the source connector.

[0086] In this embodiment, the understanding stage of the original data structure is identified through the data schema descriptor of the source connector. The read original data is analyzed to clarify its original data schema and original semantics. The data schema descriptor is a component in the source connector used to describe and identify the data structure. It can be statically predefined or dynamically inferred automatically, and includes metadata such as field names, data types, primary key constraints, and index information. The original data schema refers to the specific structural form of the original data in the source data space, including field naming conventions, data type definitions, and physical storage formats. The original semantics refers to the specific business meaning of the original data in the source data space, including measurement units, encoding rules, and value ranges. The original data schema and original semantics of the original data are parsed through the data schema descriptor.

[0087] Step S13-2-4: Using the semantic mapping rules of the source connector, the original data pattern and original semantics are mapped to a general data pattern and general semantics.

[0088] In this embodiment, the actual transformation mapping step is performed through the semantic mapping rules of the source connector. The semantic mapping rules encapsulated inside the source connector are applied to transform the original data pattern and original semantics into the system's internal general data pattern and general semantics. Semantic mapping rules are a key innovative component within the source connector. They are a pluggable rule engine or configuration unit that predefines the transformation logic from a specific data space to a general standard. The general data pattern refers to the standardized data structure used uniformly within the system, including common field naming conventions (e.g., customer_id instead of cust_id), common data type systems (e.g., STRING, FLOAT, TIMESTAMP), and common structural organization methods (e.g., flat structures). General semantics refers to the standardized business meanings used uniformly within the system, including unified units of measurement (e.g., yuan instead of fen), unified status codes (e.g., 'PAID' instead of 2), and unified time formats (e.g., standard timestamps instead of various date formats). For each piece of raw data, the semantic mapping rules are applied to modify field names using field renaming rules, adjust types using data type conversion rules, map encoded values ​​using value conversion rules, adjust measurements using unit conversion rules, and reorganize data using structure transformation rules, ultimately generating general data that conforms to the general data pattern and general semantics.

[0089] In conjunction with the above embodiments, an embodiment of this application also provides another cross-data space data synchronization method based on connector construction. In this method, step S13, "mapping the general data of the general data pattern and general semantics into target data of the target data pattern and target semantics and writing it into the target data space through the target connector," includes steps S13-3-1 to S13-3-3: Step S13-3-1: Establish a connection with the target data space through the connection parameters of the target data space recorded in the connection configuration unit of the target connector.

[0090] In this embodiment, a connection is established with the target data space using the connection parameters of the target data space recorded in the connection configuration unit encapsulated within the target connector. The connection configuration unit is a structured object within the target connector, used to centrally store all parameter information required to establish a connection with the target data space. These parameters are extracted from the connection configuration unit, and a connection request is initiated using the corresponding protocol driver. After authentication and permission checks, a physical connection is established with the target data space.

[0091] Step S13-3-2: Through the semantic mapping rules of the target connector, the general data pattern and the general semantic data are mapped to the target data pattern and the target semantic data.

[0092] In this embodiment, the final transformation mapping step is performed through the semantic mapping rules of the target connector. The semantic mapping rules encapsulated within the target connector are applied to convert the general data pattern and general semantics into the target data pattern and target semantics required by the target data space. The target data pattern refers to the specific structural requirements of the target data space, including the target system's specific field naming conventions (e.g., user_code instead of customer_id), the data types supported by the target system (e.g., KEYWORD instead of STRING, FLOAT remaining FLOAT), and the target system's structural organization method (e.g., nested documents in Elasticsearch, flat structure). The target semantics refers to the specific business meaning requirements of the target data space, including the target system's unit of measurement (e.g., dollar instead of yuan), the target system's status code (e.g., 'COMPLETED' instead of 'PAID'), and the target system's format standards (e.g., specific date formats, geographic coordinate formats). By applying the field renaming rules in the semantic mapping rules to modify each piece of general data to the target field name, applying data type conversion rules to adapt to the target type, applying value conversion rules to map to the target code, applying unit conversion rules to adjust to the target unit, and applying structure transformation rules to reorganize into the target structure, target data that fully conforms to the target data pattern and target semantics is finally generated.

[0093] Step S13-3-3: Write the target data into the target data space through the data writing interface of the target connector.

[0094] In this embodiment, the writing process is actually executed through the data writing interface of the target connector. The system utilizes the established physical connection to write the target data into the target data space in parallel through the data writing interface inside the target connector. Finally, the transformed target data is stored in the target data space, completing the entire cross-data space data synchronization process.

[0095] In conjunction with the above embodiments, one embodiment of this application also provides another cross-data space data synchronization method based on connector construction. In this method, after "writing the target data into the target data space" in step S13, steps S21 to S22 are further included: Step S21: Integrate the mapping processes from the original data to the general data, from the general data to the intermediate data, and from the intermediate data to the target data included in the data synchronization process from the source data space to the target data space to generate a data lineage map and store it in the connection sub-warehouse.

[0096] In this embodiment, after the target data is successfully written to the target data space, in order to integrate all the mapping processes included in the entire data synchronization process, a complete data lineage graph is generated and stored in the connection sub-warehouse. The flow of data between nodes is displayed in the form of a directed graph. The mapping processes of the three stages in the above data synchronization process are chained together in the execution order, recording the complete flow path of each field (such as cust_id→customer_id→user_code), the detailed definition of each transformation rule, the execution timestamp and performance indicators of each operation node, forming an end-to-end lineage link from the source data space to the target data space. This metadata is stored in the metadata index of the connection sub-warehouse and managed together with the version information of the connection sub-components and the task definition file to achieve persistence and traceability of data lineage information.

[0097] Step S22: If the data quality, data security, or data compliance of the target data in the target data space is abnormal, the mapping process of the target data is traced through the data lineage map to determine the cause of the abnormality.

[0098] In this embodiment, when the target data in the target data space exhibits anomalies in terms of data quality, data security, or data compliance, the complete mapping process of the target data is traced back by querying the data lineage graph generated in step S21 to determine the cause of the anomaly. Specifically, data quality anomalies refer to the target data's accuracy, completeness, consistency, timeliness, and other attributes not meeting expectations, such as a negative value in the `order_amount` field, a null value in the `user_code` field, or data delays exceeding a threshold. Data security anomalies refer to risks of leakage or unauthorized access during the access, transmission, and storage of sensitive data, such as failure to anonymize `customer_id` as required, or transmission of authentication information in plaintext. Data compliance anomalies refer to data processing processes that do not comply with regulations or internal standards, such as failure to record the purpose of data processing as required by GDPR, or failure to retain data deletion operation logs. The system uses a data lineage graph to trace the source field (customer_id → cust_id) from an abnormal field (such as user_code), all transformation rules (name renaming rules of the target join, name renaming rules of the source join), and all processing nodes involved (calculation logic in the user's SQL). This allows for quick identification of which step introduced the anomaly, such as a quality issue with the source data itself, an incorrect configuration of the semantic mapping rules of the source join, an error in the additional transformation logic declared by the user, or an inappropriate type adaptation of the target join. Simultaneously, the data lineage graph can also display all downstream systems and reports affected by the abnormal field, assess the scope of impact, and, based on version information in the join repository, determine whether a join version rollback or task configuration update is necessary, enabling rapid root cause analysis and precise remediation.

[0099] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another method for cross-data space data synchronization based on connector construction, illustrating the process after the connector is created. The method further includes steps S31 to S33: Step S31: Upload each developed connector to the connector repository.

[0100] In this embodiment, the developer packages the completed connector implementation package for a specific data space and its dependencies into a distributable format and uploads it to the connector repository. A developed connector refers to a software component that has undergone complete development and testing. The upload process includes transferring the connector implementation package (such as a JAR file, Python Wheel, etc.) to the repository server, and simultaneously submitting a metadata description file (including connector name, version number, supported data space types, interface specifications, dependency list, usage examples, author information, etc.) to allow the connector to enter the connector repository for full lifecycle management.

[0101] Step S32: After reviewing the code quality, security, standardization, and accuracy of semantic mapping of the connector, publish the connector and set the connector identifier.

[0102] In this embodiment, the warehouse administrator or automated system performs multi-dimensional review and verification of the uploaded connectors. Once approved, the connector is officially released and assigned a unique identifier. Code quality review includes checking code style, unit test coverage, performance metrics, and the completeness of exception handling. Security review includes checking authentication information encryption mechanisms, access control logic, vulnerability risks, and the security of dependent libraries. Standardization review includes checking whether the interface conforms to standards, configuration parameter naming conventions, and documentation completeness. Semantic mapping accuracy review includes verifying whether field renaming rules, type conversion logic, unit conversion formulas, and value mapping tables are correct and reasonable. After passing the review, the connector is officially released, assigned a unique connector identifier, which serves as the sole credential for subsequent user reference and system acquisition of the connector. Simultaneously, the connector becomes downloadable for authorized users.

[0103] Step S33: Store the metadata of the connector and set the access permissions for the connector.

[0104] In this embodiment, the metadata information of the connector is persistently stored and indexed, while fine-grained access control policies are set. The metadata includes static description information of the connector (name, version, author, creation time, supported data space versions), interface specification information (input / output field examples, data schema definition, supported configuration parameters), semantic mapping rule summary (conversion logic overview, lookup table definition), dependency graph (list of third-party libraries and their versions), and lineage information (records of data synchronization tasks participated in by the connector). The stored procedure writes this metadata into the metadata database of the connector repository, establishing full-text and structured indexes to support fast retrieval. Access control settings include defining which users or teams can view connector information, download implementation packages, reference connectors to execute tasks, modify connector configurations, and release new versions. Role-based access control (RBAC) or attribute-based access control (ABAC) mechanisms are used to implement access control, preventing unauthorized use and security risks, while also supporting full lifecycle management and governance of the connector.

[0105] In conjunction with the technical solutions of the above embodiments, one embodiment of this application also provides another method for cross-dataspace data synchronization based on connector construction, the method further comprising: Step S41: If the connection parameters of the data space change or the semantic mapping rules change, update the version of the corresponding connector in the connection sub-repository of the data space so that the task definition text referencing the connector can take effect without modification.

[0106] In this embodiment, throughout the entire lifecycle of a connector, when the connection parameters of the data space change (such as changes in host address, port number, and authentication information due to database migration) or the semantic mapping rules change (such as field renaming, unit conversion formulas, and status code mapping table updates due to business adjustments), it is only necessary to update the version of the corresponding connector for that data space in the connector repository, and all task definition files referencing that connector will automatically take effect without any modification.

[0107] Specifically, the connection configuration unit within the connector is directly modified to reflect the latest connection parameters in the data space, or the semantic mapping rules are adjusted to adapt to business changes in the data space, and then a new version of the connector is released. The connector repository retains historical versions of connectors for use by data synchronization tasks corresponding to un-upgraded task definition files, while also providing the latest version for reference by data synchronization tasks corresponding to new task definition files or actively upgraded task definition files. Since specific versions are referenced in the task definition files through connector identifiers, the corresponding version's implementation package is automatically retrieved from the repository when parsing the task. Therefore, after updating the connector version, only the relevant team needs to be notified to update the connector version number declared in the task definition file. Other configurations in the task definition file (including additional conversion logic) remain completely unchanged, automatically adapting to changes in the data space. This versioning management achieves a "one-stop modification, global effect" governance model, avoiding the tediousness and inefficiency of modifying dozens or even hundreds of synchronization scripts one by one when the data space interface changes in the traditional way. This significantly reduces operation and maintenance costs, ensures the controllability, traceability, and rollback capability of changes, and improves the maintainability and stability of the entire organization's data integration architecture.

[0108] The technical solution of this application has the following significant advantages: (1) Declarative simplification: Users are freed from writing complex code and configuration. They can complete most of the work by simply declaring "which connector to use", which greatly reduces the threshold for use and the complexity of configuration.

[0109] (2) Semantic consistency guarantee: By using the semantic mapping rules built into the connector, the originally scattered and error-prone semantic conversion logic is standardized and componentized, ensuring the accurate transmission of the meaning of data across systems and effectively bridging the semantic gap.

[0110] (3) Reusability and maintainability: As an independent component, the connector can be reused across the entire organization. When the data source interface changes or business rules are adjusted, only the corresponding connector version needs to be updated in the connector repository. All tasks referencing the connector will take effect without modification, greatly improving maintainability.

[0111] (4) Performance and scalability: The underlying layer is based on a high-performance distributed engine, which naturally supports batch and stream synchronization of massive data. The parallel read and write interface of the connector fully utilizes the distributed capabilities of the engine.

[0112] (5) Data governance support: The connecting sub-warehouse itself is a rich source of metadata, clearly recording the interface specifications for data inflow and outflow. Combined with the data lineage analysis function, a complete and accurate data lineage map from the data source to the target end can be easily constructed, providing a solid foundation for data quality, security and compliance.

[0113] (5) Eco-friendly and standardized: The connector specification of this application can become an open standard, encouraging communities and enterprises to contribute connectors for various data spaces, forming a thriving ecosystem and promoting the overall progress of data integration technology.

[0114] It should be noted that the cross-data space data synchronization method based on connector construction provided in this application embodiment can be executed by a cross-data space data synchronization device based on connector construction, or by a control module in the device for executing the cross-data space data synchronization method based on connector construction. This application embodiment uses the execution of the cross-data space data synchronization method based on connector construction by a cross-data space data synchronization device as an example to illustrate the cross-data space data synchronization method based on connector construction provided in this application embodiment.

[0115] Figure 5 This is a schematic diagram of the framework of a cross-dataspace data synchronization device based on a connector structure according to an embodiment of this application. (Refer to...) Figure 5 .

[0116] One embodiment of this application provides a cross-dataspace data synchronization device based on a connector structure, the device comprising: The connector determination module 11 is used to determine the source connector and the target connector based on the task definition file submitted by the user. The source connector is the connector corresponding to the source data space, and the target connector is the connector corresponding to the target data space. The connector includes a connection configuration unit, a data pattern descriptor, semantic mapping rules and data reading interface, and data writing interface. The data synchronization job graph generation module 12 is used to determine the data synchronization job graph according to the semantic mapping rules of the source connector and the semantic mapping rules of the target connector; The data synchronization module 13 is used to read the original data in the source data space through the source connector according to the data synchronization operation diagram, and map the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics, and to map the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics through the target connector, and write the target data into the target data space.

[0117] Optionally, if the task definition file further includes additional transformation logic, the data synchronization job graph generation module 12 includes: A data synchronization job graph generation unit is used to integrate the additional transformation logic with the semantic mapping rules of the source connector and the semantic mapping rules of the target connector to obtain the data synchronization job graph; The data synchronization module 13 includes: The first mapping unit is used to read the original data in the source data space through the source connector and map the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics. The second mapping unit is used to process the general data of the general data pattern and the general semantics according to the additional transformation logic to obtain intermediate data of the general data pattern and the general semantics. The third mapping unit is used to map the intermediate data of the general data pattern and general semantics to the target data of the target data pattern and target semantics through the target connector, and write the target data into the target data space.

[0118] Optionally, the connection sub-determination module 11 includes: The parsing unit is used to parse the task definition file and determine the source linker identifier and the target linker identifier; The request unit is configured to initiate a request to the linker repository based on the source linker identifier and the target linker identifier to obtain the implementation package and dependencies of the source linker and the implementation package and dependencies of the target linker; The connector determination unit is used to dynamically load and instantiate the implementation package and dependencies of the source connector and the implementation package and dependencies of the target connector, and determine the source connector and the target connector.

[0119] Optionally, the data synchronization module 13 includes: The first connection unit is used to establish a connection with the source data space through the connection parameters of the source data space recorded in the connection configuration unit of the source connector; The reading unit is used to read the original data in the source data space through the data reading interface of the source connector; An identification unit is used to identify the original data pattern and original semantics of the original data through the data pattern descriptor of the source connector; The fourth mapping unit is used to map the original data pattern and original semantics into a general data pattern and general semantics through the semantic mapping rules of the source connector.

[0120] Optionally, the data synchronization module 13 includes: The second connection unit is used to establish a connection with the target data space through the connection parameters of the target data space recorded in the connection configuration unit of the target connector; The fifth mapping unit is used to map the general data pattern and general semantics to the target data pattern and target semantics through the semantic mapping rules of the target connector; The writing unit is used to write the target data into the target data space through the data writing interface of the target connector.

[0121] Optionally, the device further includes: The storage module is used to integrate the mapping processes from the original data to the general data, the general data to the intermediate data, and the intermediate data to the target data included in the data synchronization process from the source data space to the target data space after the target data is written into the target data space, generate a data lineage map, and store it in the connection sub-warehouse; The tracing module is used to trace the mapping process of the target data through the data lineage map to determine the cause of the anomaly when the data quality, data security, or data compliance of the target data in the target data space is abnormal.

[0122] Optionally, the device further includes: The upload module is used to upload each developed connector to the connector repository; The review module is used to review the code quality, security, standardization, and accuracy of semantic mapping of the connector before releasing the connector and setting the connector identifier. The settings module is used to store the metadata of the connector and set the access permissions for the connector.

[0123] Optionally, the device further includes: The update module is used to update the version of the corresponding connector in the corresponding connector repository of the data space when the connection parameters or semantic mapping rules of the data space change, so that the task definition text referencing the connector can take effect without modification.

[0124] Optionally, the device further includes: The first configuration module is used to configure the connection configuration unit. The connection parameters of the connection configuration unit include at least the host address, port, database name, username, password, table name, and Topic name of the data space. The second configuration module is used to configure the data pattern descriptor, which is used to define the structure, type and constraint relationship of the data in the data space in a self-describing manner. The third configuration module is used to configure semantic mapping rules. The semantic mapping rules are pluggable components containing a rule engine, used to predefine or parse data patterns and semantics at runtime, and to map and convert between data patterns and semantics and general data patterns and semantics in terms of field naming, data types, data formats, business codes, and units of measurement. The semantic mapping rules include at least: field renaming rules, data type conversion rules, value conversion rules, unit of measurement conversion rules, and structure transformation rules. The fourth configuration module is used to configure the data reading interface, which is used to read data in parallel from the corresponding data space; The fifth configuration module is used to configure the data writing interface, which is used to write data in parallel to the corresponding data space. The cross-data space data synchronization device based on the connector structure in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not specifically limit the specific implementation.

[0125] The cross-data space data synchronization device based on connector construction in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0126] The cross-dataspace data synchronization device based on connector construction provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented by the cross-data space data synchronization device based on the connector construction in the method embodiment will not be described again here to avoid repetition.

[0127] Optionally, Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. This application also provides an electronic device; it should be noted that the electronic device in this application includes the mobile electronic device and non-mobile electronic device described above.

[0128] The electronic device includes, but is not limited to, components such as: radio frequency unit, network module, audio output unit, input unit, sensor, display unit, user input unit, interface unit, memory, and processor.

[0129] Those skilled in the art will understand that electronic devices may also include power supplies (such as batteries) that supply power to various components. The power supply may be connected to the processor logic through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0130] As an example, such as Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the cross-data space data synchronization method based on connector construction disclosed in the above embodiments of this application.

[0131] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0134] Furthermore, this application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described cross-data space data synchronization method embodiment based on connector construction and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0135] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0136] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described cross-data space data synchronization method embodiment based on connector construction, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0137] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A cross-dataspace data synchronization method based on connector construction, characterized in that, The method includes: Based on the task definition file submitted by the user, source connectors and target connectors are determined. The source connector is the connector corresponding to the source data space, and the target connector is the connector corresponding to the target data space. The connector includes a connection configuration unit, a data pattern descriptor, semantic mapping rules and data reading interface, and data writing interface. Based on the semantic mapping rules of the source connector and the semantic mapping rules of the target connector, the data synchronization operation graph is determined; According to the data synchronization operation diagram, the original data in the source data space is read through the source connector, and the original data pattern and original semantics corresponding to the original data are mapped to a general data pattern and general semantics. The general data of the general data pattern and general semantics are mapped to target data of the target data pattern and target semantics through the target connector, and the target data is written into the target data space.

2. The cross-dataspace data synchronization method based on connector construction according to claim 1, characterized in that, If the task definition file also includes additional transformation logic, determining the data synchronization job graph based on the semantic mapping rules of the source connector and the target connector includes: The additional transformation logic is integrated with the semantic mapping rules of the source connector and the semantic mapping rules of the target connector to obtain the data synchronization operation graph; The step of reading raw data from the source data space through the source connector, mapping the raw data pattern and raw semantics corresponding to the raw data to a general data pattern and general semantics, and mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics through the target connector, and writing the target data into the target data space, includes: The source connector reads the original data in the source data space and maps the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics. Based on the additional transformation logic, the general data of the general data pattern and general semantics are processed to obtain intermediate data of the general data pattern and general semantics; The intermediate data of the general data pattern and general semantics are mapped to the target data of the target data pattern and target semantics through the target connector, and the target data is written into the target data space.

3. The cross-dataspace data synchronization method based on connector construction according to claim 2, characterized in that, The step of determining the source connector and target connector based on the task definition file submitted by the user includes: Parse the task definition file to determine the source linker identifier and the target linker identifier; Based on the source connector identifier and the target connector identifier, a request is sent to the connector repository to obtain the implementation package and dependencies of the source connector and the target connector. The implementation packages and dependencies of the source connector and the target connector are dynamically loaded and instantiated to determine the source connector and the target connector.

4. The cross-dataspace data synchronization method based on connector construction according to claim 1, characterized in that, The step of reading the original data in the source data space through the source connector and mapping the original data pattern and original semantics corresponding to the original data to a general data pattern and general semantics includes: A connection is established with the source data space through the connection parameters of the source data space recorded in the connection configuration unit of the source connector; The original data in the source data space is read through the data reading interface of the source connector; The original data pattern and original semantics of the original data are identified by the data pattern descriptor of the source connector; The original data pattern and original semantics are mapped to a general data pattern and general semantics through the semantic mapping rules of the source connector.

5. The cross-dataspace data synchronization method based on connector construction according to claim 1, characterized in that, The step of mapping the general data of the general data pattern and general semantics to target data of the target data pattern and target semantics and writing it into the target data space through the target connector includes: A connection is established with the target data space through the connection parameters of the target data space recorded in the connection configuration unit of the target connector; The semantic mapping rules of the target connector are used to map the general data pattern and general semantics to the target data pattern and target semantics. The target data is written into the target data space through the data writing interface of the target connector.

6. The cross-dataspace data synchronization method based on connector construction according to claim 3, characterized in that, After writing the target data into the target data space, the process also includes: The data synchronization process from the source data space to the target data space includes the mapping processes from the original data to the general data, from the general data to the intermediate data, and from the intermediate data to the target data. These processes are then integrated to generate a data lineage map, which is stored in the connection sub-warehouse. In the event of abnormalities in the data quality, data security, or data compliance of the target data in the target data space, the mapping process of the target data is traced through the data lineage map to determine the cause of the abnormality.

7. The cross-dataspace data synchronization method based on connector construction according to any one of claims 3-5, characterized in that, The method further includes: Upload each developed connector to the connector repository; After reviewing the code quality, security, standardization, and accuracy of semantic mapping of the connector, the connector is released and a connector identifier is set. Store the metadata of the connector and set its access permissions.

8. The cross-dataspace data synchronization method based on connector construction according to any one of claims 3-5, characterized in that, The method further includes: If the connection parameters of the data space change or the semantic mapping rules change, the version of the corresponding connector in the connection sub-repository of the data space is updated so that the task definition text referencing the connector can take effect without modification.

9. The cross-dataspace data synchronization method based on connector construction according to any one of claims 3-5, characterized in that, The process of creating the connector includes: Configure the connection configuration unit, wherein the connection parameters of the connection configuration unit include at least the host address, port, database name, username, password, table name, and Topic name of the data space; Configure a data schema descriptor, which is used to define the structure, type and constraint relationships of data in the data space in a self-describing manner; Configure semantic mapping rules, which are pluggable components containing a rule engine, used to map and convert the data patterns and semantics of predefined or runtime data parsing data with common data patterns and semantics in terms of field naming, data types, data formats, business codes, and units of measurement; the semantic mapping rules include at least: field renaming rules, data type conversion rules, value conversion rules, unit of measurement conversion rules, and structure transformation rules; Configure a data reading interface, which is used to read data in parallel from the corresponding data space; Configure a data write interface, which is used to write data in parallel to the corresponding data space.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the cross-data space data synchronization method based on connector construction as described in any one of claims 1-9.

Images