A Data Sharing and Exchange Method and System Based on Multi-Source Heterogeneous Connectors

By adopting a data sharing and exchange method based on multi-source heterogeneous connectors, dynamically registering and loading data source connectors, and using a unified JSON intermediate object for data format conversion, the problems of difficult access to multi-source heterogeneous data sources and low synchronization efficiency are solved, and the horizontal expansion and efficient synchronization of data sources are realized.

CN123115250APending Publication Date: 2026-11-17HENAN POLICE ACAD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611213338.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies suffer from difficulties in accessing multi-source heterogeneous data, poor scalability, and low synchronization efficiency. In particular, the lack of a unified synchronization mode and format conversion mechanism between heterogeneous data sources leads to format incompatibility and synchronization delays during data transmission.

Method used

By adopting a data sharing and exchange method based on multi-source heterogeneous connectors, the data source connectors are dynamically registered and loaded using the Service Provider Interface (SPI) mechanism. A registry is established to map the data source type to the connector implementation class. A unified JSON intermediate object is used for data format conversion. Data is extracted and converted according to different synchronization modes, including full, incremental and real-time synchronization, to achieve efficient data flow between different systems.

Benefits of technology

It has improved the horizontal scalability of data source access, enhanced the compatibility and synchronization efficiency of data transmission, reduced system maintenance costs, and prevented data exchange link paralysis caused by the failure of a single component.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115250A_ABST
    Figure CN123115250A_ABST
Patent Text Reader

Abstract

This invention relates to the field of data integration and sharing exchange technology, and discloses a data sharing and exchange method and system based on multi-source heterogeneous connectors. The method dynamically registers and loads data source connectors through the Service Provider Interface (SPI) mechanism, maintains a registry mapping data source types to implementation classes, receives configuration information including data source type, connection parameters, and synchronization mode, retrieves connector instances from the registry based on the data source type, establishes a connection with the underlying physical data source based on the connection parameters, extracts source-end data records according to the synchronization mode, converts them into a unified intermediate state object, and reverse-parses and concatenates them into a target-end compatible physical write format. This invention also discloses a corresponding data sharing and exchange system. This invention eliminates the need to modify the core scheduling code when accessing new data sources, reduces the amount of structural parsing computation during heterogeneous data flow, and improves data compatibility and exchange efficiency across systems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data integration and sharing exchange technology, specifically a data sharing and exchange method and system based on multi-source heterogeneous connectors. Background Technology

[0002] With the advancement of business informatization, a large amount of data has accumulated in various departments and business systems. This data is typically stored in various underlying physical systems of different types, such as relational databases, time-series databases, document databases, message queues, and object storage, and their data structures and network access protocols vary considerably.

[0003] Existing data integration and exchange solutions have several technical limitations when handling the aforementioned multi-source heterogeneous data. Firstly, at the system architecture level, traditional solutions typically require developing dedicated data access adaptation components for each data source, resulting in a high degree of coupling between the core data scheduling code and the underlying physical access logic of the data sources. When a new type of data source needs to be accessed in the business environment, developers must modify the core scheduling code within the system and perform recompilation and deployment. This rigid access method leads to weak horizontal scalability of the data exchange system and high maintenance costs in the later stages.

[0004] Secondly, in the data acquisition and synchronization stage, traditional data extraction tools mainly rely on timed batch reading modes, and the underlying data extraction strategies are relatively simple. Faced with different needs in complex business scenarios, such as migrating full existing data, periodically capturing incremental data, and continuously monitoring high-timeliness data streams, existing solutions lack a unified synchronization mode scheduling interface. This leads to data synchronization delays in some businesses with high timeliness requirements, making it difficult to meet the data timeliness requirements of different businesses.

[0005] Finally, regarding data format processing, heterogeneous data sources have different underlying storage structures. Existing data synchronization processes often rely on developers directly establishing point-to-point format mapping code between two different heterogeneous data sources. This cross-system data structure alignment lacks a central, decoupling, universal conversion mechanism. As the number of data source nodes connected to the system increases, the underlying format conversion chain presents a complex mesh structure, which not only has a low degree of automation but is also prone to format incompatibility issues leading to write failures during data transmission. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a data sharing and exchange method and system based on multi-source heterogeneous connectors, aiming to solve the technical problems of difficulty in accessing heterogeneous data sources, poor scalability, and low synchronization efficiency in existing technologies.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a data sharing and exchange method based on multi-source heterogeneous connectors, comprising the following steps: The service provider interface (SPI) mechanism is used to dynamically register and load data source connectors, and maintain a registry that maps data source types to connector implementation classes. Receive data source configuration information including data source type, connection parameters, and synchronization mode; The connector instance is obtained from the mapping registry based on the data source type, and a connection to the underlying physical data source is established based on the connection parameters. Extract source data records based on synchronization mode; The source data records are converted into a unified intermediate object, and the unified intermediate object is reverse parsed to extract structured fields and values. These are then assembled into a physical write format compatible with the target end according to the target end's protocol syntax.

[0008] In one specific embodiment, the step of reverse parsing the unified intermediate object to extract structured fields and values, and then assembling them into a target-compatible physical write format according to the target's protocol syntax, includes: The system optimizes and converts source data records into a unified JSON intermediate object; it then iterates through the key-value pairs within the unified JSON intermediate object to perform reverse parsing, stripping the key names from the JSON intermediate object and extracting the data type and pure data value of the target attribute. Based on the extracted data type and the pure data value, reverse columnar encoding is performed, that is, the key-value pairs of the same property in multiple JSON objects are gathered and concatenated in memory into a data block format that is stored in columns. Alternatively, binary serialization compression can be performed based on the extracted data type and the pure data value. This involves converting the pure data value into a binary byte stream according to the target's preset strong type mode and applying a compression algorithm. Finally, the data blocks or byte streams are concatenated and assembled into a data payload that the target can write.

[0009] Preferably, the registry for maintaining the mapping relationship between data source types and connector implementation classes includes: The system calls the underlying service loading component to read the configuration file that records the fully qualified names of each class in the classpath, and uses dynamic instantiation methods to instantiate each data source connector class in memory; it constructs a data dictionary object based on a hash structure in memory, extracts the identifier characters of each data source type as the key value, and uses the corresponding connector implementation class object as the content value to obtain a mapping registry, so as to establish the memory addressing path between external data source types and internal processing components within the system.

[0010] Preferably, in S4, extracting source data records according to the synchronization mode includes: If the synchronization mode is a full synchronization mode identifier, inject a streaming query instruction to establish a cursor and obtain the source data record based on the cursor and store it in the data buffer. If the synchronization mode is an incremental synchronization mode identifier, the source data record is extracted based on the timestamp watermark mechanism or the incremental identifier mechanism built into the data table. If the synchronization mode is identified as a real-time synchronization mode, the event message object acquired by the listener is delivered to a pre-allocated thread-safe queue in the system memory and extracted using a consumer thread.

[0011] In one specific embodiment, if the synchronization mode is a full synchronization mode identifier, injecting a streaming query instruction to establish a cursor and obtaining source data records based on the cursor and storing them in the data buffer specifically includes: If the synchronization mode is a full synchronization mode, a streaming query instruction is injected into the connector instance to establish a cursor and the source data records are obtained based on the cursor and stored in the data buffer. When the number of records in the data buffer reaches the preset batch capacity threshold, the cursor reading action is suspended and the current data buffer is transferred to the downstream node as a data batch. After performing memory reclamation operation, the suspended task is woken up to continue pulling the next batch of data. In one specific embodiment, if the synchronization mode is an incremental synchronization mode identifier, the source data record is extracted based on the timestamp watermark mechanism or the incremental identifier mechanism built into the data table, specifically including: If the synchronization mode is identified as incremental synchronization mode, the highest timestamp parameter or identifier offset recorded in the local persistent component is read as the water level boundary, and the conditional filtering expression is assembled and sent to the underlying connector to extract the source data record; or the connector instance is called to establish a special log reading connection channel at the underlying level, receive the underlying transaction log file dumped by the database master node, parse the log byte stream in memory and identify the row-level change details of insertion, update and deletion. The extraction of source data records based on the timestamp watermark mechanism includes: Read the maximum timestamp of the previous round of synchronization task recorded in the local persistence component as the water level limit, use the water level limit as a filter condition to assemble into the query request, extract data records whose target time field is greater than the water level limit, and overwrite the local record with the maximum timestamp of the current round after synchronization is completed. The method of extracting source data records based on the built-in incremental identification mechanism of the data table includes: obtaining the auto-incrementing primary key ID or data row version number field of the target data table as the offset parameter for incremental tracking, and extracting data records whose identification value is greater than the offset parameter.

[0012] In one specific embodiment, if the synchronization mode is a real-time synchronization mode identifier, the event message object acquired through monitoring is delivered to a pre-allocated thread-safe queue in system memory and retrieved using a consumer thread, specifically including: If the synchronization mode is identified as real-time synchronization mode, a background resident thread is allocated and started in the system memory to maintain a persistent bidirectional long-connection communication channel. The event message objects acquired by the listener are directly delivered to the thread-safe queue pre-allocated in the system memory, and the capacity status of the queue is continuously monitored by the consumer thread maintained in parallel by the system to perform dequeueing and retrieval.

[0013] Preferably, the source data records are converted into a unified JSON intermediate object, and the unified JSON intermediate object is reverse-parsed and concatenated into a physical write format compatible with the target end, including: In the process of converting and generating a unified JSON intermediate object, the field names of the source data table are used as keys, and the content of the specific data row is used as the corresponding value. For unstructured or complex storage formats, the first line of the comma-separated value file is extracted by streaming and used as the attribute field key name, and subsequent data rows are split according to the delimiter. Alternatively, the tree-like hierarchical structure of the Extensible Markup Language file is parsed and the node element name is mapped to the key. Or, a component is introduced to parse the physical structure of the spreadsheet file and extract the header information for mapping, thus completing the forward conversion of heterogeneous data flow to the unified JSON intermediate state. Based on the target data source type in the task configuration, the system triggers the target connector instance to traverse the key-value pairs inside the unified JSON intermediate object, perform reverse columnar encoding, or perform type validation and binary serialization compression processing according to the pattern definition file to generate the physical write format.

[0014] In one specific embodiment, after establishing the connection with the underlying physical data source, the method further includes: During data synchronization, the system uses injected interception points to measure end-to-end latency metrics. The system records the initial timestamp when the source reads the first data record of the current batch into memory and records the end timestamp after the target confirms successful writing to the underlying physical data source, and calculates the difference between the two. At the same time, the system uses a built-in counter to accumulate the throughput bytes per unit time at the system flow nodes. When the end-to-end latency metrics exceed the preset maximum latency time threshold or the throughput is abnormal, an alarm message object is assembled and an alarm response program is triggered through the network protocol. A connection object storage area is pre-allocated in the heap memory as a connection pool. A background scheduled task periodically extracts all connection instances in the connection pool to perform probe message liveness status detection and actively removes failed connections that do not return a success flag within the timeout countdown.

[0015] A second aspect of the present invention provides a data sharing and exchange system based on multi-source heterogeneous connectors, applied to the method described in the first aspect above, comprising: The connector management module is used to dynamically read configuration files through the Service Provider Interface (SPI) mechanism to register and load data source connectors, and to build a hash structure in memory to maintain a registry that maps data source types to connector implementation classes. The configuration management module is used to receive data source configuration information streams, including data source type, connection parameters, and synchronization mode, and deserialize them into entity objects containing host address, service port, authentication credentials, and scheduling period. The connector implementation layer is used to locate and obtain the connector instance based on the data source type, and to initiate a network request to the target host based on the credentials of the connection parameters to establish a connection with the underlying physical data source. The data synchronization engine is used to perform data extraction operations and coordinate the scheduling of downstream data according to the synchronization mode. The format conversion engine is used to perform in-memory format pipeline operations, converting source data records into a unified intermediate state object and then parsing and concatenating it into a physical write format compatible with the target end. The monitoring and alarm module is used to monitor the end-to-end latency, throughput status, and exception events of the data synchronization process using the asynchronous collection component, and to trigger an early warning notification mechanism when the indicators deviate.

[0016] Preferably, the system further includes a unified interface layer that defines standard data source connector interfaces. The unified interface layer includes the following interfaces: A connection establishment operation that receives a connector configuration object encapsulating host and port information as input parameters to establish a physical network connection. The connection disconnection operation disconnects the connection to the data source and releases resources when the task is completed or an exception is caught. In conjunction with the heartbeat mechanism, probe messages are sent to test the connection test operation to assess the availability of the current physical communication link. This function retrieves metadata about the underlying data source, including its table structure, field names, and type information. It receives a query request object carrying the table name and filter conditions and performs data query operations on the underlying data request. The data synchronization operation receives a request object encapsulating routing information and returns a task execution status flag.

[0017] In one specific embodiment, the data synchronization engine includes: The full synchronization unit is used to perform full table data query through the cursor mechanism and complete batch write control in the memory buffer. The incremental synchronization unit is used to identify and synchronize changed data rows based on the read local persistent timestamp watermark, auto-incrementing primary key identifier, or fake slave node monitoring change data capture mechanism. The real-time synchronization unit is used to continuously monitor real-time change events of the data source by relying on a long-connection thread that resides in the background, and quickly push them to the target end through a memory queue buffer.

[0018] Preferably, the specific data source connectors included in the connector implementation layer include: This includes relational database connectors and data warehouse connectors that encapsulate the underlying JDBC communication protocol to execute pre-compiled statement queries and wide table data extraction; document database connectors and search engine connectors that execute aggregation pipeline statements, basic document read and write commands, and full-text structure search requests; cached database connectors that perform key-value pair state access; time-series database connectors that include various network API encapsulations to send tagged read and write workloads; and object storage connectors, distributed file system connectors, and message queue connectors that call dedicated Java development kits to execute file stream byte data, stream data production and delivery, and pre-signed link generation.

[0019] In one specific embodiment, the format conversion engine is configured with a local memory dictionary. The format conversion engine transfers the database table and field structure objects returned by the metadata retrieval operation to the system's local memory dictionary. When the data records are reconstructed into JSON format, the engine directly retrieves the field names and data type mapping relationships of the target table from the memory dictionary, thereby eliminating repeated network queries to the underlying system tables. The memory dictionary is configured with a lifetime parameter based on a system preset empirical constant. When the residence time of the cached object reaches the set lifetime limit, the memory cleanup logic is triggered to require subsequent tasks to retrieve the metadata again.

[0020] The present invention provides a data sharing and exchange method and system based on multi-source heterogeneous connectors, which has the following advantages: This invention dynamically registers and loads data source connectors through the Service Provider Interface (SPI) mechanism and maintains a registry in memory mapping data source types to connector implementation classes. Upon receiving data source configuration information, it can directly obtain connector instances and establish underlying physical connections using dictionary addressing in the registry. This approach separates the underlying data source's physical network communication implementation from the upper-layer data scheduling logic. When a business application needs to access a new type of data source, it only needs to add the corresponding connector implementation class to a specified directory, eliminating the need to modify and recompile the original core scheduling code and improving the horizontal scalability of the data processing workflow.

[0021] This invention introduces a data transfer and format reorganization mechanism based on a unified JSON intermediate object during the data transfer stage. After extracting data records from the source according to a set synchronization mode, a standard JSON intermediate object is generated based on the field names and row contents of the source data table. This intermediate object is then reverse-parsed and concatenated into a physical write format compatible with the target before being written to the target. This approach establishes a central, universal data format, replacing the hard-coding development work of establishing exclusive data alignment rules between heterogeneous systems. It reduces the computational load of structure parsing during the transmission of structured and unstructured data, and improves the compatibility of data sharing and exchange between different systems.

[0022] This invention relies on clearly defined system functional modules as the physical execution carrier. The connector management module, configuration management module, connector implementation layer, data synchronization engine, and format conversion engine within the system work collaboratively according to time and logical order. At the physical deployment level, this system architecture decouples and separates the operational nodes such as external configuration parsing, underlying link connection, data scheduling and extraction, and format parsing and reassembly, allowing each data flow link to be carried by an independent engine or module. This modular hardware scheduling and software operating environment ensures the independence of each program instruction during runtime, preventing the entire data exchange link from paralyzing due to a single component failure, and reducing the overall system maintenance cost. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention; Figure 2 This is a flowchart of a data sharing and exchange method based on multi-source heterogeneous connectors according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the data source connector interface definition in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the working principle of the connector manager according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the data synchronization process according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the data format conversion process according to an embodiment of the present invention.

[0024] The components are as follows: 101. Unified Interface Layer; 102. Connector Management Module; 103. Connector Implementation Layer; 104. Configuration Management Module; 105. Data Synchronization Engine; 106. Format Conversion Engine; 107. Monitoring and Alarm Module. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, this embodiment of the invention provides a data sharing and exchange system based on multi-source heterogeneous connectors. This system serves as the physical carrier for executing multi-source data exchange and flow, and its overall architecture includes seven main functional modules and three auxiliary optimization modules.

[0027] Specifically, the main functional modules include a unified interface layer 101, a connector management module 102, a connector implementation layer 103, a configuration management module 104, a data synchronization engine 105, a format conversion engine 106, and a monitoring and alarm module 107.

[0028] The unified interface layer 101 is used to define the standard operating specifications of the underlying heterogeneous data sources; the connector management module 102 acts as the scheduling center and is responsible for the dynamic registration and loading of data source connectors; the connector implementation layer 103 contains physical adaptation implementations for various specific heterogeneous data sources; and the configuration management module 104 is used to receive and parse externally input data source access parameters.

[0029] The data synchronization engine 105 is used to coordinate and execute various data sharing and exchange tasks according to the configured synchronization mode. Specifically, the data synchronization engine 105 includes: a full synchronization unit, used to perform full table data queries and batch writes; an incremental synchronization unit, used to identify and synchronize changed data based on timestamps, incremental identifiers, or CDC (Change Data Capture) mechanisms; and a real-time synchronization unit, used to continuously monitor real-time change events of the data source and quickly push them to the target end.

[0030] The format conversion engine 106 is used to automatically convert between different data formats during data synchronization. To support various complex file structures, the format conversion engine 106 integrates a variety of dedicated converters, which may include any of the following: CSV (Comma-Separated Values) converter, XML (Extensible Markup Language) converter, Excel (workbook format) converter, Parquet (Apache Parquet columnar storage format) converter, and Avro (Apache Avro data serialization format) converter, to respectively achieve bidirectional conversion between the corresponding data format and the internal unified JSON (JavaScript Object Notation) format.

[0031] The monitoring and alarm module 107 is used to monitor the status indicators, performance indicators and latency indicators of the data synchronization process, and to trigger an early warning notification mechanism when an anomaly occurs.

[0032] Specifically, in terms of status indicators, a status anomaly is defined when runtime errors such as network disconnection, connection timeout, or authentication failure are captured; in terms of performance indicators, a performance anomaly is defined when the number of bytes throughput or synchronized record rows per unit time is lower than the system's preset minimum expected throughput threshold; and in terms of latency indicators, a latency anomaly is defined when the difference in end-to-end data transfer time exceeds the preset maximum tolerable time threshold.

[0033] The aforementioned early warning notification mechanism refers to the following: when the monitoring and alarm module 107 assesses that any of the above-mentioned anomalies are established, it generates an alarm message object containing the anomaly source node, error stack code, and excessive index value according to the pre-configured monitoring and alarm template 107. Then, by calling the enterprise message bus interface or Simple Mail Transfer Protocol (SMTP), the alarm message object is pushed to the operation and maintenance personnel's terminal in real time via email, SMS, or Webhook interface callback.

[0034] In addition, the system is equipped with auxiliary optimization modules, which may include a connection pool management module, a batch processing module, and a caching module. The connection pool management module is used to standardize the creation, reuse, and recycling of data source connections; the batch processing module is used to optimize batch read and write operations in scenarios with large amounts of data; and the caching module is used to temporarily store frequently accessed metadata and query results to improve overall operational efficiency.

[0035] Based on the above system architecture, this embodiment of the invention also provides a data sharing and exchange method based on multi-source heterogeneous connectors. This method uses each module within the system architecture as a physical carrier, and they operate collaboratively according to time and logical order, constructing a complete data sharing and exchange lifecycle workflow. For example... Figure 2 As shown, the specific execution steps are as follows: S1. Define a unified data source connector interface. The memory dictionary interface includes connection establishment, connection disconnection, connection testing, metadata acquisition, data query, and data synchronization operations. S2. Implement a connector manager to dynamically register and load data source connectors through the Service Provider Interface (SPI) mechanism. The memory dictionary connector manager maintains the mapping relationship between connector types and connector implementation classes. S3. Receive data source configuration information. The memory dictionary configuration information includes the data source type (e.g., the input value is a specific type identifier such as MYSQL, MONGODB, or KAFKA), connection parameters (specifically including the target host's IP address, communication port number, database name, login username and password credentials), and synchronization mode. S4. Obtain the corresponding connector instance from the connector manager based on the data source type; S5. Call the connection method of the connector instance to establish a connection with the underlying physical data source; S6. Extract source data records according to the synchronization mode, including: if the synchronization mode is a full synchronization mode, inject a streaming query instruction to establish a cursor and obtain source data records based on the cursor and store them in the data buffer; if the synchronization mode is an incremental synchronization mode, extract source data records based on the timestamp watermark mechanism or the incremental identifier mechanism built into the data table; if the synchronization mode is a real-time synchronization mode, deliver the event message objects obtained by listening to the system to a pre-allocated thread-safe queue in the system memory and extract them using a consumer thread. S7. During data synchronization, automatically convert the source format to the target format; S8. Monitor the data synchronization process and record the synchronization status, data volume, and synchronization delay indicators. S9. An alarm is triggered when the synchronization delay exceeds a preset threshold.

[0036] Through the complete closed loop of steps S1 to S9 described above, this embodiment of the invention breaks down the physical and protocol barriers between underlying data systems in a highly decoupled manner, transforming scattered data silos into freely flowing resources, thereby truly implementing a cross-business and cross-departmental data sharing and exchange mechanism at the system level. The underlying technical logic of the execution process of the aforementioned steps S1 to S9 will be elaborated in detail below, taking into account the various components within the system: During the system initialization and access preparation phase, step S1 is first executed by the unified interface layer 101. The unified interface layer 101 defines a standard data source connector interface at the code level, abstracting and encapsulating the differences in communication protocols and access logic among various heterogeneous data sources. This interface encapsulates the differences in communication protocols and access logic among various heterogeneous data sources (including relational databases, time-series databases, document databases, message queues, and object storage, etc.), providing a unified calling specification to the outside world.

[0037] Combination Figure 3 The diagram shown illustrates the definition of the data source connector interface. This standard data source connector interface is divided into six basic operations and strictly aligns with the standard input and output parameter specifications at the code level. Specifically, the interface includes the following standard operations: Connection establishment operation: connect(ConnectorConfig config), which establishes a physical connection with the underlying data source by receiving a connector configuration object that encapsulates information such as host and port as input parameters; Disconnect operation: disconnect(), no parameters, used to disconnect from the data source and release resources when the task is completed or a fatal exception is caught; Connection test operation: testConnection(), no input parameters, uses heartbeat mechanism to send probe messages to test the availability of the current communication link; Metadata retrieval operation: getMetadata(), no input parameters, after execution outputs metadata information such as the database table, field names and types of the data source; Data query operation: query(QueryRequest request), which receives a query request object encapsulating the table name and filter conditions and performs the underlying data query; Data synchronization operation: sync(SyncRequest request), which receives a synchronization request object that encapsulates the synchronization mode and routing information, and performs the specific data transfer operation.

[0038] Through the strict constraints of the above six standard operations, the unified interface layer 101 provides a standardized implementation template for the subsequent access of various heterogeneous data sources.

[0039] Once invoked, this operation performs specific data transfer actions and ultimately returns a synchronization result object to indicate the success of the task and the actual number of data rows affected. Through the input and output parameter constraints of the above six basic operations, the unified interface layer 101 provides a standardized implementation template for the subsequent access of various heterogeneous data sources.

[0040] Based on the standardized implementation template provided by the unified interface layer 101, the system executes step S2, implementing a connector manager through the connector management module 102 to complete the scheduling of underlying heterogeneous data source access components. For example... Figure 4 As shown, this connector manager relies on the Service Provider Interface (SPI) mechanism to dynamically register and load the underlying data source connectors. In practice, the system pre-configures a directory file named after the fully qualified name of the unified interface in the META-INF / services directory of the classpath. This file records, line by line, the fully qualified names of all concrete connector classes that have implemented the aforementioned unified interface specification.

[0041] When the system starts up or receives a component loading instruction, the connector management module 102 calls the underlying service loading component to read the configuration file. Based on the fully qualified names recorded in the file, the service loading component uses dynamic instantiation methods to instantiate each data source connector class in memory. After the instantiation process is complete, the connector management module 102 extracts the data source type identifier characters corresponding to each connector instance.

[0042] The aforementioned dynamic invocation of instantiation methods refers to the mechanism by which a program can dynamically obtain class information and manipulate the class's internal attributes and methods while the system is running. Specifically, in this step, the system reads the fully qualified class name string recorded in the configuration file. Without hardcoding the specific class name, it calls the underlying Java runtime environment's class loader to load the class into memory and calls native dynamic instantiation methods such as `newInstance()` of the `Class` object. This dynamically allocates space in memory and constructs a concrete linker instance that can be directly invoked, thereby receiving the operation instructions for this data exchange task.

[0043] After obtaining the identifiers of each data source type, the system constructs a data dictionary object based on a hash structure in memory, extracts the characters of each data source type identifier as key values, and uses the corresponding connector implementation class object as content values ​​to obtain a mapping registry.

[0044] A hash-based data dictionary object is specifically represented as a key-value mapping structure (e.g., HashMap) maintained in the system heap memory. This structure uses a hash algorithm to hash the input keys to map them to physical memory addresses. In this system, the extracted data source type identifier string (e.g., MySQL) is used as the unique key, and the corresponding value is the object reference of the connector implementation class in memory for that type. Through this hash structure, the system can directly locate and extract the corresponding connector implementation class component in O(1) time complexity when the configuration type identifier is obtained.

[0045] In this mapping registry, the dictionary's key is set to the extracted data source type identifier, and the dictionary's content value is set to the connector implementation class object corresponding to that identifier. Through the construction of this memory mapping table, a direct addressing path is established between the external data source type and the internal processing components.

[0046] Based on the workflow constructed using the configuration read and memory mapping table described above, the multi-source heterogeneous data sharing and exchange system has the ability to connect to new data sources without modifying existing system code logic. When there is a new heterogeneous data source access requirement, developers only need to write a new connector implementation class that conforms to the unified interface specification and append the fully qualified name of the new implementation class to the configuration file under the aforementioned META-INF / services directory. When the system executes the loading logic again, the service loading component will sequentially read the new configuration item and update it in the memory mapping registry. This loading mechanism allows the system to avoid recompiling and deploying the scheduling architecture code, thereby supporting subsequent processing modules in allocating instances of the newly connected components.

[0047] After the aforementioned memory mapping registry is constructed, all types of connector instances registered in the system are supported by the connector implementation layer 103. The connector implementation layer 103 strictly follows the operation specifications defined by the unified interface layer 101, and provides fourteen types of heterogeneous data source lower-level feature implementations under the standard interface by encapsulating the underlying communication protocols or driver components specific to heterogeneous data sources.

[0048] To address the access requirements of relational databases, the connector implementation layer 103 includes the RelationalConnector relational database connector, which supports MySQL, Oracle, and PostgreSQL data sources. This connector uses the JDBC (Java Database Connectivity) protocol for communication at its underlying level. When responding to operation commands defined by the unified interface, the connector calls the database metadata component built into the JDBC driver to obtain the database table structure, executes data query requests using the prepared statement component, and maps the read physical data records into a standard structure output through the result set component.

[0049] For time-series data processing scenarios, the connector implementation layer 103 provides the InfluxDBConnector, TimescaleDBConnector, and OpenTSDBConnector time-series database connectors. The InfluxDBConnector uses the official Java client package to send time-series data read / write and aggregation operation requests to the server. The TimescaleDBConnector reuses and extends the PostgreSQL JDBC driver protocol, providing underlying management commands for supertable structures and support for sending continuous aggregation query requests. The OpenTSDBConnector converts operations defined by a unified interface into HTTP API requests, sending read / write loads with time-series tags to the server.

[0050] For NoSQL database access, the connector implementation layer 103 includes the MongoDbConnector document database connector, the RedisConnector cache database connector, and the ElasticsearchConnector search engine connector. The MongoDbConnector document database connector utilizes MongoDB's Java driver component to convert query operations into aggregate pipeline query statements or basic document read / write commands. The RedisConnector cache database connector supports establishing a data exchange link with the Redis server through the connection client component, performing key-value pair access and message publish / subscribe operations. The ElasticsearchConnector search engine connector encapsulates the Elasticsearch client tool, receiving query request objects, parsing them, and forwarding them into underlying index structure management and full-text search requests.

[0051] In the integration with the big data computing and storage platform, connector implementation layer 103 is configured with the HdfsConnector distributed file system connector, the HiveConnector data warehouse connector, and the KafkaConnector message queue connector. The HdfsConnector calls the Hadoop file system interface program, mapping the byte stream reading operations of the underlying files to standard data retrieval operations. The HiveConnector data warehouse connector uses a dedicated Hive JDBC driver to connect to the server, responsible for concatenating and issuing HQL query statements to extract the computed wide table data. The KafkaConnector message queue connector encapsulates the Kafka Java client environment, enabling the production, delivery, and consumption subscription of message records under a specified topic.

[0052] For object storage scenarios involving unstructured files, connector implementation layer 103 incorporates the MinioConnector object storage connector and the ObjectStorageConnector Alibaba Cloud OSS object storage connector. These two specific connector implementations rely on the official MinIO and Alibaba Cloud Java Software Development Kits (SDKs) respectively to construct the corresponding underlying network communication links. When responding to data acquisition or synchronization operations, these two connectors utilize their respective underlying toolkit interfaces to upload and download file stream byte data and generate pre-signed links for temporary access.

[0053] After the various heterogeneous connector components at the underlying level are ready at the connector implementation layer, the system executes step S3, receiving and parsing the externally input data source configuration information through the configuration management module 104. This configuration information is uniformly formatted using JSON data format during network transmission and persistent storage. After receiving the externally input JSON text stream, the configuration management module 104 deserializes it and maps it into configuration entity objects required for internal system operation.

[0054] The JSON-formatted configuration information includes several standard fields used to define data source attributes and task scheduling rules. The data source type field uses character input, such as "MYSQL" or "INFLUXDB". The value of this field strictly corresponds to the dictionary key maintained by the connector management module 102 in the memory mapping registry, providing a basis for locating specific connector instances. The connection parameter field is represented as a nested JSON object, responsible for carrying the underlying network communication and authentication credentials. This nested object specifically records the target data source's host address, service port, requested database name, username, and password.

[0055] The configuration information also defines a synchronization mode field, which indicates the data extraction strategy for subsequent task execution. The value of the synchronization mode field is limited to three specified enumeration flags: a "FULL" full synchronization mode flag indicating the extraction of all existing source data; an "INCREMENTAL" incremental synchronization mode flag indicating the extraction of only changed data; and a "REALTIME" real-time synchronization mode flag indicating continuous reception of data streams via listening. In addition, for non-real-time scheduled tasks, the configuration information also sets a synchronization interval field. This field accepts an integer input, explicitly specifying minutes as the unit of measurement, to control the trigger cycle of the next data retrieval action. The underlying operations involved in the above configuration parsing process, such as JSON string lexical analysis, tree structure construction, and in-memory object property assignment, can be completed using a conventional JSON processing library.

[0056] After the configuration management module 104 completes the parsing and extraction of the above configuration parameters, the system then executes step S4 to allocate connector instances. Specifically, the system reads the data source type field identifier from the parsing result, uses this identifier as the query key, and initiates a retrieval in the memory mapping registry maintained by the connector management module 102. Through hash addressing logic, the system locates and extracts the corresponding connector implementation class object from the data dictionary of the registry. Based on the obtained implementation class object, the system dynamically instantiates a specific connector instance in memory using a dynamic call to the instantiation method to receive the operation instructions for this data exchange task.

[0057] After obtaining the corresponding connector instance, the system proceeds to step S5 to establish a communication link with the underlying physical data source. The system calls the connection establishment method in the connector instance, which follows a unified operating specification, and injects the credential information encapsulated in the connection parameter fields parsed in the previous steps as input parameters. This credential information specifically includes the host address, service port, database name, username, and password. Upon receiving these input parameters, the underlying protocol driver component of the connector initiates a network connection request to the corresponding port of the target host according to the network protocol specifications required by the target data source, and submits authentication credential information, thereby establishing a bidirectional data communication channel between the multi-source heterogeneous data sharing and exchange system and the underlying physical data source.

[0058] After network connection and authentication are completed, to confirm that the communication channel is in a healthy, readable, and writable state, the system immediately calls the connection test method of the connector instance. This test method sends a lightweight probe message to the physical data source at the underlying level and initiates a countdown mechanism to wait for a response. When the system receives a successful verification flag from the data source within the time window, it confirms that the physical communication link has been established, signifying that the system now has the basic network environment to perform subsequent data query and write operations. If a timeout is triggered or an authentication rejection instruction is received during this probe, the system will intercept subsequent execution nodes of the current data sharing and exchange task and throw out the corresponding exception log record to avoid invalid network requests occupying system resources for an extended period.

[0059] After the physical communication link is established and verified, the system enters the data synchronization processing stage, specifically step S6. (Combined with...) Figure 5 The data synchronization process shown involves the data synchronization engine 105 acting as the control center at the execution level. It intervenes and reads the configuration information entity object generated in the previous steps. By extracting the synchronization mode field from this entity object, the data synchronization engine 105 identifies the specific execution strategy of the current task. When the value of the synchronization mode field is identified as a full synchronization mode identifier, the data synchronization engine 105 initiates a comprehensive extraction operation on the existing data in the source data table.

[0060] When performing full data extraction, in order to cope with the system operation risks caused by loading massive amounts of data at once, the data synchronization engine 105 does not directly request all data records. Instead, it injects streaming query instructions to establish a cursor and retrieves the source data records based on the cursor and stores them in the data buffer.

[0061] The aforementioned cursor refers to a result set pointer structure built on the underlying physical data source server. It does not return the query results all at once, but rather remains at a specific data row position in the result set. A streaming query command refers to a request issued by the system to the physical data source driver, requesting data to be transmitted over the network in batches according to a specified data block size. Specifically, injecting a streaming query command to establish a cursor involves the database server maintaining the query session and establishing a cursor after the system issues the streaming query command. The system continuously requests the next batch of data from the server over the network, and the cursor moves downstream accordingly until all results are read. This method ensures that only a small amount of buffered data exists in the system memory at any given time, thus avoiding the risk of memory overflow caused by loading massive amounts of data at once.

[0062] When this connector instance issues a data query operation, it carries streaming read parameters. After receiving the query request with these parameters, the underlying physical data source keeps the query session open on the server side and establishes a cursor pointing to the data result set. The system then uses this cursor to gradually retrieve data rows in the form of a data stream, rather than transferring the entire result set to system memory all at once.

[0063] During the continuous reception of data streams from the source, the data synchronization engine 105 incorporates a strict batch control mechanism. The system allocates a collection space in memory as a data buffer and pre-sets a batch capacity threshold. During the data fetching phase, whenever a data record is read from the result set cursor, the system stores it in this data buffer and synchronously increments the internal data row counter.

[0064] When the counter reaches a preset batch capacity threshold, the data synchronization engine 105 temporarily suspends reading the source data stream. The system treats the data buffer that has reached the threshold as a complete data batch and initiates a transfer process to subsequent processing modules. After the batch of data is received by downstream nodes and leaves the current scope, the system performs a memory reclamation operation, clearing the current data buffer and the counter value, and then wakes up the suspended read task to continue pulling the next batch of data from the cursor position. Relying on this control mechanism of cursor reading combined with batch-based memory flushing, the data synchronization engine 105 limits the size of data remaining in the system heap memory at any given time, avoiding memory overflow problems that are prone to occur during large-scale data migration, and ensuring the stable operation of the full synchronization task.

[0065] When the data synchronization engine 105 detects that the synchronization mode field in the configuration information entity object is set to the incremental synchronization mode identifier, the task execution trajectory shifts to a periodic capture operation of the changed data from the source. To accurately identify the changed data records in the target data table, the data synchronization engine 105 is internally configured with three differentiated incremental data identification logics to adapt to the characteristics of different underlying physical data sources.

[0066] The first identification logic is based on a timestamp watermark mechanism. Under this mechanism, the system uses a local persistent storage component to record the highest timestamp parameter when the previous round of synchronization tasks was successfully completed, using this as a clearly defined watermark. When the incremental synchronization task of the current period is triggered, the data synchronization engine 105 reads this watermark parameter and uses it as a filtering condition to assemble into the query request object issued to the connector instance. The conditional filtering expression inside this request object is updated to limit the target time field to be greater than the watermark value. The underlying connector issues instructions based on this expression to extract newly added or modified records from the physical data source within that time period. After this round of data retrieval is completed, the data synchronization engine 105 extracts the maximum timestamp parameter from the currently retrieved data and overwrites the locally stored watermark record for use in the next round of scheduling.

[0067] The second identification logic is based on the incremental identifier mechanism built into the data table. When the business data source uses an auto-incrementing primary key ID or record version number field, the data synchronization engine 105 uses this identifier as the offset parameter for incremental tracking. The system also maintains the maximum primary key ID or highest version number parameter from the previous synchronization locally. When constructing the underlying query request, this offset parameter is used as a boundary condition to limit the read range, extracting all data rows from the source end whose identifier value is greater than this offset parameter. This mechanism avoids clock synchronization problems that may arise from relying on system time.

[0068] The third identification logic is based on a change data capture mechanism, which bypasses the field restrictions of the business table structure and directly listens to the underlying transaction logs. When this logic is used, the data synchronization engine 105 calls the underlying connector instance to establish a special log reading connection channel. The system disguises itself as a slave node of the physical database at the underlying level, sending log dump requests to the master node to continuously receive transaction log file records generated during database operation, such as receiving the binary log (Binlog) generated by relational database MySQL or the write-ahead log (WAL) generated by PostgreSQL. The connector instance performs structural parsing on the received log byte stream in memory, directly identifying the insert, update, and delete events occurring in the underlying physical data source, and encapsulates the extracted row-level change details into a standard structure for the next processing stage.

[0069] When the data synchronization engine 105 identifies the synchronization mode field as a real-time synchronization mode identifier by parsing the configuration information entity object, the task execution trajectory changes from the aforementioned active polling and fetching to an event-driven push mechanism based on long connections. In this mode, the data synchronization engine 105 allocates and starts a background resident thread in system memory. This resident thread is responsible for calling the corresponding connector instance to establish and maintain a persistent bidirectional long-connection communication channel with the underlying physical data source. The lifecycle of this resident thread is strongly bound to the current data sharing and exchange task's runtime, ensuring it remains in a ready state to receive data streams actively sent by the source at any time.

[0070] Based on the persistent long-connection communication channel constructed above, the resident thread registers a data change event listener with the physical data source server using the underlying connector instance. When data write, update, or delete operations occur in the underlying physical data source, the server's internal trigger or publishing mechanism will be activated, actively pushing an event message object encapsulating the change details to the listener via the long connection. After receiving the event message object, to avoid data backlog caused by network I / O blocking, the resident thread does not directly perform subsequent data transformation and write operations, but instead quickly delivers the received event message object to a pre-allocated thread-safe queue in system memory.

[0071] While data is being queued, the data synchronization engine 105 internally maintains a set of consumer threads in parallel. These consumer threads continuously monitor the capacity of the aforementioned thread-safe queue. When there are event message objects to be processed in the queue, the consumer threads immediately perform a dequeue operation to retrieve them. The retrieved event message objects are parsed and assembled into the standard data record structure defined by the aforementioned interface specification, and then handed over to the subsequent processing module for processing. Finally, the synchronous write method of the target connector instance is called to push the data to the target data source. This event-driven architecture, which maintains the connection through resident threads and combines in-memory queue buffering for decoupling, enables low-latency data transfer between heterogeneous systems, meeting the technical support requirements for high-timeliness business scenarios such as message queue subscription and distribution and real-time synchronization of cache databases.

[0072] After extracting data records from the source end through the aforementioned full synchronization, incremental synchronization, or real-time synchronization mechanisms, in order to eliminate data structure barriers between multi-source heterogeneous underlying physical data sources, the system synchronously executes step S7 before the data is formally written to the target end. For example... Figure 6 As shown, in this step, the format conversion engine 106 intervenes in the data transmission link inside the system and undertakes the task of automatic parsing and format alignment of heterogeneous data structures.

[0073] To avoid establishing complex point-to-point hard-coded conversion relationships between multiple heterogeneous data sources, the format conversion engine 106 constructs a bidirectional conversion pipeline in memory, using JSON data format as an intermediate state. When data flows into this pipeline, the format conversion engine 106 receives source data records from the source connector instance, converts these records into a unified JSON intermediate state object, and then reverse-parses and concatenates the unified JSON intermediate state object into a physical write format compatible with the target end. During the conversion to generate the unified JSON intermediate object, the field names of the source data table are used as keys, and the content of the specific data rows is used as the corresponding values, thus completing the forward conversion from heterogeneous data flow to the unified JSON intermediate state.

[0074] After constructing a unified JSON intermediate object in memory, the format conversion engine 106 continues to drive the data flow along the pipeline to the target end, thereby implementing a data sharing and exchange mechanism based on multi-source heterogeneous connectors at the system level. During this process, the format conversion engine 106 obtains the target data source type specified in the task configuration information, triggers the corresponding target-end connector instance, and reverse-parses and concatenates the aforementioned JSON intermediate object. Reverse parsing here refers to stripping the encapsulation structure of the intermediate object (e.g., JSON) to extract the original key names (field names), data types, and pure data values. Concatenation here refers to recombinating the extracted elements according to the syntax requirements of the target database. For example, if the target is configured as a relational database, the connector instance extracts the data values ​​through reverse parsing and then concatenates them into an SQL statement format containing "INSERT INTO..." syntax and parameterized arrays; if the target is configured as a document database, the parsed pure data values ​​are concatenated and assembled into a target-specific JSON document stream structure.

[0075] Through the decoupled conversion mechanism centered on the JSON structure described above, the system only needs to maintain bidirectional mapping rules from a single data source to the JSON format, without needing to handle direct format cross-referencing between pairs of data sources. This ensures structural alignment of any multimodal data during transmission. The abstract syntax tree parsing, node traversal, and character stream reassembly operations in the underlying JSON serialization and deserialization process can be implemented using conventional text processing packages.

[0076] Building upon the aforementioned processing of regular records in structured databases, the format conversion engine 106 further configures underlying mapping logic for the mutual conversion between various typical complex data formats and a unified JSON intermediate object, targeting unstructured files and complex big data formats contained in object storage or distributed file systems.

[0077] For handling comma-separated CSV files, the format conversion engine 106 integrates the OpenCSV underlying technology component library in memory. When the source data is in CSV format, the format conversion engine 106 extracts the first line of the file as the attribute field key using a streaming read method, and then breaks down subsequent lines into specific data values ​​according to the delimiter. The system concatenates the extracted key names and data values ​​in memory to construct a standard JSON intermediate object. When writing CSV format to the target end, the format conversion engine 106 iterates backward through the JSON intermediate object, extracts the key names to generate the file header, and arranges the attribute values ​​sequentially with the specified delimiter, assembling them into a text line structure compatible with the target end.

[0078] When faced with the Extensible Markup Language (XML) format, the format conversion engine 106 calls the Java architecture's XML binding JAXB underlying component library to perform structure parsing. The system reads the tree-like hierarchical structure of the XML file, mapping the names of node elements within the XML file to keys of JSON objects, mapping the text content enclosed within nodes to corresponding values ​​of JSON objects, and synchronously reorganizing the nested child nodes of the XML into a collection of nested objects within the JSON.

[0079] For processing Excel spreadsheet files, the system uses the Apache POI component to parse binary file blocks. The format conversion engine 106 extracts worksheet objects from the underlying workbook hierarchically, sequentially traversing and reading the data rows and cell physical structure within the worksheet. During parsing, the system identifies the underlying formats of numerical values, dates, and strings stored within cells, maps them to the corresponding header information, and converts them into a unified JSON intermediate object.

[0080] In big data columnar storage and serialization scenarios, for Parquet format files, the format conversion engine 106 relies on the corresponding Hadoop ecosystem reading components to extract the metadata definition stored at the end of the Parquet file blocks. Based on the column block distribution information in the metadata definition, the system flattens the columnar storage data block structure in memory, reassembles it, and restores it into a row-organized JSON format record. For Avro binary data format, the system loads the Avro schema definition file carried in the task configuration, uses this schema to deserialize the underlying binary byte stream in the transmission link, extracts the structured fields, and forward maps them to JSON intermediate objects. When writing Avro format to the target end, the format conversion engine 106 also relies on the Avro schema definition file, reverses the JSON intermediate objects, strictly performs type verification and binary serialization compression on the structured data according to the schema definition, and finally outputs a compliant Avro byte stream.

[0081] When the target is configured as a big data platform, the system performs reverse columnar encoding or binary serialization compression on the JSON intermediate object flowing from memory, based on the storage mode definition specified by the target, and then performs streaming write. For the basic dependency import rules of the aforementioned third-party technology component libraries and the common calling methods of input / output stream API functions, please refer to the relevant technical documentation for integration. Their standard code-level implementations are well-known technologies in this field and will not be elaborated upon here.

[0082] The aforementioned reverse columnar encoding refers to the system reversing the memory structure of JSON intermediate objects organized by rows for columnar storage targets such as Parquet and ClickHouse. It extracts the key-value pairs of the same attributes from multiple JSON objects (for example, extracting the age field value from one hundred rows of data), and physically gathers and concatenates them in memory into a structure that is stored continuously by columns, so as to improve the data compression rate and query aggregation efficiency of the target.

[0083] Binary serialization compression refers to the process where, for serialization formats such as Avro, the system loads the corresponding pattern definition file, removes redundant string keys from the JSON intermediate object, converts its pure data values ​​into a compact binary byte stream according to a preset strong type (such as integer or double-precision floating-point), and then calls compression algorithms such as Snappy or Gzip to compress the byte stream, ultimately forming a streaming write workload that can be efficiently stored and transferred by big data platforms.

[0084] During the extraction, format conversion, and target end writing of the aforementioned multi-source heterogeneous data, to ensure the stable operation of the entire data sharing and exchange link, the system synchronously executes step S8, using the monitoring and alarm module 107 to perform full-link monitoring of the data task's operation. The monitoring and alarm module 107 integrates a Micrometer metric acquisition technology component within the system. This component injects interception points into the upstream and downstream nodes of the connector call link and the memory format conversion pipeline, measuring and collecting multi-dimensional operational metrics of the flowing data tasks in a non-blocking asynchronous manner.

[0085] During this process, the monitoring and alarm module 107 focuses on calculating and recording three system-level indicators: synchronization status, data volume, and end-to-end latency. For the synchronization status indicator, the system determines its status by monitoring the lifecycle and exception events thrown by each scheduling thread within the data synchronization engine. When the synchronization thread is running normally, the status is recorded as "running"; when the system captures runtime exceptions such as network timeouts or authentication failures during the read or write phase, the status indicator immediately changes to "abnormal interruption."

[0086] Regarding the calculation of data volume metrics, the monitoring and alarm module 107 intervenes between the source end read action and the target end write action, using the Micrometer's built-in counter component to accumulate the number of data record rows passing through the data buffer. Simultaneously, the system counts the total number of bytes actually transmitted at the underlying byte stream processing node, thereby quantifying the actual data throughput of the current task per unit time.

[0087] For measuring end-to-end latency, the monitoring and alarm module 107 uses timestamp difference calculation logic. When the source connector instance reads the first data record of the current batch into the system memory, the system calls the system clock to obtain and record the initial timestamp at that moment. After the batch of data completes the aforementioned JSON intermediate state conversion and is successfully written to the underlying physical data source by the target connector instance, the system records the end timestamp of the completion moment. The monitoring and alarm module 107 calculates the difference between the end timestamp and the initial timestamp to obtain the actual end-to-end latency in milliseconds for the current data batch within the system from access to outflow.

[0088] Based on the continuous collection of the above-mentioned operational indicators, the system proceeds to step S9, initiating anomaly alarm actions based on threshold judgment rules. The system internally maintains a threshold configuration object, which sets the maximum allowed latency threshold for a single synchronization task and the expected minimum data throughput threshold. The monitoring and alarm module 107 continuously compares the currently collected indicator values ​​with the parameters in the threshold configuration object in the background.

[0089] When the comparison process finds that the actual end-to-end latency in milliseconds exceeds the preset maximum latency threshold, or the accumulated data volume per unit time falls below the minimum data throughput threshold, or the synchronization status indicator experiences an abnormal interruption, the system immediately triggers an alarm response program. The alarm response program assembles an alarm message object in memory containing the name of the affected data source, the specific deviation in the indicator value, and the time of the system anomaly. Based on the contact configuration parsed during the task initialization phase, the monitoring alarm module 107 sends the alarm message object to the administrator's email address by calling the basic email transmission protocol, or pushes it to the enterprise instant messaging software via a network interface. For the indicator registry management within the Micrometer component library, the underlying interception logic of the asynchronous interceptor, and the network request encapsulation of the external alarm interface, those skilled in the art can use conventional monitoring framework configurations and network communication libraries to implement these, which are well-known technologies in the field and will not be elaborated upon here.

[0090] To further support the continuous and stable operation of the aforementioned monitoring and alarm mechanisms, and to prevent resource exhaustion during long-term data transfer, the system incorporates auxiliary optimization strategies for performance and stability during the execution of multi-source heterogeneous data sharing and exchange tasks. These optimization strategies rely on the system's underlying architecture and specifically implement connection pool management at the communication layer and batch partitioning and caching control at the data processing layer.

[0091] In managing the underlying physical communication channels, the connector implementation layer 103 introduces a connection pool management mechanism when establishing physical data source connections, replacing the direct connection mode of creating a connection for each request. The system pre-allocates a connection object storage area in the heap memory as a connection pool. When the system needs to interact with the physical data source, it pre-detects whether there are any idle network connection instances in the connection object storage area. If so, the system directly extracts and reuses the underlying connection instance, thereby avoiding the latency caused by network handshake and authentication credential verification; if there are no idle instances and the current total number of connections has not reached the preset pool capacity limit, the system calls the connection establishment operation to create a new physical connection instance according to the previously parsed authentication credential information and includes it in the management of the storage area.

[0092] Meanwhile, to prevent idle connections in the pool from being passively disconnected by the underlying physical data source server due to prolonged periods without data transmission, a background scheduled task is mounted within the connection pool management mechanism. This scheduled task periodically extracts all connection instances in the pool, performs liveness checks on probe messages, and removes failed connections, thereby ensuring the high availability of the entire data exchange link. By continuously sending lightweight probe messages to the server, the system performs liveness checks on each connection instance. For failed connections that do not return a success flag within the timeout countdown, the system proactively removes them from the memory storage area and triggers compensation creation logic to fill the gap, thus ensuring the high availability of the entire data exchange link.

[0093] In the data flow and structure alignment phase, the data synchronization engine 105 and the format conversion engine 106 work together to perform batch processing and cache control. The data synchronization engine 105 deepens the aforementioned batch control logic, fully implementing a read / write batch splitting mechanism. During the target-side write phase, the system does not immediately initiate a network write request after processing each JSON intermediate object. Instead, it accumulates multiple JSON intermediate objects processed by the format conversion engine 106 in a memory buffer. When the accumulated number of data rows reaches the batch capacity threshold, the target-side connector instance assembles the batch data into a single batch write instruction containing multiple rows of data. Through this batch splitting and batch submission mechanism, the system significantly reduces the number of underlying network I / O interactions, alleviating the concurrent connection pressure on the target-side physical data source.

[0094] To support the high-throughput data flow, the system implements a metadata caching control strategy in the format conversion engine 106. The system stores the metadata result objects returned by the initial metadata retrieval operations in a local memory dictionary. When the format conversion engine 106 reassembles the source data records into JSON format, it directly retrieves the field names and data type mappings of the target table from this memory dictionary, thus eliminating repeated query requests to the underlying physical data source system tables. To handle potential dynamic changes to the underlying table structure, the system configures a lifetime parameter for this memory dictionary based on a system-preset empirical constant. When the cached object's residency time reaches the set lifetime limit, the system automatically performs memory cleanup and re-triggers the metadata retrieval operation when the next data parsing request arrives. The concurrent acquisition lock mechanism within the connection pool, the instruction concatenation rules for underlying batch write operations, and the lifetime-based cache eviction mechanism can be implemented using conventional basic utility libraries.

[0095] Specific application examples: To verify the actual operating effect of the aforementioned data sharing and exchange method and system based on multi-source heterogeneous connectors, this paper will describe it in conjunction with specific business application embodiments.

[0096] In applications involving heterogeneous synchronization between relational databases and time-series databases, the system needs to continuously synchronize population change data from the source relational database to the target time-series database to support population flow trend analysis. Prior to this task, the system defined standard operating procedures through a unified interface layer, and the connector management module used the Service Provider Interface (SPI) mechanism to dynamically register and load connectors for each data source (corresponding to steps S1 to S2). The system configuration management module receives and parses the externally input JSON-formatted data source configuration information (corresponding to step S3). From this configuration information, the system extracts the identifier representing the source data source type as a relational database and the identifier representing the target type as a time-series database. It also obtains the synchronization mode as incremental synchronization mode and parses the execution interval parameter and the underlying credential authentication connection parameters for both parties.

[0097] Based on the data source type parsed above, the system connector management module retrieves the corresponding relational database connector instance and time-series database connector instance from the locally maintained mapping registry (corresponding to step S4). After retrieving the instances, the system calls the connection establishment operations within these two connector instances, following standard specifications, and injects the aforementioned credential authentication connection parameters to establish a bidirectional network communication link with the underlying physical MySQL master database and InfluxDB server (corresponding to step S5).

[0098] Once the network environment is ready, the data synchronization engine triggers a data extraction operation based on the incremental synchronization mode command in the configuration information (corresponding to step S6). In this scenario, the system adopts a timestamp-based incremental identification logic, reads the maximum update timestamp of the previous synchronization of the locally persisted records, and sends a query command with filtering conditions to the underlying MySQL database to accurately extract incremental population data records whose update time is greater than that timestamp. During this data extraction and transfer stage, the data format conversion engine synchronously intervenes in the transmission link (corresponding to step S7). This engine extracts the relational row-level data records returned by the query, combines them with the metadata mapping relationship obtained in memory, and maps the two-dimensional table structure into a unified JSON intermediate object. The engine then passes this JSON intermediate object to the target connector instance, which reverse-parses and concatenates it into an InfluxDB-compatible underlying time-series protocol write format, completing the automatic alignment and batch writing of multi-source heterogeneous formats.

[0099] Throughout the entire end-to-end data transmission cycle, the monitoring and alarm module continuously records the synchronization status, actual data throughput, and end-to-end latency metrics for each batch of data (corresponding to step S8). When the background comparison program detects that the actual end-to-end synchronization latency of the current batch circulating within the system exceeds the preset maximum threshold of five minutes, the module immediately generates an alarm message object containing the abnormal node and pushes it to the system administrator (corresponding to step S9). Through the above-mentioned complete business-level closed loop, automated data exchange between heterogeneous platforms is achieved.

[0100] In the implementation scenario verifying the effectiveness of highly scalable technology, when faced with the business side's requirement to integrate a new ClickHouse columnar database, the system fully demonstrated its zero-code intrusion extension capability based on the service provider interface. For this type of extension, developers do not need to modify any system scheduling or parsing logic within the unified interface layer, data synchronization engine, or format conversion engine. Developers only need to write a new connector implementation class for this columnar database, based on the aforementioned unified data source connector interface specification. Within this implementation class, the underlying execution code is written to handle standard operations such as loading the underlying JDBC communication protocol, verifying network connections, and querying columnar data.

[0101] After completing the implementation class, the developers write its fully qualified name as a configuration item into the Service Provider Interface (SPI) dedicated configuration file under the system classpath. When the system executes the next startup or component loading instruction, the underlying service loading component of the connector management module will sequentially read the configuration file, dynamically instantiate the newly added connector class in memory using reflection technology, and register it in the dictionary mapping relationship. When subsequent data synchronization tasks are configured to carry the newly added data source identifier, the system can directly schedule the connector instance to complete the physical connection establishment and data extraction. The loading of the underlying ClickHouse-specific JDBC driver package, the string concatenation of basic SQL query statements, and the general network request delivery logic can be routinely called by those skilled in the art according to the official database development documentation; these are well-known technologies in the field and will not be elaborated upon here.

[0102] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A data sharing and exchange method based on multi-source heterogeneous connectors, characterized in that, include: S1. Dynamically register and load data source connectors through the Service Provider Interface (SPI) mechanism, and maintain a registry that maps data source types to connector implementation classes; S2. Receive data source configuration information including the data source type, connection parameters, and synchronization mode; S3. Obtain a connector instance from the mapping registry according to the data source type, and establish a connection with the underlying physical data source according to the connection parameters; S4. Extract source data records according to the synchronization mode; S5. The source data record is converted into a unified intermediate object, and the unified intermediate object is reverse parsed to extract the structured fields and values. The structured fields and values ​​are then assembled into a physical write format compatible with the target according to the protocol syntax of the target.

2. The data sharing and exchange method based on multi-source heterogeneous connectors according to claim 1, characterized in that, In S4, the source data record is extracted according to the synchronization mode, including: If the synchronization mode is a full synchronization mode identifier, inject a streaming query instruction to establish a cursor and obtain the source data record based on the cursor and store it in the data buffer. If the synchronization mode is an incremental synchronization mode identifier, the source data record is extracted based on the timestamp watermark mechanism or the incremental identifier mechanism built into the data table. If the synchronization mode is identified as a real-time synchronization mode, the event message object acquired by the listener is delivered to a pre-allocated thread-safe queue in the system memory and extracted using a consumer thread.

3. The data sharing and exchange method based on multi-source heterogeneous connectors according to claim 1, characterized in that, The unified intermediate state object includes a unified JSON intermediate state object, and step S5 includes: In the process of generating the unified JSON intermediate object, the field names of the source data table are used as keys, and the content of the specific data row is used as the corresponding values ​​to construct and generate the unified JSON intermediate object; Traverse the key-value pairs inside the unified JSON intermediate object, extract the data type and pure data value of the target attribute as the reverse parsing result, perform reverse columnar encoding or binary serialization compression on the reverse parsing result to generate the underlying byte stream, and assemble and splice the underlying byte stream or the corresponding pure data value into the physical writing format according to the communication protocol syntax of the target end.

4. The data sharing and exchange method based on multi-source heterogeneous connectors according to claim 1, characterized in that, In S1, the registry for maintaining the mapping relationship between data source types and connector implementation classes includes: The system runtime environment's class loading component is invoked to read the configuration file recording the fully qualified names of each class in the classpath, and the instantiation method is dynamically invoked to instantiate each data source connector class in memory; A data dictionary object based on a hash structure is constructed in memory. The identifier characters of each data source type are extracted as keys, and the corresponding connector implementation class objects are used as content values ​​to obtain the mapping relationship registry.

5. The data sharing and exchange method based on multi-source heterogeneous connectors according to claim 1, characterized in that, In S3, after establishing the connection with the underlying physical data source, the following is also included: During data synchronization, an interception point is injected to measure the end-to-end latency indicator. When the end-to-end latency indicator exceeds the preset maximum latency time threshold, an alarm message object is assembled and an alarm response program is triggered. A connection object storage area is pre-allocated in the heap memory as a connection pool. All connection instances in the connection pool are extracted to perform probe message liveness status detection and remove failed connections.

6. A data sharing and exchange system based on multi-source heterogeneous connectors, applied to the data sharing and exchange method based on multi-source heterogeneous connectors as described in any one of claims 1 to 5, characterized in that, include: The connector management module (102) is used to dynamically register and load data source connectors through the service provider interface SPI mechanism and maintain a registry of the mapping relationship between data source types and connector implementation classes. The configuration management module (104) is used to receive data source configuration information including the data source type, connection parameters and synchronization mode; The connector implementation layer (103) is used to obtain a connector instance from the mapping registry according to the data source type and establish a connection with the underlying physical data source according to the connection parameters. A data synchronization engine (105) is used to extract source data records according to the synchronization mode; The format conversion engine (106) is used to convert the source data records into a unified intermediate object and reverse parse and concatenate them into a physical write format compatible with the target end.

7. The data sharing and exchange system based on multi-source heterogeneous connectors according to claim 6, characterized in that, It also includes a unified interface layer (101) that defines the standard data source connector interface, the unified interface layer (101) including the following interfaces: A connection establishment operation that receives a connector configuration object encapsulating host and port information as input parameters to establish a physical connection. Disconnect from the data source and release resources; Send probe messages to test the availability of the current communication link; Metadata retrieval operations that output metadata information such as database tables, field names, and types of the data source; The data query operation that receives the query request object and performs the underlying data query; Data synchronization operations that perform specific data transfer actions.

8. The data sharing and exchange system based on multi-source heterogeneous connectors according to claim 6, characterized in that, The data synchronization engine (105) includes: The full synchronization unit is used to perform full table data queries and batch writes. Incremental synchronization unit, used to identify and synchronize changed data based on timestamp, incremental identifier or change data capture mechanism; The real-time synchronization unit is used to continuously monitor real-time change events of the data source and quickly push them to the target end.

9. The data sharing and exchange system based on multi-source heterogeneous connectors according to claim 6, characterized in that, The specific data source connectors included in the connector implementation layer (103) are as follows: Relational database connectors, time-series database connectors, document database connectors, cache database connectors, search engine connectors, distributed file system connectors, data warehouse connectors, message queue connectors, and object storage connectors.

10. The data sharing and exchange system based on multi-source heterogeneous connectors according to claim 7, characterized in that, The format conversion engine (106) is equipped with a memory dictionary; The format conversion engine (106) is used to transfer the metadata result object returned by the metadata acquisition operation to the local memory dictionary of the system, and extract the field name and data type mapping relationship from the memory dictionary during reorganization; the memory dictionary is configured with a lifetime parameter based on the system preset empirical constant, and automatically performs memory cleanup when the residence time of the cached object reaches the set lifetime limit.