Heterogeneous data source synchronization method, device and system for general goods vehicle

Abstract

The invention discloses a heterogeneous data source synchronization method, device and system for a general goods vehicle. The method comprises the following steps: receiving synchronization task configuration of general goods vehicle data submitted by a user; analyzing the synchronization task configuration through a Master node, and dividing to-be-synchronized data based on a fragmentation strategy in the synchronization parameters to generate a plurality of data fragments; dynamically allocating the data fragments to the corresponding Worker nodes according to the real-time load states of the Worker nodes, and thermally loading a data source plug-in from a plug-in warehouse through the Worker nodes so as to execute an allocated data fragment task; on the Worker node, starting an execution engine by using the loaded data source plug-in to process data synchronization; wherein the execution engine concurrently reads to-be-synchronized data corresponding to the data fragments from the data source in batches through a Reader thread group, and transmits the to-be-synchronized data to a Writer thread group through a zero-copy memory channel, and the Writer thread group writes the to-be-synchronized data into the target source in batches.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a method and apparatus for synchronizing heterogeneous data sources for general freight vehicles. Background Technology

[0002] In the field of smart logistics and transportation, the data generated by general freight vehicles during operation exhibits typical characteristics of being multi-source, heterogeneous, and massive. Vehicle GPS positioning data, engine operating status, fuel consumption sensor readings, cargo temperature and humidity monitoring data, transportation business order data, and vehicle diagnostic logs are stored in different types of heterogeneous data sources, such as MySQL business databases, HDFS distributed file systems, and Kafka message queues. These data sources differ significantly in connection protocols, data formats, access interfaces, and transaction mechanisms, forming a severe data silo phenomenon. Traditional data synchronization tools such as Kettle and Sqoop require writing a large amount of customized code for each combination of data sources when facing this complex heterogeneous environment, resulting in low development efficiency and high maintenance costs. Furthermore, existing synchronization solutions generally adopt single-threaded or simple multi-threaded architectures, which cannot fully utilize modern hardware resources such as multi-core CPUs and large memory, facing serious performance bottlenecks when processing massive amounts of vehicle data and failing to meet the real-time data requirements of logistics and transportation scenarios. In terms of reliability, due to factors such as network jitter caused by vehicle movement and node failures in distributed environments, traditional solutions lack effective fault recovery mechanisms and breakpoint resumption capabilities, making it difficult to guarantee data consistency. In addition, the system lacks scalability and cannot flexibly adapt to the surge in data volume and access requirements of new data sources brought about by the rapid growth of logistics business. Resource scheduling is inflexible, operation and maintenance monitoring capabilities are weak, and there is a lack of fine-grained monitoring and intelligent alarm mechanisms for task execution status, system resource usage, etc. Summary of the Invention

[0003] In view of this, the purpose of this invention is to propose a method, device, system and computer program product for synchronizing heterogeneous data sources for general freight vehicles, aiming to solve the technical problems such as performance bottlenecks, insufficient reliability and poor scalability faced by existing synchronization schemes when processing multi-source heterogeneous data in logistics transportation.

[0004] To achieve the above objectives, the present invention provides a method for synchronizing heterogeneous data sources for general freight vehicles, the method comprising:

[0005] The synchronization task configuration for receiving general freight vehicle data submitted by users includes defining the data source, target source, and synchronization parameters.

[0006] The Master node parses the synchronization task configuration and divides the data to be synchronized based on the sharding strategy in the synchronization parameters, generating multiple data shards.

[0007] Based on the real-time load status of the Worker nodes, the data shards are dynamically allocated to the corresponding Worker nodes, and the Worker nodes hot-load data source plugins that match the data source and the target source from the plugin repository to execute the allocated data sharding tasks.

[0008] On the Worker node, the execution engine is started using the loaded data source plugin to handle data synchronization; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

[0009] Preferably, the method further includes:

[0010] During the process of the Writer thread group writing to the target source in batches, the synchronization position information of each data shard is persisted to the distributed storage module;

[0011] When a worker node failure or network interruption is detected, the master node redistributes the incomplete data fragments to idle worker nodes based on the synchronization location information, and resumes data synchronization from the corresponding breakpoint.

[0012] Preferably, the method further includes:

[0013] The system collects monitoring metrics in real time, including throughput and resource utilization of Worker nodes. When an anomaly is detected in the monitoring metrics, an alarm is triggered.

[0014] Preferably, the step of parsing the synchronization task configuration through the Master node and dividing the data to be synchronized based on the sharding strategy in the synchronization parameters to generate multiple data shards includes:

[0015] Based on the data primary key specified in the synchronization parameters, the data to be synchronized is divided according to the numerical range of the data primary key to obtain multiple consecutive and non-overlapping data fragments.

[0016] Preferably, the step of dynamically allocating the data shards to the corresponding Worker nodes based on the real-time load status of the Worker nodes includes:

[0017] ZooKeeper monitors the heartbeats and resource utilization reported by each Worker node, and allocates the data shards to Worker nodes with low load based on resource utilization.

[0018] Preferably, the step of hot-loading the data source plugin that matches the data source and the target source from the plugin repository via the Worker node includes:

[0019] According to the synchronization task configuration, the corresponding standardized data source plugin is obtained from the plugin repository through the Worker node, and the obtained data source plugin is dynamically loaded and initialized on the Worker node.

[0020] Preferably, the step of concurrently reading the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and transferring the data to be synchronized to the Writer thread group through a zero-copy memory channel, and having the Writer thread group write the data to the target source in batches, includes:

[0021] A bounded blocking queue is established between the Reader thread group and the Writer thread group as the zero-copy memory channel. The binary transmission of the data to be synchronized is realized through the zero-copy memory channel. The Reader thread group stores the data to be synchronized in batches into the bounded blocking queue in the form of binary data. The Writer thread group retrieves the binary data from the bounded blocking queue and writes it in batches to the target source.

[0022] To achieve the above objectives, the present invention also provides a heterogeneous data source synchronization device for general freight vehicles, the device comprising:

[0023] The task receiving unit is used to receive the synchronization task configuration of general freight vehicle data submitted by the user. The synchronization task configuration includes defining the data source, the target source, and the synchronization parameters.

[0024] The data partitioning unit is used to parse the synchronization task configuration through the Master node and partition the data to be synchronized based on the partitioning strategy in the synchronization parameters to generate multiple data partitions.

[0025] The allocation unit is used to dynamically allocate the data shards to the corresponding Worker nodes according to the real-time load status of the Worker nodes, and to hot-load the data source plugins that match the data source and the target source from the plugin repository through the Worker nodes.

[0026] The synchronization unit is used on the Worker node to start the execution engine to process data synchronization using the loaded data source plugin; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

[0027] To achieve the above objectives, the present invention also proposes a heterogeneous data source synchronization system, the system comprising:

[0028] The user console is used to receive synchronization task configurations submitted by users via a web console or API, and send the synchronization task configurations to the unified scheduling center.

[0029] The unified scheduling center includes a Master node and Worker nodes. The Master node is used to parse the synchronization task configuration, generate multiple data shards based on the sharding strategy, and dynamically allocate data shards to Worker nodes according to the real-time load status of the Worker nodes. The Worker nodes are used to receive data shards and hot-load the data source plugins corresponding to the data source and target source from the plugin repository to execute the allocated data sharding tasks.

[0030] The plugin repository stores standardized data source plugins and provides hot-reloading support to Worker nodes.

[0031] The execution engine, deployed in the Worker nodes, interacts with the plugin repository and distributed storage module. It includes a dynamic concurrency controller, a pipeline processor, and a breakpoint resumption manager. The dynamic concurrency controller allocates threads based on the amount of data in each data shard. The pipeline processor reads the data to be synchronized from the data source in batches from the data source via the Reader thread group, transmits the data to be synchronized to the Writer thread group via a zero-copy memory channel, and then writes it to the target source in batches by the Writer thread group. The breakpoint resumption manager persists data location information to the distributed storage module and resumes synchronization from the breakpoint position based on the data location information when a fault is detected.

[0032] The distributed storage module is used to store the data location information and data fragments persisted by the breakpoint resume manager;

[0033] The monitoring center is used to collect monitoring indicators such as throughput and resource utilization during the data synchronization process in real time, analyze the collected monitoring indicators, and trigger alarms when performance bottlenecks or data anomalies are detected.

[0034] To achieve the above objectives, the present invention also proposes a heterogeneous data source synchronization device for general freight vehicles, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a heterogeneous data source synchronization method for general freight vehicles as described in the above embodiments.

[0035] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a heterogeneous data source synchronization method for general freight vehicles as described in the above embodiments.

[0036] To achieve the above objectives, the present invention also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of a heterogeneous data source synchronization method for general freight vehicles as described in the above embodiments.

[0037] Beneficial effects:

[0038] The above solution improves task configuration efficiency by parsing task configurations and generating data shards based on sharding strategies through the Master node. Dynamic allocation based on the real-time load status of Worker nodes, combined with a hot-loading mechanism from the plugin repository, enables the system to intelligently adapt to various heterogeneous data sources, effectively breaking down data silos and reducing the development and maintenance costs of multi-source adaptation. Furthermore, data source expansion for new vehicle models or equipment requires no modification to the core code; simply loading the corresponding standardized plugin from the plugin repository completes the integration. The execution engine employs a multi-level parallel pipeline architecture with concurrent batch reading by Reader thread groups, zero-copy memory channel transmission, and batch writing by Writer thread groups. This fully utilizes multi-core CPUs and high-speed network resources, significantly improving the throughput performance of massive data synchronization and meeting the real-time or near-real-time synchronization requirements for general freight vehicle data. The distributed Master-Worker architecture supports seamless horizontal scaling, allowing elastic expansion to more node clusters as business grows, significantly improving resource utilization and achieving efficient heterogeneous data synchronization in general freight vehicle data scenarios.

[0039] By persisting synchronization location information to distributed storage and resuming synchronization based on the breakpoint location in case of failure, precise breakpoint resume functionality is achieved. This avoids bandwidth waste and data duplication caused by full retransmission, effectively solving the problem of data duplication or loss due to network jitter or node failure. In other words, the breakpoint resume mechanism based on distributed storage fundamentally ensures the reliability of data synchronization for general freight vehicles in mobile scenarios. By collecting throughput and resource utilization metrics in real time and triggering alarms, a full-link monitoring system is established, enabling maintenance personnel to quickly identify system bottlenecks or anomalies, significantly reducing maintenance complexity and improving maintenance efficiency.

[0040] By limiting the sharding strategy to a range based on the primary key values, continuous and non-overlapping data shards are generated. This enables intelligent load balancing and efficient parallel processing tailored to the characteristics of large-scale general freight vehicle data. It automatically balances the data volume of each shard based on the actual data distribution characteristics, avoiding data skew and ensuring balanced load across worker nodes, fully leveraging the parallel processing advantages of a distributed architecture. This is particularly suitable for large-scale data scenarios with natural sequence characteristics, such as general freight vehicle trajectory data, effectively improving the efficiency and stability of sharding processing. Furthermore, shard metadata can be persistently used for fault tracing and performance analysis, providing precise data for operational optimization.

[0041] By using ZooKeeper to monitor the status of Worker nodes in real time and prioritizing the allocation of shards to low-load nodes, dynamic load balancing is achieved, improving cluster resource utilization and the intelligence of task scheduling. This allows the system to automatically adapt to node performance fluctuations and traffic changes, avoiding single-point overload and improving resource utilization. Simultaneously, the standardized plugin hot-loading mechanism allows new plugins to be pulled from the repository without modifying the core code or restarting the service when adding new data source types, greatly enhancing the system's scalability and flexibility.

[0042] The zero-copy memory channel design based on bounded blocking queues fundamentally eliminates the performance overhead of frequent serialization / deserialization of data objects in traditional ETL architectures.

[0043] By specifically defining the implementation method of zero-copy memory channels and employing bounded blocking queues for binary data transmission, the performance overhead of data serialization / deserialization in traditional ETL architectures is avoided, significantly reducing CPU usage and memory copy counts. Simultaneously, the bounded queue design prevents memory overflow risks, ensuring data transmission stability. This maximizes data transmission efficiency and resolves the performance bottleneck issues in traditional synchronization schemes. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a heterogeneous data source synchronization method for general freight vehicles, provided as an embodiment of the present invention.

[0046] Figure 2This is a schematic diagram of the architecture of a heterogeneous data source synchronization system provided in an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of distributed task scheduling provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the Reader-Writer processing flow provided in an embodiment of the present invention.

[0049] Figure 5 This is a flowchart illustrating a breakpoint resume mechanism provided in an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram of a heterogeneous data source synchronization device for general freight vehicles, provided in an embodiment of the present invention.

[0051] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The present invention will be described in detail below with reference to the embodiments.

[0054] Reference Figure 1 The diagram shown is a flowchart illustrating a heterogeneous data source synchronization method for general freight vehicles according to an embodiment of the present invention.

[0055] In this embodiment, the method is implemented based on a heterogeneous data source synchronization system; wherein, the system includes:

[0056] The user console is used to receive synchronization task configurations submitted by users via a web console or API, and send the synchronization task configurations to the unified scheduling center.

[0057] The unified scheduling center includes a Master node and Worker nodes. The Master node is used to parse the synchronization task configuration, generate multiple data shards based on the sharding strategy, and dynamically allocate data shards to Worker nodes according to the real-time load status of the Worker nodes. The Worker nodes are used to receive data shards and hot-load the data source plugins corresponding to the data source and target source from the plugin repository to execute the allocated data sharding tasks.

[0058] The plugin repository stores standardized data source plugins and provides hot-reloading support to Worker nodes.

[0059] The execution engine, deployed in the Worker nodes, interacts with the plugin repository and distributed storage module. It includes a dynamic concurrency controller, a pipeline processor, and a breakpoint resumption manager. The dynamic concurrency controller allocates threads based on the amount of data in each data shard. The pipeline processor reads the data to be synchronized from the data source in batches from the data source via the Reader thread group, transmits the data to be synchronized to the Writer thread group via a zero-copy memory channel, and then writes it to the target source in batches by the Writer thread group. The breakpoint resumption manager persists data location information to the distributed storage module and resumes synchronization from the breakpoint position based on the data location information when a fault is detected.

[0060] The distributed storage module is used to store the data location information and data fragments persisted by the breakpoint resume manager;

[0061] The monitoring center is used to collect monitoring indicators such as throughput and resource utilization during the data synchronization process in real time, analyze the collected monitoring indicators, and trigger alarms when performance bottlenecks or data anomalies are detected.

[0062] Reference Figure 2As shown in the diagram. In this embodiment, the system includes a user console, a unified scheduling center (including Master nodes and Worker nodes), a plugin repository, an execution engine, a distributed storage module, and a monitoring center. The user console receives data synchronization task configurations submitted by users in JSON format via a web interface or API and transmits these configurations to the unified scheduling center. Upon receiving the task configuration, the Master node of the unified scheduling center first parses it, then divides the data to be synchronized into multiple data shards based on the sharding strategy in the task configuration. Simultaneously, it uses the ZooKeeper coordination service to monitor the resource utilization of each Worker node in real time and dynamically allocates the data shards to Worker nodes with lower loads based on the load balancing of the Worker nodes. After receiving the allocated data shards, the Worker nodes hot-load standardized plugins (such as the MySQLReader plugin and HDFS Writer plugin) corresponding to the data source and target source defined in the task configuration from the plugin repository, adapting to heterogeneous data sources without requiring a restart. The execution engine is deployed in Worker nodes, where the dynamic concurrency controller automatically allocates processing threads based on the size of the data shards (i.e., allocates the number of Reader and Writer threads according to the amount of data in each shard). The pipeline processor reads data in batches from the data source through the Reader thread group, and directly transmits the data to the Writer thread group via a zero-copy memory channel (implemented using a bounded blocking queue) for batch writing to the target source, avoiding serialization overhead. During this process, the breakpoint resume manager periodically persists the synchronization position information of each data shard to a distributed storage module (such as Redis or HBase). When a node failure or network interruption is detected, data synchronization can be accurately resumed from the breakpoint, ensuring data consistency. The monitoring center collects throughput, resource utilization, and data consistency metrics across the entire chain in real time, and analyzes anomalies (such as Writer blocking or data discrepancies) through a rule engine. When performance bottlenecks or data anomalies are detected, multi-level alarms are triggered immediately. Through the collaborative work of the above modules, the system achieves high-performance, highly reliable heterogeneous data synchronization in the big data scenario of general freight vehicles.

[0063] Reference Figure 1 As shown, the method includes:

[0064] S11, Receive the synchronization task configuration of general freight vehicle data submitted by the user. The synchronization task configuration includes defining the data source, target source and synchronization parameters.

[0065] In this embodiment, the system's user console receives the user-submitted general freight vehicle data synchronization task configuration, which is transmitted in JSON format via a web interface or API. This configuration includes the data source, target source (data source / target source such as MySQL, Oracle, HDFS, HBase, Kafka, FTP, etc.), and synchronization parameters (such as sharding strategy, data transformation rules, batch size settings, etc.). The sharding strategy may be set to divide data based on vehicle ID primary key range or timestamp. The data transformation rules may define field mapping, data filtering, encryption, or format conversion logic. The batch size parameter controls the processing granularity of the Reader and Writer thread groups. After receiving the task configuration, the user console first performs syntax validation and parameter integrity checks to ensure the configuration conforms to the preset JSON schema specification. Then, the parsed configuration information is transmitted to the Master node of the unified scheduling center to initiate the subsequent data sharding and task scheduling process. The entire process is implemented through a standardized interface, simplifying user operations and adapting to the multi-source heterogeneous scenario of general freight vehicle big data, providing an accurate configuration foundation for subsequent data synchronization operations.

[0066] S12, the Master node parses the synchronization task configuration and divides the data to be synchronized based on the sharding strategy in the synchronization parameters to generate multiple data shards.

[0067] Furthermore, in step S12, the synchronization task configuration is parsed by the Master node, and the data to be synchronized is divided based on the sharding strategy in the synchronization parameters to generate multiple data shards, including:

[0068] Based on the data primary key specified in the synchronization parameters, the data to be synchronized is divided according to the numerical range of the data primary key to obtain multiple consecutive and non-overlapping data fragments.

[0069] Reference Figure 3As shown, in this embodiment, after receiving the task configuration from the user console, the Master node first parses the JSON-formatted configuration content to identify the defined data source connection information, target source configuration, and synchronization parameters. Based on the sharding strategy explicitly specified in the synchronization parameters, when the configuration requires partitioning by data primary key, the Master node obtains the primary key distribution of the source data table through the database metadata interface and automatically calculates the numerical range boundary of each data shard based on the preset number of shards and the total amount of data. For example, for a vehicle trajectory data table, if the primary key is vehicle ID and the numerical range is 1-10000, when the number of shards is set to 4, the system will generate four consecutive and non-overlapping data shards: Shard1 (Vehicle ID 1-2500), Shard2 (Vehicle ID 2501-5000), Shard3 (Vehicle ID 5001-7500), and Shard4 (Vehicle ID 7501-10000). Each data shard contains a complete metadata description, including shard ID, starting primary key value, ending primary key value, estimated data volume, and corresponding data source information. All generated data shards are stored in a sharding scheduling queue, awaiting subsequent allocation to Worker nodes for execution. The entire process is coordinated by ZooKeeper to ensure the atomicity and consistency of sharding computation. This adaptive sharding strategy based on primary key range dynamically adjusts the sharding granularity according to data distribution characteristics, effectively guaranteeing the uniformity and integrity of data partitioning. Furthermore, metadata sampling avoids full table scans, improving sharding efficiency.

[0070] S13, based on the real-time load status of the Worker nodes, dynamically allocate the data shards to the corresponding Worker nodes, and hot-load the data source plugins that match the data source and the target source from the plugin repository through the Worker nodes to execute the allocated data sharding tasks.

[0071] Furthermore, in step S13, the step of dynamically allocating the data shards to the corresponding Worker nodes based on the real-time load status of the Worker nodes includes:

[0072] ZooKeeper monitors the heartbeats and resource utilization reported by each Worker node, and allocates the data shards to Worker nodes with low load based on resource utilization.

[0073] Furthermore, in step S13, the hot-loading of data source plugins matching the data source and the target source from the plugin repository via Worker nodes includes:

[0074] According to the synchronization task configuration, the corresponding standardized data source plugin is obtained from the plugin repository through the Worker node, and the obtained data source plugin is dynamically loaded and initialized on the Worker node.

[0075] In this embodiment, after the Master node completes the data sharding, it uses the ZooKeeper coordination service to monitor the operating status of the Worker node cluster in real time, continuously collecting heartbeat information and resource utilization indicators, such as CPU utilization, memory usage, and network I / O load, periodically reported by each Worker node. Based on these indicators, the system calculates a comprehensive load score for each Worker node and prioritizes allocating the data shards to be assigned to available Worker nodes with lower load scores. For example, when it detects that Worker node A has a CPU utilization of 30% and a memory usage of 40%, while Worker node B has a CPU utilization of 75% and a memory usage of 80%, the system will prioritize allocating the new data shards to the lower-load Worker node A.

[0076] Furthermore, the data sharding allocation process employs an intelligent scheduling strategy. For example, the Master node first allocates shards with larger data volumes to the highest-performing Worker nodes. Simultaneously, considering data locality principles, shards requiring access to the same data source are allocated to the same Worker node whenever possible to reduce network overhead. Allocation decisions are updated in real time; when a Worker node's load exceeds a threshold (e.g., CPU utilization exceeds 85%), the system automatically reassigns its unprocessed shards to other idle nodes.

[0077] After receiving the assigned data sharding task, the Worker node retrieves the corresponding standardized data source plugin from the plugin repository based on the data source type (e.g., MySQL, HDFS) and target source type (e.g., Doris, Kafka) defined in the synchronization task configuration. Specifically, the Worker node first generates a plugin dependency list based on the configuration, then downloads the corresponding Reader plugin (e.g., MySQL Reader) and Writer plugin (e.g., HDFS Writer) from the plugin repository to its local cache, and finally dynamically loads these plugin packages and initializes plugin instances using the Java class loader. This entire hot-loading process does not require restarting the system service, achieving plug-and-play functionality for the data source plugins. For example, when processing a MySQL to HDFS synchronization task, the Worker node loads both the MySQL-Reader and HDFS-Writer plugins simultaneously and establishes the corresponding data read and write channels. After plugin initialization is complete, the Worker node immediately starts the execution engine and begins processing the assigned data sharding task.

[0078] S14, on the Worker node, the execution engine is started using the loaded data source plugin to handle data synchronization; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

[0079] Further, in step S14, the step of concurrently reading the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and transferring the data to be synchronized to the Writer thread group through a zero-copy memory channel, and having the Writer thread group write the data to the target source in batches, includes:

[0080] A bounded blocking queue is established between the Reader thread group and the Writer thread group as the zero-copy memory channel. The binary transmission of the data to be synchronized is realized through the zero-copy memory channel. The Reader thread group stores the data to be synchronized in batches into the bounded blocking queue in the form of binary data. The Writer thread group retrieves the binary data from the bounded blocking queue and writes it in batches to the target source.

[0081] Reference Figure 4 As shown, in this embodiment, after loading the data source plugin, the Worker node starts the execution engine to process the assigned data sharding tasks. The execution engine first dynamically allocates thread resources based on the size of the data shards and the system resource status, that is, it allocates a corresponding number of Reader threads and Writer threads to form a concurrent pipeline processing architecture. For example, for shards with a large amount of data (such as those containing 100,000 vehicle trajectory records), more Reader and Writer threads are allocated (such as 8 threads each), while for smaller shards, fewer threads are allocated (such as 2 threads each) to optimize resource utilization.

[0082] The Reader thread group, based on the allocated data shard information, concurrently reads data to be synchronized in batches from the data source through the corresponding Reader plugin (such as MySQLReader). Each Reader thread is responsible for processing one or more data shards, using a batch fetching method (e.g., reading 1024 records at a time to form a data batch), and directly storing the data in binary format into a memory channel composed of a bounded blocking queue to improve data reading efficiency. The read data maintains its original binary format in memory, avoiding unnecessary serialization / deserialization operations.

[0083] A bounded blocking queue established between the Reader and Writer thread groups forms a zero-copy memory channel. This channel is implemented using a circular buffer, supporting efficient binary data transfer. Reader threads directly write batches of data to be synchronized into the queue in binary form, while Writer threads read binary data from the other end of the queue. The entire process eliminates the need for data copying and format conversion, significantly reducing CPU overhead and memory usage. The Writer thread group uses a corresponding Writer plugin (such as HDFS Writer) to write the acquired binary data in batches to the target data source. The execution engine also includes a dynamic backpressure mechanism. When the processing rate of Writer threads is detected to be lower than the production rate of Reader threads, causing queue backlog, the read rate of Reader threads is automatically reduced to prevent memory overflow.

[0084] The execution engine also integrates a breakpoint resume manager, which saves the current processing position (such as the MySQL binlog position or HDFS file offset) to distributed storage in real time after each batch of data is successfully written. The entire data processing flow embodies a highly efficient pipeline architecture. Through the decoupled design of Reader→Channel→Writer, it achieves read-write separation and parallel processing, fully leverages the performance of multi-core CPUs, effectively improves the throughput of data synchronization and system resource utilization, and meets the high-throughput synchronization requirements of large amounts of data from general freight vehicles.

[0085] Furthermore, in another embodiment, the method further includes:

[0086] During the process of the Writer thread group writing to the target source in batches, the synchronization position information of each data shard is persisted to the distributed storage module;

[0087] When a worker node failure or network interruption is detected, the master node redistributes the incomplete data fragments to idle worker nodes based on the synchronization location information, and resumes data synchronization from the corresponding breakpoint.

[0088] In this embodiment, during the batch writing of data to the target source by the Writer thread group, the system (such as the breakpoint resume manager) periodically persists the synchronization position information of each data shard to the distributed storage module. The position information varies depending on the data source type. For example, for database data sources such as MySQL, the system records the latest binlog position and commit timestamp; for message queue data sources such as Kafka, it records the message offset and partition information; for file-based data sources such as HDFS, it records the file offset; and for file systems, it records the file path and offset. This synchronization position information is stored in units of shards, with each data shard having its own independent breakpoint record. The persistence operation is performed immediately after each batch of data is successfully written to the target source, ensuring the accuracy and timeliness of the records.

[0089] Reference Figure 5 As shown in the diagram, when a worker node failure or network interruption is detected by the ZooKeeper heartbeat detection mechanism during monitoring, the master node sends a fault alarm to initiate the fault recovery process. First, the master node retrieves the last successful synchronization location information (breakpoint location) of all incomplete data shards on the failed node from the distributed storage module. Then, it re-marks these incomplete data shards as pending allocation. Next, based on the load of currently available worker nodes, it reassigns these data shards to worker nodes with lower loads. Upon receiving the reassigned data shards, the newly allocated worker node first loads the required data source plugins from the plugin repository, then connects to the data source and starts reading data from the breakpoint location. This allows the system to quickly and automatically recover in case of anomalies, effectively ensuring data consistency and task continuity in long-term operation and large-volume data synchronization scenarios. Precise location recording enables breakpoint resumption, guaranteeing data consistency.

[0090] Furthermore, in another embodiment, the method further includes:

[0091] The system collects monitoring metrics in real time, including throughput and resource utilization of Worker nodes. When an anomaly is detected in the monitoring metrics, an alarm is triggered.

[0092] In this embodiment, the system implements end-to-end performance monitoring and intelligent alarm mechanisms through a monitoring center. The monitoring center adopts a distributed data acquisition architecture, deploying lightweight acquisition agents at various key nodes of the system to collect key indicators in real time, including the data read rate of the Reader thread group, the data write rate of the Writer thread group, the resource utilization of Worker nodes (such as CPU utilization, memory usage, and network I / O throughput), network latency, and data consistency verification results.

[0093] The monitoring center's rule engine uses a built-in threshold detection mechanism to detect key indicators. When anomalies are detected, tiered alarms are immediately triggered. For example, if a severe imbalance in read and write rates is detected (e.g., read rate consistently exceeds write rate by more than 30%), the system determines that a write bottleneck exists and automatically sends an important alarm. If the resource utilization of a worker node exceeds a preset threshold (e.g., CPU utilization consistently exceeds 85% or memory utilization exceeds 90%), an emergency alarm is triggered. If data consistency verification finds that the difference between the records on the data source and the target source exceeds the allowable range, a data anomaly alarm is triggered.

[0094] Alarm information is pushed in real time through multiple channels, including the system's built-in message center, email notifications, and WeChat chatbots, ensuring that operations and maintenance personnel can detect system anomalies immediately. Simultaneously, the monitoring center provides intelligent diagnostic functions, automatically locating the root cause of performance bottlenecks based on the root cause analysis algorithm built into the rule engine. For example, it identifies specific issues such as "HDFS write response latency causing Writer queue blocking," providing precise guidance for system optimization and troubleshooting. Furthermore, the monitoring center offers a visualization interface, allowing all monitoring data and analysis results to be displayed visually, such as through heatmaps and trend curves, showcasing system operating status and supporting comprehensive monitoring and trend analysis of system operation by operations and maintenance personnel.

[0095] Reference Figure 6 The diagram shown is a structural schematic of a heterogeneous data source synchronization device for general freight vehicles provided in an embodiment of the present invention.

[0096] In this embodiment, the device 20 includes:

[0097] Task receiving unit 21 is used to receive the synchronization task configuration of general freight vehicle data submitted by the user. The synchronization task configuration includes defining the data source, the target source, and the synchronization parameters.

[0098] The data partitioning unit 22 is used to parse the synchronization task configuration through the Master node and partition the data to be synchronized based on the partitioning strategy in the synchronization parameters to generate multiple data partitions.

[0099] The allocation unit 23 is used to dynamically allocate the data shards to the corresponding Worker nodes according to the real-time load status of the Worker nodes, and to hot-load the data source plugin that matches the data source and the target source from the plugin repository through the Worker nodes.

[0100] Synchronization unit 24 is used on the Worker node to start the execution engine to process data synchronization using the loaded data source plugin; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

[0101] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0102] This invention also provides a heterogeneous data source synchronization device for general freight vehicles. The device includes the heterogeneous data source synchronization apparatus for general freight vehicles described above, wherein the heterogeneous data source synchronization apparatus for general freight vehicles can employ… Figure 6 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0103] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0104] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0105] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0106] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0107] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method shown is for synchronizing heterogeneous data sources for general freight vehicles. The computer-readable storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0108] This invention also provides a computer program product, including a computer program / instructions, which are loaded and executed by a processor to implement... Figure 1 This paper presents a method for synchronizing heterogeneous data sources for general freight vehicles.

[0109] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0110] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for synchronizing heterogeneous data sources for general freight vehicles, characterized in that, The method includes: The synchronization task configuration for receiving general freight vehicle data submitted by users includes defining the data source, target source, and synchronization parameters. The Master node parses the synchronization task configuration and divides the data to be synchronized based on the sharding strategy in the synchronization parameters, generating multiple data shards. Based on the real-time load status of the Worker nodes, the data shards are dynamically allocated to the corresponding Worker nodes, and the Worker nodes hot-load data source plugins that match the data source and the target source from the plugin repository to execute the allocated data sharding tasks. On the Worker node, the execution engine is started using the loaded data source plugin to handle data synchronization; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

2. The method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The method further includes: During the process of the Writer thread group writing to the target source in batches, the synchronization position information of each data shard is persisted to the distributed storage module; When a worker node failure or network interruption is detected, the master node redistributes the incomplete data fragments to idle worker nodes based on the synchronization location information, and resumes data synchronization from the corresponding breakpoint.

3. The method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The method further includes: The system collects monitoring metrics in real time, including throughput and resource utilization of Worker nodes. When an anomaly is detected in the monitoring metrics, an alarm is triggered.

4. The method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The process involves parsing the synchronization task configuration via the Master node and dividing the data to be synchronized based on the sharding strategy in the synchronization parameters to generate multiple data shards, including: Based on the data primary key specified in the synchronization parameters, the data to be synchronized is divided according to the numerical range of the data primary key to obtain multiple consecutive and non-overlapping data fragments.

5. A method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The step of dynamically allocating the data shards to the corresponding Worker nodes based on the real-time load status of the Worker nodes includes: ZooKeeper monitors the heartbeats and resource utilization reported by each Worker node, and allocates the data shards to Worker nodes with low load based on resource utilization.

6. A method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The hot loading of data source plugins matching the data source and the target source from the plugin repository via Worker nodes includes: According to the synchronization task configuration, the corresponding standardized data source plugin is obtained from the plugin repository through the Worker node, and the obtained data source plugin is dynamically loaded and initialized on the Worker node.

7. A method for synchronizing heterogeneous data sources for general freight vehicles according to claim 1, characterized in that, The process of concurrently reading the data to be synchronized corresponding to the data shards from the data source through the Reader thread group, and transferring the data to be synchronized to the Writer thread group through a zero-copy memory channel, and then having the Writer thread group write the data to the target source in batches, includes: A bounded blocking queue is established between the Reader thread group and the Writer thread group as the zero-copy memory channel. The binary transmission of the data to be synchronized is realized through the zero-copy memory channel. The Reader thread group stores the data to be synchronized in batches into the bounded blocking queue in the form of binary data. The Writer thread group retrieves the binary data from the bounded blocking queue and writes it in batches to the target source.

8. A heterogeneous data source synchronization device for general freight vehicles, characterized in that, The device includes: The task receiving unit is used to receive the synchronization task configuration of general freight vehicle data submitted by the user. The synchronization task configuration includes defining the data source, the target source, and the synchronization parameters. The data partitioning unit is used to parse the synchronization task configuration through the Master node and partition the data to be synchronized based on the partitioning strategy in the synchronization parameters to generate multiple data partitions. The allocation unit is used to dynamically allocate the data shards to the corresponding Worker nodes according to the real-time load status of the Worker nodes, and to hot-load the data source plugins that match the data source and the target source from the plugin repository through the Worker nodes. The synchronization unit is used on the Worker node to start the execution engine to process data synchronization using the loaded data source plugin; wherein, the execution engine concurrently reads the data to be synchronized corresponding to the data shard from the data source in batches through the Reader thread group, and passes the data to be synchronized to the Writer thread group through the zero-copy memory channel, and the Writer thread group writes it to the target source in batches.

9. A heterogeneous data source synchronization system, characterized in that, The system includes: The user console is used to receive synchronization task configurations submitted by users via a web console or API, and send the synchronization task configurations to the unified scheduling center. The unified scheduling center includes a Master node and Worker nodes. The Master node is used to parse the synchronization task configuration, generate multiple data shards based on the sharding strategy, and dynamically allocate data shards to Worker nodes according to the real-time load status of the Worker nodes. The Worker nodes are used to receive data shards and hot-load the data source plugins corresponding to the data source and target source from the plugin repository to execute the allocated data sharding tasks. The plugin repository stores standardized data source plugins and provides hot-reloading support to Worker nodes. The execution engine, deployed in the Worker nodes, interacts with the plugin repository and distributed storage module. It includes a dynamic concurrency controller, a pipeline processor, and a breakpoint resumption manager. The dynamic concurrency controller allocates threads based on the amount of data in each data shard. The pipeline processor reads the data to be synchronized from the data source in batches from the data source via the Reader thread group, transmits the data to be synchronized to the Writer thread group via a zero-copy memory channel, and then writes it to the target source in batches by the Writer thread group. The breakpoint resumption manager persists data location information to the distributed storage module and resumes synchronization from the breakpoint position based on the data location information when a fault is detected. The distributed storage module is used to store the data location information and data fragments persisted by the breakpoint resume manager; The monitoring center is used to collect monitoring indicators such as throughput and resource utilization during the data synchronization process in real time, analyze the collected monitoring indicators, and trigger alarms when performance bottlenecks or data anomalies are detected.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the steps of a heterogeneous data source synchronization method for general freight vehicles as described in any one of claims 1 to 7.

Images