Synchronizer-based data distribution system, and distribution method, device and medium thereof

Abstract

The application discloses a data distribution system based on a synchronizer, a distribution method, equipment and a medium, and comprises at least one synchronizer plug-in and a synchronization framework, a management module for providing a synchronization plug-in registration, configuration and state query interface, an interface module for setting interface standard data corresponding to the synchronization plug-in, a strategy configuration module for configuring a set strategy of distributing original data, an engine module for creating a corresponding synchronization plug-in instance when the management module enables any synchronization plug-in, a load balancing module for screening out a synchronization plug-in instance receiving original data from the engine module according to the set strategy, and a buffer module for controlling a data ready speed of an instance buffer queue corresponding to the enabled synchronization plug-in. The application is composed of the synchronization framework and the synchronization plug-in to form a synchronization plug-in architecture, and meets the requirements of efficient synchronization capability of data distribution and large-scale reuse and fast delivery under limited hardware resources and input cost.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a data distribution system based on a synchronizer, as well as its distribution method, device, and medium. Background Technology

[0002] With the deepening development of the Industrial Internet of Things (IIoT), its application scenarios are expanding from large enterprises to a large number of small and medium-sized manufacturing plants. These plants generally hope to accurately and in real time synchronize the massive amounts of operational data, status data, and process data generated by industrial equipment to the upper-level Manufacturing Execution System (MES), Enterprise Resource Planning (ERP), or cloud data analysis platform in a low-cost and lightweight manner, so as to achieve production visualization, process optimization, and decision support.

[0003] However, current mainstream data synchronization and distribution technologies face significant challenges when implemented. They use traditional streaming frameworks such as Flink and Spark, and consume tens or hundreds of gigabytes of memory resources, resulting in resource waste for small and medium-sized industrial IoT projects. The data bridging function of EMQX Enterprise Edition cannot meet budget requirements. Secondly, traditional IoT projects involve a lot of customized development when synchronizing data, making it impossible to replicate on a large scale between projects. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art in which it is impossible to provide efficient and stable synchronization capabilities for device data and large-scale reuse under limited hardware resources and development investment costs. The present invention provides a data distribution system based on a synchronizer, as well as its distribution method, device and medium.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution:

[0006] In a first aspect, the present invention provides a synchronizer-based data distribution system, the data distribution system comprising: at least one synchronizer plugin and a synchronization framework, the synchronization framework being built on a backend management platform; the synchronization framework comprising an interface module, a management module, a strategy configuration module, an engine module, a buffer module, and a load balancing module; the synchronizer plugin being configured according to the requirements of the target end business system to achieve data distribution capabilities and having a standard interface;

[0007] The management module is used to provide interfaces for registering, configuring, and querying the status of the synchronizer plugin, and to coordinate the work of the other modules.

[0008] The interface module is used to set the interface standard data corresponding to the synchronizer plugin, and to perform a legality check on the registered synchronizer plugin corresponding to the interface standard data.

[0009] The strategy configuration module is used to configure the setting strategy for distributing raw data and to verify the legality of the setting strategy; the raw data is the input data collected by the data acquisition system.

[0010] The engine module is used to create a corresponding synchronizer plugin instance when the management module enables any of the synchronizer plugins; the synchronizer plugin instance is used to load the preprocessing capability and the data distribution capability of the synchronizer plugin, and store the preprocessed target data in the instance buffer queue.

[0011] The load balancing module is used to broadcast the raw data to the enabled synchronizer plugin, and select synchronizer plugin instances from the engine module to receive the raw data according to the set strategy;

[0012] The buffer module is used to control the data readiness speed of the instance buffer queue corresponding to the synchronizer plugin after it is enabled, so that the synchronizer plugin instance can read the target data at a set time point and send it to the target end business system.

[0013] Preferably, the configuration strategy includes a data preprocessing strategy, a data sending strategy, and a load balancing strategy; the data preprocessing strategy is used to characterize the methods of filtering data, formatting data, and converting data to different number systems; the data sending strategy is used to characterize the maximum data sending rate, maximum buffer size, and buffering method; and the load balancing strategy is used to characterize the number of synchronizer plugin instances and the load balancing type.

[0014] Preferably, the engine module is further configured to load the data preprocessing strategy corresponding to the synchronizer plugin when the synchronizer plugin instance is enabled, and determine the processing method during the preprocessing of the raw data.

[0015] Preferably, the load balancing module is further configured to determine the target number of instances and the target load balancing type for enabling the synchronizer plugin based on the load balancing strategy.

[0016] Preferably, the buffer module is further configured to load the data sending strategy corresponding to the synchronizer plugin when each synchronizer plugin instance is created, and determine the buffering method when the original data is sent.

[0017] Preferably, the interface standard data includes at least one of initialization standard, data preprocessing standard, data distribution standard, status reporting standard, and destruction standard.

[0018] In a second aspect, the present invention provides a distribution method for a synchronizer-based data distribution system, applied to the synchronizer-based data distribution system described in the first aspect, the distribution method comprising:

[0019] The load balancing module selects a synchronizer plugin instance for each enabled synchronizer plugin to receive raw data according to the load balancing strategy.

[0020] The engine module controls the synchronizer plugin instance to preprocess the raw data according to the data preprocessing strategy and then store it in the instance buffer queue.

[0021] The buffer module controls the data readiness speed of the instance buffer queue according to the data sending strategy and downstream data status feedback, so that the synchronizer plugin instance can read the target data at a set time point and send it to the target end business system.

[0022] The synchronizer plugin instance uploads data transmission status information to the management module for display and to the buffer module for traffic shaping and closed-loop adjustment.

[0023] Preferably, the distribution method further includes:

[0024] Register several synchronizer plugins on the backend management platform and configure the setting strategy corresponding to each synchronizer plugin; the setting strategy includes at least one of data preprocessing strategy, data sending strategy and load balancing strategy.

[0025] The interface module is used to verify the legality of the synchronizer plugin after registration;

[0026] After the verification is successful, the synchronizer plugin is enabled through the backend management platform. The management module starts the corresponding set number of synchronizer plugin instances through the engine module.

[0027] The instance buffer queue corresponding to the synchronizer plugin after the buffer module is initialized and enabled;

[0028] And / or,

[0029] When the synchronizer plugin is updated, the management module loads new plugin code through the engine module to update the synchronizer plugin; the synchronizer plugin instance corresponding to the unupdated synchronizer plugin is deleted after the data in the instance buffer queue has been processed.

[0030] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distribution method of the synchronizer-based data distribution system as described in the first aspect.

[0031] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the distribution method of the synchronizer-based data distribution system as described in the first aspect.

[0032] The positive and progressive effects of this invention are as follows: This invention discloses a data distribution system, method, device, and medium based on a synchronizer, comprising: at least one synchronizer plugin and a synchronization framework; a management module for providing synchronizer plugin registration, configuration, and status query interfaces; an interface module for setting the interface standard data corresponding to the synchronizer plugin; a policy configuration module for configuring the setting policy for distributing raw data and verifying the legality of the setting policy; an engine module for creating a corresponding synchronizer plugin instance when the management module enables any synchronizer plugin; a load balancing module for selecting synchronizer plugin instances from the engine module to receive raw data according to the set policy; and a buffer module for constructing an instance buffer queue corresponding to the enabled synchronizer plugin. This invention, consisting of a synchronizer framework and synchronizer plugins, constitutes a synchronizer plugin architecture, meeting the requirements for efficient and stable synchronization of device data and large-scale reuse and rapid delivery under limited hardware resources and investment costs; the hot update mechanism of the synchronizer plugin shortens the downtime caused by data distribution function updates; the buffer module uses a multi-level buffer queue to achieve data transmission buffering and traffic shaping; and the load balancing module achieves multi-instance load balancing and horizontal scaling. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the distribution system of the synchronizer-based data distribution system according to Embodiment 1 of the present invention.

[0034] Figure 2 This is a schematic diagram of the application of the data distribution system based on the synchronizer in Embodiment 1 of the present invention.

[0035] Figure 3 This is a schematic diagram of the first process of the data distribution method of the synchronizer-based data distribution system according to Embodiment 2 of the present invention.

[0036] Figure 4 This is a schematic diagram of the second process of the data distribution method of the synchronizer-based data distribution system according to Embodiment 2 of the present invention.

[0037] Figure 5 This is a schematic diagram of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0038] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0039] Example 1

[0040] This embodiment provides a synchronizer-based data distribution system, such as... Figure 1As shown, the data distribution system includes: at least one synchronizer plugin 100 and a synchronization framework 200, the synchronization framework 200 being built on a backend management platform; the synchronization framework 200 includes a management module 210, an interface module 220, a policy configuration module 230, an engine module 240, a load balancing module 250, and a buffer module 260; the synchronizer plugin 100 is configured according to the requirements of the target business system to achieve data distribution capabilities and has a standard interface;

[0041] The management module 210 is used to provide interfaces for registering, configuring, and querying the status of synchronizer plugins, and to coordinate the work of other modules;

[0042] The interface module 220 is used to set the interface standard data corresponding to the synchronizer plugin 100, and to perform a legality check on the registered synchronizer plugin 100 corresponding to the interface standard data.

[0043] The strategy configuration module 230 is used to configure the setting strategy for distributing raw data and to verify the legality of the setting strategy; the raw data is the input data collected by the data acquisition system.

[0044] Engine module 240 is used to create a corresponding synchronizer plugin instance when management module 210 enables any synchronizer plugin; the synchronizer plugin instance is used to load the preprocessing capability and data distribution capability of the synchronizer plugin and store the preprocessed target data in the instance buffer queue.

[0045] The load balancing module 250 is used to broadcast raw data to enabled synchronizer plugins and select synchronizer plugin instances from the engine module to receive raw data according to the set strategy.

[0046] The buffer module 260 is used to control the data readiness speed of the instance buffer queue corresponding to the synchronizer plugin after it is enabled, so that the synchronizer plugin instance can read the target data and send it to the target business system at a set time point.

[0047] The strategy settings include data preprocessing strategy, data sending strategy, and load balancing strategy. The data preprocessing strategy is used to characterize the methods of filtering data, formatting data, and converting data to different number systems. The data sending strategy is used to characterize the maximum data sending rate, maximum buffer size, and buffering method. The load balancing strategy is used to characterize the number of synchronizer plugin instances and the load balancing type.

[0048] The interface standard data includes at least one of the following: initialization standard, data preprocessing standard, data distribution standard, status reporting standard, and destruction standard.

[0049] Specifically, the backend management platform is a web-based management interface integrated with an HTTP (Hypertext Transfer Protocol) interface. It provides a visual window for user interaction, allowing for configuration of synchronizer strategies, viewing synchronizer status, and other interactive operations. Synchronizer Plugin 100 is a compiled code product implementing the standard interface required by the interface module. Written in Go, it is compiled into a dynamically linked file with the ".so" extension. The plugin's capabilities can be dynamically loaded into the synchronization framework's engine module. Synchronizer Plugin 100 is customized according to the requirements of the target business system, using different network protocols such as HTTP and MQTT to achieve data distribution capabilities. The synchronization framework uses Go's built-in standard plugin module to implement a hot-update mechanism for the synchronizer. When a plugin is registered and enabled, the synchronization framework loads the code implementation from the plugin, and the engine module re-initializes the instance according to the strategy, without restarting the entire data distribution system. Thanks to the use of Go, this system supports hot updates of the synchronizer while maintaining the lightweight nature of the overall framework and plugins; a single Synchronizer Plugin 100 adds only about 10MB of memory. A synchronizer plugin instance is an independent running instance of the synchronizer plugin code, running as a goroutine lightweight thread.

[0050] The management module 210 is responsible for coordinating the other modules, realizing the full lifecycle management and data distribution process of the synchronizer plugin 100, and providing HTTP interfaces for synchronizer registration, configuration, and status query to be integrated with the backend management platform to provide a visual interactive interface.

[0051] Interface module 220 defines the interface standard data that all synchronizer plugins 100 must implement, consisting of initialization standards, data preprocessing standards, data distribution standards, status reporting standards, and destruction standards. The initialization standard determines the synchronizer configuration loaded from the strategy configuration module 230, loads resources, and initializes the synchronizer instance. The data preprocessing standard determines the data preprocessing functions such as filtering, formatting, and number system conversion. The data distribution standard determines the specific implementation method for transmitting encapsulated data to the target end. For example, it can be customized based on different data transmission types, or based on various network protocols such as HTTP, MQTT, and TCP. Each type of synchronizer can be reused in other projects as needed after development. The status reporting standard determines that the returned data transmission result will be passed to the caller, including whether the data transmission was successful and the transmission delay. The destruction standard determines that resources are released and the instance is destroyed. Interface module 220 serves as the interaction contract between synchronizer plugin 100 and the synchronization framework, verifying whether the registered plugin implements the interface standard data and checking whether the plugin code is legal and valid.

[0052] The strategy configuration module 230 manages all configured strategies during synchronizer data distribution and provides validation of these strategies. The management module 210 collaborates with the strategy configuration module 230, providing an HTTP interface for integration with the backend management platform to allow users to configure strategies on the interface. Multiple synchronizer plugin instances of each synchronizer plugin 100 share the same configuration strategy. These configuration strategies can be read and used by the synchronizer plugin 100 and other modules, supporting the configuration of data filtering strategies, load balancing strategies, and data transmission strategies, among others.

[0053] Engine module 240 maintains all synchronizer plugin instances. When management module 210 starts a synchronizer plugin, it creates a synchronizer plugin instance through engine module 240. Each synchronizer plugin instance is an independent goroutine, similar to a processor that can independently process data according to a strategy. The synchronizer plugin instance loads the preprocessing and data distribution capabilities of synchronizer plugin 100, places the preprocessed data into buffer module 260, and sends the buffered data to the target service when the time is right. Recorded data, including the success rate and latency status, is fed back to management module 210 for real-time status display and to buffer module 260 for traffic shaping.

[0054] When the load balancing module 250 starts, it reads the load balancing policy and determines the number of synchronizer plugin instances to start based on the number of instances in the load balancing policy, thus achieving horizontal scaling. When a piece of data arrives, the load balancing module 250 broadcasts the data to all enabled synchronizer plugins. If a synchronizer plugin corresponds to multiple synchronizer plugin instances, the load balancing module 250 will select the appropriate synchronizer plugin instance to process and distribute the data according to the policy.

[0055] The buffer module 260 supports data transmission buffering to improve the stability of data distribution. This data readiness rate characterizes the time from when data enters the queue to when it can be sent. It maintains an independent buffer queue for each running synchronizer plugin instance, responsible for the temporary storage of pre-processed data and for the synchronizer plugin instance to send the pre-processed data. Furthermore, the buffer module 260 also supports traffic shaping, dynamically adjusting the data outflow rate based on the synchronizer plugin's status feedback to ensure the stability of downstream business systems.

[0056] In this embodiment, the synchronizer plugin architecture decouples the system, reducing development complexity, shortening the development cycle, and lowering development costs. The concept of synchronizer plugins is introduced, encapsulating the logic of distributing data to different target business systems as independent, pluggable plugins. When a synchronizer plugin is updated, the synchronization framework dynamically loads the new synchronizer's functionality without restarting the data distribution service, reducing service downtime from hours to seconds. The buffer module 260 uses a multi-level buffer queue to temporarily store data. When the target system responds slowly or the network experiences momentary fluctuations, data can be safely accumulated on the platform side, preventing data loss or backpressure on the data acquisition end, ensuring the stability of the acquisition end. The buffer module 260 can perform traffic shaping, dynamically adjusting the data distribution rate based on the processing capacity and health status of downstream systems, achieving adaptive balance between the production and consumption ends, preventing downstream systems from being overwhelmed, and improving the stability of data synchronization. The synchronizer plugin instance is located in the engine module 240 and built into the IoT system, reducing data transmission links and lowering transmission latency. The load balancing module can distribute data to several parallel synchronizer plugin instances according to preset strategies, increasing throughput by ten times and easily coping with data surges caused by a rapid increase in the number of devices, achieving high-performance horizontal scaling under a lightweight architecture.

[0057] In one embodiment, the data distribution system further includes:

[0058] Engine module 240 is also used to load the data preprocessing strategy corresponding to synchronizer plugin 100 when a synchronizer plugin instance is enabled, and to determine the processing method during raw data preprocessing.

[0059] Specifically, the data preprocessing strategy includes filtering data configuration items that configure forward and reverse filtering rules for data features such as device ID, message type, and message topic; formatted data configuration items that format all values ​​to retain at most two decimal places; and data base conversion configuration items that configure data base conversion rules.

[0060] The load balancing module 250 is also used to determine the target number of instances and the target load balancing type of the synchronizer plugin 100 based on the load balancing strategy.

[0061] Specifically, the target load balancing type includes a round-robin type where a synchronizer instance is selected in turn to distribute the data when a piece of data arrives; the hash value of the data topic is calculated and the hash value type of the selected synchronizer is determined by taking the remainder after dividing by the total number of synchronizers; the load of the business system is perceived through the status reporting of the synchronizer plugins; and the synchronizer plugin instance with the least load is selected to distribute the data.

[0062] The buffer module 260 is also used to load the data sending strategy corresponding to the synchronizer plugin 100 when each synchronizer plugin instance is created, and to determine the buffering method when sending raw data.

[0063] Specifically, the data transmission strategy includes configuration items corresponding to the number of retries for data transmission failure, retry isolation, maximum data transmission rate, maximum buffer size, and buffer type. The buffer type may include memory buffer, disk buffer, and hybrid memory-disk buffer.

[0064] In one embodiment, such as Figure 2 As shown, after the synchronizer plugin is registered, configured, and enabled, the data acquisition system collects data and connects it to the data distribution system. This distribution system adopts a synchronizer plugin architecture, collaborating with the synchronizer plugin instance through a synchronization framework. After preprocessing the data, the synchronizer plugin's data distribution capabilities send the data to external business systems. The data acquisition system and the data distribution system can be deployed within the same service, with data flowing directly through memory, or they can be deployed independently. The overall architecture of this application adopts a pluggable architecture design concept of synchronizer plugins and a synchronization framework, abstracting repetitive functions into the synchronization framework for reuse. The data distribution system runs as a built-in module of the IoT platform, consisting of a backend management page, synchronizer plugins, and a synchronization framework. This synchronization framework abstracts the underlying framework of general data distribution capabilities, is a code module developed in the Go language, and provides complete data distribution capabilities when combined with the synchronizer plugin.

[0065] This embodiment provides a synchronizer-based data distribution system, which consists of a synchronizer framework and synchronizer plugins to form a synchronizer plugin architecture. Under limited hardware resources and investment costs, it meets the requirements of efficient synchronization capability and large-scale reuse and fast delivery of data distribution. The hot update mechanism of the synchronizer plugin shortens the downtime caused by data distribution function updates. The buffer module adopts a multi-level buffer queue to realize data transmission buffering and traffic shaping, and the load balancing module realizes multi-instance load balancing and horizontal scaling.

[0066] Example 2

[0067] This embodiment provides a distribution method for a synchronizer-based data distribution system, such as... Figure 3 As shown, the data distribution method applied to the synchronizer-based data distribution system as in Example 1 includes:

[0068] S11. The load balancing module selects a synchronizer plugin instance for each enabled synchronizer plugin to receive raw data according to the load balancing strategy.

[0069] S12. The engine module control synchronizer plugin instance preprocesses the raw data according to the data preprocessing strategy and then stores it in the instance buffer queue.

[0070] S13. The buffer module controls the data readiness speed of the instance buffer queue according to the data sending strategy and downstream data status feedback, so that the synchronizer plug-in instance can read the target data at the set time point and send it to the target business system.

[0071] S14. The synchronizer plugin instance uploads data sending status information to the management module for display and to the buffer module for traffic shaping and closed-loop adjustment.

[0072] Regarding steps S11-S14 above, after receiving the raw collected data, the data distribution system hands the data over to the load balancing module for processing. The load balancing module broadcasts the data to all enabled synchronizer plugins. Based on the configured load balancing strategy, it quickly selects a synchronizer plugin instance from the engine module to receive the data for each synchronizer plugin. After receiving the data, the synchronizer plugin instance calls the data preprocessing method loaded from the synchronizer plugin. This data preprocessing method performs preprocessing such as filtering, formatting, and number system conversion on the data according to the data preprocessing strategy defined in the strategy configuration module. The preprocessed data is placed into its dedicated buffer queue by the synchronizer instance plugin, decoupling data preprocessing from data distribution. The queueing operation is an extremely fast, non-blocking operation, ensuring that the data preprocessing process is not blocked by downstream network latency or failures, thus guaranteeing the stability of the acquisition link.

[0073] The buffer module dynamically controls the data readiness rate in the buffer queue based on the data transmission strategy and real-time downstream status feedback. This data readiness rate characterizes the time from when data enters the queue to when it can be sent. The synchronizer plugin instance listens to the buffer queue, and once data is ready, it immediately retrieves it and invokes the loaded data distribution capability to send the data to the external business system. Under normal circumstances, the buffer module marks the data in the buffer queue as ready at a rate close to the maximum rate set by the strategy. The instance listens to the buffer queue, and when data is ready, it invokes the data distribution method to send the data to the target business system. When the downstream is busy, if the synchronizer plugin instance's data distribution method continuously returns errors such as the target system being busy or connection timeout, the status reporting mechanism will immediately transmit this feedback to the buffer module. The buffer module then triggers traffic shaping, reducing the data readiness rate of the queue according to a preset exponential backoff rule (e.g., increasing the waiting time from 1 second to 2 seconds, 4 seconds, etc.), thereby reducing the pressure on the downstream system. When the downstream recovers, and data distribution is successful continuously without receiving further error feedback, the buffer module will gradually increase the data dequeue rate until it returns to normal levels, ensuring system stability.

[0074] The synchronizer plugin instance within the engine module reports data transmission success rate, latency, and other information to the management and buffer modules in real time. The management module pushes the aggregated status data to the backend management platform, displaying the real-time health and performance of data distribution to users through dashboards, charts, and other formats. This status information serves as the core input for traffic shaping in the buffer module, forming an intelligent closed loop of distribution, monitoring, adjustment, and re-distribution.

[0075] In one embodiment, the distribution method further includes:

[0076] S15. When updating the synchronizer plugin, the management module loads the new plugin code through the engine module to update the synchronizer plugin; the synchronizer plugin instance corresponding to the unupdated synchronizer plugin is deleted after the instance buffer queue data is processed.

[0077] During the continuous operation of the data distribution system, if modifications to the distribution logic or plugin repairs are needed, a new synchronizer plugin file can be uploaded to the backend management platform, and the plugin can be re-enabled to achieve hot updates. The management module notifies the engine module to dynamically load the new plugin code to replace the old plugin. Newly started data processing instances will use the new logic, while old instances will be automatically destroyed after processing the remaining data in their buffer queues. The entire process is transparent to the data stream, achieving second-level lossless upgrades of the data distribution function.

[0078] This embodiment achieves the best balance between resource cost, real-time performance, development efficiency, flexibility, and ease of operation and maintenance in data distribution by decoupling the system through a synchronizer plug-in architecture, shortening downtime caused by data distribution function updates through a hot update mechanism, and using data sending buffering and traffic shaping, as well as multi-instance load balancing and horizontal scaling. It provides a suitable dedicated data distribution solution for industrial IoT scenarios, which meets the core requirements of low cost, lightweight, and fast delivery in factory digital transformation.

[0079] In one embodiment, such as Figure 4 As shown, the distribution method also includes:

[0080] S101. Register several synchronizer plugins on the backend management platform and configure the setting strategy for each synchronizer plugin; the setting strategy includes at least one of the data preprocessing strategy, data sending strategy and load balancing strategy.

[0081] S102. Use the interface module to verify the legality of the registered synchronizer plugin;

[0082] S103. After the verification is passed, the synchronizer plugin is enabled through the backend management platform. The management module starts the corresponding set number of synchronizer plugin instances through the engine module.

[0083] S104. The instance buffer queue corresponding to the synchronizer plugin after the buffer module is initialized and enabled.

[0084] Regarding steps S101-S104 above, after determining the data reception method of the business system, several corresponding synchronizer plugins are registered on the backend management platform, and the interface module verifies the legality of the synchronizer plugins. Data preprocessing strategies and exception retry strategies are configured on the backend management platform. When the backend management platform starts a synchronizer plugin, the management module will call the engine module to start a synchronizer plugin instance according to the configuration, load the synchronizer plugin's capabilities into the synchronizer plugin instance, and notify the buffer module to initialize the buffer queue.

[0085] This embodiment provides a distribution method for a synchronizer-based data distribution system. The synchronizer plugin architecture consists of a synchronizer framework and synchronizer plugins. Under limited hardware resources and investment costs, it meets the requirements of efficient and stable synchronization of device data and large-scale reuse and fast delivery. The hot update mechanism of the synchronizer plugin shortens the downtime caused by data distribution function updates. The buffer module uses a multi-level buffer queue to realize data transmission buffering and traffic shaping, and the load balancing module realizes multi-instance load balancing and horizontal scaling.

[0086] Example 3

[0087] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the distribution method of the synchronizer-based data distribution system of Embodiment 2. Figure 5 The electronic device 60 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0088] Electronic device 60 may be in the form of a general-purpose computing device, such as a server device. Components of electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0089] Bus 63 includes a data bus, an address bus, and a control bus.

[0090] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0091] The memory 62 may also include a program tool 625 having a set (at least one) of program modules 624, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0092] The processor 61 executes various functional applications and data processing by running computer programs stored in the memory 62, such as the distribution method of the synchronizer-based data distribution system in Embodiment 2 of the present invention.

[0093] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 65. Furthermore, the model-generated device 60 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of the model-generated device 60 via bus 63. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the model-generated device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0094] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0095] Example 4

[0096] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data distribution method of the synchronizer-based data distribution system of Embodiment 2.

[0097] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0098] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to perform the steps of the distribution method of the synchronizer-based data distribution system of Embodiment 2.

[0099] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0100] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A data distribution system based on a synchronizer, characterized in that, The data distribution system includes: at least one synchronizer plugin and a synchronization framework, the synchronization framework being built on a backend management platform; the synchronization framework includes an interface module, a management module, a strategy configuration module, an engine module, a buffer module, and a load balancing module; the synchronizer plugin is configured according to the requirements of the target business system to achieve data distribution capabilities and has a standard interface; The management module is used to provide interfaces for registering, configuring, and querying the status of the synchronizer plugin, and to coordinate the work of the other modules. The interface module is used to set the interface standard data corresponding to the synchronizer plugin, and to perform a legality check on the registered synchronizer plugin corresponding to the interface standard data. The strategy configuration module is used to configure the setting strategy for distributing raw data and to verify the legality of the setting strategy; the raw data is the input data collected by the data acquisition system. The engine module is used to create a corresponding synchronizer plugin instance when the management module enables any of the synchronizer plugins; the synchronizer plugin instance is used to load the preprocessing capability and the data distribution capability of the synchronizer plugin, and store the preprocessed target data in the instance buffer queue. The load balancing module is used to broadcast the raw data to the enabled synchronizer plugin, and select synchronizer plugin instances from the engine module to receive the raw data according to the set strategy; The buffer module is used to control the data readiness speed of the instance buffer queue corresponding to the synchronizer plugin after it is enabled, so that the synchronizer plugin instance can read the target data at a set time point and send it to the target end business system.

2. The synchronizer-based data distribution system as described in claim 1, characterized in that, The configuration strategy includes a data preprocessing strategy, a data sending strategy, and a load balancing strategy; the data preprocessing strategy is used to characterize the methods of filtering data, formatting data, and converting data to different number systems; the data sending strategy is used to characterize the maximum data sending rate, maximum buffer size, and buffering method; the load balancing strategy is used to characterize the number of synchronizer plugin instances and the load balancing type.

3. The synchronizer-based data distribution system as described in claim 2, characterized in that, The engine module is also used to load the data preprocessing strategy corresponding to the synchronizer plugin when the synchronizer plugin instance is enabled, and to determine the processing method during the preprocessing of the raw data.

4. The synchronizer-based data distribution system as described in claim 2, characterized in that, The load balancing module is also used to determine the target number of instances and the target load balancing type for enabling the synchronizer plugin based on the load balancing strategy.

5. The synchronizer-based data distribution system as described in claim 2, characterized in that, The buffer module is also used to load the data sending strategy corresponding to the synchronizer plugin when each synchronizer plugin instance is created, and to determine the buffering method when the original data is sent.

6. The synchronizer-based data distribution system as described in claim 1, characterized in that, The interface standard data includes at least one of the following: initialization standard, data preprocessing standard, data distribution standard, status reporting standard, and destruction standard.

7. A distribution method for a data distribution system based on a synchronizer, characterized in that, The data distribution method, applied to the synchronizer-based data distribution system as described in any one of claims 1-6, comprises: The load balancing module selects a synchronizer plugin instance for each enabled synchronizer plugin to receive raw data according to the load balancing strategy. The engine module controls the synchronizer plugin instance to preprocess the raw data according to the data preprocessing strategy and then store it in the instance buffer queue. The buffer module controls the data readiness speed of the instance buffer queue according to the data sending strategy and downstream data status feedback, so that the synchronizer plugin instance can read the target data at a set time point and send it to the target end business system. The synchronizer plugin instance uploads data transmission status information to the management module for display and to the buffer module for traffic shaping and closed-loop adjustment.

8. The distribution method of the synchronizer-based data distribution system as described in claim 7, characterized in that, The distribution method further includes: Register several synchronizer plugins on the backend management platform and configure the setting strategy corresponding to each synchronizer plugin; the setting strategy includes at least one of data preprocessing strategy, data sending strategy and load balancing strategy. The interface module is used to verify the legality of the synchronizer plugin after registration; After the verification is successful, the synchronizer plugin is enabled through the backend management platform. The management module starts the corresponding set number of synchronizer plugin instances through the engine module. The instance buffer queue corresponding to the synchronizer plugin after the buffer module is initialized and enabled; And / or, When the synchronizer plugin is updated, the management module loads new plugin code through the engine module to update the synchronizer plugin; the synchronizer plugin instance corresponding to the unupdated synchronizer plugin is deleted after the data in the instance buffer queue has been processed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the distribution method of the synchronizer-based data distribution system as described in any one of claims 7-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distribution method of the synchronizer-based data distribution system as described in any one of claims 7-8.

Images