A data synchronization method and system

Abstract

This application discloses a data synchronization method and system. The data synchronization method includes: receiving data to be synchronized provided by a first system; processing the data to be synchronized using at least one first rule design file in a rule engine to obtain topic messages corresponding to the first rule design file; sending all topic messages to an external message queue deployed in a cluster, with each topic message sent to a corresponding first topic queue in the external message queue, and the topic messages in the first topic queue being consumed by at least one second system adapted to the first topic queue. This application edits adapted business rule design files in the rule engine according to the needs of each synchronized system and sets up multiple topic queues in the external message queue, without needing to modify the source code of the system where the data source is located for each synchronized system, which helps to reduce the workload of the data sender.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data synchronization method and system. Background Technology

[0002] In information systems, there are often situations where data synchronization occurs from one system (e.g., system A) to other systems (e.g., systems B, C, and D). Generally, there are two ways to handle this: 1) System A calls the application programming interface (API) of systems B, C, and D to write data to systems B, C, and D; 2) System A pushes messages to a message queue, and systems B, C, and D listen to the message queue, process the messages, and write data to each system.

[0003] The drawback of method 1) is that each new system requires modification of the source code of system A, adding new API calls. When many new systems are added, the source code of system A needs to be adjusted frequently, easily leading to frequent updates. An improved approach is to write the API calls in a configurable way to achieve dynamic updates; however, when there are many external systems, a set of API calls still needs to be configured for each external system, requiring the writing of a large number of configuration files and debugging.

[0004] The advantage of method 2) is that message topics can be divided according to the type of external system, and each group of external systems only listens to one message topic. This approach can significantly reduce the workflow that requires personalized coding, but it still has a disadvantage: different systems may need different message formats. If the message format is changed, we need to modify the source code of system A to implement the new logic. The change process usually requires downtime for updates. Summary of the Invention

[0005] This application provides a data synchronization method and system. According to the needs of each synchronized system, the method edits the corresponding business rule design file in the rule engine and sets up multiple topic queues in the external message queue to provide the topic messages required by each synchronized system. It does not require modifying the source code of the data source system for each synchronized system, which helps to reduce the workload of the data sender. Furthermore, since multiple synchronized systems share a single topic queue, the number of topic queues in the external message queue is less than that of the synchronized systems, which helps to reduce the complexity of the entire system and enhance the maintainability of the system.

[0006] This application provides a data synchronization method, including:

[0007] Receive data to be synchronized from the first system;

[0008] The data to be synchronized is processed using at least one first rule design file in the rule engine to obtain the topic message corresponding to the first rule design file;

[0009] All topic messages are sent to an external message queue deployed in a cluster. Each topic message is sent to the corresponding first topic queue in the external message queue. The topic messages in the first topic queue are consumed by at least one second system that is adapted to the first topic queue.

[0010] In each first topic queue, the message format and message content of all topic messages are completely identical.

[0011] Preferably, each first topic queue includes at least one first message group, and the number of first message groups is the same as the number of second systems consuming the first topic queue.

[0012] Preferably, the data synchronization method further includes:

[0013] It receives message formats and content from the newly added third system, which is used to consume topic messages from the external message queue.

[0014] If the message format and content of the third system do not match the message format and content of all first topic queues, then add a second rule design file corresponding to the third system in the rule engine;

[0015] If data to be synchronized is received, the data to be synchronized will be processed to synchronize and obtain the first topic message corresponding to all first rule design files and the second topic message corresponding to all second rule design files.

[0016] Send the first topic message to the first topic queue, and at the same time send the second topic message to the newly added second topic queue in the external message queue.

[0017] Preferably, if the message format and message content of the third system match the message format and message content of one of the third topic queues, then a second message group corresponding to the third system is added to the third topic queue.

[0018] Preferably, the data synchronization method further includes:

[0019] If an exception occurs in the second or third system while processing the topic message, the exception handling information will be written to the centrally managed and maintained log system.

[0020] This application also provides a data synchronization system, including a data synchronization module and an external message queue deployed in a cluster.

[0021] The input data for the data synchronization module is the data to be synchronized provided by the fourth system. The data synchronization module includes a rule engine, which includes at least one third rule design file. Each third rule design file is used to process the data to be synchronized into a topic message corresponding to the third rule design file.

[0022] The external message queue includes a fourth topic queue that corresponds one-to-one with all third rule design files. The input data for each fourth topic queue is the topic message corresponding to it. The topic messages in the fourth topic queue are consumed by at least one fifth system adapted to the fourth topic queue.

[0023] In each of the fourth topic queues, the message format and message content of all topic messages are completely identical.

[0024] Preferably, the data synchronization system also includes a centralized log system, which is used to record the handling of anomalies in all synchronized systems.

[0025] Preferably, each fourth topic queue includes at least one third message group, the number of which is the same as the number of fifth systems consuming the fourth topic queue.

[0026] Preferably, the data synchronization module includes a receiving module, an adding module, a data processing module, and a sending module;

[0027] The receiving module is used to receive the message format and message content of the newly added sixth system, and the sixth system is used to consume topic messages in the external message queue;

[0028] The new module is used to add a fourth rule design file corresponding to the sixth system in the rule engine when the message format and message content of the sixth system do not match the message format and message content of all fourth topic queues.

[0029] The data processing module is used to process the data to be synchronized, and to synchronously obtain the third topic messages corresponding to all third rule design files and the fourth topic messages corresponding to all fourth rule design files.

[0030] The sending module is used to send the third topic message to the fourth topic queue, and at the same time send the fourth topic message to the fifth topic queue newly added by the external message queue.

[0031] Preferably, the new module is also used to add a fourth message group corresponding to the sixth system in the sixth topic queue when the message format and message content of the sixth system match the message format and message content of one of the sixth topic queues.

[0032] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0034] Figure 1 A flowchart of the data synchronization method provided in this application;

[0035] Figure 2 A flowchart of the newly added synchronized system provided in this application;

[0036] Figure 3 The structural diagram of the data synchronization system provided in this application;

[0037] Figure 4 A structural diagram of an example of the data synchronization system provided in this application;

[0038] Figure 5 A structural diagram of the data synchronization module provided in this application. Detailed Implementation

[0039] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0042] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0043] This application provides a data synchronization method and system. According to the needs of each synchronized system, the method edits the corresponding business rule design file in the rule engine and sets up multiple topic queues in the external message queue to provide the topic messages required by each synchronized system. It does not require modifying the source code of the data source system for each synchronized system, which helps to reduce the workload of the data sender. Furthermore, since multiple synchronized systems share a single topic queue, the number of topic queues in the external message queue is less than that of the synchronized systems, which helps to reduce the complexity of the entire system and enhance the maintainability of the system.

[0044] It should be noted that in this application, when the data source system needs to synchronize some data it manages to several synchronized systems, the required synchronization data varies among the different synchronized systems. For example, some systems only need partial data, while others need to anonymize certain data fields. Generally, the data source system needs to maintain several topic queues. All synchronized systems with the same message format and content requirements can share the same topic queue; therefore, each topic queue is suitable for one or more synchronized systems. The data source system sends data to the corresponding topic queue according to its specific needs.

[0045] Based on the above explanation, as Figure 1 As shown, the data synchronization method provided in this application includes:

[0046] S110: Receives data to be synchronized from the first system (the system where the data source is located).

[0047] S120: Use at least one first rule design file in the rule engine to process the data to be synchronized and obtain the topic message corresponding to the first rule design file.

[0048] Specifically, different first rule design files correspond to different topic messages with different message formats and content. In other words, by processing the data to be synchronized, at least one topic message can be obtained. The number of first rule design files is determined by the message formats and content required by all synchronized systems (such as the second and third systems below).

[0049] S130: Send all topic messages to an external message queue deployed in a cluster. Each topic message is sent to the corresponding first topic queue in the external message queue. The topic messages in the first topic queue are consumed by at least one second system (the synchronized system) adapted to the first topic queue. Each second system can listen to different topic queues in the external message queue and consume messages according to its desired message format, writing them to its own system.

[0050] Each first rule design file corresponds to a first topic queue, and the message format and message content of all topic messages in each first topic queue are completely consistent.

[0051] Based on this, preferably, each first topic queue includes at least one first message group, and the number of first message groups is the same as the number of second systems consuming the first topic queue. That is, each second system corresponds to one message group, and the second system subscribes to and listens to topic messages in that message group. Thus, each second system has its own dedicated message group, improving the efficiency of synchronous data acquisition for the second system.

[0052] Based on the above, when a new system is added to the synchronization system, it can be determined whether a message queue needs to be added based on its required message format and content. For example... Figure 2 As shown, adding a synchronized system includes the following steps:

[0053] S210: Receives message format and message content from the newly added third system (the synchronized system), which is used to consume topic messages from the external message queue.

[0054] S220: Determine whether the message format and content of the third system do not match the message format and content corresponding to all first topic queues. If yes, it means that no first topic queue is compatible with the third system, so execute S230; otherwise, it means that there is a first topic queue (denoted as the third topic queue) that is compatible with the third system, so execute S260.

[0055] S230: Add a second rule design file corresponding to the third system to the rule engine. Simultaneously, a second topic queue will be added to the external message queue, with the second rule design file corresponding to this second topic queue. In other words, if the newly added synchronized system requires a new message format or needs to filter messages, we can add a new topic queue and its message group to the external message queue, and add a new "rule design file" to the rule engine. This does not require changes to the data source system or the data synchronization module where the rule engine resides (please refer to...). Figure 3 and 4 The source code of ) does not require downtime for updates.

[0056] S240: If data to be synchronized is received, the data to be synchronized is processed to synchronize and obtain the first topic message corresponding to all first rule design files and the second topic message corresponding to all second rule design files.

[0057] S250: Send the first topic message to the first topic queue, and at the same time send the second topic message to the second topic queue. The first topic message in the first topic queue is consumed by the second system, and the second topic message in the second topic queue is consumed by the third system.

[0058] S260: A second message group corresponding to the third system is added to the third topic queue. The third system can consume topic messages in the second message group. Therefore, no changes are needed to the data source system or the data synchronization module where the rules engine resides, which helps reduce workload and improves the speed of system data integration.

[0059] Preferably, if the second or third system (i.e., the synchronized system) encounters an anomaly while processing the topic message, the anomaly handling information is written into the centrally managed and maintained log system for subsequent troubleshooting and problem tracing.

[0060] Based on the above data synchronization method Figure 3 A structural diagram of the corresponding data synchronization system is shown. For example... Figure 3 As shown, the data synchronization system includes a data synchronization module and an external message queue deployed in a cluster.

[0061] The input data for the data synchronization module is the data to be synchronized provided by the fourth system (i.e., the system where the data source is located, shown as system A in the diagram).

[0062] The data synchronization module includes a rules engine. A rules engine is a component embedded in the application that separates business decisions from application code, allowing business decisions to be written using predefined code. Because the rules engine extracts complex business logic from the business code, it reduces the difficulty of implementing that logic. Furthermore, the extracted business rules are implemented using the rules engine, making dynamic business rules maintainable. Combined with the rules engine's robust business rule designer, complex business rules can be quickly implemented without coding. Similarly, business users with some programming experience can quickly learn to use the rules engine to define complex business rules.

[0063] In this application, the rule engine includes at least one third rule design file, each third rule design file being used to process the data to be synchronized into a topic message corresponding to the third rule design file.

[0064] By editing the rule design file of the rule engine, you can dynamically add topic queues to external message queues to accommodate the needs of newly added synchronized systems.

[0065] Specifically, the data synchronization module executes the data synchronization method described above.

[0066] The external message queue includes a fourth topic queue corresponding one-to-one with all third rule design files. The input data for each fourth topic queue is the corresponding topic message output by the data synchronization module. The topic messages in the fourth topic queue are consumed by at least one fifth system adapted to the fourth topic queue. For example, ... Figure 3 As shown, topic queue 1 is for system B to consume, topic queue 2 is for system C to consume, and topic queue 3 is for system D to consume. All topic messages in each of the four fourth topic queues have the same message format and message content.

[0067] As an example, the external message queue is a Kafka cluster deployed in a cluster.

[0068] Preferably, each fourth topic queue includes at least one third message group, the number of which is the same as the number of fifth systems consuming the fourth topic queue.

[0069] Specifically, such as Figure 5 As shown, the data synchronization module includes a receiving module 510, a adding module 520, a data processing module 530, and a sending module 540.

[0070] The receiving module 510 is used to receive the data to be synchronized from the system where the data source is located, and to receive the message format and message content of the newly added sixth system. The sixth system is used to consume topic messages in the external message queue.

[0071] The newly added module 520 is used to add a fourth rule design file corresponding to the sixth system in the rule engine when the message format and message content of the sixth system do not match the message format and message content corresponding to all fourth topic queues. Furthermore, when the message format and message content of the sixth system match the message format and message content corresponding to one of the sixth topic queues, a fourth message group corresponding to the sixth system is added to the sixth topic queue.

[0072] The data processing module 530 is used to process the data to be synchronized, and to synchronously obtain the third topic messages corresponding to all third rule design files and the fourth topic messages corresponding to all fourth rule design files.

[0073] The sending module 540 is used to send the third topic message to the fourth topic queue, and simultaneously send the fourth topic message to a newly added fifth topic queue in the external message queue. The topic messages in all message groups within each fourth and fifth topic queue are identical.

[0074] Preferably, the data synchronization system also includes a centralized log system for recording the exception handling status of all synchronized systems. Understandably, the log system may include a log module corresponding to each synchronized system, or it may be a centralized log system that stores the exception handling status of all synchronized systems.

[0075] Figure 4 illustrates an example of a data synchronization system. In this example, the data source is an e-commerce system where customers place orders. The systems being synchronized include a logistics center and a membership center. After a customer completes a purchase on the e-commerce platform, the corresponding e-commerce order information needs to be synchronized from the e-commerce system to the logistics center to facilitate the logistics center's order fulfillment. Simultaneously, it needs to be synchronized to the membership center to award points to the customer.

[0076] Logistics centers and member centers have different requirements for order data formats. For example, we might further subdivide logistics orders into several major regions based on the customer's shipping address. In this case, we can configure a "rule design file" to send order messages to different topic queues based on the customer's shipping address. Logistics centers in different regions can listen to different topic queues to process orders. Thus, the topic queues in the external message queue include a topic queue corresponding to the member center and a topic queue corresponding to each logistics center. All e-commerce order data is written to the same topic queue. If there are 5 logistics centers nationwide (Northeast, Northwest, Southwest, South China, and Central), then there will be 5 topic queues corresponding to each logistics center.

[0077] If you need to add a new logistics center (such as the North China region), you need to complete the following steps:

[0078] 1. Add a new topic queue to the external message queue to receive e-commerce order data that should be sent to the "North China" logistics region.

[0079] 2. Add a new "rule design file" to the rule engine, write the corresponding rules in it, and send the orders from the North China region to the topic queue added in step 1 above. There is no need to change the source code of the data synchronization module and the e-commerce system or republish the program.

[0080] 3. Add a new synchronized system, "North China Logistics Center", and subscribe to and listen to the topic queue added in step 1 above.

[0081] To enhance the system's ability to monitor errors and faults, error logs must be written to the log system if message processing fails or anomalies occur at the logistics center or member center. Through monitoring, operations engineers can promptly identify and resolve issues that arise during message processing.

[0082] As an example, the log system is implemented using the distributed search and analysis engine Elasticsearch, the data analysis and visualization platform Kibana, and the Logstash toolchain. Elasticsearch is used to store logs, Kibana is used to provide a graphical log access interface, and Logstash is used to pull logs from the logistics center or member center. The logistics center and member center can also use Log4j to write logs directly to Elasticsearch.

[0083] This application enables flexible data synchronization from a data source system to multiple synchronized systems. It allows for dynamic changes to the transmitted data format as needed without stopping the system, enhancing the adaptability of synchronized data. This implementation method helps simplify data synchronization, strengthens the configurability of data synchronization, and improves the efficiency of data synchronization.

[0084] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A method of data synchronization, the method comprising: The method comprises the following steps: receiving data to be synchronized provided by a first system; processing the data to be synchronized by using at least one first rule design file in a rule engine to obtain subject messages corresponding to the first rule design file; sending all the subject messages to an external message queue deployed in a cluster mode, each of the subject messages being sent to a corresponding first subject queue in the external message queue, and the subject messages in the first subject queue being for consumption by at least one second system adapted to the first subject queue; wherein the message format and the message content of all the subject messages in each of the first subject queues are completely consistent; wherein each of the first subject queues comprises at least one first message group, and the number of the first message groups is the same as the number of the second systems consuming the first subject queues; receiving the message format and the message content of a newly added third system, the third system being used for consuming the subject messages in the external message queue; if the message format and the message content of the third system do not match the message format and the message content corresponding to all the first subject queues, adding a second rule design file corresponding to the third system in the rule engine; if the data to be synchronized is received, processing the data to be synchronized to synchronously obtain first subject messages corresponding to all the first rule design files and second subject messages corresponding to all the second rule design files; sending the first subject messages to the first subject queues and sending the second subject messages to a newly added second subject queue in the external message queue; wherein if the message format and the message content of the third system match the message format and the message content corresponding to one of the third subject queues, a second message group corresponding to the third system is added in the third subject queue; wherein a second rule design file corresponding to the third system is added in the rule engine, and a second subject queue is added in the external message queue, the second rule design file corresponding to the second subject queue.

2. The data synchronization method of claim 1, wherein, The method further comprises the following steps: if an exception occurs when the second system or the third system processes the subject messages, writing the exception processing condition into a log system managed and maintained in a centralized manner.

3. A data synchronization system, characterized by The method comprises a data synchronization module and an external message queue deployed in a cluster mode; the input data of the data synchronization module is data to be synchronized provided by a fourth system, the data synchronization module comprises a rule engine, and the rule engine comprises at least one third rule design file, each of the third rule design files being used for processing the data to be synchronized into subject messages corresponding to the third rule design file; the external message queue comprises fourth subject queues corresponding to all the third rule design files in a one-to-one manner, the input data of each of the fourth subject queues being the subject messages corresponding thereto, and the subject messages in the fourth subject queues being for consumption by at least one fifth system adapted to the fourth subject queues; wherein the message format and the message content of all the subject messages in each of the fourth subject queues are completely consistent. Each fourth topic queue includes at least one third message group, and the number of the third message groups is the same as the number of fifth systems consuming the fourth topic queue; The data synchronization module includes a receiving module, an adding module, a data processing module, and a sending module; The receiving module is configured to receive the message format and the message content of the added sixth system, and the sixth system is configured to consume the topic message in the external message queue; The adding module is configured to add a fourth rule design file corresponding to the sixth system in the rule engine when the message format and the message content of the sixth system do not match the message format and the message content corresponding to all fourth topic queues; The data processing module is configured to process the to-be-synchronized data, and synchronously obtain third topic messages corresponding to all third rule design files and fourth topic messages corresponding to all fourth rule design files; The sending module is configured to send the third topic messages to the fourth topic queue, and send the fourth topic messages to a fifth topic queue added in the external message queue; The adding module is further configured to add a fourth message group corresponding to the sixth system in one of the sixth topic queues when the message format and the message content of the sixth system match the message format and the message content corresponding to the sixth topic queue.

4. The data synchronization system of claim 3, wherein, The log system is further included and is configured to record the abnormal processing of all synchronized systems.

Images