Data synchronization method and device, equipment and medium

Abstract

The invention discloses a data synchronization method, device and equipment and a medium, which are used for realizing data synchronization between a source system and a target system, and the method comprises the following steps: S1, setting a configuration center, and creating a corresponding frequency synchronization queue and a corresponding data synchronization queue for the source system and the target system; s2, setting a time sequence, and updating a frequency synchronization queue based on the time sequence; s3, the source system and the target system monitor own frequency synchronization queues and update synchronization frequencies; s4, the source system pushes the synchronous data converted into the same format into a message queue; and S5, the target system monitors the message queue and obtains synchronous data. The method has the advantages that the independent frequency synchronization queue of each system is set in the message queue, frequency synchronization is achieved, the target system consumes the data of the required frequency, the synchronization process is stable and efficient, and the integrity and accuracy of the data are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data synchronization method, device, equipment and medium. Background Art

[0002] In software systems, data is of paramount importance; when multiple systems collaborate, each system uses different data sources and different data formats, which will cause data inconsistency during the collaboration process. Inconsistent data will bring very high risks to system business judgment. For example, if the sales team and the customer service team use different systems, customer information and order data may be out of sync, resulting in customer service staff being unable to provide accurate services and sales staff being unable to accurately follow up on customer needs. If the finance department and the procurement department use different systems, it may cause data errors, payment delays and other problems.

[0003] In summary, there is an urgent need for a data synchronization method, device, equipment and medium to solve the problems existing in the prior art. Summary of the invention

[0004] The present invention aims to provide a data synchronization method, device, equipment and medium. The specific technical solution is as follows:

[0005] A data synchronization method is used to achieve data synchronization between a source system and a target system. The steps of the method are as follows:

[0006] Step S1: Setting a configuration center, wherein the configuration center creates corresponding frequency synchronization queues and data synchronization queues in the message queue for the source system and the target system, and the source system and the target system connect to the frequency synchronization queues and data synchronization queues corresponding to themselves in the message queue;

[0007] Step S2: The configuration center sets a time sequence, and the message queue updates the frequency synchronization queue and the data synchronization queue based on the time sequence;

[0008] Step S3: The source system and the target system monitor their own frequency synchronization queues in the message queue, and update their own synchronization frequencies based on the frequency synchronization queues;

[0009] Step S4: The source system converts the data into synchronization data of the same format, and pushes the synchronization data to the message queue according to the updated frequency;

[0010] Step S5: The target system monitors the message queue, obtains synchronization data through the data synchronization queue, and stores the synchronization data as consumption data.

[0011] In a specific preferred implementation case, the message queue is a Kafka message queue.

[0012] In a specific preferred implementation case, a SASL secure connection is configured in the Kafka message queue for pushing data or consuming data.

[0013] In a specific preferred implementation case, the time series is a cron expression.

[0014] In a specific preferred implementation case, the synchronization data is in JSON data format.

[0015] In a specific preferred implementation case, the frequency synchronization queue and the data synchronization queue include authentication information for authenticating a source system or a target system, and the authentication information includes an account, a key, and a queue name.

[0016] In addition, the present invention also discloses a data synchronization device for implementing the above-mentioned data synchronization method, wherein the data synchronization device includes a message queue module and a configuration center;

[0017] The message queue module includes a frequency synchronization queue, a data synchronization queue and an authentication submodule; the frequency synchronization queue is used to update its own synchronization frequency according to the time series, the data synchronization queue is used for the target system to consume data, and the authentication submodule is used to configure the account, key and queue name for the frequency synchronization queue and the message synchronization queue, and determine whether the authentication information corresponds;

[0018] The configuration center includes a queue management module and a frequency synchronization module; the queue management module is used to create corresponding frequency synchronization queues and data synchronization queues in the message queues for the source system and the target system, and the frequency synchronization module is used to configure the time series.

[0019] In addition, the present invention also discloses a computer device, including a memory and a processor;

[0020] The memory stores a computer program;

[0021] When the processor executes the computer program, the steps of the above method are implemented.

[0022] In addition, the present invention also discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0023] The application of the technical solution of the present invention has the following beneficial effects:

[0024] (1) The method of the present invention discloses setting a frequency synchronization queue in a message queue, and each source system or target system has its corresponding frequency synchronization queue, so as to realize frequency synchronization between the source system and the target system. The target system can consume data of the required frequency. The synchronization process of the present invention is stable and efficient, and the integrity and accuracy of the data are guaranteed.

[0025] (2) The method of the present invention adopts a unified data format to realize data push and consumption, which is conducive to realizing a dynamically adjustable frequency synchronization strategy; the preferred JSON data format of the present invention is easy to machine parse and generate, which effectively improves the network transmission efficiency.

[0026] (3) The method of the present invention uses the configuration center and the message queue as the middle layer for data push and consumption, thereby realizing decoupled interaction between the source system and the target system, that is, multiple source systems can realize free data interaction with multiple target systems.

[0027] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a flowchart of the steps of a data synchronization method in a preferred embodiment of the present invention;

[0030] Figure 2 is a simplified diagram of a data synchronization architecture in a preferred embodiment of the present invention;

[0031] Figure 3 It is a schematic diagram of the flow of message queue connection in a preferred embodiment of the present invention;

[0032] Figure 4 It is a schematic diagram of the process of persisting target system data in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0033] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0034] Example:

[0035] See also Figure 1 This embodiment discloses a data synchronization method for realizing data synchronization between a source system and a target system. The steps of the method are as follows:

[0036] Step S1: Set up a configuration center, which creates corresponding frequency synchronization queues and data synchronization queues in the Kafka message queue for the source system and the target system. The source system and the target system connect to the frequency synchronization queues and data synchronization queues corresponding to themselves in the Kafka message queue; the frequency synchronization queues and data synchronization queues include authentication information for authenticating the source system or the target system, and the authentication information includes an account, a key, and a queue name.

[0037] Step S2: The configuration center sets a cron expression as a time series, and the message queue updates the frequency synchronization queue and the data synchronization queue based on the cron expression.

[0038] Step S3: The source system and the target system monitor their own frequency synchronization queues in the Kafka message queue, and update their own synchronization frequencies based on the frequency synchronization queues.

[0039] Step S4: The source system converts the data into synchronization data in JSON data format, and pushes the synchronization data to the Kafka message queue according to the updated frequency;

[0040] Step S5: The target system monitors the Kafka message queue, obtains synchronization data through the data synchronization queue, and stores the synchronization data as consumption data. It should be noted that in this embodiment, the target system can process synchronization data in JSON format through FastJson.

[0041] Preferably, a SASL secure connection is configured in the Kafka message queue for pushing data or consuming data.

[0042] like Figure 2 As shown, the data synchronization framework implemented in this embodiment includes a source system, a message queue, a configuration center and a target system;

[0043] Among them, the source system serves as the data producer, which stores production data internally, and is also provided with a JSON data conversion module for converting the production data into JSON format and a synchronization frequency configuration module for monitoring the frequency synchronization queue corresponding to itself in the message queue. After monitoring the update frequency of the frequency synchronization queue, the synchronization frequency configuration module dynamically updates its own frequency in real time, and retrieves the production data of the corresponding frequency as synchronization data and pushes it to the message queue.

[0044] Furthermore, the message queue includes a data synchronization queue, a frequency synchronization queue and an authentication sub-module. The data synchronization queue is used to realize the synchronization data in the target system consumption message queue. The frequency synchronization queue realizes its own frequency adjustment according to the configuration center. The authentication sub-module is used to realize the source system or the target system connecting to the message queue, thereby realizing the source system or the target system pushing or listening to the message queue.

[0045] like Figure 3 As shown, the specific process of connecting the source system or the target system to the message queue in this embodiment is as follows: the configuration center creates a corresponding frequency synchronization queue and data synchronization queue in the message queue, and the source system and the target system connect to the frequency synchronization queue and data synchronization queue corresponding to themselves in the Kafka message queue; the frequency synchronization queue and the data synchronization queue include authentication information for authenticating the source system or the target system, and the authentication information includes an account, a key, and a queue name. The message queue pushes the authentication information to the source system or the target system, and the source system or the target system requests to connect to the message queue according to the authentication information. The authentication submodule in the message queue authenticates the corresponding information and determines whether the connection is successful.

[0046] Furthermore, the configuration center includes a queue management module and a frequency synchronization module, wherein the queue management module is used to create a frequency synchronization queue and a data synchronization queue, and the frequency synchronization module is used to send a cron expression to the message queue, and the frequency synchronization queue in the message queue updates its own synchronization frequency according to the cron expression.

[0047] The cron expression consists of 6 parts, which respectively represent the execution time of the scheduled task. The format of the cron expression is: <seconds><minutes><hours><days><months><weeks>.

[0048] The value range and meaning of each part are as follows: Seconds (0-59); Minutes (0-59); Hours (0-23); Day (1-31); Month (1-12); Week (0-6, 0 for Sunday, 1 for Monday, 2 for Tuesday, 3 for Wednesday, 4 for Thursday, 5 for Friday, 6 for Saturday)

[0049] Among them, seconds, minutes, and hours represent specific times, while days, months, and weeks indicate that wildcards can be used. Commonly used expressions include the following examples:

[0050] 0 / 15****?: Execute every 15 seconds;

[0051] 00 / 5***?: Execute every 5 minutes;

[0052] 0012**?: Executed at 12 noon every day;

[0053] 01510? **: Executed at 10:15 am every day;

[0054] 01510**?: Executed at 10:15 am every day;

[0055] 00 / 514**?: Execute every 5 minutes between 2pm and 2:55pm every day;

[0056] 00-514**?: Execute every 1 minute between 2pm and 2:05pm every day;

[0057] 00 / 514,18**?: Execute every 5 minutes between 2pm and 2:55pm and between 6pm and 6:55pm every day;

[0058] 00-514,18**?: Execute every 1 minute between 2pm and 2:05pm and between 6pm and 6:05pm every day;

[0059] 00 / 514-18**?: Execute every 5 minutes between 2pm and 6:55pm every day.

[0060] Furthermore, the target system includes a JSON data storage module and consumption data, wherein the JSON data storage module is used to persistently store consumption data, and the consumption data is synchronous data consumed from a message queue.

[0061] like Figure 4 As shown, the specific process of data persistence in the target system is as follows: the source system generates production data according to the configured synchronization frequency and pushes it to the message queue. The target system listens to the message queue and obtains synchronization data through the data synchronization queue. In this embodiment, the target system processes the synchronization data in JSON format through FastJson and stores it persistently.

[0062] In addition, the present invention also discloses a data synchronization device for implementing the above-mentioned data synchronization method, wherein the data synchronization device includes a Kafka message queue module and a configuration center;

[0063] The Kafka message queue module includes a frequency synchronization queue, a data synchronization queue and an authentication submodule; the frequency synchronization queue is used to update its own synchronization frequency according to the time series, the data synchronization queue is used for the target system to consume data, and the authentication submodule is used to configure the account, key and queue name for the frequency synchronization queue and the message synchronization queue, and determine whether the authentication information corresponds;

[0064] The configuration center includes a queue management module and a frequency synchronization module; the queue management module is used to create corresponding frequency synchronization queues and data synchronization queues in the Kafka message queue for the source system and the target system, and the frequency synchronization module is used to configure the time series.

[0065] In addition, the present invention also discloses a computer device, including a memory and a processor;

[0066] The memory stores a computer program;

[0067] When the processor executes the computer program, the steps of the above method are implemented.

[0068] In addition, the present invention also discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data synchronization method, It is characterized in that For realizing data synchronization between a source system and a target system, the method has the following steps: Step S1: Setting a configuration center, wherein the configuration center creates corresponding frequency synchronization queues and data synchronization queues in the message queue for the source system and the target system, and the source system and the target system connect to the frequency synchronization queues and data synchronization queues corresponding to themselves in the message queue; Step S2: The configuration center sets a time sequence, and the message queue updates the frequency synchronization queue and the data synchronization queue based on the time sequence; Step S3: The source system and the target system monitor their own frequency synchronization queues in the message queue, and update their own synchronization frequencies based on the frequency synchronization queues; Step S4: The source system converts the data into synchronization data of the same format, and pushes the synchronization data to the message queue according to the updated frequency; Step S5: The target system monitors the message queue, obtains synchronization data through the data synchronization queue, and stores the synchronization data as consumption data.

2. The data synchronization method according to claim 1, It is characterized in that The message queue is a Kafka message queue.

3. The data synchronization method according to claim 2, It is characterized in that Configure SASL secure connection in Kafka message queue to push or consume data.

4. The data synchronization method according to claim 1, It is characterized in that The time series is a cron expression.

5. The data synchronization method according to claim 1, It is characterized in that The synchronization data is in JSON data format.

6. The data synchronization method according to claim 1, It is characterized in that The frequency synchronization queue and the data synchronization queue include authentication information for authenticating a source system or a target system, and the authentication information includes an account, a key, and a queue name.

7. A data synchronization device, It is characterized in that Used to implement the data synchronization method according to any one of claims 1 to 6, the data synchronization device includes a message queue module and a configuration center; The message queue module includes a frequency synchronization queue, a data synchronization queue and an authentication submodule; the frequency synchronization queue is used to update its own synchronization frequency according to the time series, the data synchronization queue is used for the target system to consume data, and the authentication submodule is used to configure the account, key and queue name for the frequency synchronization queue and the message synchronization queue, and determine whether the authentication information corresponds; The configuration center includes a queue management module and a frequency synchronization module; the queue management module is used to create corresponding frequency synchronization queues and data synchronization queues in the message queues for the source system and the target system, and the frequency synchronization module is used to configure the time series.

8. A computer device, It is characterized in that including memory and processor; The memory stores a computer program; When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, It is characterized in that A computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images