A method and system for data synchronization based on double queues

Abstract

The application discloses a data synchronization method and system based on double queues, which comprises a first business system and a second business system, and further comprises a first data synchronization application end and a second data synchronization application end; the first business system sends a first message of changed data to a first message queue, wherein the first message comprises an ID value corresponding to the changed data and a change type; the first data synchronization application end is used for querying a first database in the first business system to obtain the changed data according to the first message sent by the first message queue, and then converting the changed data and the first message into a second message and sending the second message to a second message queue; the second data synchronization application end is used for checking the second message sent by the second message queue, and after the checking succeeds, synchronizing the changed data to a second database in the second business system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, in particular to a data synchronization method and system based on double queues. BACKGROUND

[0002] Data synchronization in an information system is a technology of extracting data from a source database to a target database, commonly used for data sharing between different business systems or data extraction of a data warehouse in big data analysis. Data synchronization can be generally divided into real-time data synchronization and timing data synchronization. Real-time data synchronization means that the target database can change in real time when the source database changes. Timing data synchronization means that the target database changes after a certain time interval after the source database changes.

[0003] Common technical solutions for real-time data synchronization include real-time interface calling and real-time database log parsing synchronization. Timing data synchronization generally sets a data synchronization time interval, such as 5 minutes, half an hour, or a day, and requires the use of an ETL tool to write a data extraction script to extract data from the source database at regular intervals.

[0004] If real-time data synchronization technology uses real-time interface calling to synchronize data, a data synchronization interface needs to be added to the target business system, and the source business system needs to call the interface of the target business system after each data change. This has the problems of large modification workload, impact on the performance of the source business system, and difficulty in modifying a business system that has been running for a long time.

[0005] If real-time data synchronization technology uses the technical solution of real-time database log parsing synchronization, no modification is needed for the source business system and the target business system. However, the log formats of databases in different business systems are not the same, and parsing the log and extracting data requires a log parsing technical solution for each type of database, which has a high technical threshold and is usually difficult to implement.

[0006] Timing data synchronization technology extracts data at regular intervals, and generally requires that once the data in the source database changes, the time field of the data table must also be updated synchronously. If the time field does not change, the ETL tool cannot capture the data change, and the target database will lose the data change. At the same time, if the ETL tool fails, the synchronization time needs to be adjusted manually, otherwise the target database will lose the data. Timing data synchronization requires the source business system to update the time field of the data table for each data change, and the time for the ETL tool to complete data extraction cannot exceed the timing interval. If these conditions are not met, the target database will lose data.

[0007] Chinese patent (CN 111813868 A) discloses a data synchronization method and device, by generating a first message queue in the business service end, obtaining the first data change message; obtain the data to be synchronized associated with the first data change message; update the local data by using the data to be synchronized; generate the second data change message for the update operation; the second data change message is published to the second message queue, so that the application service end associated with the storage service end carries out data update according to the second data change message. The patent needs to add storage service end between business service end and application service end to update and store local data according to time stamp, needs to add various modules in multiple service ends, the transformation amount is large; and the data is synchronized together in the synchronization process of two message queues, so that the processing amount will be large, and the synchronization efficiency is reduced. SUMMARY

[0008] In view of the low data synchronization efficiency in the prior art two business systems, the present application provides a data synchronization method and system based on double queue, which realizes data synchronization by adopting double queue technical scheme, maximally reduces the transformation of source business system and target business system, reduces the technical threshold of real-time data synchronization, makes the technical scheme more easily implemented, and improves the synchronization efficiency.

[0009] In order to achieve the above purpose, the present application provides the following technical scheme:

[0010] A data synchronization system based on double queue, comprising a first business system and a second business system, further comprising a first data synchronization application end and a second data synchronization application end;

[0011] The first business system sends the first message of change data to the first message queue, and the first message comprises the ID value and the change type corresponding to the change data;

[0012] The first data synchronization application end is used for querying the first database in the first business system to obtain the change data according to the first message sent by the first message queue, and then converting the change data and the first message into the second message and sending it to the second message queue;

[0013] The second data synchronization application end is used for checking the second message sent by the second message queue, and after the checking succeeds, synchronizing the change data to the second database in the second business system.

[0014] Preferably, the first business system is internally provided with a data synchronization agent class, which is used for real-time capturing the data change of the first database, and outputting the ID value and the change type corresponding to the change data to the first message queue.

[0015] Preferably, the data synchronization agent class adds annotations in the first database addition, deletion and modification method by using springAop aspect proxy technology, parses the values of the annotations, and finally obtains the ID value and the change type of the changed data from the parsing result by using the reflection technology of Jdk.

[0016] Preferably, the annotations include a queue name, an entity type and a field name of an ID.

[0017] The application also provides a data synchronization method based on double queues, comprising the following steps:

[0018] S1: A data synchronization agent class uses springAop aspect proxy technology to add annotations in the first database addition, deletion and modification method; when the first database changes, the changed data is captured, the annotations on the addition, deletion and modification method are parsed to obtain a first message, and the first message is output to a first message queue, wherein the first message includes an ID corresponding to the changed data and a change type;

[0019] S2: A first data synchronization application end obtains the first message from the first message queue, queries the first database according to the first message to obtain the changed data, converts the changed data into a second message, and sends the second message to a second message queue;

[0020] S3: After the second data synchronization application end obtains the second message from the second message queue, the changed data is synchronized to a second database of a second business system after verification.

[0021] Preferably, the S1 specifically comprises:

[0022] S1-1: A data synchronization agent class uses springAop aspect proxy technology to add annotations in the first database addition, deletion and modification method, wherein the annotations include a queue name, an entity type and a field name of an ID;

[0023] S1-2: After the first database addition, deletion and modification method is called, the changed data is captured in real time and the values of the annotations are parsed, and finally the ID of the changed data is obtained from the parsing result by using the reflection technology of Jdk, and the ID and the entity type are packaged into a first message;

[0024] S1-3: The first message is sent to the first message queue in the format of json.

[0025] Preferably, the S2 specifically comprises:

[0026] S2-1: A first message queue is subscribed, and the first message pushed by the data synchronization agent class is received in real time;

[0027] S2-2: According to the first message, directly query the entity table or the table association in the first database in the first business system to obtain the change data;

[0028] S2-3: Pack the change data into the second message, and push to the second message queue.

[0029] Preferably, the S3 specifically comprises:

[0030] S3-1: Subscribe to the second message queue, and receive the second message pushed by the first data synchronization application in real time;

[0031] S3-2: Verify the second message, and after verification, synchronize the change data to the second database of the second business system.

[0032] As described above, compared with the prior art, the present application has at least the following beneficial effects:

[0033] The present application uses the technical scheme of double queues to complete data synchronization, maximally reduces the modification of the source business system and the target business system, reduces the technical threshold of real-time data synchronization, makes the technical scheme more easily implemented, and improves the synchronization efficiency.

[0034] The data synchronization agent class in the first business system only obtains the ID value and the entity type corresponding to the change data, does not need to transmit the change data, only transmits the entity type and the entity ID value of the change data, maximally reduces the performance influence on the first business system, reduces the data transmission time, and improves the synchronization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a schematic diagram of a double-queue-based data synchronization system according to an exemplary embodiment of the present application.

[0036] Figure 2 It is a schematic diagram of a double-queue-based data synchronization method according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in combination with embodiments and specific implementation manners. However, it should not be understood as the scope of the above-mentioned subject matter of the present application being limited to the following embodiments only, and any technology realized based on the content of the present application belongs to the scope of the present application.

[0038] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0039] As shown in Figure 1 The present application provides a double-queue-based data synchronization system, which comprises a first business system 10, a first data synchronization application end 30, a second data synchronization application end 50 and a second business system 60 connected in sequence.

[0040] The purpose of the present application is to realize data synchronization between the first business system 10 and the second business system 60, and only the first business system 10 needs to be modified, without any modification of the second business system 60, thereby saving costs.

[0041] In this embodiment, the first business system 10 is provided with a first database 101 and a data synchronization agent class 102. The first database 101 is used to store data in the first business system 10; the data synchronization agent class 102 is used to capture data changes of the first database 101 and output the ID value and change type corresponding to the changed data to the first message queue 20.

[0042] In this embodiment, the data synchronization agent class 102 uses springAop aspect proxy technology to weave data synchronization logic into the add, delete and modify methods of the first database 101; for example, an annotation (including queue name, entity type, field name of Id, etc.) is added in the add, delete and modify methods, and the annotation is used to identify which method will cause data modification, that is, by parsing the value of the annotation, the Id value of the changed data is finally obtained from the return result through the reflection technology of Jdk.

[0043] The specific implementation steps of the data synchronization agent class 102 are as follows:

[0044] 1. Define an annotation class DataSync, which defines the queue name, entity type and field name of Id in the annotation;

[0045]

[0046] 2. The data synchronization agent class needs to dynamically proxy the class using the annotation @DataSync, then parse the value of the annotation, and finally obtain the Id value of the entity class from the return result through the reflection technology of Jdk, and publish the id value and entity type to the first message queue, and the specific implementation is as follows:

[0047]

[0048]

[0049] 3. The entity should be returned by the add, delete and update methods, and the correct value should be ensured for the annotation @DataSync.

[0050] 4. The data is sent to the first message queue in the format of json, specifically as follows:

[0051]

[0052] The first data synchronization application end 30 is configured to obtain the ID value from the first message queue 20, query the first database 101 in the first business system 10 to obtain the changed data, and send the changed data to the second message queue 40.

[0053] In the embodiment, the data synchronization proxy class 102 only obtains the ID value and the entity type of the changed data in the first business system 10, thereby minimizing the impact on the performance of the first business system 10; the first message queue 20 only needs to transmit the ID value and the entity type, without transmitting the data to be synchronized, thereby reducing the time of data synchronization and improving the synchronization efficiency. The first data synchronization application end 30 can perform any complex data query on the first database 101 by using the ID value, thereby avoiding the impact on the performance of the first business system 10 and having sufficient flexibility.

[0054] The second data synchronization application end 50 is configured to obtain the changed data from the second message queue 40 and perform verification (there are many business fields, which have certain business restrictions, and the data fields are subjected to non-empty verification, numerical value verification, length range verification, time string format verification, enumeration value verification, etc. according to different business requirements, for example, must be greater than or equal to a certain threshold or less than a certain threshold); after the verification is successful, the changed data is updated to the second database 601 in the second business system 60, thereby realizing the real-time synchronization of the data change of the second business system 60 after the data change of the first business system 10; if the verification is unsuccessful, the data that fails the verification is temporarily stored in the second database of the second business system 60, and after the reason for the unsuccessful verification is found, the data that fails the verification is corrected and manually sent to the first message queue, and the second data synchronization application end 50 obtains the corrected data and completes the operation of storing in the database.

[0055] As shown in Figure 2 , based on the above system, the embodiment further provides a data synchronization method based on double queues, specifically including the following steps:

[0056] S1: The data synchronization agent class queries the first database of the first business system in real time to capture change data, and outputs a first message to a first message queue, the first message including an ID and a change type of the change data.

[0057] In this embodiment, to reduce the performance impact on the first business system and improve data synchronization efficiency, only the ID and the change type of the change data are transmitted in the first message, and the change data is not transmitted. The specific steps are as follows:

[0058] S1-1: An annotation is added in an add, delete or modify method of the first database, including defining a queue name, an entity type and a field name of an ID.

[0059] S1-2: The first database is queried in real time to capture change data, the values of the annotations are parsed, and finally the first message is obtained from the returned result by using the reflection technology of Jdk, including the ID of the change data.

[0060] In this embodiment, the values of the annotations are obtained by using an API provided by SpringAop technology, and the values of the annotations include the queue name, the entity type and the field name of the ID.

[0061] S1-3: The first message is sent to the first message queue in the format of json.

[0062] S2: The first data synchronization application end obtains the first message from the first message queue, queries the first database to obtain change data according to the first message, packs the change data into a second message, and sends the second message to a second message queue.

[0063] In this embodiment, the first data synchronization application end is an independently deployed application, and the specific steps for development and implementation are as follows:

[0064] S2-1: The first message queue is subscribed to, so that the first message pushed by the data synchronization agent class can be received in real time.

[0065] S2-2: Different data processing classes are designed and developed according to different entity types. After the entity ID value is obtained, the entity table can be directly queried in the business database A, or a join table association query can be performed.

[0066] S2-3: After the change data is queried according to the ID value, the change data and the first message are assembled and converted into the second message, and the second message is pushed to the second message queue.

[0067] S3: After the second data synchronization application end obtains the second message from the second message queue, the change data is synchronized to the second database of the second business system after verification.

[0068] In the embodiment, the second data synchronization application end is also an independently deployed application, and the specific steps for development and implementation are as follows:

[0069] S3-1: Subscribing to the second message queue to ensure that the second message pushed by the first data synchronization application end can be received in real time.

[0070] S3-2: The second message is checked, and after the check is passed, the change data is converted into a data format recognized by the second business system, and the change data of the second database is completed.

[0071] In the embodiment, after the second message check is passed, whether the change data needs to be converted depends on the implementation of the first data synchronization application end. If the data conversion adapted to the second business system has been completed in the first data synchronization application, the second data synchronization application does not need to perform data conversion, and after the check is passed, the data is directly stored in the database. If the first data synchronization application does not perform the corresponding conversion, the data needs to be converted and stored in the database after the check is passed. The data conversion is prior art, that is, mapping data of one type or one format into data of another type or another format.

[0072] Through experiments, the second business system can basically obtain the data change of the first business system within 2 seconds, effectively improving the data synchronization efficiency.

[0073] Those skilled in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A dual-queue based data synchronization system comprising a first business system and a second business system, characterized in that, The first data synchronization application end and the second data synchronization application end are further included. The first service system sends a first message of the change data to a first message queue, and the first message includes an ID value and a change type corresponding to the change data; The first data synchronization application end is configured to query a first database in the first service system to obtain the change data according to the first message sent by the first message queue, and convert the change data and the first message into a second message and send the second message to a second message queue; The second data synchronization application end is configured to check the second message sent by the second message queue, and synchronize the change data to a second database in the second service system after the checking succeeds; The first service system is internally provided with a data synchronization agent class, which is configured to capture data changes of the first database in real time, and output an ID value and a change type corresponding to the change data to the first message queue; The data synchronization agent class uses springAop aspect proxy technology to add an annotation in the increase, decrease or modification method of the first database, obtains the ID value and the change type of the change data from the parsing result through the reflection technology of Jdk by parsing the value of the annotation.

2. The dual-queue based data synchronization system of claim 1, wherein, The annotation includes a queue name, an entity type and an ID field name.

3. A dual-queue based data synchronization method based on the system of any of claims 1-2, characterized in that, The method comprises the following steps: S1: The data synchronization agent class uses springAop aspect proxy technology to add an annotation in the increase, decrease or modification method of the first database; When the first database changes, the change data is captured, the annotation on the increase, decrease or modification method is parsed to obtain a first message, and the first message is output to the first message queue, wherein the first message includes an ID corresponding to the change data and a change type; S2: The first data synchronization application end obtains the first message from the first message queue, queries the first database according to the first message to obtain the change data, converts the change data into a second message, and sends the second message to the second message queue; S3: The second data synchronization application end obtains the second message from the second message queue, checks the second message, and synchronizes the change data to the second database of the second service system after the checking succeeds.

4. The method of claim 3, wherein, The S1 specifically comprises: S1-1: The data synchronization agent class uses springAop aspect proxy technology to add an annotation in the increase, decrease or modification method of the first database, and the annotation includes a queue name, an entity type and an ID field name; S1-2: After the increase, decrease or modification method of the first database is called, the change data is captured in real time and the value of the annotation is parsed, and finally the ID of the change data is obtained from the parsing result through the reflection technology of Jdk, and the ID and the entity type are packaged into a first message; S1-3: The first message is sent to the first message queue in the format of json.

5. The method of claim 3, wherein the method further comprises: The S2 specifically comprises: S2-1: The first message queue is subscribed, and the first message pushed by the data synchronization agent class is received in real time; S2-2: The change data is obtained by directly querying an entity table or a table association in the first database in the first service system according to the first message; S2-3: The change data is packaged into a second message and pushed to the second message queue.

6. The method of claim 3, wherein the method further comprises: The S3 specifically comprises: S3-1: subscribe to the second message queue, receive the second message pushed by the first data synchronization application in real time; S3-2: check the second message, and after the check is passed, synchronize the change data to the second database of the second business system.

Images