A data synchronization pattern across multiple data source types that supports distributed deployment

Abstract

The application relates to a data synchronization mode supporting distributed deployment across multiple data source types, including a synchronization request triggering mode and a synchronization request processing mode, the synchronization request triggering mode including a real-time triggering mode, a timing triggering mode and a manual triggering mode; the synchronization request processing mode includes the following steps: obtaining a data structure of a synchronization request from a message middleware, forming a request parameter; according to a verification rule, verifying whether the request parameter is legal, if the request parameter is not legal, recording a failure log, and ending the synchronization request; if the request parameter is legal, obtaining a mapping process definition file corresponding to the synchronization request according to the request parameter, and querying general variables of an execution task corresponding to the synchronization request according to the request parameter; starting an execution thread of a specific data mapping processing process. The application has the effects of realizing data synchronization among different systems without affecting the business of each system, and reducing the coupling among various business systems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer application, and particularly relates to a data synchronization mode supporting distributed deployment and crossing multiple data source types. BACKGROUND

[0002] With the explosive growth of data, the pressure of data storage is more and more obvious, and the loss caused by information data loss is also more and more large, and the attention is also more and more. Correspondingly, the data centers in different regions generally need special network or Internet for data transmission. In the current network background, multiple businesses often share a network, and the load condition of the network link is unpredictable. In addition, the upper business system is generally 7*24 hours running, the data generation is real-time, and the quantity and the size of a single file are usually large.

[0003] After the unified management of the business data of an enterprise, how to flexibly and efficiently distribute and use, how to realize the broadcast synchronization between the databases of multiple service modules in the micro-service architecture, how to analyze the data in multiple types of data sources, how to decouple between modules based on the process of data processing between business modules, and other problems encountered in the process of informatization construction are a big problem that needs to be solved at present. SUMMARY

[0004] The application has the advantages of realizing data synchronization between different systems without affecting the business of each system, and reducing the coupling between each business system.

[0005] A data synchronization mode supporting distributed deployment and crossing multiple data source types, comprising a synchronization request triggering mode and a synchronization request processing mode, the synchronization request triggering mode comprising a real-time triggering mode, a timing triggering mode and a manual triggering mode, and being used for processing received data sources into a unified data structure and sending the unified data structure to a message middleware;

[0006] The synchronization request processing mode comprises the following steps:

[0007] The task executor obtains the data structure of the synchronization request from the message middleware, and forms a request parameter;

[0008] According to the verification rule, whether the request parameter is legal is verified, if the request parameter is not legal, a failure log is recorded, and the synchronization request is ended; if the request parameter is legal, a mapping process definition file corresponding to the synchronization request is obtained according to the request parameter, a general variable of the execution task corresponding to the synchronization request is queried according to the request parameter, and an execution thread of a specific data mapping processing process is started.

[0009] As a further description of the application: the execution thread comprises: judging whether the synchronization request needs to be delayed for execution, if the synchronization request is non-delayed execution or there is a non-delayed execution identifier, then an immediate execution process is performed; if the synchronization request is delayed execution, then a delayed execution process is performed.

[0010] As a further description of the application: the delayed execution process comprises the following steps:

[0011] Obtaining an exclusive lock of a buffer interval waiting queue corresponding to the synchronization request;

[0012] Saving the request parameters corresponding to the synchronization request in the set of buffer interval waiting queues corresponding to the synchronization request;

[0013] Releasing the exclusive lock at the buffer interval level, forming a successful log of inserting the buffer waiting queue and recording;

[0014] Obtaining all pending execution tasks in the buffer interval waiting queue, sending a message to the message middleware, and performing a non-delayed execution identifier on all pending execution tasks in the buffer interval waiting queue.

[0015] As a further description of the application: the immediate execution process comprises the following steps:

[0016] Obtaining an exclusive lock at the data body and mapping process level;

[0017] If successful, obtaining the last successful execution timestamp of the task file of the execution task corresponding to the synchronization request, obtaining the mapping process definition file corresponding to the synchronization request, decompressing the mapping process definition file to obtain the mapping process definition, executing the mapping process definition to obtain the exclusive lock of the waiting queue at the mapping process level, processing the unprocessed task data of the execution task in the waiting queue at the mapping process level, releasing the exclusive lock of the waiting queue at the mapping process level, forming a data exchange task execution completion log and recording, and then releasing the exclusive lock at the mapping process level;

[0018] If unsuccessful, obtaining the exclusive lock of the waiting queue at the mapping process level, inserting the unprocessed task data of the execution task into the waiting queue at the level, releasing the exclusive lock of the waiting queue at the mapping process level, and forming a successful log of inserting the waiting queue and recording.

[0019] As a further description of the application: the data structure comprises a data body number, a business system name, an exchange task number, and an exchange task variable.

[0020] As a further description of the application: the request parameters comprise a data body number, a business system name, and an exchange task number.

[0021] As a further description of the application: the check rule includes:

[0022] Detecting whether the data body number exists or not;

[0023] Whether the business system name is empty or not;

[0024] Whether the exchange task number exists or not.

[0025] As a further description of the application: the mapping flow definition file is an xml format file, and is stored in a distributed file system.

[0026] As a further description of the application: the timing trigger mode and the manual trigger mode are both based on embedding a task executor into a data processing service through a distributed task scheduling system, and the data source is processed through the task executor.

[0027] As a further description of the application: the real-time trigger mode is based on AOP technology to monitor the data change in the business system in real time, and when detecting that the data has changed, the data in the business system is processed.

[0028] Compared with the prior art, the application has the following beneficial technical effects:

[0029] 1. The application realizes the data synchronization processing of different business systems by parsing the synchronization request information into data subjects and execution tasks, taking the synchronization request sent by the business party as the starting point and taking the successful synchronization execution of the data as the termination point; the unified management and distribution of business data, the broadcast synchronization between the databases of multiple service modules under the micro-service architecture, the multi-source data summary analysis and the data processing function based on the process between business modules are used, so that the data synchronization between different systems is realized without affecting the business of each system, and the coupling between each business system is reduced.

[0030] 2. The application supports the extraction, cleaning conversion, loading and synchronization processing of data between heterogeneous data sources, and in the three trigger modes of real-time trigger mode, timing trigger mode and manual trigger mode, the business system can perform full or incremental synchronization processing according to the data volume, synchronization frequency and performance requirements of the data, that is, the high-availability data synchronization of limited range data or full data. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the application. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0033] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0034] In addition, the terms "horizontal", "vertical" and the like do not mean that the components must be absolutely horizontal or vertical, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0035] In the description of the present application, it should also be noted that unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above-mentioned terms in the present application can be understood according to the specific circumstances.

[0036] For example, Figure 1As shown, a data synchronization mode supporting distributed deployment across multiple data source types includes a synchronization request triggering mode and a synchronization request processing mode. The synchronization request triggering mode includes a real-time triggering mode, a timing triggering mode, and a manual triggering mode, which are used to send the received data source to a message middleware after processing into a unified data structure. The timing triggering mode and the manual triggering mode both embed a task executor into a data processing service based on a distributed task scheduling system, and process the data source through the task executor. The real-time triggering mode monitors data changes in a business system in real time based on an AOP technology, and processes the data in the business system when detecting a data change, so that the business system can be synchronized in full or incrementally according to the data volume, synchronization frequency, performance, and other requirements in the real-time triggering mode, the timing triggering mode, and the manual triggering mode, i.e., high-availability data synchronization of limited-range data or full data.

[0037] The synchronization request processing mode includes the following steps: obtaining a data structure of a synchronization request from a message middleware to form a request parameter; the data structure includes a data ontology number, a business system name, an exchange task number, and an exchange task variable, and the request parameter includes the data ontology number, the business system name, and the exchange task number. Then, the business system checks the request parameter according to a verification rule, and if the request parameter is not legal, records a failure log and ends the synchronization request; if the request parameter is legal, obtains a mapping process definition file corresponding to the synchronization request according to the request parameter, and queries a general variable of an execution task corresponding to the synchronization request according to the request parameter; starts an execution thread of a specific data mapping processing process; the verification rule includes: detecting whether the data ontology number exists; whether the business system name is empty; and whether the exchange task number exists.

[0038] The execution thread includes: judging whether the synchronization request needs to be delayed for execution, and if the synchronization request is a non-delayed execution or has a non-delayed execution identifier, performing an immediate execution process; if the synchronization request is a delayed execution, performing a delayed execution process.

[0039] Specifically, the delay execution process includes the following steps: acquiring an exclusive lock of a buffer interval waiting queue corresponding to a synchronization request; saving request parameters corresponding to the synchronization request in a set of the corresponding buffer interval waiting queue, and then releasing the exclusive lock at the buffer interval level, forming a successful log of inserting the buffer waiting queue and recording; acquiring all to-be-executed tasks in the buffer interval waiting queue, sending a message to a message middleware, and identifying all to-be-executed tasks in the buffer interval waiting queue for non-delay execution; in this embodiment, the buffer interval is a Key-value type database, wherein the key is a data subject corresponding exchange task number, and the value is a list of request data; specifically, before the addition and deletion operations of the buffer interval waiting queue, a locking operation at the buffer interval level is required, and after the data operation of the buffer interval waiting queue is completed, the lock is released; and when the delay execution synchronization request is received by the task executor, the delay execution synchronization request is first cached in the shared database, and then based on the internally started scheduling task with the buffer interval as the period, the to-be-executed task data in the corresponding buffer interval waiting queue is periodically acquired and executed.

[0040] The immediate execution process includes the following steps: acquiring an exclusive lock at the data subject and mapping process level; if the acquisition is successful, acquiring a timestamp of the last successful execution of the task file of the execution task corresponding to the synchronization request; acquiring a mapping process definition file corresponding to the synchronization request, decompressing the mapping process definition file to obtain a mapping process definition, and acquiring an exclusive lock of the waiting queue at the mapping process level; processing the unprocessed execution task data in the waiting queue at the mapping process level; releasing the exclusive lock of the waiting queue at the mapping process level, forming a data exchange task execution completion log and recording, and then releasing the exclusive lock at the mapping process level; if the acquisition fails, acquiring an exclusive lock of the waiting queue at the mapping process level, inserting the unprocessed execution task data into the waiting queue at the level, releasing the exclusive lock of the waiting queue at the mapping process level, and forming a successful log of inserting the waiting queue and recording; in this embodiment, the mapping process definition file is an xml format file stored in a distributed file system.

[0041] The application realizes the data synchronization processing of different business systems by parsing the synchronization request information into data subjects and execution tasks, taking the synchronization request sent by the business party as the starting point and taking the successful synchronization execution of data as the termination point; the unified management and distribution of business data, the broadcast synchronization between multiple service module databases under the micro-service architecture, the multi-source data summary analysis, and the data processing function of the business modules based on the process are twisted, which realizes the data synchronization between different systems without affecting the business of each system, and reduces the coupling between each business system.

[0042] The above embodiments are preferred examples for implementing the present application, and the present application is not limited to the above embodiments. Any non-essential addition, replacement made by those skilled in the art according to the technical features of the technical solutions of the present application shall fall within the protection scope of the present application.

Claims

1. A data synchronization method supporting distributed deployment across multiple data source types, characterized in that, The synchronization request trigger mode comprises a real-time trigger mode, a timing trigger mode and a manual trigger mode, and the synchronization request processing mode comprises the following steps: The synchronization request processing mode comprises the following steps: The task executor obtains the data structure of the synchronization request from the message middleware, and forms a request parameter; According to the verification rule, if the request parameter is not legal, a failure log is recorded, and the synchronization request is ended; if the request parameter is legal, a mapping process definition file corresponding to the synchronization request is obtained according to the request parameter, and a general variable of the execution task corresponding to the synchronization request is queried according to the request parameter; An execution thread of the specific data mapping processing process is started; The execution thread comprises: judging whether the synchronization request needs to be delayed, if the synchronization request is not delayed or there is a non-delayed execution identifier, an immediate execution process is performed; if the synchronization request is delayed, a delayed execution process is performed; The delayed execution process comprises the following steps: An exclusive lock of a buffer interval waiting queue corresponding to the synchronization request is obtained; The request parameter corresponding to the synchronization request is saved in the corresponding buffer interval waiting queue set; The exclusive lock of the buffer interval level is released, a success log of inserting the buffer waiting queue is formed and recorded; All the tasks to be executed in the buffer interval waiting queue are obtained, a message is sent to the message middleware, and a non-delayed execution identifier is performed on all the tasks to be executed in the buffer interval waiting queue.

2. The method of claim 1, wherein, The immediate execution process comprises the following steps: An exclusive lock of the data body and the mapping process level is obtained; If the acquisition is successful, a timestamp of the last successful execution of the task file of the execution task corresponding to the synchronization request is obtained, a mapping process definition file corresponding to the synchronization request is obtained, the mapping process definition file is decompressed to obtain a mapping process definition, the mapping process definition is executed to obtain an exclusive lock of the waiting queue of the mapping process level, the task data of the unprocessed execution task in the waiting queue of the mapping process level is processed, the exclusive lock of the waiting queue of the mapping process level is released, a data exchange task execution completion log is formed and recorded, and then the exclusive lock of the mapping process level is released; If the acquisition fails, an exclusive lock of the waiting queue of the mapping process level is obtained, the task data of the unprocessed execution task is inserted into the waiting queue of the level, the exclusive lock of the waiting queue of the mapping process level is released, and an insertion waiting queue success log is formed and recorded.

3. The method of claim 1, wherein, The data structure comprises a data body number, a business system name, an exchange task number and an exchange task variable.

4. The method of claim 3, wherein, The request parameter comprises a data body number, a business system name and an exchange task number.

5. The method of claim 4, wherein, The verification rule comprises: detecting whether the data body number exists; whether the business system name is empty; whether the exchange task number exists.

6. The method of claim 1, wherein, The mapping process definition file is an xml format file and is stored in a distributed file system.

7. The method of claim 1, wherein, The timing trigger mode and the manual trigger mode are both based on embedding a task executor into a data processing service by a distributed task scheduling system, and processing a data source by the task executor.

8. The method of claim 1, wherein, The real-time trigger mode is based on real-time monitoring of data changes in a business system by an AOP technology, and processing data in the business system when detecting that data changes.

Images